Drobo and Linux
By andy.grover on Apr 24, 2009
The Path to Drobo
I just got a Drobo to replace an aging NAS appliance. (Note I am not a storage developer, either at the filesystem or block level, so I am just talking about this as a guy who needs to store stuff safely)
When the Drobo first came out, I was not that impressed. It was like an external USB-attached hard drive, but you could hotswap drives? It seemed overpriced. But, I took another look at it recently because we want to reduce the number of machines in our home network, for power, noise, and simplicity's sake.
Our mythtv box has to be on all the time anyways, and because our NAS box is not one of the current generation that contains a wealth of services, the myth box has already been doing a lot of mounting NAS shares and re-exporting services (e.g. iTunes) anyway. So by switching to the direct-attached Drobo, not too much changes (except the myth box becomes a Samba server, which is is way better at than the NAS's puny CPU!). This eliminates one loud machine that never spun down its disks in favor of a faster, quieter, more flexible one. Or at least that was the plan.
Considering the Drobo
Drobo has a number of nice things about it that I like:
- Storage is redundant, can use different sizes of disk, and new disk capacity is used seamlessly (i.e. don't have to break the RAID5 array)
- Takes bare 3.5" SATA disks, no hotswap caddies needed. Why aren't other companies doing this?
- USB and Firewire 800 (400 works too)
- Shorter times to re-establish redundancy on a disk removal/failure due to being filesystem-aware
- Disks spin down, and it's generally quiet
- Supports ext3 and Linux (in development)
- A speed upgrade from what I had (18MB/sec vs old NAS's 8MB/sec)
- The Drobo user forum requires Drobo serial number (i.e. customers only) to access. This is a pretty uncommon restriction and a bit worrisome to prospective customers.
- Drobo doesn't appear to report capacity above 2TB. READ_CAPACITY(16) fails? Maybe this a Linux or FW issue, because larger LUNs work on other OSes from what I hear.
- I configured Drobo with a 2TB LUN. Once I add enough HD capacity to pass that threshold, it looks like I'm back into less-fun, more complicated scenarios to use its full space. This could involve either a destructive reformat of the Drobo, or multiple LUNs.
- Linux support is still under development. So now I'm becoming a drobo-utils contributor, which looks like it will be fun to hack, but really I'd rather not have to write code to get the hw I paid for to work on Linux.
- The flipside of being FS-aware: can't do its magic (skip relay-out of empty space, do proper storage virtualization) on anything but ext3, NTFS, HFS+, and FAT32.
- The OS thinks Drobo is larger than it actually is. One must refer to lights on the front (or drobo-utils) to determine actual space remaining. This means there will be issues using Drobo for apps like MythTV that want to fill the volume and then expire the oldest data.
It didn't go as smoothly as I hoped, but in the end it wasn't the Drobo's fault.
Basically the first three adapters I used to connect it caused varying degrees of flakiness. First I tried the mythtv's built-in 1394a connector. It formatted the Drobo ok, but I was seeing errors in dmesg. So I put in an additional 1394 card. That worked better, but there were long pauses, and throughput was averaging to 3MB/sec. iotop showed bursts of 30MB/sec activity followed by long stretches of nothing.
What was going on? Nothing in dmesg. drobo-utils diagnostics was broken so it was no help. Was it's little ARM brain stressed? Impossible. Benchmarks on the web were in the 15MB/sec range. Before giving up, I tried the USB connection, and was gratified to see a much more consistent 18MB/sec rate.
Problem solved, (I thought) I proceeded to start copying all my stuff onto the Drobo. After a few minutes it would start getting USB errors and vanish.
One last thing to try: pull out the known-good, based-on-TI-chipset FW card from my desktop and try that. That did it, both copying a 2GB iso and bonnie++ ran with no issues. All my problems were caused by flaky 1394 or USB adapters.
So lessons learned. Make sure Drobo-connected 1394 card has TI chipset (both mobo and 1st 1394 card had older VIA chips on them). Motherboard USB can also be flaky. Drobo seems to have an above-average sensitivity to these issues.
I'm really glad I was able to get it sorted because I really couldn't see any device, even more expensive ones, that fulfilled my desire for easy-to-use, non-destructively-expandable, redundant filesystem.
How Drobo Works
Much of Drobo's unique featureset is made possible by combining filesystem-awareness with block-awareness. Drobo, as a block device, is just getting commands like "write this 512-byte chunk of data to sector X". Normal block devices have no knowledge about what is in the data to write. Drobo does.
The first thing it must do is sparsely map the sectors in its 2TB virtual LUN to the actual sectors it has available on its disks. It only has to allocate a real sector whenever the OS's filesystem code writes to a given sector. This is equivalent to what VMWare and VirtualBox do when you allocate a dynamically-expanding disk image. At this level, the actual storage required for a large virtual volume will start small and grow. It will never shrink, because even if a sector contains data for a file that is deleted, the sector has still been written-to, and there's no way for the block layer to know it can reclaim those now-unused sectors.
The second thing Drobo does is it looks inside the block-based data it's being sent to decode the blocks that the filesystem is using for its metadata. For example, if a large file is deleted on an ext3 filesystem, there are updates to the filesystem's on-disk structures to record this. By noticing those metadata block writes, Drobo can determine when the filesystem is no longer referencing a bunch of other sectors. Now Drobo has the correct map of sectors the filesystem is using, instead of all the sectors the filesystem has ever touched.
The benefit of this should be clear to anyone who has ever inserted a new disk into a RAID array and had to wait a whole day for it to be ready. This takes the same amount of time no matter how much data is on the array's volumes, because it is ensuring every single sector is mirrored. Drobo knows how much data you actually have, so its sync times are based on how much data is in the filesystem, not how long it takes to write to every sector. Kinda nice, BUT I'd imagine a filesystem-aware block device makes filesystem developers a little queasy.
What about new filesystems? BtrFS? Ext4? ZFS? You can't put these on a Drobo and have it work right.
Drobo and TRIM
How can Drobo handle these new filesystems? One way is for a firmware update to add support for them, like the ones it currently supports. However...
At the most recent Linux Storage and Filesystem Workshop, there was some discussion about a TRIM command, to tell a block device the filesystem is not using a region anymore. This helps SSDs manage themselves better, and also could help Drobo and others:
From there, the discussion went to the seemingly unrelated topic of "thin provisioning." This is an offering from certain large storage array vendors; they will sell an array which claims to be much larger than the amount of storage actually installed. When the available space gets low, the customer can buy more drives from the vendor. Meanwhile, from the point of view of the system, the (apparently) large array has never changed.Drobo's maker Data Robotics is probably not one of the "large" vendors referred to here, but just like for the big guys' arrays, Drobo could cease its layer-violating-yet-awesome hackery if a) new filesystems used TRIM and b) Drobo understood what it meant. No doubt Drobo would want to keep its current code in place for its use with TRIM-unaware systems, but I think a TRIM-aware BtrFS on a TRIM-aware Drobo would be great -- let Drobo do its magic but still keep that nice clean filesystem and block separation.