Linux 'n Stuff

GSOC Mentoree interviewed

2009-08-28T19:05:56Z

My GSOC 2009 mentoree, Kanika Vats, shares some thoughts on her experiences, and tech careers for women.

I think her take on open source technology and its community is incredibly positive. I have no doubt at some point the bloom will come off the rose a bit, but this is certainly a good first impression for a newcomer to have, and hopefully will make it easier to put future negativity in perspective.

I was the sole non-female in the Systers development effort to customize Mailman to their community's requirements. (I was recruited by my SO, Jen, who was also a mentor as well as overall organizer for Systers' GSOC effort.) As a first-time mentor, this was an extremely rewarding experience.

First, it made clear that being a mentor/teacher/manager-type was work, even though someone else was doing the heavy lifting. It really helps to know what you're talking about, which I didn't always, such as giving advice on how to use the Storm ORM, when my experience was with SQLObject. (An ORM's an ORM, right?)

Second, working with a new contributor was a chance to give exposure to the things that school doesn't seem to teach but that are very important in the real world, and especially around open-source development: Coding standards; Understanding and enhancing other people's code, and writing code so that other people can understand it; Using source control and branches to help one's productivity; bug tracking; and online collaboration, especially across time zones.

Finally, I will have the opportunity to attend the annual Grace Hopper Women in Computing conference, where I am led to believe people of my gender will be in the extreme minority. Will it be weird? Too early to say -- although I have heard the conference schwag may include soaps and candles instead of 2XL-sized t-shirts. Should be interesting!

RHEL and Emacs 23

2009-08-17T05:41:48Z

Dear Lazyweb,

Is there a RPM (or repo) available that has Emacs 23 for RHEL5?

Unfortunately the link to Chip Coldwell's RPM is now dead.

Thanks -- Andy

Drobo and Linux

2009-04-24T23:13:10Z

The Path to Drobo

I just got a Drobo to replace an aging NAS appliance. (Note I am not a storage developer, either at the filesystem or block level, so I am just talking about this as a guy who needs to store stuff safely)

When the Drobo first came out, I was not that impressed. It was like an external USB-attached hard drive, but you could hotswap drives? It seemed overpriced. But, I took another look at it recently because we want to reduce the number of machines in our home network, for power, noise, and simplicity's sake.

Our mythtv box has to be on all the time anyways, and because our NAS box is not one of the current generation that contains a wealth of services, the myth box has already been doing a lot of mounting NAS shares and re-exporting services (e.g. iTunes) anyway. So by switching to the direct-attached Drobo, not too much changes (except the myth box becomes a Samba server, which is is way better at than the NAS's puny CPU!). This eliminates one loud machine that never spun down its disks in favor of a faster, quieter, more flexible one. Or at least that was the plan.

Considering the Drobo

Drobo has a number of nice things about it that I like:

Storage is redundant, can use different sizes of disk, and new disk capacity is used seamlessly (i.e. don't have to break the RAID5 array)
Takes bare 3.5" SATA disks, no hotswap caddies needed. Why aren't other companies doing this?
USB and Firewire 800 (400 works too)
Shorter times to re-establish redundancy on a disk removal/failure due to being filesystem-aware
Disks spin down, and it's generally quiet
Supports ext3 and Linux (in development)
A speed upgrade from what I had (18MB/sec vs old NAS's 8MB/sec)

It also has some things I don't like:

The Drobo user forum requires Drobo serial number (i.e. customers only) to access. This is a pretty uncommon restriction and a bit worrisome to prospective customers.
Drobo doesn't appear to report capacity above 2TB. READ_CAPACITY(16) fails? Maybe this a Linux or FW issue, because larger LUNs work on other OSes from what I hear.
I configured Drobo with a 2TB LUN. Once I add enough HD capacity to pass that threshold, it looks like I'm back into less-fun, more complicated scenarios to use its full space. This could involve either a destructive reformat of the Drobo, or multiple LUNs.
Linux support is still under development. So now I'm becoming a drobo-utils contributor, which looks like it will be fun to hack, but really I'd rather not have to write code to get the hw I paid for to work on Linux.
The flipside of being FS-aware: can't do its magic (skip relay-out of empty space, do proper storage virtualization) on anything but ext3, NTFS, HFS+, and FAT32.
The OS thinks Drobo is larger than it actually is. One must refer to lights on the front (or drobo-utils) to determine actual space remaining. This means there will be issues using Drobo for apps like MythTV that want to fill the volume and then expire the oldest data.

Installing Drobo

It didn't go as smoothly as I hoped, but in the end it wasn't the Drobo's fault.

Basically the first three adapters I used to connect it caused varying degrees of flakiness. First I tried the mythtv's built-in 1394a connector. It formatted the Drobo ok, but I was seeing errors in dmesg. So I put in an additional 1394 card. That worked better, but there were long pauses, and throughput was averaging to 3MB/sec. iotop showed bursts of 30MB/sec activity followed by long stretches of nothing.

What was going on? Nothing in dmesg. drobo-utils diagnostics was broken so it was no help. Was it's little ARM brain stressed? Impossible. Benchmarks on the web were in the 15MB/sec range. Before giving up, I tried the USB connection, and was gratified to see a much more consistent 18MB/sec rate.

Problem solved, (I thought) I proceeded to start copying all my stuff onto the Drobo. After a few minutes it would start getting USB errors and vanish.

ARRG.

One last thing to try: pull out the known-good, based-on-TI-chipset FW card from my desktop and try that. That did it, both copying a 2GB iso and bonnie++ ran with no issues. All my problems were caused by flaky 1394 or USB adapters.

So lessons learned. Make sure Drobo-connected 1394 card has TI chipset (both mobo and 1st 1394 card had older VIA chips on them). Motherboard USB can also be flaky. Drobo seems to have an above-average sensitivity to these issues.

I'm really glad I was able to get it sorted because I really couldn't see any device, even more expensive ones, that fulfilled my desire for easy-to-use, non-destructively-expandable, redundant filesystem.

How Drobo Works

Much of Drobo's unique featureset is made possible by combining filesystem-awareness with block-awareness. Drobo, as a block device, is just getting commands like "write this 512-byte chunk of data to sector X". Normal block devices have no knowledge about what is in the data to write. Drobo does.

The first thing it must do is sparsely map the sectors in its 2TB virtual LUN to the actual sectors it has available on its disks. It only has to allocate a real sector whenever the OS's filesystem code writes to a given sector. This is equivalent to what VMWare and VirtualBox do when you allocate a dynamically-expanding disk image. At this level, the actual storage required for a large virtual volume will start small and grow. It will never shrink, because even if a sector contains data for a file that is deleted, the sector has still been written-to, and there's no way for the block layer to know it can reclaim those now-unused sectors.

The second thing Drobo does is it looks inside the block-based data it's being sent to decode the blocks that the filesystem is using for its metadata. For example, if a large file is deleted on an ext3 filesystem, there are updates to the filesystem's on-disk structures to record this. By noticing those metadata block writes, Drobo can determine when the filesystem is no longer referencing a bunch of other sectors. Now Drobo has the correct map of sectors the filesystem is using, instead of all the sectors the filesystem has ever touched.

The benefit of this should be clear to anyone who has ever inserted a new disk into a RAID array and had to wait a whole day for it to be ready. This takes the same amount of time no matter how much data is on the array's volumes, because it is ensuring every single sector is mirrored. Drobo knows how much data you actually have, so its sync times are based on how much data is in the filesystem, not how long it takes to write to every sector. Kinda nice, BUT I'd imagine a filesystem-aware block device makes filesystem developers a little queasy.

What about new filesystems? BtrFS? Ext4? ZFS? You can't put these on a Drobo and have it work right.

Drobo and TRIM

How can Drobo handle these new filesystems? One way is for a firmware update to add support for them, like the ones it currently supports. However...

At the most recent Linux Storage and Filesystem Workshop, there was some discussion about a TRIM command, to tell a block device the filesystem is not using a region anymore. This helps SSDs manage themselves better, and also could help Drobo and others:

From there, the discussion went to the seemingly unrelated topic of "thin provisioning." This is an offering from certain large storage array vendors; they will sell an array which claims to be much larger than the amount of storage actually installed. When the available space gets low, the customer can buy more drives from the vendor. Meanwhile, from the point of view of the system, the (apparently) large array has never changed.

Drobo's maker Data Robotics is probably not one of the "large" vendors referred to here, but just like for the big guys' arrays, Drobo could cease its layer-violating-yet-awesome hackery if a) new filesystems used TRIM and b) Drobo understood what it meant. No doubt Drobo would want to keep its current code in place for its use with TRIM-unaware systems, but I think a TRIM-aware BtrFS on a TRIM-aware Drobo would be great -- let Drobo do its magic but still keep that nice clean filesystem and block separation.

Converged fabric does not mean TCP/IP everywhere

2009-03-25T06:33:43Z

While I was not able to attend the OFA workshop in person this year, I have been making use of the provided teleservices to follow it.

The big issue this year is whether it's preferable to have the RDMA protocol used in a datacenter over Ethernet be encapsulated in TCP/IP, or just on Ethernet itself. While the trend over the past 20 years has been to use TCP/IP even for LAN traffic, the reasons for this may not apply to the two possible examples of going the other way, namely Fibre Channel over Ethernet (FCoE) and what was called RDMA over Converged Enhanced Ethernet (RoCEE or RoE)

Both FCoE and RoE have TCP/IP-based work-alikes -- iSCSI and iWARP respectively. The latter technologies face the problem that processing received packets cannot be made to be efficient! People have tried. A lot. Netchannels anyone? TCP's functionality (reordering, demultiplexing, etc) means that each packet is going to get copied in its entirety at least once, in addition to the NIC's copy. You can try to coalesce this by moving the TCP stack into the NIC with a TOE, but that sucks for about 14 reasons. You can try to hide it with an asynchronously-operating DMA engine to do the copy for you, but that has issues of its own. Finally, you can add a custom TCP stack in the kernel for your special protocol. This is a flagrant layering violation and the Linux netdev crew will also put a hit out on you.

Encapsulating in only Ethernet makes things a lot more palatable, with the two caveats that you have to do without the niceties that TCP gives you (congestion control) as well as IP (routing). In a datacenter you don't need routing, and if CEE does congestion control then you're covered. It is now possible to sanely build an adapter that can do 0-copy FCoE and RoE -- just bolt your IB^H^HRDMA macrocell and your FC macrocell next to the regular stuff on your Ethernet silicon and divert to them based on ethertype. send/recv rings all look kinda the same, don't they? Your card now shows up as three separate devices, and all three (fc, rdma, net) perform at their max efficiency.

Internet protocols have pushed far into the LAN/datacenter environment, displacing almost all equivalent LAN protocols. TCP/IP's ubiquity is a virtue above almost all others, and that same virtue is also behind the drive towards Ethernet as a common fabric. But performance and clean implementation outweigh ubiquity. Convergence on TCP/IP in the datacenter is not the big win, it's a nice-to-have. iWARP had potential downsides that people thought could be worked around...but the workarounds needed workarounds and now costs totally outweigh its benefits. The big win is converged Ethernet, not converged TCP, and I think that's what the debate should not lose sight of.

Filesystem testing

2009-03-05T03:36:22Z

SSD’s, Journaling, and noatime/relatime | Thoughts by Ted

The chief mechanism for benchmarking Linux filesystems seems to be operations on the Linux kernel's source tree.

This seems to me a dangerous over-optimization for kernel developer-centric usage. What if people want to put files other than the Linux kernel source on their disk? Other use cases are completely untested!

OK I'm being facetious.

Dumb questions: Are snapshotted filesystems like btrfs, and version control systems like git two things that could converge, or do they just look like they could converge and really never will?

RDS queued for Linux kernel inclusion!

2009-02-28T03:15:24Z

RDS is queued to be included in Linux 2.6.30.

RDS is a protocol initially designed to enable Oracle RAC clustered database nodes to exchange datagrams rapidly and reliably. It has been developed and maintained out-of-tree for a number of years.

With this will come all the usual benefits that GregKH has enumerated many times: more eyes fixing bugs, a broader spectrum of users, fewer compatibility hacks, and it's the Right Thing to do if at all possible, natch.

One interesting angle is that Linux already has a reliable-datagram protocol, SCTP, and of course it's also not too hard to implement reliable datagrams on top of TCP, so why another? RDS has grown from its roots as a "thin sockets layer over Infiniband" to have its own set of distinct and unique features tailored to its particular niche.

Working on RDS has been a very positive experience for me. It's been the right balance of exposure to new stuff and also a return to the world of Linux networking development, which I find rewarding. There's still much to do to achieve further performance gains and higher code quality, so still plenty to keep me busy.

A good week.

I think git just won

2009-01-14T19:06:57Z

Qt is moving to LGPL and a more open development process.

The article notes:

In addition to adopting the LGPL license for Qt, Nokia will also be completely changing Qt's development model to make it more inclusive and transparent. The source code will be moved to a publicly-accessible Git repository so that the latest changes will always be visible. The use of Git, a distributed version control system, will make it easier for third-party developers to participate directly in the process of improving Qt. To further reduce the barrier to participation, Nokia plans to accept code from contributors without requiring copyright assignment.

Aside from how great this is for developers using Qt, I think this pretty much indicates the DSCM race has been won. Aside from the kernel and X using Git (and Rails), both KDE and Gnome camps appear to be moving towards Git.

The one thing that Git still seems to be lacking is a nice libgit library for applications to access it. The common approach for language wrappers for Git in Ruby and Python is to shell out to the git executable to do stuff. It works, but yeesh, cmon. I want git embedded everywhere. I want Firefox to use git to store my bookmarks. That's not gonna happen until other applications can easily use Git's functionality.

Adventures in Netlink

2008-12-04T07:55:27Z

Ever spent hours writing code and then found out it's not needed? I just had the pleasure. I wrote some code using netlink (via libnl) that takes an interface name ("eth0") and returns the ipv4 addresses associated with it, but the bug I thought I was fixing was not there, so I'm making a blog entry out of it :)

Libnl has documentation. Autogenerated documentation -- which is better than nothing, but it's really an API reference, not a tutorial. Of course these days everything is in a git repo so I was able to muddle through by looking at the source, but of course this shouldn't be needed, in an ideal world...

First, include some headers. This assumes you have libnl-devel RPM installed, and don't forget to link against libnl with "-lnl":

#include #include

Here's the main function:
static uint32_t parse_iface(char *ptr) { struct nl_handle *sock; struct nl_cache *link_cache; struct nl_cache *addr_cache; int ifindex; uint32_t result = 0; struct rtnl_addr *addr; sock = nl_handle_alloc(); nl_connect(sock, NETLINK_ROUTE); link_cache = rtnl_link_alloc_cache(sock); addr_cache = rtnl_addr_alloc_cache(sock); ifindex = rtnl_link_name2i(link_cache, ptr); addr = rtnl_addr_alloc(); rtnl_addr_set_family(addr, AF_INET); rtnl_addr_set_ifindex(addr, ifindex); nl_cache_foreach_filter(addr_cache, (struct nl_object *)addr, nl_cache_callback, &result); rtnl_addr_put(addr); return ntohl(result); }
This code is connecting to the netlink socket to receive route-related information. The rtnl_*_alloc_cache() calls basically fill those caches with all the information from the kernel related to links (aka net interfaces) and addresses (all of them, whether ipv4, ipv6, appletalk, or whatever). So now all the info we need is in our address space, and we just need to sift through it!

Luckily, we can cross-reference the two separate caches because ifindex refers to the same interface in either cache. The code then gets the ifindex for the passed-in name (e.g. if ptr was "eth0").

The next part was kind of weird, due to being so flexible that simple things like what I wanted to do become nonobvious. The way you filter a cache is by creating an instance of the thing in the cache, and then only filling in the fields that you want to match the results. That's what the next few lines do, alloc an addr and fill in the limiters I wanted: ifindex (so I just get addresses for the one interface) and the family (limit to ipv4 addresses only).

libnl is designed to handle multiple returns for everything. This isn't dumb -- an interface can still have more than 1 ipv4 addresses that will get returned, so it has a callback interface. You give it a function and it calls it once for each result. I named my function nl_cache_callback, let's take a look at it:
static void nl_cache_callback(struct nl_object *obj, void *arg) { struct nl_addr *addr; uint32_t *ipv4_addr = arg; /* only get addr in 1st callback */ if (*ipv4_addr) return; addr = rtnl_addr_get_local((struct rtnl_addr *)obj); *ipv4_addr = *(uint32_t *)nl_addr_get_binary_addr(addr); }
Like I said, this could be called more than once but I really just want the first one, so I am passing an initially-zero variable in the context argument, and if it's already set I bail out. The last wrinkle is that a struct rtnl_addr can hold a variety of different address types, so if I didn't already know I'd asked for only 32-bit ipv4 addresses, I'd have to use the accessor methods to find out the address's length and type. But since I do, I just copy the addr's 4 bytes into my 4 bytes, and that's it. Back in the first function, it's a quick ntohl(ipv4_addr)[1] and Bob's your uncle[2]!

Have fun!

[1] libnl also has a helper function that fills in a sockaddr directly, pretty nice
[2] I actually have an uncle named Bob! Your results may vary.

Hello!

2008-10-08T18:27:32Z

Hi, I'm Andy Grover, and I'm a Linux kernel developer. I'm currently working on a network protocol called RDS, which is used (mostly on Infiniband today) for Oracle RAC IPC messages as well as by the recently-announced Exadata technology. RDS is poised for wider adoption, so I hope to talk more about its growth in the future, as well as technical issues around it, and Linux in general.

A tip: Sometimes enterprise distros can be a little short on having all the packages you need, if you're using them for development instead of actually serving anything. For example, git is the source control tool of choice for most people doing Linux kernel-related work. RHEL and OEL don't have it. However, the EPEL (Extra Packages for Enterprise Linux) repository does! It's run by the Fedora project and does a good job of providing a way to have the stability of an enterprise distro, and the package selection of a community distro.