This is a bit late, as OLS took place at the end of June, but I think it may be useful for people interested in knowing what's happening around debugging and low level OS kernel interaction.
The reprints can be found here
Many meetings, Many presentations.
Kris and I met with the systemtap people. IBM, Hitachi, Intel and RedHat. Had all hands meeting, to discuss roadmaps, and the like. Had also group dinner, as tradition.
For Systemtap we also had a meeting with IBM to discuss testing, and how to add the IBM test results for Systemtap for s390 and ppc to the Labrat system. Main outcome of that is that for now Kris will receive tarballs of the current test runs, and will parse them into the labrat webpage. Eventually we hope to solve the possible crash-the-system problems, by using Xen for Labrat. At that point we can run all the tests (make install-check) instead of just the "smoke" tests (makecheck).Will Cohen of Red Hat is also working on more testing, specifically adding test coverage information for the tapsets themselves (as opposed to coverage for the translator/systemtap itself).
Systemtap BOF
Vara Prasad of IBM, presented. General status of Systemtap, and discussions with audience. Main problem remain around the debugging information. Problems are twofolds and almost orthogonal:
- size of the debuginfo. Files are big, we really only read part of the info in anyway. Is there a way to shrink the size or a way to split in smaller chunks, that are related to all that a tapset needs. (I personally would not split things up too much, because then one needs to carry around the whole thing anyway for debugger usage).
- availability of the debuginfo. Customers using stap right now are running into this issue. Where is the kernel debuginfo, where to get it, it's cumbersome. Should there be a debuginfo server from which to get it. Problem with become much worse with the userspace probing. You need debugging info for all sorts of apps.
- Most apps that big enterprise customers might be using are closed source, and there is no debugging info around. for Instance Oracle is one of these apps). Is there any way to provide debug info?
Utrace/Uprobes talk
Jim Keniston of IBM. Very good overview talk (slides already sent to list). Nothing new here for me, but it was good to see most of the ptrace related stuff all in one place. There isn't such an overview around.
7 Dwarves talk: Arnaldo Carvalho de Melo
This was a bit misguided. The abstract was giving the impression that the talk was going to be about how to reingeneer Dwarf2 to make it better and smaller, but that wasn't the topic. Arnaldo wrote a bunch of utilities to gather information about the layout of structures and to rearrange the structures so that they are packed better with less space wasted. He has the utilities available for download onkernel.org:
The set of tools is called Pahole.
Kernel development overview: Greg Kroah-Hartman
I got there a bit late. This talk was a summary of who is doing what in the kernel community. Quite interesting, Greg charted all the patches that went into the 2.6.22 kernel into a giant graph, with an arch going from the patch author to each of the persons that signed off on the patch. Pretty big graph. He had that printed out and it was hung from the wall of the confroom, so people coud sign their name next to their node.
Anyway, he also ranked the top contributors by number of patches: Oracle is at 20th place with 365 patches for the current release. Top contributors are "unknown developers" (i.e: not clear who they work for), Red Hat # 2, Novell #3. Google, excluding Andrew Morton, hasonly 48 patches. Red Hat is the top $$ funding company (better said as employer) and the "consultants" group is at about 15th or so. Bottom tier are companies that we all know are modifying/using/selling Linux but are not contributing back.
Linux on Cell: Arnd Bergmann
This was a fun talk. It seems that the cell processor is improving, but it's still a huge beast, with many moving parts. To compile for it with gcc, you have to generate 2 types of .o files for the 2 types of cores and then the linker will put them together, for instance. Applications are limited to supercomputing and of course the gaming area with the PS3. This was interesting to me because of my previous work with gdb in embedded space.
Frysk: Andrew Cagney
This talk concentrated on the testing of Frysk. I.e. how frysk has been able to find kernel bugs (or better how kernel bugs manifest in debugging applications). Of course frysk pushes the boundaries set by gdb a bit harder, and the ptrace/utrace infrastructure runs into its own limitations. The Frysk tests include a set of kernel tests.
Talk was a bit disappointing because it concentrated on the Frysk command line and the GUI demo consistently didn't work.
Kris's Labrat system got mentioned in the "more info" slide at the end.
Asynch System Calls: Zach Brown
This was good and jaw dropping. In the sense that the new Syslets stuff is going to impact utrace/ptrace quite a lot. We need to synch up with Roland and talk. Read the paper.
DJProbes: Masami Hiramatsu and Satoshi Oshima of Hitachi
Description of DJprobes. DJprobes are now used to improve kprobes performance by 50% in certain cases on x86. Detailed description of the limitations of the mechanism in various cases was discussed. It seems to me that the limitations and the workarounds are getting a bit Byzantine, and that with all the workarounds implemented, the worst case scenarios cannot possibly be any faster than just using kprobes. One limitation is due to the instructions replaced by Jump being the target of other jumps, and how the boundaries of the instructions can be misaligned. Another limitation is due to concurrent execution of the same area by multiple threads. Some more problems occurred with self-modifying instructions. I kind of lost track of things after awhile, because the talk was a bit too detailed.
LSB BOF: Mats Winchmann
Just attended this one to catch up on what LSB was doing, having worked with them at previous employer. Not many big news, LSB 3.2 will contain some bugfixes and more tests.
- Missing interfaces will be added (about 45 of them)
- freetype, pft, xrender
- application testkit manager
- printing support
LSB 4.0 is still way out in the future, and include some but not necessarily all of the following items:
- runtime languages
- LSB for mobile devices (smaller footprint LSB version)
- library uplift (new versions of glibc, opengl, etc)
- cairo, dbus
- audio
- much increased test coverage (contracted out, so this will actually make it)
They have revamped the website and now it includes a database of all the interfaces and it's easily queryable. LSB is now and ISO standard, freely available for download.
Perfmon2 BOF: Stephane Eranian of HP
Ah this was interesting. It ended up lasting 2 hours, and we got kicked out of the room because they wanted to shut the place down for the night. Stephane is writing this unified framework for accessing performance monitors in the kernel for a variety of processors. HP, Intel, IBM, AMD are contributing their architecture dependent part. This Perfmon2 effort started in 2005 from and impromptu OLS bof, where it was decided that less fragmentation was needed, because there were 2 such mechanisms already and none of them was covering all the needs (perfmon and perfcounter). So 2 years later what has happened? The interface looks nice, and it's getting polished. Stephane hopes to have it for inclusion in the 2.6.24 kernel. He discussed also some of the problems he is facing, (now I don't remember all of them) one of which is getting access to the real counters from a virtualized environment. So he is facing real challenges. Unfortunately he is the main/only person working on this. He is going to set up a git repo onkernel.org, though and that hopefully will speed up the testing and encourage more participation (as opposed to publishing a patch once in a while). However there is still no sign of an in-kernel API. There is one for Userspace use. Justification is that there are no"customers" ready yet for an in kernel API. This may be bit debatable, since Mathieu (LTTng) is crying for one, and so are others. Maybe the first client of such interface could be the NMI watchdog on i386.
Branch tracing and test framework: Niro Yoshioka
This guy is from Miracle Linux, and the stuff he works on is on sourceforge:
Quite interesting. Maybe useful to look at it for our Labrat.
Another tracing BOF: Alan Stern and Jim Keniston
This was supposed to be a talk by Prasanna of IBM, but he wasn't able to come to Ottawa, so Alan Stern talked about the Hardware Breakpoint support he is working on. Basically another interface will be in place in the kernel to deal with all the architecture dependencies details of the hardware assisted breakpoints and watchpoints. Number of registers varies per processor, per architecture. Triggering also varies, and position of the IP varies also after the trap is taken from procesor to processr (ie sometimes it points after the trap instruction, sometimes it points at the trap instruction). But the same event will be presented to the user (gdb/frysk, etc) and all the details will be transparent and hidden within ptrace/utrace. Alan is actually working with Roland on this, to integrate it as part of utrace. Jim Keniston did the rest of the BOF, talking some more about utraceand uprobes. Nothing new there.
