Elena's Blog

This is a bit late, as OLS took place at the end of June, but I think it may be useful for people interested in knowing what's happening around debugging and low level OS kernel interaction. 

The reprints can be found here

Many meetings, Many presentations.

Kris and I met with the systemtap people. IBM, Hitachi, Intel and RedHat.  Had all hands meeting, to discuss roadmaps, and the like. Had also group dinner, as tradition.

For Systemtap we also had a meeting with IBM to discuss testing, and how to add the IBM test results for Systemtap for s390 and ppc to the Labrat system.  Main outcome of that is that for now Kris will receive tarballs of the current test runs, and will parse them into the labrat webpage. Eventually we hope to solve the possible crash-the-system problems, by using Xen for Labrat. At that point we can run all the tests (make install-check) instead of just the "smoke" tests (makecheck).Will Cohen of Red Hat is also working on more testing, specifically adding test coverage information for the tapsets themselves (as opposed to coverage for the translator/systemtap itself).

Systemtap BOF

Vara Prasad of IBM, presented. General status of Systemtap, and discussions with audience. Main problem remain around the debugging information.  Problems are twofolds and almost orthogonal:

• size of the debuginfo. Files are big, we really only read part of   the info in anyway. Is there a way to shrink the size or a way to   split in smaller chunks, that are related to all that a tapset   needs.  (I personally would not split things up too much, because   then one needs to carry around the whole thing anyway for debugger   usage).
• availability of the debuginfo. Customers using stap right now are   running into this issue. Where is the kernel debuginfo, where to   get it, it's cumbersome. Should there be a debuginfo server from   which to get it. Problem with become much worse with the userspace   probing. You need debugging info for all sorts of apps. 
• Most apps that big enterprise customers might be using are closed   source, and there is no debugging info around. for Instance Oracle   is one of these apps). Is there any way to provide debug info?

Utrace/Uprobes talk

Jim Keniston of IBM. Very good overview talk (slides already sent to list).  Nothing new here for me, but it was good to see most of the ptrace related stuff all in one place. There isn't such an overview around.

7 Dwarves talk: Arnaldo Carvalho de Melo

This was a bit misguided. The abstract was giving the impression that the talk was going to be about how to reingeneer Dwarf2 to make it better and smaller, but that wasn't the topic. Arnaldo wrote a bunch of utilities to gather information about the layout of structures and to rearrange the structures so that they are packed better with less space wasted. He has the utilities available for download onkernel.org:

• Git repository
• Summary

The set of tools is called Pahole.

Kernel development overview: Greg Kroah-Hartman

I got there a bit late. This talk was a summary of who is doing what in the kernel community. Quite interesting, Greg charted all the patches that went into the 2.6.22 kernel into a giant graph, with an arch going from the patch author to each of the persons that signed off on the patch. Pretty big graph. He had that printed out and it was hung from the wall of the confroom, so people coud sign their name next to their node.
Anyway, he also ranked the top contributors by number of patches: Oracle is at 20th place with 365 patches for the current release. Top contributors are "unknown developers" (i.e: not clear who they work for), Red Hat # 2, Novell #3. Google, excluding Andrew Morton, hasonly 48 patches.  Red Hat is the top $$ funding company (better said as employer) and the "consultants" group is at about 15th or so. Bottom tier are companies that we all know are modifying/using/selling Linux but are not contributing back.

Linux on Cell: Arnd Bergmann

This was a fun talk. It seems that the cell processor is improving, but it's still a huge beast, with many moving parts. To compile for it with gcc, you have to generate 2 types of .o files for the 2 types of cores and then the linker will put them together, for instance. Applications are limited to supercomputing and of course the gaming area with the PS3. This was interesting to me because of my previous work with gdb in embedded space.

Frysk: Andrew Cagney

This talk concentrated on the testing of Frysk. I.e. how frysk has been able to find kernel bugs (or better how kernel bugs manifest in debugging applications). Of course frysk pushes the boundaries set by gdb a bit harder, and the ptrace/utrace infrastructure runs into its own limitations. The Frysk tests include a set of kernel tests.
Talk was a bit disappointing because it concentrated on the Frysk command line and the GUI demo consistently didn't work.
Kris's Labrat system got mentioned in the "more info" slide at the end.

Asynch System Calls: Zach Brown

This was good and jaw dropping. In the sense that the new Syslets stuff is going to impact utrace/ptrace quite a lot. We need to synch up with Roland and talk. Read the paper.

DJProbes: Masami Hiramatsu and Satoshi Oshima of Hitachi

Description of DJprobes. DJprobes are now used to improve kprobes performance by 50% in certain cases on x86. Detailed description of the limitations of the mechanism in various cases was discussed.  It seems to me that the limitations and the workarounds are getting a bit Byzantine, and that with all the workarounds implemented, the worst case scenarios cannot possibly be any faster than just using kprobes. One limitation is due to the instructions replaced by Jump being the target of other jumps, and how the boundaries of the instructions can be misaligned. Another limitation is due to concurrent execution of the same area by multiple threads. Some more problems occurred with self-modifying instructions. I kind of lost track of things after awhile, because the talk was a bit too detailed.

LSB BOF: Mats Winchmann

Just attended this one to catch up on what LSB was doing, having worked with them at previous employer. Not many big news, LSB 3.2 will contain some bugfixes and more tests.

• Missing interfaces will be added (about 45 of them)
• freetype, pft, xrender
• application testkit manager
• printing support

LSB 4.0 is still way out in the future, and include some but not necessarily all of the following items:

• runtime languages
• LSB for mobile devices (smaller footprint LSB version)
• library uplift (new versions of glibc, opengl, etc)
• cairo, dbus
• audio
• much increased test coverage (contracted out, so this will actually make it)

They have revamped the website and now it includes a database of all the interfaces and it's easily queryable. LSB is now and ISO standard, freely available for download.

Perfmon2 BOF: Stephane Eranian of HP

Ah this was interesting. It ended up lasting 2 hours, and we got kicked out of the room because they wanted to shut the place down for the night. Stephane is writing this unified framework for accessing performance monitors in the kernel for a variety of processors.  HP, Intel, IBM, AMD are contributing their architecture dependent part. This Perfmon2 effort started in 2005 from and impromptu OLS bof, where it was decided that less fragmentation was needed, because there were 2 such mechanisms already and none of them was covering all the needs (perfmon and perfcounter). So 2 years later what has happened?  The interface looks nice, and it's getting polished. Stephane hopes to have it for inclusion in the 2.6.24 kernel. He discussed also some of the problems he is facing, (now I don't remember all of them) one of which is getting access to the real counters from a virtualized environment. So he is facing real challenges. Unfortunately he is the main/only person working on this. He is going to set up a git repo onkernel.org, though and that hopefully will speed up the testing and encourage more participation (as opposed to publishing a patch once in a while). However there is still no sign of an in-kernel API. There is one for Userspace use. Justification is that there are no"customers" ready yet for an in kernel API. This may be bit debatable, since Mathieu (LTTng) is crying for one, and so are others. Maybe the first client of such interface could be the NMI watchdog on i386.

Branch tracing and test framework: Niro Yoshioka

This guy is from Miracle Linux, and the stuff he works on is on sourceforge:

• btrax
• crackerjack

Quite interesting. Maybe useful to look at it for our Labrat.

Another tracing BOF: Alan Stern and Jim Keniston

This was supposed to be a talk by Prasanna of IBM, but he wasn't able to come to Ottawa, so Alan Stern talked about the Hardware Breakpoint support he is working on. Basically another interface will be in place in the kernel to deal with all the architecture dependencies details of the hardware assisted breakpoints and watchpoints. Number of registers varies per processor, per architecture. Triggering also varies, and position of the IP varies also after the trap is taken from procesor to processr (ie sometimes it points after the trap instruction, sometimes it points at the trap instruction). But the same event will be presented to the user (gdb/frysk, etc) and all the details will be transparent and hidden within ptrace/utrace. Alan is actually working with Roland on this, to integrate it as part of utrace. Jim Keniston did the rest of the BOF, talking some more about utraceand uprobes. Nothing new there.

Let's talk a bit about Oracle Open World. This took place November 11 through November 15 2007, in San Francisco. It is a giant expo, taking place in all the buildings of the Moscone Center with overspill in adjacent areas and hotels. I was staffing the Unbreakable Linux and the Oracle VM booths. It was incredibly busy, every day. The announcements about OVM and the demos of the OVM with live migration given in the various keynotes contributed noticeably to increase booth traffic as well. You can see the daily news from the conference here

I also gave an introductory talk about Systemtap within the developer track. There was very good turnout, and lots of questions. I have posted the slides in PDF format here.

It's almost time for the Linux Foundation Collaboration Summit. It's taking place in San Francisco, April 8-10. The agenda is pretty much finalized, you can look it up here: http://events.linuxfoundation.org/events/collaboration-summit/agenda/
There will be a session on tracing on Thursday, where we'll get an overview of the active projects such as lttng, systemtap, utrace, ftrace, uprobes, and others. Hopefully the meeting will serve as a forum to iron out how the various pieces play together, and how to better integrate them with the Linux kernel. It will be surely an interesting meeting.

The first LinuxCon conference will take place September 21 to 23 in Portland Oregon, co-located with the Linux Plumbers Conference. The speaker list and the schedule were announced last week. It will have a variety of topics, and an excellent lineup of panels and keynotes.
Check the schedule here:
http://events.linuxfoundation.org/events/linuxcon/speakers

The next End User Summit event has been scheduled for November 9 and 10 2009 in Jersey City. Take a look at the website. More details to appear there soon.

http://events.linuxfoundation.org/events/end-user-summit

Two weeks ago I attended the Linux Symposium in Montreal.
I was there mostly to participate in the tracing track, and to discuss the current state of kernel tracing infrastructure and the other projects, determining what has been accomplished since the meetings we had at the Collaboration Summit in April. Some of the key players were not at the conference however, and the discussion was a bit limited. Some of the goals that were set at the Collab Summit are still lagging a bit. On the utrace side, Frank is working on the gdb stub to talk to utrace. This is built on top of kernels with utrace integrated (RHEL and Fedora) and uses the gdb remote protocol (in usual gdb fashion) to communicate with the target process being debugged under utrace control. His latest prototype is here:
http://sourceware.org/ml/systemtap/2009-q3/msg00045.html

Ptrace cleanups: Roland and Oleg are now official ptrace kernel maintainers (since April 2009). Oleg Nesterov is making progress on this, but I am not seeing any recent posts.
I see that there was a wave of ptrace cleanup patches at the end of May from Oleg (http://lkml.org/lkml/2009/5/30/229), but I cannot see that they went in. As of last week, there were no comments on them at all. It seems we are stuck in the architecture specific cleanups for ptrace (see this thread: http://lkml.org/lkml/2009/5/3/109).

Utrace: I've seen no major changes, but just this week, Oleg and Roland started looking at the utrace implementation a bit more closely, as you can see from the utrace-devel mailing list. Of course it's all depending on the ptrace cleanups getting into the kernel.

ftrace and markers: Steve Rostedt wasn't there, and Mathieu has been busy with his dissertation. Frederic Weisbecker gave a presentation, it was an introduction to ftrace.

Systemtap: Frank gave a talk on systemtap, an overall introduction and overview.

There was a panel where people talked about their projects, not much talk unfortunately around how to make progress in the overall area. It was mostly a showcase of each project.

Many people from the Montreal area talked about Eclipse. These folks work for Ericsson which uses Eclipse a lot.

The slides from the tracing mini-summits are here:
http://ltt.polymtl.ca/tracingwiki/index.php/TracingMiniSummit2009

The first day I attended a tutorial given by IBM, about Performance Inspector.
It's a decent performance tool, sharing many things with Oprofile, plus a gui. It turns out the project is an old internal IBM project, that got open sourced much later. One of the people working on it was the author of Oprofile.

I attended Martin and Dan talks as well. I think they were very good ones. Honestly, they had some real meat to them, and some new ideas. They also were engaging speakers, and didn't put the audience to sleep. I also attended Ric Wheeler's talk and that was interesting, and a nice high level summary.
Dan's Transcendent Memory slides are here:
http://oss.oracle.com/projects/tmem/documentation/presentations/
and his work is here:
http://oss.oracle.com/projects/tmem
There is a good article on Tmem here: http://lwn.net/Articles/340080/

Jon Corbet did his usual kernel report, and mentioned Oracle a lot. That was nice to see.

A talk about autotest from Google was OK. They started from Martin Bligh's system and expanded on that, the test driver mechanism is getting more sophisticated and operates on many different machines, and layers of servers.

On November 9 and 10 I attended the Linux Foundation End User Summit in Jersey City. Ted Tso and I organized a mini session about tracing in Linux. We had Steve Rostedt talking about ftrace, and Subendhu Ghosh from RedHat talk about the perf events subystem. The session was on Tuesday afternoon, and I'd say that aside from Christoph Hellwig talk about KVM it was the most crowded session. It was nice that both Jon Corbet in his Kernel Report talk that morning, and James Bottomley in his panel the day before mentioned tracing and our session as something "hot" to follow. The Ftrace talk consited of an introduction to Ftrace, and a few demos. The audience was mostly of Wall Street type people, who are still using old versions of enterprise products, so the don't really know what has happened in the meantime to the upstream. Ftrace has been around for a while, a couple of years, but perf events is newer, and many people didn't know what it was. There was a little demo of that as well. While talking about perf events, somebody (hch, IIRC) mentioned the new tool that Arjan built on top of perf events, called "timechart". It's a tool that spits out a graphical view of the time your system is spending doing various things. http://blog.fenrus.org/?p=5.
Perf events is in Fedora 12 and so is ftrace. The timechart thing is only in mainline now.
Slides from Steve's talk are on Steve's page: http://people.redhat.com/srostedt

Besides this tracing session there was a talk done by Sergio Leunissen from my same Oracle Linux team, on OLT (Oracle Linux Testkit) and Validated Configurations and how that is helping us stabilize OEL for enterprise loads. The talk went well, the audience asked lots of questions about OLT and VC. Also were happy to see the oracle-validated RPM that we ship.

The other talk I attended on the second day was the filesystem recap by Ric Wheeler. Nothing new here, just a summary of all the things that have happened to the various FS in linux during the last year or so. It was nice that Ric mentioned Oracle and the FS folks by name (thanks Ric! ). So Chris, Martin, Jens, Chuck got honorable mentions. In general the amount of work that Oracle is doing in this space is still a bit unknown, since several people in the audience expressed surprise to see Oracle so involved in Linux.

On the first day there were no deep technical talk or separate sessions. Of the plenary sessions, an interesting one was the one by IDC about the Linux market share. While Windows keeps dominating the server side, Linux is increasing, and apparently Unix is not going away any time soon. The investment in Unix seems to still be fairly solid.

I also attended a session discussing how kernel interfaces should keep up with the times. In particular as examples there was mention of SSDs and SSSs which are going to be mainstream soon. Some complaint about epoll. Also mentioned were the interfaces for sending messages to multiple recipients, with suggestions that the calls there should be vectorized to speed up processing of messages. It's interesting that a very similar debate was going on almost simultaneously on the linux-raid mailing list, here.
Jens Axboe, also in the Oracle Linux Team, has done some work on per-bdi writeback threads, see his slides from the Linux Plumber's conference here.

Another good overview from the NYSE (Brian Clark) talked about what they focus on in their IT infrastructure.

I think this is about it.