A bit late, but nevertheless...
These are some notes I took at the Collab Summit in San Francisco. I posted these notes already on the systemtap mailing list, but I thought they are worth of a blog entry.
Some of the slides, not all of them, are here: http://events.linuxfoundation.org/slides
Our tracing session went well, I think it was most useful for the developers themselves, it was a
bit technical.
The first day of the summit was all dedicated to panels and presentations to the whole audience. It was not broken down into sessions. One of the presentations was by Edward Screven about Oracle and Linux. I thought it went well. It really showed how we use Linux internally and it explained why we got into the Linux business. Also talked about upstream contributions.
Another highlight of the first day was a panel with Jim Zemlin (the LF director), Ian Murdock of Sun, and Sam Ramji from Microsoft. It was entertaining to watch how the Microsoft person kept trying to not say anything that could be quoted by journalists (he said that himself). Quite an achievement, actually.
Detailed blogs have been posted by Gerrit Huizenga of IBM: http://gh-linux.blogspot.com/
The important stuff that for me happened at the summit was the meetings we had with the tracing people and some kernel folks. It started on Thursday, after the presentation were over, and we reached some consensus on the tracing infrastructure that is going into the kernel. (Markers, utrace, uprobes). The discussions continued on Friday and we talked more about higher level things, like the debuginfo size problem and the systemtap integration with the kernel. Detailed notes below.
Overall it was a good conference, or at least, let's say that I was very happy with the conclusions and the agreement we reached about tracing. Of course not everything is decided, but it was a good start. It seems that the tracing problem is really getting attention from the kernel community. There is going to be a tracing mini-summit at the Linux Symposium in Montreal in July where we all will reconvene and discuss the next steps. It looks like we can get agreement on the short term, and more low level items, but there is still quite a bit of uncertainty on the higher level issues. Hopefully by having periodic meetings, the dialog will be ongoing, and progress can be made.
Already Steve has posted a clean up of his markers (one of the items below). This is great, so that even systemtap can use these new trace events in the kernel: http://lkml.org/lkml/2009/4/14/414 Also keep an eye on utrace-devel mailing list: where the ptrace cleanups are being discussed (they are now also being discussed on lkml)
These are mine and Christoph's notes from the discussions we had at the Collaboration Summit in SFO, in April. I've added some clarifications, comments and updates from Roland and Frank as well. Keep in mind that the situation is not static. Every day there are new discussions on these topics on the various mailing lists, so this is a "snapshot" of about of the end of April.
Attendes: Renzo Davoli, Mathieu Desnoyers, Jake Edge, Frank Ch. Eigler, Christoph Hellwig, Masami Hiramatsu, Jim Keniston, Roland McGrath, Steven Rostedt, Josh Stone, Elena Zannoni. Plus James Bottomley (day 2), Keiichiro Tokunaga (from Fujitsu)
Thursday afternoon:
Kernel Tracing items:
- make DEFINE_TRACE work in modules (Steve)
- investigate markers removal (Christoph, Matthew)
- the 25 magic google tracpoints (Matthew)
- make the two major tracepoint implementations interchangeable (Matthew, Steve) (working on a common ring buffer API)
- get djprobes and the instruction decoder upstream (Masami)
Utrace and userspace probing:
- get arm and mips converted to regsets and uprobes, set a cut off date for others (Christoph, Roland)
*** Update 4/24/09: arm work finished by Roland. (posted to lkml?) Christoph pinged the mips maintainer, no reply seen.
- more ptrace cleanups to prepare for utrace (Oleg) [In progress]
- in-kernel gdb server for debugging userspace (Frank)
- get uprobes upstream piecemail, including backing the gdbserver(Jim)
From Friday morning:
- Include Roland in email about kbuild patch for separate debuginfo sent by Wenji Huang (Elena)
- elfutils debuginfo reduction (duplicates eliminations): ROland working on it, goal is Fedora 12 timeframe.
- Look up old patch from crashdump people about pulling type info into a common .h file for debuginfo purposes. (SLES9 or SLES10) (Christoph?)
- Talk to Arnaldo about his CTF work. Redhat folks to have a meeting with him.
*** Update 4/24/09(Roland): Acme working on a kbuild scheme using his tools herein the kernel would compile with -g but have DWARF replaced with CTF in each .o during compilation. This intends to avoid the "slow compile" problem cited by some kernel hackes, and believed to be slow link phase and disk i/o. (We'll see how quick it is.) That would yield CTF for the kernel, which has at least type layout information and some minimal symbolic info about functions. That is intended to suffice for function entry probes with parameters, but it does not handle the parameter ABI problem. (As far as we can tell, Sun only deals with one presumed function call ABI.)
- James had a comment about how difficult it is for stap/kretprobes to probe at the end of a function. Seems that epilogue or last instruction of a function (upon return) is hard to probe. He is interested not just in the return value, but in the value of the locals before exit. Gcc-4.4 (current fedora and future rhel6 compiler) should produce correct info.
Systemtap PR: 10056.
What James wants is *new* compiler information to identify a site at the beginning of the epilogue as a good probe site. He believes that at such a place, the locals (now technically dead) will in fact still be accessible in stack slots not clobbered before the epilogue. If that's in fact true for a particular site of interest, then the idea presupposes that GCC's DWARF location information for that PC is accurate about whatever happens to be accessible still. AFAIK we
don't especially suspect any problems with that location information.
The DWARF format (line info) includes an epilogue_begin marker feature. This is what would be used for finding the probe site James wants. There was past talk (and maybe even some unfinished work) to make GCC emit those, but GCC 4.4 does NOT emit those markers.
- how good is stap at working on code that has no associated debuginfo. For markers, Josh has added a patch that can generate debuginfo on the fly. To be clear, it generates sufficient information for type layouts. That is enough to do interesting things with tracepoint parameters.
(This is the more useful half of what we could get from CTF.) systemtap 0.9.6 will have two new features for this.
- Revisit which (if any) parts of systemtap code tree could be pushed into kernel tree.
- Future directions: there was agreement that doing userspace debugging via .ko's is not the best solution in the long term, but currently is the only way there is to do userspace probing.
- Complaints about stap shutting itself down (from Mathieu) when doing irq monitoring.
The germ of good idea to take away from this (beyond documenting -DINTERRUPTIBLE and some other tweaks already done) is that we may want to teach systemtap to be a little more tolerant of reentrancy. For probe handlers that simply trace (without local vars, without many
temporaries), with a reentrant-tolerant printf(), we can probably do OK -- without losing the events Mathieu was worried about, and without -DINTERRUPTIBLE=0. Future work, no PR yet.
- Idea was floated around to define a stable api into the kernel for which stap is guaranteed not to break from one kernel version to the next. Mathieu proposed to keep a part of the stap self testing module into the kernel so that kernel developers can test themselves if they break something. Idea not practical, since it puts the burden on kernel community to assure that out of tree code keeps working. Problem is that stap doesn't use a predefined set of probe points. This is the whole idea behind the current stap design, that is, to get away from the static approach of dtrace, and to let you probe everything because of the use of the debuginfo. I.e. "API" is potentially the whole kernel code. However, the translator-generated and runtime-boilerplate code use a relatively small number of exported functions as the API. This is one reason why the runtime etc. is in fact remarkably stable from kernel version to kernel version. Maybe 1-3 changes per kernel release are needed. The tapsets are a different matter. Reorienting them with a preference for tracepoints will improve them too.
WRT keeping some stap tests in the kernel tree, one problem with this idea is that while the kernel developer can then instantly fix this hypothetical self-test module, and potentially make the self-test compile ("closing the feedback loop"), there is a hole in the feedback to the systemtap developers. Unless this "self-test module" is in fact complete, we still need to carry the fuller runtime/etc. code in systemtap, so we'd need to receive & redistribute those patches
- Mathieu mentioned gdb tracepoints: continuous debugging. Similar to systemtap idea. Collect info at breakpoints and continue the program. Code Sourcery working on that.
