Elena's Blog

January 29 2008

I am currently at a two day tracing workshop in Montreal Canada. It is
mainly an industry workshop hosted by Ericsson. There are a few Open
Source folks, like Mathieu, Andrew, Nathan and me. Too bad Frank
couldn't make it.  I decided to drive from Boston, it seemed like a
good idea at the time, but now I am not so sure anymore, it was a
fairly long drive. Fortunately the rental car has a 6 CDs changer.
There was an opening reception Monday evening, with nice French and
Canadian food. I tried the caribou steak, not bad at all.

The presentations so far have been of good quality, even though some
were a bit too detail oriented. There are representatives from IBM,
Red Hat, University of Montreal and Ecole Polytechnique, Canadian
DoD, Freescale, Oracle, MontaVista, WindRiver and others.  There is
a wiki for the workshop which includes the slides. The wiki will be
publicly accessible in a week or so. I'll post the link then.

The first day talks provided an overview of the current tracing
platforms including the challenges that they face. Systemtap, Frysk,
Eclipse/TPTP, QNX. There was a presentation about old IBM projects,
and the experience gathered about tracing as part of them.  I gave a
short presentation about the use of Systemtap within Oracle, and the
problems that deploying the tool in a commercial/enterprise
environment entails. I don't think these problems are specific to
Oracle, on the contrary they are a good representative set of what
Systemtap needs to address as a project in order to step up to the
next level, and move beyond the hacker community. Let's hope we can
figure out what to do about them, as they are general challenges that
would impact gdb and frysk too. I also talked a bit about some other
usage of systemtap that we tried in my team, specifically to generate
kernel modules to force the kernel into a resource starvation
state. These scripts were based on some previous work that Bryce did,
and the source code for those fragments is available here.  Here are
my slides.  Mathieu presented about LTTng and tracing at
Google. Interesting talk, especially now that the markers
infrastructure has been released in the 2.6.24 kernel. Speaking of
which, there is still a lot to do there, because now there will need
to be negotiations about where to insert the actual markers in the
various kernel subsystems. Once those are in, we need to write the
systemtap scripts that use them.

The evening had a couple of BOFs, one about Eclipse and modeling, and
the other about tracing events. I attended the latter. Discussion was
a bit all over the place, but there was some interesting brainstorming
about tracing and time counters in virtualized environments. The notes
should be up in the wiki soon.

In the evening I met again with a bunch of folks from the workshop,
and a colleague, Martin, kernel person from Oracle who lives in Montreal.


January 30 2008

On the second day, somebody from Enea presented a tracing framework
that is based on evenpoints and actions which is very similar in
spirit to Systemtap.

The next talk was from ZealCore. They talked about replay debugging,
and tracing. They do flight recorder type tracing, with very small
overhead.

Next was Eclipse PTP. Nice CDT extensions for parallel programming,
supporting all sort of interactive static analysis for OpenMP and MPI.
This effort was mentioned, called "Scalable Tools Communication
Infrastructure" at http://www.scalabletools.org. Interesting, worth
keeping an eye on them.  The PTP project uses SDM: Scalable Debug
Manager that is based on gdb. There are multiple instances of gdb one
for each process and a manager program that controls the various gdbs.
Also these projects were mentioned. TAU (from University of Oregon)
and PAPI.

Windriver presented about OS agnostic visualization tools, they used
LTTng, and plugged their graphical interface and visualizer on top of
it.

Unfortunately at this point I had to leave, and I skipped the last
presentation, also from WindRiver, where a tool was presented that had
similar functionality to Systemtap.

The Montreal Tracing Workshop wiki is now publicly accessible.
This is the link:
https://ltt.polymtl.ca/tracingwiki/index.php/TracingSummit2008

A bit late, but nevertheless...

These are some notes I took at the Collab Summit in San Francisco. I posted these notes already on the systemtap mailing list, but I thought they are worth of a blog entry.

Some of the slides, not all of them, are here: http://events.linuxfoundation.org/slides

Our tracing session went well, I think it was most useful for the developers themselves, it was a
bit technical.

The first day of the summit was all dedicated to panels and presentations to the whole audience. It was not broken down into sessions. One of the presentations was by Edward Screven about Oracle and Linux. I thought it went well. It really showed how we use Linux internally and it explained why we got into the Linux business. Also talked about upstream contributions.

Another highlight of the first day was a panel with Jim Zemlin (the LF director), Ian Murdock of Sun, and Sam Ramji from Microsoft. It was entertaining to watch how the Microsoft person kept trying to not say anything that could be quoted by journalists (he said that himself). Quite an achievement, actually.

Detailed blogs have been posted by Gerrit Huizenga of IBM: http://gh-linux.blogspot.com/

The important stuff that for me happened at the summit was the meetings we had with the tracing people and some kernel folks. It started on Thursday, after the presentation were over, and we reached some consensus on the tracing infrastructure that is going into the kernel. (Markers, utrace, uprobes). The discussions continued on Friday and we talked more about higher level things, like the debuginfo size problem and the systemtap integration with the kernel. Detailed notes below.

Overall it was a good conference, or at least, let's say that I was very happy with the conclusions and the agreement we reached about tracing. Of course not everything is decided, but it was a good start. It seems that the tracing problem is really getting attention from the kernel community. There is going to be a tracing mini-summit at the Linux Symposium in Montreal in July where we all will reconvene and discuss the next steps. It looks like we can get agreement on the short term, and more low level items, but there is still quite a bit of uncertainty on the higher level issues. Hopefully by having periodic meetings, the dialog will be ongoing, and progress can be made.

Already Steve has posted a clean up of his markers (one of the items below). This is great, so that even systemtap can use these new trace events in the kernel: http://lkml.org/lkml/2009/4/14/414 Also keep an eye on utrace-devel mailing list: where the ptrace cleanups are being discussed (they are now also being discussed on lkml)

These are mine and Christoph's notes from the discussions we had at the Collaboration Summit in SFO, in April. I've added some clarifications, comments and updates from Roland and Frank as well. Keep in mind that the situation is not static. Every day there are new discussions on these topics on the various mailing lists, so this is a "snapshot" of about of the end of April.

Attendes: Renzo Davoli, Mathieu Desnoyers, Jake Edge, Frank Ch. Eigler, Christoph Hellwig, Masami Hiramatsu, Jim Keniston, Roland McGrath, Steven Rostedt, Josh Stone, Elena Zannoni. Plus James Bottomley (day 2), Keiichiro Tokunaga (from Fujitsu)

Thursday afternoon:

Kernel Tracing items:

- make DEFINE_TRACE work in modules (Steve)
- investigate markers removal (Christoph, Matthew)
- the 25 magic google tracpoints (Matthew)
- make the two major tracepoint implementations interchangeable (Matthew, Steve) (working on a common ring buffer API)

- get djprobes and the instruction decoder upstream (Masami)

Utrace and userspace probing:

- get arm and mips converted to regsets and uprobes, set a cut off date for others (Christoph, Roland)

*** Update 4/24/09: arm work finished by Roland. (posted to lkml?) Christoph pinged the mips maintainer, no reply seen.

- more ptrace cleanups to prepare for utrace (Oleg) [In progress]
- in-kernel gdb server for debugging userspace (Frank)
- get uprobes upstream piecemail, including backing the gdbserver(Jim)

From Friday morning:

- Include Roland in email about kbuild patch for separate debuginfo sent by Wenji Huang (Elena)

- elfutils debuginfo reduction (duplicates eliminations): ROland working on it, goal is Fedora 12 timeframe.

- Look up old patch from crashdump people about pulling type info into a common .h file for debuginfo purposes. (SLES9 or SLES10) (Christoph?)

- Talk to Arnaldo about his CTF work. Redhat folks to have a meeting with him.

*** Update 4/24/09(Roland): Acme working on a kbuild scheme using his tools herein the kernel would compile with -g but have DWARF replaced with CTF in each .o during compilation. This intends to avoid the "slow compile" problem cited by some kernel hackes, and believed to be slow link phase and disk i/o. (We'll see how quick it is.) That would yield CTF for the kernel, which has at least type layout information and some minimal symbolic info about functions. That is intended to suffice for function entry probes with parameters, but it does not handle the parameter ABI problem. (As far as we can tell, Sun only deals with one presumed function call ABI.)

- James had a comment about how difficult it is for stap/kretprobes to probe at the end of a function. Seems that epilogue or last instruction of a function (upon return) is hard to probe. He is interested not just in the return value, but in the value of the locals before exit. Gcc-4.4 (current fedora and future rhel6 compiler) should produce correct info.

Systemtap PR: 10056.

What James wants is *new* compiler information to identify a site at the beginning of the epilogue as a good probe site. He believes that at such a place, the locals (now technically dead) will in fact still be accessible in stack slots not clobbered before the epilogue. If that's in fact true for a particular site of interest, then the idea presupposes that GCC's DWARF location information for that PC is accurate about whatever happens to be accessible still. AFAIK we
don't especially suspect any problems with that location information.

The DWARF format (line info) includes an epilogue_begin marker feature. This is what would be used for finding the probe site James wants. There was past talk (and maybe even some unfinished work) to make GCC emit those, but GCC 4.4 does NOT emit those markers.

- how good is stap at working on code that has no associated debuginfo. For markers, Josh has added a patch that can generate debuginfo on the fly. To be clear, it generates sufficient information for type layouts. That is enough to do interesting things with tracepoint parameters.
(This is the more useful half of what we could get from CTF.) systemtap 0.9.6 will have two new features for this.

- Revisit which (if any) parts of systemtap code tree could be pushed into kernel tree.

- Future directions: there was agreement that doing userspace debugging via .ko's is not the best solution in the long term, but currently is the only way there is to do userspace probing.

- Complaints about stap shutting itself down (from Mathieu) when doing irq monitoring.

The germ of good idea to take away from this (beyond documenting -DINTERRUPTIBLE and some other tweaks already done) is that we may want to teach systemtap to be a little more tolerant of reentrancy. For probe handlers that simply trace (without local vars, without many
temporaries), with a reentrant-tolerant printf(), we can probably do OK -- without losing the events Mathieu was worried about, and without -DINTERRUPTIBLE=0. Future work, no PR yet.

- Idea was floated around to define a stable api into the kernel for which stap is guaranteed not to break from one kernel version to the next. Mathieu proposed to keep a part of the stap self testing module into the kernel so that kernel developers can test themselves if they break something. Idea not practical, since it puts the burden on kernel community to assure that out of tree code keeps working. Problem is that stap doesn't use a predefined set of probe points. This is the whole idea behind the current stap design, that is, to get away from the static approach of dtrace, and to let you probe everything because of the use of the debuginfo. I.e. "API" is potentially the whole kernel code. However, the translator-generated and runtime-boilerplate code use a relatively small number of exported functions as the API. This is one reason why the runtime etc. is in fact remarkably stable from kernel version to kernel version. Maybe 1-3 changes per kernel release are needed. The tapsets are a different matter. Reorienting them with a preference for tracepoints will improve them too.

WRT keeping some stap tests in the kernel tree, one problem with this idea is that while the kernel developer can then instantly fix this hypothetical self-test module, and potentially make the self-test compile ("closing the feedback loop"), there is a hole in the feedback to the systemtap developers. Unless this "self-test module" is in fact complete, we still need to carry the fuller runtime/etc. code in systemtap, so we'd need to receive & redistribute those patches

- Mathieu mentioned gdb tracepoints: continuous debugging. Similar to systemtap idea. Collect info at breakpoints and continue the program. Code Sourcery working on that.

Two weeks ago I attended the Linux Symposium in Montreal.
I was there mostly to participate in the tracing track, and to discuss the current state of kernel tracing infrastructure and the other projects, determining what has been accomplished since the meetings we had at the Collaboration Summit in April. Some of the key players were not at the conference however, and the discussion was a bit limited. Some of the goals that were set at the Collab Summit are still lagging a bit. On the utrace side, Frank is working on the gdb stub to talk to utrace. This is built on top of kernels with utrace integrated (RHEL and Fedora) and uses the gdb remote protocol (in usual gdb fashion) to communicate with the target process being debugged under utrace control. His latest prototype is here:
http://sourceware.org/ml/systemtap/2009-q3/msg00045.html

Ptrace cleanups: Roland and Oleg are now official ptrace kernel maintainers (since April 2009). Oleg Nesterov is making progress on this, but I am not seeing any recent posts.
I see that there was a wave of ptrace cleanup patches at the end of May from Oleg (http://lkml.org/lkml/2009/5/30/229), but I cannot see that they went in. As of last week, there were no comments on them at all. It seems we are stuck in the architecture specific cleanups for ptrace (see this thread: http://lkml.org/lkml/2009/5/3/109).

Utrace: I've seen no major changes, but just this week, Oleg and Roland started looking at the utrace implementation a bit more closely, as you can see from the utrace-devel mailing list. Of course it's all depending on the ptrace cleanups getting into the kernel.

ftrace and markers: Steve Rostedt wasn't there, and Mathieu has been busy with his dissertation. Frederic Weisbecker gave a presentation, it was an introduction to ftrace.

Systemtap: Frank gave a talk on systemtap, an overall introduction and overview.

There was a panel where people talked about their projects, not much talk unfortunately around how to make progress in the overall area. It was mostly a showcase of each project.

Many people from the Montreal area talked about Eclipse. These folks work for Ericsson which uses Eclipse a lot.

The slides from the tracing mini-summits are here:
http://ltt.polymtl.ca/tracingwiki/index.php/TracingMiniSummit2009

The first day I attended a tutorial given by IBM, about Performance Inspector.
It's a decent performance tool, sharing many things with Oprofile, plus a gui. It turns out the project is an old internal IBM project, that got open sourced much later. One of the people working on it was the author of Oprofile.

I attended Martin and Dan talks as well. I think they were very good ones. Honestly, they had some real meat to them, and some new ideas. They also were engaging speakers, and didn't put the audience to sleep. I also attended Ric Wheeler's talk and that was interesting, and a nice high level summary.
Dan's Transcendent Memory slides are here:
http://oss.oracle.com/projects/tmem/documentation/presentations/
and his work is here:
http://oss.oracle.com/projects/tmem
There is a good article on Tmem here: http://lwn.net/Articles/340080/

Jon Corbet did his usual kernel report, and mentioned Oracle a lot. That was nice to see.

A talk about autotest from Google was OK. They started from Martin Bligh's system and expanded on that, the test driver mechanism is getting more sophisticated and operates on many different machines, and layers of servers.

On November 9 and 10 I attended the Linux Foundation End User Summit in Jersey City. Ted Tso and I organized a mini session about tracing in Linux. We had Steve Rostedt talking about ftrace, and Subendhu Ghosh from RedHat talk about the perf events subystem. The session was on Tuesday afternoon, and I'd say that aside from Christoph Hellwig talk about KVM it was the most crowded session. It was nice that both Jon Corbet in his Kernel Report talk that morning, and James Bottomley in his panel the day before mentioned tracing and our session as something "hot" to follow. The Ftrace talk consited of an introduction to Ftrace, and a few demos. The audience was mostly of Wall Street type people, who are still using old versions of enterprise products, so the don't really know what has happened in the meantime to the upstream. Ftrace has been around for a while, a couple of years, but perf events is newer, and many people didn't know what it was. There was a little demo of that as well. While talking about perf events, somebody (hch, IIRC) mentioned the new tool that Arjan built on top of perf events, called "timechart". It's a tool that spits out a graphical view of the time your system is spending doing various things. http://blog.fenrus.org/?p=5.
Perf events is in Fedora 12 and so is ftrace. The timechart thing is only in mainline now.
Slides from Steve's talk are on Steve's page: http://people.redhat.com/srostedt

Besides this tracing session there was a talk done by Sergio Leunissen from my same Oracle Linux team, on OLT (Oracle Linux Testkit) and Validated Configurations and how that is helping us stabilize OEL for enterprise loads. The talk went well, the audience asked lots of questions about OLT and VC. Also were happy to see the oracle-validated RPM that we ship.

The other talk I attended on the second day was the filesystem recap by Ric Wheeler. Nothing new here, just a summary of all the things that have happened to the various FS in linux during the last year or so. It was nice that Ric mentioned Oracle and the FS folks by name (thanks Ric! ). So Chris, Martin, Jens, Chuck got honorable mentions. In general the amount of work that Oracle is doing in this space is still a bit unknown, since several people in the audience expressed surprise to see Oracle so involved in Linux.

On the first day there were no deep technical talk or separate sessions. Of the plenary sessions, an interesting one was the one by IDC about the Linux market share. While Windows keeps dominating the server side, Linux is increasing, and apparently Unix is not going away any time soon. The investment in Unix seems to still be fairly solid.

I also attended a session discussing how kernel interfaces should keep up with the times. In particular as examples there was mention of SSDs and SSSs which are going to be mainstream soon. Some complaint about epoll. Also mentioned were the interfaces for sending messages to multiple recipients, with suggestions that the calls there should be vectorized to speed up processing of messages. It's interesting that a very similar debate was going on almost simultaneously on the linux-raid mailing list, here.
Jens Axboe, also in the Oracle Linux Team, has done some work on per-bdi writeback threads, see his slides from the Linux Plumber's conference here.

Another good overview from the NYSE (Brian Clark) talked about what they focus on in their IT infrastructure.

I think this is about it.