Monday Jan 07, 2008

Fenxi - Performance analysis made easy

We just opensourced a nice performance analysis tool called Fenxi. Fenxi is a pluggable Java-based post-processing, performance analysis tool that parses and loads the data from a variety of tools into a database, and then allows you to query and compare different sets of performance data. Fenxi can also be used to graph data from performance tools. Fenxi (mandarin for analyze) is the successor to the Sun-internal tool called Xanadu. It is integrated with the Faban Benchmark harness.

If you have ever worked with performance data, you will pretty soon realize that
Performance Data can get huge.
Consider a benchmark running on a 64 core system with 100's of disks attached, with multiple network interfaces for 30 minutes. If you collect mpstat at 10 second intervals for the whole run, you end with more than 11,000 lines of data! (That is 400 CNTRL-F's if you are using VI in a regular sized termial). If you collect data from more tools like vmstat, iostat, trapstat, busstat, cpustat, etc you will end up with much more! Going through each of them line by line is not a scalable approach.
Performance Data is interrelated.
The tool outputs are just different views of the system behavior. We want to look at the system as a whole, rather than at its individual views. If your incoming network packets peaks, your interrupts in your mpstat most likely peaks. We may want to see if throughput was impacted as a result of a burst of writes to our disks, etc.
Some performance data makes sense visually.
For large data, a visual view gives a quick summary of the data. As Tim Cook states it, "the human brain is a powerful pattern-recognition machine - graphs allow you to spot things you would never see in numbers (like waves of CPU migrations moving across different cores)". Look at the bottom of the blog for more details
Performance Data should be queryable
We want to be able to query or ask questions to the performance data.  For ex, you might want to know "What are my hot disks?". Traditionally, people have answered such questions  by writing custom scripts using sed/awk/perl. This can get tedious very fast. We need a better way of asking questions. In Fenxi, we store the data in the database, and questions are formulated in SQL.
Performance Data should be comparable, averageable, etc.
Since I work in the performance group at Sun, we run a lot of benchmarks. Since the goal of [most] benchmarks is to maximize the performance of a system, we are always constantly trying out new changes to the system. Typically, we change a parameter and repeat the benchmark and see if it has improved performance.
Performance Data should be sharable.
We rarely work in isolation. We should be able to share data with our peers and collaborate on finding performance fixes.

Fenxi tries to solve all of the above problems.

Sample Graph

Sample Text

Fenxi text view

You can see a sample database run processed by Fenxi. I urge you to check it out!

Thursday Nov 01, 2007

Visualizing callgraphs via dtrace and ruby

The allfrom.d dscript can be used to display all function entry/exits caused by a function call. When call graphs are deep, or long, a visual representation is very helpful to understand the flow as well as how much time each function consumed. The output displayed is inspired by Roch's CallStackAnalyzer which was in turn inspired by the work on vftrace by Jan Boerhout.

I wrote a simple ruby script to post process the allfrom.d output and generate a SVG image. The advantage of using SVG is that you can use javascript to provide added functionality. For example, you can hover your mouse over any block to see the name of the function and its elapsed time. Similarly, you could add support for Zoom and Pan

Unfortunately, I am having problems with serving svg files with the right mime type. So I have included a png image below. You can save the svg files somewhere on your computer and view them using Firefox.

A sample for the connect(3socket) is shown below. The input file used to generate it is available here. The width of each box indicates how long the function took. Y axis indicates call depth.

If you are interested, you can also check out

To use the scripts to generate your own call stacks, download the following two [ 1, 2] ruby source files. For example, to generate the above callgraph, you can use

dtrace -s allfromkernel.d acccept > accept.log
ruby function_call_graph.rb accept.log > accept.svg

Note that the script does not work reliably for asynchronous function calls, or cases where the thread changes CPU. If there is sufficient interest, I might be tempted to add those in :-)

Thursday Oct 18, 2007

Marshalling trouble with JRuby/Rails?

If you are getting exceptions like the one below with JRuby and Rails, read on for the solution
Rendering /tmp/neel/jruby-1.0.1/lib/ruby/gems/1.8/gems/actionpack-1.13.5/lib/
   action_controller/templates/rescues/layout.rhtml (500 Internal Error)
no marshal_dump is defined for class Java::JavaObject
/tmp/neel/jruby-1.0.1/lib/ruby/1.8/pstore.rb:349:in `dump'
/tmp/neel/jruby-1.0.1/lib/ruby/1.8/pstore.rb:327:in `transaction'
/tmp/neel/jruby-1.0.1/lib/ruby/1.8/cgi/session/pstore.rb:81:in `update'
/tmp/neel/jruby-1.0.1/lib/ruby/1.8/cgi/session/pstore.rb:88:in `close'
/tmp/neel/jruby-1.0.1/lib/ruby/1.8/cgi/session.rb:324:in `close'
The problem is that Rails is trying to store the Java object in the session store. The default session store is the filesystem, and it fails since it cannot marshal a Java Object. Setting the session store to use memory instead of the filesystem solves the problem.

In config/environment.rb set

config.action_controller.session_store = :memory_store

Thursday Mar 22, 2007

JRuby and Derby - The easy way

Are you interested in learning to use Derby from JRuby? While there are more comprehensive tutorials/classes on the web, here is a simple way to access Derby from JRuby (tested on 0.9.8). I hope you find it useful. Make sure derby.jar is in your CLASSPATH

require "java"

conn = java.sql.DriverManager.getConnection("jdbc:derby:test;create=true")

stmt = conn.createStatement
rs = stmt.executeQuery("select TABLEID,TABLENAME from sys.systables")

while ( do
  printf("%20s %20s\\n", rs.getString(1), rs.getString(2))



« April 2014