Thursday Jun 14, 2007

Coloring dtrace output

Currently I am working on a subtle bug in KSSL. (SSL kernel proxy) In order to diagnose the root cause of the bug I use set of dtrace scripts to gather data in various probes. One of the dtrace scripts I am using looks like this:

#!/usr/sbin/dtrace -Cs
  trace mbuf which caused activation of 
  kssl-i-kssl_handle_any_record_recszerr probe

  NOTE: this presumes that every mblk seen by TCP/KSSL
        is resonably sized (cca 100 bytes)

/\* how many bytes from a mblk to dump \*/
#define DUMP_SIZE       48

  generic kssl mblk dump probe 
  (covers both input path and special cases)
  printf("\\nmblk size = %d",
      ((mblk_t \*)arg0)->b_wptr - ((mblk_t \*)arg0)->b_rptr);
  tracemem(((mblk_t \*)arg0)->b_rptr, DUMP_SIZE);
  tracemem(((mblk_t \*)arg0)->b_wptr - DUMP_SIZE, DUMP_SIZE);

The scripts usually collect big chunks of data from the various SDT probes I have put into kssl kernel module. After the data are collected I usually spend big chunks of time sifting though it. At one point of time I have got a suspicion that the problem is actually a race condition of sorts. In order to shed some light on what's going on I have used less(1) which provides highlighting of data when searching. While this is sufficient when searching for a single pattern, it does not scale when more patterns are used. This is when I got the idea to color the output from dtrace scripts to see the correlations (or lack of them) of the events with a simple Perl script. Example of the output colored by the script:

colored dtrace output (data)

This looks slightly more useful than plain black output in terminal but even with 19" display the big picture is missing. So, I have changed the dtrace-coloring script to be able to strip the data parts for probes and print just the headers:

colored dtrace output (headers)

This is done via '-n' command line option. (the default is to print everything.) The output containing just the colored headers is especially nice for tracking down race conditions and other time-sensitive misbehaviors. You can download the script for dtrace log coloring here:

The colors can be assigned to regular expressions in the hash 'coloring' in the script itself. For the example above I have used the following assignments:

my %coloring = (
  '.\*kssl_mblk-i-handle_record_cycle.\*' => '4b6983',    # dark blue
  '.\*kssl_mblk-enqueue_mp.\*'    => '6b7f0d',    # dark green
  '.\*kssl_mblk-getnextrecord_retmp.\*'   => 'a11c10',    # dark red

In the outputs above you might have noticed that a timestamp is printed when a probe fires. This is useful for pre-processing of the log file. dtrace(1) (or libdtrace to be more precise) does not sort events as they come from the kernel. (see CR 6496550 for more details) In cases when hunting down a race condition on multiprocessor machine having the output sorted is crucial. So in order to get consistent image suitable for race condition investigation a sort script is needed. You might use a crude script of mine or you can write yours :)

Technorati Profile

Tuesday Mar 27, 2007

Sound Blaster Live! on Solaris

During stressful days of libmd backport to Solaris 10 update 4 I managed to break my iPod mini by sliding on my chair backwards and running to get next coffee cup. (The catch was that I still had my headphones connected to the iPod on so the iPod fell down on the floor. It now only reports the unhappy face of Macintosh.)

Only after that I have realized how music is important for my day-to-day tasks. I cannot simply continue in the same pace as before without steady flow of rhythm into my ears.

Before buying another iPod I wanted to get some interim relief. This is when I entered the not-so-happy world of Solaris audio drivers. My workstation is Acer Aspire E300 which came with integrated TV card and whatnot but also with integrated sound card. The sound card is supported in Solaris but the level of noise coming from it was unbearable. (and I am no HiFi snob, listening mostly to 32kbps radio Akropolis streams) After some googling I have realized that there is a driver for Sound Blaster Live! 5.1 PCI card which I had in my Solaris test workstation at home. The driver was backported by J├╝rgen Keil (frequent OpenSolaris contributor) from NetBSD among other drivers.

The relevant part of prtconf -v output looks like this:

            pci1102,8027 (driver not attached)
                Hardware properties:
                    name='assigned-addresses' type=int items=5
                    name='reg' type=int items=10
                    name='compatible' type=string items=7
                        value='pci1102,2.1102.8027.7' + 'pci1102,2.1102.8027' + 
'pci1102,8027' + 'pci1102,2.7' + 'pci1102,2' + 'pciclass,040100' + 'pciclass,0401'
                    name='model' type=string items=1
                        value='Audio device'

It's quite easy to get it working:

  1. disable the on-board sound card in BIOS
  2. get the sources and extract them
      bzcat audio-1.9beta.tar.bz2 | tar -xf -
  3. compile (set the PATH if needed to get access to gcc/ar)
      export PATH=/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/sfw/sbin:/usr/ccs/bin
      cd audio-1.9beta && make
  4. install the driver (audioemu) and its dependency (audiohlp) on x86:
      cp drv/emu/audioemu /platform/i86pc/kernel/drv
      cp drv/emu/amd64/audioemu /platform/i86pc/kernel/drv/amd64
      cp drv/emu/audioemu.conf /platform/i86pc/kernel/drv
      cp misc/audiohlp /kernel/misc
      cp misc/amd64/audiohlp /kernel/misc/amd64
      cp misc/audiohlp.conf /kernel/misc
  5. attach the driver (see instructions in drv/emu/Makefile)
    add_drv -i '"pci1102,2" "pci1102,4"' audioemu
  6. reboot

Yes, I could have used the infrastructure provided in Makefiles to create the package and install it but I wanted to have minimalistic install and have just the things which are really needed.

After the reboot you should be able to play sound via e.g. xmms. (installed e.g. from Blastwave) Check for audioemu in modinfo(1M) output and dmesg(1M) output for error messages if something goes wrong. So far it has been working for me rather flawlessly. (no kernel panics ;))

During the search for the driver I have discovered number of complaints from the users trying OpenSolaris for the first time that their Sound Blaster Live! was not recognized. Looking into Device drivers community not much is going on about soundcard drivers.
I wonder how hard would it be to get the audioemu NetBSD-based driver into ON.. the CR number is 6539690.

2007-08-08 update: After asking on opensolaris-discuss mailing list I have realized that there is a project underway which will deliver OSS into (Open)Solaris. Some info can be found in PSARC/2007/238, however there is no project page at (yet). Hopefully, RFE 6539690 will be closed after OSS integrates into ONNV.

Monday Nov 13, 2006

Simple Solaris Installation

OpenSolaris was already out before I joined Sun so I had a chance to play with Solaris Express Community release for couple of months and actually look into the source code online thanks to Chandan's OpenGrok. Still, with all these goodies it required fair amount of exploration before I figured things out. Being a BSD person some of them were not surprising, whereas some of them slowed me down substantialy.

I don't remember now the origin of the thought to summarize the installation steps and basic steps after installation to make the system more useable, this is not important. Anyway, the slides containing all of this information (and more) were made.
The slides were originaly meant for CZOSUG Bootcamp where people brought their laptops and installed Solaris on them. I have created the slides together with Jan Pechanec . After watching both external people and new-hires struggle with basic steps after completing Solaris installation I think they could be used also to ease those post-install how-do-I-set-this steps.

They are not perfect, could contain errors (please do report them) but here you go:

Also, do not forget to look at recently founded Immigrants community at which contains other goodies such as links to Ben Rockwood's Accelerated introduction to Solaris. Also do not forget to subscribe to the Immigrants mailing list.

Friday Nov 03, 2006

Cleartext for now !

I am Vlad, working as "Revenue Product Engineer" in security "technology-management" team. This means that my main job is to fix security bugs (both vulnerabilities and bugs in technologies) in Solaris.

My fixes usually have to do something with following technologies/products: IPsec, SSH, OpenSSL and crypto. In this blog I will try to present not only security technologies but also Solaris/OpenSolaris-related stuff.


blog about security and various tools in Solaris


« July 2016