Saturday Dec 01, 2007

Netcat in Solaris

CR 4664622 has been integrated into Nevada and will be part of build 80 (which means it will not be part of next SXDE release but I can live with that :)).

During the course of getting the process done I have stumbled upon several interesting obstacles. For example, during ingress Open Source Review I was asked by our open-source caretaker what will be the "support model" for Netcat once it is integrated. I was puzzled. Because, for Netcat, support is not really needed since it has been around for ages (ok, since 1996 according to wikipedia) and is pretty stable piece of code which is basically no longer developed. Nonetheless, this brings some interesting challenges with move to a community model where more and more projects are integrated by people outside Sun (e.g. ksh93 project).

The nc(1) man page will be delivered in build 81. In the meantime you can read Netcat review blog entry which contains the link to updated man page. The older version of the man page is contained in the mail communication for PSARC 2007/389.
Note: I have realized that the ARC case directory does not have to include most up-to-date man page at the time of integration. Only when something _architectural_ changes, then the man page has to be updated (which was not the case with Netcat since we only added new section describing how to setup nc(1) with RBAC). Thanks to Jim for the explanation.

I have some ideas how to make Netcat in Solaris even better and will work to get them done over time. In particular, there are following RFEs: 6515928, 6573202. However, this does not mean that there is only single person who can work on nc(1). Since it is now part of ONNV, anyone is free to hack it.

So, I'd like to invite everyone to participate - if you have an idea how to extend Netcat, what features to add, it is sitting in ONNV waiting for new RFEs (or bugs) and sponsor requests (be sure to read Jayakara Kini's explanation of how to contribute if you're not OpenSolaris contributor yet).

Also, if you're Netcat user and use Netcat in a cool way, I want to hear that !

Saturday Nov 17, 2007

Chemical plants, Linux and dtrace

When my friend from Usti nad Labem (9th biggest city in Czech republic) asked me to present about OpenSolaris at local Linux conference, I got hooked. First, Usti is interesting city (the city is surrounded by beautiful countryside, yet has chemical plants in the vicinity of city center) and I haven't been there for a long time and second, having the opportunity to present about OpenSolaris technologies to Linux folks is unique.

When we (Sasha agreed to go present with me) arrived to Usti we were greeted by slightly apocalyptic weather (see pictures below). The environment where all the presentations took place compensated that, fortunately.

40 minutes to present about something which most people in the room are not very aware of is challenging. The fact that OpenSolaris is open source and it is a home for several disruptive technologies makes that both easier and harder. We have greatly leveraged Bryan Cantrill's Dtrace review video taped at Google for doing second part of the presentation where we demo'ed dtrace. I have even borrowed some of his quotes. I am pretty sure he wouldn't object since his presentations were perused in the past. To make the list of attributions complete, we have substantial material in the first part from Lukas' past presentations about OpenSolaris project.

It's much better to demo the technology than just talk about how great it is (I remember a funny moment in Bryan's presentation where a dtrace probe didn't "want" to fire where Bryan jokingly said "bad demo!" to the screen. I nearly fell off of my chair at that moment.). "So, I have finally seen dtrace in action !" was one of the great things to hear after the presentation.

The "OpenSolaris technologies" presentation can be downloaded here.

Tuesday Aug 28, 2007

Getting code into libc

In my previous entry about BSD compatibility gap closure process I have promised to provide a guide on how to get new code into libc. I will use changes done via CR 6495220 to illustrate the process with examples.

Process related and technical changes which are usually needed:

  • get PSARC case done
  • File a CR to create a manual page according to the man page draft supplied with the PSARC case. You will probably need to go through the functions being added and assign them MT-Level according to attributes(5) man page (if this was not done prior to filing the PSARC case).
  • actually add the code into libc
    This includes moving/introducing files from the SCM point of view and doing necessary changes to the Makefiles.
    In terms of symbols, the functions need to be actually delivered twice. Once as underscored (strong) symbol and second as WEAK alias to the strong symbol. This allows libraries use their own private implementation of the functions. (This is because the weak symbol is silently overridden by the private symbol in runtime linker)
  • add entries to c_synonyms.h and synonyms.h
    synonyms.h is used in libc for symbol alias contruction (see above). c_synonyms.h provides access to underscored symbols for other (non-libc) libraries. This provides a way how to call the underscored symbols directly without risking namespace clashes/pollution.
    This step is actually needed to be used in conjunction with the previous step. nm(1) can be used to check this worked as expected:
    $ nm -x /usr/lib/ | grep '|_\\?err$'
    [5783] |0x00049c40|0x00000030|FUNC |GLOB |0  |13 |_err
    [6552] |0x00049c40|0x00000030|FUNC |WEAK |0  |13 |err
  • Do the necessary packaging changes
    If you're adding new header file change SUNWhea's prototype_\* files (most probably just prototype_com)
    If the file was previously installed into proto area during build it needs to be removed from the exception files (for i386 and sparc).
  • modify lib's mapfile
    This is needed for the symbols to become visible and versioned. Public symbols belong to the latest SUNW section. After you have compiled the changes you can check this via command similar to the following:
    pvs -dsv -N SUNW_1.23 /usr/lib/ \\
      | sed -n '1,/SUNW.\*:/p' | egrep '((v?errx?)|(v?warnx?));'
    If you're adding private (underscored) symbols do not forget to add them to the SUNWprivate section. This is usually the case because the strong symbols are accompanied by weak symbols. Weak symbols go to the global part of the most recent SUNW section and strong symbols go to global part of SUNWprivate section.
  • update libc's lint library
    If you are adding private symbols then add them as well. See the entries _vwarnfp et al. for example.
    After you're done it's time to run nightly with lint checks and fix the noise. (see below)
  • Add per-symbol filters
    If you are moving stuff from a library to libc you will probably want to preserve the existing interfaces. To accomplish this per-symbol filters can be added to the library you're moving from. So, if symbol foo is moved from libbar to libc then change the line in the global section of libbar's mapfile to look like this:
    This was done with the \*fp functions in libipsecutils' mapfile. The specialty in that case was that the \*fp functions were renamed to underscored variants while moving them via redefines in errfp.h.
  • Fix build/lint noise introduced by the changes
    There could be the following noises:
    • build noise
      Can be caused by symbol type clash (there is symbol of the same name defined in libc as FUNC and in $SRC/cmd as OBJT) which is not harmful because ld(1) will do due diligence and prefer the non-libc symbol. This can be fixed by renaming the local symbol. There could also be #define clash caused by inclusion of c_synonyms.h. Fixed via renaming as well.
    • lint noise
      In the 3rd pass of the lint checks an inconsistency in function declarations can be found such as this:
      /builds/.../usr/include/err.h", line 43: warning: function argument declared
      inconsistently: warn(arg 1) in utils.c(62) char \* and llib-lc:err.h(43) const char \*
      The problem with this output is that there are cca 23 files named utils.c in ONNV. CR 6585621 is waiting someone to provide remedy for that via adding -F flag to LINTFLAGS in $SRC/lib and $SRC/cmd.
      After the right file(s) are found the fix is usually renaming again. Where the renaming is not possible -erroff=E_FUNC_DECL_VAR_ARG2 can be passed to lint(1).
  • Make sure there are not duplicate symbols in libc after the changes
    This is necessary because it might confuse debugging tools (mdb, dtrace). For err/warn stuff there was one such occurence:
    [6583] | 301316| 37|FUNC |GLOB |0 |13 |_warnx
    [1925] | 320000| 96|FUNC |LOCL |0 |13 |_warnx
    This can be usually solved by renaming the local variant.
  • Test thoroughly
    • test with different compilers
      SunStudio does different things than gcc so it is good idea to test the code with both.
    • Try to compile different consolidations (e.g. Companion CD, SFW) on top of the changes. For err/warn project a bug was filed to get RPM build fixed.
    • Test if the WEAK symbols actually work
    • Test the programs in ONNV affected by the changes
      e.g. the programs which needed to be modified because of the build/lint noise.
  • Produce annotated putback list explaining the changes
    This is handy for a CRT advocate and saves time.
  • If the change requires some sort of action from fellow gatelings, send a heads-up, e.g. like heads-up for err/warn.
  • If you are actually adding code to libc (this includes moving code from other libraries to libc) send an e-mail similar to the heads-up e-mail to opensolaris-code mailing list, e.g. like this message about err/warn.

Thursday Aug 23, 2007

Closing BSD compatibility gap

I do not get to meet customers very often but I clearly remember the last time where I participated in a pseudo-technical session with a new customer. The engineers were keen on learning details about all features which make Solaris outstanding but they were also bewildered by the lack of common functions such as daemon() (see CR 4471189). Yes, there is a number of private implementations in various places in ONNV, however this is not very useful. Until recently, this was also the case of err/warn function family.

With the putback of CR 6495220 there are now the following functions living in libc:

  void err(int, const char \*, ...);
  void verr(int, const char \*, va_list);
  void errx(int, const char \*, ...);
  void verrx(int, const char \*, va_list);
  void warn(const char \*, ...);
  void vwarn(const char \*, va_list);
  void warnx(const char \*, ...);
  void vwarnx(const char \*, va_list);

These functions were present in BSD systems for a long time (they've been in FreeBSD since 1994). The configure scripts of various pieces of software contain checks for presence and functionality of the err/warn functions in libc (and setting the HAVE_ERR_H define). For Solaris, those checks have now become enabled too.

The err(3C) man page covering these functions will be delivered in the same build as the changes, that is build 72.

The change is covered by PSARC 2006/662 architectural review and the stability level for all the functions is Committed (see Architecture Process and Tools for more details on how this works). Unfortunately, the case is still closed. Hopefully it will be opened soon.
Update 09-28-2007: the PSARC/2006/662 case is now open, including onepager document and e-mail discussion. Thanks to John Plocher, Darren Reed and Bill Sommerfeld.

As I prompted in the err/warn Heads-up there is now time to stop creating new private implementations and to look at purging duplicates (there are many of them, however not all can be gotten rid of in favour of err/warn from libc).

I will write about how to get code into libc in general from a more detailed perspective next time.

However, this does not mean there is nothing left to do in this area. Specifically, FreeBSD's err(3) contains functions err_set_exit(), err_set_file() and errc(), warnc(), verrc() and vwarnc(). These functions could be ported over too. Also, there is __progname or getprogname(3). Moreover, careful (code) reader has realized that err.c contains private implementations of functions with fp suffix. This function (sub)family could be made Committed too. So, there is still lot of work which could be done. Anyone is free to work on any of these. (see Jayakara Kini's blog entry on how to become OpenSolaris contributor if you're not already)

Thursday Jun 14, 2007

Coloring dtrace output

Currently I am working on a subtle bug in KSSL. (SSL kernel proxy) In order to diagnose the root cause of the bug I use set of dtrace scripts to gather data in various probes. One of the dtrace scripts I am using looks like this:

#!/usr/sbin/dtrace -Cs
  trace mbuf which caused activation of 
  kssl-i-kssl_handle_any_record_recszerr probe

  NOTE: this presumes that every mblk seen by TCP/KSSL
        is resonably sized (cca 100 bytes)

/\* how many bytes from a mblk to dump \*/
#define DUMP_SIZE       48

  generic kssl mblk dump probe 
  (covers both input path and special cases)
  printf("\\nmblk size = %d",
      ((mblk_t \*)arg0)->b_wptr - ((mblk_t \*)arg0)->b_rptr);
  tracemem(((mblk_t \*)arg0)->b_rptr, DUMP_SIZE);
  tracemem(((mblk_t \*)arg0)->b_wptr - DUMP_SIZE, DUMP_SIZE);

The scripts usually collect big chunks of data from the various SDT probes I have put into kssl kernel module. After the data are collected I usually spend big chunks of time sifting though it. At one point of time I have got a suspicion that the problem is actually a race condition of sorts. In order to shed some light on what's going on I have used less(1) which provides highlighting of data when searching. While this is sufficient when searching for a single pattern, it does not scale when more patterns are used. This is when I got the idea to color the output from dtrace scripts to see the correlations (or lack of them) of the events with a simple Perl script. Example of the output colored by the script:

colored dtrace output (data)

This looks slightly more useful than plain black output in terminal but even with 19" display the big picture is missing. So, I have changed the dtrace-coloring script to be able to strip the data parts for probes and print just the headers:

colored dtrace output (headers)

This is done via '-n' command line option. (the default is to print everything.) The output containing just the colored headers is especially nice for tracking down race conditions and other time-sensitive misbehaviors. You can download the script for dtrace log coloring here:

The colors can be assigned to regular expressions in the hash 'coloring' in the script itself. For the example above I have used the following assignments:

my %coloring = (
  '.\*kssl_mblk-i-handle_record_cycle.\*' => '4b6983',    # dark blue
  '.\*kssl_mblk-enqueue_mp.\*'    => '6b7f0d',    # dark green
  '.\*kssl_mblk-getnextrecord_retmp.\*'   => 'a11c10',    # dark red

In the outputs above you might have noticed that a timestamp is printed when a probe fires. This is useful for pre-processing of the log file. dtrace(1) (or libdtrace to be more precise) does not sort events as they come from the kernel. (see CR 6496550 for more details) In cases when hunting down a race condition on multiprocessor machine having the output sorted is crucial. So in order to get consistent image suitable for race condition investigation a sort script is needed. You might use a crude script of mine or you can write yours :)

Technorati Profile

Tuesday Mar 27, 2007

Sound Blaster Live! on Solaris

During stressful days of libmd backport to Solaris 10 update 4 I managed to break my iPod mini by sliding on my chair backwards and running to get next coffee cup. (The catch was that I still had my headphones connected to the iPod on so the iPod fell down on the floor. It now only reports the unhappy face of Macintosh.)

Only after that I have realized how music is important for my day-to-day tasks. I cannot simply continue in the same pace as before without steady flow of rhythm into my ears.

Before buying another iPod I wanted to get some interim relief. This is when I entered the not-so-happy world of Solaris audio drivers. My workstation is Acer Aspire E300 which came with integrated TV card and whatnot but also with integrated sound card. The sound card is supported in Solaris but the level of noise coming from it was unbearable. (and I am no HiFi snob, listening mostly to 32kbps radio Akropolis streams) After some googling I have realized that there is a driver for Sound Blaster Live! 5.1 PCI card which I had in my Solaris test workstation at home. The driver was backported by J├╝rgen Keil (frequent OpenSolaris contributor) from NetBSD among other drivers.

The relevant part of prtconf -v output looks like this:

            pci1102,8027 (driver not attached)
                Hardware properties:
                    name='assigned-addresses' type=int items=5
                    name='reg' type=int items=10
                    name='compatible' type=string items=7
                        value='pci1102,2.1102.8027.7' + 'pci1102,2.1102.8027' + 
'pci1102,8027' + 'pci1102,2.7' + 'pci1102,2' + 'pciclass,040100' + 'pciclass,0401'
                    name='model' type=string items=1
                        value='Audio device'

It's quite easy to get it working:

  1. disable the on-board sound card in BIOS
  2. get the sources and extract them
      bzcat audio-1.9beta.tar.bz2 | tar -xf -
  3. compile (set the PATH if needed to get access to gcc/ar)
      export PATH=/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/sfw/sbin:/usr/ccs/bin
      cd audio-1.9beta && make
  4. install the driver (audioemu) and its dependency (audiohlp) on x86:
      cp drv/emu/audioemu /platform/i86pc/kernel/drv
      cp drv/emu/amd64/audioemu /platform/i86pc/kernel/drv/amd64
      cp drv/emu/audioemu.conf /platform/i86pc/kernel/drv
      cp misc/audiohlp /kernel/misc
      cp misc/amd64/audiohlp /kernel/misc/amd64
      cp misc/audiohlp.conf /kernel/misc
  5. attach the driver (see instructions in drv/emu/Makefile)
    add_drv -i '"pci1102,2" "pci1102,4"' audioemu
  6. reboot

Yes, I could have used the infrastructure provided in Makefiles to create the package and install it but I wanted to have minimalistic install and have just the things which are really needed.

After the reboot you should be able to play sound via e.g. xmms. (installed e.g. from Blastwave) Check for audioemu in modinfo(1M) output and dmesg(1M) output for error messages if something goes wrong. So far it has been working for me rather flawlessly. (no kernel panics ;))

During the search for the driver I have discovered number of complaints from the users trying OpenSolaris for the first time that their Sound Blaster Live! was not recognized. Looking into Device drivers community not much is going on about soundcard drivers.
I wonder how hard would it be to get the audioemu NetBSD-based driver into ON.. the CR number is 6539690.

2007-08-08 update: After asking on opensolaris-discuss mailing list I have realized that there is a project underway which will deliver OSS into (Open)Solaris. Some info can be found in PSARC/2007/238, however there is no project page at (yet). Hopefully, RFE 6539690 will be closed after OSS integrates into ONNV.

Monday Nov 13, 2006

Simple Solaris Installation

OpenSolaris was already out before I joined Sun so I had a chance to play with Solaris Express Community release for couple of months and actually look into the source code online thanks to Chandan's OpenGrok. Still, with all these goodies it required fair amount of exploration before I figured things out. Being a BSD person some of them were not surprising, whereas some of them slowed me down substantialy.

I don't remember now the origin of the thought to summarize the installation steps and basic steps after installation to make the system more useable, this is not important. Anyway, the slides containing all of this information (and more) were made.
The slides were originaly meant for CZOSUG Bootcamp where people brought their laptops and installed Solaris on them. I have created the slides together with Jan Pechanec . After watching both external people and new-hires struggle with basic steps after completing Solaris installation I think they could be used also to ease those post-install how-do-I-set-this steps.

They are not perfect, could contain errors (please do report them) but here you go:

Also, do not forget to look at recently founded Immigrants community at which contains other goodies such as links to Ben Rockwood's Accelerated introduction to Solaris. Also do not forget to subscribe to the Immigrants mailing list.


blog about security and various tools in Solaris


« April 2014