Friday Aug 31, 2007

Adding custom load URL probes to Mozilla

Hi - having fun at the minute with an onsite workshop on DTrace with the Mozilla folks in Menlo Park. Brendan Gregg treated us to a bunch of live DTrace demos that really showed how you can get that system wide view across the stack that is impossible with conventional tools :) Now if we could just clone Brendan that would be ideal. The real learning comes from doing and just watching how Brendan formulates questions then explores the problem with dtrace, doing some quick profiling, identifying areas of interest, dumping ustacks, grabbing functions, checking timings and so on. The syntax is easy, the mind set is a lot tougher to capture.

The reason we are over with Mozilla having the workshop in the first place is to help work on the DTrace Tracing framework in Mozilla, which should really help the community identify and fix performance issues in Mozilla, be it in the Javascript XUL layer, the C++ layer or in some dependent platform or system lib.

Having a framework is pretty cool, but not much use if you have no useful probes built on top of it. Brendan has created Javascript probes which hopefully will help dig into a lot of performance issues in this layer. In addition to the Javascript probes we want to add some C++ probes to track things like page loading.

With Johnny Steinbeck's help, Brendan and crew added loadURI_start / done probes.

Here's a quick brain dump of what we needed to do to add them. I'll blog later when we start using them to do some interesting timing analysis.


Using new Load URI Probes

$dtrace -nl mozilla\*::mozdtrace_load_uri_start:load_uri_start

$ dtrace -ln "moz\*:::"
   ID   PROVIDER            MODULE                          FUNCTION NAME
71986 mozilla18214          mozdtrace_load_uri_start load-uri-start
71987 mozilla18214          mozdtrace_load_uri_done load-uri-done

$ dtrace -qn moz\*:::'{printf("%s\\n",copyinstr(arg0));}'

Adding Load URI Probes


Add the new dtrace probes:

 \* load__uri__start        (char \* uri)
 \* load__uri__done        (char \* uri)

provider mozilla {
 probe load__uri__start(char \* );
 probe load__uri__done(char \* );


Add probe lib to MOZILLA_PROBE_LIBS:

MOZILLA_PROBE_LIBS = $(DEPTH)/staticlib/components/$(LIB_PREFIX)docshell.$(LIB_SUFFIX)
MOZILLA_DTRACE_SRC = $(DEPTH)/mozilla-trace.d
include $(topsrcdir)/config/

Note: Adding probe lib to MOZILLA_PROBE_LIBS here so object file with probe can be post processed by dtrace -G
Processing step pulled in to the make from mozilla/config/

    dtrace -G -C -32 -s $(MOZILLA_DTRACE_SRC) -o $(DTRACE_PROBE_OBJ) $(OBJS)

-G: tells dtrace to process the list of .o's looking for USDT probes, change
their status to IGNORE and generate a special <probe>.o that you should link
against so the kernel linker can correctly initialize things when the shared
lib is loaded.
-C: use the C preprocessor to process the specified .d file, allows you to have
-32: compile for 32 bit architecture, we need to change this so it
will work on 64 bit arch as well
-s: source of probes, mozilla-trace.d in this instance
-o: output special <probe>.o file, call it anything you want, just need to link
against it.


Note: If building debug, components of XUL shared lib are not linked into it, so need to add probe libs below.

        $(LIBS_DIR) \\
        $(EXTRA_DSO_LIBS) \\
        $(MOZ_JS_LIBS) \\
        $(MOZ_COMPONENT_LIBS) \\

MOZILLA_DTRACE_SRC = $(DEPTH)/mozilla-trace.d

include $(topsrcdir)/config/

        -I$(srcdir) \\


Add wrapper functions for the probes to give sensible function names, otherwise get mangled C++ enclosing function names :(

#include "mozilla-trace.h"

extern "C" {
void mozdtrace_load_uri_start(nsIURI \* aURI);
void mozdtrace_load_uri_done(nsIChannel \* aChannel);




Note: Tried putting MOZILLA_LOAD_URI_START_ENABLED() around wrapper func, mozdtrace_load_uri_start, but get two func probes
if we do this.
6202 mozilla23090             __1cKnsDocShellJDoURILoad6MpnGnsIURI_2ipnLnsISupports_pkcpnOnsIInputStream_8ippnLnsIDocShell_ppnKnsIRequest_i_I_ load-uri-start
6203 mozilla23090             mozdtrace_load_uri_start load-uri-start


Define wrapper probe functions and add custom probe macros to load URI start and end points.

#include "mozilla-trace.h"

// Implementation of probe func wrapper
void mozdtrace_load_uri_start(nsIURI \* aURI)
        nsCAutoString spec;

        MOZILLA_LOAD_URI_START((char \*)spec.get());

void mozdtrace_load_uri_done(nsIChannel \* aChannel)
   /\* XXX DTrace load-uri-done \*/
       nsCOMPtr<nsIURI> uri;
       nsCAutoString spec;

       MOZILLA_LOAD_URI_DONE((char \*)spec.get());


nsDocShell::DoURILoad(nsIURI \* aURI, ....)

     /\* DTrace URI probe \*/

    rv = DoChannelLoad(channel, uriLoader);

nsDocShell::EndPageLoad(nsIWebProgress \* aProgress,
                        nsIChannel \* aChannel, nsresult aStatus)
    nsCOMPtr<nsIDocShell> kungFuDeathGrip(this);
// NOTE: if this is moved to the top of the function then we get two USDT probes instead of one:
// mozilla7863 __1cKnsDocShellLEndPageLoad6MpnOnsIWebProgress_pnKnsIChannel_I_I_ load-uri-done
// mozilla7863           mozdtrace_load_uri_done load-uri-done

DTrace request for enhancement:

Allow an optional function name to be specified in probe description:

provider mozilla {
 probe mozdtrace_load_uri_start:load__uri__start(char \* );
 probe mozdtrace_load_uri_done:load__uri__done(char \* );

This would be so handy when putting USDT probes into C++ code. No need for the wrapper funcs we are using above :) 



Tuesday Jun 26, 2007

Mercurial and C/C++ Support in NetBeans 6.0 - well almost

Hi - just thought folks might be interested. Got the latest NetBeans 6.0 building from head which has the CND C/C++ support included and ran it using the Mercurial Plugin against a simple Mercurial Hello world project that was already under Mercurial control.

The good news is that it actually comes up and things seem to work, well kinda (check out the status window below). Now if you click on the .hg directory in the Projects window all hell breaks lose, but then no ones perfect ;) We'll work on this over the next few weeks or so to flush out the issues, but so far so good.

 Mercurial and C/C++ support


Tuesday May 08, 2007

Adding your own VCS support to NetBeans 6.0 thanks to Maros' Skeleton

Note: 26th June, few changes, added externally visible link to VCS API doc and removed step due to platform API changes. 

Hi - a few folks expressed interest in doing what we've just done for Mercurial using Maros Sandor's VCS skeleton for NetBeans 6.0 with their own Version Control Systems. Andy's comment finally prompted me to put down a few rough guidelines. Make sure to touch base with the NetBeans Version Control folks as they can offer plenty of guidance and make sure your module is created in the correct location and so on.

Maros has written a VCS API Overview (Externally accessible) of the key integration points that is a really useful starting point. There are already skeletons available for Mercurial, Perforce and Clearcase. Mercurial and Perforce are under active development.


You will need to get developer access on the NetBeans versioncontrol project
$ cvs -d co 
  • Create you stub module under versioncontrol/<your vcs stub>
    • As a basis grab the mercurial skeleton code that Maros very kindly provided for us.
$ cvs -d 
export -D "8 Feb 2007" -r mercurial versioncontrol/mercurial
Change the skeleton code to whatever name you want and check it in as your own skeleton 
vcs stub. There are already existing stubs for Mercurial, ClearCase and Perforce, that
can be checked out directly
The skeletons have a very useful list of TODO's in the code that act as good set of starting 
points, see below.
  • [Jun 26th - No Longer Required due to Platform API changes] 
    As your module will not yet be in standard build, you need to patch the
    versioncontrol/nbproject/project.xml file to include <Your VCS Module> in its
    list of friends:
                <friend>org.netbeans.modules.<Your VCS Module></friend>
  • Build the IDE by running "ant build-nozip" from the nbbuild directory
    •  cd netbeans/nbbuild
    •  ant build-nozip
  •  Launch NetBeans 6.0 - Open Project versioncontrol/<Your VCS Module>
    •  Right click on <Your VCS Module>l and Build with dependencies
    •  Run - IDE launches and under Versioning see <Your VCS Module> submenu


Look at the Mercurial Plugin and the Subversion Plugin code to get hints on what to do for your own VCS. 

$ find . -exec grep "TODO" {} \\; -print
        entry = null;     
        // TODO: read your detailed information about the file here, or disregard the entry field
        // TODO: insert you status logic here, device status of a file, cache it and return it
        // TODO: make all files and folders uptodate for now
        // TODO: in case the NEW file status does not match OLD file status, fire a change event
        // TODO: FAKE now make all files uptodate and files beginning with "m" as modified
        // TODO: use your own logic here
        // TODO: scan the given folder and return information about containing files
        // TODO: do not put up-to-date files in the map
        // TODO: fake, now only return existing files and make them all uptodate
        // TODO: insert your logic here
        // TODO FAKE: what lies inside mercurial-root folder is managed by mercurial
        // TODO: make locally modified files blue, leave others default
        // TODO; new files should be green, ignored files gray, etc.
        // TODO: ignored folders should be gray


Sunday May 06, 2007

Mercurial support in NetBeans - not just wishful thinking :)

[Read More]

Friday Mar 09, 2007

Adding Javascript Dtrace probes to Mozilla

Hi, as a follow on from my earlier post on adding dynamic instrumentation to mozilla, just wanted to let you all know that we have now put up a patch for Brendan Gregg's Javascript probes that will apply on top of the infrastructure patch we posted earlier :)

All of the probes listed in Brendan's Blog on JavaScript and DTrace will work:

Brendan's Javascript blog

Demo of the Javascript probes in action

But we have changed the namespace to be in sync with the other mozilla probes, so you will need to change the probe names in the script appropriately, for example:

Change: javascript\*:::function-entry to trace_mozilla\*:::js_function-entry

Applying the Javascript patch

Follow the instructions below to apply the Mozilla Infrastructure and layout patch:

Adding Dtrace probes to Mozilla

Then apply the Mozilla Javascript probe patch:

  • Apply mozilla-js.diff downloaded from bug 370906

$ cd mozilla
$ gpatch -p0  -i mozilla-js.diff

Rebuild mozilla as described above, not forgetting to run ./configure –enable-dtrace before hand. Test the probes have been applied by running the newly patched firefox-bin and listing the probes as described below.

Available Javascript Probes

To list the available javascript probes once the patch is applied and mozilla rebuilt with --enable-dtrace, just type:

$ cd mozilla/dist/bin
$ ./firefox-bin &
$ dtrace -n
'trace_mozilla\*:::js\*' -l
ID  PROVIDER             MODULE                  FUNCTION              NAME
35 trace_mozilla1815 jsdtrace_execute_done js_execute-done 35 trace_mozilla1815 jsdtrace_execute_done jsdtrace_execute_start 35 trace_mozilla1815 jsdtrace_execute_done js_execute-done



This script counts all of the javascript function calls that are being executed by firefox and lists the originating file for the javascript code, listed under FUNC and FILE respectively. You could use a predicate to exclude javascript being used by the browser's chrome UI if you wished, but this simple script just counts all of the function calls. See Brendan's blog for lots more examples :)

D option quiet
printf("Tracing...Hit Ctrl-C to end.\\n";);
@funcs[basename(copyinstr(arg0)), copyinstr(arg2)] = count();
printf("%-32s %-36s %8s\\n", "FILE", "FUNC","CALLS";);
printa("%-32s %-36s %@8d\\n", @funcs);
dtrace -s js_funcalls.d


FILE           FUNC                             CALLS
autocomplete.xml createEvent 1
autocomplete.xml dispatchEvent 1
: browser.xml QueryInterface 130
nsSessionStore.js QueryInterface 987
nsSessionStore.js getNext 988
nsSessionStore.js hasMoreElements 990

Wednesday Mar 07, 2007

Adding Dtrace Probes to Mozilla

Hi - we've been looking at adding a general framework for dynamic instrumentation to mozilla. On OpenSolaris this means dtrace probes :) The blurb from the submitted  bugzilla RFE gives you the general idea:


The main goal for this Dynamic Tracing Framework for Mozilla is to provide a common interface for adding instrumentation points or probes to Mozilla so its behavior can be easily observed by developers and administrators even in production systems. This framework will allow Mozilla to use the appropriate monitoring/tracing facility provided by each OS. OpenSolaris, FreeBSD and MacOS will use DTrace and other OSes can use their own respective tool.

All the gory details are up with the bug including the full proposal:

Bug 370906: [RFE] Dynamic Tracing Framework for Mozilla

Example 1 

A little sample output from Firefox built with --enable-dtrace and using the layout probes. The layout probes fire when a page is being loaded by Firefox. The phases that are triggered during the auto layout are frame construction, reflow [elements on the page need to be changed] and paint. All of these phases can be tracked by the probes, the times taken for each phase and whether one phase has triggered other phases, such as reflow causing new frame construction.

This script just counts the number and type of each phase that is triggered for a specific Presentation context. 

$ dtrace -qs layout_count.d -p `pgrep firefox-bin` -o out.txt

Snip from layout_count.d [see bug for full details]:
@[arg0, phaseStr[arg1], probefunc] = count();

Sample Output

$ cat out.txt | c++filt 
Pres Ctx Phase Count Function
153556424 [ paint ] 2 unsigned PresShell::Paint(nsIView\*,
nsIRenderingContext\*,const nsRegion)
169258584 [ frameCon ] 2 unsigned nsCSSFrameConstructor::CharacterDataChanged
140305640 [ frameCon ] 4 unsigned nsCSSFrameConstructor::ContentRemoved

This information is very important when tracking down hard to debug reflow bugs or when trying to understand performance timing data for specific pages. As we work with the probes we will start to look at how Firefox is interacting with the system as these phases are being triggered, such as what XServer traffic is being generated. This will be possible by combining the Firefox layout probes and the Xserver probes.


We are hacking away to improve the patch given feedback from the mozilla folks. But thought others might enjoy having a go now, so I've put down some general instructions if you are keen to try this on OpenSolaris :) We are putting in some layout probes which allow you to see the progress of frame construction, reflow and paint as a new page is loaded as a proof of concept. We will also provide a patch to add Brendan Gregg's Javascript probes in the next few days which should be an interesting adjunct to Mozilla's Javascript debugger. The hope is that others who have a lot more knowledge of the mozilla code base will start adding their own probes, once the infrastructure is in place to support them. The patch is still in review so a ways to go yet, but hopefully as others use them, the benefits will become clear. We have tried to make sure we minimize the probe impact and by default dtrace is not enabled in the build, you must build mozilla with ./configure --enable-dtrace to add any of the dtrace infrastructure.


All of the patches discussed below can be obtained from: Bug 370906


Getting Sources

  • Mozilla sources
    • Grab the mozilla sources from head: Mozilla CVS
    • If you need to do this from behind a firewall, you can use runsocks and set your CVS_RSH=ssh and your SOCKS5_SERVER=<socks server>
$ export CVSROOT=""
$ cvs login -> just hit return for password
$ cvs co mozilla

Applying Patches

  • Mozilla infrastructure + layout probe patch:
    • To get the diff patch, just go to the bug  370906 and look for "Mozilla Dynamic Tracing Framework + layout probes patch" in the Attachment table, click on it and do a save as mozilla-layout.diff
    • Apply mozilla-layout.diff downloaded from bug 370906 
$ cd mozilla
$ gpatch -p0 -i mozilla-layout.diff

Building Mozilla

  • Build mozilla
    • Setup mozilla for building:
$ export LD_LIBRARY_PATH=/usr/sfw/lib
$ export CC="/usr/bin/cc"
$ export CXX="/usr/bin/CC"
$ export CFLAGS="-xlibmil"
$ export CXXFLAGS="-xlibmil -xlibmopt -features=tmplife -norunpath"
$ export LDFLAGS="-R'\\$\\$ORIGIN:\\$\\$ORIGIN/..' -R/usr/sfw/lib"
  • Create a .mozconfig file under mozilla/.mozconfig containing the following:
ac_add_options --enable-xft
ac_add_options --disable-freetype2
ac_add_options --disable-tests
ac_add_options --disable-debug
ac_add_options --enable-svg
ac_add_options --enable-canvas
ac_add_options --enable-optimize
ac_add_options --enable-xinerama
ac_add_options --disable-auto-deps
ac_add_options --enable-application=browser
. $topsrcdir/browser/config/mozconfig
  • You must run autoconf to regenerate configure with the appropriate dtrace options; it needs to be version 2.13.
$ autoconf
$ ./configure --enable-dtrace
Note: this will generate a mozilla-trace.h from the mozilla-trace.d if one is not
present in the mozilla root directory.
$ gmake

Testing the Probes

  • The probes should now be present
    • Check Layout Probes are there.
$ cd mozilla/dist/bin
$ ./firefox-bin &
$ dtrace -n 'trace_mozilla\*:::layout\*' -l

Example 2

This sample script is using the layout start and end probes to time the various layout phases as they occur when a page is loaded by Firefox. The script will print out the Presentation context for each phase [identifies which context each phase is being triggered for], which allows them to be grouped together, along with the time taken for each phase and the function which is triggering that phase.

#!/usr/sbin/dtrace -s
#pragma D option quiet

enum eLayoutPhases { EPaint, EReflow, EFrameC};
phaseStr[EPaint] = "paint";
phaseStr[EReflow] = "reflow";
phaseStr[EFrameC] = "frameCon";

self->ts[probefunc, arg0, arg1] = timestamp;

/self->ts[probefunc, arg0, arg1]/
@phase_time[arg0, phaseStr[arg1], probefunc ] =
sum((timestamp - self->ts[probefunc, arg0, arg1])/1000);
self->ts[probefunc, arg0, arg1] = 0;

printf("%10s %10s %8s %-30s \\n", "Pres Ctx", "Phase", "Time(usec)", "Function" );
printa("%10d [ %8s ] %5@d %-30s \\n", @phase_time);

/x++ == $1/
{ exit(0);}

Sample Output

$ dtrace -qs layout_time.d -p `pgrep firefox-bin` -o out.txt
$ cat out.txt | c++filt
Pres Ctx Phase Time(usec) Function
156084568 [ frameCon ] 34 unsigned nsCSSFrameConstructor::CharacterDataChanged
141161216 [ frameCon ] 91 unsigned nsCSSFrameConstructor::ContentRemoved
141161216 [ frameCon ] 331 unsigned nsCSSFrameConstructor::ContentInserted
141161216 [ reflow ] 3458 unsigned PresShell::ProcessReflowCommands(int)
152558200 [ paint ] 6216 unsigned PresShell::Paint(nsIView\*,
nsIRenderingContext\*,const nsRegion )
141161216 [ paint ] 90459 unsigned PresShell::Paint(nsIView\*,
nsIRenderingContext\*,const nsRegion )


Wednesday Jun 14, 2006

OpenSolaris - Lá breithe mhaith agat! from Ireland

Well first and foremost a very happy birthday to you :) OpenSolaris. It's been too long since I blogged it's true, but some times you just fall off the rails, so new year resolution to get back on the horse. If I haven't mixed enough metaphors for you yet, humble apologies ;) \*Looking Back\* It's been an interesting year for the "Desktop": group on OpenSolaris. With releases poping up like "Belenix": , "Shillix": and "Nexenta": , people have certainly been busy. Our own group got "JDS": out on OpenSolaris thanks to Gman, Dermot, Laca and a cast of thousands. The latest Vermillion builds (JDS 4) have been going up regulary now and the latest Gnome 2.14 based cut is integrated into "nevada": . So happy days indeed. What's too like, hey it's Gnome 2.14 based, what's not to like ;) Love evince, love the nimbus theme (very cool Erwann and Mike), love rhythm box. Love the performance improvements, gdm login flying along, menus poping up at lightening speed, nimbus 3x faster than blueprint, gedit and gterminal on fire, by the way did I say nimbus rocks :) \*What's Next\* Can't wait for all the HAL/ DBUS pieces to get there, so we can really start to kick butt on removable media, camera, ipod, you name it. Also support for Avahi coming down the pike and lots of discussion on how to make printing totally awesome (shorthand - easy for mortals ;) Lots of cool work being done to port the Gnome System tools onto JDS, coming to a store near you pretty soon. Then we have "Ekiga": for all you meeting junkies. Further out, well maybe not, give me sizzle - 3D accel support, already there for nvidia, coming this summer for intel and end of year for ati. This will really help us have fun, where are my transparent desklets and fancy fade outs, I know, I know it's eye candy, but what can I say I love it! \*Developer's Desktop\* Going forward want to get Desktop Developers flocking to OpenSolaris. Lets give them a happy home where they are really free to invoate, using whatever tools/ lanugages ring their bell. All about freedonm of choice. Make it easy for Gnome developers used to Gtk/ Glade to work with Java on Solaris if they want with Java-Gnome. Tempt them to take the leap to Swing with all the crazy stuff in "Java2D": and Java "timing framework": [anyone seen the "Aerith": demo - it really rocks, and for a Java Desktop app that's worth repeating :) ]. Lots of discussion on the alias about getting all the tools we need to "build OSS on OpenSolaris": just there by default, rock on ... \*A bientot\* So gotta go back and do a little more to help make the desktop rock. Java-Gnome bindings for me at the mo, the joys of pkgtool/ spec files, for another post (see I'm even thinking about my next blog, I must be cured). OpenSolaris rocks, it's fantastic to see the communities evolving and growing, there are a lot of commited folks out there who love working on a Unix with the pedigree of OpenSolaris and one whose future they can now guide. Roll on the 10 yr aniversary, expect to be using my "3D holographic desktop": or did I just wake up from the latest episode of StarTrek or was it Minority Report ;) Who knows ...

Thursday Dec 08, 2005

Solaris Desktop Summit: Performance Day 3

Just a quick update on the Performance Hackfest. Having lots of fun and learning loads. Had plenty of interesting talks this morning. Alan C gave us all the gory details of the XServer, past, present and future. Great to see just what was being planned and all the goodies coming down the pike with XOrg 6.9 and beyond. Oh for hardware acceleration support, but it is coming. Alan gave us an overview of the XServer dtrace provider, looks very interesting. Should be easy to see what xlib calls are being made across the Gnome stack, will allow the XServer guys to tune particulary hot commands.
Bob Doolittle, Bob Terek and Charles Le Brac gave us a great preso on the Sun Ray architecture. It's very helpful to understand precisely what we can and can't do with the Sun Ray device. We really need to play well with our desktop in a multi user server based environment. Plenty of challenges, but with the various teams working together who knows what we can do. Would love to see an interface added in the XServer to allow the Sun Ray to communicate it's knowledge of the current bandwidth environment up the stack, so the Gnome desktop could respond dynamically to limited bandwidth environments, such as when a user goes from the office to their home.
Sean Meighan gave an excellent talk on Canary, the tool used to monitor the 700+ Sun Ray servers throughout Sun. It can give a very dynamic view of the Sun Ray's, their current performance and the performance of various applications running on the server. Was entertaining to see the different work patterns across the various Sun campuses by using Cananry to dynamically monitor the CPU usage. Thankfuly he seemed to miss Dublin ;) \*Performance Update\* \* Firefox: using the Pagein command used by Staroffice to page in libraries used by Firefox [determined using pldd], the lads saw an improvement in cold start from 18 sec to 10 sec. Will want to see if we can use a general preload mechanism to heat up the page cache. Would be interesting if during the compile/ link cycle we could have a build option to indicate to the dynamic linker that it should page in the entire library. This would remvoe the need for an external preload mechanism such as pagein.
\* Brian saved another 1 sec by removing a redundent read of a 20 Meg file from Firefox startup. Dependency patch added to g_open_module to use RTLD_FIRST to reduce lookup of uneeded callback dependenices. Still need to test this against a Firefox startup benchmark. Jack and Patrick have a neat way to timing the startup, by getting to load a piece of Javascript to shutdown Firefox as soon as it starts. \* Timsstamp bug: trying to get a good repeatable benchmark for Firefox startup, we uncovered a bug in the high res timer used by dtrace timestamp. I was seeing a drift of 0.6 sec per sec, when comparing timestamp timings to walltimestamp timings. The problem is being caused when I enable powernow automatic cpu stepper, setting this to manual removes the problem. Following script used to confirm the issue from Charles. #pragma D option quiet BEGIN { ts = timestamp; wts = walltimestamp; } tick-1s { this->dts = timestamp - ts; this->dwts = walltimestamp - wts; printf("%15d %15d %15d\\n", this->dts, this->dwts, (long) (this->dts --- this->dwts)); } \* Nautilus Open: Yukon and Oleg gave us a cool demo of Sun Studio 11 Analyzer. When we ran it on Nautilus and did an open of /usr/bin [1100 files], it takes 9 sec ona cold start to do the listing. Over 6 secs is spent in reading the files to determine the mime type. It is also rereading a number of png files lots of time. Will dig into this tomorrow to see what's going on. Nautilus should be caching its icons for the view so need to see why this isn't happening. Also want to see why the mime type detection isn't just using the extension initially and then in the background do the more expensive read to confirm or adjust the mime type if necessary. \* Bootime: looks like we should be able to have the initial login screen appear some 10-15 sec sooner than it currently does. Need to modify the SMF service definitions for both dtlogin and gdmlogin. The various services still need started, but it will certainly feel faster to the user. This type of behaviour is the norm in Windows, wher user sees the login in 20 secs, but from the disc activity you can see the system is still woring hard int eh background to bring up various services. \* Gtk: icon cache seems to be a little deranged, checking for the default High Color theme cache over a 100 times, to be sure to be sure :( Erwann will dig into this tomorrow. Looking across all the Gtk calls on login see a lot of Gtlkrcpixmap & path calls. \* Java: changing caching of translucent pixmaps to opaque ones leads to 5% improvement. Further Gtk improvements in removing redundent value lookups, will save 2% on the swing mark tests. More changes on the Java Gtk L&F fidelity which Chris will post. \* Bonobo: Alo has a patch for g_rand_new() to remove a redundent read, shaving a few miliseconds off startup of all gnome apps :) \* Sun Ray: few issues uncovered. DDX being more aggresive than it needed to be in hotdesking. XScreensaver doing 0 length polls, need to see why. New brand to be used in Sun Ray on tests showed a 10x improvement in bandwidth usage from the previous one. Always good news, Erwann muttered something about knowing where the old 400k background image came from, but refused to reveal more even after extensive libations in Gordons brewery ;) \* Gnome Terminal: interesting that the 30-70% improvement estimate yesterday by optimising some of the vte redraw code, was not seen when running the modified Gnome terminal on a Sun Ray. Want to use Alan C's XServer dtrace provider to dig into what and how long the various requests are taking to carry out the client redraw requests. Off for a few quick hours shut eye, before I head out for my 6.15am run with Alo. Sanity definitely is not my strong suits :) More dtrace fun tomorrow, awesome.

Tuesday Dec 06, 2005

Solaris Desktop Summit: Performance Day 2

Had plenty of interesting talks in the morning. Bart Smalders gave us a tour de force of what not to do when looking at performance, which was very cool. \*Bart's Anti Patterns\*
\* Fix performance at the end of the project - all been there :( \* Measure the wrong thing - not just regressions, need to compare against what our customers care about, other desktops :) \* Avoid rewriting algorithms at all cost. Best performance gains in Solaris 10 came from going back and rewriting key algorithms, generally where you get the biggest wins. \* What work do we not need to do? Best performance win is not doing anything ;) Avoid needless work, redrawing at a refresh rate beyond human perception makes little sense, but you'd be surprised how many apps do it, hello Gnome-Terminal. Collapse those redraw events where it makes sense. \* Premature optimisations. As Knuth would say, the root of all evil. \* Focusing only on what you can see readily. Better to look more carefully to see what can be tuned higher up the stack. \* Not optimising for the common case. Lay out data structs in traversal order, coallocate data structs, reuse info where it makes sense. General Advice
\* Have real runnable benchmarks that everyone can run easily and regulary. \* Evaluate key algorithms and data structs. \* Find work that doesn't need to be done and get rid of it. \* Tune hot algorithms [last thing to do, not the first] And above all else remember Disc access is slow, random disc access even slower and file access across the network well even slower. In 1 second you can carry out around 1000,000,000 instructions and about 100 disc head moves, not hard to do the maths. \*Linker Tips\* Rod Evan's gave a great talk on all things to do with the Linker and how to tune things to avoid asking the linker to do any more work than it needs to. Lots of LD_DEBUG flags to use to see what exactly is going on with the dynamic linker, namely bindings, symbols, unused. Check out "Rod's blog": More details for another blog ... \*Performance Update\*
Lots of interesting progress with the various breakouts: \* Firefox, on cold boot kinda bizzare seems to be getting to its main gtk loop [gtk_main] and executing various XMapWindow calls in 3.5 sec, but the user isn't seeing the running app for another 9 secs. Need to dig some more, what's the window manager and xserver up to?
Rod's talked sparked some ideas on unused dependencies and Erwann has a patch to remove unecessary gloabl dependency checking for a number of undefined call back functions. Patch just restricts the check to the local lib [use RTLD_FIRST], will benchmark the patch tomorrow. \* Gnome-Terminal, issues around reuse of graphic contexts being created and torn down in the redraw routine. Initial optimisation bere looks like there could be a win of 40-70% :) No doubt Stephen and others will blog the details. \* Boot Time, folks looking at optimising the services dependency graph by removing disabled services. Seeing a 10 sec gain in a 90 sec bootup. More testing to be done, but looks promising. \* Applets, "Glynn": has the details. More to be investigated. \*Fresh Air\* Great trip to Yosemite with "Gman": , though the encounter with the State Police was a little unexpected, story for a pint ... "Another hard day in the office ..."

Solaris Desktop Summit: Performance Day 1

Into performance, lots of fun with a very dream like presentation from Bryan Cantrill ;) We got a whirlwind tour of some of the new features in dtrace in Solaris 10 Update 1. Some very cool stuff indeed. A few random highlights for me: \*Late Binding\*
Hallelujah - this will make such a difference in tracking things down in gnome, where we have so many library dependencies and when doing pid probes often various libs are not loaded in when setting up the pid probes [see pango example below]. \*-Z\*: no probes yet but tell dtrace to ignore this for the minute $ dtrace -n 'pid${@[probefunc] = count()}' -q -c gucharmap > dtrace: invalid probe specifier pid${@[probefunc] = count()}: probe description does not match any probes
$ dtrace -Z -n 'pid${@[probefunc] = count()}' -q -c gucharmap
  arabic_engine_fc_class_init                                       1
  script_engine_create                                              1
  script_engine_init                                                1
  Arabic_Assign_Properties                                         89
  arabic_engine_shape                                              89
  get_ruleset                                                      89
  Get_Joining_Class                                               178
\*String Manipulation\*
All of the string manipulation funcs now supported like substr, index, strtok, strrchr ... #!/usr/sbin/dtrace -s
  self->filename = arg0;

/self->filename != NULL/
 this->filename = basename(copyinstr(self->filename));
 self->filename = NULL;

 this->ext = strrchr(basename(this->filename), '.');
 @[this->ext == NULL ? "": this->ext , this->filename] = count();
Lots of other things including support for default args, new ring buffer policy, support for multiple aggregations in printa, auxillary files, a file descriptor array .... I'll blog these up when I get a few moments \*Performance\* The performance breakouts are underway, we have 6 teams of 5 with a good mixture of different skills looking into things such as Firefox startup, Boot time, Java Application performance, Gnome Terminal, Gtk Performance and Panel Applets. We'll be picking other topics as the week progresses. The Boot Time team already look like they have a win thanks to Brian Cameron, Alan Coopersmith and Bryan Cantrill. Check out "Alan's blog": for all the details. Brian had an initial hack running to optimise the socket gdm code and hopefully will trim off 3 secs from boot time :) We are digging into Firefox and trying to get a good startup benchmark, which is proving surprisingly tricky, but we'll get there. Issue is C++ name mangling, inlined funcs and so on ... Bart has an initial hack with an endpoint on an access of the history.dat file, but we seem to be losing about 6 secs in the timing :( More to follow.

Monday Dec 05, 2005

Solaris Desktop Summit - Day 3

Well survived the first few days and so got time to play in the 3rd day developing user types. Found it really interesting, trying to get inside the user's skin so to speak. \*SMB Sys Admin\* We even turned up a few interesting ideas on Small and Medium sized business System Admin. Basically this person has been sucker punched, is under pressure, would be happy if it weren't for his/her users and generally is not in a good spot. So anything we can do to make their lives easier is good news. Couple of really simple ideas: 1. Login Changes --- lots of users can't login for pretty silly reasons. Help by adding additional info in the login screen or accessible from it. So when a fail to login happens check if cap locks is on and report it. We know what the IP is so make this available for the sys admin to easily find. Often network issues stop login, have a network status indicator, green good to go, red in trouble. 2. System logs --- boy all over the map. At least make it easier to view them all with a log viewer. Second phase would be parse the data in some way to extract meaningful error messages and also provide links to online documentation from the error. Small steps, just get the viewer - next phase would be to see which are the most critical logs and at least get their error info extracted in a way the sys admin can use. \*User Centric Design\* Plenty of other groups looked into the developer, the mobile user, knowledge worker and transaction worker. All the write ups are up on the twiki. What I'd really like to see is us collectively moving to a much more user centric design process. I think Frank Ludolph and the XDesign team have planted the seeds so I'd like to see us take this forward for our key user types [Sys Admin for SMB and Developers]. Would be fun to develop these user personas with input from real users. Here's a few photos from the summit with me sending everyone to sleep:

Friday Dec 02, 2005

Solaris Desktop Summit - Day 2

Well as Glynn mentioned I've been just a little stressed out over the last few days "Gman": I wonder why? Well trying to heard a bunch of 50 very talented and creative folks through the summit agenda may have something to do with it :) Thank goodness for Frank Ludolph's steady hand on the tiller and Jeff Cheeney's many talents, of which the wolf whistle is certainly the most impressive ;) Kudos to all my bosses and bosses bosses for letting me pull this together. It is just fantastic to get all these folks together, to see them chatting, to hear plenty of "ahh I didn't know that's been sorted, great..." So first disaster was no network in Central Park. I thought we all where going to have withdrawal symptoms, as Bart said a disconnected laptop just ain't no fun at all. But once we all got used to the idea of bing disconnected things moved along well. I know plenty of folks had been disappointed that day 1 was more of a lecture format and less interaction/ discussion. Always a problem when you have to frame the discussions in anytype of workshop scenario. On reflection we could have had a Desktop Gripe session in the first day to let folks vent a little, but I am learning (slowly it's true). I'll hesitate before making any further suggestions over a pint to my VP, it's scary when they actually take you seriously ;) \*Glenn\* Kicked off with a preso from Glenn Weinberg, who managed to make it despite having a really bad dose of the some cold/ flu thing. Many thanks to Glenn for dragging himself in. Talked about change on the street, we are back in the spotlight, which is great. Not where we need to be at the minute, but can get there by moving faster, taking more risks. Keep the core Solaris solid, but inovate and experiment with other parts of the stack such as the desktop where our customers want and expect a higher rate of change. In targeting the developer want to give them a superb integrated develoepr apps experience on Solaris for Java and native app development. Solaris 10 adoption rates still cranking along over 3.5 million and no sign of slowing down. The street thinks we are inovators in the OS space but not in User Space. Up to us to prove them wrong. Need to leverage the OpenSolaris community and really experiment in this space, get the source code management sorted. Experiment with different release models, get the latest bits into customers hands who want to be at the bleeding edge, let them see that we are inovating in this space. Starting to see OpenSolaris coming onto University curriculum which is very cool. Need to drive volume and as I'm fond of saying make the Solaris Desktop rock ! \*Brand\* Had a fun preso on the Sun Brand. Very cool stuff from Bruce Lee how brand is all about soul = values + character + vision. Not just about the looks, but when you interact with Sun's products its gotta be a good experience, so behaviour is crucial and then we need to spread the word within Sun as well as outside. Core strengths doing things differently, never give up, brilliance, passion, trust and FUN. Think with all the changes over the past few years a lot of the Fun has gone and it's great to see it coming back. That's a lot of what the summit is about, getting the teams to together and having fun seeing how we can move forward, helping with our focus and so on. Some pretty cool stuff about the Ted prize that Bono won, how Sun technology enabled the texting system at the U2 concerts. \*Sizzle\* Had a sizzle session looking at all the things that the various OS's are up to and the use of GPU graphics. Use of transperancy, transition effects, animation. All very sexy it has to be said, my favouite was the splash effect when you drop a new deskbar widget onto you dashboard on the Mac, utterly useless but hey made me smile and isn't that what it's all about ;) \*Breakouts\* Finally we get to brainstorm, discuss and have a bit of fun. List of topics which we had to cull down to 8. Broke up into groups to look into the issue, explore what the key parts of the problem where and come up with next steps. Thanks to Jeff, Marny, Frank and Dee for facilitating, not a job for the faint hearted. Turned up some interesting things which are all on the "twiki": or will get up there in the next few days. I was in the Single Sys Admin session, which turned up some surprises when we applied the filter of the two main user types we were looking at Sys Admin for SMB's and Developers. Developer top issues --- How to find, Task oriented, Network and ISV. Sys Admin top issues --- Model, Security, Install/Upgrade/Deploy and Fault Management. Java on the desktop, the good news is after all the brainstorming, no real surprises, many of the concerns around performance, integration in the desktop, better look & feel, native support and so on are being addressed in Mustang. The great thing was to see the Gnome heads and Java folks hooking up, which will be the key to giving our Developers the best dam experience possible on Solaris. \*Wrapup\* Lots of interesting things here, but getting late and I need breakfast, so will keep it short. One session on things we could do with the GPU in the Sun Ray DTU. There are limitations with a remote framebuffer, that you can't get around for some of the sizzle type desktop features, but the cool thing is the one key usability feature I'd really love namely Anti ALiased fonts can be supported directly in the DTU. As long as X can tell the Sun Ray it's sending it down text the AA compositing can happen on the DTU. This is just frigging awesome. I'll leave the splash effects and rotating globes on my workstation, but if can get AA fonts on my Sun Ray at home I'll be a very happy man. \*Beer and Pizza\* Need I say more ... bunch on Irishmen, loads of beer. Gman was seen leaving with a whole rack of beer, Alo has the photos. Scary stories of 6.5 litre trucks tearing down the highway at 120 mph ... So off for Desktop Gripe session today which should be a lot of fun and then a bit of a deep dive on User Types under Frank's very capable direction. Then the weekend and I'll collapse in a puddle somewhere. P.s Boy do I need a good back massage, though I suspect nothing short of dynamite is going to get the knots out of my back muscles ;) P.P.s. Thanks to Donna who got me a No Jet Lag remedy to try out when I'm heading home. Will be very interesting to see if it works - would be truely awesome if it did. P.P.P.s Ron Shipper, who has been an absolute star on the Sun Ray organisation front has some cool photos, but as my mail server in ireland seems to be refusing to talk to me, I wonder why, will have to wait for another blog.

Wednesday Nov 23, 2005

Solaris Live Upgrade - the next installment ;)

OK so you've done your first Live Upgrade. See my earlier blog for all the gory details, "Solaris Live Upgrade": Now what about the next one? I assume you have 3 slices (partitions), one Active "/", one inactive to be upgraded and one "/export" that will remain unchanged on upgrade. My Active "/" is the solenv_s3 Boot Environment [BE] and I want to upgrade the old inactive solenv_s0 BE to nevada b27a.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status

solenv_s0                  yes      no     no        yes    ---
solenv_s3                  yes      yes    yes       no     ---

\* Steps \* I need to upgrade some bits on solenv_s3 (frkit & lu tools), copy the current solenv_s3 to solenv_s0 (lumake), upgrade solenv_s0 (luupgrade) and finally make solenv_s0 my new active "/" BE (luactivate). The solenv_s3 then becomes my old inactive BE, ready for the next live upgrade. Here you go:

1. Update frkit
# /usr/sfw/bin/wget http://vaticaan.holland/~casper/tools/frkit ; chmod 755 frkit ; ./frkit <- as root

2. Get nevada iso and mount it
> ftp://nana.sfbay/solaris_media/nv/27a/x86/
> get solarisdvd-iso.gz  [2.23 Gigs]
# gunzip solarisdvd-iso.gz
# lofiadm ---a /export/home/john/build/solarisdvd.iso
# mount ---r ---F hsfs /dev/lofi/1 /mnt

3. Update lu tools from the iso
# cd /mnt/Solaris_11/Tools/Installers/
# ./liveupgrade20

# pkgrm SUNWluu SUNWlur
# pkgadd ---d /mnt/Solaris_11/Product SUNWlur SUNWluu
# pkginfo ---l SUNWluu

4. Copy from current Active BE [solenv_s3] to Inactive target BE [solenv_s0] (1-2 hrs)
# lumake ---n solenv_s0

5. Upgrade the target BE [solenv_s0] (1-2 hrs)
# lupgrade ---u ---n solenv_s0 -s /mnt
[Note: add ---N if you want to do a dry run and check out what's going to happen]

6. Activate target BE
# luactivate solenv_s0

Reboot - DONE! It's getting easier ...

Monday Nov 07, 2005

Solaris Live Upgrade - the joys

Well thought I should upgrade from Solaris 11 (nevada b22) to the latest (nevada b26) on my Ferrari 3400 x86 box, just to see what's new and wanted to install the new Gnome 2.12 tarballs on the latest build. So had heard of Solaris Live Upgrade, the ability to upgrade your system on the fly and thought himm sounds like something to try. So checked out our friendly sun docs "Solaris 10 Installation Guide: Solaris Live Upgrade and Upgrade Planning":http://docsview.eng/app/docs/doc/817-5505/6mkv5m1l0?a=view and wished I hadn't ;) Very long winded, hey I'm an engineer attention span of a gnat when it comes to manuals ;) So in the best engineering tradition went and picked someone else's brain [thanks "Darren": ]. \* Where's Partition Magic when you need it?\* OK so first problem, when I setup Solaris on my dual boot box initially I went for 2 slices (partitions - why slices who knows?), one for root and another for my /export/home. Not so good. In order to use the Live Upgrade I needed another slice to setup my boot environment in :( Now they tell me. So thankfully with a little housekeeping I reduced my home enough to be able to copy it all into the root slice (cpio is your friend). Once that's copied off then I was able to resize my /export/home and create a new slice to hold my boot environment. Once that was done I just copied my home back to the resized /export/home. $ cd /export/home $ find . --- depth ---print | cpio -pvd /export/home2 \* Format \* Boy do I want a good partition manager on Solaris. Please if anyone is listening ... So off to format, then select partition, edit my home slice to change its size, setup one of the unassigned slices to 9 Gig as another root slice and I'm done. No don't forget to type label, himm obvious not. $ format > partition > print - prints out partition table showing slices [partitions] select partition to edit - change cylinder size as required Slice 3 will be our new boot environment, so set it to be same size as current root partition, adjusted slice 7 to give us enough room for new boot env slice: 0 root wm 3 - 5082 9.77GB (5080/0/0) 20482560 3 unassigned wm 5384 - 10463 9.77GB (5080/0/0) 20482560 7 home wm 10464 - 29613 36.82GB (19150/0/0) 77212800 > label > quit Need to create new file system on home slice 7 - will wipe this slice $ newfs /dev/dsk/c0d0s7 \* Grabbing the ISO \* Too painful to describe getting this remotely (/usr/sfw/bin/ncftp is your dearest friend, thank god for resume), but eventually got the nevada b26 iso and cut it to disc: $ cdrw ---iv solarisdvd.iso [ create DVD of ISO image ] \* NOTE: Using the latest Live Upgrade tools \* [Comment from UVR] You need to make sure to be using the Live Upgrade tools from the release you are upgrading too :( I checked that the lu\* packages hadn't changed between b22 and b26, so thought I was safe enough skipping this step. But if they have changed you will get an error. So to be safe: 1. Install the Solaris lu packages from the ISO you have just grabbed above and install them in the current image you are running and want to upgrade from. 2. Use the new lu tools for the live upgrade.

\* Create Boot Environment \*
Now all ready to go. Create your new boot env from the command line (can do it from a curses app, xterm -e lu &, but command line simpler and faster).

$ lucreate ---c solenv_s0 ---m /:/dev/dsk/c1d0s3:ufs -n solenv_s3
Creation of boot environment  successful.
$ lustatus solenv_s3

Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
solenv_s3                  yes      no     no        yes    -

\* Upgrade and Activate \* Now that I have my boot environment and my iso all I had to do to upgrade the boot environment was: $ luupgrade ---u ---n solenv_s3 -s /cdrom/sol_11_x86/ $ luactivate solenv_s3 -> activates new upgraded boot environment To check it out the upgrade before running just put in -N as the first param and this will do a dry run to check out your params and so on. 2 hrs later, the upgrade completed, did the activate, rebooted and it was all done, hooray !! Then I just went and downloaded the Gnome 2.12 tarball and installed it on my new nevada b26 boot environment and all was well with the world. \* Next Time \* This will be so much easier next time round, isn't it always ;) Will just have to grab the ISO, and type the above luupgrade comand targeting my now old version of Solaris solenv_0. Follow this with an luactivate, reboot and we are good to go. \* Install suggestion \* Wouldn't it be nice if you where installing Solaris for the first time on an x86 Windows machine and the installer setup everything by default to make LiveUpgrade easy. So during install it could setup for dual boot, make Solaris the default (optionally), then create two root partitions (slices) and one /export/home. Explaining to the user as you go that this will allow users to upgrade their version of Solaris, whilst still maintaining their current version completely intact.

Monday Oct 24, 2005

JDS/ Gnome Performance - leaner meaner login - thanks Lorenzo

Well - as you know there's been plenty of interest in login times on JDS Solaris. So wanted to give people a little taster for the brave and foolish if they wish strip up to 20% off Gdm login and just can't wait for a 2.14 based release of Gnome ;) \* gconf and login \* With "Lorenzo Coletti's work": in the Gnome community, gconf was identified as a real hog on startup. Reading zillions of little config files in deeply nested directories, not good. He managed to shave quite a chunk off startup time by changing its behaviour to read a single merged settings file incrementally, which was also reduced in size by stripping non default language keys. These changes will be supported in future releases of Gnome, but for now if you'd like to experiment I will explain how below. Note this will work with vanilla Solaris Express builds (nevada), but as I'd need to give you a patched gconf to get the incremental reads that wrinkle can wait for another day. Enjoy, as always your mileage may vary, but I've tested this with no ill effects on a Ferrari 3400 nevada b22, my login went from 25 sec down to 20 sec. Laca has managed to reduce his further to 17-18 sec by using a Gnome stack optimised for lazy loading, which will make it into subsequent nevada builds. Still plenty to do, dare I mention Cold Boot to Gdm [100 sec agggghhh]. Compare to Windows 20 sec cold boot to login, and 3 sec from login to desktop, but Rome wasn't built in a day :) Nor will I give my fellow travellers on the road nightmares by mentioning Mac OSX resume from sleep in 2 sec ;) JR \* Steps: \* 1. Create a merged gconf settings defaults file. 2. Strip all the language keys out of the file except the default "C" locale. 3. Put the merged stripped file in the appropriate location and reboot [then cross your fingers and take out your stop watch ;) ]. Any problems, just go into failsafe mode from Gdm and delete the merged file to reset things. If you change any default settings within JDS, you will need to run through the steps above again to regenerate the new default settings. \* 1. Merged Gconf File \* As root: $ gconf-merge-tree /etc/gconf/gconf.xml.defaults/ Generates: /etc/gconf/gconf.xml.defaults/%gconf-tree.xml Copy this file to ~/gconf-tree.xml [Had to do this as every time I tried to use the %gconf-tree.xml on the bash line with the java command below it entered command mode, no matter what I did - if you tell me how to prevent this happening great] \* 2. Strip the langauge strings \* Run: $ cd ~ $ cp /etc/gconf/gconf.xml.defaults/%gconf-tree.xml gconf-tree.xml $ java -Xmx100m -jar gconf-strip-langs.jar gconf-tree.xml > gconf-tree-stripped.xml \* 3. Put the defaults file in place \* $ cp ~/gconf-tree-stripped.xml /etc/gconf/gconf.xml.defaults/%gconf-tree.xml \* What is the gconf-strip-langs.jar doing: \* It's a simple java program I wrote last night in NetBeans to load up the gconf xml file, trundle through its DOM, stripping all local_schema elements [along with children] whose locale attribute is not equal to "C" and then spitting this modified XML out to stdout. If you want to preserve another locale other than the default "C" locale, just hack the attached java file. Example local_schema node, with short description attribute and long description child node: Empty notes are always deleted without confirmation Sizes before and after stripping: 22478144 Oct 19 20:13 gconf-tree.xml 4100783 Oct 19 20:13 gconf-tree-stripped.xml \* gconf-strip-langs \* "Source:": "gconf-strip-langs.jar":



Top Tags
« April 2014