Tuesday Oct 20, 2009

O'Reilly's Beautiful Testing (Thanks Adam, Tim, Emily, Derek...)

Emily Chen and I coauthored a chapter in an O'Reilly book entitled "Beautiful Testing" which was edited by Adam Goucher and Tim Riley.

Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed ideas of beauty in testing a wide range of products -- valuable knowledge that you can apply to your own projects.

The electronic version is already available here: http://oreilly.com/catalog/9780596159825

The bound version should be available in bookstores in the next few days.

Sunday May 03, 2009

Django framework, a good balance of convenience and dicipline

Some of the strongest criticisms of Django come from users of PHP or other frameworks which allow you to do almost anything in the templates. This can encourage developers to put business logic into the presentation layer which eventually results in an unmaintainable mess. It's the web2.0 equivalent of a GOTO statement.

But on several occasions while working on SourceJuicer it really would've been handy to be able to store or increment variables or directly access a slightly complicated query in the template. For example, I was attempting to pass a list indexed by an existing column (JobID) into a template when I noticed that there was no way to convert the JobID string to a numeric within the template. Then Luis reminded me that if it seems like you're trying to do too much in the template, you're probably trying to do too much in the template. So I went back to the model and saw that I could add the 'slightly complicated' query to an access method in my model. Django does allow you to access methods from the template but it doesn't allow you to pass parameters to these methods. This seems a reasonable balance between strictness and ease of use. It simplified the code while maintaining good separation between business logic and presentation.

Monday Nov 24, 2008

Good news about a bank

We often teased the shyest member of my family by reminding her of the bad joke about the kid who didn't speak a word until he was eating breakfast on his 7th birthday when he said, "My porridge is cold!" When asked why he never spoke before, he said, "Up until now everything has been alright." This is how I felt about the silence which followed my work on a project which installed and provided support for over 7000 opensource JDS desktops at a bank.1 We called the customer occasionally to see if everything was O.K. We helped them through one upgrade which was necessary because the Linux kernel needed to be upgraded to support modern hardware but didn't have a stable ABI so the entire application stack also had to be upgraded. After the upgrade, one of our customers gave us some upgrades/minute statistics that were well beyond what is possible given network bandwidth limitations so I'll just say that the upgrade went well.

Shortly after the upgrade, we helped solve a peculiar focus bug whose root causes were spread across gtk, Java, Firefox and Star/OpenOffice. But overall things were very quiet. Sun was also quiet about this deployment, first of all because we hadn't yet finalized the disclosure agreement and later because Sun decided to drop our Linux-based desktop product and focus on OpenSolaris. So between our "are you still there?" pings to the customer's 2 person technical support staff, I was left wondering if no news is good news?

Then when I gave my presentation at the Irish Opensource Technology Conference, I noticed that two knowledgeable IT managers from this bank were giving presentations on their opensource desktop (a.k.a. JDS) roll out. I finally had the opportunity to be the "fly on the wall" and hear how things really went. I don't have links to their presentations, but these gentlemen said that the project was a success, that the deployment saved money and IT support costs compared to traditional Microsoft Windows based desktop solutions. They said the project completed ahead of schedule and under budget and that they were telling other banks the secrets of their success. I don't know if the other banks were paying attention to the potential savings in deploying opensource alternatives back when easy money was still flowing, but I would think they should take a hard look at such cost-effective alternatives now. In any case, it seems likely that the number of successful cost-saving "invisible" opensource deployments is understated.

"The art and science of interface design depends largely on making the transaction with the computer as transparent as possible in order to minimize the burden on the user" -- S. Joy Mountford

"Any sufficiently advanced technology is indistinguishable from magic!"
-- Arthur C. Clarke.

1 The deployment was of Sun's linux based "Java Desktop System." If we were to do it now, the obvious choices in Sun's product portfolio would be Solaris 10 or OpenSolaris. Since the customer's network is now fast enough to support Sun Ray over WAN, we could potentially save them another $500,000 in annual electricity costs by deploying their desktop via Sun Ray clients instead of X86 PCs.

Thursday Nov 13, 2008

Django newbie gotchas

I'm working on a project which makes use of the Django framework. Django is a web framework which abstracts the back-end database away from the python object and view models which use that database. Since (aside for a small netbean project) I've stayed away from web development for several years, I seem to have missed a lot of noise around ASP, Perl and PHP web development. I hope I haven't missed much. From the looks of the internet, It wouldn't surprise me if only half the web-developers out there only understand half of what they know about web development. There is more than a grain of truth in the saying "If cities were built like web sites, the first woodpecker would destroy civilization".

I'm finding Django 1.0 somewhat challenging, but I'm also finding that I can pull off usable web pages and forms without too much trouble. The django documentation is well above average (especially djangobook.com), and the IRC and mailing list communities are active and helpful. I did run into a few gotchas. Here are some of them:

[Read More]

Tuesday Sep 09, 2008

Is Sun behind more than 1/2 of all corporate OpenSource contributions?

Corporate FOSS Contributions
The EC's Study on the Economic impact of open source software on innovation and the competitiveness of the Information and Communication Technologies (ICT) sector in the EU is an interesting document. I was fortunate to attend a presentation by one of the authors at an OpenIreland event a couple of years ago. The above StarCalc graph uses corporate FOSS contribution numbers from this document. A picture is worth 1000 words, isn't it? This study was published in November 2006, the same month Java was GPL'd so I doubt the Java codebase was included in these calculations. The open sourcing of Solaris was also in early stages. Add these and the MySQL code and it wouldn't surprise me if more than 1/2 of the corporate contributed OpenSource code is from a division of Sun Microsystems. I know we can do better. But quite a few big FOSS consumers (e.g. those selling beautifully branded FreeBSD or web services) are notably absent from the top 10 corporate contributor list. What percentage of corporate FOSS contribution would quench some of the hottest alternate kernel fanboy flames? 60%, 75%? Is it sufficient to contribute to the whole software ecosystem Joe sixpack thinks of as "Linux" or do you have to commit directly to Linus's kernel? What does it take to be cool? Are we there yet?

Tuesday Jul 15, 2008

Thanks to the organizers of GUADEC 2008!

Istanbul Sunset

Many thanks to the organizers, volunteers and Gnomers who helped make GUADEC 2008 possible! There were a few glitches (please open up SSL and VPN ports next time!), but overall it was a success. I also appreciate being able to present my talk on measuring the GNOME footprint. If I'd anticipated the interest, I would have gone into more detail regarding the extensive use of dtrace in this study, but John Rice covered that well in his talk. I'll try to post the details here as soon as possible. I also appreciate the fact that my friend Andrei, a Google Summer of Code student, is eager to use some of my techniques in his memory profile study. It was great to see my friends again and I look forward to seeing them again at a future GUADEC!

The attached photo is an QtpfsGUI tonemapping of a RAW image I took from the roof of our hotel one night. Istanbul is truly an amazing city and though this image doesn't exactly capture the experience of being there, I hope it captures something of the mood.

Friday May 23, 2008

HDR Images with Qtpfsgui

Blossoms near estuary Malahide Castle woods photo

I've been experimenting with tone mapping RAW images with the Qtpfsgui OpenSource tool. It's always difficult to get the sky and foreground things (like spring tree leaves or cherry blossoms) to be simultaneously exposed properly. In the chemical photography days, you'd have to dodge and burn in tiny areas of the photo to properly compress reality's tonal range onto the limited dynamic range of paper prints. I'd like to build Qtpfsgui for OpenSolaris, but it looked like the GNU/Linux versions depended on a particular development environment. Now I see that some versions seem to have been built with cmake which is portable to OpenSolaris. I'll let you know if I figure out how to build it.

Friday May 02, 2008

Linux crash at 35000 feet (what happened to QA?)

The Aer Lingus Airbus 330-300 that my family took to the U.S. a few weeks ago was equipped with video screens on every seat. Each passenger had the choice of several video games, songs, TV shows and movies. My wife watched Cecelia Ahern's tearjerker "P.S. I love you", and after checking the options, I started the same movie about 30 minutes later on my screen and started a Snoopy/Charlie brown cartoon on my daughter's screen. It's pretty impressive when you think about it. The Airbus 330-300 can hold over 200 passengers and I'd expect that decent quality small-screen video requires about 1MB/sec. Hmm, come to think of it, 200MB/sec isn't that impressive, I think my old G3 powerbook usually managed to keep up with that speed of video data on its firewire port, but somewhere, one or more CPUs is very busy uncompressing up to 200 streams of video data.

The most interesting part of our experience came when my daughter decided that the sardonic "Peanuts" cartoon wasn't what she wanted so I hit the home menu button about 3/4ths of the way through her movie. The very instant I hit the home menu button, her screen went black and began booting a version of Red Hat Linux. Then I noticed that my screen and my wife's screen were also booting Linux. I looked around me, everyone's screen was scrolling the ugly text of the Linux kernel boot sequence, a tiny penguin peered from the upper left corner of each screen. "Did I do that?", I wondered. A few minutes later everyone was back to their movie as though nothing had happened. My wife commented that people have become so accustomed to computers crashing that this didn't surprise anyone. It looks like I'm not the first to have seen Linux crash onboard a flight. Another friend told us of a similar failure which wasn't resolved even after someone came from the cockpit on their flight in a failed attempt to repair and reboot the same entertainment system. Now if it might be necessary for someone to come from the cockpit to fly an entertainment system, shouldn't that entertainment system go through some good A?

Even with the occasional crash, X86 hardware running Red Hat Linux with MythTV or some other PVR software may be adequate for an in-flight entertainment system. Let's just hope the software used for fly-by-wire and air traffic control is more robust and more well-tested.

Tuesday Nov 27, 2007

Obama steps into the open file format debate

During a talk at Google, Barack Obama mentioned that he would like to promote "universally accessible formats" for government information. News analysts are wondering why Obama would want to enter this debate. Government de facto support for proprietary standards owned by Microsoft impacts nearly everyone by forcing us to buy Microsoft's products. But this is understood only by a small technical minority and so normally wouldn't get any play in an electoral culture centered around "sound bites" and visceral reactions to pseudo issues which impact virtually no one but which make for emotive campaign advertisements. The fact that Obama mentioned file formats is probably an accident, but it would be nice to pretend that he is trying to drag electoral debates up to a higher level.

Tuesday Sep 25, 2007

Soffice is compatible with actual math instead of Microsoft

Thanks to Slashdot for revealing a serious bug in Microsoft Excel 2007. It seems that in Microsoft's popular spreadsheet application, multiplying values which would result in a product of 65535 (e.g. 10.2\*6425), produce wildly inaccurate results.

[Read More]

Monday Sep 03, 2007

Irish standards authority disapproves (with comments) OOXML ISO fast track

The National Standards Authority of Ireland (NSAI) has voted Disapproval-with Technical Comments in response to Microsoft's OOXML submission to ISO/IEC JTC 1 as a candidate ISO/IEC standard. While Ireland's conditional disapproval might not carry the clout of China's and India's no votes, it is reassuring that such a small country with a big Microsoft presence has the courage to say no.

[Read More]

Saturday Aug 18, 2007

Google trends, ODF vs OOXML

I was wondering whether Microsoft's OOXML was as popular as the Open Document Format (ODF) in the U.S. I expected that since Microsoft's market stranglehold is more complete in the U.S., OOXML should be more popular in there but less popular in Europe and the rest of the world. But according to Google trends there are very few people searching Google for OOXML anywhere. Counts of files on the web also show that beyond a few odd websites such as Microsoft.com, very few people are publishing documents in Microsoft's OOXML format but hundreds of thousands of documents are being published in the open ODF document standard.

I wonder why Microsoft is going to all of the trouble of creating yet another open document standard when a very popular one already exists which can be made compatible with Microsoft Office by simply installing an ODF plug-in?

Thursday Jul 19, 2007

Internet Fences, is IIS "ProvincialNET" a bug or feature?

Me a Fence and Horse on Inish mor, Aran Islands August 2001

"Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offence.
Something there is that doesn't love a wall,
That wants it down.' I could say 'Elves' to him,
But it's not elves exactly..."

From "Mending Wall" by Robert Frost.

In applying for my residency permit and dealing with other Irish government issues, I've noticed that some important Irish citizen and resident information websites are unavailable. Browsing to most government websites results in connection timeouts and other failures which indicate either overuse, undersizing or configuration issues with the servers. I've since found that the problem appears to be that the websites are unavailable outside of a few local ".ie" domains. Here are some examples:

[Read More]

Wednesday Jun 20, 2007

Avoid storage lock-in (goodbye NTFS hello ZFS)

I wanted to transfer a few gigabytes of photos to my father's external USB storage drive a few days ago . I copied the photos from my laptop's ZFS partition to a FAT32 partition so Ubuntu on the same laptop could read them. My father's External USB drive was preformatted NTFS which meant our Solaris, Ubuntu Linux and OSX machines could read it, but none of these could write to it! (Yes I did try the Ubuntu FUSE plug-in, no dice.) This got me to thinking, what is the best cross platform filesystem? Most digital camera and USB keychain drives use FAT or FAT32 as their least common denominator filesystem. The assumption is that everyone is either running a variant of Microsoft Windows or something which can read this ancient kludge of a filesystem (FAT32 somehow reminds me of VGER in Star Trek 1.) As drives grow larger, FAT scalability problems appear and NTFS can't be reliably read or written to by anything other than Microsoft Windows. Clearly we need a new common file system but what should it be? HFS and HFS+ are pretty good, but not popular and getting old. One of my filesystems classes used Amiga's weird FFS as an example of something which had clear advantages over FAT, but even though its scalability and usability were fantastic by 1980s standards, it doesn't quite cut it for 2007. Ditto for UFS. For a while it seemed that ReiserFS was gaining traction in the GNU/Linux community but license and other issues prevented it from growing outside of the GNU/linux community. ZFS has some clear advantages and it's becoming apparent that it will at least be an option on Apple's OSX 10.5 leopard operating system. And now ZFS is available on GNU/Linux. The fact ZFS will be available on 3 out of the 4 operating systems which run on my family's computers convinced me that ZFS can meet my personal storage requirements. Now we only need to convince one more company to consider adopting a 21st century filesystem. Unfortunately that company seems to have a long history of subverting standards in order to lock consumers into its monopoly. Let's hope consumers have learned something from history.

Friday Apr 20, 2007

Watching kernel modules load

I'm taking a course in Solaris Internals. This bit of information our instructor mentioned isn't in the textbook and might be useful to others. How to watch kernel modules load:

/usr/openwin/bin/xterm -C
In resulting terminal do this:
mdb -kw
Loading modules: [ unix krtld genunix specfs dtrace uppc ufs ip sctp usba uhci fctl nca s1394 lofs random nfs audiosup sppp crypto ptm ipc ]
> moddebug/W 0,8000000
moddebug:       0               =       0xfec4cd3c
> $q

Now load a module.

modload /kernel/drv/lofi
# modload /kernel/drv/lofi  
Apr 20 12:15:08 solarisvaio genunix: kobj_open: '/kernel/drv/lofi'
Apr 20 12:15:08 solarisvaio genunix:  vp = d65b7f00
Apr 20 12:15:08 solarisvaio genunix: kobj_close: 0xd65b7f00
Apr 20 12:15:08 solarisvaio genunix: Holding /kernel/drv/lofi
Apr 20 12:15:08 solarisvaio genunix: Releasing /kernel/drv/lofi

Tuesday Apr 10, 2007

Risk of closed proprietary "RAW" digital imaging formats

After I published the entry on building the dcraw "RAW" image format converter for OpenSolaris I found an interesting post on the photography forums regarding Pentax PEF files:
Default Opening Pentax \*ist DS RAW .PEF files in Windows 98? Pentax say on the box that the Windows system requirement is Windows ME/2000/XP and they're not kidding... However, the software with the \*ist DS (including the software for processing RAW images) requires some methods in Share32.dll which simply don't exist in Win98...[Read More]

Wednesday Mar 07, 2007

OpenIreland's Open Source Briefing

I attended OpenIreland's Open Source Software Intelligence Briefing last Friday. (The linked brochure document is a PDF which became an open standard in January 2007. If you aren't able to open this document with Adobe's proprietary acroread product, try open source evince.)

Rishab Ghosh gave some interesting highlights from the FlossImpact report released for the E.C. last autumn. I understood his explanation that Sun's contribution to opensource may be overstated because some OpenOffice.org code was developed outside Sun and retains a Sun copyright. But since these statistics were gathered, 10 million lines of OpenSolaris and I don't know how many millions of lines of Java were open sourced. I think it's a safe bet that Sun still comes out as the leading contributer to the free and open source software movement.

If Douglas Heintzman faithfully represented IBM's philosophy towards open source in his "The Reality" talk, we are very close to the same wavelength. Who can argue with the fact that very few are making money selling packaged software? I believe those few profitable shrink-wrap software companies represent monopolies in their particular market. Who can argue with the fact that software was free long before "open source" became a popular buzzword and long before the Linux kernel was a gleam in Linus's eye? Who can argue with the fact that encoding important Government documents to a proprietary format controlled by a single company is an issue of sovereignty? Shortly before this conference I trolled E.U. and Irish government websites for documents in proprietary Microsoft formats. It's scary how much is locked to one company, especially when the goal is to have these documents viewable 100 years from now.

Who can argue with the fact that too many standards is a bad thing, which is why I hope neither Microsoft nor IBM take advantage of open sourced Java and create their own incompatible releases. I was aware of enormous opportunities for document indexing and display technologies now that we aren't tied to a closed binary format that lives and dies at the whim of one company, but I wasn't aware how far remote collaborative editing and role-based access to portions of an ODF document has progressed. This would fit in well with Sun's role-based access control and trusted JDS desktop. (Which if you think security is a good thing, could be the most under-marketed Sun product since Sun Ray.) I was happy to hear that Mr. Heintzman was impressed with Sun's accessibility demonstration in Germany. Imagine if the important content of business and government documents could be viewed regardless of whether you are a sighted person on a Linux, Microsoft or OpenSolaris desktop or a blind executive viewing the document on a Nokia phone.

I also spoke with employees from OpenApp who were doing really cool things with open source geographic database software but who understandably wondered why Ireland, which has become an IT powerhouse, is so slow to accept the open source paradigm. I also spoke with some very good people from the Blackrock education center. OpenApp and Blackrock worked together to develop the first European Computer Driver's License (ECDL) training material for OpenOffice.org. I gave them copies of Solaris Express Developer Edition. If open source continues to gain acceptance in the E.U., training companies such as Blackrock will play a crucial role in helping people learn new ways of increasing workplace productivity.

Tuesday Jan 23, 2007

Whoah! Microsoft's 6000 page OOXML standard ECMA fast-tracked by Feb 5?

Standards are good. Open standards are better. But when a company with a long history of anti-competitive "standards" submits a 6000 page proposal for its new "open standard" to ECMA and fast-tracked so that any objections must be raised within a few weeks,doesn't it make you wonder what they're up to? As the groklaw article above notes:

Only qualified "P" member bodies of the ISO/IEC Joint Technical Committee 1 ("JTC-1") have legal standing to submit such objections. Fast-track processing is the default JTC-1 status for the proposed Ecma-376 standard, more informally known as the Ecma Office Open XML ("EOOXML") standard.

All we can do is try to make sure that those with the power to slow this decision so that it can be properly considered are aware of this fast-approaching deadline. If Microsoft's lock-in history is any indication, missing this deadline could become one of the costliest mistakes in the computer industry.

Wednesday Aug 30, 2006

libumem traces an evince leak to libz/zlib, is the bug in your dist?

Someone at Sun noticed that the evince process on a recent Solaris Express build consumes a huge amount of memory when it loaded a particular pdf document from IBM. pmap -x showed that it was definitely heap space and that it grew over time. A big leak! I logged the bug in bugzilla because it didn't seem to be specific to Solaris. I used a couple of malloc dtrace scripts but the libumem technique on Adam Leventhal's blog turned out to be the easiest way to track the source of the leak to libz. This build of Solaris was using zlib 1.1.4. When I built zlib 1.2.3 and ran evince with that library, the problem disappeared and performance improved significantly.

For some reason, Adobe still seem to reluctant to recompile and release acroread on Solaris X86. Maybe if enough people ask nicely here, they will do this. But its reassuring to know that the alternative opensource pdf viewer 'evince' continues to improve. It looks like pdf form support is coming soon, thanks to the google summer of code project.

Thursday Aug 24, 2006

pygtkconsole pygtk tutorial update

OpenSolaris Nevada build 46 contains GNOME 2.14.x, python and pyGTK. The pyGTK tutorial has an example of a program which lets you play with Gtk widgets from the python console. Unfortunately the example is based on an earlier version of pyGTK in GNOME 2.14, this should work:

>>> import pygtk
>>> import gtk
>>> w=gtk.Window()
>>> b=gtk.Button('Hello')
>>> w.add(b)
>>> def hello(b):
...     print "Hello, World!"
>>> b.connect('clicked',hello)
>>> w.show_all()
Python is pretty cool, but some of the bindings aren't necessarily write once, run everywhere yet.

Wednesday Jul 05, 2006

GUADEC followup: evolution mallocs by callstack

Evolution memory usage has been a sore spot for many GNOME desktop users, including those of us who run GNOME on a thin client. In his GUADEC talk on GNOME performance, Federico Mena Quintero mentioned that it would be useful to profile evolution memory usage. This entry shows the dtrace oneliner which profiles arg0 (malloc size) for the calls to malloc during the startup of the evolution process. During our talk I tried to also profile mallocs by callstack, but Federico's blog on this topic, indicated that profiling evolution-data-server mallocs while visiting all email folders might be useful. Here is the dtrace script:

#!/usr/sbin/dtrace -s

   @memwhere[ustack()] = quantize(arg0);
I call it with:
./g_evolutionmalloc.d -x aggsize=10m -c /usr/lib/evolution-data-server-1.2/evolution-data-server-1.6  > edatamalloc.out

(aggsize is the size of the aggregation buffer). Then I launch evolution and visit all of the folders in my mailbox. The output is quite large, remember for each unique callstack which calls malloc, I quantize the size of the malloc. Using the oneliner shows the overall distribution of sizes by evolution-data-server when I run evolution and browse all of my folders:

    value  ------------- Distribution ------------- count
               0 |                                         0
               1 |                                         27
               2 |@                                        311
               4 |@@@                                      663
               8 |@@@@@@@@@@                               2046
              16 |@@@@@@@@@@@@                             2553
              32 |@@@@@@@                                  1471
              64 |@@@                                      721
             128 |@                                        177
             256 |@                                        142
             512 |@                                        132
            1024 |                                         91
            2048 |                                         21
            4096 |                                         5
            8192 |                                         48
           16384 |                                         2
           32768 |                                         0
           65536 |                                         4
          131072 |                                         0
          262144 |                                         2
          524288 |                                         0

Lots of 8 and 16 byte mallocs, is this death by a thousand cuts? Gslice (and/or Solaris's slab allocator) should at least help reduce the time for these small mallocs. But where are we when these allocs take place? You are here. (bzip2 compressed because our roller admin doesn't like files > 3M.) As you might expect, most of the unique callstacks only have one value for arg0 (within a power of 2). Here are a few excerpts that caught my eye: Calendar timezone stuff calls malloc with a wide range of sizes:


           value  ------------- Distribution ------------- count
             512 |                                         0
            1024 |@@@                                      1
            2048 |@@@                                      1
            4096 |@@@@@@@@@@                               3
            8192 |@@@@@@@@@@@@@@@@@@@@                     6
           16384 |@@@                                      1
           32768 |                                         0

And there are some individual calls to malloc from calendar timezone code which are relatively large.


           value  ------------- Distribution ------------- count
          131072 |                                         0
          262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
          524288 |                                         0

Uggh, too much data! So I can add a predicate to the original script so I only have look at malloc calls between 8 and 32 bytes:

#!/usr/sbin/dtrace -s

/arg0 > 8 && arg0 < 32/
   @memwhere[ustack()] = quantize(arg0);

That does thin it out a bit, but there are still 578 unique callstacks. Maybe I could sum the total amount of memory allocated for each callstack and only print the ones which exceed 1M? I'm open to suggestions.

Similar memory profiles of the evolution process itself are also interesting. During evolution startup and folder browse, there are 30 malloc requests for 64K from this evolution process callstack:


           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 30

If anyone finds this useful, I can upload a dump of this or any simililar information.

Friday Jun 30, 2006

GUADEC dtrace, normalized time spent in Cairo

Those attending the GUADEC performance bof may be interested in this little dtrace script based on normtime.d:
#!/usr/sbin/dtrace -s
#pragma D option quiet

         \* Get the start time, in nanoseconds.
        start = timestamp;

        @func[probefunc] = count();

         \* Normalize the aggregation based on the number of seconds we have
         \* been running.  (There are 1,000,000,000 nanoseconds in one second.)
        normalize(@func, (timestamp - start) / 1000000000);

run it with cairo.d -c gedit and play with gedit:
  _cairo_ft_unscaled_font_keys_equal                                0
  cairo_font_iface_init                                             0
  cairo_font_map_iface_init                                         0
  cairo_font_options_set_hint_metrics                               0
  gdk_cairo_rectangle                                               0
  pango_cairo_fc_font_class_intern_init                             0
  pango_cairo_fc_font_map_class_intern_init                         0
  pango_cairo_fc_font_map_init                                      0
  pango_cairo_font_map_new                                          0
  pango_cairo_renderer_class_intern_init                            0
  pango_cairo_renderer_get_type                                     0
  pango_cairo_renderer_init                                         0
  _cairo_cache_init                                                 0
  _cairo_font_face_init                                             0
  _cairo_ft_unscaled_font_init                                      0
  _cairo_hash_table_create                                          0
  _cairo_unscaled_font_init                                         0
  cairo_font_options_set_antialias                                  0
  cairo_font_options_set_hint_style                                 0
  cairo_font_options_set_subpixel_order                             0
  cairo_line_to                                                     0
  _cairo_ft_font_face_create                                        0
  _cairo_ft_font_face_scaled_font_create                            0
  _cairo_ft_scaled_font_create                                      0
  _cairo_ft_scaled_font_font_extents                                0
  _cairo_ft_unscaled_font_create_for_pattern                        0
  _cairo_scaled_font_font_extents                                   0
  _cairo_scaled_font_init                                           0
  _cairo_unscaled_font_destroy                                      0
  cairo_ft_font_face_create_for_pattern                             0
  cairo_scaled_font_extents                                         0
  _cairo_matrix_get_affine                                          0
  cairo_ft_font_options_substitute                                  0
  pango_cairo_fc_font_map_context_substitute                        0
  _cairo_ft_unscaled_font_init_key                                  0
  _cairo_ft_unscaled_font_map_lock                                  0
  _cairo_ft_unscaled_font_map_unlock                                0
  _cairo_hash_string                                                0
  _cairo_hash_table_insert                                          0
  _cairo_hash_table_resize                                          0
  _cairo_matrix_compute_determinant                                 0
  _cairo_matrix_compute_scale_factors                               0
  _cairo_surface_composite_trapezoids                               1
  _cairo_traps_extents                                              1
  _cairo_xlib_surface_composite_trapezoids                          1
  _cairo_array_append                                               2
  _cairo_array_grow_by                                              2
  _cairo_clip_intersect_to_region                                   2
  _cairo_surface_fill_region                                        2
  _cairo_user_data_array_set_data                                   2
  cairo_surface_set_device_offset                                   2
  cairo_surface_set_user_data                                       2
  cairo_xlib_surface_create                                         2
  gdk_pixmap_ref_cairo_surface                                      2
  gdk_x11_cairo_surface_destroy                                     2
  gdk_x11_ref_cairo_surface                                         2
  pango_cairo_font_map_create_context                               2
  pango_cairo_font_map_get_default                                  2
  pango_cairo_context_set_resolution                                2
  cairo_font_options_create                                         2
  pango_cairo_context_set_font_options                              2
  _pango_cairo_fc_font_new                                          3
  pango_cairo_fc_font_get_type                                      3
  pango_cairo_fc_font_init                                          3
  pango_cairo_fc_font_map_create_font                               3
  pango_cairo_fc_font_map_get_type                                  3
  cairo_matrix_scale                                                3
  pango_cairo_fc_font_map_context_key_copy                          3
  _cairo_gstate_clip_and_composite_trapezoids                       3
  _cairo_gstate_fill                                                3
  _cairo_surface_clip_and_composite_trapezoids                      3
  _cairo_surface_fill_path                                          3
  cairo_fill                                                        3
  cairo_fill_preserve                                               3
  _cairo_image_surface_assume_ownership_of_data                     3
  _cairo_ft_scaled_font_glyph_extents                               3
  _cairo_ft_unscaled_font_create_glyph                              3
  _cairo_image_surface_create_for_pixman_image                      3
  _cairo_pixman_format_create                                       3
  _cairo_pixman_format_create_masks                                 3
  _cairo_pixman_format_destroy                                      3
  _cairo_pixman_format_init                                         3
  _cairo_pixman_image_create_for_data                               3
  _cairo_pixman_image_get_data                                      3
  _cairo_pixman_image_get_depth                                     3
  _cairo_pixman_image_get_height                                    3
  _cairo_pixman_image_get_stride                                    3
  _cairo_pixman_image_get_width                                     3
  _cairo_scaled_font_glyph_extents                                  3
  cairo_image_surface_create_for_data                               3
  cairo_scaled_font_glyph_extents                                   3
  _cairo_clip_reset                                                 4
  _cairo_gstate_reset_clip                                          4
  _cairo_gstate_translate                                           4
  cairo_identity_matrix                                             4
  cairo_reset_clip                                                  4
  cairo_translate                                                   4
  gdk_cairo_create                                                  4
  gdk_window_ref_cairo_surface                                      4
  cairo_font_options_destroy                                        5
  _cairo_unscaled_font_reference                                    6
  _cairo_gstate_show_glyphs_draw_func                               6
  _cairo_scaled_font_get_glyph_cache_key                            6
  _cairo_scaled_font_show_glyphs                                    6
  _cairo_surface_show_glyphs                                        6
  _cairo_xlib_surface_show_glyphs                                   6
  _cairo_xlib_surface_show_glyphs8                                  6
  _cairo_ft_scaled_font_glyph_bbox                                  6
  _cairo_gstate_backend_to_user                                     6
  _cairo_gstate_ensure_font_face                                    6
  _cairo_gstate_ensure_scaled_font                                  6
  _cairo_gstate_set_font_face                                       6
  _cairo_gstate_set_font_matrix                                     6
  _cairo_gstate_set_font_options                                    6
  _cairo_gstate_show_glyphs                                         6
  _cairo_path_fixed_get_current_point                               6
  _cairo_scaled_font_glyph_bbox                                     6
  _cairo_xlib_surface_get_font_options                              6
  _pango_cairo_font_install                                         6
  _pango_cairo_font_map_get_renderer                                6
  cairo_get_current_point                                           6
  cairo_scaled_font_destroy                                         6
  cairo_scaled_font_reference                                       6
  cairo_set_font_face                                               6
  cairo_set_font_matrix                                             6
  cairo_set_font_options                                            6
  cairo_show_glyphs                                                 6
  cairo_surface_get_font_options                                    6
  pango_cairo_fc_font_install                                       6
  pango_cairo_fc_font_map_get_renderer                              6
  pango_cairo_renderer_draw_glyphs                                  6
  pango_cairo_show_glyph_string                                     6
  _cairo_scaled_font_keys_equal                                     6
  cairo_scaled_font_create                                          6
  pango_cairo_fc_font_get_font_face                                 6
  cairo_font_face_reference                                         6
  _cairo_hash_table_lookup                                          6
  _cairo_scaled_font_init_key                                       6
  _cairo_hash_table_lookup_internal                                 6
  gdk_cairo_region                                                  6
  _cairo_clip_clip                                                  6
  _cairo_clip_init                                                  6
  _cairo_color_compute_shorts                                       6
  _cairo_color_init_rgb                                             6
  _cairo_color_init_rgba                                            6
  _cairo_gstate_clip                                                6
  _cairo_gstate_create                                              6
  _cairo_gstate_set_source                                          6
  _cairo_path_fixed_init                                            6
  _cairo_surface_allocate_clip_serial                               6
  _cairo_surface_get_clip_mode                                      6
  _cairo_surface_set_clip_region                                    6
  _cairo_xlib_surface_set_clip_region                               6
  cairo_clip                                                        6
  cairo_clip_preserve                                               6
  cairo_create                                                      6
  cairo_destroy                                                     6
  cairo_pattern_create_rgb                                          6
  cairo_set_source                                                  6
  cairo_set_source_rgb                                              6
  gdk_cairo_set_source_color                                        6
  pango_cairo_fc_font_get_metrics                                   7
  _cairo_matrix_is_integer_translation                              7
  _cairo_pattern_acquire_surface                                    7
  _cairo_pattern_acquire_surface_for_solid                          7
  _cairo_pattern_release_surface                                    7
  _cairo_surface_create_similar_scratch                             7
  _cairo_surface_create_similar_solid                               7
  _cairo_surface_fill_rectangle                                     7
  _cairo_xlib_surface_create_similar                                7
  _cairo_xlib_surface_set_attributes                                7
  _cairo_xlib_surface_set_repeat                                    7
  cairo_xlib_surface_create_with_xrender_format                     7
  get_cairo_context                                                 7
  _cairo_box_round_to_rectangle                                     8
  _cairo_clip_intersect_to_rectangle                                8
  _cairo_gstate_clip_and_composite                                  8
  _cairo_region_create_from_rectangle                               8
  _cairo_pixman_region_num_rects                                    9
  _cairo_pixman_region_rects                                        9
  _gdk_drawable_ref_cairo_surface                                   9
  cairo_font_options_merge                                          9
  cairo_font_options_copy                                           9
  _cairo_font_options_init_default                                  9
  _cairo_array_fini                                                 9
  _cairo_surface_fill_rectangles                                    9
  _cairo_user_data_array_fini                                       9
  _cairo_xlib_screen_info_get                                       9
  _cairo_xlib_surface_create_internal                               9
  _cairo_xlib_surface_fill_rectangles                               9
  _cairo_xlib_surface_finish                                        9
  cairo_surface_finish                                              9
  _cairo_gstate_copy_transformed_pattern                           10
  _cairo_gstate_copy_transformed_source                            10
  _cairo_pattern_init_copy                                         10
  _cairo_pattern_transform                                         10
  _cairo_pixman_region_extents                                     10
  _cairo_pixman_region_intersect                                   10
  _cairo_region_extents_rectangle                                  10
  _cairo_surface_get_current_clip_serial                           10
  _cairo_surface_set_clip                                          10
  cairo_matrix_translate                                           10
  cairo_matrix_init_scale                                          10
  _cairo_path_fixed_fill_to_traps                                  10
  _cairo_path_fixed_interpret                                      10
  _cairo_polygon_fini                                              10
  _cairo_polygon_init                                              10
  _cairo_traps_extract_region                                      10
  _cairo_traps_fini                                                10
  _cairo_traps_grow_by                                             10
  _cairo_traps_init                                                10
  _cairo_traps_tessellate_polygon                                  10
  _cairo_gstate_apply_device_inverse_transform                     11
  _cairo_gstate_apply_device_transform                             11
  _cairo_gstate_identity_matrix                                    11
  cairo_rectangle                                                  11
  pango_cairo_context_get_resolution                               11
  pango_cairo_fc_font_map_get_resolution_fc                        11
  _cairo_filler_close_path                                         11
  _cairo_path_fixed_close_path                                     11
  cairo_close_path                                                 11
  _cairo_traps_add_trap                                            11
  _cairo_get_global_image_glyph_cache                              12
  _cairo_lock_global_image_glyph_cache                             12
  _cairo_unlock_global_image_glyph_cache                           12
  _compare_cairo_edge_by_current_x_slope                           12
  pango_cairo_fc_font_map_context_key_equal                        12
  cairo_font_face_destroy                                          12
  _compare_cairo_edge_by_top                                       13
  _cairo_surface_init                                              13
  _cairo_array_init                                                13
  _cairo_path_arg_buf_create                                       13
  _cairo_path_fixed_add_arg_buf                                    13
  _cairo_path_fixed_add_op_buf                                     13
  _cairo_path_op_buf_create                                        13
  _cairo_pattern_create_solid                                      13
  _cairo_user_data_array_init                                      13
  _cairo_stock_color                                               14
  _cairo_xlib_surface_set_picture_clip_rects                       14
  _cairo_filler_move_to                                            14
  pango_cairo_font_map_get_type                                    15
  cairo_new_path                                                   15
  _cairo_format_from_content                                       15
  _cairo_fixed_integer_ceil                                        16
  _cairo_ft_scaled_font_get_glyph_cache_key                        16
  _cairo_clip_init_copy                                            17
  _cairo_gstate_clone                                              17
  _cairo_pen_init_copy                                             17
  cairo_restore                                                    17
  cairo_save                                                       17
  _cairo_xlib_surface_ensure_dst_picture                           17
  _cairo_operator_bounded                                          17
  _cairo_path_fixed_move_to                                        18
  cairo_move_to                                                    18
  _cairo_cache_shrink_to                                           18
  cairo_font_options_equal                                         19
  _cairo_fixed_integer_floor                                       19
  pango_cairo_fc_font_map_context_key_get                          19
  pango_cairo_font_get_type                                        19
  _cairo_scaled_font_map_lock                                      19
  _cairo_scaled_font_map_unlock                                    19
  cairo_matrix_init_translate                                      19
  pango_cairo_fc_font_map_context_key_hash                         20
  _cairo_restrict_value                                            20
  _cairo_pattern_init                                              21
  _cairo_pattern_init_solid                                        21
  _cairo_path_fixed_fini                                           22
  _pango_cairo_context_get_merged_font_options                     22
  _cairo_clip_fini                                                 24
  _cairo_gstate_destroy                                            24
  _cairo_pen_fini                                                  24
  cairo_pattern_reference                                          24
  _cairo_pixman_region_union                                       25
  _cairo_pattern_fini                                              25
  _cairo_pixman_region_union_rect                                  25
  cairo_ft_scaled_font_lock_face                                   26
  cairo_ft_scaled_font_unlock_face                                 26
  pango_cairo_fc_font_lock_face                                    26
  pango_cairo_fc_font_unlock_face                                  26
  cairo_font_options_hash                                          27
  _cairo_pixman_region_destroy                                     29
  _cairo_ft_unscaled_font_lock_face                                30
  _cairo_ft_unscaled_font_set_scale                                30
  _cairo_ft_unscaled_font_unlock_face                              30
  _cairo_pen_init_empty                                            31
  cairo_matrix_multiply                                            33
  _cairo_gstate_user_to_device_distance                            35
  _cairo_path_fixed_rel_line_to                                    35
  cairo_rel_line_to                                                35
  _cairo_filler_line_to                                            35
  _cairo_path_fixed_line_to                                        35
  _cairo_polygon_line_to                                           35
  _cairo_gstate_unset_scaled_font                                  35
  _cairo_polygon_close                                             37
  _cairo_pixman_region_create                                      37
  _cairo_pixman_region_create_simple                               37
  _cairo_pixman_region_copy                                        38
  cairo_pattern_destroy                                            38
  cairo_surface_destroy                                            39
  _cairo_fixed_integer_part                                        40
  pango_cairo_fc_font_get_glyph                                    41
  cairo_surface_reference                                          47
  _cairo_polygon_add_edge                                          50
  _cairo_gstate_user_to_backend                                    52
  _cairo_fixed_is_integer                                          57
  cairo_matrix_init_identity                                       59
  _cairo_polygon_move_to                                           64
  _cairo_path_fixed_add                                            65
  _cairo_cache_lookup                                              74
  _cairo_glyph_cache_keys_equal                                    75
  cairo_matrix_transform_point                                     82
  cairo_matrix_init                                                89
  _cairo_glyph_cache_hash                                          92
  _cairo_fixed_to_double                                          108
  cairo_matrix_transform_distance                                 117
  pango_cairo_fc_font_get_glyph_extents                           201
  pango_cairo_fc_font_get_scaled_font                             254
  _cairo_fixed_from_double                                        315

When I have more time I'll try this on the benchmark and also try it in a thin client environment where I expect things will be different. But now it is time for lunch!

Wednesday Apr 12, 2006

Linux company still runs Windows????

I assumed that Sun's "eat thy own dogfood" policy wasn't terribly unusual. So I was suprised to learn that Novell still runs Microsoft Windows on many of its corporate desktops. In the spirit of fun competition, I wonder if Johnathan or whoever has the numbers and authorization could publish:

How many JDS (SuSE Linux based) desktops do we run?
How many JDS (Solaris/OpenSolaris based) desktops do we run?
How many GNOME desktops (including Sun Ray) do we run?
How many Windows "dual boot" desktops we run?
How many GNOME (Linux and Solaris based) desktops have we delivered to our customers?

I don't have the answers but it would really surprise me if we didn't beat SuSE both in number of SuSE based (JDS1,JDS2) desktops and number of Solaris GNOME (JDS3+) desktops deployed internally. We certainly have a lower percentage of Microsoft Windows and Microsoft office desktops! I wonder if SuSE could let us in on what is taking them so long to migrate away from MS Windows. The obstacles to their migration could be useful for others in the open source desktop community. I'd have to admit that Sun doesn't have as much experience with internal migration from Microsoft Windows because most of us could never justify migrating to MS-Windows in the first place.

Monday Feb 27, 2006

StarOffice Computer Driver's Licence


The European Computer Driving Licence is a standard of computer literacy training which is recognised by many European employers. I'd thought ECDL training was focused only on Microsoft Office products, but this weekend I met a man who told me about OpenApp and Blackrock Training Center's Training for ECDL: A complete Course in StarOffice 8. This is great news! It makes sense to have training standards which aren't focused on a single closed proprietary product. After all, my automobile driver's licence isn't just valid for the 1969 American Motors Rambler (or was it the '68 Fury?) that I learned on.

The photo is of dad's 1966 AMC Rambler 232, somewhere in the desert of the southwestern U.S. in the mid 1970s. It doesn't have nearly enough tail fins and chrome strips for a car of that era. Maybe that's what caused their decline in popularity. Oddly, a couple of the Japanese models imported to Europe this year remind me of the AMC Pacer.

Thursday Jan 05, 2006

Wisconsin requires source access to evoting software

After the year 2000 florida election fiasco, some states decided to hide the problem behind a computer running close source software. Thankfully my old home state of Wisconsin held out for a better solution requiring a paper trail and:

5.84 (3) If a municipality uses an electronic voting system at any election, the municipal clerk shall provide to any person, upon request, at the expense of the municipality, the coding for the software that the municipality uses to operate the system and to tally the votes cast.


The law isn't perfect, how would the average citizen know that the screen he/she voted on isn't hooked up to a closed source box running a random number generator? A receipt should be required but it should be encrypted in such a way to discourage vote selling. Also, what if a municipality mistakingly uses a closed source product such as Microsoft Windows instead of an open source product such as OpenSolaris in evoting machines? Will Oshkosh, Wisconsin be required to purchase the source code to Microsoft Windows? And the municipality must pay for providing the source code to "any person?" Wow, even if they used a typical linux distribution and slashdot got wind of it... this could get expensive! Another question which would be familiar to fans of GPL is, where do you stop? As written it appears that the law requires the municipality to provide source code for the entire system. This would probably include the voting software, the operating system, BIOS, drivers, peripheral cards (video, I/O...) and the firmware of CRTs, printers. It's a commendable goal, but I suspect it will be a while before electronic computers can match the pure openness of other vote tallying solutions.

Fortunately, the "If" at the start of paragraph 5.84 allows non-electronic solutions and the expense of distribution favors simplicity and open source. Let's just hope the municipalities make wise choices.

Wisconsin doesn't yet require that the voter present a form of identification and this law doesn't seem to have any A11Y, i18n requirements, voter assurance or prevention of vote selling. But it is an improvement over laws passed in other states requiring electronic voting but not requiring a paper trail or any access to evoting internals.

Pay no attention to that man behind the curtain. -- The Wizard of Oz.

Monday Dec 19, 2005

Microsoft recommends more modern browser

I'll admit that I experienced a tiny disturbance in the force when I heard that Microsoft will end support for Internet Explorer on Apple computers. But then when I saw:

Microsoft will end support for Internet Explorer for Mac on December 31st, 2005, and will provide no further security or performance updates.

I chuckled at the reference to security and performance updates. IE seems to bring our old powerbook and every other computer within a 5km radius to a screeching halt as it swaps itself in as though it owned the memory resources of the planet. I suspect that no one has willingly chosen IE for the past 5 years. Only those with no alternative would use it. Fortunately there are alternatives. From Microsoft's announcement:

It is recommended that Macintosh users migrate to more recent web browsing technologies such as Apple's Safari.

Apple Browsers with recent technologies such as tabbed browsing, configurable popup blocking and lightning speed include safari, firefox, mozilla, netscape, opera... in fact practically every web browser except for Internet Explorer. I wonder if this will help convince web developers to develop standards compliant web sites. Or will they continue ignoring the growing alternative browser and Apple market share?

Tuesday Nov 01, 2005

Nexenta: Debian + OpenSolaris

This is interesting. From the posting:

What is Nexenta OS today

Several things:

1) A working prototype that runs on our 32-bit laptops and AMD64 box; the
latter is used for development of the system itself, and it in turn runs our
entire development environment, and hosts the web portal (below).

2) 2,300 Debian packages available for immediate usage.

3) Developer's portal at http://www.gnusolaris.org - fully functional, with
downloads, APT repository, discussion forums, developer's "hack zone", bug
database, blogs, and numerous Solaris and free software related resources.

This will be 100% open and free-of-any-charge easy-to-install easy-to-use
distribution. Coming out soon!

I'm glad to see that the opensolaris community continues to grow. I hope this will make it easier for the traditional news media (and a few people here at Sun) to understand the relationship between open source software, Linus's kernel and Open Solaris. This new distribution was also mentioned on Slashdot, though for some reason they put it in the Linux category.

gtkperf 0.4 with scriptable features, builds on opensolaris

Kaj Grönholm, the gtkperf maintainer upgraded gtkperf to 0.4. He added new command line options which make it easier to run it from within a test script. It also builds cleanly on opensolaris. Thanks Kaj!

Saturday Oct 29, 2005

JDS source is on opensolaris.org

Java Desktop System is now available for building on opensolaris.org! I test drove Laca and Gman's Building JDS instructions and scripts several times on a sparc and an X86. Builds can take quite a long time on average hardware, but the docs and build scripts make it fairly simple.

Friday Oct 28, 2005

Avoid Wintel meltdown with open source utility

I've used Solaris, Linux and Apple desktops almost exclusively for the past few years. But about a year ago we purchased a brand-name centrino laptop for use with my wife's embroidery and patchwork software and as a VNC client to a powerbook with a broken display. It didn't surprise me that alternative OSs often require new drivers or tweaks in order to get power management working. What does surprise me is that this laptop, which was designed for Microsoft Windows, has terrible power management when used with the preinstalled XP professional OS. When I close the lid it sleeps. When I open the lid, it fails to wakes up about 40% of the time. Most of the time it gives a "drive letter component failure" error and offers to email microsoft. About half the time it wakes up without restoring network access. Repairing the network doesn't work so I must reboot. This completely defeats the purpose of sleep mode. What's more, running a CPU intensive application caused the laptop to overheat and crash ungracefully. Fortunately there is a solution to the overheating problem. The fan utility on the Toshiba Linux utility website can be used to force the fan on. Open source software can help those who use Microsoft Windows.

P.S. A few months ago while making myself a cup of coffee, Damien Farnham explained to me how dtrace scripts can be used for power management. I hadn't thought about all of the possible uses of dtrace's awesome instrumentation capabilities. Maybe we can get Damien, Casper or Bryan to explain how it works.




« July 2016