Friday Dec 05, 2008

asc("") and printf("%s",NULL)

A few days ago I was reminded of one of the differences between Solaris and GNU/Linux which caused a few headaches for Sun's desktop team back in the days of GNOME 1.2. The problem is that while you can printf("%s",NULL); in most Linux distributions, doing the same in Solaris caused the executable to exit and generate a core. There were some debates about the correctness of each approach. Should the program crash to tell the developer that he shouldn't be trying to print a NULL string (the Solaris behavior) or should the program continue happily along printing a NULL pointer? I can see some advantages in both approaches I suspect this and other "Linuxisms" will end up in OpenSolaris simply because they make it more convenient for average coders to throw together a quick and dirty program or for someone to compile and run the thousands of source packages out there which (perhaps unknowingly) take advantage of this Linuxism.

But thinking about this reminded me of a similar "bug/feature" in my very first computer, the Commodore 64 back in 1982 when 64K seemed an unbelievably excessive amount of memory for a computer which cost only $595. The built in blitter and 4 channel 16 bit synthesizer made it a really fun computer for me to write simulations and sound generator programs for my father's physical science class. If you look closely at many of the programs which were published for Compute! and other magazines of the time, you might notice something strange. When a character was read from the user (e.g. via get (a$) ), the asc(a$) function would convert the character to its numeric ASCII value. But in the code you would usually see something like this:

n = asc(a$+chr$(0))

J64 java emulator

What is going on here? There was a bug/feature in Commodore 64 BASIC V2 which raised an "Illegal quantity error." whenever a null string was passed to the asc() function. The Commodore 64's 6510 processor had the unusual ability of being able to peek the ROM and write to shadow RAM which shared the same address space and then disable ROM so that BASIC was running from RAM. This allowed modifications to the BASIC interpreter. Jim Butterfield, a Commodore expert and author once demonstrated a one byte poke which fixed this asc("") bug. Ever since I learned of this simple fix, I wondered why so many BASIC's had this same one byte bug. The Commodore 64, Vic-20, Atari, Amiga, and at least some versions of the Apple and IBM PC Basic's shared this same bug! What was going on? Well, as it happens, a little company known as Microsoft wrote versions of BASIC for nearly all 8 bit computers of the 1980s and 1990s. Was this one byte bug overlooked by Microsoft and propagated to all Microsoft inspired codebases or were Microsoft's developers following the same purist philosophy as Solaris developers who assert that "good coders shouldn't pass NULL into string functions?" Either way, when such a company grows to what it has now become, it can decide that this one byte bug is actually a feature.

Monday Nov 24, 2008

Good news about a bank

We often teased the shyest member of my family by reminding her of the bad joke about the kid who didn't speak a word until he was eating breakfast on his 7th birthday when he said, "My porridge is cold!" When asked why he never spoke before, he said, "Up until now everything has been alright." This is how I felt about the silence which followed my work on a project which installed and provided support for over 7000 opensource JDS desktops at a bank.1 We called the customer occasionally to see if everything was O.K. We helped them through one upgrade which was necessary because the Linux kernel needed to be upgraded to support modern hardware but didn't have a stable ABI so the entire application stack also had to be upgraded. After the upgrade, one of our customers gave us some upgrades/minute statistics that were well beyond what is possible given network bandwidth limitations so I'll just say that the upgrade went well.

Shortly after the upgrade, we helped solve a peculiar focus bug whose root causes were spread across gtk, Java, Firefox and Star/OpenOffice. But overall things were very quiet. Sun was also quiet about this deployment, first of all because we hadn't yet finalized the disclosure agreement and later because Sun decided to drop our Linux-based desktop product and focus on OpenSolaris. So between our "are you still there?" pings to the customer's 2 person technical support staff, I was left wondering if no news is good news?

Then when I gave my presentation at the Irish Opensource Technology Conference, I noticed that two knowledgeable IT managers from this bank were giving presentations on their opensource desktop (a.k.a. JDS) roll out. I finally had the opportunity to be the "fly on the wall" and hear how things really went. I don't have links to their presentations, but these gentlemen said that the project was a success, that the deployment saved money and IT support costs compared to traditional Microsoft Windows based desktop solutions. They said the project completed ahead of schedule and under budget and that they were telling other banks the secrets of their success. I don't know if the other banks were paying attention to the potential savings in deploying opensource alternatives back when easy money was still flowing, but I would think they should take a hard look at such cost-effective alternatives now. In any case, it seems likely that the number of successful cost-saving "invisible" opensource deployments is understated.

"The art and science of interface design depends largely on making the transaction with the computer as transparent as possible in order to minimize the burden on the user" -- S. Joy Mountford

"Any sufficiently advanced technology is indistinguishable from magic!"
-- Arthur C. Clarke.

1 The deployment was of Sun's linux based "Java Desktop System." If we were to do it now, the obvious choices in Sun's product portfolio would be Solaris 10 or OpenSolaris. Since the customer's network is now fast enough to support Sun Ray over WAN, we could potentially save them another $500,000 in annual electricity costs by deploying their desktop via Sun Ray clients instead of X86 PCs.

Monday Apr 09, 2007

RAW digital photos on OpenSolaris and gcc vs Sun Studio

Howth Dolman

Ireland was blessed with clear skies during the lunar eclipse of 3 March 2007. My combination of a Celestron 500mm F5.6 Maksutov mirror lens telescope and a Pentax \*ist DL worked well. I set the white balance to daylight so the camera wouldn't try to compensate for the unearthly red tint earth's atmosphere caste on the moon. I also set the camera to RAW mode, to avoid JPEG artifacts and allow me to make use of the full dynamic range of the sensor chip. Unfortunately because I had previously used the camera to photograph documents in support of my Irish residency (important documents which have not yet been returned), the 6 Megapixel camera was set to 1 Megapixel. So Simon Phipps and Tim Foster both have better photos of this event.

I did learn something from this experiment; gimp, eog and other GNOME applications which are available on recent OpenSolaris desktops don't know how to handle RAW images. There's a very good reason for this. RAW isn't a digital image standard. RAW is a word which is used to refer to one of a number of relatively unprocessed proprietary digital camera image formats. These formats range from Sony's encrypted "RAW" format to formats which resemble tiff or Adobe's Digital Negative (DNG.) Ideally the manufacturers would agree on a standard such as Adobe's DNG the lossless JPEG used in medical imaging, or even Microsoft's "JPEG Killer", as long as patents, closed standards and other all too familiar tactics aren't used to "embrace and extinguish" digital cameras and internet imaging. Can you imagine if the open standard (JPEG and PNG) hadn't replaced GIF on the web and IBM and Unisys had decided to collect royalties?

Until camera manufacturers standardize on an open format, we will need conversion utilities. Fortunately these utilities exist. Dave Coffin wrote one such utility called "dcraw". I like the program, it's a small command line utility which only depends on standard C libraries. A portable command line utility probably makes more sense than a library because next week a camera manufacturer might come up with yet another "RAW" format and you'll have to change the source code and rebuild. If you're building this on Solaris or OpenSolaris, you'll probably want to apply a tiny patch:

bash-3.00$ diff -u dcraw.c.orig dcraw.c
--- dcraw.c.orig        Tue Mar  6 11:18:08 2007
+++ dcraw.c     Sat Mar 24 00:04:15 2007
@@ -317,7 +317,7 @@
   fread (pixel, 2, count, ifp);
   if ((order == 0x4949) == (ntohs(0x1234) == 0x1234))
-    swab (pixel, pixel, count\*2);
+    swab ((const char \*) pixel, (char \*) pixel, count\*2);
 void CLASS canon_600_fixed_wb (int temp)
@@ -1923,7 +1923,7 @@
   size_t nbytes;
   nbytes = fread (jpeg_buffer, 1, 4096, ifp);
-  swab (jpeg_buffer, jpeg_buffer, nbytes);
+  swab ((const char \*) jpeg_buffer, jpeg_buffer, nbytes);
   cinfo->src->next_input_byte = jpeg_buffer;
   cinfo->src->bytes_in_buffer = nbytes;
   return TRUE;
@@ -7162,7 +7162,7 @@
           FORCC ppm [col\*colors+c] = lut[image[soff][c]];
       else FORCC ppm2[col\*colors+c] =     image[soff][c];
     if (output_bps == 16 && !output_tiff && htons(0x55aa) != 0x55aa)
-      swab (ppm2, ppm2, width\*colors\*2);
+      swab ((const char \*) ppm2, (char \*) ppm2, width\*colors\*2);
     fwrite (ppm, colors\*output_bps/8, width, ofp);
   free (ppm);


Building with Sun Studio requires a couple of options and still gives warnings about some of the unsigned char data exceeding 127, but gcc ignores it so we'll ignore it for now also:

bash-3.00$ cc -o sdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 309: warning: implicit function declaration: getc_unlocked
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./sdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m21.769s
user    0m19.671s
sys     0m0.222s

Guess why I named the executable sdcraw? Because it's slow, 21 seconds for a 6 megapixel Pentax file. That's because the Sun Studio in Solaris Express Developer edition still defaults to no optimization! Let's see how gcc does with no optimization flags:


bash-3.00$ /usr/sfw/bin/gcc -o sgdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./sgdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m15.308s
user    0m13.339s
sys     0m0.211s

That's a bit faster.  By default gcc doesn't seem to care if you're initializing
unsigned chars to values greater than 127, but it's default optimization seems 
somewhat better than Sun Studio's.  Let me try enabling some gcc optimization:

bash-3.00$ /usr/sfw/bin/gcc -O4 -o gdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./gdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m6.767s
user    0m5.220s
sys     0m0.189s

That's getting respectable!  Now let me try Sun Studio with the -fast, -xpentium and
-xspace flags.  You have to be careful with -fast, because by itself it assumes that
the target architecture is the same as the build architecture.  Since I'll be using it
on the same laptop I'm building it on, that's O.K.


bash-3.00$ cc -xc99=none -i -xspace -xstrconst -xpentium -mr -o fdcraw -fast dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./fdcraw ../Documents/Photos/20070401/IMGP3420.PEF
real    0m5.659s
user    0m4.160s
sys     0m0.208s

Sun Studio's default of no optimization is probably a good thing. I'm sure someone will complain if (when?) this default is changed, but until it is changed, newcomers to OpenSolaris might take a quick glance at Sun Studio and say "this sucks". If you're willing to read some man pages and explore some optimizations and tools (e.g. Sun Studio's analyzer profiler), you might grow to prefer Sun Studio.

If you're looking for a Solaris application and find it doesn't exist, try to find the source of a GNU/Linux or OSX application which does what you need and rebuild it for Solaris. And thank the maintainer of the application, Dave Coffin's dcraw is pretty useful. Once I've converted my Pentax PEF "RAW" original into IMGP3420.tiff, I can pull it into GIMP where I crop, resize and upload it to this blog entry. The above photo was taken at the Howth dolman a couple of weekends ago. The rhododendrons were already blooming.




« July 2016