Wednesday Sep 24, 2008

What lies beyond the archaic 8 char FAT filesystem limit?

I took my family out to the Donegal this weekend. By coincidence my Pentax digital camera "rolled over" from IMGP9999.PEF to IMGP0001.PEF just before I took this picture:
So now I must move all new files to a new folder or change the names before I rsync to my photo backup space to avoid overwriting old files. Whoever decided 10,000 photos was a sufficient limit for a digital SLR never had kids or lived in such a beautiful country. I'm still amazed at how many modern standards (e.g. ISO-9660, DCIM, DICOM) still bow to the 8.3 limitations of Microsoft's old FAT filesystem. As far as I know, DOS and RT-11 were the only popular operating systems made after 1970 with such a SHRTFLNM.LMT. Solaris, AmigaDOS, VMS, Gem, Geos, even the Vic-20 and Commodore 64 filenames exceeded this limit! Yes, it's possible to work around this limit with CLVRABRV.NMS or DIRECTRY STRUCTRS, but it seems that of all of the Digital camera manufacturers, only Casio, HP and Kodak have reasonably long term naming conventions, but none of them is as forward thinking as ZFS would allow them to be.

We really enjoyed our stay at the friendly family run Ocean Spray Bed and Breakfast on Muckross head, just east of the Slieve League cliffs in Donegal. The fact that there were no nearby restaurants or other late season tourists contributed to that "edge of the earth" feel of Donegal.

Wednesday May 14, 2008

Spring in Wisconsin

Wisconsin Spring 2008 photo

I recently returned to a visit with the family in Wisconsin. After more than 100 inches of snow in what was one of the snowiest winters since 1979, very few people were happy to see April snow showers. But we were. After a few grey wet Dublin winters, we enjoyed the April snowfall. The snow only lasted a few hours and spring came back in full force a few days later with temperatures in the 70s.

[Read More]

Tuesday Apr 10, 2007

Risk of closed proprietary "RAW" digital imaging formats

After I published the entry on building the dcraw "RAW" image format converter for OpenSolaris I found an interesting post on the photography forums regarding Pentax PEF files:
Default Opening Pentax \*ist DS RAW .PEF files in Windows 98? Pentax say on the box that the Windows system requirement is Windows ME/2000/XP and they're not kidding... However, the software with the \*ist DS (including the software for processing RAW images) requires some methods in Share32.dll which simply don't exist in Win98...[Read More]

Monday Apr 09, 2007

RAW digital photos on OpenSolaris and gcc vs Sun Studio

Howth Dolman

Ireland was blessed with clear skies during the lunar eclipse of 3 March 2007. My combination of a Celestron 500mm F5.6 Maksutov mirror lens telescope and a Pentax \*ist DL worked well. I set the white balance to daylight so the camera wouldn't try to compensate for the unearthly red tint earth's atmosphere caste on the moon. I also set the camera to RAW mode, to avoid JPEG artifacts and allow me to make use of the full dynamic range of the sensor chip. Unfortunately because I had previously used the camera to photograph documents in support of my Irish residency (important documents which have not yet been returned), the 6 Megapixel camera was set to 1 Megapixel. So Simon Phipps and Tim Foster both have better photos of this event.

I did learn something from this experiment; gimp, eog and other GNOME applications which are available on recent OpenSolaris desktops don't know how to handle RAW images. There's a very good reason for this. RAW isn't a digital image standard. RAW is a word which is used to refer to one of a number of relatively unprocessed proprietary digital camera image formats. These formats range from Sony's encrypted "RAW" format to formats which resemble tiff or Adobe's Digital Negative (DNG.) Ideally the manufacturers would agree on a standard such as Adobe's DNG the lossless JPEG used in medical imaging, or even Microsoft's "JPEG Killer", as long as patents, closed standards and other all too familiar tactics aren't used to "embrace and extinguish" digital cameras and internet imaging. Can you imagine if the open standard (JPEG and PNG) hadn't replaced GIF on the web and IBM and Unisys had decided to collect royalties?

Until camera manufacturers standardize on an open format, we will need conversion utilities. Fortunately these utilities exist. Dave Coffin wrote one such utility called "dcraw". I like the program, it's a small command line utility which only depends on standard C libraries. A portable command line utility probably makes more sense than a library because next week a camera manufacturer might come up with yet another "RAW" format and you'll have to change the source code and rebuild. If you're building this on Solaris or OpenSolaris, you'll probably want to apply a tiny patch:

bash-3.00$ diff -u dcraw.c.orig dcraw.c
--- dcraw.c.orig        Tue Mar  6 11:18:08 2007
+++ dcraw.c     Sat Mar 24 00:04:15 2007
@@ -317,7 +317,7 @@
   fread (pixel, 2, count, ifp);
   if ((order == 0x4949) == (ntohs(0x1234) == 0x1234))
-    swab (pixel, pixel, count\*2);
+    swab ((const char \*) pixel, (char \*) pixel, count\*2);
 void CLASS canon_600_fixed_wb (int temp)
@@ -1923,7 +1923,7 @@
   size_t nbytes;
   nbytes = fread (jpeg_buffer, 1, 4096, ifp);
-  swab (jpeg_buffer, jpeg_buffer, nbytes);
+  swab ((const char \*) jpeg_buffer, jpeg_buffer, nbytes);
   cinfo->src->next_input_byte = jpeg_buffer;
   cinfo->src->bytes_in_buffer = nbytes;
   return TRUE;
@@ -7162,7 +7162,7 @@
           FORCC ppm [col\*colors+c] = lut[image[soff][c]];
       else FORCC ppm2[col\*colors+c] =     image[soff][c];
     if (output_bps == 16 && !output_tiff && htons(0x55aa) != 0x55aa)
-      swab (ppm2, ppm2, width\*colors\*2);
+      swab ((const char \*) ppm2, (char \*) ppm2, width\*colors\*2);
     fwrite (ppm, colors\*output_bps/8, width, ofp);
   free (ppm);


Building with Sun Studio requires a couple of options and still gives warnings about some of the unsigned char data exceeding 127, but gcc ignores it so we'll ignore it for now also:

bash-3.00$ cc -o sdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 309: warning: implicit function declaration: getc_unlocked
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./sdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m21.769s
user    0m19.671s
sys     0m0.222s

Guess why I named the executable sdcraw? Because it's slow, 21 seconds for a 6 megapixel Pentax file. That's because the Sun Studio in Solaris Express Developer edition still defaults to no optimization! Let's see how gcc does with no optimization flags:


bash-3.00$ /usr/sfw/bin/gcc -o sgdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./sgdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m15.308s
user    0m13.339s
sys     0m0.211s

That's a bit faster.  By default gcc doesn't seem to care if you're initializing
unsigned chars to values greater than 127, but it's default optimization seems 
somewhat better than Sun Studio's.  Let me try enabling some gcc optimization:

bash-3.00$ /usr/sfw/bin/gcc -O4 -o gdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./gdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m6.767s
user    0m5.220s
sys     0m0.189s

That's getting respectable!  Now let me try Sun Studio with the -fast, -xpentium and
-xspace flags.  You have to be careful with -fast, because by itself it assumes that
the target architecture is the same as the build architecture.  Since I'll be using it
on the same laptop I'm building it on, that's O.K.


bash-3.00$ cc -xc99=none -i -xspace -xstrconst -xpentium -mr -o fdcraw -fast dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./fdcraw ../Documents/Photos/20070401/IMGP3420.PEF
real    0m5.659s
user    0m4.160s
sys     0m0.208s

Sun Studio's default of no optimization is probably a good thing. I'm sure someone will complain if (when?) this default is changed, but until it is changed, newcomers to OpenSolaris might take a quick glance at Sun Studio and say "this sucks". If you're willing to read some man pages and explore some optimizations and tools (e.g. Sun Studio's analyzer profiler), you might grow to prefer Sun Studio.

If you're looking for a Solaris application and find it doesn't exist, try to find the source of a GNU/Linux or OSX application which does what you need and rebuild it for Solaris. And thank the maintainer of the application, Dave Coffin's dcraw is pretty useful. Once I've converted my Pentax PEF "RAW" original into IMGP3420.tiff, I can pull it into GIMP where I crop, resize and upload it to this blog entry. The above photo was taken at the Howth dolman a couple of weekends ago. The rhododendrons were already blooming.




« June 2016