Monday Apr 09, 2007

RAW digital photos on OpenSolaris and gcc vs Sun Studio

Howth Dolman

Ireland was blessed with clear skies during the lunar eclipse of 3 March 2007. My combination of a Celestron 500mm F5.6 Maksutov mirror lens telescope and a Pentax \*ist DL worked well. I set the white balance to daylight so the camera wouldn't try to compensate for the unearthly red tint earth's atmosphere caste on the moon. I also set the camera to RAW mode, to avoid JPEG artifacts and allow me to make use of the full dynamic range of the sensor chip. Unfortunately because I had previously used the camera to photograph documents in support of my Irish residency (important documents which have not yet been returned), the 6 Megapixel camera was set to 1 Megapixel. So Simon Phipps and Tim Foster both have better photos of this event.

I did learn something from this experiment; gimp, eog and other GNOME applications which are available on recent OpenSolaris desktops don't know how to handle RAW images. There's a very good reason for this. RAW isn't a digital image standard. RAW is a word which is used to refer to one of a number of relatively unprocessed proprietary digital camera image formats. These formats range from Sony's encrypted "RAW" format to formats which resemble tiff or Adobe's Digital Negative (DNG.) Ideally the manufacturers would agree on a standard such as Adobe's DNG the lossless JPEG used in medical imaging, or even Microsoft's "JPEG Killer", as long as patents, closed standards and other all too familiar tactics aren't used to "embrace and extinguish" digital cameras and internet imaging. Can you imagine if the open standard (JPEG and PNG) hadn't replaced GIF on the web and IBM and Unisys had decided to collect royalties?

Until camera manufacturers standardize on an open format, we will need conversion utilities. Fortunately these utilities exist. Dave Coffin wrote one such utility called "dcraw". I like the program, it's a small command line utility which only depends on standard C libraries. A portable command line utility probably makes more sense than a library because next week a camera manufacturer might come up with yet another "RAW" format and you'll have to change the source code and rebuild. If you're building this on Solaris or OpenSolaris, you'll probably want to apply a tiny patch:

bash-3.00$ diff -u dcraw.c.orig dcraw.c
--- dcraw.c.orig        Tue Mar  6 11:18:08 2007
+++ dcraw.c     Sat Mar 24 00:04:15 2007
@@ -317,7 +317,7 @@
 {
   fread (pixel, 2, count, ifp);
   if ((order == 0x4949) == (ntohs(0x1234) == 0x1234))
-    swab (pixel, pixel, count\*2);
+    swab ((const char \*) pixel, (char \*) pixel, count\*2);
 }
 
 void CLASS canon_600_fixed_wb (int temp)
@@ -1923,7 +1923,7 @@
   size_t nbytes;
 
   nbytes = fread (jpeg_buffer, 1, 4096, ifp);
-  swab (jpeg_buffer, jpeg_buffer, nbytes);
+  swab ((const char \*) jpeg_buffer, jpeg_buffer, nbytes);
   cinfo->src->next_input_byte = jpeg_buffer;
   cinfo->src->bytes_in_buffer = nbytes;
   return TRUE;
@@ -7162,7 +7162,7 @@
           FORCC ppm [col\*colors+c] = lut[image[soff][c]];
       else FORCC ppm2[col\*colors+c] =     image[soff][c];
     if (output_bps == 16 && !output_tiff && htons(0x55aa) != 0x55aa)
-      swab (ppm2, ppm2, width\*colors\*2);
+      swab ((const char \*) ppm2, (char \*) ppm2, width\*colors\*2);
     fwrite (ppm, colors\*output_bps/8, width, ofp);
   }
   free (ppm);

Slow

Building with Sun Studio requires a couple of options and still gives warnings about some of the unsigned char data exceeding 127, but gcc ignores it so we'll ignore it for now also:

bash-3.00$ cc -o sdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 309: warning: implicit function declaration: getc_unlocked
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./sdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m21.769s
user    0m19.671s
sys     0m0.222s

Guess why I named the executable sdcraw? Because it's slow, 21 seconds for a 6 megapixel Pentax file. That's because the Sun Studio in Solaris Express Developer edition still defaults to no optimization! Let's see how gcc does with no optimization flags:

Faster

bash-3.00$ /usr/sfw/bin/gcc -o sgdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./sgdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m15.308s
user    0m13.339s
sys     0m0.211s

That's a bit faster.  By default gcc doesn't seem to care if you're initializing
unsigned chars to values greater than 127, but it's default optimization seems 
somewhat better than Sun Studio's.  Let me try enabling some gcc optimization:

bash-3.00$ /usr/sfw/bin/gcc -O4 -o gdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./gdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m6.767s
user    0m5.220s
sys     0m0.189s

That's getting respectable!  Now let me try Sun Studio with the -fast, -xpentium and
-xspace flags.  You have to be careful with -fast, because by itself it assumes that
the target architecture is the same as the build architecture.  Since I'll be using it
on the same laptop I'm building it on, that's O.K.

Fastest!

bash-3.00$ cc -xc99=none -i -xspace -xstrconst -xpentium -mr -o fdcraw -fast dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./fdcraw ../Documents/Photos/20070401/IMGP3420.PEF
real    0m5.659s
user    0m4.160s
sys     0m0.208s

Sun Studio's default of no optimization is probably a good thing. I'm sure someone will complain if (when?) this default is changed, but until it is changed, newcomers to OpenSolaris might take a quick glance at Sun Studio and say "this sucks". If you're willing to read some man pages and explore some optimizations and tools (e.g. Sun Studio's analyzer profiler), you might grow to prefer Sun Studio.

If you're looking for a Solaris application and find it doesn't exist, try to find the source of a GNU/Linux or OSX application which does what you need and rebuild it for Solaris. And thank the maintainer of the application, Dave Coffin's dcraw is pretty useful. Once I've converted my Pentax PEF "RAW" original into IMGP3420.tiff, I can pull it into GIMP where I crop, resize and upload it to this blog entry. The above photo was taken at the Howth dolman a couple of weekends ago. The rhododendrons were already blooming.

Friday Jun 09, 2006

grepping /etc, who put these pipes here?

One minor annoyance I've encountered when switching between GNU/Linux and Solaris is that Solaris lays some pipes in the /etc directory. So if you happen to be logged in as root and you do something like this: grep foobar /etc/\* Then your grep may hang for a very long time, seemingly trying to read from the /etc/initpipe pipe. The solution is to only grep files (not pipes) with something like this: grep foobar `find /etc/\* -type f -print` This searches all of the subdirectories under /etc so it still isn't as quick as I'd like. I'm sure someone out there has a more elegant way of doing this. And I suspect there is a very good reason why rusty old pipes are laying around where folks can trip over them.

Wednesday May 24, 2006

Linux Overview for Solaris Users

It doesn't include important Solaris 10 features such as dtrace and zfs, but this (pdf) document entitled Linux Overview for Solaris Users is a good introduction to some of the similarities and differences between Solaris and Linux.

Wednesday Feb 22, 2006

Solaris pkg equivalent to rpm -q --whatprovides

The Rosetta stone for unix isn't quite up to date with Solaris 10+/OpenSolaris and its linux entries could be expanded to include other linux distributions and versions. But it is a good overview of some differences a sysadmin would encounter when migrating between various flavors of unix.

Since linux and Solaris have several package formats to keep track of, I occasionally forget the syntax of installing and querying various package databases. For example, the Solaris SVR4 package equivalent to:

-bash-2.05b# rpm -q --whatprovides /bin/ls
coreutils-4.5.3-26
is this:
 bash-3.00$ grep "/usr/bin/ls" /var/sadm/install/contents
/usr/bin/ls f none 0555 root bin 23480 47244 1131589504 SUNWcsu
I occasionally forget this so I figured that I should write it down here in case anyone else is looking for it.

Update:

Thanks Jason for the comment. I agree that in most circumstances it would be better to use the pkgchk command:

pkgchk -l -p /usr/bin/ls

About

bnitz

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today