Porting OpenSolaris Project Indiana to SPARC - Part 2

Step 4 - The Distribution Constructor (DC)

The DC is a useful tool. Grabs the bits from the IPS and installs em into a 'proto' area. This proto area will become the new root dir of the liveCD. The next task of the DC is to carve up this proto area into two main subsections. The microroot, and all those bells and whistles. Remember, it is the task of the microroot to find and mount these goodies as /usr, /etc, /var, and /opt. These complete the user-land functionality the microroot otherwise lacks. These goodies are solaris.zlib and solarismisc.zlib, as seen in the figure. Below is a diagram showing which bits get carved into which subsections. The only complicated bit of the diagram is that some of /usr actually lives in the microroot. The zlibs have the option of being compressed using gzip or LZMA.

/mnt-------|           \^
/platform--|           |
/proc------|           |
/sbin------|   (the smaller bits)
/system----|           |
/tmp-------|           |
/usr----(most of it)---|-> solaris.zlib

/etc-----|-----------> solarismisc.zlib

/jack----|--------->exist at / of cdrom

The first task that needed tackling was that because this tool is x86-centric it needed to be gutted of any of these x86isms. The largest of these changes was the difference in how x86/GRUB reads its microroot and how SPARC/OBP reads its microroot - compression.

On x86 architectures, the entire microroot, containing a file system of uncompressed files, is compressed. On SPARC systems, the microroot is an uncompressed filesystem containing compressed files. These files on the SPARC microroot have been marked to decompress on the fly using a dcfs kernel module. The benefit of using the SPARC approach is that it not only saves CD space using this compression, but it also shrinks the memory footprint of the microroot on bootup. The utility we used in the DC to do this is fiocompress which uses gzip underneath.

The crazy problem with fiocompress was a very strange error that ended up making everything go fubar. Fiocompress does not carry over permissions provided by it's source file to its destination; it instead uses permissions from its destination file (only if it exists). If the destination file doesn't exist yet it gives -rw-r--r--. I'll let this example speak for itself and leave the reader to extrapolate the mess this can make.

# rm init
# fiocompress -mc /sbin/init ./init
# ls -l init
-rw-r--r--   1 root     root       67664 Jul  8 15:28 init
# Note the permissions when dest doesn't exist. (Ow!)

# rm init
# cp /sbin/init .
# Making dest exist now

# ls -l init
-r-xr-xr-x   1 root     root       67664 Jul  8 15:28 init
# Permissions brought over from cp command

# du -sh init
67K   init
# It isn't compressed yet

# fiocompress -mc /sbin/init ./init
# ls -l init
-r-xr-xr-x   1 root     root       67664 Jul  8 15:28 init
# Permissions have been kept because dest existed (Happy Panda!)

# du -sh init
30K   init
# Confirming it has been compressed

So in the end, while waiting for the bug to get fixed, the quick work around is to first straight up copy all the files we wish to have microroot over and _then_ compress them back on top of themselves. We get to keep permissions and they're smaller. Victory!

Step 5 - The microroot

Here's the bit where the most hours were spent, and many other things as well. Sweat. Anger. Despair. Annoyance. And Love - but only after it started working. :)

An early tackled problem was OBP complaining that our microroot didn't appear to be executable.

Sun Blade 2500, No Keyboard
Copyright 1998-2003 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.9.5, 8192 MB memory installed, Serial #57202056.
Ethernet address 0:3:ba:68:d5:88, Host ID: 8368d588.

Rebooting with command: boot disk1 -kv -F /sparc3.microroot        
Boot device: /pci@1d,700000/scsi@4/disk@1,0  File and args: -kv -F /sparc3.microroot

The file just loaded does not appear to be executable.
{1} ok

It turns out you need to install boot-blocks to something you wish to directly boot from! Note: We only had to do this because we were booting from a file as the above OBP command shows - not from a device. Here's the command to imbue those magic ones and zeros:

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rlofi/1
</dev/rlofi/1 is where I have the microroot loopback mounted to>

This next issue was by far the hardest and most elusive bug we ran into. The microroot's kernel couldn't find /devices and panics. It was totally crazy! The kernel can't find /devices but we can when we loopback mount the microroot on a already running system. It's right there! Arg! It was funny, we would talk about how we were going to find the problem its going to be so small we'll all have a simultaneous 'Doh!' moment when we find it.

{1} ok boot disk1 -k -F /sparc13.microroot
Boot device: /pci@1d,700000/scsi@4/disk@1,0  File and args: -k -F /sparc13.microroot
Loading kmdb...
SunOS Release 5.11 Version snv_88 64-bit
Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
panic[cpu1]/thread=180e000: Cannot find /devices

The answer was to add two lines into the microroot's /etc/system. So while we were busy interpreting assembly back to C and stepping through the kernel source code of vfs_mountdevices() and ufs_dirlook() in the kernel debugger, /etc/system was staring silently at us, laughing to itself. Doh! Two tiny lines >.<

# cat etc/system
set root_is_ramdisk=1
set ramdisk_size=90000

These lines tell the kernel to mount / as the ramdisk that had been loaded into memory, not the device the microroot happened to be stored at. It couldn't find /devices because our HDD (in OBP speak, 'disk1') that we used to store our microroots didn't have said directory. Another funny one to watch out for is setting that second line correctly. You gotta make sure that that is actually larger than the amount of bits you have living in your ramdisk otherwise your kernel and system come to a screeching halt and the only thing you see printed a very beautiful, very lonely, 'H'. 'H' as in 'Hi, I'm not useful at all.'

The rest of the problems solved seem rather minor in comparison now. For instance on SPARC a ramdisk is called /devices/ramdisk-root:a, where as on x86 it's /devices/ramdisk:a. This is helpful information when attempting to remount the ramdisk with rw permissions.

Or when sshd spats out 'no kex alg' upon ssh'ing in... That means you need to run these two commands:

# ssh-keygen -t dsa -f etc/ssh/ssh_host_dsa_key
# ssh-keygen -t rsa -f etc/ssh/ssh_host_rsa_key

Step 6 -
Licensing & Icing on the cake: Gnome

The next and last step towards making this truly OpenSolaris (and not just a Nevada LiveCD) is having the licensing wizards go through the code base and make sure its all kosher. You know, all the legal mumbo-jumbo.

Due to a lack of time and limited availability of open-sourced SPARC graphics drivers I will leave adding X/Gnome functionality to the experts. This would have been a fun problem to whack at, but alas I'm off to a long needed vacation and then classes start again.

Step 7 - Candy Time! \^_\^

And now here's your candy for reading with me for this long. OpenSolaris Project Indiana @ multi-user milestone (without X/Gnome) on my trusty sun4u machine you've seen a lot of already !!AND!! a blazingly fast, snazzy new sun4v machine as well. Enjoy :)

    - Chris

Sun Blade 2500, No Keyboard
Copyright 1998-2003 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.9.5, 8192 MB memory installed, Serial #57202056.
Ethernet address 0:3:ba:68:d5:88, Host ID: 8368d588.

{1} ok boot disk1 -kv -F /sparc14.microroot
Boot device: /pci@1d,700000/scsi@4/disk@1,0  File and args: -kv -F /sparc14.microroot


opensolaris console login: jack
Jul 18 16:12:25 opensolaris pseudo: pseudo-device: devinfo0
Jul 18 16:12:25 opensolaris genunix: devinfo0 is /pseudo/devinfo@0
Sun Microsystems Inc.   SunOS 5.11      snv_88  January 2008
jack@opensolaris:~$ uname -a
SunOS opensolaris 5.11 snv_88 sun4u sparc SUNW,Sun-Blade-2500 Solaris
jack@opensolaris:~$ svcs | grep "maintenance"
maintenance    16:10:27 svc:/application/x11/xfs:default

SPARC Enterprise T5120, No Keyboard
Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.27.1, 16256 MB memory available, Serial #81038714.
Ethernet address 0:14:4f:d4:8d:7a, Host ID: 84d48d7a.

{0} ok boot disk -kv -F /sparc14v.microroot 


opensolaris console login: jack
Jul 18 16:04:14 opensolaris pseudo: pseudo-device: devinfo0
Jul 18 16:04:14 opensolaris genunix: devinfo0 is /pseudo/devinfo@0
Sun Microsystems Inc.   SunOS 5.11      snv_88  January 2008
jack@opensolaris:~$ uname -a
SunOS opensolaris 5.11 snv_88 sun4v sparc SUNW,SPARC-Enterprise-T5120 Solaris
jack@opensolaris:~$ svcs | grep "maintenance"
maintenance    16:03:10 svc:/application/x11/xfs:default

[Trackback] Bookmarked your post over at Blog Bookmarker.com!

Posted by elusive on July 18, 2008 at 07:13 PM MDT #

this is really nice work. very cool indeed.

As of ~b91 SPARC can now do ZFSboot too. Do you think you'll have to change a lot to make that work here too, or should those pieces pretty much fall inline as part of the "installer" (not livecd) code?

Posted by guest on July 20, 2008 at 09:04 AM MDT #

I don't believe ZFSboot functionality will change much in the way the liveCD comes up. Right now the microroot is just a ufs filesystem and the zlibs are hsfs. My guess (though I'm not very familiar with ZFSboot) is that those changes will fall inline as part of the system installer code. I'm now working on making that b93 repo I mentioned in the post, so if something does come up - I'll make note of it here.


Posted by cwalsh on July 21, 2008 at 06:04 AM MDT #

The funny that we fought with sparc miniroot at the same time ;)

Posted by Alexander R. Eremin on July 25, 2008 at 02:45 AM MDT #

Very cool indeed, mate. Congrats :) I went browsing through your site and see that you have a sparc liveCD as well, was your development process similar to mine? Did you hit the same problems?

Posted by cwalsh on July 25, 2008 at 04:45 AM MDT #

Yea. So far I plan to move next x86 release to IPS. If you continue work it's will be great to share experience:)

Posted by Alexander R. Eremin on July 25, 2008 at 05:40 AM MDT #

Hi, I had been going through exactly the same procedure last February, but Sun didn't want my help.

Yes, the two lines that were missing from /etc/system were killing a few days until I found the solution in ON's src code.
A bity nobody wanted my work.

Good luck, anyway.

Posted by Martin Bochnig on September 06, 2008 at 09:34 PM MDT #


BTW: Thanks for the information on how the compression mechanism of sparc.miniroot differs from x86.
I didn't find out myself how I can compress it (on sparc).
That's the only thing which is new to me, but it is good to finally learn about it.
I wished Sun would have wanted me to be in the Indiana-SPARC team, but they didn't. All my effort of the recent years (e.g. SPARC-Xorg) is worth nothing to them: No Dime, no Sheckel, no "thanks". N o t h i n g.

Posted by Martin Bochnig on September 06, 2008 at 09:43 PM MDT #

Hey again, one thing needs to be added: It sounds like you took a ready2run Nevada release as base, in this case you missed most of the really odd time-consuming things which I experienced (not only with the Studio-self-built ON binaries, but also with the un-cpio'ed BFU archives [b83 and b96]). Strange driver related bugs in hme and lofi. This costed the most time. Things like "lofiadm -a /path/to/iso" would give "lofiadm: file cannot be mapped, file not found". Or that you could in no circumstances somehow plumb the hme0 interface on U30 and U60's, despite everything in /etc being 100% correct, including path_to_inst.
Things like that happen to make trivial things difficult-to-impossible. That's why it took over two weeks to upgrade my b83 related proto area to b96. Otherwise I could and would have released Natamar_0.4 RC0 ten days ago, before I left home for a week.
But right now I got the lofi and hme related issues worked around (not really, but it works). Now I'm only having two problems with gdm (#0._loads two root-windows over each other instead of one, #1._still the start-up-refusal due to invalid RandR 1.2 - based Xinerama self-identification of server 1.3 when operating on the on the sunffb driver).
Also: Should I release a big DVD with JDS and everything, or only a small CD like Indiana? In case of the latter I need to compress the sparc.miniroot.
Also: /usr/sfw will only fit on a DVD, whatever I do. So I think I should offer both. Actually I would also like to put Stefan Teleman's KDE port-binaries on it as well. And currently I didn't care much about packaging. First it shall only be a monolithic demo LiveCD and LiveDVD. Okay?

Posted by Martin Bochnig on September 07, 2008 at 01:55 AM MDT #

First of all I'm not "whining", no: I'm \*shouting\*. But not about the amazing www.x.org project, but more specificly about SMI's X11-group. Until last year I had thought they were my friends (when I had the honor to shake hands with them in Menlo park), but it was an illusion. They didn't offer me any job, nor whatever other material or non-material help. Instead they hired one or more other persons. That's their way of saying "thank you for the music".
The same with SPARC-Indiana. I offered Sun at least 10\^24 times to work on this, at very low (even zero) payment. All I wanted was to be a Sun-employee. It had always been my dream. And I neglected my private life, University studies, everything, I'm more than EUR 10K in the reds now (14K USD).
But those "friends" of mine don't even say hello when I try to talk to them.

BTW: The Natamar DVD iso is still under construction, but the limited CD.iso is here already:

DVD-image and more info to be posted asap.

Posted by Martin Bochnig on September 08, 2008 at 11:52 PM MDT #

Hi Martin,

I'm sad to hear about your less than desirable relationship with Sun considering all the work you've put in. I can relate to your situation.

I wanted to thank you for taking the time to read through my process; and I'm glad it was able to uncover some of the unknowns of your compression problem.

More specifically though, you mentioned having problems with the Studio-self-built ON binaries, and the un-cpio'ed BFU archives [b83 and b96]. I never ran into those from just using the SVR4 packages from a Nevada DVD. What base of code are you using for your releases? By using the Nevada base, am I missing something? Could your work be simplified if you followed my steps?

Thanks again,

Posted by cwalsh on September 10, 2008 at 04:34 AM MDT #

Martin, it's better to make LiveCD then DVD;)
MilaX Sparc CD (230 Mb) for example has no X but contains all Nevada drivers + Apache+PHP, so I think it's possible to put base KDE (or XFCE) in Natamar LiveCD, you can play with miniroot size - for example you can move unnecessary drivers at booting in /usr/kernel.

Posted by Alexander R. Eremin on September 15, 2008 at 11:48 PM MDT #

Thanks for your post. Unfortunately I am reading it now. I just found the problem with /etc/system and was googling around to see what OTHER arguments were useful in /etc/system for boot cdroms. I am working on a rescue cd that pulls contents from a running system into a bootable cd.

Posted by David Huffman on February 20, 2009 at 10:12 AM MST #

Post a Comment:
  • HTML Syntax: NOT allowed

Christopher Walsh


« July 2016