By moinakg on May 22, 2008
From now on I will be blogging at my personal blog site: moinakg.wordpress.com some interesting tidbits coming up there regarding BeleniX.
I spent some time in the weekend doodling around with OpenOffice Draw to try and come up with some big diagram explaining the components of Indiana aka OpenSolaris 2008.05 since a picture is worth N words. Many may know that BeleniX contributed some stuff to Indiana but I don't think people realize how much contributions have gone in. So I have also tried to identify the components that are directly or through modifications, based on my work on BeleniX (all done in my spare time, weekends and late nights of course).
In addition I have tried to capture a timeline that covers the evolution of BeleniX and subsequently Indiana covering the features that were added. There is an addendum covering the features added via the LiveKit. This timeline does not cover the other distros namely Nexenta, Schillix and Milax. I intend to prepare a much more comprehensive timeline covering all those in the coming weekend. Click on the images below for larger versions:
LiveCD Architecure Overview:
As you might notice from the above a major part of Indiana's foundations are based directly on my work on BeleniX. The technology readily available via BeleniX drastically reduced the time to market for Project Indiana. In addition I was directly involved in Indiana from time the project was kicked off and did a bunch of critical stuff like - Integration of HSFS additions into OpenSolaris, Initial builds and workspace setup for Fully Open X, Implementing the Transfer Module that is responsible for transferring livecd contents to harddisk, Implementing Python based Transfer Module via embedded and extended Python interpreter, helping Alok with getting the lofi compression stuff integrated.
The HSFS optimizations involved some major R&D effort for I/O scheduling, elevator, readahead etc. I actually ended writing a whitepaper in the description field of RFE #6588256.
BeleniX today has also benefitted from Indiana and the 0.7 release has evolved to be a source-level derivate that complements Indiana. I can of course see BeleniX contributing back to OpenSolaris going forward - a healthy community development. The story of getting to 0.7 is essentially a story of Unix From Scratch. That is the topic of another blog post (in a different blog site) another day.
I was quite shocked to learn yesterday that SciFi behemoth Arthur C. Clarke passed away. I am deeply saddened having had read and enjoyed most of his writings. His clarity of vision and expression has been just amazing. I am waiting to get my hands on his last novel: " The Last Theorem".
There are times when Solaris is unable to detect the system's Ethernet card and most commonly it turns out that one of Masayuki Murayama's Solaris NIC drivers will work. We end up pointing people to the source tarball download page and instruct them to do a make install. Though not too difficult it is kinda troublesome esp. for Solaris newbies. It is always nice to have an easy install bundle with binaries packaged up as SVR4 packages.
With this in mind I went about adding a Pkgbuild SPEC file for Masayuki-san's drivers to the SFE repository. Thanks to the flexibility and amazing ease provided by the SPEC file mechanism it took me a few hours on a Sunday to nicely package up the drivers (21 packages in all) with proper postinstall actions and checks to add the drivers. It would probably have taken a couple days to achieve this result using a Makefile - driven system.
Instead of building from the SPEC file you can download this small bundle of binary packages. Use the following simple steps to install:
The other piece I added into SFE is the Scorched 3D game. It is a 3D remake of the Scorched Earth DOS game. It builds on Solaris after a small source patch and runs without problems. Unfortunately the SDL library bundled with Solaris Express is not built with OpenGL support. So I had to remove SUNWlibsdl and SUNWlibsdl-devel and build and install SFEsdl to get the game to work.
I used to play this in Ubuntu on my Ferrari laptop. However the ATI's 3D accelerated driver for Linux sucks Big Time. It hard hangs the system now and then. I will now be able to enjoy the game on by Solaris box. Here is an obgligatory screenshot:
While browsing the Cscope homepage I found out about Kscope a KDE based GUI for Cscope. The screenshots looked inviting enough for me to quickly grab it. After a little tweak to a shell script that assumes /bin/sh to be bash (ouch) it worked perfectly. Being a Cscope user myself, I found this GUI to be extremely powerful, a great frontend. It has interesting features with the ability to use Graphviz and graphically display call graphs, call trees, ease of querying, save cscope queries. Overall adds a whole new dimension to Cscope.
I was quickly able to create a project to index ON kernel (/usr/src/uts). Here are a few screenshots:
Partial call graph showing functions calling door_ki_upcall and functions called by it:
Another screenshot of the call tree from the same function:
Screenshot after jumping to the source line of the highlighted mutex_exit call above:
It should be an interesting project to write a Netbeans plugin with these features.
I have recently been burning some evening and weekend oil pushing a lot of spec files into the SFE repository. I have finally been able to put spec files for all of the KDE 3.5.8 packages up there and at long last have a complete KDE desktop running on my home system on Nevada Build78 in all it's glory with proper antialiased fonts.
It is built using Gcc 3.4 that comes with Nevada. I have been using it for a couple of weeks now and do not see any performance or stability issues apart from some rare weirdness from aRts. Apart from the SPEC files I had check-in a bundle of patches to KDE source most of which I have pushed (or is pushing) to upstream KDE projects and a few have already been integrated by the maintainers.
Drop into the next BOSUG meet if you want to grab the binary packages of KDE and whole bunch of other FOSS stuff. Some screenshots follow (click on the images for a larger version).
Found a nice wallpaper to go with the default blue theme. KDE is probably the only desktop that comes with an appealing default theme after you compile and install it from source!
Konqueror, Krita image editor and Control Center
AmaroK rocks - the ultimate music player created by Mankind.
The Fully Open X (FOX) project's mercurial repository is now fairly well populated with Makefiles for all the Xorg modules available and has reached a point where the whole wad is now buildable on Nevada. In early December, I had pushed all the Xorg modules that I added to Solaris Xorg sources to create a complete X environment for BeleniX. However that was just the initial step.
Alan Coopersmith had also added package definitions for all the SUNW\* X11 packages that are redistributable to the repository. These package definitions refer to Solaris legacy X11 pathnames and will need to be modified to work with the FOX built binaries since FOX moves the whole X environment to /usr/X11 and just keeps compatibility symlinks around in /usr/openwin.
Over this weekend I added package definitions for all the FSW\* X11packages that are in BeleniX and Indiana and updated the repository with a series of modifications and patches. So now it is possible to build the whole repository and get a working set of X11 packages which includes the FSW ones. However ultimately the intention is to get rid of the FSW packages and update all the SUNW packages to work with FOX builds. That is a big drudgery of a job and will need some time.
There were some X11 compatibility issues reported against Indiana that caused many Solaris GUI apps including StarOffice to not work on Indiana. This was mainly had to do with missing symbols in Xorg's libXext that were present in the Solaris version of libXext. Alan has now added patches to the FOX repository to fix these problems so StarOffice should work out of the box in the upcoming Indiana beta release.
One thing that is still a known limitation is the lack of proper l10n support in the libX11 provided by FOX. The Globalization team is working on this. As you can see there is still a lot of work to do. If you wish to play with the FOX repo you can follow the build instructions at: http://www.opensolaris.org/os/project/fox/build_inst/
hg clone ssh://firstname.lastname@example.org/hg/fox/fox-gate
You can even schedule it to run during certain times. There are various actions possible with blacklisted numbers: divert the call to another phone, send back busy/ignore tone, tell that the number does not exist, tell that the phone is powered off etc. It also maintains a log of the date-time, calls, numbers and actions. It even has the ability to auto-update from a pre-configured WAP site if GPRS is enabled on your phone.
Quite a comprehensive list of features and absolutely must-have. In addition it is free. Now I can block unwanted callers esp. the tele-marketing types and recently people asking me to sign up as Amway distributor! I do not have to subscribe to a paid call blocking service from the service provider. A few screenshots:
Apart from this I completely agree with the author's points. In fact just see what our much vaunted and hyped IT industry is doing. BPO, Call center, Maintenance, Documentation, low-end Sustenance, Customer support form the vast bulk of the business. There is very little innovation in technology (with few exceptions as the author noted and a few MNCs). How many software products have come out of India ? What new technology have we created ? The Industry here has essentially created assembly line workers out of intelligent minds. There is no long-term technical career path and brilliant technical minds are clobbered into stagnation.
How long is this primarily services model going to work ? Costs and Salaries in India are increasing and services/outsourcing are increasingly going to move onto other cheaper shores like China or Czech republic etc. Unless we move up the Technology value chain, unless we innovate, unless we get world class products onto the marketplace I do not think the IT boom is going to sustain.
I recently read an article in the November issue of Linux For You titled "SUN Microsystems: A Friend or a Foe" authored by the well known John "Maddog" Hall. I felt there are a couple of misconceptions in the article that should be pointed out.
He mentions a comment by a LUG speaker: "... how hard it was to work on a Solaris system because of the antiquated and non-standard set of utilities on the upper levels of the Operating System ...". I am not sure what is meant by non-standard here. POSIX compliance is followed on Solaris and there are conformance testsuites for the commands and utilities. Is he equating non-standard to meaning non-GNU ? Well for that it requires a tweak of PATH to put /usr/sfw/bin on Solaris 10 to get a whole host of modern FOSS tools. If more is needed Blastwave.ORG is a very good resource. In addition SXDE is a very complete and modern Operating Environment with a complete GNU CLI environment under /usr/gnu/bin. Again a PATH tweak is all that is required. In adition SXDE bundles a plethora of FOSS including latest GNOME releases in /usr/bin . SXDE today is indeed very comfortable to use. It is amazing to see how far Solaris has come from the earlier days in a relatively short period of it's history.
"Sun also claims binary compatibility between releases, ... Sun guarantees your program to run as long as you did not use any interfaces that did not change or were not deleted. This is a guarantee ?". HUH, I smell FUD in this statement. What does one expect ? Absolute backward compatibility till the point when the OS was born! Every interface is Solaris is meticulously documented for interface stability guidelines and indicated which are stable and which are evolving etc. There is an ARC process that oversees every public interface being introduced, their implications and backward compatibility criteria. What more can one expect ? If you are using an evolving interface of course you know what the implications are. Of course changes do happen, interfaces do get obsolete - an inevitable part of software evolution. But these changes are "managed", one does not see APIs disappearing or changing suddenly without warning in micro-releases/updates for eg.
There is also a point in the article that mentions about lack of clear information about claimed several hundred benefits of Solaris. This is something to consider IMHO. Everyone talks about the big features of Solaris: DTrace, ZFS, SMF, Zones et. al. but there is much less talk about loads of other stuff that make the Operating System unique and compelling. For eg. Fine Grained Resource Management, RBAC, Integrated Real-Time features and so much more. How many people know that Interrupt Shielding has been present in Solaris since Solaris 7 ? There are so many more such little-known things about Solaris that are just crying out to be better highlighted. We need a well-organized how-to collection.
Forgot about one point:
"... some of those features are offered by major Linux distributions, and others can be added by 'rolling your own' in the Linux space."
Correct me if I am wrong here. AFAIK, apart from ZFS there is one other thing that cannot be had even by rolling your own: Comprehensive Resource Management (CPU Caps, Memory Caps, Dynamic Resource Pools, Processor sets, Fair Share Scheduler, Zones + Resource Controls etc.).
Some partial things are available for eg. the project: ckrm.sourceforge.net (Class Based Kernel Resource Management) but there is a long way to go.
A couple of new features introduced in BeleniX have now made their way into OpenSolaris core. These features are critical to making a LiveCD possible: Lofi compression and HSFS performance enhancements.
The second feature also benefits data CD/DVD usage on OpenSolaris in the generic case. The HSFS filesystem module in OpenSolaris used to issue I/O requests only in 2K chunks and had no readahead. This caused it to only achieve a max sequential read throughput of 3.5 MB/s whereas an 8X DVD for eg. is capable of supporting a much higher throughput. The changes as I had mentioned in earlier posts enable much better performance for both sequential and random access esp. on the newer faster boxes. There are a few other tweaks (like metadata prefetch) that I have in mind for later. The HSFS enhancements were integrated into Build 77. However subsequently there were a couple of problems that cropped up (a panic 6621609, and an issue with xVM 6627899) that had me slogging over 2 weekends to resolve these under tough deadlines.
Integrating lofi compression also required a lot of work over the original changes for BeleniX including a PSARC case (2007/569) and Alok Aggarwal did a fine job of this. This was integrated into Build 80. All this while lofi compression was only used in 32Bit mode since the BeleniX and Indiana microroots only have a 32Bit kernel till date. Testing and making lofi compression work in 64Bit mode had Alok breaking his head to figure out some weird rounding and data type issues. The code was cleaned up and enhanced a lot in addition to making the compression algorithm pluggable so that additional compression techniques can be supported in future, if desired.
Well the interesting and hectic days are coming back. FOSS.IN, one of the most prestigious FOSS conferences in Asia will be held from Dec 4th to 9th, 2007. SUN is participating this time as well. This time the conference has been expanded to 5 days from the usual 3 days and includes some changes.
The aim of the conference this time is to kick-start more widespread FOSS contributions from India. India has long been known as big FOSS consumer but not contributor, and this is an attempt to change that. The conference this time has less number of talks and a bunch of in-depth sessions called Project Days. First two days of the conference will be Project Days where there will be a series of deep technical/contributory sessions/workshops on a bunch of topics and OpenSolaris is one of them. See here for the first list of main conference talks and Project Days sessions. I am taking a Project Days session on Indiana where I will be taking the audience through a deep technical session as well as discussing how to get involved. There are other folks from SUN giving talks like Jim Grisanzio on getting involved in OpenSolaris, Josh Berkus on PostgreSQL, Roger Brinkley on OpenJDK, Glynn Foster on Introducing OpenSolaris. The OpenSolaris project day has a good number of community speakers.
I like this idea of Project Days. This time round there is an experimental Hacka-center where there will be PCs with net access. So folks can have impromptu hacking/experimentation sessions. BOFs are mostly being pre-planned. In all a good collection of activities to foster more contributions.
This time the organizers have been particularly strict about the nature of the talks and the background of the speakers. If you see the list of talks all newbie-style and evangelism talks have been avoided. Rather they wanted talks which discuss advanced technical topics and also focus on how to contribute or get involved. So they are trying to attract a more mature and developer oriented crowd rather than FOSS-newbies.
The new participatory content is welcome and probably should be expanded upon in future. The list of talks is impressive. Hey, India with the huge technically talented population has to have a corresponding impact on FOSS development. Anyway it is gonna be an interesting ride. If you are nearby Bangalore in the beginning of Dec do not forget to register and visit.
Sometime back an interesting feature got added to the Live Media Kit. This kit allows one to take a Solaris Express install DVD and build a Solaris Express Live bootable DVD out of it. This kit has formed the basis of the initial prototype work that we did for Slim Install. Eventually we will be moving to using the Distro Constructor once it stabilizes. Though the Distro Constructor has taken the same core ideas and some of the actual logic from the Live Media Kit.
Towards the middle of last year I did some performance enhancements that dealt with laying out file data on the CDROM to reduce booting time. That is explained at "The BeleniX LiveCD Performance Story Part 3.
I used this technique for the Live Media Kit as well to gain big performance benefits. However initially I/O tracing required one to build a modified DVD image collect trace data and process it. This is typically been done for every subsequent BeleniX release. I spent some time looking at making it more automatic and did some initial changes to easily build I/O tracing enabled LiveCD/DVD images. Subsequently Dave Miner took those changes and refined them and got them into the Live Media Kit. This requires the DTraceToolkit to be present in the LiveCD/DVD image.
So now it is easy to boot an OpenSolaris LiveCD and select "Solaris (32 bit) I/O tracing" boot option in the Grub menu. One dosen't even have to build a tracing enabled image. The trace data gets collected into a file called traceout on the root filesystem on harddisk. This means that this has to be necessarily on a system that already has Solaris installed on the harddisk. Subsequently one has to run the "proc_tracedata" script in the LiveKit to build a mkisofs sortfile called iso.sort.
This, to my knowledge, is probably the first time that any LiveCD kit is making it so easy to do this optimization. Of course thanks in no small terms to DTrace. Similar optimization, though with a different logic, is done for many Linux LiveCDs as well but those sometimes lack in effectiveness and require system call tracing to collect huge trace data. Subsequently that tracedata is processed via cut-ing, sort-ing, awk-ind, grep-ing and so-on in a nightmarish shell pipeline. And this is mostly done manually by those LiveCD maintainers. More often than not folks re-mastering Linux LiveCDs complain that their re-mastered LiveCDs are slower to boot than the original.
With the LiveKit tracing is now a part of the startup scripts and is as easy as selecting a boot option as mentioned above. The script "live-devices-local" that replaces devices-local in the Live environment has code to enable tracing if that boot option is detected. Scroll to the end to see the actual code - it is quite simple. This means that DVDs that other folks create will boot as fast as the original without too much of work needing to be done.
And of course that has already gone into the distro constructor. This optimization does make a huge difference in the bootup time. You can checkout the DIstro Constructor mercurial repository from here: ssh://hg.opensolaris.org/hg/caiman/distro_constructor
While having a general discussion on smartphones Nagki suggested utilizing a phone's USB storage mode and putting a USB-bootable OpenSolaris on the SD card. Since I own a Motorola Ming Linux smartphone with a 2GB SD card this immediately got me interested. After experimenting a couple of days I managed to get the Indiana LiveCD (Slim Install) booting on my home PC (Dell Dimension E521) from the phone in USB storage mode. The Motoming is a great hacker's phone.
It was quite fun to watch and also make calls from the phone while OpenSolaris was happily booting from it. I can now carry my Motoming as around as a Slim Install demo piece with the ability to even install off it. Here are some pictures followed by more details on how I did it and what problem I faced, so that you don't think I am bluffing
First a full view of my home desktop booted into Indiana LiveCD preview release from the phone, red arrow pointing to the phone:
Here is a bad Digicam shot of a dmesg snippet showing the USB storage on the phone being recognized (click to see larger size):
A Screenshot showing /.cdrom and /usr (via lofi) mounted off the USB storage from the phone (click to see larger size):
Finally here is another pic of the phone taking a call while OpenSolaris is running of it's USB storage. Fun eh. Rather good for demos as well (click to see larger image):
It took a couple of days of experimentation to work this out. Firstly I had to put the Slim Install LiveCD onto the SD card. I have a 2GB SD card which I decided to split up into 2 1GB partitions. So first I took a backup of all existing stuff on the card, not much, mostly some useful apps and a few pictures. Then I booted into Solaris and inserted the phone which promptly got mounted. I unmounted it and used fdisk to create 2 partitions on the phone's SD card. First one being a DOS-BIG partition because that's what was present on the card originally. The other was obviously Solaris2. Then I ran devfsadm to refresh the devices and used format -e to create one big root slice in the Solaris2 partition. Finally I used newfs to put a UFS filesystem on that. It will be uber-cool to have this as ZFS on the phone but the LiveCD mechanism can't yet handle that; should be a relatively simple enhancement.
Now I lofi-mounted the LiveCD ISO image and used cpio to transfer all it's contents to the newly created UFS filesystem on USB storage. I then manually updated the microroot on the USB storage to indicate that we want a LiveUSB boot. Finally an installgrub onto the phone's USB storage completed the steps. Thus I manually performed all the steps that usbdump/usbcopy would have performed to put the LiveCD on a USB pendrive. This was required since usbdump uses the whole USB device whereas I wanted to use one partition. It should not be too difficult to enhance the usbdump script to allow for partitioning.
Now I needed to recover my original phone data and apps back. So I unmounted everything and pulled out the USB cable. Then I opened the File Manager app and browsed to the TransFlash directory via which you can access the SD card. the phone said that the card is not formatted and asked whether to format it. Doubtfully I hit on Yes and it completed formatting. Then I re-inserted the phone's USB cable and thankfully the phone has only formatted the FAT-16 partition leaving the Solaris2 partition intact - cool. Now while in Solaris I recovered all backed-up data/apps onto the phone's FAT-16 partition and pulled out the USB cable. I was able to see all my data in the File Manager's TransFlash directory but unfortunately none of the apps showed up. Pondering a bit I booted into WinXP and used WinRAR to once again uncompress the backup (tar.gz) onto the Phone's FAT partition. This time it worked and I was able to access all the apps. Probably looks like some weirdo problem with PCFS and FAT-16.
So I had the phone working properly and having 2 partitions one with an OpenSolaris LiveUSB image. All I had to do was boot it on my laptop. This is where another unfortunate problem came up. The Ferrari 3400 BIOS would hang if rebooted with the phone plugged in and a Ferrari 4000 would simply not recognize the phone even though it had switched into USB Mass-Storage mode. I tried on a few other desktops with the same result. It appears that there is some problem with the way the phone switched into USB Mass-Storage mode. It takes a while to switch and it switches when the BIOS comes up. So this might be causing the phone to not respond properly to the BIOS probe and the BIOS probably does not retry and ignores the device. Hum, so much of effort wasted! Not quite.
There are some boxes whose BIOS is unable to boot off USB. It is possible to use OpenSolaris on USB even on those. All one needed to do was to get the USB microroot booted from some other media like a CDROM and it will scan and mount the USB. So I mounted the phone and copied the directories: boot, platform dev and devices to another directory. Then I used mkisofs to create a small bootable 30MB ISO of this directory. I burned this onto a DVD-R/W. I then inserted the phone's USB cable into my home PC and inserted the CDROM and rebooted the machine from the CDROM. Once the kernel messages came up I ejected the CDROM and at last the system continued upto JDS desktop from the phone Only glitch being the phone is recognized as a USB 1.x device so it is kinda slow. But it works just fine.
This is one cool Indiana demo that I will be carrying around. I will next try to install off the phone.
Recently I had an interesting discussion with a relative about how technology can touch lives even in remote places. Arunachal Pradesh is a state in the Northeast of India situated in the Himalayas. You can see the map here.
You can find Wild cows (Bos Frontails) freely roaming among habitation and wilderness. These are locally known as Mithuns and they have religious and traditional value. Most of these Mithuns are actually owned by the mountainfolk and represent wealth more than money. These animals are allowed to roam free and occasionally used for beef during ceremonies like marriages. Since they represent wealth that can also be used for barter.
One adult Mithun can cost around Rs 22000 - 25000 ($550 - $650) and people tend to own anywhere from scores to hundreds of these animals. These are flaunted as a show of wealth esp. during marriages. Each animal is branded with the owner's seal.
Since these animals are allowed to roam free they represent floating property which can be hard to keep track of. What if an animal is eaten by a predator or stolen - though stealing these is a serious crime and it rarely happens. These animals can wander over a 10KM radius and are manually rounded up into a barn for counting once in a while.
So now an entrepreneur is looking at a cost effective way of using technology to keep track of these animals. One idea is to use active RFID tags embedded under the skin of each animal. However active RFID tags cannot have a 10KM range so they would need a system of towers spread across the area to pick up the signals. They can also co-locate the receivers with nearby cellular phone towers to reduce cost as well.
Another idea is to embed tiny radio/GPS transmitters. But that is more expensive. Also since the per-unit cost has to be kept at a minimum the service-provider can have monthly service charges to cover the initial expenses and have a recurring revenue stream.
Further there are other aspects being considered as well. Even if an animal is killed, for eg. by a Leopard the tag will keep on transmitting so how do you know whether the animal is dead or alive. The tag not moving from one place from a long time can give clues, but what if the tag is in the Leopard's stomach and still transmitting So there can a facility to monitor the animal's vital signs like body temperature, blood pressure and transmit the information.