Monday Jun 09, 2008

OpenSolaris update to build 90

Just noting that updated packages for snv_90 were pushed to the pkg.opensolaris.org repository over the weekend, see Alan's announcement for some very important instructions that need to be followed. Two things that I'll add:
  1. You should not use the Package Manager GUI to perform the update; instead use the pkg image-update command. The reason is that the GUI is not yet able to create an alternate boot environment for running the update, so you'll end up updating your running system and not have a clean fallback if something goes wrong. Yes, this will be fixed.
  2. We aren't yet publishing new ISO images, due to some fixes we need to make to the installer to synchronize with the ZFS boot project's integration. You wouldn't use those for updating from the 2008.05 bits anyway, so it's no real loss at the moment. We expect to resume publishing ISO's of every build via BitTorrent soon, possibly as soon as build 91.
Happy updating!

Friday Jun 06, 2008

OpenSolaris Requirements Meeting

This has been rescheduled for June 19... 

We've scheduled a meeting for next Thursday, June 12 19, at 2 PM US/Pacific to have a planning and requirements discussion with the community around the OpenSolaris distribution. See Tim Cramer's announcement mail for more background and details; the working agenda is on the wiki. I'll be presenting our draft plans for the work covered in the Caiman project, and Stephen will do the same for IPS, the rest are still being worked out Hope to see you there!

Monday May 19, 2008

Clearing the backlog

Thankfully, I was able to take a much-needed week off last week to recover from the OpenSolaris 2008.05 release. I'm still not sure which was more exhausting: the engineering work, or the parties afterwards. A few things that accumulated while I was gone:

  • Barton published the impromptu podcast we did just before the big launch.

  • Ars Technica published a nice review of OpenSolaris. My favorite, of course:
    The most impressive aspect of OpenSolaris is the installation experience, which is painless, intuitive, and easily on par with Ubuntu and Fedora.
    That's the nicest thing anyone's said about one of my projects in a long time.

  • Unfortunately, we also got a less than outstanding result on another review. Looks like a known hang, possibly, but obviously we have work to do here.

And yes, the vacation was very nice, it included a couple of days at the fabulous Bandon Dunes resort. Considering that I haven't been playing as much as I normally would have and links golf is about as different from the parkland layout I'm a member at as you can imagine, I was pretty happy with the 85 and 84 I shot on Pacific Dunes and Bandon Dunes, respectively. Much credit to my caddy, Jeff Pack, who did all the hard work of reading the putts and telling me what to hit. 2 days as a golf robot were kind of fun ;-) I've been to a lot of the top resorts in the Golf Magazine rankings, and this is one of my favorites, as it's not at all pretentious; just "golf as it was meant to be". I will definitely be going back!

Wednesday Feb 13, 2008

Indiana Preview 2 now playing

The second preview of Indiana was pushed out for download about 15 hours ago as I write this.  As those who are following closely know, we'd actually made a testing image available via Bittorrent a couple of weeks ago in order to broaden the testing exposure.  This time the image actually got some professional and community testing, as opposed to Preview 1, which basically had about 1 hour of some of us trying to boot it on every machine we could get our hands on.  Thanks to those who took the time to test it!

The announcement has the general highlights and links off to all of the bugs we've fixed.  From an installation point of view, the changes in this preview are mostly under the hood.  The most notable ones include:

  • Most visibly, at least to those who encountered it, we fixed the bug that prevented you from using 'v' in the passwords of the root user and the user created during installation.
  • A better PATH and MANPATH for the user account created at installation.
  • The ZFS dataset hierarchy has been renamed and reorganized a little, it is now similar to what will be used in the ZFS boot/install project that'll be added to Nevada and Solaris 10.  For the preview, you'll now see the root dataset named rpool/ROOT/preview2. In the same vein, we now take a snapshot of all of the datasets at installation (snapshot name is @install) so that it's easy to tell what's been changed, and to get back to the start state of the system should that be needed for some reason.
  • /boot/grub/menu.lst now points you off to the "real" menu, which is at /rpool/boot/grub/menu.lst.
  • The introduction of pkg verify pointed out some errors in the installation code, primarily around zero-length files, which we've fixed.  A verification after installation will show some issues with modes of Python compiled files, but otherwise it'll be quite clean.
I've been running this on my laptop and desktop for a week now, and I've had no issues.  If you're feeling adventurous, a couple of things I've done on my laptop.


First, I'm running with a compressed root.  The installer doesn't support this right now, but you can easily fire up a terminal window and, once the installer's created the data sets, do a zfs set compression=on rpool/ROOT/preview2 and all of the installed software will be in compressed data sets. You need to do this quickly in order to get as much compressed as possible, since anything copied onto the data sets before the compression is enabled will not be compressed. After adding OpenOffice and a lot of other packages, I'm at 1.94 GB for the root dataset, with compression ratios reported at 1.8x or so.


The second thing is running Preview 2 as an xVM dom0.  Three steps for this:
  1. pkg install SUNWvirtinst SUNWurlgrabber SUNWlibvirt SUNWxvmhvm SUNWxvmdom SUNWxvm
  2. Manually edit /rpool/boot/grub/menu.lst and add the xVM entry. The normal method for doing this is bootadm -m upgrade, but it doesn't work for Indiana because of the menu being moved for ZFS root.  The entry should look like
  3. title OpenSolaris xVM
    kernel$ /boot/$ISADIR/xen.gz
    module$ /platform/i86xpv/kernel/$ISADIR/unix
    /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
    module$ /platform/i86pc/$ISADIR/boot_archive
  4. Run the following commands to update the sysevent registrations so that you can install domU's (fixing this is bug 509):
  5. pfexec syseventadm add -c EC_xendev /usr/lib/xen/scripts/xpvd-event \\
     'action=$subclass' 'domain=$domain' 'vdev=$vdev' \\
    'device=$device' 'devclass=$devclass' 'febe=$fob'
    pfexec syseventadm add -c EC_xpvsys /usr/lib/xen/scripts/xpvsys-event \\
    'subclass=$subclass' 'shutdown=$shutdown'

If you do give Indiana a spin, please report any bugs you find at defect.opensolaris.org.

Friday Jan 11, 2008

10 Steps to Caiman Development on Indiana

In one of our meetings the other day I was pointing out to the team that I could tell most of them weren't using Indiana as their primary development platform, because I was finding bugs in the Caiman source release when attempting to build it using Indiana.  Usually we're pretty good at eating our own dog food, but that hasn't really happened much yet with Indiana, in part because we've had a couple of other things to do at the same time.  Some of the team suggested that one of the issues holding them back is that there's a bunch of additional stuff you need to add to the base Indiana install to make it suitable for development use, and they didn't feel they had the time to figure it out.  Well, fortunately, I already have, at least for Caiman development, since I put Indiana on my laptop right after we released it 2.5 months ago, and my desktop a few weeks later (my desktop at home is still running SXCE for a bunch of mostly lame reasons).  So, without further ado, here's my brief cheat-sheet on how to get an Indiana system into shape for OpenSolaris development:
  1. Follow the instructions for downloading the live CD, burning it, booting it, installing from it
  2. Once you've booted from the disk, make sure you update the system (which is mentioned in the above instructions, but bears repeating since it will fix some critical problems, such as the broken SVR4 packaging software on the preview media).
  3. Install the header files: pkg install SUNWhea
  4. Install Mercurial: pkg install SUNWmercurial
  5. Install make: pkg install SUNWsprot
  6. Install Java: pkg install SUNWj6rt
  7. Follow the instructions to download and install the compilers and build tools, though you'll want the tools from the SCM Migration project so that they work with Mercurial.
  8. In order to build the Caiman sources, download and pkgadd SUNWwbint, SUNWzoneint, and SUNWldskint (SUNWldskint isn't up there yet, we're working on getting that corrected).
  9. If you have an NVIDIA graphics controller, then you probably want to install NVDAgraphics and NVDAgraphicsr from the Nevada media or NVIDIA's web site.
  10. A few other things you might find yourself wanting; first two are on the Nevada media.
    • SUNWflash-player-plugin
    • SUNWrealplayer
    • OpenOffice 2.3
I haven't tried building the OS/Net sources on Indiana since I haven't had a need lately, but I don't know of any reason why it wouldn't work.

Thursday Nov 15, 2007

Install source release

I'm happy to report that we've just released more installation source code to OpenSolaris.  What we've released today is the source code which corresponds to the Dwarf Caiman installer which appeared in SXDE 9/07, and the Slim Install installer that's used in the OpenSolaris Developer Preview.  It also encompasses the SVR4 packaging code which had been released previously, but which is now being made available via the Mercurial repository rather than the tarball releases we'd been doing.

This is a significant milestone, as it represents essentially all of the source base that we're using in building the new installer in the Caiman project and finally opens us up for meaningful contributions from the community.  Thanks to Moriah for her hard work on this, as well as the others on the team who helped with research and reviews!



Thursday Nov 01, 2007

OpenSolaris Developer Preview on USB flash drives

One of the things we developed between BeleniX and the Live Media project is the ability to run the live CD bits on USB flash drives, and indeed, that's what I mostly demo, because it's a lot faster (boot time is under a minute, install time for the OpenSolaris Developer Preview  is about 7 minutes) - Jonathan loved it when we demo'ed it a couple of weeks ago.  We don't push USB images yet because right now you need to already have Solaris installed in order to copy it to the USB stick.  But if you do have Solaris Nevada (or have installed the preview!) already, and a copy of the preview ISO, you can make a USB flash drive for yourself, as follows:
  1. Use mercurial to get a copy of the Distribution Constructor repository:

    hg clone ssh://anon@hg.opensolaris.org/hg/caiman/distro_constructor

  2. Go into the distro_constructor/tools directory and run usbgen, this will take roughly 10 minutes:

    ./usbgen <path_to_iso_file> <path_to_usb_image> `pwd` <tmpdir>

  3. Plug your USB flash drive into the system, give the system a few seconds to see it, then run usbcopy, this will usually take 3-5 minutes, depending on your flash device; usbcopy will discover all your removable media and let you pick the right device:

    ./usbcopy <path_to_usb_image>

You'll need a 1 GB or larger USB flash device, since the image is 600+ MB.

Why Indiana uses bash

One the indiana-discuss and caiman-discuss lists a poster wrote:

> /bin/sh -c 'print ${.sh.version}'
> /bin/sh: bad substitution
>
> I hoped Indiana would improve things and deliver the ksh 93 as
> /bin/sh, but no, it is just the same bourne shell as in Solaris. No
> improvement. Indiana doesn't improve things for developers.
>
> The choice for /bin/bash as user shell for 'Jack' and 'root' is a SHAME.

As I think this is representative of a broader set of issues in OpenSolaris, I decided to post this as a blog entry rather than just email to the lists.  Let me explain how this "choice" came to be...

When the OpenSolaris project opened a couple of years ago, Moinak Ghosh and friends built a distro called BeleniX, a very nice piece of work which I respect even more after what we've gone through in building this preview.  One of the things they developed was the tools for building a live CD, and in that kit they'd made a number of choices, which included that bash was the shell for root.  Sensible, since ksh93 wasn't ported at that time.

A bit later, I started collaborating with Moinak on bringing the live CD technology to the Solaris base, in the OpenSolaris Live Media project.  We worked in the background on it for a year or so, tweaking it and updating it for later Nevada builds as I worked on the political end of getting Solaris management to buy into this as our future approach.  I was doing this project in my spare time, as was Moinak, and we didn't really have any other particularly active contributors other than Anil Gulecha (who did the original USB work) and a couple of patches I got from others.  For our purposes, bash was fine, it worked, and I didn't need to bother changing it, so I didn't.

When Ian came along and got Sun management to buy into doing Indiana with a preview in October and including the new installer, live CD, new packaging system, we were already working in that direction, but by the time we got staffing freed up from other things, we had about 10 weeks to get to this preview release.  You don't do that by making lots of random changes and creating lots of new code (IPS is an exception to the latter) so we used what we had, cobbling things together, and that meant we borrowed from Live Media to create the Distribution Constructor, which used bash.  And in the last 4 weeks of 14 hour days, this wasn't a priority to change, so it didn't.

The moral of the story above is that if you get involved and contribute code to projects, you \*will\* make decisions in that code which will show up in distributions; Live Media would have welcomed contributions along the way.  We need coders, not managers, as Sun pays plenty enough of them already.  If you're going to pass final judgment on this distro based on this preview, then that's unfortunate; we've done some good work, I think, but we've also cut a \*lot\* of corners to get to where we are, by focusing on the things we knew we had to do and leaving the other things for later.  That won't be quite so necessary going along, but my years of delivering projects tells me we're going to postpone something from every milestone, and make someone unhappy every single time, so I'm disappointed by the tone of some of these comments, but not surprised.  The code for much of this is out there now, between the Distribution Constructor and IPS repositories, and now is the time to start offering patches.  We'll welcome them.

Wednesday Oct 31, 2007

No candy, just an OpenSolaris treat

Well, we went a little later today than planned, but the preview release of Project Indiana is out in the wild.  It's too late in my US/Eastern timezone to write much right now, but I'll just say before I go to bed that I'm glad to have reached this milestone.  And those of you in the Boston area can come to the NEOSUG meeting Thursday night where I definitely will say more, and show what we've done so far.  And if you're really up for jumping in, help you install it!

Friday Sep 14, 2007

Jumpstart with MIcrosoft DHCP server

Jumpstart usage with non-Sun DHCP servers is a question that comes up fairly often.  A good posting I noticed today on the sunmanagers mailing list:

Solaris JumpStart vs the Microsoft DHCP Server - Larry Rosenman's personal blog

(reposted to update link to Larry's blog)

Monday Aug 20, 2007

Try Out the "Dwarf Caiman" Installer

Over the weekend, Solaris Nevada build 70 was pushed out to the Sun Download Center.  This build is the first candidate for the next Solaris Express Developer Edition (SXDE) release, which means the work of the Dwarf Caiman team is now generally available.  Due to a number of bugs that have already been found in internal testing we know there'll be at least one "re-spin" build, 70a, before we'll call it the SXDE release.  In the meantime, please try it out and let us know how it works via caiman-discuss (or on our new IRC channel, #caiman-discuss at chat.freenode.net).  While you're downloading, read on for a couple of comments I'd like to make in relation to this new installer.

Probably the most important thing to understand is that we're closer to the beginning of producing a truly new installer here than the end.  This project was specifically scoped to provide a new graphical interface for SXDE as soon as possible, replacing the tired old GUI that's been in place for several Solaris releases now and which we'd done some minor tweaking of late last year when the SXDE program started.  To allow for delivery in the time frame we've done this on (the project went from zero to delivery in 6 months), we specifically didn't change the underlying install engine, a moldy old warhorse known as pfinstall.  The code to discover and identify the available storage (what we call "target discovery" in the architecture) is mostly new, and there's a new thing called the "orchestrator" which provides the interfaces the GUI uses.  But the actual installation and upgrade process still relies on pfinstall, which is driven by a Jumpstart profile generated by the GUI/orchestrator combo.  Replacing the rest of pfinstall is the subject of current and future projects within the Installation and Packaging community.

Because we were specifically scoping this project for SXDE, we could make a couple of simplifying assumptions - the old installer is still there for all the things we don't choose to support yet.  For those of you who are experienced Solaris users, the one you may notice is that there's no way for you to set up the UFS slices within the Solaris FDISK partition.  That was intentional, and it's because you're an experienced Solaris hand that this is even particularly noticeable; however, if that's true, then you're not the target audience for this iteration of the installer.  The target users here are people new to Solaris, and the concept that you have to both set up an FDISK partition plus lay out slices within that partition is confusing to every single one of them.  So we just left it out for now, in favor of a canned layout for fresh installs that is more appropriate than the one the old installer uses (read the discussion thread on this for details); its most important attributes are that the root slice will be larger than before, and if the FDISK partition specified is sufficiently large, you'll get a second, alternate root for use with Live Upgrade by default.  Later projects will be adding more control over the file systems than is provided here, but even before that we're expecting we'll move to ZFS as the root file system.  The pooled storage model of ZFS is specifically meant to eliminate slicing as a common exercise for system administration, so I'm not expecting it to be a particularly commonly-used function in the finished installer, either.

You might also notice that we've added configuration of an initial user account, rather than just setting the root password and leaving you to your own devices to figure out how to create a user.  Again, this is a simplified interface - you don't get to choose uid's or groups, just the necessary basics, which is essentially identical the to Gnome Users & Groups tool's basic settings tab.  If you're part of a NFS environment where uid's and gid's matter, then by all means adjust these once you're installed, and if the administrative tools don't do what you need, please file bugs.  The philosophy here is to configure a few basics necessary to get started, but not require you to get everything exactly right to start with, so if there's something in the tools that makes that not work, we'll get it fixed.

The last thing I'll mention at this point isn't new, but was new in the SXDE 5/07 release and thus may not be particularly familiar yet.  The installer doesn't ask you to do any network configuration, because it enables Network Auto-Magic instead.  If you find that NWAM's a little too magical (or perhaps not magical enough yet, since it's still an early version, too) for your taste, you can switch to the standard Solaris networking configuration with sys-unconfig, which will force a reboot and walk you through the standard old configuration dialogs.

Tuesday Jul 31, 2007

Putting the Open into Solaris Installation

Just a note that the project meetings for the rest of the new OpenSolaris installer project are finally starting, and are open to anyone interested:

New OpenSolaris Installer Weekly Meeting at OpenSolaris.org

We're days away from integrating the Dwarf Caiman project, which provides the initial implementation of the new GUI for the next Solaris Express Developer Edition release, shortly after which we should be releasing much more source code.  At that point we'll be about as open as open can be.

Friday Jul 20, 2007

Replacing aging infrastructure

I've been in the New York City area this week, taking part in presentations to the local customer base on a variety of Solaris topics; in my case, focused on Solaris 10 patching.  Wednesday evening turned out to be a bit more intense than I bargained for, as Spencer, Brian Wong and I were having an impromptu discussion with a customer after the day's event had concluded when there was a fairly loud bang, followed by a sustained, roaring rumble.  At first it sounded like the thunder we'd had earlier in the day, but when it continued for more than 30 seconds, we looked out the window, and realized people were running away from our building.  Taking that as a cue, we hurriedly shook hands and headed for the emergency exits.  Turned out to be the steam pipe eruption (though we didn't know that for roughly an hour), which was only a couple hundred yards from Sun's NYC office on Park Ave.  That chaotic scene also turned out to be the context for my first introduction to Ian and Sara Dornsife as we scrambled down 41st St.  Hopefully this isn't a harbinger for our collaboration on Indiana!  It was a relief to hear that the cause of all that chaos was a failure of some aging infrastructure rather than an intentional act, but that will do little to ease the pain of those injured in the accident.  My deepest condolences to them; those of us who were in the area and got out unscathed really have to feel lucky.  My thanks to Ambreesh for his aid in getting to our hotel in NJ that night, as well as Isaac for retrieving my stranded bags from Park Ave. Thursday morning.

In Solaris, we've got our share of aging infrastructure as well, very notably in packaging and patching.  Its failures haven't been as dramatic as that steam pipe, but it's been failing nonetheless and becoming ever harder to maintain as the demands on it have increased.  We've been doing our best to band-aid the patching situation for Solaris 10, the latest being the addition of Deferred Activation Patching, which provides a mechanism to safely apply complex changes to the system in contexts where Live Upgrade can't be used (I still recommend Live Upgrade as the preferred alternative).  Even as we've been patching patching, though, we've well understood that a fundamental reconstruction of our packaging system is in order to solve the core issues and have been exploring approaches to that task.  I'm delighted that Stephen has posted his initial observations on the topic, and I'd encourage everyone to give it a read and join the conversation.  There will be much more to come in the next few weeks as this turns into a full-blown project on OpenSolaris.

Friday Jul 13, 2007

Indiana Installer prototype

As I promised back on June 19, I've pushed out the changes to the Live Media kit which include hooking up the Dwarf Caiman user interface.  If you download the kit, download Solaris Express Community Edition (build 67 works well), build the live image, burn it, boot it, and login as root, you'll have an "Install Solaris" icon on the desktop.

It's still rather rough, since the version of the UI package that I'm using wasn't fully functional at the time I grabbed it.  In particular, you must already have a Solaris FDISK partition created, upgrade won't work, and none of the configuration settings the GUI creates will be applied - the first boot will end up running sysidtool to configure the system.  Oh, and the root in this version will be on UFS, not ZFS, though if you inspect the dummy_install script in the kit it'll be obvious how you can change it to use ZFS instead.

Now, if you just imagine all the rough spots above fixed, cut the image down to CD size, and add a few enhancements to the UI, you'll see what Project Indiana's installer should look like.

[Thanks to Glynn for posting the screen shots that I should have included!]

Tuesday Jun 19, 2007

Live Media installer

A bit over a week ago I pushed out an update to the OpenSolaris Live Media project which a few people have asked about, so I figured I'd answer them here.

Since we opened the project in August last year, most of the work for a few months was basically just maintenance, updating the construction kit to keep it functioning as OpenSolaris kept evolving around it. Some of those updates were easier than others; Gnome 2.16 in particular presented a few challenges (which turn out to have been useful to understand, as we're now dealing with some of them all over again in the Dwarf Caiman project as it modifies the current install miniroot so that it's capable of running a GTK-based install GUI).

The past few months, though, have seen the project start to evolve more rapidly. In March, we added the ability to generate flash memory images, which place an entire Solaris Express release on a 2 GB memory stick. This really helps the pace of development, as now we're not stuck exclusively using a read-only piece of media which has to be completely rebuilt to make even the smallest change. April brought improved networking support in the form of NWAM, and May's update added the WiFi drivers, while all along the improvements in the X server and drivers have continued to improve the display support options.

The most recent update, though, gets to the reason I initiated this project: using it as an environment for an installer. Truth be told, this installer is really primitive, not as functional or easy to use as what Belenix or Nexenta or Solaris offers; you first have to partition and slice out disk space using fdisk and format (so it's definitely an advanced user process at present), but once that's done, the installer's synopsis is simply:

/.cdrom/installer/install_live <ufs | zfs> <root slice> <swap slice>

Let 'er rip, and you'll have a complete installation of Solaris Express; on my test laptop (Toshiba Tecra M5) it takes about 20 minutes with a UFS root, 15 minutes with ZFS. The actual install_live script is quite simple: newfs or zpool create + zfs create to generate the root file system, a series of cpio's to copy the files to the new root, undo the customizations that make the live image work, generate the vfstab, configure GRUB, and finally run sys-unconfig so that the usual Solaris configuration procedures will run on first boot.

The primitive nature of this installer is something we're well on the way to addressing already; I've hooked it up as the back-end to the new GUI from Dwarf Caiman I mentioned above, and we've done successful installs with it to a half-dozen or so different machines. It also includes Moinak's work to optimize the DVD layout, so DVD boot times to reach the Gnome desktop are down to roughly five minutes rather than the nearly 15 that's been the case up 'til now. I'll be pushing those changes out as soon as I've cleaned them up a bit.

Which leaves the question that at least a couple of people asked: how does this relate to Caiman, or Project Indiana? Indiana's goals are well-aligned with the initial portions of our plans for Caiman all along (a project I've been calling “Slim Install”), so I look at this experiment as providing a demonstrable prototype of what Indiana's installer could look like to a user. The major challenge is in cutting down the Live Media image to fit in a CD; right now it's a bit more than twice the 700 MB limit we'd have for a CD. There are plenty of other big problems to be solved in many areas to build an open, maintainable installation and software maintenance system for the OpenSolaris family of distros, but at least now we've got something tangible to show for the graphical installer.

About

I'm the architect for Solaris installation, with a lot of background in networking and system management. I also play a lot of golf.

Search

Archives
« July 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today
SMF blogs
News

No bookmarks in folder

Blogroll

No bookmarks in folder

Networking blogs