Wednesday Apr 18, 2012

Migrating Mirror Root Pool To New Laptop ... Recovery !!

A few years down the line and it's time to upgrade laptop as my once trusty laptop has decided to give up the ghost completely now, the on board graphics memory has decided it's had enough. So time to replace the motherboard or get a new one... so I got a new one.

So I think about how to migrate my nice Mirrored root pool configuration over to my new laptop, would be really nice to not have to actually install on the new laptop, and turns out I really don't have to and I've managed to keep the new laptop as a dual boot machine as well.

NOTE : Ensure the new disk/partition size you will be adding is at LEAST as large as the existing zpool device.

Here's what I did to achieve this :

1. Shrink Windows 7 Volume

Laptop comes pre-installed with Windows 7 and I'd like to actually keep a minimal install of this just in case I need IE, but I'd rather it not use the entire disk obviously. Boot into Windows 7, and run the disk management tool (diskmgmt.msc). In here you can remove Data volumes (assuming they don't have any actual data that you need). Then attempt to shrink the main C: volume down to a preferred size.

For me it's a 320GB disk so I'd like to shrink C: partition down to about 50GB. Right click on the partition in disk management tool and select shrink volume. Initially it would only allow me to shrink to 100GB, as windows was encountering some unmovable files. To resolve, I turned of Paging, Hibernation and Restore and Backup, I could then shrink the volume. Before rebooting out of windows ensure to re-enable these services.

Once shrunk I have about 260GB of free unused space to use, to ensure windows functions later on, I create one large NTFS partition on this unused space, we'll be redefining this later on for our root zpool.

2. Configure New Disk For Solaris.

Before attaching our USB mirror and the device we need to perform a few configuration steps on the new Laptop's internal disk. Specifically creating a Solaris2 partition and labeling it.

Boot new laptop into a Solaris Live CD or Text Install environment, and get to a shell prompt. From here (as root), run format(1) to create a Solaris2 partition and label the partition.

  $ format
  Searching for disks...done

  AVAILABLE DISK SELECTIONS:
         0. c5t0d0 
            /pci@0,0/pci1043,15d7@1f,2/disk@0,0

From this we can see the internal disk is device c5t0d0, using the format(1) command and fdisk option we need to remove the NTFS partition we originally created and replace it with a Solaris2 partion.

  $ format c5t0d0

From menu displayed, choose fdisk option :

  $ format> fdisk

This should show what current partition layout is. For me it's showing three NTFS partitions, the largest being partition 3, which is the one I created in Windows 7 of all the available left over space. Simply delete this partition via option 3 from the displayed menu. Before deleting make a mental note of the size of the partition specifically the starting cylinder and the length in cylinders. If using the whole disk then you can simply specify 100%.

Now to create the new Solaris2 partition choose Option 1 from the displayed menu :

  SELECT ONE OF THE FOLLOWING:
     1. Create a partition
     2. Specify the active partition
     3. Delete a partition
     4. Change between Solaris and Solaris2 Partition IDs
     5. Edit/View extended partitions
     6. Exit (update disk configuration and exit)
     7. Cancel (exit without updating disk configuration)
  Enter Selection: 1

  Select the partition type to create:
   1=SOLARIS2  2=UNIX        3=PCIXOS     4=Other
   5=DOS12     6=DOS16       7=DOSEXT     8=DOSBIG
   9=DOS16LBA  A=x86 Boot    B=Diagnostic C=FAT32
   D=FAT32LBA  E=DOSEXTLBA   F=EFI        0=Exit? 1

  Specify the percentage of disk to use for this partition
  (or type "c" to specify the size in cylinders). c
 
  Enter starting cylinder number: 7769
  Enter partition size in cylinders : 41144

  Should this become the active partition? If yes, it  will be activated
  each time the computer is reset or turned on.
  Please type "y" or "n". y

As can be seen from above, choose option 1 to create a new partition, select 1, for SOLARIS2 type partition, and 100% if using the entire disk, other wise specify the starting cylinder and number of cylinders to use. These you will have written down from the NTFS partition you deleted. You may need to +1 to the starting point and -1 from total number of cylinders, and finally "y" to make partition active. You should then see something like the following :

               Total disk size is 38912 cylinders
               Cylinder size is 16065 (512 byte) blocks

                                                 Cylinders
        Partition   Status    Type          Start   End   Length    %
        =========   ======    ============  =====   ===   ======   ===
            1                 NTFS              1     10       10    1
            1                 NTFS             10   7768      700   18  
            1       Active    Solaris2       7768  38912    31145   80


  SELECT ONE OF THE FOLLOWING:
     1. Create a partition
     2. Specify the active partition
     3. Delete a partition
     4. Change between Solaris and Solaris2 Partition IDs
     5. Edit/View extended partitions
     6. Exit (update disk configuration and exit)
     7. Cancel (exit without updating disk configuration)
  Enter Selection: 

Choose 6 to update disk configuration and exist fdisk and back to format(1) menu.

Now to label the disk, ensure a VTOC label is on the Solaris partition, sizes listed above and below are examples only and may not be what you will see on your actual disks.

 
  $ format> partition

  PARTITION MENU:
        0      - change '0' partition
        1      - change '1' partition
        2      - change '2' partition
        3      - change '3' partition
        4      - change '4' partition
        5      - change '5' partition
        6      - change '6' partition
        7      - change '7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
  $ partition> print
  Current partition table (original):
  Total disk cylinders available: 9419 + 2 (reserved cylinders)

  Part     Tag    Flag   Cylinders      Size         Blocks
  0 unassigned     wm     0             0       (0/0/0)             0
  1 unassigned     wm     0             0       (0/0/0)             0
  2     backup     wm     0 - 19453   149.03GB  (19454/0/0) 312528510 
  3 unassigned     wm     0             0       (0/0/0)             0
  4 unassigned     wm     0             0       (0/0/0)             0
  5 unassigned     wm     0             0       (0/0/0)             0
  6 unassigned     wm     0             0       (0/0/0)             0
  7 unassigned     wm     0             0       (0/0/0)             0
  8       boot     wm     0 -     0     7.84MB  (1/0/0)         16065
  9 unassigned     wm     0             0       (0/0/0)             0

  $ partition>

We need to ensure slice 0 is configured, in my case I configure it to be the whole size of the available partition, then actually label the partition.


  $ partition> 0
  Part     Tag    Flag   Cylinders      Size         Blocks
  0 unassigned     wm     0             0       (0/0/0)             0

  Enter partition id tag[unassigned]: root
  Enter partition permission flags[wm]: wm
  Enter new starting cyl[1]: 1
  Enter partition size[0b, 0c, 0e, 0mb, 0gb]: 19453c
  $ partition> print
  Current partition table (original):
  Total disk cylinders available: 9419 + 2 (reserved cylinders)

  Part     Tag    Flag   Cylinders      Size         Blocks
  0       root     wm     1 - 19453   149.02GB  (19453/0/0) 312512445
  1 unassigned     wm     0             0       (0/0/0)             0
  2     backup     wm     0 - 19453   149.03GB  (19454/0/0) 312528510 
  3 unassigned     wm     0             0       (0/0/0)             0
  4 unassigned     wm     0             0       (0/0/0)             0
  5 unassigned     wm     0             0       (0/0/0)             0
  6 unassigned     wm     0             0       (0/0/0)             0
  7 unassigned     wm     0             0       (0/0/0)             0
  8       boot     wm     0 -     0     7.84MB  (1/0/0)         16065
  9 unassigned     wm     0             0       (0/0/0)             0

  $ partition> label
  Ready to label disk, continue? y
  $ partition> quit

Again ensure disk is definitely labeled from the format prompt :

  $ format> label
  Ready to label disk, continue? y

Choose label from the format(1) menu, and select "y". If prompted for specific label type, ensure to choose SMI. Once labeled choose quit to get back to shell prompt.

  $ format> quit

3. Replacing Mirror Device.

My old laptop configuration consisted of an internal 200GB disk mirrored with an external 300GB USB drive, this resulted in a 200GB mirrored root pool. To migrate to new laptop I simply need my USB disk.

Again ensure new laptop is booted into a Solaris Live DVD or Text Install environment, and get to a prompt. Ensure the USB drive is attached to the machine, then simply import the degraded rpool.

  $ zpool import -f rpool
  $ zpool status
    pool: rpool
   state: DEGRADED
  status: One or more devices could not be opened. Sufficient replicas exist for
          the pool to continue functioning in a degraded state.
  action: Attach the missing device and online it using 'zpool online'
     see: http://www.sun.com/msg/ZFS-8000-2Q
    scan: resilvered 181M in 0h0m with 0 errors on Fri Mar 30 10:32:13 2012
  config:

	NAME                     STATE     READ WRITE CKSUM
	rpool                    DEGRADED     0     0     0
	  mirror-0               DEGRADED     0     0     0
	    2000330655777608537  UNAVAIL      0     0     0 was /dev/dsk/c4t0d0s0
	    c6t0d0s0             ONLINE       0     0     0

  errors: No known data errors

The UNAVAIL device from above is the original laptop's internal drive, which obviously is not available on the new machine. c6t0d0s0 is the USB device.

Now all we have to do is replace the unavailable device with the new internal disk partition we created above. Make sure you specify slice 0 e.g. s0

  $ zpool replace -f rpool 2000330655777608537 c5t0d0s0

The old device will now get replaced with the new internal disk device and depending on the size will may take a few hours to resilver everything from the USB disk onto the new internal device. Once completed zpool status should look like following :

  $ zpool status
    pool: rpool
   state: ONLINE
  config:

	NAME          STATE     READ WRITE CKSUM
	rpool         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    c5t0d0s0  ONLINE       0     0     0
	    c6t0d0s0  ONLINE       0     0     0

  errors: No known data errors

4. Install Grub To Internal Disk

Finally we need to install grub to the internal disk and ensure grub menu entries are correct for booting.

  $ installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t0d0s0

If you have other Dual bootable OS's you will need to edit /rpool/boot/grub/menu.lst and add entries for them here otherwise you won't be able to get to them. In my previous configuration I had consumed the entire disk on both devices for my root pool, in this case the grub menu entries are exactly the same for locating the disk and partition e.g. findroot (0,a) However my new scenario is slightly different, I am consuming the whole disk on the external USB disk but only partition 3 on the internal disk. Because of this I will need two separate grub menu entries depending on which disk I'd like to boot from. Here's an example of what mine looks like :

title Windows 7
rootnoverify (hd0,1)
chainloader +1

title Internal Disk Solaris 11
findroot (pool_rpool, 2, a)
bootfs rpool/ROOT/s11
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS -v
module$ /platform/i86pc/$ISADIR/boot_archive

title USB Disk Solaris 11
findroot (pool_rpool, 0, a)
bootfs rpool/ROOT/s11
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS -v
module$ /platform/i86pc/$ISADIR/boot_archive

Notice from above the only difference between the Internal disk and the USB is the findroot line for the internal disk showing 2 (partition 3), and on the USB disk 0 (partition 1).

Now export your new rpool and reboot, ensuring to remove the Live DVD/Text Install disc so that you boot from your new rpool.

  $ zpool export rpool
  $ init 5

Power on your laptop and at the grub prompt edit one of the Solaris entries and put "-r" at end of kernel line, and then select 'b' to boot. This will boot solaris in reconfigure mode and will attempt to ensure all the correct devices are loaded for your new laptop. This only needs to be done once e.g. initial boot.

Now you may find other issues with various devices on your laptop, like display, wifi etc... but that's an entirely new blog...

5. Auto expanding to maximize available space.

In my situation my rpool size was constrained by my original Laptop's internal disk size of 200GB. After following the above steps I now have a 200GB rpool recreated however I know the USB drive is 300GB in size and I know the SOLARIS2 partition on my new internal disk is 260GB, so I have 60GB of unused space.

Zpools have a property 'autoexpand' which by default is set to 'off'. This property if set to true will auto expand the pool to whatever space it can on the devices attached to the pool. So I can now simply set this property to 'on' and I will get the unused space automatically in my pool: Before :

  $ zpool list
  NAME   SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
  rpool  180G   135G  51.2G  72%  1.00x  ONLINE  -
Reset the autoexpand property, and re run zpool list.
  $ zpool set autoexpand=on rpool
  $ zpool list
  NAME   SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
  rpool  260G   135G 131.2G  51%  1.00x  ONLINE  -

If the size does not automatically expand as you expect you may need need to check the s0 size on your disks, one of them may be hard set to the lower size than the actual space on that solaris partition. Changing the VTOC s0/s2 (change both if necessary), will not effect anything as long as you are not changing the starting cylinder. After the expansion has completed, reset autoexpand property back to it's default of off.

  $ zpool set autoexpand=off rpool

6. Ensuring boot succeeds when one device is missing.

Potentially a bug but on a default solaris install when I disconnect the USB device the laptop fails to boot, mainly as the root pool is in a degraded state because a device is UNAVAIL. Pools now have a property "failmode". By default this is set to 'wait'. If you change this to a value of "continue', boot should succeed.
  $ zpool set failmode=continue rpool

Friday May 07, 2010

Bootable USB Mirror

Recently my laptop hard drive simply decided it had enough, unfortunately I had become a little lazy when it came to doing backup's so I got bitten pretty hard.

To ensure I don't get hit by this again, I got myself a USB portable drive and decided to use existing ZFS technology available on Solaris. I could have just attached the new device, set up a new pool on that device and performed regular zfs send's and I would have had a nice backup solution.

Another means would be to attach the new USB drive to my existing single pool laptop drive as a mirror and my data would get backed up automatically every time I attached the USB drive.

But I thought I'd go one step further, why not set up the attached device as bootable, and that way I can simply attach it to any other laptop/PC and boot up via USB and have the exact same environment that I have on my original laptop.

Here's the steps I performed to do this :

1. Format/Partition/Label New USB Disk

New drive needs to be specifically partitioned for Solaris. We need to identify the physical devices. Firstly lets identify the current laptop hard drive device and pool setup.

  $ pfexec zpool status
    pool: rpool
   state: ONLINE
  config:

	NAME          STATE     READ WRITE CKSUM
	rpool         ONLINE       0     0     0
	    c5t0d0s0  ONLINE       0     0     0

  errors: No known data errors

From above we can see laptop disk device is c5t0d0s0, which is a 200GB drive with a single default pool "rpool" for the entire disk.

Using the format(1) command we can identify the USB drive, ensure it's attached and power is on.

  $ pfexec format
  Searching for disks...done


  AVAILABLE DISK SELECTIONS:
         0. c5t0d0 
            /pci@0,0/pci1043,15d7@1f,2/disk@0,0
         1. c6t0d0 
            /pci@0,0/pci1043,15d7@1d,7/storage@1/disk@0,0

From this we can see the USB disk is device c6t0d0, using the format(1) command and fdisk option we need to remove any existing partitions and create just one new partition of type SOLARIS2 for 100% of the disk.

  $ pfexec format c6t0d0

From menu displayed, choose fdisk option :

  $ format> fdisk

This should show what current partition layout is, if there are existing partitions just delete them via option 3 from the displayed menu.

Once all existing partitions have been deleted, choose option 1 from displayed menu :

  SELECT ONE OF THE FOLLOWING:
     1. Create a partition
     2. Specify the active partition
     3. Delete a partition
     4. Change between Solaris and Solaris2 Partition IDs
     5. Edit/View extended partitions
     6. Exit (update disk configuration and exit)
     7. Cancel (exit without updating disk configuration)
  Enter Selection: 1

  Select the partition type to create:
   1=SOLARIS2  2=UNIX        3=PCIXOS     4=Other
   5=DOS12     6=DOS16       7=DOSEXT     8=DOSBIG
   9=DOS16LBA  A=x86 Boot    B=Diagnostic C=FAT32
   D=FAT32LBA  E=DOSEXTLBA   F=EFI        0=Exit? 1

  Specify the percentage of disk to use for this partition
  (or type "c" to specify the size in cylinders). 100

  Should this become the active partition? If yes, it  will be activated
  each time the computer is reset or turned on.
  Please type "y" or "n". y

As can be seen from above, choose option 1 to create a new partition, select 1, for SOLARIS2 type partition, and 100% for entire disk and finally "y" to make partition active. You should then see something like the following :

               Total disk size is 38913 cylinders
               Cylinder size is 16065 (512 byte) blocks

                                                 Cylinders
        Partition   Status    Type          Start   End   Length    %
        =========   ======    ============  =====   ===   ======   ===
            1       Active    Solaris2          1  38912    38912    100


  SELECT ONE OF THE FOLLOWING:
     1. Create a partition
     2. Specify the active partition
     3. Delete a partition
     4. Change between Solaris and Solaris2 Partition IDs
     5. Edit/View extended partitions
     6. Exit (update disk configuration and exit)
     7. Cancel (exit without updating disk configuration)
  Enter Selection: 

Choose 6 to update disk configuration and exist fdisk and back to format(1) menu.

Now to label the disk, ensure a VTOC label is on the Solaris partition, sizes listed below are examples only and may not be what you will see on your actual disks.

 format> partition

PARTITION MENU:
        0      - change '0' partition
        1      - change '1' partition
        2      - change '2' partition
        3      - change '3' partition
        4      - change '4' partition
        5      - change '5' partition
        6      - change '6' partition
        7      - change '7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> print
Current partition table (original):
Total disk cylinders available: 9419 + 2 (reserved cylinders)

Part     Tag    Flag   Cylinders      Size         Blocks
0 unassigned     wm     0             0       (0/0/0)             0
1 unassigned     wm     0             0       (0/0/0)             0
2     backup     wm     0 - 19453   149.03GB  (19454/0/0) 312528510 
3 unassigned     wm     0             0       (0/0/0)             0
4 unassigned     wm     0             0       (0/0/0)             0
5 unassigned     wm     0             0       (0/0/0)             0
6 unassigned     wm     0             0       (0/0/0)             0
7 unassigned     wm     0             0       (0/0/0)             0
8       boot     wm     0 -     0     7.84MB  (1/0/0)         16065
9 unassigned     wm     0             0       (0/0/0)             0

partition>

We need to ensure slice 0 is configured, in my case I configure it to be the whole size of the disk, also ensure disk is labeled :


  partition> 0
  Part     Tag    Flag   Cylinders      Size         Blocks
  0 unassigned     wm     0             0       (0/0/0)             0

  Enter partition id tag[unassigned]: root
  Enter partition permission flags[wm]: wm
  Enter new starting cyl[1]: 1
  Enter partition size[0b, 0c, 0e, 0mb, 0gb]: 19453c
  partition> print
  Current partition table (original):
  Total disk cylinders available: 9419 + 2 (reserved cylinders)

  Part     Tag    Flag   Cylinders      Size         Blocks
  0       root     wm     1 - 19453   149.02GB  (19453/0/0) 312512445
  1 unassigned     wm     0             0       (0/0/0)             0
  2     backup     wm     0 - 19453   149.03GB  (19454/0/0) 312528510 
  3 unassigned     wm     0             0       (0/0/0)             0
  4 unassigned     wm     0             0       (0/0/0)             0
  5 unassigned     wm     0             0       (0/0/0)             0
  6 unassigned     wm     0             0       (0/0/0)             0
  7 unassigned     wm     0             0       (0/0/0)             0
  8       boot     wm     0 -     0     7.84MB  (1/0/0)         16065
  9 unassigned     wm     0             0       (0/0/0)             0

  partition> label
  Ready to label disk, continue? y
  partition> quit

Again ensure disk is definitely labeled from the format prompt :

  $ format> label
  Ready to label disk, continue? y

Choose label from the format(1) menu, and select "y". If prompted for specific label type, ensure to choose SMI. Once labeled choose quit to get back to shell prompt.

  $ format> quit

3. Attach USB Disk To Main Pool Creating Mirror

Now to attach the new USB drive to the existing rpool and thus creating a mirror. This time ensure you use s0 (slice 0) at end of device.

  $ pfexec zpool attach -f rpool c5t0d0s0 c6t0d0s0

Once this completes run zpool status again, and you will see the new device as part of your new mirror configuration, and will also show percentage re-silvering. It took 1hr 45mins to resilver a 200GB disk, and 120GB of data.

  $ zpool status
    pool: rpool
   state: ONLINE
  config:

	NAME          STATE     READ WRITE CKSUM
	rpool         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    c5t0d0s0  ONLINE       0     0     0
	    c6t0d0s0  ONLINE       0     0     0  1.18G resilvered

  errors: No known data errors

4. Install Grub on USB Disk

Finally we need to install grub on the USB disk to ensure it is bootable :

  $ pfexec installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c6d0s0

5. Some Caveats

Whilst I definitely think this is a great backup solution, it doesn't come without some caveats/kinks that I've come across over the first few days with this setup.

  • Ensure your USB disk is at least as big as your laptop drive. In a ZFS mirror configuration, the smallest disk dictates the size of the pool.
  • Always have your USB drive connected a boot time. For some reason Solaris will not boot unless your drive is connected at boot time. This to me seems like a bug.
  • When shutting down, remember the exact USB port your USB drive was connected to, on some occasions I've noticed Solaris would not boot unless the USB drive was connected to the exact same port when shutdown.
  • NEVER shutdown whilst your USB drive is on offline mode. You can take your USB drive offline via zfs offline pool disk, however if you shutdown whilst it is offline (and disconnected) Solaris simply hangs when attempting to boot, again another possible bug. To recover from this situation I had to do the following :
    • Boot into LiveCD environment of a Solaris build of the same or more recent than currently installed build on your laptop. Ensure USB drive is connected when booting.
    • Once booted import your pool, and bring USB drive back online :
        $ zpool import -f rpool
        $ zpool online rpool %ltusbdisk%gt
      
    • Now reboot and you should be good to go.

Monday Feb 15, 2010

Automated Installer Setup And Virtual Box

The best way to investigate and play with a technology is within a virtual environment. So with that in mind I wanted to try and setup an Automated Install (AI) in a VirtualBox environment.

NOTE : Blogs by Chris Armes, Configuring an auto install client / server setup in VirtualBox and Joe Vlcek, Automated Install Using VirtualBox, were used extensively as references in following this process.

NOTE : All of the commands I performed as root. Run following command before starting :

   $ pfexec su

NOTE : One caveat I will mention, I've noticed that getting the network set up correctly BEFORE creating the install service and creating the initial DHCP server settings is vital, and changes to the AI server VM network setup, requires and reset of the DHCP server and the install-services.

1. Create And Install VirtualBox Server VM

Create a new guest VM in VirtualBox for the AI server, with following setup, ensure you configure network interfaces before booting up, Network Adapter 1, should be left as the default value of NAT.

  Name :
    ai-server

  Network Interfaces :
    Adapter 1: Intel PRO/1000 MT Desktop (NAT)
    Adapter 2: Intel PRO/1000 MT Desktop (internal network, 'AI')

  Base Memory :
    1024 MB

Install OpenSolaris 2009.06 or later in this guest. ISO can be got from http://hub.opensolaris.org/bin/view/Main/TryOpenSolaris.

NOTE: Remeber the name you provided when installing as you'll use this later, the default is "opensolaris".

2. Configure AI-Server VM.

Once installed, reboot into the AI-Server VM, and perform the following steps :

2.1. Set preferred pkg repository

By default OpenSolaris install's will point to publisher http://pkg.opensolaris.org/release, if you want to continue using the released repository then thats fine, however I prefer to use the latest dev repository.
   $ pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
Make sure your preferred publisher is actually working, try doing a simple search to determine if it's contactable and returning information e.g
   $ pkg search -r entire
Should return loads of information, if not you could try the internal only repo http://ipkg.sfbay/dev.

2.2. Install required installadm tools

   Recent Builds of OpenSolaris :
   $ pkg install install/installadm

   Old package name :
   $ pkg install SUNWinstalladm-tools 
   

2.3. Configure network SMF services

By default NWAM is turned on, you need to disable this and enable network/physical:default SMF instead :

   $ svcadm disable network/physical:nwam
   $ svcadm enable network/physical:default

2.4. Configure Network Adapter 1 to be DHCP

   $ ifconfig e1000g0 dhcp
   $ touch /etc/dhcp.e1000g0

Everytime you boot your ai-server VM you will need to ensure this interface is plumbed and the set up for dhcp, I've yet to determine why.

   $ ifconfig e1000g0 plumb
   $ ifconfig e1000g0 dhcp

2.5. Configure Network Adaptor 2, Step 1 : /etc/hosts

Edit /etc/hosts adding the following lines

   10.0.2.15       ai-server
   192.168.2.50    ai-server-private
   192.168.2.60    ai-client0
   192.168.2.61    ai-client1
   192.168.2.62    ai-client2
   192.168.2.63    ai-client3
   192.168.2.64    ai-client4

ai-server is the name I called my Server VM when installing, by default this is called opensolaris. Just check output from uname -n and use the output from that. You can add as many ai-client lines as you like, I just added 5.

Also comment out the lines ::1 and 127.0.0.1 as follows

   #::1 ai-server ai-server.local localhost loghost
   #127.0.0.1   ai-server ai-server.local localhost loghost

2.6. Configure Network Adaptor 2, Step 2 : /etc/netmasks

Edit /etc/netmasks adding the following lines :

   10.0.2.0       255.255.255.0
   192.168.2.0    255.255.255.0

2.7. Configure Network Adaptor 2, Step 3 : hostname.e1000g1

Enter the IP assigned to the server in /etc/hosts into this file:

Enter the server name assigned to interface /etc/hosts :

   $ cat hostname.e1000g1
   ai-server-private

2.8. Configure Network Adaptor 2, Step 3 : ifconfig

   $ ifconfig e1000g1 plumb
   $ ifconfig e1000g1 inet 192.168.2.50 netmask 255.255.255.0 broadcast 192.168.2.0
   $ ifconfig e1000g1 up

2.9. Enable ipv4 fowarding and routing:

   $ routeadm -e ipv4-fowarding
   $ routeadm -e ipv4-routing
   $ routeadm -u

2.10. Ensure Your name service is using DNS on ai-server VM

   $ grep \^ipnodes /etc/nsswitch.conf | grep dns
   ipnodes: files dns # Added by DHCP

   $ grep \^hosts /etc/nsswitch.conf | grep dns
   hosts: files dns # Added by DHCP

2.11. Ensure /etc/resolv.conf contains valid IP's of DNS servers

On ai-server VM :

   $ cat /etc/resolv.conf
   domain ireland.sun.com
   111.111.22.01
   111.111.22.02

NOTE : These IP's are for illustration purposes only, and don't actually exist.

2.12. Ensure dns SMF services are enabled :

   $ svcs dns/client:default 
   online         12:44:41 svc:/network/dns/client:default

   $ svcadm enable network/dns/multicast:default
   $ svcs dns/multicast:default
   online         12:44:41 svc:/network/dns/multicast:default

2.13. Configure NAT and enable ipfilter SMF service

Create /etc/ipf/ipnat.conf :

   $ echo "map e1000g0 192.168.2.0/24 -> 0/32" > /etc/ipf/ipnat.conf

Enable ipfilter SMF service :

   $ svcadm enable network/ipfilter:default

This can sometimes timeout and fail, depending on your hardware setup, to get around this simply increase the start/timeout_seconds SMF property, and try restarting.

   $ svcadm disable network/ipfilter
   $ svccfg -s network/ipfilter setprop start/timeout_seconds = 90
   $ svcadm refresh network/ipfilter
   $ svcadm enable network/ipfilter

2.14. Ensure route:default SMF service is enabled

Ensure route:default SMF service is online :

   $ svcs routing/route:default
   STATE     STIME    FMRI
   online    10:21:13 svc:/network/routing/route:default

If not online then enable it :

   $ svcadm enable /network/routing/route:default

3. Download AI Client Image

Download an AI ISO image to use with your automated install, this can be got from http://www.opensolaris.com/get/index.jsp.

Before downloading I created image and server location areas :

   $ mkdir /export/home/ai-images
   $ mkdir /export/home/ai-servers

Here's a convenient download command :

   $ mkdir /export/home/ai-images
   $ wget http://dlc.sun.com/osol/opensolaris/2009/06/osol-0906-ai-x86.iso 

4. Create Install Service

When creating your first install service you need to supply DHCP network information, e.g -i and -c flags. Subsequent create-service calls do not require this information.

   $ installadm create-service -n osol-0906-x86 -i 192.168.2.60 -c 5 -s /export/home/ai-images/osol-0906-ai-x86.iso /export/home/ai-servers/osol-0906-ai-x86
   Setting up the target image at /export/home/ai-servers/osol-0906-x86 ...
   Registering the service osol-0906-x86._OSInstall._tcp.local
   Creating DHCP Server
   Created DHCP configuration file.
   Created dhcptab.
   Added "Locale" macro to dhcptab.
   Added server macro to dhcptab - ai-server.
   DHCP server started.
   dhtadm: Unable to signal the daemon to reload the dhcptab
   Unable to determine the proper default router
   or gateway for the 192.168.2.0 subnet. The default
   router or gateway for this subnet will need to
   be provided later using the following command:
      /usr/sbin/dhtadm -M -m 192.168.2.0 -e  Router=<address> -g
   Added network macro to dhcptab - 192.168.2.0.
   Created network table.
   adding tftp to /etc/inetd.conf
   Converting /etc/inetd.conf
   copying boot file to /tftpboot/pxegrub.I86PC.OpenSolaris-1
   Service discovery fallback mechanism set up

This will create install service osol-0906-x86. If you wanted to create another service with a different AI Iso Image, e.g. dev-131 you would use the following :

   $ installadm create-service -n dev-131 -s /export/home/ai-images/dev-131.iso /export/home/ai-servers/dev-131
   Setting up the target image at /export/home/ai-servers/dev-131 ...
   Registering the service dev-131._OSInstall._tcp.local
   Service discovery fallback mechanism set up

5. Ensure tftp SMF service is enabled

Initial call to installadm create-service will add an entry to /etc/inetd.conf to start the tftp SMF Service, at this point just make sure it's enabled :

   $ svcs -a | grep tftp
   online   11:46:48 svc://network/tftp/udp6:default

If it's not online then enable it :

   $ svcadm enable network/tftp/udp6:default   

6. Configure DHCP

Run dhcpmgr GUI tool

Select macros tab and double click on the dhcp_macro_osol-0906-x86 to bring up a macro properties dialog ensure the following macros are set :

  Option Name : Router
  Option Value : 192.168.2.50

  Option Name : DNSserv
  Option Value : <IP address(s) from /etc/resolv.conf, space delimeted>

  Option Name : BootSrvA
  Option Value : 10.0.2.15

NOTE : DNSserv will need to be changed when you move the system around to the correct DNS entries specifed in /etc/resolv.conf.

You will need to set these macros for each service that is created using create-service.

7. Verify Install Server Available

Launch browser and browse to http:/localhost:5555, you should see "Index of /" page. You can also use installadm list :

  $ installadm list
  Service Name  Status  Arch Port  Image Path
  ------------  ------  ---- ----  ----------
  osol-0906-x86 on      x86  46501 /export/home/ai-servers/osol-0906-x86
  dev-131       on      x86  46502 /export/home/ai-servers/dev-131

8. Create And Configure VirtualBox Client VM

Create another VirtualBox VM for the client :

  Name :
    ai-client0

  Network Interfaces :
    Adapter 1: Intel PRO/1000 MT Desktop (internal network, 'AI')

  Base Memory :
    1024 MB

  Boot Sequence :
    Network, Floppy, CD/DVD ROM, Hard Disk

You can assign a specific client to use a specific install service using the network adapters MAC address. This can be seen by clicking on advanced when viewing the network adaptor, before the VM guest is started.

To assign this client to the dev-131 install service you would do the following on the server VM, lets assume MAC address is 08:00:27:28:C3:1D :

   $ installadm create-client -n dev-131 -e 08:00:27:28:C3:1D
   Setting up X86 client...
   Service discovery fallback mechanism set up
   Enabled network boot by adding macro named 08002728C31D
   to DHCP server with:
     Boot server IP     (BootSrvA) : 192.168.2.50
     Boot file          (BootFile) : 08002728C31D

Now when you boot the client it will specificaly attempt to install using the dev-131 install service.

9. Boot and Install Client

Start up your new VirtualBox Client, it should boot from the network and get a DHCP address assigned from the ai-server VM, and should present you with a grub menu item to perform an automated install.

10. Amending the default manifest.

For more information see the section "How to change or replace the default manifest?" at http://dlc.sun.com/osol/docs/content/dev/AIinstall/customai.html#changeai.

10.1. Determine the target image path for your specific service :

   $ svccvg -s :/system/install/server:default listprop AI<svcname>/image_path
Where svcname is your newly created install service e.g. :
   $ svccvg -s :/system/install/server:default listprop AIosol-0906-x86/image_path
   AIosol-0906-x86/image_path  astring  /export/home/ai-servers/osol-0906-x86

10.2. Make a copy of the default manifest

   $ cp /export/home/ai-servers/osol-0906-x86/auto_install/default.xml /tmp

10.3. Edit your copied /tmp/default.xml

Ensure <ai_manifest name> tag value remains "default".

Revise <ai_embedded_manifest> and <sc_embedded_manifest> sections as necessary.

To change the publisher from the default of pkg.opensolaris.org/release, simply change the value of <main utl> tag under <ai_pkg_repo_default_publisher>.

  From
   <main url="http://pkg.opensollaris.org/release" publisher="opensolaris.org"/>

  From
   <main url="http://ipkg.sfbay/dev" publisher="opensolaris.org"/>

now using installadm add this new manifest to the service overriding the previous default.xml

   $ installadm add -m /tmp/default.xml -n osol-0906-x86

Installing a new client from this service will now reference the new default.xml manifest.

11. Enabling Debug/Verbose.

When doing an AI install verbose can be enabled, which simply outputs extra debug info to /tmp/install_log. This is achieved via a boot argument on the install client. There are three methods of enabling this.

11.1. Enabling debug at install service creation time

To enable debug output when creating your install service, just use the -b option for installadm as follows :

   $ installadm create-service -n dev-131 -b install_debug=enable -s /export/home/ai-images/dev-131.iso /export/home/ai-servers/dev-131
   Setting up the target image at /export/home/ai-servers/dev-131 ...
   Registering the service dev-131._OSInstall._tcp.local
   Service discovery fallback mechanism set up

Remember any subsequent clients created using installadm create-client for this install service MUST also have the -b install_debug=enable, set via the create-client command line, as it will not be turned on by default. (Potentially a bug).

11.2. Enabling debug for specific client via installadm.

Specific client's can be assigned to boot from specific install servers, this is achieved by associating the clients MAC address with a specific install service, as shown in section 8 above. the create-client installadm subcommand can also take a -b option to specify boot arguments.

   $ installadm create-client -b install_debug=enabled -n dev-131 -e 08:00:27:28:C3:1D
   Setting up X86 client...
   Service discovery fallback mechanism set up
   Enabled network boot by adding macro named 08002728C31D
   to DHCP server with:
     Boot server IP     (BootSrvA) : 192.168.2.50
     Boot file          (BootFile) : 08002728C31D

11.3. Enabling debug via grub menu editing.

For already existing services you can enable debug output by editing the grub menu entry at boot time and adding the boot argument. When the install client gets to grub menu, select grub entry you want to edit and press "e", then press "e" again to edit the kernel line, scroll back using arrow key until you get to "-B". Just after this add " install_debug=enabled,", then press "b" to boot.

12. Troubleshooting

The following is a list of issues that may arise and how one might resolve them.

Auto Install DNS Failure

If client auto install is failing with the following install_log excerpt :

    <TRANSFER_MOD_E Nov  2 06:56:35> pkg image-create: The URI 'http://pkg.opensolaris.org' does not appear to point to a valid pkg server.
    <TRANSFER_MOD_E Nov  2 06:56:35> Please check the server's address and client's network configuration.
    <TRANSFER_MOD_E Nov  2 06:56:35> Additional details:
    <TRANSFER_MOD_E Nov  2 06:56:35>
    <TRANSFER_MOD_E Nov  2 06:56:35> Unable to contact valid package server
    <TRANSFER_MOD_E Nov  2 06:56:35> Encountered the following error(s):
    <TRANSFER_MOD_E Nov  2 06:56:35> Unable to contact any configured publishers. This is likely a network configuration problem.
    <TRANSFER_MOD_E Nov  2 06:56:35> Unable to initialize the pkg image area at /a
    ...
    <AI Nov  2 06:56:36> Automated Installation failed in Transfer module
    <AI Nov  2 06:56:36> Transferring the files from the source failed. Please see previous messages for more details

From within the Vbox Client VM try the following tests :

Is the repository pingable :

   $ ping pkg.opensolaris.org

If you get back: 'ping: unknown host pkg.opensolaris.org' the error may be DNS related.

If you get back: 'no answer from pkg.opensolaris.org' the error may be connectivity related

Is DNS Serer Reachable

Firstly get the server IP addresses via :

   $ grep \^nameserver /etc/resolve
   nameserver 192.168.0.1

Verify the server is reachable by pinging the ip addresses returned :

   $ ping 192.168.0.1
   192.168.0.1 is alive

If you get don't get returned IP is alive, then there is a DNS issue, talk to your DNS/Network administrator.

If /etc/resolv.conf is empty

If /etc/resolv.conf is empty, you can run :
   $ dhcpinfo DNSServ
and see if you get any addresses back. If nothing is returned your DHCP is mis-configured. If address are return, try pinging them as stated above.

DNS Server is alive

If the DNS server was alive, you can see if you are getting the right IP address for pkg.opensolaris.org by running :

   $ getent hosts pkg.opensolaris.org

If the IP address differs from a machine able to go to pkg.opensolaris.org, then you should contact your DNS/network administrator.

If getent is not returning anything, there is a routing issue between your Vbox network and the host network on which VirtualBox is running.

Check the value of the BootSrvA DHCP Macro property for your install service :

   $ dhcpinfo BootSrvA
   10.0.2.15

This should be the IP address of network adaptor 1 on your AI server VM. If not go back to your server, and set this value correctly and refresh the dhcp-server SMF service, and try rebooting the install AI client VM.

Another more extreme method is to re-initialize DHCP and re-install the install services :

Firstly delete the install service :

   $ installadm delete-service osol-0906-x86
   Boot archive /tftpboot/I86PC.OpenSolaris-1 for services is not a mountpoint.
   Boot archive (/tftpboot/I86PC.Opensolaris-1) for service osol-0906-x86 nto in vfstab
   Unable to find path /tftpboot/I86PC.OpenSolaris-1
   To delete DHCP macro, run the following command:
   /usr/sbin/dhtadm -D -m dhcp_macro_osol-0906-x86

Remove dhcp macro for this service :

   $ dhtadm -D -m dhcp_macro_osol-0906-x86

Now unconfgiure the DHCP server settings completly :

   $ dhcpconfig -U -x
   Unconfigure will stop the DHCP service and remove the DHCP configuration file.
   Are you SURE you want to unconfigure the DHCP service? (Y/N): y
   DHCP server shutdown.
   Deleted the server macro from dhcptab.
   Deleted table for network 192.168.2.0.
   Deleted the dhcptab.
   Deleted the DHCP configuration file.

Now recreate the install service and the dhcp starting from Step 4 above, and try booting into the AI client VM again.

Useful Resources

Friday Sep 04, 2009

How To : Use Distro Constructor To Add Custom Slim Install Packages

This is a Quick 'n' Dirty guide to creating an LiveCD ISO image for OpenSolaris on OpenSolaris.

NOTE: these notes are written assuming you have access to Sun's internal servers. You may be able to populate your own local IPS repo from some other source, I don't know how off hand but if you find out let me know and I'll add that info here aswell.

1. Some Environment setup

Install ss-dev and SUNWonbld packages, this will ensure all required dev tools are installed such as SunStudioExpress and create SUNWspro symbolic link :

  $ pfexec pkg install ss-dev SUNWonbld
  $ pfexec ln -s /opt/SunStudioExpress /opt/SUNWspro

Some other dependencies that I also installed were :

  $ pfexec pkg install SUNWgnome-common-devel
  $ pfexec pkg install SUNWpython-setuptools
  $ pfexec pkg install SUNWpython-pycurl

If the build fails, see what the build was looking for and use pkg search filename to see what other package might be necessary.

Ensure your PATH has SunStudioExpress and onbld included, and also has /usr/bin before /usr/gnu/bin :

  $ export PATH=/usr/bin:/opt/SunStudioExpress/bin:/opt/onbld/bin:$PATH

2. Create local IPS Repository

First clone the gate (IPS) source and build/install it :

  $ cd
  $ hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate
  $ cd ~/gate/src
  $ pfexec dmake packages

All gate related packages are available under : ~/gate/packages/i386.

On OpenSolaris, just enure you have the most up to date versions of these package installed from your default authority :

  $ pfexec pkg install SUNWipkg SUNWipkg-brand SUNWipkg-gui SUNWipkg-gui-data  SUNWipkg-gui-l10n SUNWipkg-um SUNWpython-cherrypy SUNWpython-mako SUNWpython-ply SUNWpython-pycurl

On SXDE you should remove the old versions of these packages and install the newly built ones. Before installing these packages ensure the pkg/server SMF service is disabled.

  $ svcs -a | grep pkg/server
  online         Feb_18   svc:/application/pkg/server:default

If online then disable :

  $ pfexec svcadm disable pkg/server
  $ svcs -a | grep pkg/server:default
  disabled       11:26:58 svc:/application/pkg/server:default

Now as root :

    $ pkgrm SUNWipkg
    $ cd ~gate/packages/i386
    $ ls
    SUNWipkg               SUNWipkg-gui           SUNWipkg-gui-l10n        
    SUNWpython-cherrypy    SUNWpython-ply         pkg5-c700981b0af2.pkg
    SUNWipkg-brand         SUNWipkg-gui-data      SUNWipkg-um              
    SUNWpython-mako        SUNWpython-pycurl      pkg5.pkg
    $ pkgadd -d pkg5.pkg
    $ pkginfo SUNWipkg

3. Set up Local IPS Repository.

By default pkg/server uses /var/pkg/repo for it's package directory. If you choose to use the default the first time you import just ensure it's contents are clear out :

  $ svcadm disable pkg/server
  $ pfexec rm -rf /var/pkg/repo/\*

Best practice though would be to create a new zfs location for local repo:

  $ pfexec zfs create rpool/ips
  $ pfexec zfs create rpool/ips/repo
  $ pfexec zfs set mountpoint=/export/ips/repo rpool/ips/repo

If you do create a new directory you need to set the pkg/inst_root property of the SMF service to point to this new location :

  $ pfexec svccfg -s pkg/server setprop pkg/inst_root = /export/ips/repo

You need to update the default port number from 80 to 10000, 10000 is the port used by distro-import :

  $ pfexec svccfg -s pkg/server setprop pkg/port = 10000

Refresh and enable pkg/server, and check your settings :

  $ pfexec scvadm refresh pkg/server
  $ pfexec svcadm enable pkg/server
  $ svcprop pkg/server | grep port
  pkg/port count 10000
  $ svcprop pkg/server | grep inst_root
  /export/ips/repo (or /var/pkg/repo)

You can browse the repo in browser with following URL :

     http://localhost:10000

4. Create your own slim install packages.

Get the sources:

  $ cd
  $ hg clone ssh://anon@hg.opensolaris.org/hg/caiman/slim_source

Read ~/slim_source/usr/src/README and follow the instructions on how to build.

You need five other packages of which four are not on the IPS servers but are available on http://dlc.sun.com/downloads. These are SUNWzoneint, SUNWwbint, SUNWldskint, SUNWinstallint, and SUNWswig. Just download and install them using pkgadd. The following shows a set of single line commands that will download and install in one go :

  $ pfexec pkgadd -d http://dlc.sun.com/osol/install/downloads/current/SUNWldskint.`uname -p`.pkg all
  $ pfexec pkgadd -d http://dlc.sun.com/osol/install/downloads/current/SUNWwbint.`uname -p`.pkg all
  $ pfexec pkgadd -d http://dlc.sun.com/osol/install/downloads/current/SUNWzoneint.`uname -p`.pkg all
  $ pfexec pkgadd -d http://dlc.sun.com/osol/install/downloads/current/SUNWinstallint.`uname -p`.pkg all
  $ pfexec pkg install SUNswig

Before you build if you have pulled your slim source to a location other than the default "slim_source", you need modify developer.sh and change CODEMGR_WS to point to this location. To build your packages just :

  $ cd ~/slim_source/usr/src
  $ ./nightly developer.sh

This will build packages under : ~/slim_source/packages/i386/nightly-nd, and the nightly build log is located in ../../log/\*/nightly.log.

5. Import packages to local Repo.

Now that we have some custom packages we want to generate a local repo which will contain latest nevada and your custom packages. To import latest nevada packages to your local IPS repo you need access to Sun internal servers. You can edit the Makefile located at :

    ~/gate/src/util/distro-import/Makefile

And set the following variables :

Name Description Value
WOS_PKGS preferred internal repository, I use the default WOS_PKGS=/net/netinstall.sfbay/export/nv/x/$(BUILDID)/Solaris_11/Product
REPO set to your local IPS repository that you are importing to. REPO=http://localhost:10000
NONWOS_PKGS Set this to the location of your distro specific packages, these include IPS and Caiman Install packages. I just leave it as the default : NONWOS_PKGS=/net/paradise.sfbay/export/integrate_dock/nv/$(NONWOS_DOCK)/all /net/paradise.sfbay/export/integrate_dock/nv/$(NONWOS_DOCK)/$(ARCH)
TEST_PKGS Set this to the location of your custom packages you want in the repo. Default is blank : TEST_PKGS=

NOTE : If you don't have any custom packages leave TEST_PKGS blank.

Save and exit the Makefile. The following manifest file can be used to initialize your pkg/server service back to default values. Before importing you should edit the file and ensure the pkg/inst_root and pkg/port are assigned the correct values e.g. /export/ips/repo and 10000 :

  $ cd ~/gate/src/svc
  $ pfexec svcadm disable pkg/server
  $ pfexec svccfg import pkg-server.xml
  $ pfexec svcadm refresh pkg/server
  $ pfexec svcadm enable pkg/server

Change directory back into distro-import, and using a the specific build number use redist_import script to populate your local IPS repo. e.g. for build 121 :

  $ cd ~/gate/src/util/distro-import
  $ make 121/redist_import

NOTE:This will fail if you have the JDS CBE included on your PATH, e.g. ensure /opt/dtbld/bin is NOT included in your PATH.

This step can take quite a while. In your browser, keep an eye on http://localhost:10000 — it starts get new packages after a while.

After an initial import to your local IPS repo, if you created a new zfs filesystem for your local repo, it's best to create a snapshot at this time, this will allow you to return to a clean initial repo without haveing to re-import everything again :

  $ pfexec zfs snapshot rpool/ips/repo@virgin 

To roolback to this snapshot at a later time you can run the following :

  $ pfexec zfs rollback rpool/ips/repo@virgin 

There are some alternative imports which can be run but they have caveats:

make 121/redist_import will pull in all redistributable packages and make the entire package
make 121/slim_import will pull in just slim (install) packages
make 121/redist_import -j will pull in just the command-line listed packages

6. Install or Push local test packages into your local repo.

There are a number of methods that can be used to push some custom packages into your local repo.

  • 1. Use solaris.py to push into your local repo

    To push custom SUNWgui-install into http://localhost:10000 repo :

      $ cd ~/gate/src/util/distro-import/
      $ mkdir gui-install
      $ cat > gui-install/SUNWgui-install << EOF
      > package SUNWgui-install
      > import SUNWgui-install
      > end package
      > EOF
    

    The above script may also contain other commands if required such as adding users/groups which might be required by specific packages. See ~/gate/src/utils/distro-import/105/common/SUNWslim-utils as an example of how to create a user. Now create a metafile pointing to the gui-install stuff :

      $ cd ~/gate/src/util/distro-import/
      $ cat > gui-install.list << EOF
      > include gui-install/SUNWgui-install
      > EOF
    

    Finally run solaris.py to push the SUNWslim-utils package into the repo :

      $ ./solaris.py -b 6.6 -d -s http://localhost:10000 -w ~/slim_source/packages/i386/nightly-nd slim-utils.list
    

    Pay attention to the above. The "-b 6.6" specifies the revision. You just bump it up higher and higher to make sure that version is found. The ~/slim_source/packages/i386/nightly-nd is the directory containing the packages.

    You can verify that the right version was pushed with the following :

      $ pkg image-create -F -a test=http://localhost:10000 /tmp/test-image
    
      $ pkg -R /tmp/test-image install SUNWgui-install
      DOWNLOAD                                  PKGS       FILES    XFER (MB)
      Completed                                  1/1       30/30      0.0/0.0
    
      PHASE                                        ACTIONS
      Install Phase                                  58/58
      PHASE                                          ITEMS
      Reading Existing Index                           8/8
      Indexing Packages                                1/1
    
      $ pkg -R /tmp/test-image list -av SUNWgui-install
      FMRI                                                           STATE      UFIX
      pkg:/SUNWgui-install@0.5.11,5.11-6.6:20090219T082311Z           installed  ----
    

    Note the -6.6 at the end of the version shown in the list above.

    In order for distro_const to include the correct version of your custom package in the ISO, the package entire needs to be rebuilt so that it's manifest includes your package version.

    To achieve this firstly go back to your IPS gate source and edit Makefile :

      $ cd ~/gate/src/util/distro-import
      $ vi Makefile
    

    Edit the Makefile, and comment out the recreation of entire.incorporation, by changing line :

          From : entire: $(BUILDID)/entire.incorporation
          To   : entire: #$(BUILDID)/entire.incorporation
    

    Now edit the specific build entire.incorporation file :

      $ vi 121/entire.incorporation
    

    and change the version of the package listed here to your local version e.g. for SUNWgui-install :

          From : depend fmri=SUNWgui-install@0.5.11-0.121 type=incorporate
          To   : depend fmri=SUNWgui-install@0.5.11-6.6 type=incorporate
    

    Now re-build the entire package :

      $ make 121/entire
      PKG_REPO=http://localhost:10000 ./import_manifest_file \\
          entire@0.5.11,5.11-0.`echo 121 | tr -d '[a-z]'` \\
          121/entire.incorporation
      PUBLISHED
      pkg:/entire@0.5.11,5.11-0.121:20090903T143856Z
    

    Do a quick restart of the pkg/server, even though I'm not sure it's definitely required. Now check http://localhost:10000, search for package entire, once found click the "manifest" link and browse down to your custom package, the version listed here should be the one you published.

  • 2. Import at initial local repo creation time

    When your first populate your local IPS repo you could set TEST_PKGS as stated in the section above to point to the location of your custom SVR4 packages, it will automatically grab all packages here and install them in your local repo.

  • 3. Re-run redist_import to re-populate your local repo.

    Quite similar to previous method, except in this case you already have a populated local repo, you can re-run redist_repo to re-populate your local IPS repo. Before doing so you can set the environment variable TEST_PKGS to point to your custom packages :

      $ export TEST_PKGS=~/slim_source/packages/i386/nightly-nd
    

    Then re-run redist_import specifying -e make flag to override environment variables in the Makefile from current shell environment:

      $ cd ~/gate/src/util/distro_import
      $ /usr/bin/make -e 122/redist_import
    

NOTE:Options 2 & 3 above can take quite a bit of time depending on network connectivity and hardware specification.

7. Build Distribution ISO.

NOTE:If you have run disto_const on a build of OpenSolaris prior to snv_129, I would recommend that you destroy the default zfs distro constructor datasets. distro_const will recreate these when run again. One of the possible errors you may encounter is an error where distro_const cannot find directory boot_archive, this is because bootroot was renamed to boot_archive. To destroy run the following commands :

  $ pfexec zfs destroy -r rpool/dc

Ensure SUNWdistro-const is installed :

  $ pfexec pkg install SUNWdistro-const

Now make a copy of the distro XML manifest that you want to use to generate the ISO :

  $ mkdir ~/mydistro
  $ cp /usr/share/distro_const/slim_cd/all_lang_slim_cd_x86.xml ~/mydistro/mydistro.xml
  $ cd ~/mydistro

Now edit ~/mydistro/mydistro.xml :

  • Change the distribution name to one of your choice :
         From : <distribution name="OpenSolaris">
         To   : <distribution name="MyDistro">
    
  • Change url element of <pkg_repo_default_authority> section from :
          From : pkg.opensolaris.org/release
          To   : localhost:10000
    
  • Change authname element of <pkg_repo_default_authority> section from :
          From : opensolaris.org
          To   : myrepo
    
  • Optional to remove generation of usb image.
    If you don't want the distro.usb image to be generated then comment out or remove the usb finalizer script from within mydistro.xml. Go to section <finalizer> section and remove/comment out the following <finalizer> section :
       <script name="/usr/share/distro_const/create_usb">
          <checkpoint
             name="usb"
             message="USB image creation"/>
       </script>
    
  • If you are using a custom finalizer script to pkgadd custom packages as detailed in the last section below, ensure the required changes are made to the <finalizer> section.

Now run distro_const on this mydistro.xml file to generate your ISO :

  $ pfexec distro_const build ~/mydistro/mydistro.xml

By default this will generate an installation under ZFS dataset :

    rpool/dc

If this filesystem does not exist it will be created. The media once generated will exist in :

    /rpool/dc/media

8. Test Your LiveCD ISO Using VirtualBox.

Boot from /rpool/dc/media/MyDistro.iso.
9. Use of Custom Finalizer script to install custom packages.

An alternative to pushing custom packages into your local repo, and regenerateing the entire package, you could simply use a finalizer script to add these packages to distro_const's default pkg_image area when it is creating the image.

When distro_const is run to create your distro, a number of finalizer scripts are run for the various steps to generate the distribution ISO image. These scripts are defined in the <'finalizer> section within the xml manifest used by distro_const. e.g. mydistro.xml mentioned above.

You can add your own finalizer script to pkgadd your custom packages to the image install area before the distro itself is constructed. The first script is "pre_bootroot_pkg_image_mod", add your custom finalizer install script just after this and just before the "slimcd_pre_bootroot_pkg_image_mod" script. It might look like the following :

   <script name="~/mydistro/add-custom-pkgs">
      <checkpoint 
         name="add-custom-pkgs" 
         message="Add Custom pkgs"/>
   </script>

The script add-custom-pkgs might contain the following :

  $ cat ~/mydistro/add-custom-pkgs
  #!/usr/bin/ksh93

  ADMIN_FILE=/tmp/admin.$$
  cat << \\ADMIN_EOF > $ADMIN_FILE
  mail=
  instance=unique
  partial=nocheck
  runlevel=nocheck
  idepend=nocheck
  rdepend=nocheck
  space=nocheck
  setuid=nocheck
  conflict=nocheck
  action=nocheck
  networktimeout=60
  networkretries=3
  authentication=quit
  keystore=/var/sadm/security
  proxy=
  basedir=default
  ADMIN_EOF

  pkgadd  -a ${ADMIN_FILE}  -d ~/slim_source/packages/i386/nightly-nd -R /rpool/dc/build_data/pkg_image SUNWgui-install

This script simply takes a custom SVR4 package version of SUNWgui-install which I have built in ~/slim_source/packages/i386/nightly-nd and installs it to the default package image area /rpool/dc/build_data/pkg_image.

Once you've edited your manifest xml file and added your custom finalizer script entry, just re-run distro_const to generate your ISO as specified in the previous section.

10. Using Distro Constructor Checkpoints

Distro Constructor very conveniently builds your image in stages, each of these stages is checkpointed with a zfs snapshot, which makes it really simple to pause or restart a build from a given checkpoint. See Building An Image In Stages for complete details, I'll briefly describe what I do here.

To get a list of checkpoints from my presiously run mydistro.xml described in previous section :

  $ pfexec distro_const build -l mydistro.xml
  /usr/share/distro_const/DC-manifest.defval.xml validates
  /tmp/slim_cd_x86_temp_28221.xml validates

  Step           Resumable Description
  -------------- --------- -------------
  im-pop             X     Image area creation
  add-custom-pkgs    X     Add custom pkgs
  im-mod             X     Image area modifications
  slim-im-mod        X     Slim CD Image area Modifications
  ba-init            X     Boot archive initialization
  slim-ba-config     X     Slim CD boot archive configuration
  ba-config          X     Boot archive configuration
  ba-arch            X     Boot archive archiving (64-bit)
  ba-arch-32         X     Boot archive archiving (32-bit)
  slim-post-mod      X     Slim CD post boot archive image area modification
  grub-setup         X     Grub menu setup
  post-mod           X     Post boot archive image area modification
  gen-slim-cont      X     Generate Slim CD image content list
  iso                X     ISO image creation
  usb                      USB image creation

The "X" in the Resumable column indicates you can resume operation from this point with the -r flag. To resume from a give checkpoint this step must have already completed in a previous run. When using a finalizer script to generate my ISO's, If I have changed something and would like to regenerate my ISO, rather than having to regenerate the complete pkg-image from your preffered IPS Repo (step "im_pop"), I tend to simply re-run the distro_const build from the step "add-custom-pkgs".

The steps that I follow are :

  • When a new development build of OpenSolaris is made available I run a complete distro_const build from scratch e.g.
      $ pfexec distro_const build mydistro.xml
    
    If nothing has changed in the development repository that I am dependent on, then there is no need to re-run the population of the pkg-image for your ISO from an IPS repository, as this takes some time, it makes sense ot skip it if you can.
  • For each subsequent ISO generation I simply update my add-custom-pkgs script, and re-run distro_const starting at the add-custom-pkgs step :

      $ pfexec distro_const build -r add-custom-pkgs mydistro.xml
    
    This will skip the image population step (saving potentially hours of time), and will start with adding your custom packages. The -r option rolls back to the start of that step, e.g. before custom packages were added (ZFS Rocks !), and will re-add your custom packages all over again.

11. Summary And Links

Well done you made it to the end of what turned out to be a lot longer that I had originally planned. If you find errors in the above, please let me know and I will update the blog.

Some handy resources :

About

Install engineer at Oracle who is passionate about Music, Sport and has a soft spot for Solaris

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today