Friday May 07, 2010

Bootable USB Mirror

Recently my laptop hard drive simply decided it had enough, unfortunately I had become a little lazy when it came to doing backup's so I got bitten pretty hard.

To ensure I don't get hit by this again, I got myself a USB portable drive and decided to use existing ZFS technology available on Solaris. I could have just attached the new device, set up a new pool on that device and performed regular zfs send's and I would have had a nice backup solution.

Another means would be to attach the new USB drive to my existing single pool laptop drive as a mirror and my data would get backed up automatically every time I attached the USB drive.

But I thought I'd go one step further, why not set up the attached device as bootable, and that way I can simply attach it to any other laptop/PC and boot up via USB and have the exact same environment that I have on my original laptop.

Here's the steps I performed to do this :

1. Format/Partition/Label New USB Disk

New drive needs to be specifically partitioned for Solaris. We need to identify the physical devices. Firstly lets identify the current laptop hard drive device and pool setup.

  $ pfexec zpool status
    pool: rpool
   state: ONLINE
  config:

	NAME          STATE     READ WRITE CKSUM
	rpool         ONLINE       0     0     0
	    c5t0d0s0  ONLINE       0     0     0

  errors: No known data errors

From above we can see laptop disk device is c5t0d0s0, which is a 200GB drive with a single default pool "rpool" for the entire disk.

Using the format(1) command we can identify the USB drive, ensure it's attached and power is on.

  $ pfexec format
  Searching for disks...done


  AVAILABLE DISK SELECTIONS:
         0. c5t0d0 
            /pci@0,0/pci1043,15d7@1f,2/disk@0,0
         1. c6t0d0 
            /pci@0,0/pci1043,15d7@1d,7/storage@1/disk@0,0

From this we can see the USB disk is device c6t0d0, using the format(1) command and fdisk option we need to remove any existing partitions and create just one new partition of type SOLARIS2 for 100% of the disk.

  $ pfexec format c6t0d0

From menu displayed, choose fdisk option :

  $ format> fdisk

This should show what current partition layout is, if there are existing partitions just delete them via option 3 from the displayed menu.

Once all existing partitions have been deleted, choose option 1 from displayed menu :

  SELECT ONE OF THE FOLLOWING:
     1. Create a partition
     2. Specify the active partition
     3. Delete a partition
     4. Change between Solaris and Solaris2 Partition IDs
     5. Edit/View extended partitions
     6. Exit (update disk configuration and exit)
     7. Cancel (exit without updating disk configuration)
  Enter Selection: 1

  Select the partition type to create:
   1=SOLARIS2  2=UNIX        3=PCIXOS     4=Other
   5=DOS12     6=DOS16       7=DOSEXT     8=DOSBIG
   9=DOS16LBA  A=x86 Boot    B=Diagnostic C=FAT32
   D=FAT32LBA  E=DOSEXTLBA   F=EFI        0=Exit? 1

  Specify the percentage of disk to use for this partition
  (or type "c" to specify the size in cylinders). 100

  Should this become the active partition? If yes, it  will be activated
  each time the computer is reset or turned on.
  Please type "y" or "n". y

As can be seen from above, choose option 1 to create a new partition, select 1, for SOLARIS2 type partition, and 100% for entire disk and finally "y" to make partition active. You should then see something like the following :

               Total disk size is 38913 cylinders
               Cylinder size is 16065 (512 byte) blocks

                                                 Cylinders
        Partition   Status    Type          Start   End   Length    %
        =========   ======    ============  =====   ===   ======   ===
            1       Active    Solaris2          1  38912    38912    100


  SELECT ONE OF THE FOLLOWING:
     1. Create a partition
     2. Specify the active partition
     3. Delete a partition
     4. Change between Solaris and Solaris2 Partition IDs
     5. Edit/View extended partitions
     6. Exit (update disk configuration and exit)
     7. Cancel (exit without updating disk configuration)
  Enter Selection: 

Choose 6 to update disk configuration and exist fdisk and back to format(1) menu.

Now to label the disk, ensure a VTOC label is on the Solaris partition, sizes listed below are examples only and may not be what you will see on your actual disks.

 format> partition

PARTITION MENU:
        0      - change '0' partition
        1      - change '1' partition
        2      - change '2' partition
        3      - change '3' partition
        4      - change '4' partition
        5      - change '5' partition
        6      - change '6' partition
        7      - change '7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> print
Current partition table (original):
Total disk cylinders available: 9419 + 2 (reserved cylinders)

Part     Tag    Flag   Cylinders      Size         Blocks
0 unassigned     wm     0             0       (0/0/0)             0
1 unassigned     wm     0             0       (0/0/0)             0
2     backup     wm     0 - 19453   149.03GB  (19454/0/0) 312528510 
3 unassigned     wm     0             0       (0/0/0)             0
4 unassigned     wm     0             0       (0/0/0)             0
5 unassigned     wm     0             0       (0/0/0)             0
6 unassigned     wm     0             0       (0/0/0)             0
7 unassigned     wm     0             0       (0/0/0)             0
8       boot     wm     0 -     0     7.84MB  (1/0/0)         16065
9 unassigned     wm     0             0       (0/0/0)             0

partition>

We need to ensure slice 0 is configured, in my case I configure it to be the whole size of the disk, also ensure disk is labeled :


  partition> 0
  Part     Tag    Flag   Cylinders      Size         Blocks
  0 unassigned     wm     0             0       (0/0/0)             0

  Enter partition id tag[unassigned]: root
  Enter partition permission flags[wm]: wm
  Enter new starting cyl[1]: 1
  Enter partition size[0b, 0c, 0e, 0mb, 0gb]: 19453c
  partition> print
  Current partition table (original):
  Total disk cylinders available: 9419 + 2 (reserved cylinders)

  Part     Tag    Flag   Cylinders      Size         Blocks
  0       root     wm     1 - 19453   149.02GB  (19453/0/0) 312512445
  1 unassigned     wm     0             0       (0/0/0)             0
  2     backup     wm     0 - 19453   149.03GB  (19454/0/0) 312528510 
  3 unassigned     wm     0             0       (0/0/0)             0
  4 unassigned     wm     0             0       (0/0/0)             0
  5 unassigned     wm     0             0       (0/0/0)             0
  6 unassigned     wm     0             0       (0/0/0)             0
  7 unassigned     wm     0             0       (0/0/0)             0
  8       boot     wm     0 -     0     7.84MB  (1/0/0)         16065
  9 unassigned     wm     0             0       (0/0/0)             0

  partition> label
  Ready to label disk, continue? y
  partition> quit

Again ensure disk is definitely labeled from the format prompt :

  $ format> label
  Ready to label disk, continue? y

Choose label from the format(1) menu, and select "y". If prompted for specific label type, ensure to choose SMI. Once labeled choose quit to get back to shell prompt.

  $ format> quit

3. Attach USB Disk To Main Pool Creating Mirror

Now to attach the new USB drive to the existing rpool and thus creating a mirror. This time ensure you use s0 (slice 0) at end of device.

  $ pfexec zpool attach -f rpool c5t0d0s0 c6t0d0s0

Once this completes run zpool status again, and you will see the new device as part of your new mirror configuration, and will also show percentage re-silvering. It took 1hr 45mins to resilver a 200GB disk, and 120GB of data.

  $ zpool status
    pool: rpool
   state: ONLINE
  config:

	NAME          STATE     READ WRITE CKSUM
	rpool         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    c5t0d0s0  ONLINE       0     0     0
	    c6t0d0s0  ONLINE       0     0     0  1.18G resilvered

  errors: No known data errors

4. Install Grub on USB Disk

Finally we need to install grub on the USB disk to ensure it is bootable :

  $ pfexec installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c6d0s0

5. Some Caveats

Whilst I definitely think this is a great backup solution, it doesn't come without some caveats/kinks that I've come across over the first few days with this setup.

  • Ensure your USB disk is at least as big as your laptop drive. In a ZFS mirror configuration, the smallest disk dictates the size of the pool.
  • Always have your USB drive connected a boot time. For some reason Solaris will not boot unless your drive is connected at boot time. This to me seems like a bug.
  • When shutting down, remember the exact USB port your USB drive was connected to, on some occasions I've noticed Solaris would not boot unless the USB drive was connected to the exact same port when shutdown.
  • NEVER shutdown whilst your USB drive is on offline mode. You can take your USB drive offline via zfs offline pool disk, however if you shutdown whilst it is offline (and disconnected) Solaris simply hangs when attempting to boot, again another possible bug. To recover from this situation I had to do the following :
    • Boot into LiveCD environment of a Solaris build of the same or more recent than currently installed build on your laptop. Ensure USB drive is connected when booting.
    • Once booted import your pool, and bring USB drive back online :
        $ zpool import -f rpool
        $ zpool online rpool %ltusbdisk%gt
      
    • Now reboot and you should be good to go.

Wednesday Dec 17, 2008

How To : Build JDS (GNOME) on OpenSolaris

The second release of OpenSolaris is now officially available 2008.11, and it is a vast improvement over the first release 2008.05. A lot of new hardware drivers have been integrated helping things like audio to just work out of the box.

As OpenSolaris is being targeted at developers it makes sense that you should be able to build the Desktop on OpenSolaris with relative ease. Initially I thought it was going to be difficult but it's actually not, once you follow the following steps.

Bear in mind this is probably not the complete set of steps required and is a work in progess list :), So if you encounter something that I've missed let me know and I will update this blog entry.

1. Install Sun Studio compiler

This can be installed via the ss-dev package.

   $ pfexec pkg install ss-dev

2. Install Required Packages.

Install GNOME development tools package SUNWgnome-common-devel :

   $ pfexec pkg install SUNWgnome-common-devel

Install Perl XML::Parser package SUNWperl-xml-parser :

   $ pfexec pkg install SUNWperl-xml-parser

Install GNU gettext package SUNWgnu-gettext for po procesing :

   $ pfexec pkg install SUNWgnu-gettext

Install GNOME xml documentation packages SUNWgnome-xml\* :

   $ pfexec pkg install SUNWgnome-xml-root
   $ pfexec pkg install SUNWgnome-xml
   $ pfexec pkg install SUNWgnome-xml-share
   $ pfexec pkg install SUNWdoxygen

3. Install docbook catalog.

Once you've installed all the xml stylesheets, you still need to manually update the docbook catalog, this is to ensure xsltproc knows that DTD's are installed locally and it won't try to access the net for DTD's :

   $ pfexec /usr/share/sgml/docbook/docbook-catalog-install.sh

4. Install JDS CBE (Common Build Environment) 1.7+.

Download the CBE from :

   http://dlc.sun.com/osol/jds/downloadds/cbe/test/

Ensure you select the correct architecture. Once downloaded, extract the contents from the tarball.

To install run cbe-install. This needs to be run by a user with Primary Administrator privileges. Users wishing to run and CBE once installed successfully should have at least Software Installation privileges. Privilages are easy enough to add, by either manually editing the /etc/user_attr file or using usermod(1) with the -P CLI option.

NOTE : cbe-install will build and install SVR4 packages.

5. install latest pkgbuild to get IPS support.

CBE 1.7+ includes a version of pkgbuild which at the moment does not generate IPS packages. If you just want to compile packages and generate SVR4 packages then there is no need to perform this step.

Download latest pkgbuild from here :

   http://pkgbuild.sourceforge.net/download.php

Which at time of writing is version 1.3.98. Remove the CBE installed version :

   $ pfexec pkgrm SFpkgbuild

Use the installed CBE to build the downloaded latest pkgbuild package

   $ cd pkgbuild-1.3.98
   $ ./configure --prefix=/opt/dtbld/
   $ make
   $ pfexec make install

6. ACLOCAL

When building some packages you may encounter ACLOCAL build failure messages.

This is caused by aclocal being unable to locate CBE m4 files installed under /opt/dtbld/share/aclocal. By default it looks in /usr/share/aclocal, it also needs to look in /opt/dtbld/share/aclocal.

To fix this ensure the file /usr/share/aclocal/dirlist exists and contains the following :

 
   /usr/sfw/share/aclocal
   /opt/dtbld/share/aclocal

7. JDS Source and spec-files.

In order to build JDS packages you need to download the spec-files that are used by pkgbuild. You can get the latest set of JDS sources from the SVN repository with anonymous access :

   $ svn co svn+ssh://anon@svn.opensolaris.org/svn/jds/spec-files/trunk spec-files-trunk

This will check the trunk version of spec-files into the local directory spec-files-trunk.

Bear in mind trunk is the latest development branch of JDS and some components my not build as they are being currently developed. The GNOME version on !OpenSolaris 2008.11 is 2.24, and the spec-files and JDS sources for this version can be downloaded with anonymous access as follows :

   $ svn co svn+ssh://anon@svn.opensolaris.org/svn/jds/spec-files/branches/gnome-2-24 spec-files-2-24

This will check out the gnome-2-24 branch of spec-files into the local directory spec-files-2-24.

Before you can use _pkgtool_ to build JDS components, you need to ensure manpage tarballs and po-sun tarballs are created. To do this :

   $ cd spec-files-2-24/manpages
   $ make
   $ cd ../po-sun
   $ make

8. Building JDS packages.

You are now ready to build packages for JDS packages using pkgtool. Before using pkgtool or pkgbuild you need to ensure you have the correct environment conifgured, pkgtool comes with two convenience scripts for this very purpose. For csh shell users :

   $ source /opt/dtbld/bin/env.csh

For non csh shell users :

   $ . /opt/dtbld/bin/env.sh

Now pkgtool will be on your path and you can build your spec file :

   $ cd spec-files-2-24
   $ pkgtool --download build-only SUNWblah.spec

At the moment I would recommend using the flag --with-indiana-branding. But this will likely go away in the future.

9. Generating IPS Packages

This will of course build SVR4 packages, if you want to play with IPS you need to use the --ips CLI option for pkgtool. This will build local SVR4 packages, and generate IPS packages from these and by default will populate your local package repository, http://localhost:80/

Before doing this you will of course need to enable your local repository via the smf service pkg/server as follows :

   $ pfexec svcadm enable pkg/server

This will put your local IPS server online http://localhost:80/.

Now you can build your IPS package and have pkgtool populate the local server :

   $ pkgtool --download --ips build-only SUNWblah.spec

10. Installing Your IPS Packages

To actually install and test your local IPS repository and the packages that you have built. Firstly you need to add your local server as an authority :

   $ pfexec pkg set-authority -e -O http://localhost:80/ MyLocalRepo

This will create an authority called MyLocalRepo and enable it. To see what authorities exist :

   $ pkg authority
   AUTHORITY                           URL
   MyLocalRepo                         http://localhost:80/
   opensolaris.org (preferred)         http://pkg.opensolaris.org/dev/

You can see what packages exist in your local repository via Package Manager GUI application or the following command will also list them :

   $ pkg list -a | grep "MyLocalRepo"
or
   $ pkg list -av SUNWblah

If you don't see your package name listed you may need to refresh and indexes :

   $ pfexec pkg refresh --full
   $ pfexec pkg rebuild-index

Before proceeding and installing packages from your local repository it is recommended to take a snapshot of where you are currently at. This will give you the option to rollback to a given point if something goes wrong, ZFS rocks !. Use beadm to create a boot environment snapshot before you install from your local repository :

  $ beadm create -a opensolaris-snap-1

This will create a boot environment snapshot which you can boot from and will return you to a point before you installed anything from your local repo. See beadm man page for more information.

Before installing your packages, if it is already installed on your system but from another authority, probably best to uninstall it first :

   $ pfexec pkg uninstall SUNWblah

To install a package from your local repository you can use the Package Manager GUI application or the following command :

   $ pfexec pkg install pkg://MyLocalRepo/SUNWblah

Summary

So in 8 easy steps you are building desktop packages, in 10 steps you building IPS packages!!

These steps are relatively simple to follow, especially if you are already familiar with using the JDS CBE.

There are plans on making this process really seamless, and provide a meta IPS package, which when installed will simply pull down and install all the required packages in one swoop... it's a work in progress...

Happy building !!!

About

Install engineer at Oracle who is passionate about Music, Sport and has a soft spot for Solaris

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today