SVM root mirroring and GRUB

Although I haven't been working on SVM for over 6 months (I am working on Zones now), I still get questions about SVM and x86 root mirroring from time to time. Some of these procedures are different when using the new x86 boot loader (GRUB) that is now part of Nevada and S10u1. I have some old notes that I wrote up about 9 months ago that describe the updated procedures and I think these are still valid.

Root mirroring on x86 is more complex than is root mirroring on SPARC. Specifically, there are issues with being able to boot from the secondary side of the mirror when the primary side fails. On x86 machines the system BIOS and fdisk partitioning are the complicating factors.

The x86 BIOS is analogous to the PROM interpreter on SPARC. The BIOS is responsible for finding the right device to boot from, then loading and executing GRUB from that device.

All modern x86 BIOSes are configurable to some degree but the discussion of how to configure them is beyond the scope of this document. In general you can usually select the order of devices that you want the BIOS to probe (e.g. floppy, IDE disk, SCSI disk, network) but you may be limited in configuring at a more granular level. For example, it may not be possible to configure the BIOS to probe the first and second IDE disks. These limitations may be a factor with some hardware configurations (e.g. a system with two IDE disks that are root mirrored). You will need to understand the capabilities of the BIOS that is on your hardware. If your primary boot disk fails you may need to break into the BIOS while the machine is booting and reconfigure to boot from the second disk.

On x86 machines fdisk partitions are used and it is common to have multiple operating systems installed. Also, there are different flavors of master boot programs (e.g. LILO) in addition to GRUB which is the standard Solaris master boot program. The boot(1M) man page is a good resource for a detailed discussion of the multiple components that are used during booting on Solaris x86.

Since SVM can only mirror Solaris slices within the Solaris fdisk partition this discussion will focus on a configuration that only has Solaris installed. If you have multiple fdisk partitions then you will need to use some other approach to protect the data outside of the Solaris fdisk partition.

Once your system is installed you create your metadbs and root mirror using the normal procedures.

You must ensure that both disks are bootable so that you can boot from the secondary disk if the primary fails. You use the installgrub program to setup the second disk as a Solaris bootable disk (see installgrub(1M)). An example command is:

/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t1d0s0

Solaris x86 emulates some of the behavior of the SPARC eeprom. See eeprom(1M). The boot device is stored in the "bootpath" property that you can see with the eeprom command. The value should be assigned to the the device tree path of the root mirror. For example:


Next you need to modify the GRUB boot menu so that you can manually boot from the second side of the mirror, should this ever be necessary. Here is a quick overview of the GRUB disk naming convention.

(hd0),(hd1) -- first & second BIOS disk (entire disk)
(hd0,0),(hd0,1) -- first & second fdisk partition of first BIOS disk
(hd0,0,a),(hd0,0,b) -- Solaris/BSD slice 0 and 1 on first fdisk partition on the first BIOS disk

Hard disk names starts with hd and a number, where 0 maps to BIOS disk 0x80 (first disk enumerated by the BIOS), 1 maps to 0x81, and so on. One annoying aspect of BIOS disk numbering is that the order may change depending on the BIOS configuration. Hence, the GRUB menu may become invalid if you change the BIOS boot disk order or modify the disk configuration. Knowing the disk naming convention is essential to handling boot issues related to disk renumbering in the BIOS. This will be a factor if the primary disk in the mirror is not seen by the BIOS so that it renumbers and boots from the secondary disk in the mirror. Normally this renumbering will mean that the system can still automatically boot from the second disk, since you configured it to boot in the previous steps, but it becomes a factor when the first disk becomes available again, as described below.

You should edit the GRUB boot menu in /boot/grub/menu.lst and add an entry for the second disk in the mirror. It is important that you be able to manually boot from the second side of the mirror due to the BIOS renumbering described above. If the primary disk is unavailable, the boot archive on that disk may become stale. Later, if you boot and that disk is available again, the BIOS renumbering would cause GRUB to load that stale boot archive which could cause problems or may even leave the system unbootable.

If the primary disk is once again made available and then you reboot without first resyncing the mirror back onto the primary drive, then you should use the GRUB menu entry for the second disk to manually boot from the correct boot archive (the one on the secondary side of the mirror). Once the system is booted, perform normal metadevice maintenance to resync the primary disk. This will restore the current boot archive to the primary so that subsequent boots from that disk will work correctly.

The previous procedure is not normally necessary since you would replace the failed primary disk using cfgadm(1M) and resync but it will be required if the primary is simply not powered on, causing the BIOS to miss the disk and renumber. Subsequently powering up this disk and rebooting would cause the BIOS to renumber again and by default you would boot from the stale disk.

Note that all of the usual considerations of mddb quorum apply to x86 root mirroring, just as they do for SPARC.

Post a Comment:
Comments are closed for this entry.



Top Tags
« July 2016