Solaris Volume Manager root mirror problems on S10

There are a couple of bugs that we found in S10 that make it look like Solaris Volume Manager root mirroring does not work at all. Unfortunately we found these bugs after the release went out. These bugs will be patched but I wanted to describe the problems a bit and describe some workarounds

On a lot of systems that use SVM to do root mirroring there are only two disks. When you set up the configuration you put one or more metadbs on each disk to hold the SVM configuration information.

SVM implements a metadb quorum rule which means that during the system boot, if more than 50% of the metadbs are not available, the system should boot into single-user mode so that you can fix things up. You can read more about this here.

On a two disk system there is no way to set things up so that more than 50% of the metadbs will be available if one of the disks dies.

When SVM does not have metadb quorum during the boot it is supposed to leave all of the metadevices read-only and boot into single-user. This gives you a chance to confirm that you are using the right SVM configuration and that you don't corrupt any of your data before having a chance to cleanup the dead metadbs.

What a lot of people do when when they set up a root mirror is pull one of the disks to check if the system will still boot and run ok. If you do this experiment on a two disk configuration running S10 the system will panic really early in the boot process and it will go into a infinite panic/reboot cycle.

What is happening here is that we found a bug related to UFS logging, which is on by default in S10. Since the root mirror stays read-only because there is no metadb quorum we hit a bug in the UFS log rolling code. This in turn leaves UFS in a bad state which causes the system to panic.

We're testing the fix for this bug right now but in the meantime, it is easy to workaround this bug by just disabling logging on the root filesystem. You can do that be specifying the "nologging" option in the last field of the vfstab entry for root. You should reboot once before doing any SVM experiments (like pulling a disk) to ensure that UFS has rolled the log and is no longer using logging on root.

Once a patch for this bug is out you will definitely want to remove this workaround from the the vfstab entry since UFS logging offers so many performance and availability benefits.

By they way, UFS logging is also on by default in the S9 9/04 release but that code does not suffer from this bug.

The second problem we found is not as serious as the UFS bug. This has to do with an interaction with the Service Management Facility (SMF) which is new in S10 and again, this is related to not have metadb quorum during the boot. What should happen is that the system should enter single-user so you can clean up the dead metadbs. Instead it boots all the way to multi-user but since the root device is still read-only things don't work very well. This turned out to be a missing dependency which we didn't catch when we integrated SVM and SMF. We'll have a patch for this too but this problem is much less serious. You can still login as root and clean up the dead metadbs so that you can then reboot with a good metadb quorum.

Both of the problems result because there is no metadb quorum so the root metadevice remains read-only after a boot with a dead disk. If you have a third disk which you can use to add a metadb onto, then you can reduce the likelihood of hitting this problem since losing one disk won't cause you to lose quorum during boot.

Given these kinds of problems you might wonder why does SVM bother to implement the metadb quorum? Why not just trust the metadbs that are alive? SVM is conservative and always chooses the path to ensure that you won't lose data or use stale data. There are various corner cases to worry about when SVM cannot be sure it is using the most current data. For example, in a two disk mirror configuration, you might run for a while on the first disk with the second disk powered down. Later you might reboot off the second disk (because the disk was now powered up) and the first disk might now be powered down. At this point you would be using the stale data on the mirror, possibly without even realizing it. The metadb quorum rule gives you a chance to intervene and fix up the configuration when SVM cannot do it automatically.

Post a Comment:
  • HTML Syntax: NOT allowed



Top Tags
« April 2014