Live Upgrade, /var/tmp and the Ever Growing Boot Environments
By user12611829 on Jan 13, 2012
Let's start with a clean installation of Solaris 10 10/09 (u8).
# df -k / Filesystem kbytes used avail capacity Mounted on rpool/ROOT/s10x_u8wos_08a 20514816 4277560 13089687 25% /So far, so good. Solaris is just a bit over 4GB. Another 3GB is used by the swap and dump devices. That should leave plenty of room for half a dozen or so patch cycles (assuming 1GB each) and an upgrade to the next release.
Now, let's put on the latest recommended patch cluster. Note that I am following the suggestions in my Live Upgrade Survival Guide, installing the prerequisite patches and the LU patch before actually installing the patch cluster.
# cd /var/tmp # wget patchserver:/export/patches/10_x86_Recommended-2012-01-05.zip . # unzip -qq 10_x86_Recommended-2012-01-05.zip # wget patchserver:/export/patches/121431-69.zip # unzip 121431-69 # cd 10x_Recommended # ./installcluster --apply-prereq --passcode (you can find this in README) # patchadd -M /var/tmp 121431-69 # lucreate -n s10u8-2012-01-05 # ./installcluster -d -B s10u8-2012-01-05 --passcode # luactivate s10u8-2012-01-05 # init 0After the new boot environment is activated, let's upgrade to the latest release of Solaris 10. In this case, it will be Solaris 10 8/11 (u10).
Yes, this does seem like an awful lot is happening in a short period of time. I'm trying to demonstrate a situation that really does happen when you forget something as simple as a patch cluster clogging up /var/tmp. Think of this as one of those time lapse video sequences you might see in a nature documentary.
# pkgrm SUNWluu SUNWlur SUNWlucfg # pkgadd -d /cdrom/sol_10_811_x86 SUNWluu SUNWlur SUNWlucfg # patchadd -M /var/tmp 121431-69 # lucreate -n s10u10-baseline' # echo "autoreg=disable" > /var/tmp/no-autoreg # luupgrade -u -s /cdrom/sol_10_811_x86 -k /var/tmp/no-autoreg -n s10u10-baseline # luactivate s10u10-baseline # init 0As before, everything went exactly as expected. Or I thought so, until I logged in the first time and checked the free space in the root pool.
# df -k / Filesystem kbytes used avail capacity Mounted on rpool/ROOT/s10u10-baseline 20514816 10795038 2432308 82% /Where did all of the space go ? Back of the napkin calculations of 4.5GB (s10u8) + 4.5GB (s10u10) + 1GB (patch set) + 3GB (swap and dump) = 13GB. 20GB pool - 13GB used = 7GB free. But there's only 2.4GB free ?
This is about the time that I smack myself on the forehead and realize that I put the patch cluster in the /var/tmp. Old habits die hard. This is not a problem, I can just delete it, right ?
Not so fast.
# du -sh /var/tmp 5.4G /var/tmp # du -sh /var/tmp/10* 3.8G /var/tmp/10_x86_Recommended 1.5G /var/tmp/10_x86_Recommended-2012-01-05.zip # rm -rf /var/tmp/10* # du -sh /var/tmp 3.4M /var/tmpImagine the look on my face when I check the pool free space, expecting to see 7GB.
# df -k / Filesystem kbytes used avail capacity Mounted on rpool/ROOT/s10u10-baseline 20514816 5074262 2424603 68% /We are getting closer. At least my root filesystem size is reasonable (5GB vs 11GB). But the free space hasn't changed at all.
Once again, I smack myself on the forehead. The patch cluster is also in the other two boot environments. All I have to do is get rid them too, and I'll get my free space back.
# lumount s10u8-2012-01-05 /mnt # rm -rf /mnt/var/tmp/10_x86_Recommended* # luumount s10u8-2012-01-05 # lumount s10x_u8wos_08a /mnt # rm -rf /mnt/var/tmp/10_x86_Recommended* # luumount s10x_u8wos_08aSurely, that will get my free space reclaimed, right ?
# df -k / Filesystem kbytes used avail capacity Mounted on rpool/ROOT/s10u10-baseline 20514816 5074265 2429261 68% /This is when I smack myself on the forehead for the third time in one afternoon. Just getting rid of them in the boot environments is not sufficient. It would be if I were using UFS as a root filesystem, but lucreate will use the ZFS snapshot and cloning features when used on a ZFS root. So the patch cluster is in the snapshot, and the oldest one at that.
Let's try this all over again, but this time I will put the patches somewhere else that is not part of a boot environment. If you are thinking of using root's home directory, think again - it is part of the boot environment. If you are running out of ideas, let me suggest that /export/patches might be a good place to put them.
Doing the exercise again, with the patches in /export/patches, I get similar results (to be expected), but with one significant different.This time the patches are in a shared ZFS dataset (/export) and can be deleted.
# lustatus Boot Environment Is Active Active Can Copy Name Complete Now On Reboot Delete Status -------------------------- -------- ------ --------- ------ ---------- s10x_u8wos_08a yes no no yes - s10u8-2012-01-05 yes no no yes - s10u10-baseline yes yes yes no - # df -k / Filesystem kbytes used avail capacity Mounted on rpool/ROOT/s10u10-baseline 20514816 5184578 2445140 68% / # df -k /export Filesystem kbytes used avail capacity Mounted on rpool/export 20514816 5606384 2445142 70% /exportThis time, when I delete them, the disk space will be reclaimed.
# rm -rf /export/patches/10_x86_Recommended* # df -k / Filesystem kbytes used avail capacity Mounted on rpool/ROOT/s10u10-baseline 20514816 5184578 8048050 40% /Now, that's more like it. With this free space, I can continue to patch and maintain my system as I had originally planned - estimating a few hundred MB to 1.5GB per patch set.
The moral to the story is that even if you follow all of the best practices and recommendations, you can still be tripped up by old habits when you don't consider their consequences. And when you do, don't feel bad. Many best practices come from exercises just like this one.