Sunday Jan 17, 2010

More Sun Ray on OpenSolaris build 130 workarounds

One more thing for the Sun Ray on build 130. Whether this is the last remains to be seen.

Now that gdm is run via smf gnome-session is now run via ctrun(1) so that it gets it's own contract and therefore any problems it has do not result in gdm being restarted.

However the Sun Ray sessions are not started that way. Hence I was seeing all users logged out if just one session had a problem:

So a rather trivial failure such as this:

Jan 16 11:10:47 pearson genunix: [ID 603404 kern.notice] NOTICE: core_log: metacity[7448] core dumped: /var/cores/core.metacity.7448 

would result in gdm restarted:


[ Jan 16 00:06:08 Method "start" exited with status 0. ]
[ Jan 16 11:10:47 Stopping because process dumped core. ]
[ Jan 16 11:10:47 Executing stop method ("/lib/svc/method/svc-gdm stop"). ]
[ Jan 16 11:10:47 Method "stop" exited with status 0. ]

which in turn means all the users were logged out. Ooops.

The solution was simple but like the previous workarounds leaves your warranty in tatters!

# mv /usr/bin/gnome-session /usr/bin/gnome-session.orig
# cat > /usr/bin/gnome-session << EOF
> #!/bin/ksh -p
> exec ctrun -l child \\$0.orig \\$@
> EOF
# chmod 755 /usr/bin/gnome-session

This results in all your gnome sessions having their own contract as their parent is ctrun:

: pearson FSS 33 $; ptree $(pgrep -o -u gdm gnome-session)
22433 /usr/sbin/gdm-binary
  22440 /usr/lib/gdm-simple-slave --display-id /org/gnome/DisplayManager/Displa
    22965 ctrun -l child /usr/bin/gnome-session.orig --autostart=/usr/share/gdm
      22967 /usr/bin/gnome-session.orig --autostart=/usr/share/gdm/autostart/Lo
        23062 /usr/lib/gdm-simple-greeter
        23063 gnome-power-manager
: pearson FSS 34 $; 

and means that any failures are now ring-fenced to just that session.

Wednesday Jan 06, 2010

ZFS resliver performance improved!

I'm not being lucky with the Western Digital 1Tb disks in my home server.

That is to say I'm extremely pleased I have ZFS as both have now failed and in doing so corrupted data which ZFS has detected (although the users detected the problem first as the performance of the drive became so poor).One of the biggest irritations about replacing drives, apart from having to shut the system down as I don't have hot swap hardware is waiting for the pool to resliver. Previously this has taken in excess of 24 hours to do.

However yesterday's resilver was after I had upgraded to build 130 which has some improvements to the resilver code:


: pearson FSS 1 $; zpool status tank
  pool: tank
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scrub: resilver completed after 6h28m with 0 errors on Wed Jan  6 02:04:17 2010
config:

        NAME           STATE     READ WRITE CKSUM
        tank           ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            c21t0d0s7  ONLINE       0     0     0
            c20t1d0s7  ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            c21t1d0    ONLINE       0     0     0  225G resilvered
            c20t0d0    ONLINE       0     0     0

errors: No known data errors
: pearson FSS 2 $; 
Only 6 ½ hours for 225G which while not close to the theoretical maximum is way better than 24 hours and the system was usable while this was going on.

Friday Oct 09, 2009

Preparing for OpenSolaris @ home

Since the "nevada" builds of Solaris next are due to end soon and for some time the upgrade of my home server has involved more than a little bit of TLC to get it to work I will be moving to an OpenSolaris build just as soon as I can.

However before I can do this I need to make sure I have all thesoftware to provide home service. This is really a note to myself to I don't forget anything.

  • Exim Mail Transfer Agent (MTA). Since I am using certain encryption routines, virus detection and spamassassin I was unable to use the standard MTA, sendmail, when the system was originaly built and have been using exim, from blastwave. I hope to build and use exim without getting all the cruft that comes from the Blastwave packaged. So far this looks like it will be simple as OpenSolaris now has OpenSSL.

  • An imapd. Currently I have a blastwave version but again I intend to build this from scratch again the addition of OpenSSL and libcrypto should make this easy.

  • Clamav. To protect any Windows systems and to generally not pass on viri to others clamav has been scanning all incoming email. Again I will build this from scratch as I already do.

  • Spamassassin. Again I already build this for nevada so building it for OpenSolaris will be easy.

  • Ddclient. Having dynamic DNS allows me to login remotely and read email.

  • Squeezecenter. This is a big issue and in the past has proved hard to get built thanks to all the perl dependacies. It is for that reason I will continue to run it in a zone so that I don't have to trash the main system. Clearly with all my digital music loaded into the squeezecentre software this has to work.

I'm going to see if I can jump through the legal hoops that will allow me to contribute the builds to the contrib repository via Source Juicer. However as this is my spare time I don't know whether the legal reviews will be funded.

Due to the way OpenSolaris is delivered I also need to be more careful about what I install. rather than being able to choose everything. First I need my list from my laptop. Then in addtion to that I'll need

  • Samba - pkg:/SUNWsmba

  • cups - pkg:/SUNWcups

  • OpenSSL - pkg:/SUNWopenssl

Oh and I'll need the Sun Ray server software.

Tuesday May 26, 2009

Why everyone should be using ZFS

It is at times like these that I'm glad I use ZFS at home.


  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        tank           ONLINE       0     0     0
          mirror       ONLINE       0     0     0
            c20t0d0s7  ONLINE       6     0     4
            c21t0d0s7  ONLINE       0     0     0
          mirror       ONLINE       0     0     0
            c21t1d0    ONLINE       0     0     0
            c20t1d0    ONLINE       0     0     0

errors: No known data errors
: pearson FSS 14 $; 

The drive with the errors was also throwing up errors that iostat could report and from it's performance was trying heroicially to give me back data. However it had failed. It's performance was terrible and then it failed to give the right data on 4 occasions. Anyother file system would, if that was user data, just had deliviered it to the user without warning. That bad data could then propergate from there on, probably into my backups. There is certainly no good that could come from that. However ZFS detected and corrected the errors.


Now I have offlined the disk the performance of the system is better but I have no redundancy until the new disk I have just ordered arriaves. Now time to check out Seagate's warranty return system.

Friday Apr 10, 2009

Native solaris printing working

The solution to my HP printer not working on Solaris is to use CUPS. Since I have a full install and solaris now has the packages installed all I had to do was switch the service over:

$ pfexec /usr/sbin/print-service -s cups

then configure the printer using http://localhosts:631 in the same way as I did with ubunto. Now I don't need the virtual machine running which is a bonus. I think cups may be a better fit for me at home since I don't have a nameservice running so the benefits of the lp system are very limited.


The bug is 6826086: printer drivers from package SUNWhpijs are not update with HPLIP

Saturday Feb 21, 2009

[Open]Solaris logfiles and ZFS root

This week I had reason to want to see how often the script that controls the access hours of Sun Ray users actually did work so I went off to look in the messages files only to discover that there were only four and they only went back to January 11.

: pearson FSS 22 $; ls -l mess\*
-rw-r--r--   1 root     root       12396 Feb  8 23:58 messages
-rw-r--r--   1 root     root      134777 Feb  8 02:59 messages.0
-rw-r--r--   1 root     root       53690 Feb  1 02:06 messages.1
-rw-r--r--   1 root     root      163116 Jan 25 02:01 messages.2
-rw-r--r--   1 root     root       83470 Jan 18 00:21 messages.3
: pearson FSS 23 $; head -1 messages.3
Jan 11 05:29:14 pearson pcplusmp: [ID 444295 kern.info] pcplusmp: ide (ata) instance #1 vector 0xf ioapic 0x2 intin 0xf is bound to cpu 1
: pearson FSS 24 $; 

I am certain that the choice of only four log files was not a concious decision I have made but it did make me ponder whether logfile management should be revisted in the light of ZFS root. Since clearly if you have snapshots firing logs could go back a lot futher:

: pearson FSS 40 $; head -1 $(ls -t /.zfs/snapshot/\*/var/adm/message\*| tail -1)
Dec 14 03:15:14 pearson time-slider-cleanup: [ID 702911 daemon.notice] No more daily snapshots left
: pearson FSS 41 $; 

It did not take long for this shell function to burst into life:

function search_log
{
typeset path
if [[ ${2#/} == $2 ]]
then
        path=${PWD}/$2
else
        path=$2
fi
cat $path /.zfs/snapshot/\*$path | egrep $1 | sort -M | uniq
}

Not a generalized solution but one that works when you root filesystem contains all your logs and if you remember to escape any globbing on the command line will search all the log files:

: pearson FSS 46 $; search_log block /var/adm/messages\\\* | wc
      51     688    4759
: pearson FSS 47 $; 

There are two ways to view this. Either it it great that the logs are kept and so I have all this historical data or it is a pain as getting red of log files becomes more of a chore, indeed this is encouraging me to move all the logfiles into their own file systems so that the management of those logfiles is more granular.

At the very least it seems to me that OpenSolaris should sort out where it's log files are going and end the messages going in /var/adm and move them to /var/log which then should be it's own file system.

Tuesday Feb 17, 2009

VirtualBox pause and resume

I have a VirtualBox running ubuntu that is acting as a print server for my HP Officejet J6410 and although that works well it chews a significant amount of CPU resource which given that the printer is not in use most of the time seems a bit of a waste, especially when the Sun Ray server can have 4 active sessions running on it. So now I am running this simple script:

#!/bin/ksh93 -p 

logger -p daemon.notice -t ${0##\*/}[$$] |&

exec >&p 2>&1

while :
do
        echo resuming vbox
        su - vbox -c "VBoxManage -nologo controlvm ubunto resume"

        ssh print-server sudo ntpdate 3.uk.pool.ntp.org

        while ping printer 2 > /dev/null 2>&1
        do
                sleep 60
        done
        echo pausing vbox
        su - vbox -c "VBoxManage -nologo controlvm ubunto pause"
        until ping printer 2 > /dev/null 2>&1
        do
                sleep 1
        done
done

After the usual magic to redirect it's output to syslog this just loops forever pinging the printer. If the printer is alive the VrtualBox is resumed and if the printer is not alive the VirtualBox is paused. So as soon as the printer is turned on the virtual box is ready and then within 60 seconds of the printer being turned off the virtualbox is paused.


So that the clock on the VirtualBox is kept correct (the guest additions are, I am told, supposed to do this for free but in practice they do not for me) after resuming there is the ssh onto the virtual machine to set it's clock\*, so I have a work around in place for VirtualBox which is itself a work around for a defect in Solaris and that work around is also working around another issue with Solaris.


Life on the bleeding edge is always fun!



\*) I would normally use the address of my server to sync with but thanks to the issues with build 108 I currently don't have a working ntp server on the system.

Sunday Feb 15, 2009

Build 108

I've managed to upgrade my home server to build 108 which is an important mile-stone for me as it has the fix for:

6763600: nautilus becomes unusable on a system with 39000 snapshots.

Which was rendering nautilus close to unusable for any users who moved out of their lofs automounted home directories. In partilcular any attempt to use it to manage the photo directories was painful.

However all was not smooth as again I hit this bug:

6795892: Sun Ray X servers (both Xsun and Xorg) suffer network problems in snv_106

but since I was expecting this I tried the workaround from bug 6799655 which is the same as the one for 6763600:


In /etc/sock2path change the following lines:

    2   2   0   tcp
    2   2   6   tcp

    26  2   0   tcp
    26  2   6   tcp

    2   1   0   udp
    2   1   17  udp

    26  1   0   udp
    26  1   17  udp

to:

    2   2   0   /dev/tcp
    2   2   6   /dev/tcp

    26  2   0   /dev/tcp6
    26  2   6   /dev/tcp6

    2   1   0   /dev/udp
    2   1   17  /dev/udp

    26   1  0   /dev/udp6
    26   1  17  /dev/udp6

While this got the Sun Rays up it also stopped named from working, spewing errors like this:

Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] 71/Protocol error
Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] socket.c:4315: unexpected error:

So have had to revert to some internal to Sun binaries that work around this while the underlying bug is fixed. It is slighly worring as I'm left wondering what other victims are out there. One I have already found is ntp, which is a known bug:

6796948: NTP completely broken by I_SETSIG semantics change in snv_106

I suspect that the system will have to revert to build 105 soon.

Sunday Feb 08, 2009

New Home printer VirtualBox & Ubuntu to the rescue

I've got a new printer for home, an HP OfficeJet J6410, but the bad news is on Solaris the hpijs server for ghostscript fails and while I have made some progress debugging this it will clearly take some time to get to the root cause. As part of the debugging I ran up a Ubuntu linux in VirtualBox to see if the problem was a generic ghostscript/CUPS problem.

Since Ubuntu had no problem printing for the short term I have given the Ubuntu system a virtual network interface so that I can configure the Sun Ray server to print via the Ubuntu print-server, which if I use the cups-lpd server, will do the formatting for the remote hosts. The remote hosts therefore just have to send PostScript to the print-server and only it needs to be able run the hpijs server.

I'm running the VirtualBox headless and gdm turned off, infact everything except CUPS, xinetd and sshd turned off. I've not written an SMF service to start the VirtualBox yet in the hope that this is infact a very short lived situation as the Solaris print service will be up and running soon. Anyway I now have working printing on Solaris

Now back to what I know about ghostscript and the hpijs service. If you know more (which would not be hard) then let me know.

The problem is that ghostscript called from foomatic-gswrapper fails with a range check error:

: pearson FSS 51 $; truss -faeo /tmp/tr.$$ /usr/lib/lp/bin/foomatic-gswrapper>
truss -faeo /tmp/tr.$$  /usr/lib/lp/bin/foomatic-gswrapper  -D -dBATCH -dPARANOIDSAFER -dQUIET -dNOPAUSE -sDEVICE=ijs -sIjsServer=hpijs -dIjsUseOutputFD -sOutputFile=-
foomatic-gswrapper: gs '-D' '-dBATCH' '-dPARANOIDSAFER' '-dQUIET' '-dNOPAUSE' '-sDEVICE=ijs' '-sIjsServer=hpijs' '-dIjsUseOutputFD' '-sOutputFile=/dev/fd/3' 3>&1 1>&2
unable to set Quality=0, ColorMode=2, MediaType=0, err=4
following will be used Quality=0, ColorMode=2, MediaType=0
unable to set paper size=17, err=4
Unrecoverable error: rangecheck in setscreen
Operand stack:
    -0.0  0  --nostringval--
: pearson FSS 52 $; 

The errors, unable to set Quality and unable to set paper size appear to home from the hpijp server that returns NO_PRINTER_SELECTED (4) if these routines are called prior to the printer context is set up.

Quite how and why the printer context is not getting set up is the question. I've now rebuilt both ghostscript and the hpijs service from source and worked out how to get foomatic to use the new ones and I still have the issue but at least I should now be able to do diagnosis in my own time while the printer is usable by everyone on the local network.

Building ghostscript was simple:

$ ./configure –prefix=/opt/cjgsw

Then make and make install.

Building hpijs was less simple and I am not entirely sure I have it built correctly in fact I stronly suspect it is not built correctly. At least I have my binaries behaving the same way as the Solaris binaries, ie the failure mode is the same. My configure line was:

 $ ./configure LIBS=”-lsocket -lnsl” --prefix=/opt/cjgsw --disable-pp-build --ena
ble-static –disable-shared

All the –enable-static and –disable-shared are required or the linker gets upset during the build I think the next stop would be an update to gcc....

Wednesday Feb 04, 2009

40" Sun Ray Display

I managed to buy 2 Sun Ray 2's off Ebay and one of them was is now in place in the living room driving our 40” TV.


Combine this with a KeySonic wireless mini keyboard and the DTU does not only act as a photo frame. The Sun Ray unit is attached to the underside of the shelf as the top unit in the pile is a Virgin cable TV recorder which does not like having anything on top blocking the air flow. Thanks to the Sun Ray 2 being so light 5 strips of sticky back velcro do the trick so well that it really is going nowhere to the point that I could not remove it to plug the USB keyboard adapter directly in the back of the unit. The keyboard adapter has a button you have to press once plugged in to pair it with the keyboard. Alas with the Sun Ray in this configuration the button faces upwards. So there is a short USB cable hidden back there.

Networking is provided via Ethenet over mains.

The keyboard has impressive range and a really nice touch pad that pretends to have a scroll wheel down one side. However I've not yet got the keyboard map for it right but it only arrived an hour ago so there is time.

Wednesday Jan 07, 2009

Access hours for Sun Ray users

Having installed a Sun Ray in my daughters bedroom I am now faced with the inevitable problem of her being online all night not getting any sleep and then being generally grumpy. The irony here is that I was sent an email asking how I handle access control to the DTUs and I said I just trusted the children to be sensible (what was I thinking!).

So a solution was required that gave access to the systems only between certain hours. The hours would depend on the user and would have to not loose all their “work” in case this was a late night finishing their homework session.

After asking around no one came back to me and said how it can be done so I wrote my own script. It works by having a file that contains lines with a format

user:starttime:endtime

The times are specified in 24 hour format and only accurate to the minute.

# cat /etc/opt/local/access_hours             
user1:1915:1900
user2:0630:2300
user3:0630:2230
user4:0630:2000
# 

The top line is just really for testing only not allowing access from 1900 to 1915. Then you need a user who has system admin privs which does not have a crontab file. Since I already have a kroot role I'm overloading this. Running the script as with the -c flag and the name of the user will write the crontab file. Note it also writes an entry to keep the crontab file uptodate on an hourly basis.


# /usr/local/sbin/check_access_hours -c kroot
# crontab -l kroot
46 \* \* \* \* /usr/local/sbin/check_access_hours -c kroot
00 19 \* \* \* /usr/local/sbin/check_access_hours user1
00 23 \* \* \* /usr/local/sbin/check_access_hours user2
30 22 \* \* \* /usr/local/sbin/check_access_hours user3
00 20 \* \* \* /usr/local/sbin/check_access_hours user4
#  

Finally I added a line to the utaction script that is already run for every user when they connect to a Sun Ray DTU:

if ! /usr/local/sbin/check_access_hours -t 0 $1
then
        exit 1
fi

The way it disallows access is that it adds the DTU's IP address to the ipfilter, which you have to have configured, so that all traffic from the DTU is blocked. It also submits an at(1) job to run 2 minutes in the future to remove the block so that the Sun Ray can burst back into life. The effect is that the user can no longer use any Sun Ray outside of the defined hours. But after about 2 minutes the DTU is usable again by others or indeed as a photo frame.


A word of warning. Having got all this running the system has paniced twice which is disappointing on one level, that it panics, but pleasing on another, I've found a bug that can now be fixed. The bug is:

6791062: System panic in ip_tcp_input when a rule is added to ipfilter

I look forward to the fix!


The script is here but check that that bug has been fixed before you use it.

Friday Dec 05, 2008

Who forgot to update named.root ?

Me.

So the latest script added to my list of cron jobs running on my home server is:

#!/bin/ksh

TMPDIR=/tmp/named_check_$$.$RANDOM

trap "cd / && test -d $TMPDIR && rm -r $TMPDIR" EXIT
mkdir $TMPDIR || exit
cd $TMPDIR || exit

wget -q ftp://ftp.internic.net/domain/named.root
if ! cmp -s named.root /var/named/db.cache
then
        mv named.root ~
        mailx -s "new named.root file " $LOGNAME << EOF

There is a new named.root file in ~/named.root it needs to be verified and then
installed as /var/named/db.cache.


EOF
fi

All after the internet broke in the house. My initial suspicion that one of the kids had typed google into google was wide of the mark. My old named.root file was old, really, really old. So now I'm checking once a month that my named.root file is upto date

Thursday Dec 04, 2008

Doing more with less

As I have mentioned before I have an ancient Sun Ray 1 that drives the TV in our kitchen to look like a photo frame. The network is provided my an ethernet over mains bridge that is rated at 85Mbit/sec and the network drop from the server is a 1Gbit/sec. Since the switch I have is very cheap this results in a significant packet drop to that DTU with the result that the picture transition is less than ideal and can stutter somewhat.

So last night after reading an email on an internal list I finally got around to reading the documentation so I could set a bandwidth limit on this one DTU to see if things could be improved, With no bandwidth limit the very excellent utbw gives:

 lost      0/00% pkts     62 cpu   0% kbytes     53 0.021 Mbps 4.3(4.2) ms
 lost   1243/46% pkts   2652 cpu  11% kbytes   1614 0.631 Mbps 4.9(4.6) ms
 lost      0/00% pkts     60 cpu   3% kbytes     51 0.020 Mbps 4.1(4.3) ms
 lost      0/00% pkts     64 cpu   2% kbytes     55 0.022 Mbps 4.8(4.6) ms
 lost      0/00% pkts     64 cpu   2% kbytes     56 0.022 Mbps 4.2(4.7) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.3(4.5) ms
 lost      0/00% pkts     62 cpu   3% kbytes     53 0.021 Mbps 4.9(4.7) ms
 lost    266/11% pkts   2393 cpu   6% kbytes   2314 0.904 Mbps 4.4(4.6) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.4(4.5) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 4.7(4.6) ms
 lost      0/00% pkts     64 cpu   3% kbytes     56 0.022 Mbps 4.1(4.3) ms
 lost      0/00% pkts     58 cpu   0% kbytes     48 0.019 Mbps 4.4(4.4) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 6.0(5.2) ms
 lost    229/09% pkts   2377 cpu   8% kbytes   2320 0.907 Mbps 4.7(4.9) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.1(4.5) ms
 lost      0/00% pkts     62 cpu   0% kbytes     53 0.021 Mbps 4.1(4.3) ms
 lost      0/00% pkts     64 cpu   2% kbytes     56 0.022 Mbps 4.2(4.2) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.2(4.2) ms
 lost      0/00% pkts     62 cpu   0% kbytes     53 0.021 Mbps 4.4(4.3) ms
 lost    597/23% pkts   2532 cpu   9% kbytes   2123 0.830 Mbps 4.2(4.3) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.1(4.2) ms
 lost      0/00% pkts     62 cpu   3% kbytes     53 0.021 Mbps 4.1(4.1) ms
 lost      0/00% pkts     64 cpu   0% kbytes     56 0.022 Mbps 5.1(4.6) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.1(4.4) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 4.3(4.4) ms
 lost      0/00% pkts   2133 cpu   8% kbytes   2322 0.907 Mbps 4.1(4.2) ms
 lost      0/00% pkts     60 cpu   0% kbytes     51 0.020 Mbps 7.3(5.7) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 4.2(5.0) ms
 lost      0/00% pkts     64 cpu   2% kbytes     56 0.022 Mbps 4.2(4.6) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 6.5(5.6) ms
 lost      0/00% pkts     62 cpu   0% kbytes     53 0.021 Mbps 4.1(4.9) ms
 lost    462/18% pkts   2509 cpu   9% kbytes   2251 0.879 Mbps 4.2(4.5) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.2(4.3) ms

What is more the transition that is observed often jumps or will have one block of the picture updated after all the others. Having tuned the bandwidth down to 20Mb/sec:

 lost    114/04% pkts   2295 cpu   4% kbytes   2344 0.916 Mbps 4.4(4.7) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.2(4.5) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 7.2(5.8) ms
 lost      0/00% pkts     63 cpu   2% kbytes     53 0.021 Mbps 4.2(5.0) ms
 lost      0/00% pkts     60 cpu   0% kbytes     51 0.020 Mbps 4.4(4.7) ms
 lost      0/00% pkts     62 cpu   3% kbytes     53 0.021 Mbps 4.1(4.4) ms
 lost    216/09% pkts   2304 cpu   7% kbytes   2295 0.897 Mbps 4.4(4.4) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.2(4.3) ms
 lost      0/00% pkts     63 cpu   0% kbytes     55 0.022 Mbps 4.4(4.4) ms
 lost      0/00% pkts     63 cpu   2% kbytes     53 0.021 Mbps 4.4(4.4) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.1(4.2) ms
 lost      0/00% pkts     63 cpu   2% kbytes     55 0.022 Mbps 5.0(4.6) ms
 lost    168/07% pkts   2174 cpu   6% kbytes   2230 0.871 Mbps 7.0(5.8) ms
 lost      0/00% pkts     60 cpu   2% kbytes     51 0.020 Mbps 4.9(5.3) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 5.9(5.6) ms
 lost      0/00% pkts     63 cpu   2% kbytes     56 0.022 Mbps 4.1(4.9) ms
 lost      0/00% pkts     60 cpu   0% kbytes     51 0.020 Mbps 4.5(4.7) ms
 lost      0/00% pkts     62 cpu   2% kbytes     53 0.021 Mbps 4.2(4.5) ms
 lost      0/00% pkts   1938 cpu   8% kbytes   2118 0.827 Mbps 4.3(4.4) ms
 lost      0/00% pkts     60 cpu   0% kbytes     50 0.020 Mbps 8.4(6.4) ms
 lost      0/00% pkts     62 cpu   2% kbytes     52 0.021 Mbps 17.7(12.1) ms
 lost      0/00% pkts     64 cpu   2% kbytes     55 0.022 Mbps 4.2(8.1) ms
 lost      0/00% pkts     60 cpu   2% kbytes     50 0.020 Mbps 6.3(7.2) ms
 lost      0/00% pkts     62 cpu   0% kbytes     52 0.021 Mbps 4.4(5.8) ms
 lost    214/09% pkts   2224 cpu   7% kbytes   2170 0.848 Mbps 4.0(4.9) ms
 lost      0/00% pkts     60 cpu   2% kbytes     50 0.020 Mbps 4.1(4.5) ms
 lost      0/00% pkts     62 cpu   2% kbytes     52 0.021 Mbps 4.6(4.6) ms
 lost      0/00% pkts     64 cpu   0% kbytes     55 0.022 Mbps 5.7(5.1) ms
 lost      0/00% pkts     60 cpu   2% kbytes     50 0.020 Mbps 4.6(4.8) ms
 lost      0/00% pkts     64 cpu   2% kbytes     55 0.022 Mbps 5.8(5.3) ms
 lost    194/08% pkts   2278 cpu   9% kbytes   2274 0.888 Mbps 4.2(4.8) ms
 lost      0/00% pkts     60 cpu   0% kbytes     50 0.020 Mbps 4.2(4.5) ms
 lost      0/00% pkts     63 cpu   2% kbytes     53 0.021 Mbps 4.2(4.3) ms
 lost      0/00% pkts     63 cpu   3% kbytes     53 0.021 Mbps 4.2(4.3) ms
 lost      0/00% pkts     60 cpu   1% kbytes     50 0.020 Mbps 4.9(4.6) ms
 lost      0/00% pkts     62 cpu   1% kbytes     52 0.021 Mbps 4.1(4.3) ms
 lost      0/00% pkts   2149 cpu   6% kbytes   2319 0.906 Mbps 4.3(4.3) ms
 lost      0/00% pkts     60 cpu   2% kbytes     50 0.020 Mbps 4.2(4.3) ms

The numbers improve greatly however they hide the truth somewhat. Since the operation of the picture frame is that except for the clock that updates every second there are no updates at all for most of the time then every 2 minutes a new photo is displayed. So those averages which appear to be every 20 seconds hide the very high burst of data that happens.


You can see the difference here:

100Mb/sec:

10Mb/sec

Thursday Nov 27, 2008

Adding dependancies to exim

I finally got around to adding dependancies to the smtp (mail) server I am using on my home server so that it depends on both spamassassin and the clam anti virus services. While there is probably a way to do this using individual commands it was much quicker to export the XML edit that and reimport it having added these lines:

    <dependency name='spamd' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/network/spamd'/>
    </dependency>
    <dependency name='clam' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/network/clam'/>
    </dependency>

Having refreshed the service and restarted I, it now shows as depending on the other two services:

: pearson FSS 3 $; svcs -d cswexim
STATE          STIME    FMRI
online         Nov_24   svc:/network/loopback:default
online         Nov_24   svc:/milestone/name-services:default
online         Nov_24   svc:/system/filesystem/local:default
online         Nov_24   svc:/network/clam:default
online         Nov_26   svc:/network/spamd:default
: pearson FSS 4 $; 

and any failure of the dependant services results in cswexim being restarted after the dependant service restarts. Depressingly I had found that small amounts of spam could sneak through thanks to exim not depending on spamassasin.

Sunday Nov 23, 2008

Two pools on one drive?

Now I'm committed to ZFS root I'm left with a dilemma. Given the four drives I have in the system and that I have too much data and the drives are of different sizes so raid2Z is not an option even though it would give the greatest protection for the data the next best solution is some form of mirroring. Initially I simply had two pools which offers good redundancy and allows ZFS root to work but is suboptimal performance. If I could stripe the pool that would be better but then that does not work with ZFS root.

However since I used to run with a future proof Disk Suite, UFS based root I still have the space that used to contain the two boot environments that were on UFS into which I intended to grow the pool once they were not needed. What if I did not grow the pool but instead put a second pool on that partition? Then I would have a pool, “rpool” mirrored across part of the disk and then the data pool, “tank” mirrored over the rest of the boot drives and striped across a second mirror consisting of the entire second pair of drives.

Clearly the solution is suboptimal but given the constraints of ZFS root and the hardware I have would this perform better?

I should point out that the system as is does not perform badly, but I don't want to leave performance on the table if I don't have to. I'm not going to rush into this (that is I've not already done it) since growing the pool is a one way operation there being no way to shrink it again although at the moment I am minded to do it.

Comments welcome

Saturday Nov 22, 2008

Forced to upgrade

Build 103 and ZFS root have come to the home server. While I was travelling the system hit bug 6746456 which resulted in the system panicing every time it booted. So I was forced to return to build 100 and have now upgraded to build 103. Live upgrade using UFS would not work at all and since I have the space I've moved over to ZFS root. However the nautilus bug is still in build 103 so I'm either going to have to live with it, which is impossible, disable nautilus completely or work to get the time slider feature disabled until it is usable. Disabling nautilus while irritating is effectively what I have had to do now so could be the medium term solution.

The other news for the home server was the failure of the power supply. So it was good bye to the small Antec case that used to house the server since it did not really save any space a more traditional desk side unit has replaced it which also allows upto six internal drives. Since ZFS root will not support booting of stripes the extra two drives I have form a second pool.

# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
pool2   294G  36.8G   257G    12%  ONLINE  -
tank    556G   307G   249G    55%  ONLINE  -
# 

The immediate effect of two pools is being able to have the Solaris image from which I upgraded on a different pair of disks from the ones being upgraded with a dramatic performance boost. The other is that I can let the automatic snapshot service take control of the other pool rather than add it to my old snapshot policy. Early on I realise I need to turn off snapshots on the swap volumes which are on both pools (to get some striping):

# zfs set com.sun:auto-snapshot=false pool2/swap
# 
zfs set com.sun:auto-snapshot=false tank/swap 
# 

should do it.

Friday Oct 24, 2008

Brief visit to build 100 at home.

I finally managed to upgrade the home server to build 100 by deleting all the zones and then upgrading. However a few minutes of running resulted in this bug:

6763600 nautilus becomes unusable on a system with 39000 snapshots

which is a pity as I was really excited about time slider. The positive part of this is that this is yet another feature that will be improved by users running OpenSolaris variants at home.

Alas while I could live with out nautilus (the only file chooser I recall on a vt220 was vsh and I never really got on with that) the other users at home can not so the system is back running build 96.

Saturday Sep 13, 2008

Native CIFS and samba on the same system

I've been using samba at home for a while and now but would like to migrate over to the new CIFS implentation provided by solaris. Since there are somre subtle differences in what each service provides\* this means a slower migration.

Obviously you can't configure both services to run on the same system so to get around this I am going to migrate all the SMB services into a zone running on the server and then allow the global zone to act as the native CIFS service.

So I configured a zone called, rather dully, “samba” with loop back access to all the file systems that I share via SMB and added the additional priviledge “sys_smb” so that the daemons could bind to the smb service port.

zonecfg:samba> set limitpriv=default,sys_smb

The end command only makes sense in the resource scope.
zonecfg:samba> commit
zonecfg:samba> exit

Now you can configure the zone in the usual way to run samba. I simply copied the smb.conf and smbpasswd files from the global zone using zcp.


Once that was done and samba enabled in smf I could then enable the natives CIFS server in the global zone and have the best of both worlds.



\*) The principal difference I see is that the native smb service does not cross file systems mount points. So if you have a hierarchy of file systems you have to mount each one on the client. With samba you can just mount the root and it will see everything below.

Sunday Sep 07, 2008

Upgrading disks

Having run out of space in the root file systems and being close to full on the zpool the final straw was being able to get 2 750Gb sata drives for less than £100, that and knowing that sanpshots no longer cause re livering to restart which greatly simplifies the data migration. So I'm replacing the existing drives with new ones. Since the enclosure I have can only hold three drives this involved a two stage upgrade so that at no point was my data on less than two drives. First stage was to install one drive and label it:

partition> print
Current partition table (unnamed):
Total disk cylinders available: 45597 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm   39383 - 41992       39.99GB    (2610/0/0)    83859300
  1 unassigned    wm       0                0         (0/0/0)              0
  2     backup    wu       0 - 45596      698.58GB    (45597/0/0) 1465031610
  3 unassigned    wm       0                0         (0/0/0)              0
  4 unassigned    wm   36773 - 39382       39.99GB    (2610/0/0)    83859300
  5 unassigned    wm   45594 - 45596       47.07MB    (3/0/0)          96390
  6 unassigned    wm   36379 - 36772        6.04GB    (394/0/0)     12659220
  7 unassigned    wm       3 - 36378      557.31GB    (36376/0/0) 1168760880
  8       boot    wu       0 -     0       15.69MB    (1/0/0)          32130
  9 alternates    wm       1 -     2       31.38MB    (2/0/0)          64260

partition>

These map to the partitions from the original set up, only they are bigger. I'm confident that when the 40Gb root disks are to small I will have migrated to ZFS for root. So this looks like a good long term solution.

pearson # dumpadm -d /dev/dsk/c2d0s6
      Dump content: kernel pages
       Dump device: /dev/dsk/c2d0s6 (dedicated)
Savecore directory: /var/crash/pearson
  Savecore enabled: yes
pearson # metadb -a -c 3 /dev/dsk/c2d0s5
pearson # egrep c2d0 /etc/lvm/md.tab
d12 1 1 /dev/dsk/c2d0s0
d42 1 1 /dev/dsk/c2d0s4
pearson # metainit d12
d12: Concat/Stripe is setup
pearson # metainit d42
d42: Concat/Stripe is setup
pearson # metattach d0 d12
d0: submirror d12 is attached
pearson # 

Now wait until the disk has completed resyning. While you can do this in parallel this causes the disk heads to move more so overall it is slower. Left to just do one partition at a time it is really quite quick:

                 extended device statistics                 
device    r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b 
cmdk0   357.2    0.0 18321.8    0.0  2.6  1.1   10.4  52  58 
cmdk1     0.0  706.4    0.0 36147.4  1.0  0.5    2.2  23  27 
cmdk2   350.2    0.0 17929.6    0.0  0.4  0.3    2.1  12  15 
md1      70.0   71.0 35859.2 36371.5  0.0  1.0    7.1   0 100 
md3       0.0   71.0    0.0 36371.5  0.0  0.3    3.8   0  27 
md15     35.0    0.0 17929.6    0.0  0.0  0.6   16.5   0  58 
md18     35.0    0.0 17929.6    0.0  0.0  0.1    4.3   0  15 
pearson # metastat d0 
d0: Mirror
    Submirror 0: d10
      State: Okay         
    Submirror 1: d11
      State: Okay         
    Submirror 2: d12
      State: Resyncing    
    Resync in progress: 70 % done
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 20482875 blocks (9.8 GB)

d10: Submirror of d0
    State: Okay         
    Size: 20482875 blocks (9.8 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c1d0s0          0     No            Okay   Yes 


d11: Submirror of d0
    State: Okay         
    Size: 20482875 blocks (9.8 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c5d0s0          0     No            Okay   Yes 


d12: Submirror of d0
    State: Resyncing    
    Size: 83859300 blocks (39 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c2d0s0          0     No            Okay   Yes 


Device Relocation Information:
Device   Reloc  Device ID
c1d0   Yes      id1,cmdk@AST3320620AS=____________3QF09GL1
c5d0   Yes      id1,cmdk@AST3320620AS=____________3QF0A1QD
c2d0   Yes      id1,cmdk@AST3750840AS=____________5QD36N5M
pearson # 

Once complete do the other root disk:

pearson # metattach d4 d42 
d4: submirror d42 is attached
pearson # 

Finally attach slice 7 to the zpool:

pearson # zpool attach -f tank c1d0s7 c2d0s7
pearson # zpool status
  pool: tank
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 0.00% done, 252h52m to go
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            c1d0s7  ONLINE       0     0     0
            c5d0s7  ONLINE       0     0     0
            c2d0s7  ONLINE       0     0     0

errors: No known data errors
pearson # 

The initial estimate is more pessimistic than reality but it still took over 11hours to complete. The next thing was to shut the system down and replace one of the old drives with the new. Once this was done the final slices in use from the old drive can be detached and in the case of the meta devices cleared.

: pearson FSS 4 $; zpool status
  pool: tank
 state: ONLINE
 scrub: scrub completed after 11h8m with 0 errors on Sat Sep  6 20:58:05 2008
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            c5d0s7  ONLINE       0     0     0
            c2d0s7  ONLINE       0     0     0

errors: No known data errors
: pearson FSS 5 $; 
: pearson FSS 5 $; metastat
d6: Mirror
    Submirror 0: d62
      State: Okay         
    Submirror 1: d63
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 12659220 blocks (6.0 GB)

d62: Submirror of d6
    State: Okay         
    Size: 12659220 blocks (6.0 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c5d0s6          0     No            Okay   Yes 


d63: Submirror of d6
    State: Okay         
    Size: 12659220 blocks (6.0 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c2d0s6          0     No            Okay   Yes 


d4: Mirror
    Submirror 0: d42
      State: Okay         
    Submirror 1: d43
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 83859300 blocks (39 GB)

d42: Submirror of d4
    State: Okay         
    Size: 83859300 blocks (39 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c5d0s4          0     No            Okay   Yes 


d43: Submirror of d4
    State: Okay         
    Size: 83859300 blocks (39 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c2d0s4          0     No            Okay   Yes 


d0: Mirror
    Submirror 0: d12
      State: Okay         
    Submirror 1: d13
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 83859300 blocks (39 GB)

d12: Submirror of d0
    State: Okay         
    Size: 83859300 blocks (39 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c5d0s0          0     No            Okay   Yes 


d13: Submirror of d0
    State: Okay         
    Size: 83859300 blocks (39 GB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c2d0s0          0     No            Okay   Yes 


Device Relocation Information:
Device   Reloc  Device ID
c5d0   Yes      id1,cmdk@AST3750840AS=____________5QD36N5M
c2d0   Yes      id1,cmdk@AST3750840AS=____________5QD3EQEX
: pearson FSS 6 $; 

The old drive is still in the system but currently only has a metadb on it:

: pearson FSS 6 $; metadb -i
        flags           first blk       block count
     a m  p  luo        16              8192            /dev/dsk/c1d0s5
     a    p  luo        8208            8192            /dev/dsk/c1d0s5
     a    p  luo        16400           8192            /dev/dsk/c1d0s5
     a    p  luo        16              8192            /dev/dsk/c5d0s5
     a    p  luo        8208            8192            /dev/dsk/c5d0s5
     a    p  luo        16400           8192            /dev/dsk/c5d0s5
     a       luo        16              8192            /dev/dsk/c2d0s5
     a       luo        8208            8192            /dev/dsk/c2d0s5
     a       luo        16400           8192            /dev/dsk/c2d0s5
 r - replica does not have device relocation information
 o - replica active prior to last mddb configuration change
 u - replica is up to date
 l - locator for this replica was read successfully
 c - replica's location was in /etc/lvm/mddb.cf
 p - replica's location was patched in kernel
 m - replica is master, this is replica selected as input
 t - tagged data is associated with the replica
 W - replica has device write errors
 a - replica is active, commits are occurring to this replica
 M - replica had problem with master blocks
 D - replica had problem with data blocks
 F - replica had format problems
 S - replica is too small to hold current data base
 R - replica had device read errors
 B - tagged data associated with the replica is not valid
: pearson FSS 7 $; 

I'm tempted to leave the third disk in the system so that the disk suite configuration will always have a quorum if a single drive files. However since the BIOS only seems to be able to boot from the first disk drive this may be pointless.

I'm now keenly interested in bug 6592835 “resliver needs to go fasterâ€ since if a disk did fail I don't fancy waiting more than 24hours after I have sourced a new drive for the data to sync when the disks fill. The disk suite devices managed to drive the disk at over 40Mb/sec while ZFS achieved 5Mb/sec.

Thursday Aug 28, 2008

It scrubbed up good

Since the home server has been snapping regularly I have had to choose between snapshots and scrubbing and I chose snapshots. User error is more likely than hardware failures and scrubbing is really about seeing those errors sooner so you don't get a unrecoverable failure due to having two problems at once. However I would rather not have to choose.

So I was particularly pleased to see that build 94 contains the fix for this bug:

6343667 scrub/resilver has to start over when a snapshot is taken

So today the home server had it´s first scrub in years and it scrubbed up well:

: pearson FSS 5 $; pfexec zpool status
  pool: tank
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scrub: scrub completed after 12h42m with 0 errors on Thu Aug 28 20:12:36 2008
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            c1d0s7  ONLINE       0     0     0
            c5d0s7  ONLINE       0     0     0

errors: No known data errors
: pearson FSS 6 $; 

When I upgrade the pool, after the other live upgrade boot environment can support this pool version, there is the promise of a faster scrub but since this scrub happened during the day and I also backed up the pool using zfs_backup during the same time.

About

This is the old blog of Chris Gerhard. It has mostly moved to http://chrisgerhard.wordpress.com

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today