Tuesday Mar 10, 2009

Update to scsi.d

I've just released a new version of scsi.d that fixes a strange bug that only effected 10 but would result in scsi.d not starting:

# dtrace -qCs scsi.d-1.16  
dtrace: failed to compile script scsi.d-1.16: "/usr/include/sys/scsi/scsi_pkt.h", line 33: incomplete struct/union/enum struct scsi_address: pkt_address
# 

The solution was simple even if I still don't really understand why the header included fine in OpenSolaris but not in 10.

Sunday Mar 08, 2009

AV codecs for Solaris

One of the big complaints about using OpenSolaris and Solaris is multi media support. The lack of codecs to allow you to play "common" if not "open" formats is irritatings.

I've been using the Fluendo mp3 codecs for a while as they have a compelling price point, being free. Fluendo also have a bundle of other codecs that they supply for OpenSolaris containing codecs for a number of other popular formats. I decided the €28 was would be worth it and have now installed the codecs on my home server. I now have gstreamer support for:

: pearson FSS 1 $; gst-inspect | grep -i flue
fluasf:  flurtspwms: Fluendo WMS RTSP Extension
fluasf:  fluasfcmdparse: Fluendo ASF Command Parser
fluasf:  fluasfdemux: Fluendo ASF Demuxer
fluwmvdec:  fluwmvdec: Fluendo WMV Decoder
fluaacdec:  fluaacdec: Fluendo AAC Decoder
flumms:  flummssrc: Fluendo MMS source
fludivx3dec:  fludivx3dec: Fluendo DivX 3.11 Decoder
flulpcm:  flulpcmdec: Fluendo LPCM decoder
fluwma:  fluwmsdec: Fluendo WMS Decoder
fluwma:  fluwmadec: Fluendo WMA Decoder (STD + PRO + LSL + WMS)
fluh264dec:  fluh264dec: Fluendo H264 Decoder
flumpeg4vdec:  flumpeg4vdec: Fluendo MPEG-4 ASP Video Decoder
flumpeg2vdec:  flumpeg2vdec: Fluendo MPEG-2 Video Decoder
flump3dec:  flump3dec: Fluendo MP3 Decoder (C build)
: pearson FSS 2 $; 

Alas on the Sun Rays the video performance is still less than great but on the console it is faultless and I am seeing the Totem is not quite as useless as I had previously thought.

It would be really cool if you could get OpenSolaris codecs via IPS repository.

The next thing I want is to be able to edit video.

Saturday Feb 21, 2009

[Open]Solaris logfiles and ZFS root

This week I had reason to want to see how often the script that controls the access hours of Sun Ray users actually did work so I went off to look in the messages files only to discover that there were only four and they only went back to January 11.

: pearson FSS 22 $; ls -l mess\*
-rw-r--r--   1 root     root       12396 Feb  8 23:58 messages
-rw-r--r--   1 root     root      134777 Feb  8 02:59 messages.0
-rw-r--r--   1 root     root       53690 Feb  1 02:06 messages.1
-rw-r--r--   1 root     root      163116 Jan 25 02:01 messages.2
-rw-r--r--   1 root     root       83470 Jan 18 00:21 messages.3
: pearson FSS 23 $; head -1 messages.3
Jan 11 05:29:14 pearson pcplusmp: [ID 444295 kern.info] pcplusmp: ide (ata) instance #1 vector 0xf ioapic 0x2 intin 0xf is bound to cpu 1
: pearson FSS 24 $; 

I am certain that the choice of only four log files was not a concious decision I have made but it did make me ponder whether logfile management should be revisted in the light of ZFS root. Since clearly if you have snapshots firing logs could go back a lot futher:

: pearson FSS 40 $; head -1 $(ls -t /.zfs/snapshot/\*/var/adm/message\*| tail -1)
Dec 14 03:15:14 pearson time-slider-cleanup: [ID 702911 daemon.notice] No more daily snapshots left
: pearson FSS 41 $; 

It did not take long for this shell function to burst into life:

function search_log
{
typeset path
if [[ ${2#/} == $2 ]]
then
        path=${PWD}/$2
else
        path=$2
fi
cat $path /.zfs/snapshot/\*$path | egrep $1 | sort -M | uniq
}

Not a generalized solution but one that works when you root filesystem contains all your logs and if you remember to escape any globbing on the command line will search all the log files:

: pearson FSS 46 $; search_log block /var/adm/messages\\\* | wc
      51     688    4759
: pearson FSS 47 $; 

There are two ways to view this. Either it it great that the logs are kept and so I have all this historical data or it is a pain as getting red of log files becomes more of a chore, indeed this is encouraging me to move all the logfiles into their own file systems so that the management of those logfiles is more granular.

At the very least it seems to me that OpenSolaris should sort out where it's log files are going and end the messages going in /var/adm and move them to /var/log which then should be it's own file system.

Tuesday Feb 17, 2009

VirtualBox pause and resume

I have a VirtualBox running ubuntu that is acting as a print server for my HP Officejet J6410 and although that works well it chews a significant amount of CPU resource which given that the printer is not in use most of the time seems a bit of a waste, especially when the Sun Ray server can have 4 active sessions running on it. So now I am running this simple script:

#!/bin/ksh93 -p 

logger -p daemon.notice -t ${0##\*/}[$$] |&

exec >&p 2>&1

while :
do
        echo resuming vbox
        su - vbox -c "VBoxManage -nologo controlvm ubunto resume"

        ssh print-server sudo ntpdate 3.uk.pool.ntp.org

        while ping printer 2 > /dev/null 2>&1
        do
                sleep 60
        done
        echo pausing vbox
        su - vbox -c "VBoxManage -nologo controlvm ubunto pause"
        until ping printer 2 > /dev/null 2>&1
        do
                sleep 1
        done
done

After the usual magic to redirect it's output to syslog this just loops forever pinging the printer. If the printer is alive the VrtualBox is resumed and if the printer is not alive the VirtualBox is paused. So as soon as the printer is turned on the virtual box is ready and then within 60 seconds of the printer being turned off the virtualbox is paused.


So that the clock on the VirtualBox is kept correct (the guest additions are, I am told, supposed to do this for free but in practice they do not for me) after resuming there is the ssh onto the virtual machine to set it's clock\*, so I have a work around in place for VirtualBox which is itself a work around for a defect in Solaris and that work around is also working around another issue with Solaris.


Life on the bleeding edge is always fun!



\*) I would normally use the address of my server to sync with but thanks to the issues with build 108 I currently don't have a working ntp server on the system.

Sunday Feb 15, 2009

Build 108

I've managed to upgrade my home server to build 108 which is an important mile-stone for me as it has the fix for:

6763600: nautilus becomes unusable on a system with 39000 snapshots.

Which was rendering nautilus close to unusable for any users who moved out of their lofs automounted home directories. In partilcular any attempt to use it to manage the photo directories was painful.

However all was not smooth as again I hit this bug:

6795892: Sun Ray X servers (both Xsun and Xorg) suffer network problems in snv_106

but since I was expecting this I tried the workaround from bug 6799655 which is the same as the one for 6763600:


In /etc/sock2path change the following lines:

    2   2   0   tcp
    2   2   6   tcp

    26  2   0   tcp
    26  2   6   tcp

    2   1   0   udp
    2   1   17  udp

    26  1   0   udp
    26  1   17  udp

to:

    2   2   0   /dev/tcp
    2   2   6   /dev/tcp

    26  2   0   /dev/tcp6
    26  2   6   /dev/tcp6

    2   1   0   /dev/udp
    2   1   17  /dev/udp

    26   1  0   /dev/udp6
    26   1  17  /dev/udp6

While this got the Sun Rays up it also stopped named from working, spewing errors like this:

Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] 71/Protocol error
Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] socket.c:4315: unexpected error:

So have had to revert to some internal to Sun binaries that work around this while the underlying bug is fixed. It is slighly worring as I'm left wondering what other victims are out there. One I have already found is ntp, which is a known bug:

6796948: NTP completely broken by I_SETSIG semantics change in snv_106

I suspect that the system will have to revert to build 105 soon.

Friday Feb 13, 2009

Mobile internet on OpenSolaris

I have a new mobile phone which is less steam driven than the one it replaces and, hopefully, will not power itself off and reset to factory defaults every few days. Being modern it can act as a USB modem device and therefore allows me to have my OpenSolaris Laptop work with it.

Following the instructions from James Leggs blog and some from an Email I had been sent (James' instructions were the same as the email) it all worked just fine except: The phone is a Nokia E71 and by default the USB setting was menu->connectivity->usb was set with "USB mode" to mass storage and "Ask on connection" to Yes. It seems that that was enough for OpenSolaris to see it as a disk drive and then not plumb up the serial devices. Changing the "USB mode" to "PC Suite" resolved this.

The only addition I have made was to add these scripts into /etc/ppp to disable nwam and fix resolv.conf when the link comes up and reverse that when it is taken down.

cjg@brompton:~$ cat /etc/ppp/ip-up
#!/bin/ksh -p

/usr/sbin/svcadm disable -t svc:/network/physical:nwam

mv /etc/resolv.conf /etc/resolv.conf.preppp

cp /etc/ppp/resolv.conf /etc/resolv.conf
chmod 644 /etc/resolv.conf
cjg@brompton:~$ 

and

cjg@brompton:~$ cat /etc/ppp/ip-down
#!/bin/ksh -p

test -f /etc/resolv.conf.preppp && mv /etc/resolv.conf.preppp /etc/resolv.conf

/usr/sbin/svcadm enable nwam
cjg@brompton:~$ 

Not rocket science and when nwam knows about ppp they will surely go but until then quite satisfactory.


Finally create a launcher that will start this at the click of a button. I have cheated here and simply have it starting in a terminal so I can stoop it by quitting the terminal:

cjg@brompton:~/Desktop$ cat 3G\\ network.desktop 

[Desktop Entry]
Encoding=UTF-8
Version=1.0
Type=Application
Terminal=true
Icon[en_GB]=/usr/share/icons/gnome/24x24/actions/call-start.png
Name[en_GB]=3G network
Exec=\\spfexec pppd call vodafone\\n
Name=3G network
Icon=/usr/share/icons/gnome/24x24/actions/call-start.png
cjg@brompton:~/Desktop$ 

Wednesday Feb 04, 2009

40" Sun Ray Display

I managed to buy 2 Sun Ray 2's off Ebay and one of them was is now in place in the living room driving our 40” TV.


Combine this with a KeySonic wireless mini keyboard and the DTU does not only act as a photo frame. The Sun Ray unit is attached to the underside of the shelf as the top unit in the pile is a Virgin cable TV recorder which does not like having anything on top blocking the air flow. Thanks to the Sun Ray 2 being so light 5 strips of sticky back velcro do the trick so well that it really is going nowhere to the point that I could not remove it to plug the USB keyboard adapter directly in the back of the unit. The keyboard adapter has a button you have to press once plugged in to pair it with the keyboard. Alas with the Sun Ray in this configuration the button faces upwards. So there is a short USB cable hidden back there.

Networking is provided via Ethenet over mains.

The keyboard has impressive range and a really nice touch pad that pretends to have a scroll wheel down one side. However I've not yet got the keyboard map for it right but it only arrived an hour ago so there is time.

Saturday Jan 31, 2009

Suspend and resume really does just work

The whole rebooting thing last night and the surprise that my laptop had been "up"for 18 days left me wondering how many times I had suspended the system. I turns out this is easy to check in /var/adm/messages, which thanks to the laptop being hibernated most of the night fails to get cleaned out by cron, using a short script:

cjg@brompton:~$ nawk '/SunOS/ { if (count > max) { max=count} ; count=0 } /resume/ { count++ } END { print "current",count; print "max",max }' /var/adm/messages 
current 3
max 109
cjg@brompton:~$ 

109 sucessful suspend and resume cycles is not bad. I can't find a kernel statistic that counts them directly which I think is a pity. Also I need to keep an eye on the messages files however with ZFS root and snapshots going on the whole log file rotation and clean up could do with rethinking even in the non laptop case.

Friday Jan 30, 2009

OpenSolaris updates virtualbox

After a short delay VirtualBox for OpenSolaris has been upgraded. Update manager popped up to tell me and ask if I wanted the upgrade and created a new boot environment with the package installed. Very cool. Alas now I have to reboot to install it so my laptop won't reach three weeks uptime:

cjg@brompton:~$ uptime
 10:12pm  up 18 days  0:05,  2 users,  load average: 0.16, 0.27, 0.84
cjg@brompton:~$ beadm list
BE            Active Mountpoint Space  Policy Created          
--            ------ ---------- -----  ------ -------          
opensolaris   -      -          89.37M static 2008-11-27 12:14 
opensolaris-1 -      -          8.39M  static 2008-11-30 13:14 
opensolaris-2 -      -          13.36M static 2008-12-02 11:27 
opensolaris-3 N      /          12.70M static 2008-12-06 20:02 
opensolaris-4 R      -          9.44G  static 2009-01-30 21:42 
cjg@brompton:~$ 

Thursday Jan 29, 2009

lwp_park and lwp_unpark

When you have a program that uses the locking primitives mutex_lock() and mutex_unlock and their POSIX equivlients if you truss the process you will often see calls to lwp_park() and lwp_unpark() appering:

/1:	 1.3658	lwp_park(0x00000000, 0)				= 0
/2:	 1.3659	lwp_unpark(1)					= 0
/1:	 1.3659	lwp_park(0x00000000, 0)				= 0
/2:	 1.3660	lwp_unpark(1)					= 0
/1:	 1.3660	lwp_park(0x00000000, 0)				= 0
/1:	 1.3661	lwp_unpark(2)					= 0
/2:	 1.3661	lwp_park(0x00000000, 0)				= 0

These system calls are, as their names imply, the calls that cause the current LWP to stop (park) and allow the current LWP to allow another parked LWP to run (unpark). If we consider the noddy example code:


#include <stdlib.h>
#include <unistd.h>
#include <thread.h>
#include <synch.h>

mutex_t global_lock;
int global_count;

static void \*
locker(void \*arg) 
{
	while (1) {
		mutex_lock(&global_lock);
		global_count++;
		mutex_unlock(&global_lock);
	}
}

int
main(int argc, char \*\*argv)
{
	int count;

	if (argc == 2) {
		count = strtol(argv[1], NULL, NULL);
	}  else {
		count = 2;
	}
	while (--count > 0) {
		thr_create(0, 0, locker, NULL, NULL, NULL);
	}
	locker((void \*)time);
}

The mutex global_lock is going to be battled over by all the threads that are created if one of those threads needs to sleep, as it can't get the mutex, then it has to make a system call so that it can stop using the CPU. The system call is lwp_park(). When the other thread, the one that has the mutex, releases the mutex it signals the blocked thread using lwp_unpark(), with the LWP id of the thread to which to start. This can be seen in the truss:


/1:	 1.3658	lwp_park(0x00000000, 0)				= 0
/2:	 1.3659	lwp_unpark(1)					= 0

/1:	 1.3659	lwp_park(0x00000000, 0)				= 0

/2:	 1.3660	lwp_unpark(1)					= 0
/1:	 1.3660	lwp_park(0x00000000, 0)				= 0
/1:	 1.3661	lwp_unpark(2)					= 0
/2:	 1.3661	lwp_park(0x00000000, 0)				= 0

however the truss can be a bit misleading. You have to remember that truss only reports on system calls when the system call returns (unless they block long enough to be reported as sleeping). So for a call like lwp_park which will sleep until there is a corresponding lwp_unpark call from another thread. In the truss output above you can see this. LWP 2, on the first line in red calls lwp_unpark(1) to unpark LWP 1, at this point LWP 1 returns from what was the blocked lwp_park() call and continues on it's merry way. Alas as can be seen it does not get very far before it once again finds itself blocked but that is what happens if your code is just fighting for a lock. If this were a real application then there are many D scripts that could be used to help track down your issue not least one like this:

pfexec /usr/sbin/dtrace -n 'syscall::lwp_park:entry /execname == "mutex"/
    { @[ustack(5)] = count() }'

Wednesday Jan 28, 2009

Cardiff System Administration Mash up

Big thank you Clive for arranging and Cardiff University for hosting the System Admin Mash up today. I was paritcularly pleased to have 100% of the Cardiff OpenSolaris User group present and to hear of people exploring COMSTAR on Thumpers to produce high performance OpenStorage at a very affordable price.

Lewis gave an excellent overview of the VSCAN & CIFS server with some brave demostrations using VirtualBox to host servers and clients. If you can persuade him to repeat it then do so.

However good though that was, for me, that was not the highlight of the day. That goes to Gwent Police Crime Forensics Unit who gave a excellent and and informative presentation about the challenges and successes of investigating issues around computer forensics. As storage devices get bigger the problems can only increase.

Tuesday Jan 27, 2009

New version of scsi.d required for build 106

This version supports some more filters. Specifically you can now specify these new options:

  • MIN_BLOCK only report on IO to less than or equal to this value.

  • MAX_BLOCK only report on IO to blocks greater or equal to this value.

This is most useful for limiting your trace to particular block ranges, be they file system or as was the case that caused me to add this to see who is trampling on the disk label.

In this contrived example it was format:

pfexec /usr/sbin/dtrace -o /tmp/dt.$$ -Cs  scsi.d -D MAX_BLOCK=3 
<SNIP>
00058.529467684 glm0:-> 0x0a  WRITE(6) address 01:00, lba 0x000000, len 0x000001
, control 0x00 timeout 60 CDBP 60012635560 1 format(3985) cdb(6) 0a0000000100
00058.542945891 glm0:<- 0x0a  WRITE(6) address 01:00, lba 0x000000, len 0x000001
, control 0x00 timeout 60 CDBP 60012635560, reason 0x0 (COMPLETED) pkt_state 0x1
f state 0x0 Success Time 13604us

While this answered my question there are neater ways of answering the question just by using the IO provider:

: s4u-nv-gmp03.eu TS 68 $; pfexec /usr/sbin/dtrace -n 'io:::start / args[0]->b_blkno < 3 && args[0]->b_flags & B_WRITE / { printf("%s %s %d %d", execname, args[1]->dev_statname, args[0]->b_blkno, args[0]->b_bcount) }'
dtrace: description 'io:::start ' matched 6 probes
CPU     ID                    FUNCTION:NAME
  0    629             default_physio:start format sd0 0 512
  0    629             default_physio:start format sd0 0 512
  0    629             default_physio:start format sd0 0 512
  0    629             default_physio:start format sd0 0 512
  0    629             default_physio:start format sd0 0 512
  0    629             default_physio:start format sd0 0 512

Also build 106 of nevada has changed the structure definition for scsi_address and in doing so this breaks scsi.d which has intimate knowledge of scsi_address structures. I have a solution that you can download but in writing it I also filed this bug:

679803 dtrace suffers from macro recursion when including scsi_address.h

which scsi.d has to work around. When that bug is resolved the work around may have to be revisited.

All versions of scsi.d are available here and this specific verison, version 1.16 here.

Thank you to Artem Kachitchkine for bringing the changes to scsi_address.h and their effects on scsi.d to my attention.

Friday Jan 23, 2009

Update to Sun Ray access hours script

I have just uploaded and update to my script that controls the access hours of users to Sun Ray DTUs. The change is to not block the DTU if the user accepts the warning and disconnects voluntarily.

With that test in place the number of forced disconnects has fallen to zero which is good as the warning about bug 6791062 still applies so this reduces the chances of hitting the bug (if it is indeed triggered by the change of the ipfilter rules).

Thursday Jan 22, 2009

Powerline network control from OpenSolaris


At last I can configure my ethernet over power devices from OpenSolaris. I do have to run Windows XP in VirtualBox in seamless mode with a virtual network adapter but you can hardly tell.


My requirement for Windows on the bare metal has just disappeared.

Wednesday Jan 21, 2009

Getting the right CTF

I just spent to long, way too long, working out why a system dump's CTF did not seem to match the source code or for that matter the assembler that had been generated.

When a Solaris release is released all the CTF is merged into the unix file. As updates are released any structures that change are not updated in the unix file (since the old definition may still be being used) so the CTF definition is held in the module in which they are defined.

So faced with a dump where I needed to look at the “conn_udp” element in “struct conn_s” mdb kept saying there was no element “conn_udp”:

> ::print -at conn_t conn_udp
mdb: failed to find member conn_udp of conn_t: no such member of structure or union
> 

since the assembler made it abundantly clear that we were indeed using this element (I would show you the source but this is Solaris 10 and the source is very different from the OpenSolaris code). The thing to recall was that the structure is really defined in the “ip” module so to get the correct definition you need this:

> ::print -at ip`conn_t conn_udp
30 struct udp_s \*conn_udp 
>

This also effects dtrace as that is also a consumer of CTF (note this dtrace is entirely pointless):

# dtrace -n 'fbt::udp_bind:entry / ((conn_t \*)(args[0]->q_ptr))->conn_udp / { tot++ }'
dtrace: invalid probe specifier fbt::udp_bind:entry / ((conn_t \*)(args[0]->q_ptr))->conn_udp / { tot++ }: in predicate: conn_udp is not a member of struct conn_s
# 

and again by getting the definition from the original module gets the right answer:

# dtrace -n 'fbt::udp_bind:entry / ((ip`conn_t \*)(args[0]->q_ptr))->conn_udp / { tot++ }'
dtrace: description 'fbt::udp_bind:entry ' matched 1 probe
\^C

#

Since “ip`conn_t” will always give the right answer even in the case where the merged CTF data is in unix and that is the current version it is best to understand where the object was declared.

I kind of wish that at least in the case of dtrace it should get this right at the very least when you have specified the module since it knows what module you are in it could choose the CTF from that module.


# dtrace -n 'fbt:ip:udp_bind:entry / ((conn_t \*)(args[0]->q_ptr))->conn_udp / { tot++ }'
dtrace: invalid probe specifier fbt:ip:udp_bind:entry / ((conn_t \*)(args[0]->q_ptr))->conn_udp / { tot++ }: in predicate: conn_udp is not a member of struct conn_s
#


Should IMO work.

Wednesday Jan 07, 2009

Access hours for Sun Ray users

Having installed a Sun Ray in my daughters bedroom I am now faced with the inevitable problem of her being online all night not getting any sleep and then being generally grumpy. The irony here is that I was sent an email asking how I handle access control to the DTUs and I said I just trusted the children to be sensible (what was I thinking!).

So a solution was required that gave access to the systems only between certain hours. The hours would depend on the user and would have to not loose all their “work” in case this was a late night finishing their homework session.

After asking around no one came back to me and said how it can be done so I wrote my own script. It works by having a file that contains lines with a format

user:starttime:endtime

The times are specified in 24 hour format and only accurate to the minute.

# cat /etc/opt/local/access_hours             
user1:1915:1900
user2:0630:2300
user3:0630:2230
user4:0630:2000
# 

The top line is just really for testing only not allowing access from 1900 to 1915. Then you need a user who has system admin privs which does not have a crontab file. Since I already have a kroot role I'm overloading this. Running the script as with the -c flag and the name of the user will write the crontab file. Note it also writes an entry to keep the crontab file uptodate on an hourly basis.


# /usr/local/sbin/check_access_hours -c kroot
# crontab -l kroot
46 \* \* \* \* /usr/local/sbin/check_access_hours -c kroot
00 19 \* \* \* /usr/local/sbin/check_access_hours user1
00 23 \* \* \* /usr/local/sbin/check_access_hours user2
30 22 \* \* \* /usr/local/sbin/check_access_hours user3
00 20 \* \* \* /usr/local/sbin/check_access_hours user4
#  

Finally I added a line to the utaction script that is already run for every user when they connect to a Sun Ray DTU:

if ! /usr/local/sbin/check_access_hours -t 0 $1
then
        exit 1
fi

The way it disallows access is that it adds the DTU's IP address to the ipfilter, which you have to have configured, so that all traffic from the DTU is blocked. It also submits an at(1) job to run 2 minutes in the future to remove the block so that the Sun Ray can burst back into life. The effect is that the user can no longer use any Sun Ray outside of the defined hours. But after about 2 minutes the DTU is usable again by others or indeed as a photo frame.


A word of warning. Having got all this running the system has paniced twice which is disappointing on one level, that it panics, but pleasing on another, I've found a bug that can now be fixed. The bug is:

6791062: System panic in ip_tcp_input when a rule is added to ipfilter

I look forward to the fix!


The script is here but check that that bug has been fixed before you use it.

Saturday Jan 03, 2009

Making the zfs snapshot service run faster

I've not been using Tim's auto-snapshot service on my home server as once I configured it so that it would work on my server I noticed it had a large impact on the system:

: pearson FSS 15 $; time /lib/svc/method/zfs-auto-snapshot \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    1m22.28s
user    0m9.88s
sys     0m33.75s
: pearson FSS 16 $;

The reason is two fold. First reading all the properties from the pool takes time and second it destroys the unneeded snapshots as it takes new ones. Something the service I used cheats with and does only very late at night. Looking at the script there are plenty of things that could be made faster and so I wrote a python version that could replace the cron job and the results , while and improvement were disappointing:

: pearson FSS 16 $; time ./zfs.py \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    0m47.19s
user    0m9.45s
sys     0m31.54s
: pearson FSS 17 $; 

still too slow to actually use. The time was dominated by cases where the script could not use a recursive option to delete the snapshots. The problem being that there is no way to list all the snapshots of a filesystem or volume but not it's decendents.


Consider this structure:

# zfs list -r -o name,com.sun:auto-snapshot tank
NAME                                  COM.SUN:AUTO-SNAPSHOT
tank                                  true
tank/backup                           false
tank/dump                             false
tank/fs                               true
tank/squid                            false
tank/tmp                              false

The problem here is that the script wants to snapshots and clean up “tank” but can't use recustion without backing up all the other file systems that have the false flag set and set for very good reason. Howeve If I did not bother to snapshot “tank” then tank/fs could be managed recusively and there would be no need for special handling. The above list does not reflect all the file systems I have but you get the picture. The results of making this change brings the timing for the service

: pearson FSS 21 $; time ./zfs.py \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    0m9.27s
user    0m2.43s
sys     0m4.66s
: pearson FSS 22 $; time /lib/svc/method/zfs-auto-snapshot \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    0m12.85s
user    0m2.10s
sys     0m5.42s
: pearson FSS 23 $; 

While the python module still gets better results than the korn shell script the korn shell script does not do so badly. However it still seems worthwhile spending the time to get the python script to be able to handle all the features of the korn shell script. More later.

Wednesday Dec 24, 2008

Timezone aware cron finally pushed to OpenSolaris

With this “push” yesterday:

changeset:   8439:51a23ac0d2a6
user:        Chris Gerhard <Chris.Gerhard@sun.com>
date:        Tue Dec 23 15:44:14 2008 +0000
files:       usr/src/cmd/cron/Makefile usr/src/cmd/cron/cron.c usr/src/cmd/cron/cron.h usr/src/cmd/cron/crontab.c usr/src/cmd/cron/funcs.c
description:
PSARC/2007/503 crontab entry environment variables
6518038 cron & crontab should support multiple timezones

OpenSolaris finally contains a version of cron that understands and correctly handles having different timezones. You can also specify a different home directory (useful when you don't want NFS to get involved in your cron job for any reason) and shell to run jobs in. It should be in build 106 of OpenSolaris & Nevada.

This brings you crontab in line with at(1) which has been timezone aware for some time.

To use simply set the variables HOME, TZ and SHELL in your crontab file and all subsequent lines will use those values until the next HOME, TZ and SHELL lines are found:

SHELL=/usr/bin/ksh
TZ=Africa/Abidjan
HOME=/tmp/cg13442
23 \* 1-9,11-26,28-29,31 2-10,12 \* exec /var/tmp/cron/crontest.sh.1.1 23 \\\* 1-9,1
1-26,28-29,31 2-10,12 \\\* Africa/Abidjan
HOME=/var/tmp/cg13442
3 0-7,9-10,12-22 1-6,8-9,11-21,24-26,28 1-7,9-10,12 \* exec /var/tmp/cron/crontes
t.sh.1.1 3 0-7,9-10,12-22 1-6,8-9,11-21,24-26,28 1-7,9-10,12 \\\* Africa/Abidjan
TZ=Africa/Accra
37 0-2,4-5,7-17,20-22 \* \* \* exec /var/tmp/cron/crontest.sh.1.1 37 0-2,4-5,7-17,2
0-22 \\\* \\\* \\\* Africa/Accra

Tuesday Dec 16, 2008

The best code is code you have forgotten.

Here is some shell code I had forgotten about:

if (( $(date +%m) == 12 ))
then
        ( while  (( $(date +%d) >= 15 ))
        do
        test -f /var/run/snow-opts && snow_opts=$(</var/run/snow-opts)
        /opt/csw/bin/xsnow ${snow_opts:--santa 2}
        done ) &
fi

all for the the Sun Ray picture frames. The kids, all of us, love it.

Friday Dec 12, 2008

Don't reboot to add swap on solaris.

Today I wish I had blogged this earlier after the second person who should have known better suggested that you need to reboot a system to increase the swap space when using ZFS for swap. The second one asserted it in such a way that the system I was using was rebooted!

Just like when we did not use zvols for swap you always had the option to have multiple swap devices that option is still there if you are using zvols. You can have multiple zvols just create them and add them to the vfstab so that when you do reboot you still have them. If however you don't like that you can always remove the existing swap device and add a new one of the new size.

Or if you really only want one swap volume and don't want to edit /etc/vfstab and like to live a little bit on the edge you can grow the swap device and then add the extra space as a new swap volume.

cjg@brompton:~$ swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 182,1 8 4126712 4111472
cjg@brompton:~$ pfexec zfs set volsize=4G rpool/swap
cjg@brompton:~$ pfexec zfs set reservation=4G rpool/swap
cjg@brompton:~$ pfexec env NOINUSE_CHECK=1 swap -a /dev/zvol/dsk/rpool/swap $((8+4126712))
cjg@brompton:~$ swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 182,1 8 4126712 4111472
/dev/zvol/dsk/rpool/swap 182,1 4126720 4261888 4261888
cjg@brompton:~$

When the system is finally rebooted it will end up with just a single swap device again.

Obviously the correct answer is that the operating system should just “do the right thing” and to that end I have filed this RFE: 6783886The bug should turn up on opensolaris.org in the next few days/hours.

About

This is the old blog of Chris Gerhard. It has mostly moved to http://chrisgerhard.wordpress.com

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today