Tuesday Nov 24, 2009

Clear up those temporary files

One of my (many) pet peeves are shell scripts that fail to delete any temporary files they use. Included in this pet peeve are shell scripts that create more temporary files than they absolutely need, in most cases the number is 0 but there are a few cases where you really do need a temporary file but if it is temproary make sure you always delete the file.

The trick here is to use the EXIT trap handler to delete the file. That way if your script is killed (unless it is kill with SIGKILL) it will still clean up. Since you will be using mktemp(1) to create your temporary file and you want to minimize any race condition where the file could be left around you need to do (korn shell):

trap '${TMPFILE:+rm ${TMPFILE}}' EXIT

TMPFILE=$(mktemp /tmp/$0.temp.XXXXXX)

if further down the script you delete or rename the file all you have to do is unset TMPFILE eg:

mv $TMPFILE /etc/foo && unset TMPFILE

Friday Jun 05, 2009

Possibly the best shell programming mistake ever

A colleague, lets call him Lewis, just popped over with the most bizarrely behaving shell script I have seen.

The problem was that the script would hang while the automounter timed out an attempt to NFS mount a file system on the customer's system.

I narrowed it down to something in a shell function that looked like this:

# Make a copy even if the destination already exists.
 	typeset src="$1"
	typeset dst="$2"

	/\* Nothing to copy \*/
	if [ ! -f $src ] ; then

        if [ ! -h $src -a ! -h $dst -a ! -d $dst ] ; then
		cp -p $src $dst || exit 1

safe_copy was called with a file as the $1 and a file as $2.

I laughed when saw the problem. Funny how you can read something and miss such an obvious mistake!

Thankfully the script has quietly been fixed.

Tuesday Nov 25, 2008

Redirecting output to syslog

People are always asking this and often when they are not they should be. How do you redirect all the output from a script to syslog?

The obvious is:

my_script | logger -p local6.debug -t my_script 2>&1

but how can you do that from within the script? Simple put this at the top of your script:


logger -p daemon.notice -t ${0##\*/}[$$] |&

exec >&p 2>&1

Clearly this is korn shell specific but then who still writes bourne shell scripts. If you script was called redirect you get messages logged thus:

Nov 25 17:40:41 enoexec redirect[17449]: [ID 702911 daemon.notice] bar

Friday Sep 07, 2007

PATH manipulation...

I just read this blog entry about reducing your path and realised there were two reasons I would not be doing that:

  1. Calling perl from a dot file would be one step to far.

  2. I have some korn shell functions for manipulating PATH variables that I have been using for years that amongst other things prevent duplicate entries in your path.

I realise I should share these with the world, however since the SCCS is dated 1996 I will use the defence that they work and are very useful when you all tell me that the scripts are not nice.

Shell function


apath [ -p PATH_VARIABLE][-d|-f|-n] path ...

Append the paths to end of the PATH_VARIABLE listed. If the path is already in the PATH_VARIABLE then the path is moved to the end of the variable.

ipath [ -p PATH_VARIABLE][-d|-f|-n] path ...

Insert the paths at the beginning of the PATH_VARIABLE listed. If the path is already in the PATH_VARIABLE then the path is moved to the beginning of the variable.

rpath [ -p PATH_VARIABLE][-d|-f|-n] path ...

Delete the path from the PATH_VARIABLE .

tpath [ -p PATH_VARIABLE] path ....

Test if the path is in the PATH_VARIABLE

All the scripts, except tpath take the same arguments. -p lets you select the variable you are working on. -d, which is the default says that this is a path of directories, -f says it is a path of files and -n that it is a path of objects that don't exist in the file system (useful for NIS_PATH when I was supporting NIS+).

To use them you simply save the script in a directory that is part of your FPATH. Then make hard links to it for all the names:

$ for i in rpath tpath ipath
> do
> ln apath $i
> done

and now you can use them. Here is a typical use from my .profile:

ipath /opt/SUNWspro/bin
ipath -p MANPATH /opt/SUNWspro/man

This adds /opt/SUNWspro/bin to my PATH and /opt/SUNWspro/man to my MANPATH. (Actually this is a fabrication as I have a shell function like this to cover the common case:

function addpath
     typeset i
     for i in $@
           if apath $i
                 apath -p MANPATH ${i%/\*}/man

so my MANPATH remains close to my PATH but lets keep things simple.)

Not rocket science but really useful, even more so for interactive sessions when you want to manipulate your path.

Friday Jun 08, 2007

korn shell programming advice from David Korn

There has been some great stuff about shell scripts on the shell-discuss OpenSolaris list I'm lurking on (my cron changes were discussed there so I got added and have not removed myself).

Starting with the this previously internal advice. I particularly like the c-shell advice, even if the words “for any scripts” seem to be superfluous.

Then last night David Korn posted some advice for shell scripts.

I did not even know you could do

$ > foo date

Now I do and also know it is to be avoided.

Thursday Jun 07, 2007

zfs_versions of a file

This post reminds me that I should have posted my zfs_versions script a while back. I did not as it had a theoretical bug where if you could move a new file into place which had the same age as the old file it would not see this as a new version. I've fixed that now at the expense of some performance and the script is called zfs_versions.

The script lists all the different versions of a file that live in the snapshots.

Here is the output:

: pearson FSS 124 $; /home/cjg/bin/sh/zfs_versions  ~/.profile 
: pearson FSS 125 $; 

Compare this to the number of snapshots:

: pearson FSS 128 $; ls -1 ~/.zfs/snapshot/\*/.profile | wc -l
: pearson FSS 129 $; 

So I have 705 snapshots that contain my .profile file but actually only three of them contain different versions.


The addition of the check to fix the theoretical bug slows down the script enough that the programmer in me could not let it lie. Hence I now have the same thing in TCL.

Update 2

See http://blogs.sun.com/chrisg/entry/new_wish_and_tickle

Wednesday Jun 07, 2006

Update to icheck.sh

I have updated my icheck.sh script so that it now finds fragments. If you request it look for a fragment it will find all the inodes that use the block that contains that fragment.

# ~cg13442/lang/sh/icheck -d /dev/rdsk/c0t0d0s0  dd6d7 2> /tmp/err
dd6d7 is a fragment address. Rounding to block 16#dd6d0
inode 1ade4: file block: 16#0 device block: 16#dd6d7
inode 1adc8: file block: 16#0 device block: 16#dd6d0
# find / -xdev \\( -inum $((16#1ade4)) -o -inum $(( 16#1adc8 )) \\) -print

The script is here.


Monday Jun 05, 2006

Mapping disk blocks to UFS file blocks.

Ever since Solaris 2.0 people have been asking for a way to map from a block on a disk back to the file which contains that block. In SunOS 4 and earlier used to have icheck(8) but that was never available in Solaris 2.0.

The answer that is usually given is short: “Use fsdb”. However fsdb is slightly less than friendly and in fact doing an exhaustive search of a file system for a particular block would be close to impossible to do by hand.

I was left thinking this must be able to be scripted. Since my particular issue was on Solaris 8 I had the added constraint that it would have to be a shell script from the choice of shells on Solaris 8.

As an example of things you can do I have written a script that will drive fsdb and can be used to:

  1. Copy files out of unmounted file systems (with the caveat that they get padded to be whole number of blocks). I used this to test the script then compare the source file and target. I have left it in more amusement.

  2. Find which inode and offset contains a particular disk block (blocks get specified in hex):

    # icheck  -d  /dev/rdsk/c0d0s6  007e1590 007dd2a0 008bb6c0
    inode 5c94: file block: 16#80b device block: 16#007e1590
    inode 5c94: file block: 16#1ffff device block: 16#007dd2a0
    inode 5c94: file block: 16#7ffffff device block: 16#008bb6c0

    This search can be directly limited to a single inode using the -i option and an inode number (in hex).

  3. Print the extents of a file. Again this is just mildly amusing but shows how well or badly UFS is doing laying out your files.

    # icheck -d /dev/rdsk/c0t0d0s0 -x -i 186e
    file off 0 dev off 6684 len 1279
    file off 1279 dev off 6683 len 1

The user interface could live with being tidied up but my original goal has been satisfied.

The script itself is not for those with a weak stomach as it works around some bugs and features in fsdb. The script is here if you wish to see the full horror.


Thursday Mar 02, 2006

Logging commands in korn shell

Yet another blast from the past, but I was asked for this again today.

How can you log every command typed into a korn shell session? Here is the cheap and dirty but surprisingly useful way that logs them all into syslog.

Type this into your shell and you can capture the command, it's return code and the current working directory.

function dlog
        typeset -i stat=$?
        typeset x
        x=$(fc -ln -0)
        logger -p daemon.notice -t "ksh $LOGNAME $$" Status $stat PWD $PWD \\'${x#       }\\'
trap dlog DEBUG

(note that there is a tab after the # in “${x# }”)

You might want to use a different logging facility but that one gets it into /var/adm/messages:

Mar  2 14:44:15 estale ksh cg13442 497922: [ID 702911 daemon.notice] Status 0 PWD /home/cg13442 'ls'
Mar  2 14:44:18 estale ksh cg13442 497922: [ID 702911 daemon.notice] Status 1 PWD /home/cg13442 'false'
Mar  2 14:45:09 estale ksh cg13442 497922: [ID 702911 daemon.notice] Status 0 PWD /home/cg13442 'ls -la'

I had run ls, false and “ls -la” which is dutifully logged.


Thursday Dec 15, 2005

Letting users create ZFS file systems

Darren has just posted his fast bringover script that solves some of my desire to be able to have a file system per workspace. I'm not commenting on the script since it manages to trip one of my shell script peeves that of calling a program and then calling exit $?. What is wrong with exec? I'll keep taking the tablets.

However it does not solve my wanting to be able to let users be able to create their own ZFS file systems below a file system that they own.

Like I said in the email this can mostly be done via an RBAC script, well here it is:

#!/bin/ksh -p


if [ "$_" != "/usr/bin/pfexec" -a -x /usr/bin/pfexec ]; then
        exec /usr/bin/pfexec $0 $@

function get_owner
	echo $(ls -dln ${PARENT} | nawk '{ print $3 }')

function create_file_system
	typeset mpt name

	zfs list -H -t filesystem -o mountpoint,name,quota | \\
		 while read mpt name quota
		if [[ $mpt == $PARENT ]]
			zfs create ${DIR#/} && chown $uid $DIR && \\
				zfs set quota=${quota} ${DIR#/}
			exit $?
	echo no zfs file system $PARENT >&2
	exit 1

function check_quota
	typeset -i count
	typeset mpt name

	zfs list -H -t filesystem -o mountpoint,name | while read mpt name
		if [[ $(get_owner $name) == $uid ]]
			let count=count+1
	echo $count


test -f /etc/default/zfs_user_create && . /etc/default/zfs_user_create

if [[ $# -ne 1 ]]
	echo "Usage: $1 filesystem" >&2
	exit 1


if ! [[ -d $PARENT ]]
	echo "$0: Failed to make directory \\"$1\\"; No such file or directory" >&2
	exit 1

uid=$(id | sed -e s/uid=// -e 's/(.\*//')
owner=$(get_owner $1)

if [[ $uid != $owner ]]
	echo "$0: $1 not owner" >&2
	exit 1

if [[ $(check_quota) -gt ${MAX_FILE_SYSTEMS_PER_USER} ]]
	echo "too many file systems"
	exit 1


It has a hack in it to limit the number of file systems that a user can create just to stop them being silly. Then you just need the line in /etc/security/exec_attr:


Now any user can create a file system under a file system they already own. The file systems don't share a single quota which would be nice but for my purposes this will do.

Next trick to let them destroy them and take snapshots of them. The snapshots being the real reason I want all of this.


Friday Nov 18, 2005

ZFS snapshot on boot

ZFS hits my Toshiba laptop and so I have moved the home directories onto a zfs file system.

The big advantage for me is that I can now have the system snapshot the filesystem each time it boots. First this script does the work:

#!/bin/ksh -p
date=$(date '+%F-%T')

for fs in $(zfs list -H -o name -t filesystem)
        zfs snapshot ${fs}@${date}

And this manifest gets it to be run:

<?xml version="1.0"?>

<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='snapshot'>


        <create_default_instance enabled='true' />

                <service_fmri value='svc:/system/filesystem/local' />

                timeout_seconds='10' />

                timeout_seconds='3' />

        <property_group name='startd' type='framework'>
                <propval name='duration' type='astring' value='transient'/>


Now import the manifest into smf:

 # svccfg import  snapshot.xml

Now I get a snapshot of the home directories everytime I reboot. These are mounted under .zfs/snapshot so users can pull back anything they need when they want to. Just what you want from a file system.

Tags: , , ,

Thursday Sep 01, 2005

Sad shell programming problem

While I was away an email ended up in my mail box like this:

I have a question on scripting ...

I want a "timedwait" for a background process started from a shellscript, i.e. do something like:

while [ 1 ]; do
        start_bg_job &
        "timedwait $! 30" || kill -9 $?

i.e. wait for the background job termination - but no more than 30 secs, if it didn't finish then kill it hard.

There were a couple of replies but none of them were “pure” shell scripts but it was weeks ago so who cares. Well it nagged at me last night as I cycled home. There has to be away to do this from the korn shell, and as usual the answer came to me just outside Fairoaks Airport (that is the answer came to me as usual while cycling home rather than outside the Airport.. If all the answers came to me outside the Airport I would be there a lot).

The trick (hack may be more appropriate) is to use a co process and start both the command you wish to run and a sleep command in the background in a sub shell that is the co process. When each of the commands return you echo information as to what to do back to the parent process which reads from the co process and takes the appropriate action.

So I ended up with this shell function:

#!/bin/ksh -p

# run_n_wait “command” [timeout]
function run_n_wait
        typeset com command time pid arg


        ( ( ( $command ) > /dev/null 2>&1 & \\
                echo pid $! ; wait $! ;\\
                echo Done $? ) & \\
         (sleep $time ; echo kill ) & ) |&

         while read -p com arg
                case $com in
                kill)  if [[ "${pid}" != "" ]]
                                kill ${pid} > /dev/null 2>&1
                                wait ${pid}
                        return -1 ;;
                pid) pid=$arg ; arg="" ;;
                Done) return $arg ;;

run_n_wait "/bin/false" 3
echo Slept for $(($SECONDS - $x)) seconds ret $?
run_n_wait "sleep 5 " 
echo Slept for $(($SECONDS - $x)) seconds ret $?
run_n_wait "sleep 60" 3
echo Slept for $(($SECONDS - $x)) seconds ret $?

Yes there are lots of shells that could just do this out of the box but that was not the question. If you have a better way (korn shell, bourne shell only) let me know.


Saturday Jul 02, 2005

A new root shell.

Fed up with the bourne shell for root? All the power of root but with a proper shell, not csh, a proper shell! You can add a role with the korn shell or any other shell and then assign that role to the users you wish to be able to access it. They still have to type the password of the role but they get a sensible shell when they get it right, plus others don't even get the option.

Here is how. For the a korn shell “root” account:

# roleadd -d /root -P "Primary Administrator" -s /usr/bin/pfksh kroot
# usermod -R root,kroot me
# passwd kroot
New Password:
Re-enter new Password:
passwd: password successfully changed for kroot

Now I have a role, kroot, to which only I can su(1M) and it has a decent shell. I can still use the root role if I want pain and I have not changed root's shell which is probably a good thing. Make sure /root already exists, it did for me as it is root's home directory already.


Thursday May 05, 2005

Base Conversion

Some one in a chat room asked about converting from hex to decimal. The usual answers came up, use mdb or adb or dtcalc oddly the vastly superior gnome-calculator did not get a mention. Anyway my solution was declared “cool” by one person so that is enough to get on the blog.

The solution is the korn shell. Converting from hex to decimal do this:

echo $(( 16#4000 ))

Will convert 4000 to decimal

Converting to hex is slightly harder:

typeset -i16 x
echo $x

Which cries out for a shell function:

function convert_base
        typeset -i${2:-16} x
        echo $x

Putting that in a file called convert_base in your FPATH thus allowing:

: enoexec.eu FSS 12 $; convert_base 4000
: enoexec.eu FSS 13 $; convert_base 4000 2
: enoexec.eu FSS 14 $; convert_base 16#4000 2
: enoexec.eu FSS 15 $; convert_base 16#4000 10
: enoexec.eu FSS 16 $;

Well I like it.

Tag: , ksh


This is the old blog of Chris Gerhard. It has mostly moved to http://chrisgerhard.wordpress.com


« August 2016