Monday Feb 02, 2009

About planet and some Python lessons

I have some exciting plans for using the planet feed aggregator and have over the last couple of weeks using the “Venus” code line. I can now install on Ubuntu and the install passes all its tests. I want to be able to write a filter for the my plazes and also see how the foaf output might be used. I met up with my colleague, Dave Edmondson and we discussed the strengths and weaknesses of Python and planet.

The script runs two other python programs called spider and splice.

Spider gets the feeds defined in config.ini and creates a local cahce version, splice, reads the cache and generates the new formats from templates. The diagram below does not illustrate the template source files for the output formats and I shall probably need to dig further into the code in order to understand what needs to be done.

The diagram also indicates the location of Venus' plugin, where the plazes filter should be located.

We also discussed Python. I have been trying to write a game theory solver for a 2x2 formal game. I was representing the game as a dictionary so that I could retreive game scores using the strategy names. One problem is that two dimensional dictionaries get syntactically combersome. I had ended up with a list as the key. In theory it should make the programming easier, where game is a dictionary attached to class instance g.

i.e. score =[('decoy', 'defend')]

makes great sense where decoy and defend are blue and red strategies, however, I have usually placed the evaluation of a score in an interation, and so coding the strategy names is rare e.g.

strategies=['heads', 'tails'];
for s in strategies:
   # some iterated code

It is probably simpler to represent the game as a 2x2 matrix held in a list and to use the classic technique of holding the names of the strategies in an ordered list so we can translate the matrix cell location such as n(1,1) into n(tails,tails) by looking co-ordinates up in one, or two name lists.

score=matrix(strategies.index('heads'), strategies.index('heads'))

This would also have the advantage that I could look for and use the matrix manipulation packages that exist to avoid writing a lot of code. The code would look a lot simpler, and not just because I have put a lot of it in an external package; this is usually a good clue that the answer is correct.

Lesson 1: Be careful when using dictionaries.


Tuesday Jun 03, 2008

Python bites

I wrote my first Python program over the weekend. I very foolishly ran out space in my root filesystem on my cobalt qube. This is v. stupid and had two causes.

  • I have installed an application in the root disk volume. While this is not unusual, it has a database and its log files in the same file system. An error caused the log files to grow and burst the file system. [ So RFE the app dude, the install should offer locations for the database and logs.].
  • The OS very sensibly mailed me quite a lot that I had a problem, but it mailed it to the administrator account on the qube and while I can access this mail account remotely, I didn't; I was busy.

So I decided to write a script that mailed me on my phone (via SMS) should this happen again. (I also need to move the data files and logs to sensible places).

I would normally do this in ksh, but the qube doesn't have this, so I started in bash. I quickly discovered that my version of bash doesn't have associative (or any) arrays. It does have the string handling facilities of the ksh, but I couldn't find them; I had forgotten the syntax.

    line="/root=OK"; state=${line##\*=}; echo $state
    $ OK

actually works. NB I tried the arrays on my virtual box ubuntu 7 build, which I now use as a terminal host for the qube; I get them inside a re-sizable x-window. The arrays seem to work, but not associative arrays, so its another nail in the Qube's coffin; the Qube's bash has no arrays at all.

When I say I've written it, its not yet finished, but what I have done shows me that its a very powerful and economic language. Given that this sort of script, 90% of the code is string handling and enviromental discovery, with one command at the end of the script doing the work. I actually only use the UNIX 'df' utility and 'mail' program. I invoked the mail program via


where 'despatchmail' is an external shell script and it means that I can publish the program without stating the destination e-mail addresses. It could also invoke mail directly if I choose. I provide the df reply via a pipe to the program. (I did this because a coding example was more easily available it could be done in a number of ways.)

I am particularly impressed with python's dictionary feature and created one to hold the previous state, one to hold the current state and one to hold the utilisations. I can then use the file system mount point name as the retrieval key for all three arrays. e.g.

   >>>states={};      # states is an empty dictionary
   >>>states[s[:s.index('=')]]=s[s.index('=') + 1:]
     # assigns the state clause to the dictionary, the filesystem name is the index.
   >>>print states
   >>>{'/root': 'OK'}

and the real one looks like

    {'var': 'OK', 'home': 'OK', 'root': 'OK'}

and the utilisations from the df are held as

    {'var': 9, 'home': 38, 'root': 77}

N.B. The utilisation values are held as integers and its now easy enough to write a test such that if a directories utilisation is above a threshold, then set the status code to something else

    for keys in states.keys():
        if utilisations[keys] > threshold:

the first line ensures the tests are performed for each value pair in the states dictionary object.

So I was pleased to find a decent problem to test the language out on. When I have finished it, I might publish it in full. It might be useful to others, and you might be able to point out where my COBOL trained brain is still using tricks I learned 25 years ago. I know the parser is very powerful, and hence a line of code can perform a number of function, which means that what I would expect to take several lines can usually be done in one.

tags: technology programming language python linux utility

Tuesday Mar 04, 2008


Tim Bray noted that two of Python's leading developers are joining Sun. I'm pleased, along with all the others that have welcomed them. You can read what they have to say on their own blogs at Ted Leung, and  Frank Wierzbicki.

When deciding to re-invest in my scripting skills last year I choose Python. I didn't do this for necessarily the best of reasons but I need to look inside planet planet and also a 3rd party plazes script, both of which are written in Python. I also spent several hours getting lost inside Red Hat's apt-get when I failed to install it properly a couple of years ago.

I bought  the O'Reilly book, Learning Python but have got stuck on the exercises on method inheritance and overlays. I must get them finished this week, so I can read the next chapters on my next plane journey.


Thursday Feb 21, 2008

Organisation and Communication

Just trying to sort myself out from my last trip abroad.

Firstly, I have sorted out my twitter so that I am now receiving on the phone from all my correspondents. I have looked at forwarding my twits onto my personal planet, but wget fails for some reason. I need to investigate further, although twitter's FAQ talks about a configuration feature. I shall only be using it occasionally, probably in emergencies to let people know that facebook/e-mail is not available to me, and to use other means to contact me.

It might be sensible to ensure that really important people subscribe and follow me, although they might expect a personal message, and also don't care about my internet connectivity.

I have also revised the my flickr application on facebook so that it forwards everything, not just the London pictures, and my facebook correspondents can now see the pictures as I upload them. I am not sure if it auto-updates, but I can force it to write into my mini-feed, so hopefully people get notified when new pictures go up. Now I just have to find time to upload them. The most recent uploads are from Spain and my Madrid trip, the most recent pictures are from Dubai. This will hopefully improve the quality of my facebook feed for those who follow me there.

Its a shame that I am trapped on such an old version of planet-planet, there's a couple of things that don't work as well as I'd hope. The slynkr posts are still broken, they don't display the description, but a generated comment, I have never got round to including either this blog, nor the flickr pictures. The plazes feed My Plazes - Atom feed is also mis-mapped, but  I am working my way through the O'Reilly python book to see if this helps me build a personal feed that does what I want.


Saturday Jun 16, 2007

Running Python on Windows XP

Learning PythonI got myself a copy of the O'Reilley "Learning Python" book and started working myway through it. For various reasons, I decided to try it on Windows; I thought the binary install would be easier and I am in the process of upgrading the Solaris version and I wanted to finish that first. The windows install from is quite cute, as you would expect but it requires some further configuration before the python imports and module search works properly.

Firstly my "My Documents" is on a different disk to the Program Files directory. The Python interpreter is installed by default into the C:\\Program Files directory. This means that Python is not in the %PATH% string and that the icon for the Python CLI starts a Python instance that can't see/find my programs. My first instinct was to write a script that invokes the interpreter from the python scripts directory and I came across this link.

This pointed me to the windows help center's A-Z command reference. (Obvious, but so easy not to know about; its available from the [Start] button or using F1 while the desktop is current).

I wrote the following mighty script, called run_python.

C:\\PROGRA~1\\PYTHON2.5\\python %1

which is located in my python modules directory/folder. This can be run from the command line or explorer to start the interpreter or takes a single file argument from a command line to run a python module. This still isn't what we want, so I found

which pointed out that the PATH variable can be edited using

My Computer - Properties - Advanced - Environment Variables

so this is what I did, which means that is when I [Start] [Run] [command], and then type 'python', the command interpreter starts. This means I can use the command line to run programs, or I can iconise the modules, or create batch files containing the python module invocation. I think my super script above will become redundant, but it proves to me that scripting will never go away and I am gratefull for the help of the two web sites above, which I though I'd share with you.


Tuesday Jul 25, 2006

My personal planet

I now have an instance of the Planet RSS feed aggregator running on my Qube under Linux 2.2 which is aggregating my Snipsnap and feeds, or would be if I hadn't discovered a misconfiguration that makes it difficult to blog on my snipsnap. (This seems to be a windows firewall problem!)

How do I do this?

Planet requires Python(v2,2 or better). So I downloaded the most recent Python (2.4) from and tried to compile it into an install directory, this failed because my TCL library was too old, and I couldn't replace this with an up to date version; the ActiveState TCL distribution complained about my libc's age and version, and with Linux, that's that. For more on this unfortunate state of affairs, check my personal wiki here.... Fortunately I have a friend (Chris Gerhard) who installed Python 2.x a long time ago on a Qube and I borrowed his installation directory, copied it to my system and ran a make install. The install points at /usr/local/bin which means I can use a ${PATH} variable to control if my new Python V2.x or old Python V1.x is invoked. it claims to be V2.4 so I have no idea how it works, but I then downloaded the Planet code from its home site,, which I'd like to say is very simple but that wouldn't be true for me since I don't read Python, so with a bit of trial and error, this is what to do -

  1. I created a user called planet and ensured that $PATH & $MANPATH were set appropriately. i.e. to invoke my new version of python and then unpacked the archive into the user home directory. A sub-directory to hold the run time logs was created and also a sub-directory to hold files for a specific planet instance. (./Logs & ${HOME}/${PLANETNAME}.
  2. I then copied the fancy templates into ${HOME}/${PLANETNAME}, together with planet.css.
  3. I created a target directory which needs to be in the web server's file system. I chose ${WEBSERVER_ROOT}/planet/${PLANETNAME} and placed the image files into the target directory. The image files include the planet logo, the feed format badges and the face icon(s). (The face icon is exactly that, it allows the logoing of articles in the HTML version of the feed and the authors suggest a picture of the article author). Ensure the planet user has write permissions to this directory
  4. I then created the config file, I used explicit file and file directory names and placed the config file in the planet instance directory i.e. ${HOME}/${PLANETNAME}. The config file references a cache directory which planet uses. Planet uses a cache directory, this needs to be created and referenced in the config.ini. I placed this in the /var file system.
  5. The planet script, needs to be run occasionally. I chose to use cron to do this and wrote a script to ensure that all output was kept and written to a log file. (I also wrote a second script to keep these under control). I run planet every 15 minutes which given there's only one author is frequent enough. The log tidy up I run once/day. These jobs run as user planet, which is`why the planet user needs write privilidges to the target directory.
  6. I amended the index.html.tmpl and the planet.css files to get a colour conformity with the rest of my web space (except this blog) and discovered that the atom logo is missing, so I went and got myself one of those. Most of the amendments I made are to the sidebar. My planet is here..., PlanetCycling is here... which has a vanilla html page. I changed H1 to get my green banner and left justification.

Here's my H1 rule, which is used to implement the page banner -

h1 {
margin-top: 0px;
padding-top: 20px;

background-color: #690;
border: thin dotted #808080;

font-family: "Bitstream Vera Sans", sans-serif; font-weight: normal;
letter-spacing: -2px;
text-transform: lowercase;
text-align: right;
color: white;

Hope this helps!





« July 2016