News, tips, partners, and perspectives for the Oracle Solaris operating system
- October 18, 2019
Reimplementing a Solaris command in Python gained 17x performance improvement from C
As a result of fixing a memory allocation issue in the /usr/bin/listusers command, that had caused issues when it was converted to 64 bit, I decided to investigate if this ancient C code could be improved by conversion to Python.
The C code was largely untouched since 1988 and was around 800 lines long, it was written in an era when the number of users was fairly small and probably existed in the local files /etc/passwd or a smallish NIS server.
It turns out that the algorithm to implement the listusers is basically some simple set manipulation. With no arguments listusers just dumps a sorted list of all users in the nameservice, with the -l and -g it filters down the list of users and groups.
My rewrite of listusers in Python 3 turned out to be roughly a 10th of the number of lines of code - since Python includes native set manipulation and the C code had implemented the set operations it via linked lists.
But Python would be slower right ? Turns out it isn't and in fact for some of my datasets (that had over 100,000 users in them) it was 17 times faster. I also made sure that the Python version doesn't pull the entire nameservice into memory when it knows it is going to be filtering it based on the -l and -g options.
It also turned out to be much easier to fix a long standing bug that listusers did not properly expand nested groups. Nested groups did not exist as a concept when the original C code was written, but with LDAP they became possible.
It also turns out that 100 lines of Python 3 code will be a lot easier to maintain going forward - not that I expect listusers to need updating again given it survived several decades in its original form!
Update: A few of the comments asked about the availability of the Python version. The listusers command in Oracle Solaris 11.4 SRU 9 and higher. Since we ship the /usr/bin/listusers file as the Python code you can see it by just viewing the file in an editor. Note though that is not open source and is covered by the Oracle Solaris licenses.
As for the use of Python 3.5, that will automatically change over to Python 3.7 in a future SRU. At the time I did this work Python 3.5 was the latest I had available to do the work. Since this is a very simple Python program all we need to do to port it is update our Makefile to target it to 3.7 instead of 3.5. We will be doing this as part of a much bigger sweep through the Oracle Solaris code as we migrate away from Python 2.7.
Darren
https://cython.org/
https://www.youtube.com/watch?v=lxh4lsvEBhI