X

News, tips, partners, and perspectives for the Oracle Solaris operating system

  • October 18, 2019

Reimplementing a Solaris command in Python gained 17x performance improvement from C

Darren Moffat
Senior Software Architect

As a result of fixing a memory allocation issue in the /usr/bin/listusers command, that had caused issues when it was converted to 64 bit, I decided to investigate if this ancient C code could be improved by conversion to Python.

The C code was largely untouched since 1988 and was around 800 lines long, it was written in an era when the number of users was fairly small and probably existed in the local files /etc/passwd or a smallish NIS server.

It turns out that the algorithm to implement the listusers is basically some simple set manipulation.  With no arguments listusers just dumps a sorted list of all users in the nameservice, with the -l and -g it filters down the list of users and groups.

My rewrite of listusers in Python 3 turned out to be roughly a 10th of the number of lines of code - since Python includes native set manipulation and the C code had implemented the set operations it via linked lists.

But Python would be slower right ?  Turns out it isn't and in fact for some of my datasets (that had over 100,000 users in them) it was 17 times faster.  I also made sure that the Python version doesn't pull the entire nameservice into memory when it knows it is going to be filtering it based on the -l and -g options.

It also turned out to be much easier to fix a long standing bug that listusers did not properly expand nested groups.  Nested groups did not exist as a concept when the original C code was written, but with LDAP they became possible.

It also turns out that 100 lines of Python 3 code will be a lot easier to maintain going forward - not that I expect listusers to need updating again given it survived several decades in its original form!

Update: A few of the comments asked about the availability of the Python version.  The listusers command in Oracle Solaris 11.4 SRU 9 and higher.  Since we ship the /usr/bin/listusers file as the Python code you can see it by just viewing the file in an editor.  Note though that is not open source and is covered by the Oracle Solaris licenses.

As for the use of Python 3.5, that will automatically change over to Python 3.7 in a future SRU. At the time I did this work Python 3.5 was the latest I had available to do the work.  Since this is a very simple Python program all we need to do to port it is update our Makefile to target it to 3.7 instead of 3.5. We will be doing this as part of a much bigger sweep through the Oracle Solaris code as we migrate away from Python 2.7.

Darren

Join the discussion

Comments ( 5 )
  • Phil Wiggins Friday, October 18, 2019
    Great work Darren. You may want to consider incrementally adding some static typing to your Python3 and compile using cython. There are probably additional speed improvements beyond the 17x (fabulous, BTW!) by removing lots of the python overhead.
    https://cython.org/
    https://www.youtube.com/watch?v=lxh4lsvEBhI
  • JensE Saturday, October 19, 2019
    The new algorithm with bugfixes in C++ should yield a much faster execution time.
  • chiefartificer Sunday, October 20, 2019
    Very! Very! Interesting. Thanks for sharing it.
  • wzis Sunday, October 20, 2019
    To make Solaris more secure,only our software can do that. Solaris has UFS, ZFS and could also possible have VxFS, and attackers can create hidden files on these filesystems. Only our software can detect hidden files on these filesystems,and only our software can combat software based password stealing attacks and help users to run other software to also be able to combat software based password stealing attacks. Our software can detect trust,dtrace, TTY keylogger, memory snoopers,and also can help to prevent Trojan horse.
  • Paul Johnston Monday, October 21, 2019
    Nice one! It's cool that you took the time to look into this. Do you plan on releasing the code on github or is it throwaway code?
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.