Friday Aug 07, 2009

Big speed-ups for OpenSSL

I just integrated the fix for

6850713 32bit openssl x86 performance can be greatly improved by enabling hand-crafted asm

The result is 32bit applications on x86 using OpenSSL will see large performance increases for many ciphers. Most modern Linux distributions enable the same ASM so fixing this brings us up to par with Linux. Especially nice are the speed-ups for AES. On my test machines I see a 2-3x speedup for AES.

One of the consumers of OpenSSL in OpenSolaris is SunSSH. By default SunSSH will prefer to use AES ciphers when possible - see Ciphers in ssh_config(1)/sshd_config(4).I ran a couple of quick tests to see what sort of performance improvement would be seen when using SunSSH with the updated OpenSSL libraries. I ran the following on my x2100 test machine with the old libraries and then again with the new libraries.

time dd if=/dev/zero bs=1024k count=500 | ssh hst cat >/dev/null'
and got back:

First run (with original OpenSSL libraries).

real       17.7
user        0.0
sys         0.6

Second run (with updated OpenSSL libraries).

real        9.5
user        0.0
sys         0.6

Overall a very nice speed-up! Expect to see this in build 122.

Thursday Jun 11, 2009

OpenSSL 0.9.8k

I just upgraded OpenSSL to version 0.9.8k. 0.9.8k is the latest stable version of OpenSSL. OpenSSL in OpenSolaris before this was at version 0.9.8a with backported security fixes. I moved OpenSSL from ON to SFW a couple of weeks ago so as to make the work of upgrading simpler. There were a couple of motivators for upgrading but the one with the biggest impact is undoubtedly performance - particularly on amd64 platforms. OpenSSL 0.9.8k comes with most ciphers implemented in hand-coded assembly for maximal performance on amd64. Almost all of this can be enabled on OpenSolaris (to maintain ABI compatibility I couldn't enable this for rc4). Simple comparisons of openssl speed of before and after show larger performance gains in many ciphers.

Here are a couple of specific speed-ups I saw:

Sun Fire 2100 - AMD Opteron CPU

md5 (8192):1.55x
sha256 (8192):2.30x
aes128 (8192):1.73x
rsa2048 (sign):3.43x
rsa2048 (verify):2.83x

Sun Fire x4150 - Intel Xeon CPU

md5 (8192): 1.71x
sha256 (8192): 2.05x
aes128 (8192): 1.40x
rsa2048 (sign): 3.00x
rsa2048 (verify): 2.55x

There is also hand-coded assembly for 32bit x86. This wasn't enabled with the latest putback but expect to see it soon giving very nice speed-ups for 32bit applications on x86.

Of course there are other advantages to keeping OpenSSL up-to-date - bug fixes being the other primary advantage.

Thursday Feb 26, 2009

Truecrypt on OpenSolaris

Recently I've been playing around with TrueCrypt on Solaris. TrueCrypt is a cross-platform (Linux, MacOS, Windows, FreeBSD) application which provides disk encryption. On the non-windows platforms it uses FUSE.

As I've been involved with getting FUSE ported to OpenSolaris and work on security technologies for Sun it seemed to be a suitable project. Porting TrueCrypt to Solaris hasn't been entirely straightforward.

Apart from all the usual issues with OS specific code, TrueCrypt is also a C++ application which uses wxWidgets. Luckily for me wxWidgets is now included in later builds of OpenSolaris so I didn't need to port that, unluckily for me the GNU C++ compiler ABI and the Solaris C++ compiler ABI are incompatible which meant that I had to port TrueCrypt to SunStudio. Not only had I to contend with TrueCrypt but FUSE on Solaris is also relatively immature and I hit a number of issues (including panics) when trying to make FUSE and TrueCrypt work well together. The relevant FUSE bugs (fixed a week or two ago) are 4067, 3523 and 6441.

The port is by no means finished, in fact there are still some major problems but I have managed to get as far as creating a new volume, mounting that volume and reading and writing data. Here's a screen-shot:

Thursday Jan 01, 2009

Enhanced command-line editing support in Kerberos admin tools

Back in November I added enhanced command line editing support to the Kerberos administration tools kadmin(1M), kadmin.local(1M) and ktutil(1M).

When run interactively these commands support a vastly improved interface - things like tab-completion of sub-commands, command-line editing and command history are suddenly available. The tecla library was used to add the enhanced functionality. Other commands using libtecla on Solaris/OpenSolaris are zonecfg(1M), svccfg(1M) and elfedit(1),.

libtecla can be configured by creating a ~/.teclarc file. As I use vi keybindings in my shell it makes sense to have the same keybindings when using libtecla applications. libtecla by default uses emacs keybindings. My ~/.teclarc looks like this:

$ cat ~/.teclarc 
edit-mode vi
I'm planning on contributing back the changes to MIT soon.

Tuesday Dec 30, 2008

Multiple changeset pushes to ON

When ON switched to Mercurial from Teamware earlier this year ON went from having per-file delta's to per-repository changesets. In many ways Mercurial is a superior SCM to Teamware but this particular difference makes something I regularly find useful more difficult - fixing multiple unrelated Change-Requests (bugs and RFEs, referred to as CRs) with a single putback/push whilst maintaining a per-CR delta/changeset. Here is what I now do when fixing multiple unrelated CRs.

Create clones and snapshots of ON

It is generally a good idea to clone from the nightly tag as it ensures that usr is in sync with usr/closed. This is probably not so important if ON is cloned from ZFS is nicely suited for use with Mercurial as painless clones and snapshots can be extremely handy.
$ zfs create -o compress=on local/onnv-clone
$ cd /local/onnv-clone
$ hg clone -r nightly onnv-clone /local/onnv-clone
$ hg clone -r nightly onnv-closed-clone usr/closed
$ zfs snapshot local/onnv-clone@`date +%Y%m%d`
I like to maintain one repository per independent CR. With ZFS there is almost no downside as clones can be created almost instantly.
$ zfs clone local/onnv-clone@`date +%Y%m%d` local/5047971
$ zfs clone local/onnv-clone@`date +%Y%m%d` local/6704459
$ zfs clone local/onnv-clone@`date +%Y%m%d` local/6763503
I then work on each CR in its own repository individually. Building, unit-testing etc. can be performed for each CR in isolation.

Mercurial Queue

Mercurial Queue (MQ) is an extension to Mercurial. MQ is patch focused and maintains a queue of patches which can be pushed or popped. For historical reasons MQ commands begin with q. The MQ extension is included with OpenSolaris. It is not enabled by default. It can be enabled by modifying ~/.hgrc.
$ ggrep -A 1 extensions ~/.hgrc 
hgext.Tq =
A new clone of onnv-clone is created which will be managed by the MQ extension of Mercurial. It will contain patches for all the individual CRs. The -c option to qinit indicates that the patches themselves should be managed in a Mercurial repository. By default this repository is .hg/patches.
$ zfs clone local/onnv-clone@`date +%Y%m%d` local/miscbugs
$ cd /local/miscbugs
$ hg qinit -c
For each of the individual workspaces (one per CR) the changes are exported as a patch and then imported into the MQ repository.
$ hg -R ../5047971 pdiffs > /tmp/5047971.patch
$ hg qimport /tmp/5047971.patch
adding 5047971.patch to series file
$ hg -R ../6704459 pdiffs > /tmp/6704459.patch
$ hg qimport /tmp/6704459.patch
adding 6704459.patch to series file
$ hg -R ../6763503 pdiffs > /tmp/6763503.patch
$ hg qimport /tmp/6763503.patch
adding 6763503.patch to series file
$ hg qseries
The comments should adhere to the ON convention. By default the comment will be the same as the patch name which can be set during qimport with the -n option. To change the comment of an existing patch the patch must be pushed to the top of the queue and the comment set with the qrefresh command.
$ hg qpush
applying 6763503.patch
Now at: 6763503.patch
$ hg qrefresh -m "6763503 Prompts for kadmin and ktutil use too much space"
$ hg qpush
applying 6704459.patch
Now at: 6704459.patch
$ hg qrefresh -m "6704459 assert defined in k5-thread.h produces number of false positives"
$ hg qpush
applying 5047971.patch
Now at: 5047971.patch
$ hg qrefresh -m "kadmin could use libtecla for enhanced command history and editing"
The patches are committed to the MQ repository so they can be cloned later.
$ hg qcommit -m "Patches for 6763503, 6704459, 5047971"

Build machine

A new clone on a build machine is created.
$ hg clone -r nightly onnv-clone
$ hg clone -r nightly onnv-closed-clone onnv-clone/usr/closed
The MQ repository is cloned so the build machine has all the MQ controlled patches.
$ cd onnv-clone
$ hg clone ssh://dev_host//local/miscbugs/.hg/patches .hg/patches
Patches are applied to the build-machine gate.
$ hg qpush -a
The fixes for the individual CRs can now be tested as a combined whole. Building, regression testing etc. are performed.

Push changes

The recommended pre-push procedure (hg pull-u, hg merge, hg commit -m "merge turd", hg recommit) will collapse all the changesets into a single changeset. As the CRs are independent this is undesirable - it would be better to have one changeset per CR. MQ provides a means to do that. The patches are popped off, the repository is updated, and a snapshot is created of the updated repository without the applied patches.
$ hg qpop -a
Patch queue now empty
$ hg pull -u
$ zfs snapshot onnv-clone@prepush
$ hg qpush -a
applying 6763503.patch
applying 6704459.patch
applying 5047971.patch
Now at: 5047971.patch
The MQ patches are converted into regular Mercurial changesets which are then pushed into the gate.
$ hg qrm -r 6763503.patch
$ hg qrm -r 6704459.patch
$ hg qrm -r 5047971.patch
$ hg push ssh://onhg@elpaso.eng//export/onnv-gate
If someone sneaks a push to onnv-gate in the window between the "hg qrm" and the "hg push" then
$ zfs rollback onnv-clone@prepush
$ hg pull -u
$ zfs destroy onnv-clone@prepush
$ zfs create onnv-clone@prepush
$ hg qrm -r -a
$ hg push ssh://onhg@elpaso.eng//export/onnv-gate
The above is repeated until the push succeeds.

Thursday Apr 17, 2008

What's new for Kerberos in Solaris 10 5/08

Solaris 10 5/08 was just released and it contains a number of significant enhancements to Kerberos. I've drawn up a list of new features, singling out the ones I think are most significant. Even apart from these new features there were many bug fixes and other minor improvements.

  • Support for Kerberos principal and policy records in LDAP

    Adds support for accessing Kerberos principal and policy records stored on a Directory Server, via LDAP. Storing Kerberos records in LDAP has a number of advantages over the default db2 database such as simplified administration and allowing for multi-master KDC configuration. The KDC now supports a pluggable system for database storage. Both the new LDAP support and the older db2 database storage system are now implemented as plugins. Configuring a KDC with an LDAP database backend is a little different from the older db2 administration. A new utility kdb5_ldap_util and new configuration options are needed. You can find an updated Kerberos administration guide here which covers this new feature.

  • Zero-configuration Kerberos clients

    This is not a single feature per-se but rather a number of relatively minor changes which can obviate the need to explicitly configure Kerberos clients. The following changes were made

    • dns_lookup_kdc is enabled by default
    • The Kerberos realm for a given host (or default realm) is determined heuristically from the DNS domain name of the host ( or local host)
    • krb5.conf has been modified so that it is not mis-configured by default
    • Client-side referral support was added which allows the KDC to inform the client what realm they are in. Microsoft's Active Directory is an example of a KDC which supports this

  • Full resync with MIT 1.4.3 The Kerberos daemons, utilities and libraries are now fully in sync with MIT Kerberos 1.4.3 and contain some features of MIT 1.6. A new utility was introduced - k5srvutil and new options were added to krb5.conf. One particularily useful option is auth_to_local, it allows for a more flexible mapping of principal names to local user names.




« July 2016