Wednesday Jan 29, 2014

Ksplice SNMP Plugin

The Ksplice team is happy to announce the release of an SNMP plugin for Ksplice, available today on the Unbreakable Linux Network. The plugin will let you use Oracle Enterprise Manager to monitor the status of Ksplice on all of your systems, but it will also work with any monitoring solution that is SNMP compatible.

Installation

You'll find the plugin on Ksplice channel for your distribution and architecture. For Oracle Linux 6 on x86_64 that's ol6_x86_64_ksplice. Install the plugin by running (as root):

yum install ksplice-snmp-plugin

Configuration

If you haven't set up SNMP before, you'll need to do a little bit of configuration. Included below is a sample /etc/snmp/snmpd.conf file to try out the plugin:


# Setting up permissions
# ======================
com2sec local localhost public
com2sec mynet 10.10.10.0/24 public

group local v1 local
group local v2c local
group local usm local
group mynet v1  mynet
group mynet v2c mynet
group mynet usm mynet

view all included .1 80

access mynet "" any noauth exact all none none
access local "" any noauth exact all all none

syslocation Oracle Linux 6
syscontact sysadmin <root@localhost>

# Load the plugin
# ===============
dlmod kspliceUptrack /usr/lib64/ksplice-snmp/kspliceUptrack.so

 

You'll want to replace the IP address next to com2sec with the address of your local network, or the address of your SNMP monitoring software. If you are running on a 32 bit architecture x86), replace lib64 in the dlmod path with lib. The above configuration is an example, intended only for testing purposes. For more information about configuring SNMP, check out the SNMP documentation, including the man pages for snmpd and snmpd.conf.

Examples

You can test out your configuration by using the snmpwalk command and verifying the responses. Some examples:

Displaying the installed version of Ksplice:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceVersion
KSPLICE-UPTRACK-MIB::kspliceVersion.0 = STRING: 1.2.12

Checking if a kernel has all available updates installed:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate

Displaying and comparing the kernel installed on disk with the Ksplice effective version:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceBaseKernel
KSPLICE-UPTRACK-MIB::kspliceBaseKernel.0 = STRING: 2.6.18-274.3.1.el5

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceEffectiveKernel
KSPLICE-UPTRACK-MIB::kspliceEffectiveKernel.0 = STRING: 2.6.18-274.3.1.el5

Displaying a list of all installed updates:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable

In this case, there are none. This is why the base kernel version and effective kernel version are the same, and why this kernel is out of date.

Displaying a list of updates that can be installed right now, including their description:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable
KSPLICE-UPTRACK-MIB::kspliceavailIndex.0 = INTEGER: 0
KSPLICE-UPTRACK-MIB::kspliceavailIndex.1 = INTEGER: 1
KSPLICE-UPTRACK-MIB::kspliceavailIndex.2 = INTEGER: 2
KSPLICE-UPTRACK-MIB::kspliceavailIndex.3 = INTEGER: 3
KSPLICE-UPTRACK-MIB::kspliceavailIndex.4 = INTEGER: 4
KSPLICE-UPTRACK-MIB::kspliceavailIndex.5 = INTEGER: 5
KSPLICE-UPTRACK-MIB::kspliceavailIndex.6 = INTEGER: 6
KSPLICE-UPTRACK-MIB::kspliceavailIndex.7 = INTEGER: 7
KSPLICE-UPTRACK-MIB::kspliceavailIndex.8 = INTEGER: 8
KSPLICE-UPTRACK-MIB::kspliceavailIndex.9 = INTEGER: 9
KSPLICE-UPTRACK-MIB::kspliceavailIndex.10 = INTEGER: 10
KSPLICE-UPTRACK-MIB::kspliceavailIndex.11 = INTEGER: 11
KSPLICE-UPTRACK-MIB::kspliceavailIndex.12 = INTEGER: 12
KSPLICE-UPTRACK-MIB::kspliceavailIndex.13 = INTEGER: 13
KSPLICE-UPTRACK-MIB::kspliceavailIndex.14 = INTEGER: 14
KSPLICE-UPTRACK-MIB::kspliceavailIndex.15 = INTEGER: 15
KSPLICE-UPTRACK-MIB::kspliceavailIndex.16 = INTEGER: 16
KSPLICE-UPTRACK-MIB::kspliceavailIndex.17 = INTEGER: 17
KSPLICE-UPTRACK-MIB::kspliceavailIndex.18 = INTEGER: 18
KSPLICE-UPTRACK-MIB::kspliceavailIndex.19 = INTEGER: 19
KSPLICE-UPTRACK-MIB::kspliceavailIndex.20 = INTEGER: 20
KSPLICE-UPTRACK-MIB::kspliceavailIndex.21 = INTEGER: 21
KSPLICE-UPTRACK-MIB::kspliceavailIndex.22 = INTEGER: 22
KSPLICE-UPTRACK-MIB::kspliceavailIndex.23 = INTEGER: 23
KSPLICE-UPTRACK-MIB::kspliceavailIndex.24 = INTEGER: 24
KSPLICE-UPTRACK-MIB::kspliceavailIndex.25 = INTEGER: 25
KSPLICE-UPTRACK-MIB::kspliceavailName.0 = STRING: [urvt04qt]
KSPLICE-UPTRACK-MIB::kspliceavailName.1 = STRING: [7jb2jb4r]
KSPLICE-UPTRACK-MIB::kspliceavailName.2 = STRING: [ot8lfoya]
KSPLICE-UPTRACK-MIB::kspliceavailName.3 = STRING: [f7pwmkto]
KSPLICE-UPTRACK-MIB::kspliceavailName.4 = STRING: [nxs9cwnt]
KSPLICE-UPTRACK-MIB::kspliceavailName.5 = STRING: [i8j4bdkr]
KSPLICE-UPTRACK-MIB::kspliceavailName.6 = STRING: [5jr9aom4]
KSPLICE-UPTRACK-MIB::kspliceavailName.7 = STRING: [iifdtqom]
KSPLICE-UPTRACK-MIB::kspliceavailName.8 = STRING: [6yagfyh1]
KSPLICE-UPTRACK-MIB::kspliceavailName.9 = STRING: [bqc6pn0b]
KSPLICE-UPTRACK-MIB::kspliceavailName.10 = STRING: [sy14t1rw]
KSPLICE-UPTRACK-MIB::kspliceavailName.11 = STRING: [ayo20d8s]
KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd]
KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k]
KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]
KSPLICE-UPTRACK-MIB::kspliceavailName.15 = STRING: [qsajn0ce]
KSPLICE-UPTRACK-MIB::kspliceavailName.16 = STRING: [5tx9tboo]
KSPLICE-UPTRACK-MIB::kspliceavailName.17 = STRING: [2nve5xek]
KSPLICE-UPTRACK-MIB::kspliceavailName.18 = STRING: [w7ik1ka8]
KSPLICE-UPTRACK-MIB::kspliceavailName.19 = STRING: [9ky2kan5]
KSPLICE-UPTRACK-MIB::kspliceavailName.20 = STRING: [zjr4ahvv]
KSPLICE-UPTRACK-MIB::kspliceavailName.21 = STRING: [j0mkxnwg]
KSPLICE-UPTRACK-MIB::kspliceavailName.22 = STRING: [mvu2clnk]
KSPLICE-UPTRACK-MIB::kspliceavailName.23 = STRING: [rc8yh417]
KSPLICE-UPTRACK-MIB::kspliceavailName.24 = STRING: [0zfhziax]
KSPLICE-UPTRACK-MIB::kspliceavailName.25 = STRING: [ns82h58y]
KSPLICE-UPTRACK-MIB::kspliceavailDesc.0 = STRING: Clear garbage data on the kernel stack when handling signals.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.1 = STRING: CVE-2011-1160: Information leak in tpm driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.2 = STRING: CVE-2011-1585: Authentication bypass in CIFS.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.3 = STRING: CVE-2011-2484: Denial of service in taskstats subsystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.4 = STRING: CVE-2011-2496: Local denial of service in mremap().
KSPLICE-UPTRACK-MIB::kspliceavailDesc.5 = STRING: CVE-2009-4067: Buffer overflow in Auerswald usb driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.6 = STRING: CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.7 = STRING: CVE-2011-2699: Predictable IPv6 fragment identification numbers.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.8 = STRING: CVE-2011-2723: Remote denial of service vulnerability in gro.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.9 = STRING: CVE-2011-2942: Regression in bridged ethernet devices.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.10 = STRING: CVE-2011-1833: Information disclosure in eCryptfs.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.11 = STRING: CVE-2011-3191: Memory corruption in CIFSFindNext.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.12 = STRING: CVE-2011-3209: Denial of Service in clock implementation.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.13 = STRING: CVE-2011-3188: Weak TCP sequence number generation.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.14 = STRING: CVE-2011-3363: Remote denial of service in cifs_mount.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.15 = STRING: CVE-2011-4110: Null pointer dereference in key subsystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.16 = STRING: CVE-2011-1162: Information leak in TPM driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.17 = STRING: CVE-2011-2494: Information leak in task/process statistics.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.18 = STRING: CVE-2011-2203: Null pointer dereference mounting HFS filesystems.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.19 = STRING: CVE-2011-4077: Buffer overflow in xfs_readlink.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.20 = STRING: CVE-2011-4132: Denial of service in Journaling Block Device layer.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.21 = STRING: CVE-2011-4330: Buffer overflow in HFS file name translation logic.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.22 = STRING: CVE-2011-4324: Denial of service vulnerability in NFSv4.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.23 = STRING: CVE-2011-4325: Denial of service in NFS direct-io.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.24 = STRING: CVE-2011-4348: Socking locking race in SCTP.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.25 = STRING: CVE-2011-1020, CVE-2011-3637: Information leak, DoS in /proc.

And here's what happens after you run uptrack-upgrade -y, using Ksplice to fully upgrade your kernel:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: uptodate

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.0 = INTEGER: 0
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.1 = INTEGER: 1
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.2 = INTEGER: 2
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.3 = INTEGER: 3
[ . . . ]

The plugin displays that the kernel is now up-to-date.

SNMP and Enterprise Manager

Once the plugin is up and running, you can monitor your system using Oracle Enterprise Manager. Specifically, you can create an SNMP Adapter to allow Enterprise Manager Management Agents to query the status of Ksplice on each system with the plugin installed. Check out our documentation on SNMP support in Enterprise Manager to get started, including section 22.6, "About Metric Extensions".

This plugin represents the first step in greater functionality between Ksplice and Enterprise Manager and we're excited about what is coming up. If you have any questions about the plugin or suggestions for future development, leave a comment below or drop us a line at ksplice-support_ww@oracle.com.

Monday Oct 14, 2013

Best Practice: Ksplice Deployment

Ksplice is designed to work in many different computing environments. Because upgrading the kernel on any running system is a hassle, we want you to be able to deploy Ksplice as widely as possible. As a consequence of this, there are a number of ways to set up a Ksplice installation for your Oracle Linux infrastructure.

As the first part of a series of articles on Ksplice best practices, we've published a guide to deploying Ksplice on the Oracle Technology Network. This guide explains the different types of Ksplice deployments, helps you choose which one is right for your environment, and shows you how to get started. We've also produced a video describing these deployment options, as well as explaining how you can use Ksplice to diagnose kernel issues.

Once you've decided on how you'll install Ksplice you can find more information in the Ksplice User's Guide or through your normal Oracle support channels. Ksplice is available exclusively as part of Oracle Linux Premier Support.

Happy Ksplicing!


Was this best practice article helpful? Do you have a question about Ksplice, or an idea for a future best practice post? Let us know by leaving a comment below.

Thursday Aug 08, 2013

CVE-2013-2224: Denial of service in sendmsg().

In September 2012, CVE-2012-3552 was reported which could allow an attacker to corrupt slab memory which could lead to a denial-of-service or possible privilege escalation depending on the target machine workload.  This bug had originally been fixed in the mainline kernel in April 2011 and was a fairly large patch for a security fix.  The RedHat backport for this fix introduced a new bug which has been assigned CVE-2013-2224 which again could allow for a denial-of-service or possible privilege escalation.  Rack911 & Tortoiselabs created a reproducer in June 2013 which would allow an unprivileged user to cause a denial-of-service.

RedHat have not yet released a kernel with this CVE fixed, but CentOS have released a custom kernel with the vendor fix for CentOS 6.

We have just released a Ksplice update to address this issue for releases 5 and 6 of Oracle Linux, RedHat Enterprise Linux, CentOS and Scientific Linux.  We recommend that all users of Ksplice on these distributions install this zero-downtime update.

[Read More]

Friday May 31, 2013

CVE-2013-2850: Remote heap buffer overflow in iSCSI target subsystem.

We have just released a rebootless update to deal with a critical security vulnerability:

CVE-2013-2850: Remote heap buffer overflow in iSCSI target subsystem.

If an iSCSI target is configured and listening on the network, a remote
attacker can corrupt heap memory, and gain kernel execution control over
the system and gain kernel code execution.

As this vulnerability is exploitable by remote users, Ksplice is issuing an update for all affected kernels immediately.

This update was embargoed for release until today (May 30th), when the information regarding this vulnerability has been made public. We are pushing updates for Ubuntu Precise, Quantal, and Raring, as well as for Debian Wheezy, Fedora 17 and Fedora 18. This bug was introduced in version 3.1 of the Linux kernel and so does not affect Oracle UEK kernels, or any RedHat 6 derivatives or earlier.

We recommend Oracle Linux Premier Support for receiving rebootless kernel updates via Ksplice.

Wednesday May 15, 2013

Ksplice update for CVE-2013-2094

This is a 0-day local privilege escalation found by Tommi Rantala while fuzzing the kernel using Trinity. The cause of that oops was patched in 3.8.10 in commit 8176cced706b5e5d15887584150764894e94e02f.

'spender' on Reddit has an interesting writeup on the details of this exploit.

We've already shipped this for Fedora 17 and 18 for the 3.8 kernel, and an update for Ubuntu 13.04 will ship as soon as Canonical releases their kernel.

We have a policy of only shipping updates that the vendor has shipped, but in this case we are shipping an update for this CVE for Oracle's UEK2 kernel early. Oracle is in the process of preparing an updated UEK2 kernel with the same fix and will be released through the normal channels.

All customers with Oracle Linux Premier Support should use Ksplice to update their kernel as soon as possible.

[EDITED 2013-05-15]: We have now released an early update for Oracle RHCK 6, RedHat Enterprise Linux 6, Scientific Linux 6 and CentOS 6.

[EDITED 2013-05-15]: We have released an early update for Wheezy. Additionally, Ubuntu Raring, Quantal and Precise have released their kernel, so we have released updates for them.

Tuesday Apr 02, 2013

Ksplice Inspector

With so many kernel updates released, it can be difficult to keep track. At Oracle, we monitor kernels on a daily basis and provide bug and security updates administrators can apply without a system reboot. To help out, the Ksplice team at Oracle has produced the Ksplice Inspector, a web tool to show you the updates Ksplice can apply to your kernel with zero downtime.

The Ksplice Inspector is freely available to everyone. If you're running any Ksplice supported kernel, whether it is Oracle's Unbreakable Enterprise Kernel, a Red Hat compatible kernel, or the kernel of one of our supported desktop distributions, visit https://www.ksplice.com/inspector and follow the instructions and you'll see a list of all the available Ksplice updates for your kernel.

But what if you're running a system without a browser, or one without a GUI at all? We've got you covered: you can get the same information from our API through the command line. Just run the following command:

(uname -s; uname -m; uname -r; uname -v) | \
curl https://uptrack.api.ksplice.com/api/1/update-list/ \
-L -H "Accept: text/text" --data-binary @-

Once you've seen all the updates available for your kernel, you can quickly patch them all with Ksplice. If you're an Oracle Linux Premier Support customer, access to Ksplice is included with your subscription and available through the Unbreakable Linux Network. If you're running Red Hat Enterprise Linux and you would like to check it out, you can try Ksplice free for 30 days.

Let us know what you think by commenting below or sending us an email at ksplice-support_ww@oracle.com.

Friday Nov 09, 2012

Introducing RedPatch

The Ksplice team is happy to announce the public availability of one of our git repositories, RedPatch. RedPatch contains the source for all of the changes Red Hat makes to their kernel, one commit per fix and we've published it on oss.oracle.com/git. With RedPatch, you can access the broken-out patches using git, browse them online via gitweb, and freely redistribute the source under the terms of the GPL. This is the same policy we provide for Oracle Linux and the Unbreakable Enterprise Kernel (UEK). Users can freely access the source, view the commit logs and easily identify the changes that are relevant to their environments.

To understand why we've created this project we'll need a little history. In early 2011, Red Hat changed how they released their kernel source, going from a tarball that had individual patch files to shipping the kernel source as one giant tarball with a single patch for all Red Hat-introduced changes. For most people who work in the kernel this is merely an inconvenience; driver developers and other out-of-kernel module developers can see the end result to make sure their module still performs as expected.

For Ksplice, we build individual updates for each change and rely on source patches that are broken-out, not a giant tarball. Otherwise, we wouldn’t be able to take the right patches to create individual updates for each fix, and to skip over the noise — like a change that speeds up bootup — which is unnecessary for an already-running system. We’ve been taking the monolithic Red Hat patch tarball and breaking it into smaller commits internally ever since they introduced this change.

At Oracle, we feel everyone in the Linux community can benefit from the work we already do to get our jobs done, so now we’re sharing these broken-out patches publicly. In addition to RedPatch, the complete source code for Oracle Linux and the Oracle Unbreakable Enterprise Kernel (UEK) is available from both ULN and our public yum server, including all security errata.

Check out RedPatch and subscribe to redpatch-users@oss.oracle.com for discussion about the project. Also, drop us a line and let us know how you're using RedPatch!

Tuesday Feb 15, 2011

Happy Birthday Ksplice Uptrack!

One year ago, we announced the general availability of Ksplice Uptrack, a subscription service for rebootless kernel updates on Linux. Since then, a lot has happened!

Adoption

Over 600 companies have deployed Ksplice Uptrack on more than 100,000 production systems, on all 7 continents (Antarctica was the last hold-out). More than 2 million rebootless updates have been installed!

Ksplice at the South Pole

You see the greatest system administration time and cost savings when you use Ksplice Uptrack on all of your machines, and we've designed and priced Uptrack with this large-scale usage in mind. We've been very happy to see that our customers share this view: most of our customers either rolled out Ksplice across the board upon sign-up or have substantially expanded their usage after first purchasing for a particular environment.

We've introduced several features this year to make managing large Uptrack installations easier:

  • autoinstall: the Uptrack client can be configured to install rebootless updates on its own as they become available, making kernel updates fully automatic. Autoinstall is now our most popular configuration.
  • the Uptrack API: programmatically query the state of your machines through our RESTful API.
  • Nagios plugins: easily integrate Uptrack monitoring into your existing Nagios infrastructure with plugins that monitor for out of date, inactive, and unsupported machines.

Supported kernels

We continue to expand the distributions and flavors we support. You can now use Ksplice Uptrack on (deep breath) RHEL 4 & 5, CentOS 4 & 5 (and CentOS Plus), Debian 5 & 6, Ubuntu 8.04, 9.10, 10.04, & 10.10, Virtuozzo 3 & 4, OpenVZ for EL5 and Debian, CloudLinux 5, and Fedora 13 & 14. Within these distributions, we continue to support more flavors and older kernels; our launch may have been last year, but we support kernels back to 2007 and earlier!

You've always been able to use Ksplice in virtualized environments like VMWare, Xen, and Virtuozzo, This year, we've given you even more virtualization options by adding support for virtualization kernels like Debian OpenVZ and Debian Xen. Starting with Ubuntu Maverick, we support all Ubuntu kernel flavors, including the Maverick private cloud/EC2 kernel.

Thanks to some changes at Amazon and Rackspace, you can now even use Ksplice Uptrack on stock Linux kernels in Amazon EC2 and Rackspace Cloud.

As always, you can roll out a free trial on any of these distributions, for an unlimited number of systems.

This year, we also added Fedora Desktop to our free version options, so now you can use Ksplice Uptrack on both Ubuntu Desktop and Fedora Desktop completely for free.

Reboots saved

Did you know that between RHEL 4 and 5, Red Hat released 22 new kernels this past year?

chart: reboots on RHEL this past year

Without Ksplice, that's coordination and downtime for 22 reboots, including the hat trick of font-size: 125%; ">3 kernels in 3 weeks for RHEL 5. Gartner estimates that 90% of exploited systems are exploited using known, patched vulnerabilities, so if you're not rebooting and you're not using Ksplice, you are putting your servers (and your customers) at risk.

Looking forward

What do you want to see from Ksplice this year? What features can we add to help you deploy and monitor your Uptrack installations? Tell us what you want, and we'll do our best to deliver!

~jesstess

Monday Aug 30, 2010

Ksplice for Fedora!

In response to many requests, Ksplice is proud to announce we're now providing Uptrack free of charge for Fedora!

Fedora will join Ubuntu Desktop among our free platforms, and will give Fedora users rebootless updates as long as Fedora maintains each major kernel release.

However, of note: Fedora is the only Linux distribution that migrates to a new Linux kernel version family (e.g. 2.6.33 to 2.6.34) during the lifetime of the product. This kernel version family migration is such a major version change that Ksplice recommends a reboot for this version change.  These migrations occur roughly twice per year and only in Fedora; all of the other important Fedora kernel updates can be applied rebootlessly, as can the kernel updates for the rest of our supported Linux distributions.

We've also submitted the Uptrack client for integration into a later version of Fedora and are working with the Fedora folks to help make rebootless updates part of the distribution itself. Thanks for all your feedback about Uptrack and keep it coming! 

Best, 
the Ksplice Team

Tuesday Mar 09, 2010

How to quadruple your productivity with an army of student interns

Startup companies are always hunting for ways to accomplish as much as possible with what they have available. Last December we realized that we had a growing queue of important engineering projects outside of our core technology that our team didn't have the time to finish anytime soon. To make matters worse, we wanted the projects completed right away, in time for our planned product launch in early February.

So what did we do? The logical solution, of course. We quadrupled the size of our company's engineering team for one month using paid student interns.

20 men and women posing for a group photo

The Ksplice interns, ca. January 2010.

Now, if you happen to know Fred Brooks, please don't tell him what we did. He managed the Windows Vista of the 1960s---IBM's OS/360, a software project of unprecedented size, complexity, and lateness---and wrote up the resulting hard-earned lessons in The Mythical Man-Month, which everyone managing software projects should read. The big one is Brooks's Law: "adding manpower to a late software project makes it later". Oops.

Brooks's observation usually holds. New people take time to get up to speed on any system---both their own time, and the time of your experienced people mentoring them. They also add communication costs that grow quadratically with the size of the team.

Fortunately, Ksplice benefits from a bit of an engineering reality distortion field (our very product is supposed to be technologically impossible) and, with the right techniques, quadrupling our engineering team's size for one month worked out great. Every intern was responsible for one of the company's top unaddressed priorities for the month, and every intern successfully completed their project.

So, how do you quadruple the size of your engineering team in one month and still keep everyone productive?

Ten people working at computers in a room
At work in the Ksplice engineering office.

  • Tolerate a little crowding. It took a little creativity to suddenly find a dozen new workspaces in our two-room office. Fortunately, we've found that a room can always fit one more person---and by induction, you can fit as many as you need. (All those years we spent proving math theorems came in handy after all.) Seating everyone close to each other has an important advantage, too: when lots of people on your team have just started, it's handy for them to work right next to the mentors who are answering their questions and helping them ramp up on the learning curve of the organization. With the right team, the crowding can also create an energetic office environment that makes people love to come in to work. (Sometimes it gets in the way of concentration, though---that's when I put on a good pair of headphones.)
  • Locate next to a deep pool of hackers. OK, so we're a bit spoiled by being headquartered a few blocks away from the Massachusetts Institute of Technology. At MIT, January is set aside for students to pursue projects outside of the curriculum---perfect for hiring an intern army. Many other institutions have either a similar "January term", or a program for students to spend time working in industry during the term.
  • Know who the best people are and only hire them. Ksplice was born four years ago at SIPB, MIT's student computing group. When a group of students run computing services thousands of people rely on, and spend hours each week discussing, dreaming, collaborating, and learning from each other on computer systems---some of them get really good at it. Even better, everyone sees everyone else in action and knows exactly what it's like to work with them. Investing some time into getting involved with technical communities makes it possible to hire people based on personal experience with them and their work, which is so much better than hiring based on resumes and interviews. Companies like Google and Red Hat have known for years that being involved in the open source community can provide an excellent source of vetted job candidates.
  • Pay well. In some industries, "intern" means "unpaid"---but computer science students have plenty of options, and you want to be able to hire the best people. We looked at pay rates for jobs on campus, and pegged our rate to the high end of those.
  • Divide tasks to be as loosely-coupled as possible. Our internship program would never have worked if we had assigned a dozen new people to hack on our kernel code---the training time and communication costs that drive Brooks' Law would have swallowed their efforts whole. Fortunately, like any growing business, we had a constellation of tasks that lie around the edges of our core technology: infrastructure upgrades, additional layers of QA, business analytics, and new features in the management side of our product. These had manageable technical interfaces our existing software, so our interns were able to become productive with minimal ramp-up and rely on relatively little communication to get their projects done.
  • Design your intern projects in advance. A key challenge when scaling up your engineering team quickly is making sure that the interfaces are all well designed and the new projects will meet the company's needs. So we spent a good deal of time getting these designs together before the interns started. We also allocated plenty of our core engineers' time for code reviews and other feedback with our interns in order to make sure their work would be maintainable after they left.
Have you achieved more in one month than anyone thought should be possible? How did you do it?

[Edited to make clear our interns were paid and to say more about how we designed their projects for high-quality output.]

~price

Thursday Mar 04, 2010

/etc/init.d/blog start

Welcome to the Ksplice blog!

I struggled for quite some time coming up with a grand opening for this blog. Something profound like "We the people, in order to form more perfect computing..." or "The history of the world is the history of rebooting". But that proved too hard, so instead, I want to take a moment to briefly tell the Ksplice story, in the hopes of providing a bit more insight into who we are and what we're all about.

Ksplice was born on a beautiful summer day back in 2006. Crickets chirping, little kids splashing each other by the pool, ice cream trucks, etc. Ksplice's CEO, Jeff Arnold, then a student, was administering some very popular servers at MIT. A software update came out in the middle of the week, and, as any prudent system administrator would, he scheduled an outage window for the following Sunday at 2am, so as to disrupt as few users as possible, and went along his merry way.

Swimming pool
Where I wish I had been, Summer 2006
Of course, something much more sinister was lying in wait. Someone took advantage of the very bug that the update would have fixed, and used it to break into the system. I'm told that this event inspired the Alanis Morissette song, Ironic, but that "It's like the security update / when you're already hacked" didn't quite have the right rhythm to it.

You know how every superhero has some sort of traumatic childhood event that totally explains them? Like Bruce Wayne, Peter Parker, or even Fox Mulder? This was it for Ksplice---and really got Jeff thinking "Why do software updates have to be so disruptive?"

So he set out to solve the problem, in his award-winning master's thesis. The rest was a no-brainer: the need was obviously there, so he, Tim Abbott, Anders Kaseorg, and I founded Ksplice, Inc. in June of 2008.

Declaration of Independence
Ksplice, Inc.'s founding charter


A lot has stayed the same since June 2008: we're still slaving away in Cambridge, Massachusetts, working hard to bring you the finest-quality updates made only from fresh, organic free-range kernel source code. But a lot has also changed. For example, we're now up to twenty-something men and women. I've honestly lost count of the exact number (good thing I don't do payroll).

We're also a really tight-knit group, and share an interesting background: we're all humans, we all went to MIT, we all studied computer science, and we all were in MIT's student computing group, SIPB, together.

As you might have guessed, we like to geek out about, well, basically everything, and I'd be very surprised if that doesn't come out in this blog. The authorship of posts here will also rotate fairly regularly among my fellow Ksplicians (or is it Ksplicers?), and I'm definitely looking forward to their posts and your comments.

So go and add this to your Google Reader, Twitter, or whatever it is the kids are using to read blogs these days--you'll be glad you did.

~wdaher

About

Tired of rebooting to update systems? So are we -- which is why we invented Ksplice, technology that lets you update the Linux kernel without rebooting. It's currently available as part of Oracle Linux Premier Support, Fedora, and Ubuntu desktop. This blog is our place to ramble about technical topics that we (and hopefully you) think are interesting.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today