Wednesday Jan 29, 2014

Ksplice SNMP Plugin

The Ksplice team is happy to announce the release of an SNMP plugin for Ksplice, available today on the Unbreakable Linux Network. The plugin will let you use Oracle Enterprise Manager to monitor the status of Ksplice on all of your systems, but it will also work with any monitoring solution that is SNMP compatible.

Installation

You'll find the plugin on Ksplice channel for your distribution and architecture. For Oracle Linux 6 on x86_64 that's ol6_x86_64_ksplice. Install the plugin by running (as root):

yum install ksplice-snmp-plugin

Configuration

If you haven't set up SNMP before, you'll need to do a little bit of configuration. Included below is a sample /etc/snmp/snmpd.conf file to try out the plugin:


# Setting up permissions
# ======================
com2sec local localhost public
com2sec mynet 10.10.10.0/24 public

group local v1 local
group local v2c local
group local usm local
group mynet v1  mynet
group mynet v2c mynet
group mynet usm mynet

view all included .1 80

access mynet "" any noauth exact all none none
access local "" any noauth exact all all none

syslocation Oracle Linux 6
syscontact sysadmin <root@localhost>

# Load the plugin
# ===============
dlmod kspliceUptrack /usr/lib64/ksplice-snmp/kspliceUptrack.so

 

You'll want to replace the IP address next to com2sec with the address of your local network, or the address of your SNMP monitoring software. If you are running on a 32 bit architecture x86), replace lib64 in the dlmod path with lib. The above configuration is an example, intended only for testing purposes. For more information about configuring SNMP, check out the SNMP documentation, including the man pages for snmpd and snmpd.conf.

Examples

You can test out your configuration by using the snmpwalk command and verifying the responses. Some examples:

Displaying the installed version of Ksplice:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceVersion
KSPLICE-UPTRACK-MIB::kspliceVersion.0 = STRING: 1.2.12

Checking if a kernel has all available updates installed:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate

Displaying and comparing the kernel installed on disk with the Ksplice effective version:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceBaseKernel
KSPLICE-UPTRACK-MIB::kspliceBaseKernel.0 = STRING: 2.6.18-274.3.1.el5

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceEffectiveKernel
KSPLICE-UPTRACK-MIB::kspliceEffectiveKernel.0 = STRING: 2.6.18-274.3.1.el5

Displaying a list of all installed updates:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable

In this case, there are none. This is why the base kernel version and effective kernel version are the same, and why this kernel is out of date.

Displaying a list of updates that can be installed right now, including their description:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable
KSPLICE-UPTRACK-MIB::kspliceavailIndex.0 = INTEGER: 0
KSPLICE-UPTRACK-MIB::kspliceavailIndex.1 = INTEGER: 1
KSPLICE-UPTRACK-MIB::kspliceavailIndex.2 = INTEGER: 2
KSPLICE-UPTRACK-MIB::kspliceavailIndex.3 = INTEGER: 3
KSPLICE-UPTRACK-MIB::kspliceavailIndex.4 = INTEGER: 4
KSPLICE-UPTRACK-MIB::kspliceavailIndex.5 = INTEGER: 5
KSPLICE-UPTRACK-MIB::kspliceavailIndex.6 = INTEGER: 6
KSPLICE-UPTRACK-MIB::kspliceavailIndex.7 = INTEGER: 7
KSPLICE-UPTRACK-MIB::kspliceavailIndex.8 = INTEGER: 8
KSPLICE-UPTRACK-MIB::kspliceavailIndex.9 = INTEGER: 9
KSPLICE-UPTRACK-MIB::kspliceavailIndex.10 = INTEGER: 10
KSPLICE-UPTRACK-MIB::kspliceavailIndex.11 = INTEGER: 11
KSPLICE-UPTRACK-MIB::kspliceavailIndex.12 = INTEGER: 12
KSPLICE-UPTRACK-MIB::kspliceavailIndex.13 = INTEGER: 13
KSPLICE-UPTRACK-MIB::kspliceavailIndex.14 = INTEGER: 14
KSPLICE-UPTRACK-MIB::kspliceavailIndex.15 = INTEGER: 15
KSPLICE-UPTRACK-MIB::kspliceavailIndex.16 = INTEGER: 16
KSPLICE-UPTRACK-MIB::kspliceavailIndex.17 = INTEGER: 17
KSPLICE-UPTRACK-MIB::kspliceavailIndex.18 = INTEGER: 18
KSPLICE-UPTRACK-MIB::kspliceavailIndex.19 = INTEGER: 19
KSPLICE-UPTRACK-MIB::kspliceavailIndex.20 = INTEGER: 20
KSPLICE-UPTRACK-MIB::kspliceavailIndex.21 = INTEGER: 21
KSPLICE-UPTRACK-MIB::kspliceavailIndex.22 = INTEGER: 22
KSPLICE-UPTRACK-MIB::kspliceavailIndex.23 = INTEGER: 23
KSPLICE-UPTRACK-MIB::kspliceavailIndex.24 = INTEGER: 24
KSPLICE-UPTRACK-MIB::kspliceavailIndex.25 = INTEGER: 25
KSPLICE-UPTRACK-MIB::kspliceavailName.0 = STRING: [urvt04qt]
KSPLICE-UPTRACK-MIB::kspliceavailName.1 = STRING: [7jb2jb4r]
KSPLICE-UPTRACK-MIB::kspliceavailName.2 = STRING: [ot8lfoya]
KSPLICE-UPTRACK-MIB::kspliceavailName.3 = STRING: [f7pwmkto]
KSPLICE-UPTRACK-MIB::kspliceavailName.4 = STRING: [nxs9cwnt]
KSPLICE-UPTRACK-MIB::kspliceavailName.5 = STRING: [i8j4bdkr]
KSPLICE-UPTRACK-MIB::kspliceavailName.6 = STRING: [5jr9aom4]
KSPLICE-UPTRACK-MIB::kspliceavailName.7 = STRING: [iifdtqom]
KSPLICE-UPTRACK-MIB::kspliceavailName.8 = STRING: [6yagfyh1]
KSPLICE-UPTRACK-MIB::kspliceavailName.9 = STRING: [bqc6pn0b]
KSPLICE-UPTRACK-MIB::kspliceavailName.10 = STRING: [sy14t1rw]
KSPLICE-UPTRACK-MIB::kspliceavailName.11 = STRING: [ayo20d8s]
KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd]
KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k]
KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]
KSPLICE-UPTRACK-MIB::kspliceavailName.15 = STRING: [qsajn0ce]
KSPLICE-UPTRACK-MIB::kspliceavailName.16 = STRING: [5tx9tboo]
KSPLICE-UPTRACK-MIB::kspliceavailName.17 = STRING: [2nve5xek]
KSPLICE-UPTRACK-MIB::kspliceavailName.18 = STRING: [w7ik1ka8]
KSPLICE-UPTRACK-MIB::kspliceavailName.19 = STRING: [9ky2kan5]
KSPLICE-UPTRACK-MIB::kspliceavailName.20 = STRING: [zjr4ahvv]
KSPLICE-UPTRACK-MIB::kspliceavailName.21 = STRING: [j0mkxnwg]
KSPLICE-UPTRACK-MIB::kspliceavailName.22 = STRING: [mvu2clnk]
KSPLICE-UPTRACK-MIB::kspliceavailName.23 = STRING: [rc8yh417]
KSPLICE-UPTRACK-MIB::kspliceavailName.24 = STRING: [0zfhziax]
KSPLICE-UPTRACK-MIB::kspliceavailName.25 = STRING: [ns82h58y]
KSPLICE-UPTRACK-MIB::kspliceavailDesc.0 = STRING: Clear garbage data on the kernel stack when handling signals.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.1 = STRING: CVE-2011-1160: Information leak in tpm driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.2 = STRING: CVE-2011-1585: Authentication bypass in CIFS.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.3 = STRING: CVE-2011-2484: Denial of service in taskstats subsystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.4 = STRING: CVE-2011-2496: Local denial of service in mremap().
KSPLICE-UPTRACK-MIB::kspliceavailDesc.5 = STRING: CVE-2009-4067: Buffer overflow in Auerswald usb driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.6 = STRING: CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.7 = STRING: CVE-2011-2699: Predictable IPv6 fragment identification numbers.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.8 = STRING: CVE-2011-2723: Remote denial of service vulnerability in gro.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.9 = STRING: CVE-2011-2942: Regression in bridged ethernet devices.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.10 = STRING: CVE-2011-1833: Information disclosure in eCryptfs.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.11 = STRING: CVE-2011-3191: Memory corruption in CIFSFindNext.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.12 = STRING: CVE-2011-3209: Denial of Service in clock implementation.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.13 = STRING: CVE-2011-3188: Weak TCP sequence number generation.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.14 = STRING: CVE-2011-3363: Remote denial of service in cifs_mount.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.15 = STRING: CVE-2011-4110: Null pointer dereference in key subsystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.16 = STRING: CVE-2011-1162: Information leak in TPM driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.17 = STRING: CVE-2011-2494: Information leak in task/process statistics.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.18 = STRING: CVE-2011-2203: Null pointer dereference mounting HFS filesystems.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.19 = STRING: CVE-2011-4077: Buffer overflow in xfs_readlink.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.20 = STRING: CVE-2011-4132: Denial of service in Journaling Block Device layer.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.21 = STRING: CVE-2011-4330: Buffer overflow in HFS file name translation logic.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.22 = STRING: CVE-2011-4324: Denial of service vulnerability in NFSv4.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.23 = STRING: CVE-2011-4325: Denial of service in NFS direct-io.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.24 = STRING: CVE-2011-4348: Socking locking race in SCTP.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.25 = STRING: CVE-2011-1020, CVE-2011-3637: Information leak, DoS in /proc.

And here's what happens after you run uptrack-upgrade -y, using Ksplice to fully upgrade your kernel:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: uptodate

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.0 = INTEGER: 0
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.1 = INTEGER: 1
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.2 = INTEGER: 2
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.3 = INTEGER: 3
[ . . . ]

The plugin displays that the kernel is now up-to-date.

SNMP and Enterprise Manager

Once the plugin is up and running, you can monitor your system using Oracle Enterprise Manager. Specifically, you can create an SNMP Adapter to allow Enterprise Manager Management Agents to query the status of Ksplice on each system with the plugin installed. Check out our documentation on SNMP support in Enterprise Manager to get started, including section 22.6, "About Metric Extensions".

This plugin represents the first step in greater functionality between Ksplice and Enterprise Manager and we're excited about what is coming up. If you have any questions about the plugin or suggestions for future development, leave a comment below or drop us a line at ksplice-support_ww@oracle.com.

Thursday Aug 08, 2013

CVE-2013-2224: Denial of service in sendmsg().

In September 2012, CVE-2012-3552 was reported which could allow an attacker to corrupt slab memory which could lead to a denial-of-service or possible privilege escalation depending on the target machine workload.  This bug had originally been fixed in the mainline kernel in April 2011 and was a fairly large patch for a security fix.  The RedHat backport for this fix introduced a new bug which has been assigned CVE-2013-2224 which again could allow for a denial-of-service or possible privilege escalation.  Rack911 & Tortoiselabs created a reproducer in June 2013 which would allow an unprivileged user to cause a denial-of-service.

RedHat have not yet released a kernel with this CVE fixed, but CentOS have released a custom kernel with the vendor fix for CentOS 6.

We have just released a Ksplice update to address this issue for releases 5 and 6 of Oracle Linux, RedHat Enterprise Linux, CentOS and Scientific Linux.  We recommend that all users of Ksplice on these distributions install this zero-downtime update.

[Read More]

Tuesday Oct 18, 2011

The Ksplice Pointer Challenge

Back when Ksplice was just a research project at MIT, we all spent a lot of time around the student computing group, SIPB. While there, several precocious undergrads kept talking about how excited they were to take 6.828, MIT's operating systems class.

"You really need to understand pointers for this class," we cautioned them. "Reread K&R Chapter 5, again." Of course, they insisted that they understood pointers and didn't need to. So we devised a test.

Ladies and gentlemen, I hereby do officially present the Ksplice Pointer Challenge, to be answered without the use of a computer:

What does this program print?

#include <stdio.h>
int main() {
  int x[5];
  printf("%p\n", x);
  printf("%p\n", x+1);
  printf("%p\n", &x);
  printf("%p\n", &x+1);
  return 0;
}

This looks simple, but it captures a surprising amount of complexity. Let's break it down.

To make this concrete, let's assume that x is stored at address 0x7fffdfbf7f00 (this is a 64-bit system). We've hidden each entry so that you have to click to make it appear -- I'd encourage you to think about what the line should output, before revealing the answer.

printf("%p\n", x);
What will this print?

Well, x is an array, right? You're no stranger to array syntax: x[i] accesses the ith element of x.

If we search back in the depths of our memory, we remember that x[i] can be rewritten as *(x+i). For that to work, x must be the memory location of the start of the array.

Result: printf("%p\n", x) prints 0x7fffdfbf7f00. Alright.

printf("%p\n", x+1);
What will this print?

So, x is 0x7fffdfbf7f00, and therefore x+1 should be 0x7fffdfbf7f01, right?

You're not fooled. You remember that  in C, pointer arithmetic is special and magical. If you have a pointer p to an int, p+1 actually adds sizeof(int)to p. It turns out that we need this behavior if *(x+i) is properly going to end up pointing us at the right place -- we need to move over enough to pass one entry in the array. In this case, sizeof(int) is 4.

Result: printf("%p\n", x) prints 0x7fffdfbf7f04. So far so good.

printf("%p\n", &x);
What will this print?

Well, let's see. & basically means "the address of", so this is like asking "Where does x live in memory?" We answered that earlier, didn't we? x lives at 0x7fffdfbf7f00, so that's what this should print.

But hang on a second... if &x is 0x7fffdfbf7f00, that means that it lives at 0x7fffdfbf7f00. But when we print x, we also get 0x7fffdfbf7f00. So x == &x.

How can that possibly work? If x is a pointer that lives at 0x7fffdfbf7f00, and also points to 0x7fffdfbf7f00, where is the actual array stored?

Thinking about that, I draw a picture like this:


That can't be right.

So what's really going on here? Well, first off, anyone who ever told you that a pointer and an array were the same thing was lying to you. That's our fallacy here. If x were a pointer, and x == &x, then yes, we would have something like the picture above. But x isn't a pointer -- x is an array!

And it turns out that in certain situations, an array can automatically "decay" into a pointer. Into &x[0], to be precise. That's what's going on in examples 1 and 2. But not here. So &x does indeed print the address of x.

Result: printf("%p\n", &x) prints 0x7fffdfbf7f00.

Aside: what is the type of &x[0]? Well, x[0] is an int, so &x[0] is "pointer to int". That feels right.

printf("%p\n", &x+1);
What will this print?

Ok, now for the coup de grace. x may be an array, but &x is definitely a pointer. So what's &x+1?

First, another aside: what is the type of &x? Well... &x is a pointer to an array of 5 ints. How would you declare something like that?

Let's fire up cdecl and find out:

cdecl> declare y as array 5 of int;
int y[5]
cdecl> declare y as pointer to array 5 of int;
int (*y)[5]

Confusing syntax, but it works:
int (*y)[5] = &x; compiles without error and works the way you'd expect.

But back to the question at hand. Pointer arithmetic tells us that &x+1 is going to be the address of x + sizeof(x). What's sizeof(x)? Well, it's an array of 5 ints. On this system, each int is 4 bytes, so it should be 20 bytes, or 0x14.

Result &x+1 prints 0x7fffdfbf7f14.

And thus concludes the Ksplice pointer challenge.

What's the takeaway? Arrays are not pointers (though they sometimes pretend to be!). More generally, C is subtle. Oh, and 6.828 students, if you're having trouble with Lab 5, it's probably because of a bug in your Lab 2.


P.S. If you're interested in hacking on low-level systems at a place where your backwards-and-forwards knowledge of C semantics will be more than just an awesome party trick, we're looking to hire kernel hackers for the Ksplice team.

We're based in beautiful Cambridge, Mass., though working remotely is definitely an option. Send me an email at waseem.daher@oracle.com with a resume and/or a github link if you're interested!

Tuesday Feb 15, 2011

Happy Birthday Ksplice Uptrack!

One year ago, we announced the general availability of Ksplice Uptrack, a subscription service for rebootless kernel updates on Linux. Since then, a lot has happened!

Adoption

Over 600 companies have deployed Ksplice Uptrack on more than 100,000 production systems, on all 7 continents (Antarctica was the last hold-out). More than 2 million rebootless updates have been installed!

Ksplice at the South Pole

You see the greatest system administration time and cost savings when you use Ksplice Uptrack on all of your machines, and we've designed and priced Uptrack with this large-scale usage in mind. We've been very happy to see that our customers share this view: most of our customers either rolled out Ksplice across the board upon sign-up or have substantially expanded their usage after first purchasing for a particular environment.

We've introduced several features this year to make managing large Uptrack installations easier:

  • autoinstall: the Uptrack client can be configured to install rebootless updates on its own as they become available, making kernel updates fully automatic. Autoinstall is now our most popular configuration.
  • the Uptrack API: programmatically query the state of your machines through our RESTful API.
  • Nagios plugins: easily integrate Uptrack monitoring into your existing Nagios infrastructure with plugins that monitor for out of date, inactive, and unsupported machines.

Supported kernels

We continue to expand the distributions and flavors we support. You can now use Ksplice Uptrack on (deep breath) RHEL 4 & 5, CentOS 4 & 5 (and CentOS Plus), Debian 5 & 6, Ubuntu 8.04, 9.10, 10.04, & 10.10, Virtuozzo 3 & 4, OpenVZ for EL5 and Debian, CloudLinux 5, and Fedora 13 & 14. Within these distributions, we continue to support more flavors and older kernels; our launch may have been last year, but we support kernels back to 2007 and earlier!

You've always been able to use Ksplice in virtualized environments like VMWare, Xen, and Virtuozzo, This year, we've given you even more virtualization options by adding support for virtualization kernels like Debian OpenVZ and Debian Xen. Starting with Ubuntu Maverick, we support all Ubuntu kernel flavors, including the Maverick private cloud/EC2 kernel.

Thanks to some changes at Amazon and Rackspace, you can now even use Ksplice Uptrack on stock Linux kernels in Amazon EC2 and Rackspace Cloud.

As always, you can roll out a free trial on any of these distributions, for an unlimited number of systems.

This year, we also added Fedora Desktop to our free version options, so now you can use Ksplice Uptrack on both Ubuntu Desktop and Fedora Desktop completely for free.

Reboots saved

Did you know that between RHEL 4 and 5, Red Hat released 22 new kernels this past year?

chart: reboots on RHEL this past year

Without Ksplice, that's coordination and downtime for 22 reboots, including the hat trick of font-size: 125%; ">3 kernels in 3 weeks for RHEL 5. Gartner estimates that 90% of exploited systems are exploited using known, patched vulnerabilities, so if you're not rebooting and you're not using Ksplice, you are putting your servers (and your customers) at risk.

Looking forward

What do you want to see from Ksplice this year? What features can we add to help you deploy and monitor your Uptrack installations? Tell us what you want, and we'll do our best to deliver!

~jesstess

Tuesday Mar 09, 2010

How to quadruple your productivity with an army of student interns

Startup companies are always hunting for ways to accomplish as much as possible with what they have available. Last December we realized that we had a growing queue of important engineering projects outside of our core technology that our team didn't have the time to finish anytime soon. To make matters worse, we wanted the projects completed right away, in time for our planned product launch in early February.

So what did we do? The logical solution, of course. We quadrupled the size of our company's engineering team for one month using paid student interns.

20 men and women posing for a group photo

The Ksplice interns, ca. January 2010.

Now, if you happen to know Fred Brooks, please don't tell him what we did. He managed the Windows Vista of the 1960s---IBM's OS/360, a software project of unprecedented size, complexity, and lateness---and wrote up the resulting hard-earned lessons in The Mythical Man-Month, which everyone managing software projects should read. The big one is Brooks's Law: "adding manpower to a late software project makes it later". Oops.

Brooks's observation usually holds. New people take time to get up to speed on any system---both their own time, and the time of your experienced people mentoring them. They also add communication costs that grow quadratically with the size of the team.

Fortunately, Ksplice benefits from a bit of an engineering reality distortion field (our very product is supposed to be technologically impossible) and, with the right techniques, quadrupling our engineering team's size for one month worked out great. Every intern was responsible for one of the company's top unaddressed priorities for the month, and every intern successfully completed their project.

So, how do you quadruple the size of your engineering team in one month and still keep everyone productive?

Ten people working at computers in a room
At work in the Ksplice engineering office.

  • Tolerate a little crowding. It took a little creativity to suddenly find a dozen new workspaces in our two-room office. Fortunately, we've found that a room can always fit one more person---and by induction, you can fit as many as you need. (All those years we spent proving math theorems came in handy after all.) Seating everyone close to each other has an important advantage, too: when lots of people on your team have just started, it's handy for them to work right next to the mentors who are answering their questions and helping them ramp up on the learning curve of the organization. With the right team, the crowding can also create an energetic office environment that makes people love to come in to work. (Sometimes it gets in the way of concentration, though---that's when I put on a good pair of headphones.)
  • Locate next to a deep pool of hackers. OK, so we're a bit spoiled by being headquartered a few blocks away from the Massachusetts Institute of Technology. At MIT, January is set aside for students to pursue projects outside of the curriculum---perfect for hiring an intern army. Many other institutions have either a similar "January term", or a program for students to spend time working in industry during the term.
  • Know who the best people are and only hire them. Ksplice was born four years ago at SIPB, MIT's student computing group. When a group of students run computing services thousands of people rely on, and spend hours each week discussing, dreaming, collaborating, and learning from each other on computer systems---some of them get really good at it. Even better, everyone sees everyone else in action and knows exactly what it's like to work with them. Investing some time into getting involved with technical communities makes it possible to hire people based on personal experience with them and their work, which is so much better than hiring based on resumes and interviews. Companies like Google and Red Hat have known for years that being involved in the open source community can provide an excellent source of vetted job candidates.
  • Pay well. In some industries, "intern" means "unpaid"---but computer science students have plenty of options, and you want to be able to hire the best people. We looked at pay rates for jobs on campus, and pegged our rate to the high end of those.
  • Divide tasks to be as loosely-coupled as possible. Our internship program would never have worked if we had assigned a dozen new people to hack on our kernel code---the training time and communication costs that drive Brooks' Law would have swallowed their efforts whole. Fortunately, like any growing business, we had a constellation of tasks that lie around the edges of our core technology: infrastructure upgrades, additional layers of QA, business analytics, and new features in the management side of our product. These had manageable technical interfaces our existing software, so our interns were able to become productive with minimal ramp-up and rely on relatively little communication to get their projects done.
  • Design your intern projects in advance. A key challenge when scaling up your engineering team quickly is making sure that the interfaces are all well designed and the new projects will meet the company's needs. So we spent a good deal of time getting these designs together before the interns started. We also allocated plenty of our core engineers' time for code reviews and other feedback with our interns in order to make sure their work would be maintainable after they left.
Have you achieved more in one month than anyone thought should be possible? How did you do it?

[Edited to make clear our interns were paid and to say more about how we designed their projects for high-quality output.]

~price

Thursday Mar 04, 2010

/etc/init.d/blog start

Welcome to the Ksplice blog!

I struggled for quite some time coming up with a grand opening for this blog. Something profound like "We the people, in order to form more perfect computing..." or "The history of the world is the history of rebooting". But that proved too hard, so instead, I want to take a moment to briefly tell the Ksplice story, in the hopes of providing a bit more insight into who we are and what we're all about.

Ksplice was born on a beautiful summer day back in 2006. Crickets chirping, little kids splashing each other by the pool, ice cream trucks, etc. Ksplice's CEO, Jeff Arnold, then a student, was administering some very popular servers at MIT. A software update came out in the middle of the week, and, as any prudent system administrator would, he scheduled an outage window for the following Sunday at 2am, so as to disrupt as few users as possible, and went along his merry way.

Swimming pool
Where I wish I had been, Summer 2006
Of course, something much more sinister was lying in wait. Someone took advantage of the very bug that the update would have fixed, and used it to break into the system. I'm told that this event inspired the Alanis Morissette song, Ironic, but that "It's like the security update / when you're already hacked" didn't quite have the right rhythm to it.

You know how every superhero has some sort of traumatic childhood event that totally explains them? Like Bruce Wayne, Peter Parker, or even Fox Mulder? This was it for Ksplice---and really got Jeff thinking "Why do software updates have to be so disruptive?"

So he set out to solve the problem, in his award-winning master's thesis. The rest was a no-brainer: the need was obviously there, so he, Tim Abbott, Anders Kaseorg, and I founded Ksplice, Inc. in June of 2008.

Declaration of Independence
Ksplice, Inc.'s founding charter


A lot has stayed the same since June 2008: we're still slaving away in Cambridge, Massachusetts, working hard to bring you the finest-quality updates made only from fresh, organic free-range kernel source code. But a lot has also changed. For example, we're now up to twenty-something men and women. I've honestly lost count of the exact number (good thing I don't do payroll).

We're also a really tight-knit group, and share an interesting background: we're all humans, we all went to MIT, we all studied computer science, and we all were in MIT's student computing group, SIPB, together.

As you might have guessed, we like to geek out about, well, basically everything, and I'd be very surprised if that doesn't come out in this blog. The authorship of posts here will also rotate fairly regularly among my fellow Ksplicians (or is it Ksplicers?), and I'm definitely looking forward to their posts and your comments.

So go and add this to your Google Reader, Twitter, or whatever it is the kids are using to read blogs these days--you'll be glad you did.

~wdaher

About

Tired of rebooting to update systems? So are we -- which is why we invented Ksplice, technology that lets you update the Linux kernel without rebooting. It's currently available as part of Oracle Linux Premier Support, Fedora, and Ubuntu desktop. This blog is our place to ramble about technical topics that we (and hopefully you) think are interesting.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today