We have an awesome new Oracle Infrastructure blog where we will be covering the Oracle Infrastructure products like, servers, engineered systems, OSes, storage and more. Subscribe to get the latest updates and news.
"When I talk to Linux administrators about the one thing they hate doing, it has to be patching.
Certainly one of the reasons is finding a time to take a system down to patch is next to impossible. That usually means having to miss your kid’s soccer game, your favorite sports team’s match on TV, or a rare night out with friends. And just how do you find time when you’re running systems in the cloud?
That’s why Oracle KSplice is such a game-changer—it allows for zero-downtime kernel updates. And that’s not even the whole story about Oracle Linux, with support for Docker, Linux Containers, OpenStack, Oracle Clusterware for High Availability (HA), and more—at no additional charge with an Oracle Linux Premier Support contract. In addition to the Red Hat Compatible Kernel (RHCK), Oracle supplies the optimized Unbreakable Enterprise Kernel (UEK), which was first developed to support highly scalable Oracle Engineered Systems. ..."
If you're unfamiliar with Ksplice, it provides zero-downtime patching (also known as “live patching") for Linux. With Ksplice, updates and errata (CVEs, etc.) can be applied to running systems without restarting applications or rebooting. This is a boon for organizations since scheduling downtime across a server farm is a significant event, and many companies using a variety of operating systems only perform patches periodically in a regular cycle. But being able to apply critical updates immediately–or remove them, if necessary–without impacting servers even under heavy load allows administrators to reduce the window of vulnerability for security issues to the bare minimum, making systems more secure. A fantastic primer on Ksplice can be found in Wim Coekaerts' blog post The Magic of Ksplice.
While Ksplice has been a great tool for applying patches without downtime, it has been focused exclusively on kernel patches. That is, until last week with the announcement of userspace patching for Ksplice.
Why is userspace patching important? Well, patching the kernel as Ksplice has done for years solves a number of issues. It lets you patch security vulnerabilities at the kernel level, it allows you to apply patches to do diagnostics, and so on. But applications rely on services available above the kernel (in “userspace”) to do many of their tasks. For example, glibc, the GNU C Library, is the standard C and C++ library used by applications on Linux. And OpenSSL provides secure networking services for applications so applications don’t have to implement these features from scratch. But patching the kernel doesn’t help with changes that need to be made in these system wide libraries.
These standard libraries are very useful because they allow the community to optimize and standardize on how common tasks are accomplished. But since they are used by so many applications, a flaw in one of these libraries creates widespread exposure. Just such an exposure happened with the Heartbleed Bug, where an issue with OpenSSL left thousands of servers at risk.
With userspace patching in Ksplice, Oracle can now provide you with the tools to patch these userspace libraries without downtime. At Oracle OpenWorld last week, we were showing attendees in the Oracle Linux, Oracle VM, and OpenStack Showcase how we can apply userspace patches for the Ghost and Heartbleed vulnerabilities, check that the systems were indeed secured, and then roll those patches back to their unsecured state, all in a couple of minutes and without stopping running applications. This is the power of the new userspace feature of Ksplice. In this first release of userspace patching, the focus is on glibc and OpenSSL since this will cover many of the security related issues customers will run into.
With the addition of userspace patching, a great tool just got even better. Ksplice can now patch the running Linux kernel and also patch userspace glibc and OpenSSL without downtime. Userspace patching is a huge development in zero-downtime patching, and another tool in the toolbox for administrators needing to cope with critical updates. This brings the magic of Ksplice from the kernel up to userspace, making your systems safer.
Userspace patching for Ksplice is available now in the enhanced Ksplice client for customers with Oracle Linux Premier Support. You can read about it in the latest Ksplice documentation and when you’re ready to try it out, you can enable it on the Unbreakable Linux Network and install it on your systems.
I'm really excited to share this week's Friday Spotlight with you. Oracle Senior Vice President of Linux and Virtualization Wim Coekaerts sat down with Director of Product Management Michele Casey and Senior Development Manager Jamie Iles to talk about Ksplice.
In the video, they go into a lot of detail about why Ksplice is a production-ready tool for keeping systems up to date. A highlight for me is the discussion featuring a real world example of an Oracle Linux 6.2 system from 2011, and how a system like that can be patched over time with all the important CVEs and security updates without a single reboot -- no rebooting literally for years. For production systems, you can just keep the system up and running, and still be up to date. Click this screenshot to head on over to Oracle Media Network to watch the video:
Happy Friday! And for those of us in the US, happy Independence Day!
Our spotlight this week is on a new paper written by Oracle's Robert Chase (if you read the Oracle Linux articles over on Oracle Technical Network, you'll be familiar with his work). This paper focuses on the use of Oracle Ksplice -- well known as a kernel patching tool -- as a diagnostic tool to help you when working with Oracle Support. It's really a great use of the technology, have a read below:
Oracle Linux provides two complimentary technologies for patching and updating the operating system.
yum for updating RPM packages. Applications and libraries are packaged and distributed in the form of RPM packages, which are collected in yum repositories. Updates are installed by downloading the packages from the yum repository and installing them locally using the RPM package manager.
It's probably worth repeating that Oracle also provides updates (errata) for free from our public-yum server - you can keep your system up to date and fully patched against security threats without the need of purchasing a support subscription. This makes Oracle Linux and ideal choice to install on both your development and production systems - it is up to you to individually choose which of these systems you want to have covered by a support subscription and at which level.
We also provide updates to the Linux operating system kernel in RPM format. However, these changes only take effect after the system has been rebooted, which can be quite disruptive in certain environments. Scheduling downtime for a reboot is never easy.
This is where Ksplice enters the picture. It is a technology that allows you to apply critical fixes to the Linux kernel at run time, without the need to reboot your system. This is a feature that is unique to Oracle Linux. The system connects to the Ksplice server to obtain the individual rebootless patches, split up by security issues (which are usually tracked by CVE numbers). You can install all of the patches in one go, or choose to install only selected patches, without any service interruption or downtime. Ksplice patches can also be removed at run-time, in case they show any any unwanted or unexpected side-effects.
Both yum and ksplice require downloading patches from a remote server, so the client system needs to be able to connect to a remote server. In many cases, connecting to an update server located on the public Internet directly is not an option, due to security policies.
In the case of yum, it's possible to create a local copy of a repository and simply point all clients to obtain their patches from there instead. There are several ways to create and manage such local repositories, and Oracle Enterprise Manager 12c Cloud Control and Ops Center both provide built-in functionality to support this. We also published a script on OTN that automates the task of downloading RPM packages from the Unbreakable Linux Network.
For Ksplice, it was already possible to set up a local server that would act as a caching proxy server for all available patches - the client systems would only have to connect to this server instead of contacting the remote Ksplice server over the Internet directly. However, this solution requires setting up a dedicated system just for this particular task, so many customers were not too happy about this solution.
The Ksplice team at Oracle now came up with an alternative solution - instead of providing the Ksplice patches as individual downloadable items, they are bundled inside an RPM package, one for each Linux kernel version we support. Any time a new ksplice patch is available, the respective RPM package will be refreshed. This way we can now deliver Ksplice patches via yum repositories, which is a well-established transport mechanism and can utilize already existing infrastructure. The process involves two steps: first you download the ksplice patch RPM using yum, then you run the local ksplice client, which has been modified to check for updates on the local file system instead of contacting the remote server. Even though you are using RPM to download the Ksplice patch bundle RPM, you still use the local ksplice client to apply the individual patches at run time.
This new Ksplice offline mode gives you the best of both worlds: being able to patch your Linux kernel at run-time without disrupting any services, while not requiring you to manage any additional infrastructure or services, or having to negotiate any exceptions to your firewall rules in order to allow your systems to contact the remote Ksplice server.
For more information about the Ksplice offline mode, please see Wim's blog post or check out the following video, which outlines the basic principles of how to apply updates to your Oracle Linux system:
With today’s RedPatch announcement, the Oracle Linux team provides the Linux community a repository containing the source for all the changes Red Hat makes to their kernel, free of charge, available from our public git site. This repository simplifies life for administrators by providing them a straightforward method for determining the type of fixes a patch contains.
As you probably know by now, we in the Oracle Linux Team are quite enthusiastic about Ksplice. Just in case you haven't heard about Ksplice yet, this technology allows you to apply security fixes and upgrades to the Linux kernel while your system is running, without having to reboot your server. Your applications remain unaffected — there is no service disruption or performance impact involved.
Ksplice was added to our Oracle Linux Premier support subscription some time ago (at no additional cost) and customers rely on it to ensure their systems are always up to date and fully patched, even if the next scheduled maintenance window is still days (or weeks) ahead.
Before, Ksplice patches were only provided for customers running Oracle Linux with the Unbreakable Enterprise Kernel. Today, we’re extending our support offering: Ksplice zero downtime kernel updates are now available for the Red Hat compatible kernel on Oracle Linux 5 and 6 as well. And if you are running Red Hat Enterprise Linux without Oracle Linux support and you’d like to try Ksplice, sign up for our 30-day free trial.
In case you missed the live webcast about "Zero Downtime Updates with Ksplice" on Sept. 20th, a replay of this session is now available from here. In this session, Monica Kumar (Senior Director, Product Marketing) is joined by Wim Coekaerts (Senior VP Linux and Virtualization Engineering) and Waseem Daher, former COO and Co-Founder of Ksplice, to provide an introduction and explain the benefits of this technology for our customers. Ksplice for Oracle Linux Premier customers is available now, the Getting Started with Ksplice document (pdf) outlines the steps required to enable it.
If you want to learn more about Ksplice and Oracle Linux, please also take a look at our Ksplice Data Sheet (pdf) and this general presentation (pdf). Waseem will also talk about this exciting technology in detail at Oracle OpenWorld, where he will be joined by Chris Mason (Director Linux Kernel Engineering) and myself in a joint presentation about new features and developments in Oracle Linux 6 and the Unbreakable Enterprise Kernel.