X

News, tips, partners, and perspectives for the Oracle Linux operating system and upstream Linux kernel work

5 things you may not know about Ksplice

Ksplice is a cool technology and I wanted to share a few things that you might not know about it along with a few tips on how to get started if you aren't already using it.

1. New patching advances

As the security landscape evolves, so does Ksplice to keep up with more and more complex patches.  These new changes and techniques allow Ksplice to safely patch even more of the kernel entry assembly code, even on heavily loaded systems.  To date, Ksplice is the only technology that has been able to live patch the CVE-2018-3639 (Spectre v4) and CVE-2018-3620+CVE-2018-3646 (L1 terminal fault), the latter comprising of thousands of lines of changes across the kernel.

The Linux kernel continues to advance over time gaining new features and optimizations to scale to all kinds of workloads. Oracle Ksplice continues to advance in kind. The Ksplice team is actively developing Ksplice, making sure that we can give the best patching experience with every supported kernel in all configurations.  This includes safe integration with DTrace probes in Oracle UEKR4 and UEKR5, full support for Meltdown mitigations including both KAISER and KPTI with no reduction in patch coverage, and support for linker optimization in modern Fedora toolchains.

We have optimized some of the Ksplice core to minimize the period that the system is paused during the safety checks to ensure that we can scale to larger SMP systems with no visibility to the running workload.

These enhancements allow us to make sure that we are patching all of the issues that you care about on a running system.  With recent developments in speculative side channel attacks, we have seen an increase in number and complexity of patches to some of the lowest levels of the kernel that have resulted in some new patching techniques for Ksplice.

2. A dizzying number of supported kernels

Ksplice supports a wide range of kernels, from Oracle Linux 5 2.6.18 (32-bit, 32-bit PAE, 64-bit and Xen paravirt) to the latest Fedora 28 4.17 64-bit kernels.  At any one time, Ksplice supports around 5,000 binary kernels.  In the extreme case of a wide-reaching vulnerability affecting all kernels, Ksplice releases updates for all of those in short order.  Today, Ksplice supports kernels almost 8 years old with the oldest kernels having over 700 unique fixes applied through Ksplice - that's a lot of reboots saved.

When Ksplice updates are created, we take fixes from the most recently released kernel in a series and then iteratively apply those fixes to all older kernels in that series as far as they are applicable.  In some of the older series this means a lot of versions - at the time of writing, Oracle UEKR2 has 118 distinct source releases with each requiring new patches backporting.  Ksplice will take each of the new fixes that are applicable to already running systems and iteratively backport to all of those older releases where appropriate.

You can check if your kernel is supported by Ksplice with our inspector which will helpfully show you a list of fixes that have been applied with Ksplice and you could fix today without any downtime.

3. User-space + Xen

Since Oracle acquired Ksplice in 2011, we have continued to invest heavily in the technology and added several noteworthy new features that other Linux distributions have no competitive offering for.  In 2015, Oracle introduced Ksplice patching for user-space on Oracle Linux 6 and 7 for key components - glibc and OpenSSL.  glibc is fundamental to almost all Linux applications, providing the core functions for memory management, networking, threading and many other essentials.  OpenSSL is the Oracle Linux library used for SSL/TLS and many other common cryptographic functions seeing use in many security sensitive applications such as web servers, SSH, postfix, NTP and many other network clients and servers.

When a vulnerability is found in one of these core libraries, a new RPM is created and can be installed on systems with newly executed processes using the patched version.  However, already running applications will continue to use the vulnerable code and it can be extremely hard to even determine what applications are still using the old libraries and then schedule those to be restarted without any customer visible downtime.  Patching these vulnerabilities with Ksplice means no application downtime or reboots with all of the same deployment options that you are used to with Ksplice kernel updates.  Ksplice has patched some critical, high profile vulnerabilities since we started supporting these libraries including CVE-2015-7547, a remote code execution bug in the glibc DNS resolver and CVE-2016-0800 (DROWN), a cipher downgrade in OpenSSL.

With OVM 3.4.5, Ksplice can now patch the Xen hypervisor and user-space components such as xenstored, libxenctrl and qemu.  This means you can have a full live patchable virtualization stack, from the hypervisor, through the Dom0 kernel+user-space and the guests themselves, something that no other Linux vendor can offer.

4. OCI

Ksplice is just as important in a cloud environment as an on-premise environment and we've made it incredibly easy to get started with in Oracle Cloud Infrastructure IaaS.  OCI Oracle Linux images come with an Oracle Linux Premier Support entitlement and have Ksplice pre-installed with no registration required.  Simply create an instance and you're one "uptrack-upgrade" away from having the latest security patches applied to your kernel with zero downtime.  For legacy or custom images, Ksplice can be installed with a simple script, again without registration. Simply run the following commands in your instance:

# wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc # sh install-uptrack-oc --autoinstall

and the system will start automatically installing Ksplice kernel updates without any further interaction and no downtime.

Ksplice isn't just available inside OCI tenancies though. Ksplice powers it.  The same technology you use inside your instances is also used to proactively patch OCI without any disruptive downtime, keeping everything running securely and stably.

5. Safety

One of the key principles behind Ksplice is that security and safety come first.  This means delivering the right patches quickly and applying them to the running system without any visible side-effects.

Doing so means that we need to handle a variety of edge cases and unexpected setups. Ksplice is the only live patching system that fully covers these cases.  We'll look at a couple of these here, but as always, the devil is in the detail and there are a lot of details in live patching!

Firstly, Ksplice performs integrity checks - we want to make sure that the code your are running on your system matches what we expect so that a patch doesn't get applied to incorrect code and either make things worse or fail to close the vulnerability.  A mismatch sounds unlikely, but there are a number of things that could cause it to happen:

  • You could be running modules from a different kernel or compatible module provided by a hardware vendor.
  • You could have another application that has modified the running code such as antivirus or intrusion detection, or even a root kit or virus

Ksplice handles these conditions automatically, checking every byte of the compilation units that we want to patch making sure that they match exactly.  If a mismatch is found then we'll safely abort patching, explaining why.

Secondly, we make sure that we're not replacing anything that is in use.  We employ conservative checks here doing full stack walks. Ksplice makes sure that not only is a function to be patched not currently being called, but additionally no local function pointers or data pointers that could call the wrong version of code are present.  Simple frame pointer based stack walks do cover these cases. Simple walks might work most of the time, but failure to make thorough safety checks could result in a crash or even worse, a new security vulnerability.

Lastly, Ksplice isn't just in-memory patching.  The Linux kernel can be changed at runtime by loading new modules either on explicit request by the user or automatically when doing things like mounting filesystems, opening a network socket, hot plugging hardware, or using a new cryptographic algorithm.  Ksplice handles these unloaded modules gracefully, providing on-disk patched modules and arranging for the patched to be loaded rather than the old vulnerable versions.  Without this, it would be possible to load vulnerable module code and leaving a system unstable or open to exploitation.

Conclusions

Ksplice offers unparalleled functionality and safety, allowing system administrators to take control of patching in all deployments from on-premise to OCI.  By leveraging Ksplice in your environment you can avoid unplanned reboots, and rapidly patch against the latest vulnerabilities with minimal configuration and maintenance.  If you aren't already using Ksplice, why not give it a go?  Oracle Linux instances in OCI come preconfigured with Ksplice, and for all other uses, please visit the Ksplice website to learn how to get started in a few minutes.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.