Ten years of Oracle Ksplice

August 5, 2021 | 5 minute read
Text Size 100%:

We're celebrating a milestone: ten years of zero-downtime operating system (OS) patches with Oracle Ksplice! Ksplice is a reliable, production solution that has been keeping Linux systems up to date for twice as long as other Linux solutions. Ksplice is so popular, it is used to patch more than 2.5 million servers every month. Requiring reboots dramatically delays patching and creates significant risk from exploits of known vulnerabilities. With Ksplice technology, users can keep Oracle Linux systems current with patching for both the kernel and critical userspace OS components, without reboots.

In this blog post, we'll take a look at how Ksplice has improved in the ten years since Ksplice for Oracle Linux was introduced -- and how we continue to evolve to meet the changing needs of the enterprise. We'll discuss how Ksplice technology has evolved to patch kernel assembly code and userspace, added additional architectures and cloud platforms, and contributed to vulnerability research and threat detection.

Technology advances

As the security landscape changes, so does Ksplice to keep up with more and more complex patches. These changes and techniques allow us to safely patch even more of the kernel's assembly code, even on heavily loaded systems. During 2018, we saw an uptick in the number of hardware-related vulnerabilities, and Ksplice was the only technology in the market that was able to live patch CVE-2018-3639 (Spectre v4) and CVE-2018-3620/CVE-2018-3646 (L1 terminal fault), the latter comprising thousands of lines of changes across the kernel.

The Linux kernel gains new features and optimizations over time to scale to all kinds of workloads. We actively develop Ksplice to make sure that we can give the best patching experience with every supported kernel in all configurations. This includes safe integration with DTrace probes in Oracle Linux, full support for Meltdown mitigations (including both KAISER and KPTI with no reduction in patch coverage), and support for linker optimization in modern toolchains.

Userspace patching

While many high profile and high severity vulnerabilities target the Linux kernel, our goal has always been to allow customers to stay secure while allowing them to reboot on their own schedule. To accomplish that, we announced the addition of the GNU C library (glibc) and OpenSSL patching in 2015. The C library is fundamental to almost all Linux applications, providing the core functions for memory management, networking, threading, and other essential functionality. OpenSSL is the Oracle Linux library used for SSL/TLS and many other common cryptographic functions used by security-sensitive applications such as web servers, SSH, postfix, NTP, and other network clients and servers.

While we don't see nearly as many security vulnerabilities for these libraries as we do for the kernel, it is nevertheless really important that we have the capability of patching them without downtime when a vulnerability does show up. Since we started supporting userspace libraries we have already patched some critical, high profile vulnerabilities, perhaps most notably CVE-2015-7547, a remote code execution bug in the glibc DNS resolver and CVE-2016-0800 (DROWN), a cipher downgrade attack in OpenSSL.

In 2018, we also announced the addition of Xen and QEMU to Ksplice's repertoire. This means you can have a fully patchable virtualization stack, from the hypervisor, through the Dom0 kernel, user-space, and the guests themselves.

Arm support

Enterprise customers have started to explore a variety of architectures for their workloads, and Ksplice is changing to support these new configurations. In 2018, we announced Ksplice support on Arm. This covers both the kernel and userspace components!

Porting Ksplice to Arm brought plenty of low-level technological challenges with it and also required us to overhaul our build, test, and distribution infrastructure to support the new architecture.

In May of this year, Oracle announced the general availability of Arm compute shapes in Oracle Cloud Infrastructure (OCI), including Always Free Arm instances with Ksplice support included!

Known exploit detection

It has traditionally been difficult or impossible to know if an attacker attempts to exploit a security vulnerability after it has been patched.

In April 2019, we announced a new feature exclusive to Oracle Linux users: Known exploit detection. By installing tripwires in the area of the code where a bug was fixed, we can alert the system administrators the moment we detect that somebody seems to be doing something they shouldn't -- and potentially head off a more serious security incident, as an attempt at exploiting a kernel bug typically means somebody already has unauthorized access to the machine.

Cloud offerings

In addition to on-premises Oracle Linux deployments and Oracle Linux Premier Support customers, Ksplice was also made completely free for users in OCI and Oracle Linux users in Azure.

For more information about using Ksplice in OCI, refer to this blog post by Senior Vice President, Wim Coekaerts: Oracle Ksplice for Oracle Linux in Oracle Cloud.

Vulnerability research

While Ksplice itself is primarily focused on patching security issues once they have been discovered, the Ksplice team occasionally takes an active part in looking for vulnerabilities too. Members of the Ksplice team are credited with more than 15 CVEs discovered in the last 10 years, including multiple local privilege escalation vulnerabilities. Some of these were found by fuzzing, but several were also found by code review. In fact, code review is an essential part of what we do every day; the Ksplice team has to look at and understand every single patch that gets released for a kernel we support. We cannot simply take every patch and turn it into a Ksplice update; we need to actually understand each patch to help ensure that it will be safe to apply to a running kernel and running programs. In the course of these reviews, we sometimes discover other bugs as well, which has resulted in more than 275 patches to the mainline Linux kernel in the last 10 years. Not bad for a team whose main activity is not upstream development!

Looking ahead

Ksplice already provides protection for several layers of Oracle Cloud, from deep in its network and virtualization infrastructure through its web services up to the Linux instances run by every customer. We will always keep the security of your systems as our top priority, and, as the complexity of operations continues to increase, we are focused on improving management of that security. Services like Autonomous Linux and OS Management Service in Oracle Cloud will provide tools to better manage the security of your cloud fleet, and those capabilities will be brought back to those who are using Ksplice on-premises.

On that note, if you share our vision of security and availability and you would like to make a real impact, let us know -- check out our open positions or shoot us an email at <ksplice-support_ww@oracle.com>.

Ksplice is part of Oracle Linux and included with an Oracle Linux Premier Support subscription, and is free for Oracle Cloud Infrastructure (OCI) users as well. We also continue to offer Ksplice for free for Ubuntu and Fedora desktop users and Oracle Linux users in Azure.

Oracle Linux downloads

Individual RPM packages are available on the Unbreakable Linux Network (ULN) and the Oracle Linux yum server. ISO installation images are available for download from the Oracle Linux yum server and container images are available via Oracle Container Registry, GitHub Container Registry and Docker Hub.

Oracle Linux can be downloaded, used, and distributed free of charge and all updates and errata are freely available. Customers decide which of their systems require a support subscription. This makes Oracle Linux an ideal choice for development, testing, and production systems. The customer decides which support coverage is best for each individual system while keeping all systems up to date and secure.

Resources

Vegard Nossum


Previous Post

Oracle Verrazzano Enterprise Container Platform debuts and is certified on Oracle Linux

Honglin Su | 2 min read

Next Post


Intro to Kernel and Userspace Tracing Using BCC, Part 1 of 3

Jonah Palmer | 15 min read