Oracle developers are prolific and prominent contributors to core areas of the Linux kernel. One question we are often asked is how we manage to do both upstream development and produce a stable, enterprise kernel at the same time. As part of our commitment to keeping Linux Open and Free, we’ve decided to make our upstream kernel builds public so you can see exactly how we do this and even try it out on your own systems. We believe that allowing customers to validate their applications and workloads with the latest versions of Linux is the best way to prevent “vendor lock-in” by the OS provider—while also allowing customers to confirm the ways that upstream Linux can benefit their applications.
The Oracle Linux team is excited to introduce a new way to preview the latest developments from upstream Linux and try them out using a kernel built by Oracle. We call it UEK-next. UEK-next releases are upstream Linux kernels with Oracle Linux patches for use to evaluate new features in upstream Linux, to experiment with the latest hardware support, and to validate application compatibility with the latest kernels. UEK-next is not supported for production use, however we will provide limited test and development support for these kernels to help validate applications and workloads.
Linux Upstream Continuous Integration
Inside Oracle we developed a system called Linux Upstream Continuous Integration, or LUCI, to manage our Linux kernel patches. Any patches merged into UEK (the Unbreakable Enterprise Kernel) by Oracle are required to be added into the LUCI system and continually merged and validated agains the latest upstream Linux kernel releases. LUCI serves as an early-warning system for conflicts between those patches and upstream Linux, and encourags developers to pay attention to upstream kernel development in addition to the customer- or product- specific development they may be working on.
Before we implemented LUCI, it would routinely take more than 6 months to modernize our patchsets onto a prospective new UEK target. With LUCI, that process has been reduced to weeks. With LUCI, we can detect upstream problems that may affect UEK much sooner, and can test upstream features on key Oracle workloads before committing to backporting those features.
As much as we try to do everything in upstream Linux, there are always some patches that don’t make sense for upstream—either because they aren’t the long-term solution to a problem, or because the upstream consensus has yet to converge on a solution. LUCI gives us a way of maintaining some patches out of tree, without losing our ability to rapidly rebase onto newer kernels. That’s part of the challenge of maintaining a distribution, and a reality of working with upstream development. We find this challenge absolutely worthwhile given the greater benefits we get from staying close to upstream Linux kernel development.
We also use LUCI to encourage developers to upstream their kernel patches. As long as the patch is in LUCI and not in upstream, developers are responsible for fixing and updating their patches every time an upstream change affects their code; once the code is upstreamed, then it’s automatically part of the next version of Linux and the developer can let the upstream processes take over. This is also why upstream contributions and development are important to kernel developers at Oracle, because it means they can keep their work focused on the cutting edge of Linux development.
Within Oracle, we use LUCI to build nightly Linux kernel binaries, validating that patches apply and alerting developers for potentially breaking changes. Due to the rapid nature of Linux kernel development, patches can often break 2 or 3 times in a development cycle, so it can take several weeks for changes to settle before updated patches are ready for integration.
What are you releasing?
We plan to release a new version of UEK-next a few weeks after each new upstream kernel is released—as source, binaries, and public git repositories.
Starting with the 6.8 Linux kernel, we’ll be releasing UEK-next in a developer yum channel to allow developers and end users to experiment easily with the latest upstream Linux kernels on an Oracle Linux 9 system. UEK-next kernels are not intended for production use and do not receive production support or Ksplice patching. We will review issues via GitHub, and if you are seeing compelling features in these UEK-next builds, we encourage you to reach out to us so we can help ensure those features are part of the next supported release of UEK!
Can I contribute to UEK-next?
If you have kernel patches, please contribute to upstream Linux—that’s the best and fastest way to get code into UEK-next. Patches accepted by upstream will be part of the next UEK-next build a few weeks after that kernel is released.
If you’d like to suggest alternative kernel configuration parameters for the kernel build (CONFIG_xxx) or have other requests, feel free to file an issue with your suggestion. We also accept bug reports at github.com/oracle/linux-uek/issues .
Where can I find the code?
Check out UEK-next on our Oracle Linux 9 yum repos today at yum.oracle.com/oracle-linux-9. Looking for a virtual machine environment? You can find Oracle Linux VirtualBox images here, or run a VM with UTM or QEMU.
UEK-next is intended for developers, not production use, so it’s signed with a different gpg key than our production RPMs (this will also complicate installation where secure boot is enabled). The development key should be installed before installing the kernel. These are the steps to install UEK-next:
# rpm --import https://yum.oracle.com/RPM-GPG-KEY-oracle-development # dnf config-manager --add-repo 'https://yum.oracle.com/repo/OracleLinux/OL$releasever/developer/UEKnext/$basearch' # dnf install kernel-ueknext
Sources for uek-next can be found at https://github.com/oracle/linux-uek/tree/ueknext/latest.