Announcing UEK-next for kernel 6.10

UEK-next for kernel 6.10 was posted last week, Oracle Linux’s preview of the latest development in upstream Linux. There’s been a lot of Linux news this week, from low-level discussions of SFrame at GNU Cauldron to hand-delivered pull requests, and we’re excited about the new features that are hopefully on track for 6.12!

There have been a few questions about what goes into a UEK-next release. In this blog, we’ll take a look at the work it takes to build and test UEK-next. Part of the reason for the delay in UEK-next releases is a question of priorities: we focus on our production, supported use cases first. But there’s more to it than that, and some of the complexity that goes into UEK-next may not be immediately apparent. The phases of UEK-next creation are: collecting patches, updating the base kernel, applying changes, building the code, code reviews, and testing. With the 6.10 release, we were able to detect, report and fix an number of upstream issues, but this process takes time, and that’s why I’m writing about UEK-next 6.10 even as 6.11 is available upstream.

For instructions on downloading and installing an RPM for UEK-next, check out the previous post on UEK-next.

Collecting Patches

Collecting non-upstream patches used to be an onerous process of manual review; but to support UEK-next we changed how we handle patches for UEK. Today, any patch being offered for any version of UEK must also have a commit submitted to UEK-next to ensure that patch is picked up as part of UEK-next. This also means the developer is notified as soon as an upstream change breaks their patch, so they can engage immediately with the upstream community discussion (if necessary) or fix up their patch.

Updating the Base Kernel

Updating the base kernel means bringing in the latest code from upstream Linux. There’s nothing really “UEK specific” about the process of bringing in a new kernel, but it does require that developers review and sign off on any new kernel compile options, since those options can introduce (or remove) dependencies. While there’s not much different from general upstream kernel maintenance here, we still find it important to carefully review any changed options, as those options will become part of the next production UEK build (not just UEK-next!).

Applying Patches

Now, it’s time to take the changes from Step 1, and apply them to the kernel. While we often try and notify developers as soon as their patches stop being compatible with upstream, that doesn’t necessarily mean that developers can immediately fix their code. For example, a developer may hold off checking in fixes to the UEK-next code line while upstream is still debating or iterating on the original breaking changes, so as to not have to repeatedly fix the same area.

The question of patch application has also grown in complexity over time; in the beginning, most changes were independent and could be applied or skipped without affecting other patches in the series. Today, we have more interdependent patches with chains of dependencies between those patches. For example, a change of patches may need to be applied in a specific order, first A, then B then C. If A wont apply, B and C shouldn’t be applied either, even if they “apply cleanly” as the underlying functionality won’t exist. The worst case scenario would be a developer “fixing” B and C to while A is broken. Fortunately, we’ve been able to automate this case, removing specific sections of “patch chains” when problems are detected.

Another question of patch application is simply, what order to apply the patches. For this, we start with the patch ordering in the latest (production) UEK version, followed by specific heuristics to create a repeatable ordering of patches. The important part, though, is the repeatability, so each build of UEK-next has the patches applied in a repeatable ordering, minimizing issues derived from patch ordering.

Building the Code

Once the above steps are done, we have clean merges for all the UEK-next patches. Unfortunately, there’s no guarantee that the code is going to compile. One of the most important developments underpinning UEK-next is the ability to automatically track which patches are broken.

We depend heavily on automated git-bisection tools to identify which of the patches in UEK-next is causing the issue. When an issue is detected, it sets off a whole chain of events. First, the patch and its dependencies are removed from UEK-next (so the rest of the build can continue), then the tracking bug is reopened and the developer notified, and a development environment containing the broken patches is created so the developer can log in and fix the patch.

Tracking Broken Patches

As we do this, building kernels and patching bugs, upstream updates continue to come in. So we continue building UEK-next candidates, even though some patches are still broken, and multiple times a day we’re running test builds, automatically skipping over known broken bugs and focusing on clean patches.

We broadly classify all patches into one of three categories, Maintainer, Infrastructure, and Developer patches. This also helps ensure that generic patches (ones that are required to get a UEK-like build from Mainline) are applied before any individual bug fixes (Infrastructure bugs are similar to Maintainer bugs in their broad-applicability, but are owned by individual developers). Finally, all Developer bugs are applied.

Developer bugs are further divided into their dependency chains, to ensure that multiple developers can work on their bugs if the commits are not inter-dependent.

All of these phases are backed up by automation, including the automatic generation of developer branches, so developers can concentrate on their own patchset. The “automation branches” are used by developers to fix their patch set, and also used by our internal bug scrubbing tools to validate the changes.

Code Reviews

We require code reviews even for simple changes to UEK-next patches. This means that a developer might just be adding ‘static’ to a bunch of declarations, but it still requires review by a developer. And, as you can imagine, the code review can find suggestions pertaining to the original patches (and not the incremental changes). All these things ultimately make the end product better, but can take time to do properly, even when we’re trying to keep up with the speed of upstream Linux.

Testing

Finally, we have to test UEK-next. Even though it’s technically an unsupported, technology preview of what’s coming in the next UEK, we still take testing very seriously. As part of our test process, we’ll look at functionality (ensuring things work like they’re supposed to), check for regressions (make sure known bugs haven’t crept back in), and even evaluate the overall workload performance. It’s not the same comprehensiveness that we apply to production UEK releases, but it still takes time and effort to do it right.

Some of this testing can be done in parallel with bug fixing (specifically, functional testing), but in general, we have to ensure that all the bug fixes and code reviews are complete before we can initiate testing – we don’t want to spend too long testing a kernel that we know is missing key patches.

Conclusion

If that seems like a lot of steps to get patches into UEK-next … that’s because it is! Usually, developers can skip this whole process by submitting their code to mainline (and getting it accepted). Unfortunately there are always a number of patches which aren’t accepted by upstream, but are important that we include with UEK — and publish the source for all those changes on github.com/oracle/linux-uek. Between the development work to keep patches current and the test workloads to validate those, this can add time to the release cycle for UEK-next. But the delays this introduces are more than offset by the benefits of keeping patches current against upstream, of testing and fixing kernels as soon as they are available. This is only the third public release of UEK-next, but we’ve been building and testing these internally for years and remain as focused as ever on releasing UEK-next previews!