X

News, tips, partners, and perspectives for the Oracle Solaris operating system

Moving Oracle Solaris to LP64 bit by bit

Alan Coopersmith
Senior Principal Software Engineer

A few people have noticed a trend in the Oracle Solaris 11 update releases of delivering more and more Solaris commands as 64-bit binaries, so I figured it was time to write a detailed explanation to answer some of the questions and help prepare users & developers for further change, as it now becomes more critical to deliver shared objects in both 32-bit (ILP32) and 64-bit (LP64) versions.

I’d like to thank Ali, Gary, Margot, Rod, and Jeff for their feedback on this post, and most especially to Sharon for helping rework it to get to the most important bits first.

For the developers and administrators who are familiar with Solaris history, I’ll start with info about what you should be doing to make sure you’re ready for increased use of 64-bit software in Solaris. You can refer to the section that describes issues that arise when converting binaries — I cover examples from the X Window System packages for Oracle Solaris that I’ve done the conversion work for, and discuss LP64 conversion work by other Solaris engineers.

For those who need more background, see the sections about LP64 history in Solaris and the Application Binary Interface (ABI) differences between 32-bit and 64-bit binaries.

What do you need to do?

You need to do what you have always done — ensure you have both 32-bit and 64-bit versions of any shared objects. This requirement is being highlighted because the consequences of not providing both binary versions is becoming more disruptive in each subsequent Solaris release.

Development requirements

If you develop software for Solaris, the requirement is to provide both 32-bit and 64-bit versions of any shared objects you provide for other software to use, whether as libraries to link their programs against or as loadable objects for frameworks such as localized input methods, PAM service modules, custom crypt(3C) password hashes, or dozens of other shared object uses in Solaris.

Additionally, you should make sure all of your software is 64-bit clean, even if you still support it on 32-bit systems. While Solaris 10 still has more than 6 years left of active support, eventually you’ll move on to only supporting Solaris 11 and later and be able to go 64-bit only as well. The Solaris 64-bit Developer’s Guide and Solaris Studio Guide to Converting Applications for a 64-Bit Environment can help here. Oracle’s ISV support team is also looking to provide more assistance in future versions of the Oracle Solaris Preflight Applications Checker tool to help find possible issues for you.

User requirements

Administrators and users should verify that the software that they install and use provides both 32-bit & 64-bit shared objects. If they are not provided, contact the developers to provide both binary versions.

You should also keep an eye on the End of Features (EOF) Planned for Future Releases of Oracle Solaris page to see if something you may still need is being removed because it was determined not to be useful any more when it was reviewed for LP64 conversion. The page is updated regularly as new items make their way through the internal obsolescence review processes. If something appears there that would cause you major grief, let Solaris development know through your sales or support channels, so we can supply a better transition or replacement plan where possible.

And of course, if you find bugs in the Solaris 64-bit converted software, or find that you need a 64-bit version of a particular Solaris library or shared object that’s not already available, file bugs via Oracle Support or the Oracle Partner Network for Solaris.

Delivery of X commands as LP64 binaries

Because Solaris 11 no longer requires programs run on a 32-bit kernel, and the minimum supported system has 64 times as much RAM as the first Ultra 1, Oracle Solaris can now ship programs directly as 64-bit binaries, which better equips them to run on modern sized data sets, while utilizing the full capabilities of today’s hardware, and have started doing so.

For instance, before Solaris 11 shipped, I switched the default build flags for the X Window System programs in Solaris to 64-bit (with a few exceptions). Solaris has long shipped all the non-obsolete public libraries for X as both 32-bit and 64-bit, and the upstream open source versions of this software were made 64-bit clean long ago, originally by DEC for their Alpha workstations, and maintained since then by the BSD & Linux platforms that delivered 64-bit only distributions instead of multilib models. For the most part, this was just an implementation detail for the X programs - their functionality didn’t change, nor did their interfaces.

One case with a visible impact from becoming LP64-only was the Xorg server, which uses dlopen() to load shared object drivers for the specific hardware in use. Solaris 10 8/07 moved from Xorg 6.9 to 7.2 and started delivering Xorg as LP64 binaries for SPARC & x64, since video cards would soon have more VRAM than a 32-bit X server could access. This was also the first delivery of Xorg for SPARC, and did not need to support 32-bit SPARC platforms, so on SPARC Xorg was LP64 from the beginning. On x86, since Solaris was still supporting 32-bit platforms, the 64-bit Xorg was added alongside the existing 32-bit version. As of the 64-bit only release of Solaris 11, we could drop the 32-bit version of Xorg in Solaris, but because Solaris wasn’t the only source of the loadable driver modules (for instance, nvidia & VirtualBox both provided drivers for their graphics), we ensured that 64-bit versions were available, before announcing the end of support for 32-bit driver modules.

One less visible impact was in xdm, the old style login GUI for X, which Solaris still ships, though most Solaris systems use the more modern GNOME display manager (gdm) instead. In order to authenticate users, xdm uses the PAM framework, which allows administrators to configure a variety of login methods, such as Kerberos or SmartCards. Administrators can also install additional PAM methods to work with authentication systems Solaris doesn’t have built-in support for, or additional pluggable crypt modules to handle other password hashing methods. While PAM has supported 64-bit modules since Solaris 7, and the crypt framework has supported 64-bit modules since it was introduced in Solaris 9, most programs calling PAM and the crypt framework have been 32-bit. Installing only the 32-bit version of a crypt or PAM module thus worked most of the time. However, now that xdm is 64-bit, a 32-bit-only module will generate failure messages for users trying to login via xdm, because the system won’t find a 64-bit module to load. While xdm and PAM show that a non-existent binary can prevent system login, other less prominent 64-bit shared objects are going to be required over time, so providers of shared objects need to ensure they are installing both 32-bit & 64-bit versions of their software going forward.

Some users of custom input methods for different languages may also notice that their input methods are not available in 64-bit programs, since input methods are similarly provided as shared objects that are loaded via dlopen() and thus also have to exist in the same ABI variants as the calling programs.

Conversion of Solaris commands to LP64 binaries

This effort isn’t limited to X11 software — engineers in all areas of Solaris are evaluating the various programs and determining what needs to be done. They have two tasks - to ensure the program itself is 64-bit clean, and to ensure that the surrounding ecosystem (such as the PAM modules example above) is 64-bit ready. In some cases, they’ve found software Solaris doesn’t need to ship any longer, like the gettable tool for maintaining Internet host tables in the days before DNS, and are publishing End of Support notices for them. But for most cases, they’re working to deliver 64-bit versions into Solaris as time allows.

The results of this work are already visible — the number of LP64 programs in /usr/bin and /usr/sbin in the full Oracle Solaris package repositories has climbed each release:

Release 32-bit 64-bit total
Solaris 11.0 1707 (92%) 144 (8%) 1851
Solaris 11.1 1723 (92%) 150 (8%) 1873
Solaris 11.2 1652 (86%) 271 (14%) 1923
(These numbers only count ELF binaries, not python programs, shell scripts, etc.)

In Solaris 11.0, X11 programs provided the bulk of the LP64 programs, but there were some from other subsystems, such as gdb, emacs, and the NSS crypto commands. Solaris 11.1 added LP64 crypto commands, including digest, decrypt, elfsign, pktool, and tpmadm; as well as other commands like top & gzip. Solaris 11.2 added a number of LP64 GNU programs, including the GNU coreutils and groff. New tools added in 11.2 were 64-bit in their first delivery, such as mlocate, the Intel GPU tools, and the jsl JavaScript Lint package. The bzip2 compression program was made LP64 in 11.2 at the request of a customer who uses it on large files and wanted the extra performance on x64.

Solaris 11.2 also adds Java 8 packages, alongside Java 7, and the now deprecated Java 6 packages. Java 8 for Solaris is 64-bit-only, dropping the 32-bit binary option found in previous Java releases. As noted under the Removal of 32-bit Solaris item in the Features Removed From JDK8 list, the Java plugin for web browsers was also removed from Java 8 for Solaris, because a 64-bit Java plugin cannot be run in the 32-bit Firefox browser.

Solaris Data Model History

Solaris 11.2 represents the latest step in a 20 year journey.

The original Solaris 2.x ABI used a data model referred to as ILP32, in which the size of the C language types “int”, “long”, and pointers are all 32-bit numbers. This data model matched the SPARC & x86 CPUs available at the time.

In 1995, Sun introduced its first SPARC v9 CPU’s, the UltraSPARC I, which offered 64-bit integers and addresses. Solaris 7 followed, bringing a second ABI to Solaris, using the LP64 model, in which “int” remained 32-bits wide, but “long” and pointer sizes doubled to 64-bits.

This affected both the kernel and user space code, and Solaris 7 delivered both 32-bit (ILP32) and 64-bit (LP64) kernels for UltraSPARC systems. The Solaris kernel implementation only allowed running 64-bit user space processes if a 64-bit kernel was loaded, but 32-bit user space software could be used with either a 32-bit or 64-bit kernel.

For user-space programs, the class (32 or 64-bit) of libraries must match the program that links to them. In order to support both 32 and 64-bit programs, it was necessary to provide both 32 and 64-bit versions of libraries. To preserve binary compatibility with existing 32-bit software, the libraries in directories such as /usr/lib were left as the 32-bit versions, and the 64-bit versions were added in a new sparcv9 subdirectory. On modern Linux systems, this approach is now called “multilib.”

Because the UltraSPARC CPUs did not impose any significant performance penalty when running existing 32-bit code on a 64-bit CPU, Sun continued to ship 32-bit user-space programs in the ILP32 ABI so that the same binary could be used on both 32-bit-only and 64-bit-capable CPUs, reducing development, testing, and support costs. It also reduced by half the memory requirements for pointers and long ints (thus more easily fitting them in the CPU caches) in programs which wouldn’t benefit from the larger sizes, an important consideration in systems with only 32MB of RAM. For the small number of programs that had to run 64-bit to be able to read 64-bit kernel structures or debug other 64-bit binaries, Solaris shipped both 32-bit & 64-bit versions, with isaexec used to execute the version matching the kernel in use.

On the x86 side, nearly a decade later AMD’s first AMD64 CPUs provided similar hardware support, and Solaris 10 introduced a matching LP64 ABI for x86 platforms in 2005, with the 64-bit libraries delivered in amd64 subdirectories. Even though 32-bit binaries did not run as fast as fully 64-bit binaries, Solaris followed the same model of providing mostly ILP32 programs to get the release to market faster; to save on development, test, and support costs; and to be consistent with Solaris on SPARC platforms.

In these transitional phases, 32-bit & 64-bit software and hardware coexisted. For SPARC, support for 32-bit-only CPUs was phased out in Solaris 10, when support for the last pre-sun4u platforms was dropped. Support for the 32-bit kernel was dropped at the same time, because all remaining supported hardware could run the 64-bit kernel. For x86, support for 32-bit-only CPUs was dropped in Solaris 11.

Therefore, as of the shipping of Oracle Solaris 11 in November 2011, the supported set of platforms have 64-bit kernels that can run 32-bit or 64-bit user space binaries.

Other Differences Between 32-bit & 64-bit ABIs

While the size of the types is the defining difference between the two ABI models, the opportunity to introduce a fresh new ABI after learning of the mistakes and limitations of the old ABI was hard to resist, and other changes were made as well. Significant differences include:

Additionally, since not all 64 bits in pointers are needed to address current memory sizes, unused ones can be used for additional tasks, such as some of the new features coming in the SPARC M7 CPU.

Some other platforms followed a different strategy. For example, Linux introduced “x32”, a new 32-bit ABI, and is considering a proposal for year-2038-safe ABIs. Engineers at Sun long ago debated a “large time” extension to the 32-bit ABI like the large file interfaces, but decided to concentrate efforts on LP64 instead. Because Solaris is not trying to maintain 32-bit kernel support for embedded devices, that is not a problem we have to solve as we move forward. The result should be a simpler system, which is always a benefit for developers and ISVs.

We don’t know yet when we’ll finish this journey, but hopefully we’ll get there before the industry starts converting software to run on CPUs with 128-bit addressing.


Disclaimer: The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services