Thursday Jan 02, 2014

Book Review: “Oracle Solaris 11: First Look”

I saw a tweet this weekend from Packt Publishing (@packtpub) offering their entire catalog of ebooks for $5 apiece until Friday, January 3, 2014, so I decided to check out their selection.

One of the books I found there was Oracle Solaris 11: First Look by long-time Solaris admin Phil Brown, whose website bolthole.com was a staple of the Solaris x86 mailing list a decade ago, and whose advocacy of Solaris x86 helped save it from permanent cancellation. I first met Phil when we were undergrads at U. C. Berkeley, where we served as sysadmin staff on the student-run cluster of Unix systems together, and we’ve kept in touch via the Solaris & OpenSolaris communities we both found ourselves in later.

Since it was only $5, I picked it up to see if it was worth recommending to others. When not on sale, the ebook is only $15, since it’s not a large book - the PDF is 168 pages, the epub on my iPad was 199 pages - and in both forms that includes an index of about 25 pages. That also made it a quick enough read that I could get through in an afternoon, skimming over the examples and reference materials.

This book is intended to get existing Solaris admins up to speed quickly on Solaris 11 - it’s not going to introduce the basics of system administration, but will tell you what commands to run now. If you don’t know what routers, subnets, or tunnels are, you probably want a more introductory book - if you know what they are, and need to know what to run instead of ifconfig or editing /etc/hostname.e1000g0 to configure them on Solaris 11, this book will help.

Phil’s biases as a long time server admin are obvious in some sections, such as the introduction to NWAM, the Network AutoMagic feature, which he suggests “from a server sysadmin perspective, it might perhaps be better named "Never Wake A Monster"” though he admits on a laptop it can be useful to adjust to networks in different locations. There’s also a few areas where you can tell the book was written before Solaris 11.1 was out and didn’t get updated for the latest changes.

As the author of the classic pkg-get tool for installing Solaris SVR4 packages from network repositories, he has plenty to say about the new IPS packaging system in Solaris 11 as well, providing some useful tips on finding packages and setting up local repositories, though he does discourage use of many of the more advanced pkg subcommands that can help admins take more control over exactly what gets installed and updated on their systems.

Overall it’s a decent aide, and something I may refer to in the future, as I don’t do system administration that often these days, and often need a refresher, especially when old habits no longer work. It’s more detailed in many sections that the official Transitioning From Oracle Solaris 10 to Oracle Solaris 11.1 manual, which mainly points off to the other Solaris 11 manuals for details; but concentrated only on the areas a typical system administrator will be configuring, not developer or end-user visible changes. I’d recommend it to experienced admins looking for a hands on guide for dealing with Solaris 11 systems, but it’s not the right level for those trying to plan a migration or learn Solaris administration from scratch.

As always, the above is solely my personal opinion, not an official Oracle corporate position or endorsement.

Sunday Nov 11, 2012

Solaris 11.1 changes building of code past the point of __NORETURN

While Solaris 11.1 was under development, we started seeing some errors in the builds of the upstream X.Org git master sources, such as:

"Display.c", line 65: Function has no return statement : x_io_error_handler
"hostx.c", line 341: Function has no return statement : x_io_error_handler
from functions that were defined to match a specific callback definition that declared them as returning an int if they did return, but these were calling exit() instead of returning so hadn't listed a return value.

These had been generating warnings for years which we'd been ignoring, but X.Org has made enough progress in cleaning up code for compiler warnings and static analysis issues lately, that the community turned up the default error levels, including the gcc flag -Werror=return-type and the equivalent Solaris Studio cc flags -v -errwarn=E_FUNC_HAS_NO_RETURN_STMT, so now these became errors that stopped the build. Yet on Solaris, gcc built this code fine, while Studio errored out. Investigation showed this was due to the Solaris headers, which during Solaris 10 development added a number of annotations to the headers when gcc was being used for the amd64 kernel bringup before the Studio amd64 port was ready. Since Studio did not support the inline form of these annotations at the time, but instead used #pragma for them, the definitions were only present for gcc.

To resolve this, I fixed both sides of the problem, so that it would work for building new X.Org sources on older Solaris releases or with older Studio compilers, as well as fixing the general problem before it broke more software building on Solaris.

To the X.Org sources, I added the traditional Studio #pragma does_not_return to recognize that functions like exit() don't ever return, in patches such as this Xserver patch. Adding a dummy return statement was ruled out as that introduced unreachable code errors from compilers and analyzers that correctly realized you couldn't reach that code after a return statement.

And on the Solaris 11.1 side, I updated the annotation definitions in <sys/ccompile.h> to enable for Studio 12.0 and later compilers the annotations already existing in a number of system headers for functions like exit() and abort(). If you look in that file you'll see the annotations we currently use, though the forms there haven't gone through review to become a Committed interface, so may change in the future.

Actually getting this integrated into Solaris though took a bit more work than just editing one header file. Our ELF binary build comparison tool, wsdiff, actually showed a large number of differences in the resulting binaries due to the compiler using this information for branch prediction, code path analysis, and other possible optimizations, so after comparing enough of the disassembly output to be comfortable with the changes, we also made sure to get this in early enough in the release cycle so that it would get plenty of test exposure before the release.

It also required updating quite a bit of code to avoid introducing new lint or compiler warnings or errors, and people building applications on top of Solaris 11.1 and later may need to make similar changes if they want to keep their build logs similarly clean.

Previously, if you had a function that was declared with a non-void return type, lint and cc would warn if you didn't return a value, even if you called a function like exit() or panic() that ended execution. For instance:

#include <stdlib.h>

int
callback(int status)
{
    if (status == 0)
        return status;
    exit(status);
}
would previously require a never executed return 0; after the exit() to avoid lint warning "function falls off bottom without returning value".

Now the compiler & lint will both issue "statement not reached" warnings for a return 0; after the final exit(), allowing (or in some cases, requiring) it to be removed. However, if there is no return statement anywhere in the function, lint will warn that you've declared a function returning a value that never does so, suggesting you can declare it as void. Unfortunately, if your function signature is required to match a certain form, such as in a callback, you not be able to do so, and will need to add a /* LINTED */ to the end of the function.

If you need your code to build on both a newer and an older release, then you will either need to #ifdef these unreachable statements, or, to keep your sources common across releases, add to your sources the corresponding #pragma recognized by both current and older compiler versions, such as:

#pragma does_not_return(exit)
#pragma does_not_return(panic) 
Hopefully this little extra work is paid for by the compilers & code analyzers being able to better understand your code paths, giving you better optimizations and more accurate errors & warning messages.

Sunday Oct 28, 2012

Solaris 11.1: Changes to included FOSS packages

Besides the documentation changes I mentioned last time, another place you can see Solaris 11.1 changes before upgrading is in the online package repository, now that the 11.1 packages have been published to http://pkg.oracle.com/solaris/release/, as the “0.175.1.0.0.24.2” branch. (Oracle Solaris Package Versioning explains what each field in that version string means.)

When you’re ready to upgrade to the packages from either this repo, or the support repository, you’ll want to first read How to Update to Oracle Solaris 11.1 Using the Image Packaging System by Pete Dennis, as there are a couple issues you will need to be aware of to do that upgrade, several of which are due to changes in the Free and Open Source Software (FOSS) packages included with Solaris, as I’ll explain in a bit.

Solaris 11 can update more readily than Solaris 10

In the Solaris 10 and older update models, the way the updates were built constrained what changes we could make in those releases. To change an existing SVR4 package in those releases, we created a Solaris Patch, which applied to a given version of the SVR4 package and replaced, added or deleted files in it. These patches were released via the support websites (originally SunSolve, now My Oracle Support) for applying to existing Solaris 10 installations, and were also merged into the install images for the next Solaris 10 update release. (This Solaris Patches blog post from Gerry Haskins dives deeper into that subject.)

Some of the restrictions of this model were that package refactoring, changes to package dependencies, and even just changing the package version number, were difficult to do in this hybrid patch/OS update model. For instance, when Solaris 10 first shipped, it had the Xorg server from X11R6.8. Over the first couple years of update releases we were able to keep it up to date by replacing, adding, & removing files as necessary, taking it all the way up to Xorg server release 1.3 (new version numbering begun after the X11R7 split of the X11 tree into separate modules gave each module its own version). But if you run pkginfo on the SUNWxorg-server package, you’ll see it still displayed a version number of 6.8, confusing users as to which version was actually included.

We stopped upgrading the Xorg server releases in Solaris 10 after 1.3, as later versions added new dependencies, such as HAL, D-Bus, and libpciaccess, which were very difficult to manage in this patching model. (We later got libpciaccess to work, but HAL & D-Bus would have been much harder due to the greater dependency tree underneath those.) Similarly, every time the GNOME team looked into upgrading Solaris 10 past GNOME 2.6, they found these constraints made it so difficult it wasn’t worthwhile, and eventually GNOME’s dependencies had changed enough it was completely infeasible. Fortunately, this worked out for both the X11 & GNOME teams, with our management making the business decision to concentrate on the “Nevada” branch for desktop users - first as Solaris Express Desktop Edition, and later as OpenSolaris, so we didn’t have to fight to try to make the package updates fit into these tight constraints.

Meanwhile, the team designing the new packaging system for Solaris 11 was seeing us struggle with these problems, and making this much easier to manage for both the development teams and our users was one of their big goals for the IPS design they were working on. Now that we’ve reached the first update release to Solaris 11, we can start to see the fruits of their labors, with more FOSS updates in 11.1 than we had in many Solaris 10 update releases, keeping software more up to date with the upstream communities.

Of course, just because we can more easily update now, doesn’t always mean we should or will do so, it just removes the package system limitations from forcing the decision for us. So while we’ve upgraded the X Window System in the 11.1 release from X11R7.6 to 7.7, the Solaris GNOME team decided it was not the right time to try to make the jump from GNOME 2 to GNOME 3, though they did update some individual components of the desktop, especially those with security fixes like Firefox. In other parts of the system, decisions as to what to update were prioritized based on how they affected other projects, or what customer requests we’d gotten for them.

So with all that background in place, what packages did we actually update or add between Solaris 11.0 and 11.1?

Core OS Functionality

One of the FOSS changes with the biggest impact in this release is the upgrade from Grub Legacy (0.97) to Grub 2 (1.99) for the x64 platform boot loader. This is the cause of one of the upgrade quirks, since to go from Solaris 11.0 to 11.1 on x64 systems, you first need to update the Boot Environment tools (such as beadm) to a new version that can handle boot environments that use the Grub2 boot loader. System administrators can find the details they need to know about the new Grub in the Administering the GRand Unified Bootloader chapter of the Booting and Shutting Down Oracle Solaris 11.1 Systems guide. This change was necessary to be able to support new hardware coming into the x64 marketplace, including systems using UEFI firmware or booting off disk drives larger than 2 terabytes.

For both platforms, Solaris 11.1 adds rsyslog as an optional alternative to the traditional syslogd, and OpenSCAP for checking security configuration settings are compliant with site policies.

Note that the support repo actually has newer versions of BIND & fetchmail than the 11.1 release, as some late breaking critical fixes came through from the community upstream releases after the Solaris 11.1 release was frozen, and made their way to the support repository. These are responsible for the other big upgrade quirk in this release, in which to upgrade a system which already installed those versions from the support repo, you need to either wait for those packages to make their way to the 11.1 branch of the support repo, or follow the steps in the aforementioned upgrade walkthrough to let the package system know it's okay to temporarily downgrade those.

Developer Stack

While Solaris 11.0 included Python 2.7, many of the bundled python modules weren’t packaged for it yet, limiting its usability. For 11.1, many more of the python modules include 2.7 versions (enough that I filtered them out of the below table, but you can always search on the package repository server for them.

For other language runtimes and development tools, 11.1 expands the use of IPS mediated links to choose which version of a package is the default when the packages are designed to allow multiple versions to install side by side.

For instance, in Solaris 11.0, GNU automake 1.9 and 1.10 were provided, and developers had to run them as either automake-1.9 or automake-1.10. In Solaris 11.1, when automake 1.11 was added, also added was a /usr/bin/automake mediated link, which points to the automake-1.11 program by default, but can be changed to another version by running the pkg set-mediator command.

Mediated links were also used for the Java runtime & development kits in 11.1, changing the default versions to the Java 7 releases (the 1.7.0.x package versions), while allowing admins to switch links such as /usr/bin/javac back to Java 6 if they need to for their site, to deal with Java 7 compatibility or other issues, without having to update each usage to use the full versioned /usr/jdk/jdk1.6.0_35/bin/javac paths for every invocation.

Desktop Stack

As I mentioned before, we upgraded from X11R7.6 to X11R7.7, since a pleasant coincidence made the X.Org release dates line up nicely with our feature & code freeze dates for this release. (Or perhaps it wasn’t so coincidental, after all, one of the benefits of being the person making the release is being able to decide what schedule is most convenient for you, and this one worked well for me.) For the table below, I’ve skipped listing the packages in which we use the X11 “katamari” version for the Solaris package version (mainly packages combining elements of multiple upstream modules with independent version numbers), since they just all changed from 7.6 to 7.7.

In the graphics drivers, we worked with Intel to update the Intel Integrated Graphics Processor support to support 3D graphics and kernel mode setting on the Ivy Bridge chipsets, and updated Nvidia’s non-FOSS graphics driver from 280.13 to 295.20.

Higher up in the desktop stack, PulseAudio was added for audio support, and liblouis for Braille support, and the GNOME applications were built to use them.

The Mozilla applications, Firefox & Thunderbird moved to the current Extended Support Release (ESR) versions, 10.x for each, to bring up-to-date security fixes without having to be on Mozilla’s agressive 6 week feature cycle release train.

Detailed list of changes

This table shows most of the changes to the FOSS packages between Solaris 11.0 and 11.1. As noted above, some were excluded for clarity, or to reduce noise and duplication. All the FOSS packages which didn't change the version number in their packaging info are not included, even if they had updates to fix bugs, security holes, or add support for new hardware or new features of Solaris.

Package11.011.1
archiver/unrar 3.8.5 4.1.4
audio/sox 14.3.0 14.3.2
backup/rdiff-backup 1.2.1 1.3.3
communication/im/pidgin 2.10.0 2.10.5
compress/gzip 1.3.5 1.4
compress/xz not included 5.0.1
database/sqlite-3 3.7.6.3 3.7.11
desktop/remote-desktop/tigervnc 1.0.90 1.1.0
desktop/window-manager/xcompmgr 1.1.5 1.1.6
desktop/xscreensaver 5.12 5.15
developer/build/autoconf 2.63 2.68
developer/build/autoconf/xorg-macros 1.15.0 1.17
developer/build/automake-111 not included 1.11.2
developer/build/cmake 2.6.2 2.8.6
developer/build/gnu-make 3.81 3.82
developer/build/imake 1.0.4 1.0.5
developer/build/libtool 1.5.22 2.4.2
developer/build/makedepend 1.0.3 1.0.4
developer/documentation-tool/doxygen 1.5.7.1 1.7.6.1
developer/gnu-binutils 2.19 2.21.1
developer/java/jdepend not included 2.9
developer/java/jdk-6 1.6.0.26 1.6.0.35
developer/java/jdk-7 1.7.0.0 1.7.0.7
developer/java/jpackage-utils not included 1.7.5
developer/java/junit 4.5 4.10
developer/lexer/jflex not included 1.4.1
developer/parser/byaccj not included 1.14
developer/parser/java_cup not included 0.10
developer/quilt 0.47 0.60
developer/versioning/git 1.7.3.2 1.7.9.2
developer/versioning/mercurial 1.8.4 2.2.1
developer/versioning/subversion 1.6.16 1.7.5
diagnostic/constype 1.0.3 1.0.4
diagnostic/nmap 5.21 5.51
diagnostic/scanpci 0.12.1 0.13.1
diagnostic/wireshark 1.4.8 1.8.2
diagnostic/xload 1.1.0 1.1.1
editor/gnu-emacs 23.1 23.4
editor/vim 7.3.254 7.3.600
file/lndir 1.0.2 1.0.3
image/editor/bitmap 1.0.5 1.0.6
image/gnuplot 4.4.0 4.6.0
image/library/libexif 0.6.19 0.6.21
image/library/libpng 1.4.8 1.4.11
image/library/librsvg 2.26.3 2.34.1
image/xcursorgen 1.0.4 1.0.5
library/audio/pulseaudio not included 1.1
library/cacao 2.3.0.0 2.3.1.0
library/expat 2.0.1 2.1.0
library/gc 7.1 7.2
library/graphics/pixman 0.22.0 0.24.4
library/guile 1.8.4 1.8.6
library/java/javadb 10.5.3.0 10.6.2.1
library/java/subversion 1.6.16 1.7.5
library/json-c not included 0.9
library/libedit not included 3.0
library/libee not included 0.3.2
library/libestr not included 0.1.2
library/libevent 1.3.5 1.4.14.2
library/liblouis not included 2.1.1
library/liblouisxml not included 2.1.0
library/libtecla 1.6.0 1.6.1
library/libtool/libltdl 1.5.22 2.4.2
library/nspr 4.8.8 4.8.9
library/openldap 2.4.25 2.4.30
library/pcre 7.8 8.21
library/perl-5/subversion 1.6.16 1.7.5
library/python-2/jsonrpclib not included 0.1.3
library/python-2/lxml 2.1.2 2.3.3
library/python-2/nose not included 1.1.2
library/python-2/pyopenssl not included 0.11
library/python-2/subversion 1.6.16 1.7.5
library/python-2/tkinter-26 2.6.4 2.6.8
library/python-2/tkinter-27 2.7.1 2.7.3
library/security/nss 4.12.10 4.13.1
library/security/openssl 1.0.0.5 (1.0.0e) 1.0.0.10 (1.0.0j)
mail/thunderbird 6.0 10.0.6
network/dns/bind 9.6.3.4.3 9.6.3.7.2
package/pkgbuild not included 1.3.104
print/filter/enscript not included 1.6.4
print/filter/gutenprint 5.2.4 5.2.7
print/lp/filter/foomatic-rip 3.0.2 4.0.15
runtime/java/jre-6 1.6.0.26 1.6.0.35
runtime/java/jre-7 1.7.0.0 1.7.0.7
runtime/perl-512 5.12.3 5.12.4
runtime/python-26 2.6.4 2.6.8
runtime/python-27 2.7.1 2.7.3
runtime/ruby-18 1.8.7.334 1.8.7.357
runtime/tcl-8/tcl-sqlite-3 3.7.6.3 3.7.11
security/compliance/openscap not included 0.8.1
security/nss-utilities 4.12.10 4.13.1
security/sudo 1.8.1.2 1.8.4.5
service/network/dhcp/isc-dhcp 4.1 4.1.0.6
service/network/dns/bind 9.6.3.4.3 9.6.3.7.2
service/network/ftp (ProFTPD) 1.3.3.0.5 1.3.3.0.7
service/network/samba 3.5.10 3.6.6
shell/conflict 0.2004.9.1 0.2010.6.27
shell/pipe-viewer 1.1.4 1.2.0
shell/zsh 4.3.12 4.3.17
system/boot/grub 0.97 1.99
system/font/truetype/liberation 1.4 1.7.2
system/library/freetype-2 2.4.6 2.4.9
system/library/libnet 1.1.2.1 1.1.5
system/management/cim/pegasus 2.9.1 2.11.0
system/management/ipmitool 1.8.10 1.8.11
system/management/wbem/wbemcli 1.3.7 1.3.9.1
system/network/routing/quagga 0.99.8 0.99.19
system/rsyslog not included 6.2.0
terminal/luit 1.1.0 1.1.1
text/convmv 1.14 1.15
text/gawk 3.1.5 3.1.8
text/gnu-grep 2.5.4 2.10
web/browser/firefox 6.0.2 10.0.6
web/browser/links 1.0 1.0.3
web/java-servlet/tomcat 6.0.33 6.0.35
web/php-53 not included 5.3.14
web/php-53/extension/php-apc not included 3.1.9
web/php-53/extension/php-idn not included 0.2.0
web/php-53/extension/php-memcache not included 3.0.6
web/php-53/extension/php-mysql not included 5.3.14
web/php-53/extension/php-pear not included 5.3.14
web/php-53/extension/php-suhosin not included 0.9.33
web/php-53/extension/php-tcpwrap not included 1.1.3
web/php-53/extension/php-xdebug not included 2.2.0
web/php-common not included 11.1
web/proxy/squid 3.1.8 3.1.18
web/server/apache-22 2.2.20 2.2.22
web/server/apache-22/module/apache-sed 2.2.20 2.2.22
web/server/apache-22/module/apache-wsgi not included 3.3
x11/diagnostic/xev 1.1.0 1.2.0
x11/diagnostic/xscope 1.3 1.3.1
x11/documentation/xorg-docs 1.6 1.7
x11/keyboard/xkbcomp 1.2.3 1.2.4
x11/library/libdmx 1.1.1 1.1.2
x11/library/libdrm 2.4.25 2.4.32
x11/library/libfontenc 1.1.0 1.1.1
x11/library/libfs 1.0.3 1.0.4
x11/library/libice 1.0.7 1.0.8
x11/library/libsm 1.2.0 1.2.1
x11/library/libx11 1.4.4 1.5.0
x11/library/libxau 1.0.6 1.0.7
x11/library/libxcb 1.7 1.8.1
x11/library/libxcursor 1.1.12 1.1.13
x11/library/libxdmcp 1.1.0 1.1.1
x11/library/libxext 1.3.0 1.3.1
x11/library/libxfixes 4.0.5 5.0
x11/library/libxfont 1.4.4 1.4.5
x11/library/libxft 2.2.0 2.3.1
x11/library/libxi 1.4.3 1.6.1
x11/library/libxinerama 1.1.1 1.1.2
x11/library/libxkbfile 1.0.7 1.0.8
x11/library/libxmu 1.1.0 1.1.1
x11/library/libxmuu 1.1.0 1.1.1
x11/library/libxpm 3.5.9 3.5.10
x11/library/libxrender 0.9.6 0.9.7
x11/library/libxres 1.0.5 1.0.6
x11/library/libxscrnsaver 1.2.1 1.2.2
x11/library/libxtst 1.2.0 1.2.1
x11/library/libxv 1.0.6 1.0.7
x11/library/libxvmc 1.0.6 1.0.7
x11/library/libxxf86vm 1.1.1 1.1.2
x11/library/mesa 7.10.2 7.11.2
x11/library/toolkit/libxaw7 1.0.9 1.0.11
x11/library/toolkit/libxt 1.0.9 1.1.3
x11/library/xtrans 1.2.6 1.2.7
x11/oclock 1.0.2 1.0.3
x11/server/xdmx 1.10.3 1.12.2
x11/server/xephyr 1.10.3 1.12.2
x11/server/xorg 1.10.3 1.12.2
x11/server/xorg/driver/xorg-input-keyboard 1.6.0 1.6.1
x11/server/xorg/driver/xorg-input-mouse 1.7.1 1.7.2
x11/server/xorg/driver/xorg-input-synaptics 1.4.1 1.6.2
x11/server/xorg/driver/xorg-input-vmmouse 12.7.0 12.8.0
x11/server/xorg/driver/xorg-video-ast 0.91.10 0.93.10
x11/server/xorg/driver/xorg-video-ati 6.14.1 6.14.4
x11/server/xorg/driver/xorg-video-cirrus 1.3.2 1.4.0
x11/server/xorg/driver/xorg-video-dummy 0.3.4 0.3.5
x11/server/xorg/driver/xorg-video-intel 2.10.0 2.18.0
x11/server/xorg/driver/xorg-video-mach64 6.9.0 6.9.1
x11/server/xorg/driver/xorg-video-mga 1.4.13 1.5.0
x11/server/xorg/driver/xorg-video-openchrome 0.2.904 0.2.905
x11/server/xorg/driver/xorg-video-r128 6.8.1 6.8.2
x11/server/xorg/driver/xorg-video-trident 1.3.4 1.3.5
x11/server/xorg/driver/xorg-video-vesa 2.3.0 2.3.1
x11/server/xorg/driver/xorg-video-vmware 11.0.3 12.0.2
x11/server/xserver-common 1.10.3 1.12.2
x11/server/xvfb 1.10.3 1.12.2
x11/server/xvnc 1.0.90 1.1.0
x11/session/sessreg 1.0.6 1.0.7
x11/session/xauth 1.0.6 1.0.7
x11/session/xinit 1.3.1 1.3.2
x11/transset 0.9.1 1.0.0
x11/trusted/trusted-xorg 1.10.3 1.12.2
x11/x11-window-dump 1.0.4 1.0.5
x11/xclipboard 1.1.1 1.1.2
x11/xclock 1.0.5 1.0.6
x11/xfd 1.1.0 1.1.1
x11/xfontsel 1.0.3 1.0.4
x11/xfs 1.1.1 1.1.2

P.S. To get the version numbers for this table, I ran a quick perl script over the output from:

% pkg contents -H -r -t depend -a type=incorporate -o fmri \
  `pkg contents -H -r -t depend -a type=incorporate -o fmri entire@0.5.11,5.11-0.175.1.0.0.24` \
  | sort >> /tmp/11.1
% pkg contents -H -r -t depend -a type=incorporate -o fmri \
  `pkg contents -H -r -t depend -a type=incorporate -o fmri entire@0.5.11,5.11-0.175.0.0.0.2` \
  | sort >> /tmp/11.0

Thursday Oct 25, 2012

Documentation Changes in Solaris 11.1

One of the first places you can see Solaris 11.1 changes are in the docs, which have now been posted in the Solaris 11.1 Library on docs.oracle.com. I spent a good deal of time reviewing documentation for this release, and thought some would be interesting to blog about, but didn't review all the changes (not by a long shot), and am not going to cover all the changes here, so there's plenty left for you to discover on your own.

Just comparing the Solaris 11.1 Library list of docs against the Solaris 11 list will show a lot of reorganization and refactoring of the doc set, especially in the system administration guides. Hopefully the new break down will make it easier to get straight to the sections you need when a task is at hand.

Packaging System

Unfortunately, the excellent in-depth guide for how to build packages for the new Image Packaging System (IPS) in Solaris 11 wasn't done in time to make the initial Solaris 11 doc set. An interim version was published shortly after release, in PDF form on the OTN IPS page. For Solaris 11.1 it was included in the doc set, as Packaging and Delivering Software With the Image Packaging System in Oracle Solaris 11.1, so should be easier to find, and easier to share links to specific pages the HTML version.

Beyond just how to build a package, it includes details on how Solaris is packaged, and how package updates work, which may be useful to all system administrators who deal with Solaris 11 upgrades & installations. The Adding and Updating Oracle Solaris 11.1 Software Packages was also extended, including new sections on Relaxing Version Constraints Specified by Incorporations and Locking Packages to a Specified Version that may be of interest to those who want to keep the Solaris 11 versions of certain packages when they upgrade, such as the couple of packages that had functionality removed by an (unusual for an update release) End of Feature process in the 11.1 release.

Also added in this release is a document containing the lists of all the packages in each of the major package groups in Solaris 11.1 (solaris-desktop, solaris-large-server, and solaris-small-server). While you can simply get the contents of those groups from the package repository, either via the web interface or the pkg command line, the documentation puts them in handy tables for easier side-by-side comparison, or viewing the lists before you've installed the system to pick which one you want to initially install.

X Window System

We've not had good X11 coverage in the online Solaris docs in a while, mostly relying on the man pages, and upstream X.Org docs. In this release, we've integrated some X coverage into the Solaris 11.1 Desktop Adminstrator's Guide, including sections on installing fonts for fontconfig or legacy X11 clients, X server configuration, and setting up remote access via X11 or VNC. Of course we continue to work on improving the docs, including a lot of contributions to the upstream docs all OS'es share (more about that another time).

Security

One of the things Oracle likes to do for its products is to publish security guides for administrators & developers to know how to build systems that meet their security needs. For Solaris, we started this with Solaris 11, providing a guide for sysadmins to find where the security relevant configuration options were documented. The Solaris 11.1 Security Guidelines extend this to cover new security features, such as Address Space Layout Randomization (ASLR) and Read-Only Zones, as well as adding additional guidelines for existing features, such as how to limit the size of tmpfs filesystems, to avoid users driving the system into swap thrashing situations.

For developers, the corresponding document is the Developer's Guide to Oracle Solaris 11 Security, which has been the source for years for documentation of security-relevant Solaris API's such as PAM, GSS-API, and the Solaris Cryptographic Framework. For Solaris 11.1, a new appendix was added to start providing Secure Coding Guidelines for Developers, leveraging the CERT Secure Coding Standards and OWASP guidelines to provide the base recommendations for common programming languages and their standard API's. Solaris specific secure programming guidance was added via links to other documentation in the product doc set.

In parallel, we updated the Solaris C Libary Functions security considerations list with details of Solaris 11 enhancements such as FD_CLOEXEC flags, additional *at() functions, and new stdio functions such as asprintf() and getline(). A number of code examples throughout the Solaris 11.1 doc set were updated to follow these recommendations, changing unbounded strcpy() calls to strlcpy(), sprintf() to snprintf(), etc. so that developers following our examples start out with safer code. The Writing Device Drivers guide even had the appendix updated to list which of these utility functions, like snprintf() and strlcpy(), are now available via the Kernel DDI.

Little Things

Of course all the big new features got documented, and some major efforts were put into refactoring and renovation, but there were also a lot of smaller things that got fixed as well in the nearly a year between the Solaris 11 and 11.1 doc releases - again too many to list here, but a random sampling of the ones I know about & found interesting or useful:

Wednesday Nov 09, 2011

S11 X11: ye olde window system in today's new operating system

Today's the big release for Oracle Solaris 11, after 7 years of development. For me, the Solaris 11 release comes a little more than 11 years after I joined the X11 engineering team at what was then Sun, and finishes off some projects that were started all the way back then.

For instance, when I joined the X team, Sun was finishing off the removal of the old OpenWindows desktop, and we kept getting questions asking about the rest of the stuff being shipped in /usr/openwin, the directory that held both the OpenLook applications and the X Window System software. I wrote up an ARC case at the time to move the X software to /usr/X11, but there were various issues and higher priority work, so we didn't end up starting that move until near the end of the Solaris 10 development cycle several years later. Solaris 10 thus had a mix of the recently added Xorg server and related code delivered in /usr/X11, while most of the existing bits from Sun's proprietary fork of X11R6 were still in /usr/openwin.

During Solaris 11 development, we finished that move, and then jumped again, moving the programs directly into /usr/bin, following the general Solaris 11 strategy of using /usr/bin for most of the programs shipped with the OS, and using other directories, such as /usr/gnu/bin, /usr/xpg4/bin, /usr/sunos/bin, and /usr/ucb for conflicting alternate implementations of the programs shipped in /usr/bin, no longer as a way to segregate out various subsystems to allow the OS to better fit onto the 105Mb hard disks that shipped with Sun workstations back when /usr/openwin was created. However, if for some reason you wanted to build your own set of X binaries, you could put them in /usr/X11R7 (as I do for testing builds of the upstream git master repos), and then put that first in your $PATH, so nothing is really lost here.

The other major project that was started during Solaris 10 development and finished for Solaris 11 was replacing that old proprietary fork of X11R6, including the Xsun server, with the modernized, modularized, open source X11R7.* code base from the new X.Org, including the Xorg server. The final result, included in this Solaris 11 release, is based mostly on the X11R7.6 release, including recent additions such as the XCB API I blogged about last year, though we did include newer versions of modules that had upstream releases since the X11R7.6 katamari, such as Xorg server version 1.10.3.

That said, we do still apply some local patches, configuration options, and other changes, for things from just fitting into the Solaris man page style or adding support for Trusted Extensions labeled desktops. You can see all of those changes in our source repository, which is searchable and browsable via OpenGrok on src.opensolaris.org (or via hgweb on community mirrors such as openindiana.org) and available for anonymous hg cloning as well. That xnv-clone tree is now frozen, a permanent snapshot of the Solaris 11 sources, while we've created a new x-s11-update-clone tree for the Solaris 11 update releases now being developed to follow on from here.

Naturally, when your OS has 7 years between major release cycles, the hardware environment you run on greatly changes in the meantime as well, and as the layer that handles the graphics hardware, there have been changes due to that. Most of the SPARC graphics devices that were supported in Solaris 10 aren't any more, because the platforms they ran in are no longer supported - we still ship a couple SPARC drivers that are supported, the efb driver for the Sun XVR-50, XVR-100, and XVR-300 cards based on the ATI Radeon chipsets, and the astfb driver for the AST2100 remote Keyboard/Video/Mouse/Storage (rKVMS) chipset in the server ILOM devices. On the x86 side, the EOL of 32-bit platforms let us clear out a lot of the older x86 video device drivers for chipsets and cards you wouldn't find in x64 systems - of course, there's still many supported there, due to the wider variety of graphics hardware found in the x64 world, and even some recent updates, such as the addition of Kernel Mode Setting (KMS) support for Intel graphics up through the Sandy Bridge generation.

For those who followed the development as it happened, either via watching our open source code releases or using one of the many development builds and interim releases such as the various Solaris Express trains, much of this is old news to you. For those who didn't, or who want a refresher on the details, you can see last year's summary in my X11 changes in the 2010.11 release blog post. Once again, the detailed change logs for the X11 packages are available, though unfortunately, all the links in them to the bug reports are now broken, so browsing the hg history log is probably more informative.

Since that update, which covered up to the build 151 released as 2010.11, we've continued development and polishing to get this Solaris 11 release finished up. We added a couple more components, including the previously mentioned xcb libraries, the FreeGLUT library, and the Xdmx Distributed Multihead X server. We cleaned up documentation, including the addition of some docs for the Xserver DTrace provider in /usr/share/doc/Xserver/. The packaging was improved, clearing up errors and optimizing the builds to reduce unnecessary updates. A few old and rarely used components were dropped, including the rstart program for starting up X clients remotely (ssh X forwarding replaces this in a more secure fashion) and the xrx plugin for embedding X applications in a web browser page (which hasn't been kept up to date with the rapidly evolving browser environment). Because Solaris 11 only supports 64-bit systems, and most of the upstream X code was already 64-bit clean, the X servers and most of the X applications are now shipped as 64-bit builds, though the libraries of course are delivered in both 32-bit and 64-bit versions for binary compatibility with applications of each flavor. The Solaris auditing system can now record each attempt by a client to connect to the Xorg server and whether or not it succeeded, for sites which need that level of detail.

In total, we recorded 1512 change request id's during Solaris 11 development, from the time we forked the “Nevada” gate from the Solaris 10 release until the final code freeze for todays release - some were one line bug fixes, some were man page updates, some were minor RFE's and some were major projects, but in the end, the result is both very different (and hopefully much better) than what we started with, and yet, still contains the core X11 code base with 24 years of backwards compatibility in the core protocols and APIs.

Monday Nov 15, 2010

X11 changes in the 2010.11 release

Another OS release came out today, 2010.11, and as usual, it has a number of X11 changes. The biggest change in X is probably... Hmm, I can see by the look on your face, you're not buying the casual use of “as usual” there. Okay, you caught me, this OS release isn't quite following our previous pattern, so I guess we better get that out of the way first. Please remember I am not an Oracle spokesman, and can't speak on behalf of Oracle, so don't even think of quoting this as “Oracle says...”

In many ways, this release is simply the continuation of the OpenSolaris distro releases of the last few years. It's built the same way, using the IPS packaging system and repositories, and Caiman installers, as the OpenSolaris 2009.06 and prior releases were. Where OpenSolaris 2009.06 (the last full release) was the biweeekly build numbered 111b, and the release we'd planned to put out as OpenSolaris 2010.03 earlier this year (and which made it to the package repository, but was not put up as downloadable ISO's) would have been biweekly build 134b, this release is 151a. You should be able to upgrade to it from OpenSolaris 2009.06 or OpenSolaris /dev builds via the package repository following the instructions in the 2010.11 release notes.

So what's different about this OS release? Well, it's not named OpenSolaris anymore for starters - it's Oracle Solaris 11 Express. We'd always said that OpenSolaris releases were leading up to Solaris 11 eventually, and this name emphasizes we're getting closer to that (though still not there yet). It also recognizes that this release is built by Oracle, not Sun nor the OpenSolaris community. While it's built on the work done by the OpenSolaris community, and many portions of it are still developed as open projects on opensolaris.org, the kernel and core utilities are once again being developed behind closed doors, and the final assembly and testing are similarly done in house. The license terms for the free downloads have changed as well (though it's still offered under support contract for commercial production use as well), and the OS images include some of the encumbered packages we'd had to keep out of OpenSolaris in order to allow OpenSolaris to be freely redistributable. (Not all of them, since some were simply EOL'ed as they were for hardware well past the end of its supported lifetime, like many of the old SPARC frame buffers.)

So with that out of the way, back to the topic at hand - what's new in the X Window System in this release? Well that depends on how far back you're coming from. You can browse the complete changelogs for X going back to the point we branched the Nevada branches from the Solaris 10 release, so I'll try to stick to the highlights.

Changes since the last OpenSolaris X11 source release

None, since the X sources on opensolaris.org are still updated automatically from our internal master gate on each commit. (In fact, since the source gates currently reflect a point between biweekly builds 153 & 154, they have changes newer than this release, such as the integrations of libxcb and FreeGLUT.)

Changes since the last OpenSolaris developer build release (b134)

There were 17 biweekly builds between the last one published to pkg.opensolaris.org/dev in March and this release. The biggest change in the X packages in this period was their packaging. Previously we built our packages using the old SVR4 package format that was used since Solaris 2.0, and in many cases following the breakdown used in the old Solaris 2 releases (SUNWxwinc for most headers, SUNWxwplt for most libraries, SUNWxwman for most man pages), and then the release team converted those to the IPS format used in the OpenSolaris releases. Like several of the other consolidations, X has now converted to building IPS packages directly, and in the process refactored the X packages to better follow the way the upstream X.Org sources were split into modules at X11R7, which also happens to be more similar to the way most Linux distros break them up. This should allow easier creation of minimized environments with the subset of X packages you need.

As for headers and man pages, they are now included in the packages they are used with - for instance all the libX11 headers and API man pages are directly in the x11/library/libx11 package. System admins can still decide to include or exclude them in their installs though, since they are tagged with the devel and docfacets”, which are the IPS mechanism for controlling optional package components. To read more about how to use these with X or the other changes in the refactoring, see the heads up messages I posted when this work integrated.

Of course, there were also the usual updates to new upstream releases - Xorg 1.7.7, freetype 2.4.2, fontconfig 2.8.0, among many others. The X server packages now also include the mdb modules and scripts for getting client and grab information from the server that I blogged about back in April.

Changes since the last OpenSolaris stable release (2009.06 / b111b)

This period saw the completion of our multiyear project to completely replace the old Solaris X code base with the X11R7 open source code base from X.Org. Solaris 10 and earlier shipped with Sun's proprietary fork of X11R6, with bits of X11R5, X11R6.4, X11R6.6, & X11R6.8 mixed in. We're now set up to much more easily track upstream and are deviated from upstream in much fewer places than before (partially due to pushing a number of our previous fixes back upstream, in other cases, we determined the upstream code was better and went with it).

We also had a very large user-visible change in build 130: all the files moved from /usr/X11 directly into /usr/bin & /usr/lib, following the work done in other parts of Solaris to move files from locations like /usr/ccs/bin and /usr/sfw to the common /usr directories. We still have symlinks in /usr/openwin and /usr/X11 for backwards compatibility, so we shouldn't break your .xinitrc calls to /usr/openwin/bin/xrdb or /usr/X11/bin/xmodmap.

Since 2009.06, we moved from Xorg 1.5 to 1.7.4. Of course, with this upgrade, we got the HAL support for input device configuration working just as X.Org started moving off HAL upstream, something we still need to deal with for Solaris - for this release, input devices are still configured in HAL .fdi files. The xorgcfg and xorgconfig programs did go away as part of this move though - fortunately more and more systems are working without any xorg.conf at all, and when one is needed, only the sections being changed have to be included, lessening the utility of programs to generate full configuration files. The new Xorg also includes support for virtual consoles on systems with the necessary kernel driver support (all x86 systems and SPARCs with frame buffers supporting “coherent console”).

We also added the synaptics touchpad driver, synergy software for sharing input devices with multiple systems, the simple xcompmgr composite manager, the xinput client for input device configuration, and finally provided IPS packaged versions of the classic xdm display manager and xfs legacy font server. The Xprint server and several related commands did go away, but the libXp library was kept for binary compatibility.

Our VNC implementation was converted from RealVNC 4.1.3 to TigerVNC 1.0.1, which is being kept up-to-date with new Xorg releases, unlike RealVNC, which hasn't really been updating it's open source release in the last few years. xscreensaver was finally updated from 5.01 to 5.11, and was actually moved out of the X gate in OpenSolaris to building as a RPM-style pkgbuild spec file with the other higher-level desktop software - hopefully in the process we fixed some long-standing bugs in our forked code.

Graphics updates included Nvidia's driver support for various new devices and OpenGL 4.0, and Intel's DRI updates, including GEM support in their DRM module. Mesa was added on SPARC to provide a matching OpenGL implementation, but with only the software renderer, no hardware acceleration.

What else has changed?

Besides the official Solaris 11 Express release information, you can find more details on changes in this release on a bunch of other blogs, such as:

But here's some changes in other parts of the OS you may not see listed on those:

Of course, that's just a small sample, the full changelogs are a few thousand items long (and unfortunately, some of the consolidations haven't published theirs outside the firewall).

About

Engineer working on Oracle Solaris and with the X.Org open source community.

Disclaimer

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle, the X.Org Foundation, or anyone else.

See Also
Follow me on twitter

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today