Tuesday Sep 08, 2009

Java RTS 2.2 Early Access 2 Released

Sun recently announced the availability of a second edition of its Java Real-Time System (Java RTS) version 2.2 Early Access.  In a previous blog post, many of the important new features of this release were highlighted.  But I felt compelled to jot down a few additional words, not only because there's a newer version of the software to evaluate, but because there's one interesting component that deserves a little more exposure.

I spent some time today speaking with my compadre, Eric Bruno, author of Real-Time Java Programming: With Java RTS.  He clued me in on the fact that our engineers have performed the daunting task of backporting the JDK 7 client compiler onto Java RTS 2.2.  The engineering team is particularly proud of this feat, and ought to be.  It not only enhances Java RTS performance, it also enables them to better stay in lock step with the latest client hotspot VM developments, something the latest Java SE 6 releases may not entirely do.  For some insight into the engineering trade-offs associated creating this latest version of Java RTS, check out Roland Westerlin's entry.

You can find further information about Java RTS here.  And you can download a version of the software for evaluation here.


Wednesday Jun 10, 2009

Early Access version of Java RTS 2.2 Released

Just in time for the start of JavaOne 2009, Sun released the latest version of its real-time Java platform, Java RTS 2.2, in early access form.  New features of this release include but are not limited to:

  • OS Platform support for Solaris 10 (x86 and Sparc - Update 5 & 6), Red Hat MRG 1.1 and  SUSE Linux Enterprise Real-Time 10 (SP2 Update 4)
  • 64-bit support for the three operating systems mentioned above. In addition, 32-bit support continues to be supported.
  • Faster throughput. Java RTS 2.2 utilizes a new dynamic (JIT) compiler targeted for Java SE 7 which contains several performance improvements including a new register allocator.
  • TSV (Thread Scheduling Visualizer) enhancements.
  • NetBeans plug-in support for Java RTS and TSV. 
  • New Initialization-Time-Compilation (ITC) options including the ability to wild card a set of methods vs listing single methods and support for specifying compilation based on thread type.
  • Auto-tuning and startup improvements made to the Real-time Garbage Collector (RTGC).
  • New documentation including a Java RTS Quick Start Guide to get new users started quickly with minimum programming and configuration.

A 90-day evaluation is available here: http://java.sun.com/javase/technologies/realtime/index.jsp.  For more detailed information on OS support and new functionality, check out the 2.2 Release Notes.

Tuesday Jun 17, 2008

Real-Time Java in a Zone

As is often the case, Sun's technologies and offerings are being applied in ways which we hadn't necessarily anticipated.  Yet another example has reared its head in the govenrment/military space where customers have expressed interest in using Sun's Java Real-Time System with Solaris Trusted Extensions.  As it stands right now, Java RTS will neither operate nor install within the confines of such an environment.  Let's investigate why this is so, and see what current opportunities there are for working around this shortcoming.

So what is it that causes Trusted Extensions and Java RTS not to play together nicely?   It happens to revolve around Trusted Extension's extensive usage of Solaris zones to limit access between differing security levels.  Solaris packages must be specifically configured to accommodate zones, which has yet to formally take place with Java RTS.  As zones are a core component of Solaris, we can, for the sake of simplicity, just use standard Solaris to demonstrate how we can work around this temporary limitation.  These modifications should apply to Trusted Extensions just as well.   To get Java RTS to work within a zone, follow these steps:

1. Install the Java RTS cyclic driver (only) in the global zone.

global# pkgadd -d . SUNWrtjc

Processing package instance <SUNWrtjc> from </cyclic/src/packages_i386>
## Installing package <SUNWrtjc> in global zone

Java Real-Time System cyclic driver(i386) 2.1.0,REV=2008.
Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Using </> as the package base directory.
## Processing package information.
## Processing system information.
   5 package pathnames are already properly installed.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user
permission during the process of installing this package.

Do you want to continue with the installation of <SUNWrtjc> [y,n,?] y

Installing Java Real-Time System cyclic driver as <SUNWrtjc>

2. Create a zone called 'rtjzone':
global# mkdir -p /zone
bash-3.00# zonecfg -z rtjzone
rtjzone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:rtjzone> create
zonecfg:rtjzone> set zonepath=/zone/rtjzone
zonecfg:rtjzone> verify
zonecfg:rtjzone> commit
zonecfg:rtjzone> exit
global# zoneadm list -vc
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   - rtjzone          configured /zone/rtjzone                  native   shared
global# zoneadm -z rtjzone install
Preparing to install zone <rtjzone>.
Creating list of files to copy from the global zone.
Copying <6984> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1074> packages on the zone.
Initialized <1074> packages on zone.
Zone <rtjzone> is initialized.
Installation of <1> packages was skipped.
The file </zone/rtjzone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

3.  Modify the zone to allow access to the cyclic device, and to allow additional privileges

global# zonecfg -z rtjzone
zonecfg:rtjzone> set limitpriv=default,proc_priocntl,proc_lock_memory,proc_clock_highres
zonecfg:rtjzone> add device
zonecfg:rtjzone:device> set match=/dev/cyclic
zonecfg:rtjzone:device> end
zonecfg:rtjzone> verify
zonecfg:rtjzone> commit
zonecfg:rtjzone> exit

global# zoneadm -z rtjzone reboot
Note:  One privilege that is useful with Java RTS is sys_res_config.  This is typically used to assign a real-time process to a processor set.  Unfortunately zones cannot currently be given this privilege.  You can however, from the global zone, assign a processor set to a zone, which might be a reasonable workaround.

4.  Get a copy of the SUNWrtjv package and modify it so that it will install in a zone.  The postinstall script and postremove script must replaced with those provided by these hyperlinks just mentioned.

rtjzone# cd /scratch
rtjzone# ls
SUNWrtjv	postinstall	postremove
rtjzone# cp postinstall SUNWrtjv/install/
rtjzone# cp postremove SUNWrtjv/install/

5. Modify the SUNWrtjv pkgmap file with the appropriate sizes, checksums and last access dates.  The source code for a sample C program, called pkgmap_info, which prints out the necessary information, can be found here.

rtjzone# cd SUNWrtjv
rtjzone# grep post pkgmap 1 i postinstall 5402 42894 1194344457 1 i postremove 2966 34854 1194344457 rtjzone# cp pkgmap_info.c /tmp rtjzone# cc -o /tmp/pkgmap_info /tmp/pkgmap_info.c rtjzone# cd /scratch/SUNWrtjv/install/ rtjzone# /tmp/pkgmap_info postinstall postinstall 5820 9488 1213727841 rtjzone# /tmp/pkgmap_info postremove postremove 3092 45039 1213727538
Replace the postinstall and postremove entries in the pkgmap file with those produced by the pkgmap_info program.  You cannot simply use the example data above because the last access times will not match.  Doing so will cause the install to fail.

6. Install the Java RTS SUNWrtjv package inside the zone.

rtjzone# cd /scratch
rtjzone# pkgadd -d . SUNWrtjv

Processing package instance <SUNWrtjv> from </scratch>

Java Real-Time System runtime environment(i386) 1.5.0_13_Java-RTS-2.0_01-b08_RTSJ-1.0.2,REV=2007.
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.

Where should this package be installed? (/opt): /opt

To achieve predictable temporal behavior, the Java Real-Time System
must be granted access to a number of privileged Solaris resources.
By default, access to these privileged resources is only granted to
the superuser (root). They can also be granted to additional users
by creating a rights profile, that is, a collection of authorizations,
that can later be assigned to an administrative role or directly
to a user.

As part of this package installation, a local 'Java Real-Time System User'
rights profile can be created on this machine.
This rights profile should NOT be created if such an action conflicts
with your computer security management policies. If unsure, contact
your system administrator or your computer security manager.
Also refer to the product's release notes for further details regarding
the privileges required by the Java Real-Time System.

Should a local 'Java Real-Time System User' rights profile be created? [y,n] (no): y
Using </opt> as the package base directory.
## Processing package information.
## Processing system information.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user
permission during the process of installing this package.

Do you want to continue with the installation of <SUNWrtjv> [y,n,?] y


## Executing postinstall script.
Creating the 'Java Real-Time System User' rights profile.

Refer to the 'System Administration Guide: Security Services'
documentation for further information regarding the way to assign the
'Java Real-Time System User' rights profile to users, groups, or
administrative roles using the Solaris Management Console, smc(1M) or
the usermod(1M), groupmod(1M) and rolemod(1M) commands.

Installation of <SUNWrtjv> was successful.

6.  Try running a real-time Java application in the zone.

rtjzone# /opt/SUNWrtjv/bin/java -version
java version "1.5.0_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13_Java-RTS-2.0_01-b08_RTSJ-1.0.2)
Java Real-Time System HotSpot(TM) Client VM (build 1.5.0_13-b08, mixed mode)

We hope to more formally support Solaris zones usage with Java RTS in the future.  In the interim this workaround can help you get started.  Many thanks to Jim Clarke, who did the lion's share of the legwork to find this solution.

Thursday May 29, 2008

Are Solaris RAM Disks Swappable?

As memory access is typically orders of magnitude faster than disk access, the idea of using a part of RAM as an in-memory storage device has been one of the earliest performance optimizations realized by the computer science community.  Even though today this type of optimization takes place transparently inside modern operating systems (via mechanisms like the disk buffer cache),  there are still circumstances where manually creating a RAM disk might still be quite useful.

A Solaris customer using the ramdiskadm(1m) utility to create a Ram disk device for a real-time application posed the following question: "Are RAM disks swappable?". From a real-time perspective, a RAM disk not only yields significantly better seek performance but also provides for more deterministic behavior compared to the traditional rotating disk platter.  The customer's concern here is that, under dire circumstances, is it possible for operating system swap out the RAM disk memory?  Needless to say, if a swap out occurred it would put a big crimp in the real-time application's predictability.

To get an idea of what's going on when a RAM disk is created, let's use the Solaris kstat(1m) kernel statitics utility to see how memory is being allocated.  First let's see what memory looks like before creating a RAM disk:

# pagesize
# kstat -n system_pages | grep pagesfree
       pagesfree                       96334
# kstat -n system_pages | grep pageslocke      pageslocked                     26170

So, on this particular system, where a page is 4096 bytes,  there are currently 96334 pages free, and 26170 pages that are locked.  Now let's create a 50MB RAM disk:

# ramdiskadm -a rd 50m
# ramdiskadm
Block Device                                                  Size 
/dev/ramdisk/rd                                           52428800    Yes 
# kstat -n system_pages | grep pagesfree
       pagesfree                       83507
# kstat -n system_pages | grep pageslocked
       pageslocked                     38988

Let's subtract the original number of pageslocked from the latest value and multiply by the pagesize:

# pagesize
# bc

The increase in locked pages can be attributed to the creation of the RAM disk (50m + a small amount of overhead).  So yes, these pages are locked into memory.  But it would be nice to get a definitive statement on what pageslocked actually means.  According to McDougal, Mauro and Gregg's Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris, pageslocked is "Total number of pages locked into memory by the kernel and user processes".  Furthermore, the man page for plock(3C), a related library routine which enables programmers to lock memory segments, states that "Locked segments are immune to all routine swapping".  What's routine swapping?

Hmm.  Anybody care to shed some light on this?

Thursday Jan 31, 2008

Good Things Come To Those Who Wait

Java RTS 2.1 EA (Early Access) marks the arrival of a commitment made some time back, namely that Sun would provide a version of the Java Real-Time System for Linux.  Perhaps to some, it was long time in the making, but in fact there are at least 2 good reasons why a Linux version wasn't available till now:

  1. Until recently, there was no standard Linux release/kernel which had true real-time support.  Typically the versions available were non-standard and did not constitute any considerable volume.  Mainstream Linux distributions are only now incorporating the necessary real-time underpinnings.
  2. Porting the Java Real-Time System VM to additional platforms is non-trivial.

Support and testing for Java RTS 2.1 EA at this time is limited to the currently shipping SUSE Linux Enterprise Real Time10 platform and the upcoming Red Hat Enterprise MRG 1.0 release.  It is however possible that other versions of Linux could run Java RTS 2.1 EA as it utilizes the real-time POSIX programming interface.  At minimum they would require a 2.6.21 kernel or greater and a glibc of 2.5 or greater.  In addition, the latest RT patch would also be needed.

This announcement pertains only to Linux, but of course a 2.1 EA version for both Solaris x86/x64 and Sparc will be shortly forthcoming. In the interim, a version of Java RTS 2.0 update 1 is readily available.  Documentation for both Java RTS 2.0 and 2.1 EA can be found here.

Regardless of platform, an evaluation version of the software is available for download at the Java RTS download page.


Wednesday Dec 19, 2007

Java RTS 2.0 Update 1 Released

After blogging about the relationship between high resolution timers and Java RTS, a comment was posted by a reader stating that they couldn't use Java RTS 2.0 with the latest Solaris 10 update.  Unfortunately that reader was correct.

At the time of the post, Sun had released Solaris 10 Update 4, while Java RTS 2.0 would only support Solaris 10 update 3.  To further exacerbate the situation, once Sun releases a new version of Solaris, they do their best to encourage users to use the latest version by making it difficult (if not impossible) for them to locate previous update versions.

There is a dependency between the aforementioned high resolution timer interface and Solaris which is update specific.  We're working to rectify this situation, but as the process behind Solaris is deliberately bureaucratic, this might take a while.  In the interim we'll need to keep in better lock step with Solaris releases.

On November 27, Java RTS 2.0 update 1 was released: 

  • It supports the latest Solaris 10 update (called 08/07 or update 4)
  • Support for Java SE has been updated to Java SE 5 update 13
  • A new Java RTS DTrace Provider is available
  • Other enhancements are listed here


Monday Oct 15, 2007

General Purpose Always Wins ... Why Not for the Real-Time Market Too?

The brief but brilliant era of computing has seen companies come and go, many relegated to the ash heap of history because they failed to heed this simple rule:  
  In the long run general purpose solutions will always win out over specialized proprietary ones. 
As a long time employee of Sun Microsystems, I've witnessed firsthand the effects, both positive and negative, that this law has had on my company.  Sun can attribute it's initial meteoric rise to the fact that it offered a viable alternative to the popular minicomputer platform of the day.  The first Sun workstations were built from commercial off-the-shelf components, and although nearly equal in performance to the minicomputer, they were so much more affordable that they became good enough.  And of course over time, as Moore's law might dictate, those initial deficiencies quickly dissipated, and in fact surpassed traditional minicomputer capabilities.
Somewhere along the line Sun lost sight of this ideal and began incorporating more proprietary technology into their products.  At first the strategy appeared to be successful as Sun was well positioned to reap huge benefits during the Internet bubble.  Meanwhile, low-cost general purpose servers were continuously improving.  When the bubble burst they in turn became good enough alternatives to the powerful but costly Sun servers.  The resulting decline of Sun was rapid, and it's taken the better part of a decade for us to recover.   This story has been told -- and will continue to be again and again -- for those refusing to learn this lesson.  A professor of mine once told me, "If there's anything we've learned from history, it's that we haven't learned anything from history".

When markets mature, even those where technology is paramount, economic considerations dominate. General purpose systems scale on every dimension (unit, management, training and integration costs) whereas proprietary systems do not.  A switch to more standard components should in no way be construed as stifling innovation.  Rather, general purpose systems help create new innovation by building from common elements economically appealing in their own right, and presumably even more economically beneficial once combined.1

[1] The above paragraph was taken in bits and pieces from an email exchange with Dave Hofert, Java Embedded and Real-Time Marketing Manager.  His points were so compelling I couldn't help myself.

Real-time industrial controllers could be the next market ripe for disruption.  Admittedly this is an entrenched and conservative lot.  But the economics of general purpose computing cannot be denied.  As organizations strive to further eliminate cost out of their system, revisiting usage and deployment of industrial controllers, typically via custom proprietary PLCs, is falling under review.  Consequently, at the behest of one of the world's largest industrial corporations, Sun has partnered with iGoLogic, a systems integrator, and Axiomtek, an industrial PC board manufacturer, to create the real-time controller platform pictured below.
Among others, here are some of the compelling benefits of this platform: 

  • It's based on standard x86 hardware.  The motherboards aren't much larger than a postcard, are energy efficient, and yet have PC class processing power.  The number of competing manufacturers in this space eliminates vendor lock-in and insures price sensitivity and further rapid innovation.
  • Real-time applications are developed using Sun's Java Real-Time System, enabling you to leverage the largest development community on the planet.  Obscure development languages and highly specialized environments are longer needed.
  • Industrial Networking Protocols (e.g PROFIBUS, EtherCAT) are easily migrated to this platform, partly because of the wealth of development tools available.
  • The system utilizes an IDE flash drive from Apacer.  In addition to eliminating the moving parts associated with a traditional disk drive, it consumes less power and makes the system more resistant to shock and vibration.  Overcoming the longevity limitations of flash memory, Apacer has done some interesting work on wear leveling algorithms effectively extending the lifetime of the flash device well past the expected lifetime of the industrial controller.

Let this be our first salvo fired over the proprietary industrial encampment.  We believe the opportunity is immense, but also understand that to achieve any measure of success, partnering with organizations who are truly experts in this arena is critical.  If you think you can add further value, we'd love to talk.

Tuesday Jun 19, 2007

Real-Time Java and High Resolution Timers

Any modern x86/x64 processor worth its salt comes equipped with an Advanced Programmable Interrupt Controller, or APIC.  Among the features that an APIC provides is access to high resolution timers.  Without such capability, the default interrupt source for timer and cyclic operations has a precision on the order of 10 milliseconds -- hardly fine-grained enough for any serious real-time work.

The cyclic subsystem, introduced in Solaris 8, gives Solaris the capability to support high resolution timers.  The Sun Java Real-Time System version 2.0, available for Solaris on both x86 and Sparc platforms, includes an additional package called SUNWrtjcJava Real-Time System cyclic driver.  This package exposes an interface to the cyclic subsystem, normally only available to the kernel.  For those already familiar with Sun's Java RTS, you've no doubt noticed that you either need to run as superuser or assign a set of fine-grained privileges to an ordinary user account. (sys_res_config, proc_priocntl, proc_lock_memory and proc_clock_highres).  The proc_clock_highres privilege gives access to those timers.

Originally developed on an AMD Athlon-based PC, I recently moved a Real-Time Java project over to my Toshiba Tecra A1 laptop running the same version of Solaris and Java RTS.  With the goal of getting in a little development time during a long flight, that migration suddenly casued timings to be all wrong.  Why, might you ask, would moving an application from one seemingly identical Solaris environment to another cause this unusual behavior?  Turns out that even though the laptop, a Pentium 4M based system, has an APIC, it was never exposed by the laptop BIOS.  In this scenario, regardless what you do from a Solaris perspective, you'll never get access to the high-res timers. 

This phenomenon appears to be most prevelant in laptops.  As they account for about 50% (or more?) of PCs sold today, developers have a realistic chance of running into this situation.  You can issue this magic Solaris incantation to determine if your system has high-res timer support:

   # echo "psm_timer_reprogram::print"  | mdb -k

If anything but 'apic_timer_reprogram' is then displayed, your machine has no exposed APIC, and is probably unsuitable for Java RTS.  In some cases the BIOS may be configurable with regards to APIC support; in many others it is simply not available.

In the absence of an APIC, there is the potential to improve the high-resolution timing by setting the following tunable in /etc/system:

   set hires_tick=1

Following a reboot, this would change the clock tick frequency from 100 to 1000 ticks per second. This frequency can then be further tuned by setting the hires_hz tunable to the desired value:

   set hires_hz=10000

The default value is 1000 ticks per second; higher values are not officially supported.

Note that tuning your machine in this way does not come without cost.  It is likely to degrade overall performance, as the system will need to spend a larger proportion of time handling the larger frequency of clock interrupts.1

   [1] Thank you  Christophe Lizzi for your explanation of the problem and potential workaround.


Monday Jun 11, 2007

Audio Transcript and Slides Available for "Real-Time Java Meets Wall Street"

Better late than never ...

As Promised, you can find the audio transcript for the Java ONE 2007 technical session TS-1205: The Sun  Java Real-Time System Meets Wall Street at the aforementioned link.  The slide deck should also be available there, or can be downloaded here.

Friday Mar 30, 2007

Real-time Java Meets Wall Street

Jim Clarke, Eric Bruno and I have been selected to present this year at Java ONE.  The title of our session is called "TS-1205 The Sun Java Real-Time System Meets Wall Street".  We're scheduled to present on Wednsday, May 9 at 10:55AM - 11:55AM Pacific in the Moscone Convention Center in San Francisco.  If you plan on attending, please stop by and see us.  For those that can't make the event, we'll be sure to post the slides once we're permitted.

To get an idea of what we'll discuss, take a look at the abstract below; 


Today's financial institutions use program trading systems to automatically execute split-second trades based on sophisticated algorithms that assess current market conditions. Trade decisions and execution must occur in a timely fashion to capitalize on the market. Missing a trade opportunity, even by a few seconds, can lead to significant losses.

With improvements in Java technology performance, firms are starting to use Java technology to implement these algorithms and thereby realizing productivity gains over more-traditional C/C++ development. However, in these time-critical systems, there is still no guarantee that at any instant, the process will not be interrupted by the Java platform garbage collector. Although trade execution occurs most of the time within an acceptable response time, the trade execution is delayed every once in a while, due to garbage collection or some other system event.

Sun's implementation of the Real-Time Specification for Java (JSR 001), the Sun Java Real-Time System, enables real-time processing through techniques that protect important threads from garbage collection and other system interrupts. This means that trading systems can confidently monitor the market and take action consistently within a calculated window of opportunity.

To demonstrate the impact of these techniques, Sun's OEM Software Systems Engineering technical team has written a demonstration of a trading system that uses real trade data. The demonstration compares a regular Java virtual machine against the Sun Java Real-Time System. For each run, a graph shows the difference in the actual trade price and the price when the trade should have executed. Running this trading system with the Sun Java Real-Time System shows that no money is lost due to garbage collection latencies. The results will be contrasted with the same application run with the standard (non-real-time) Java virtual machine.

This presentation demonstrates a working system and proves real-time Java technology's ability to satisfy hard real-time trading requirements.

Friday Feb 23, 2007

Solaris Was Real-time Before Real-time Was Cool

In the financial services market, there is a general trend to move key systems close to, or even right inside the exchanges themselves -- the idea being that the nearer you are to the source, the less network infrastructure and latency you'll experience.  With this advantage firms can potentially take on additional transaction loads at higher transaction rates.  These systems typically use the latest Intel or AMD processors and run a commercial distribution of Linux.1

[1] Thank you Eric Bruno for your brief description, and for unknowingly letting me (slightly) plagiarize your comments.

Indeed these co-located systems perform as expected almost all the time.  But there are periodic intervals where the latency increases by several orders of magnitude, the ramifications of which could be financially disastrous.  After eliminating other components, the street seems to be focusing its wrath on commercial Linux distributions and their lack of real-time capabilities.

The linux community is actively working to include underpinnings to support real-time, but as of yet these capabilities are not part of a standard major (i.e. Red Hat, SuSE) distribution.  Instead, an alternate version of linux with real-time extensions is offered.  These extensions are in effect separate non-standard OS releases, and have not had the soak time required by many institutions.

Back in the early 90's, I volunteered to move over to Sun's newly formed SunSoft business unit.  One of it's main charters was to push the concept of running Solaris on alternate, i.e. Intel, platforms.  (Don't get me started here, can you imagine where Solaris would be right now if Sun had actually taken this initiative seriously back then?)  As part of that transition, I had the opportunity to take a Solaris Internals course, and couldn't help but notice the effort put in architecturally to address short latencies.  I still have the course notebook; it is dated September 1993.

The point is Solaris already has the real-time capabilities claimed by these add-on linux extensions.  It is built into the operating system, has been for quite some time, is rock solid and doesn't require any additional components.  A partial list of features include:

  • Real-time Scheduling Class - In order to allow equal opportunity to system resources, traditional Unix schedulers transparently change process priorities to give competing processes a fair chance.  Although well suited for timesharing systems, this is unacceptable real-time behavior.  Since its outset, Solaris through its SVR4 roots, introduced the concept of alternate scheduling classes.  It includes a real-time scheduling class, which furnishes fixed-priority process scheduling at the highest priority levels in the system.
  • Fine-Grained Processor Control / Processor Sets - Solaris allows threads and applications to be bound to specific individual processors. In addition, processors within a system can be grouped together as a processor set and dedicated to real-time tasks.2  Here's a nice article describing processor sets. Dated June 2001,  processor sets have been a part of Solaris since release 2.6.
  • Interrupt Sheltering - Available since Solaris 7, this feature enables CPUs to be sheltered from unbound interrupts.  It can be used in conjunction with processor sets to shelter all CPUs in a processor set.  Note: At least one processor in the system must be kept unsheltered.
  • Priority Inheritance - Priority inversion occurs when a high-priority thread blocks on a resource that is held by a lower-priority thread. A runnable thread with a priority between the high and low-priority threads creates a priority inversion because it can receive processor resources ahead of the high-priority thread. 
To avoid priority inversions with kernel synchronization primitives, Solaris employs a transient priority inheritance protocol. The protocol enables the low-priority thread holding the resource to “inherit” the higher priority of the blocked high-priority thread. This approach gives the blocking low-priority thread the CPU resources it needs to complete its task as soon as possible so that it can release the synchronization primitive. Upon completion, all threads are returned to their respective priorities by the kernel.3
  • High Resolution Timers - Solaris 8 introduces the cyclic subsystem; this allows for timers of much better granularity -- in the microsecond and nanosecond range -- without burdening the system with a high interrupt rate.
  • Memory Locking - The paging in and out of data from disk to memory may be considered normal behavior for virtual memory systems, but it is unacceptable for real-time applications.  Solaris addresses this problem by allowing the locking down of a process' pages into memory, using mlock(3C) or mlockall(3C) system calls.
  • Early Binding - By default, linking of dynamic libraries in the Solaris is done on an as-needed basis. The runtime binding for a dynamically linked function isn't determined until its first invocation. Though flexible, this behavior can induce indeterminism and unpredictable jitter in the timing of a real-time application. To avoid jitter, the Solaris provides for early binding of dynamic libraries. By setting the LD_BIND_NOW environment variable to "1", libraries are bound at application startup time. Using early binding together with memory locking is a very effective approach to avoiding jitter.4

[2,3,4] Shameful plagiarism from Scalable Real-Time Computing in the Solaris ™ Operating Environment. A Technical White Paper. To further prove the maturity of Solaris' real-time features, this document was written in the Solaris 8 time frame.  It was copyrighted in 2000.

So why not give Solaris more consideration?  It's way more mature.  And in the unlikely event (chcukle, chuckle) that a lower-latency OS might not solve all your performance problems, I'd put my money on Solaris and DTrace over anything Linux could offer in finding the real problem.


Jim Connors


« April 2014