Monday Apr 06, 2009

Node Count and JavaFX Performance

In a recent blog entry entitled Best Practices for JavaFX Mobile Applications (Part 2), Michael Heinrichs espouses that keeping the scenegraph as small as possible helps JavaFX applications perform optimally. Regardless what version of JavaFX you're using, this is sage advice.  Having spent some time trying to create components for a scoreboard-like application, I was concerned over the amount of CPU time being consumed by the clock component pictured directly below.


You can click on the preceding image to run this mini application via Java WebStart.   By placing your mouse over any of the digits and typing in, via keyboard input, a valid number, you can set the clock.  Clicking on the "START/STOP" text will toggle the clock on and off.  Like many scoreboard clocks, when the time remaining is less than one minute, 10ths of seconds are displayed.  It is during this phase, when digits are being updated every tenth of a second, that this application can be especially taxing.  You can imagine how troublesome this clock might be if it were to be part of say a hockey scoreboard which could have an additional 4 penalty clocks ticking simultaneously.

The major factor affecting performance appears to be the sheer number of nodes in the scenegraph that require recalculation for every clock tick.  For this first implementation, each of the five clock digits is comprised of 27 BulbNodes, (my naming) which are switched on or off depending upon what value needs to be displayed.

In an effort to see how decreasing the node count might effect performance, this second implementation of the clock component uses the same underlying framework, except that each digit is now composed of 7 LED SegmentNodes (my naming again) instead of 27 BulbNodes.   You can run this version of the clock component by clicking on the image that follows.


For our final example, in order to truly minimize node count, each digit is represented by a single ImageView node. When the value of a digit changes, a new Image is displayed.  By caching all of the possible digit values (blank, 0-9) you can very quickly switch images.  No doubt, prettier images can be created, but I think you get the point.  Click on the image that follows to try this version.


The Results

The slower the compute platform, the more pronounced the differences in performance should be.  Thinking along those lines, a very modest 1.4 GHz Pentium M laptop was chosen as the test environment to compare CPU utilization for these applications.  OpenSolaris provides an easy-to-use well-known command-line tool called vmstat(1M), which was chosen as the mechanism to analyze the individual applications. In contrast, the Performance Tab which is part of the Windows Task Manager, seemed to produce wild performance variations.

For each run,  the clocks were set to one minute, and run until the time expired.  The data shown below represents the average CPU utilization, after startup, for each of the three implementations.  In particular we'll look at the following fields returned by vmstat:

  • us - percentage usage of CPU time in user space
  • sy - percentage usage of CPU time in system space
  • id - percentage usage of CPU time idling
The sum of (us + sy + id) should approximate 100%.


Number of Nodes per Digit
CPU Utilization
Implementation 1: BulbClockNode
 27 BulbNodes
 us: 22%  sy: 2%  id: 76%
Implementation 2: LEDClockNode
 7 SegmentNodes
 us: 9%    sy: 2%  id: 89%
Implementation 3: ImageClockNode
 1 ImageNode
 us: 3%    sy: 2%  id: 95%


The JavaFX engineering team is well aware of this limitation, and hopes to redesign the underlying scenegraph plumbing in the future.  Regardless, it's still a good idea to take into consideration the size of your scenegraph.

JavaFX book status:  Our upcoming book, JavaFX: Developing Rich Internet Applications, is in copy edit.  Coming soon: Rough cuts of certain chapters will be available on Safari.

Tuesday Mar 24, 2009

Hooray! Acrobat Reader for [Open]Solaris x86

Today truly marks a milestone in the history of Solaris for the x86/x64 platform.  One of the most ubiquitous applications, Adobe Reader (notably Acrobat), is now available on [Open]Solaris for both Sparc and x86.

It's been a long time coming, one argument for the lack of Solaris x86 support till now was that [Open]Solaris didn't have the critical mass.  That's a hard one to swallow for the following reasons:

  • There's been a Solaris Sparc version for a long time.  Solaris x86 downloads outnumber Sparc downloads by a large factor.
  • Versions for AIX and HP-UX, which have a significantly smaller installed base, are available.  Maybe IBM and HP payed a lot of money for this?
  • Until recently there were versions even for the likes of OS/2 and UnixWare.

Perhaps open source alternatives are becoming good enough to pose a threat?  Whatever the reason, we welcome the arrival of Acrobat Reader and Adobe's change of heart.

You can get your copy here.

Happy Downloading!

Thursday Mar 19, 2009

Bindstorming

It is within our nature, even in the most infinitesimal way, to leave our mark on this world before we exit it.  I'd like to coin the following term, heretofore unseen in the JavaFX space, and submit it as my humble contribution to the human collective:

bindstorm \\'bïnd•storm\\ (noun): condition where a multitude of JavaFX bind recalculations severely hampers interactive performance

Yeah, I know, using the word you wish to define inside its definition is bad, but there is precedent for this: (1) Fancy-schmancy, hoity-toity college dictionaries do it all the time. (2) Mathematicians and computer scientists call this recursion: that mysterious concept which developers use to impress others of their programming prowess.

Don't get me wrong, JavaFX binding is incredibly powerful.  Heck, we dedicated a whole chapter to it in our soon-to-be-released book JavaFX: Developing Rich Internet Applications.  But binding does come with a price, and like most anything else, over-consumption can lead to abuse.

Consider this use case: you've got a JavaFX application with dozens or maybe even hundreds of Nodes that are part of the scenegraph.  Each of the Nodes are ultimately sized and positioned in proportion to height and width instance variables that are passed on down.  If you define width and height at startup and have no interest in a resizeable interface, then you stand a good chance of avoiding the use of many bind expressions.  The one potential twist here is that if you're sincerely interested in a non-resizeable application, but want it to consume the entire screen, what do you do?  As screens come in all shapes and sizes, you may not know what the resolution is at start time.  JavaFX has an elegant solution for this which uses binding.

Here's a simple application which defines a Rectangle and Circle that fill the entire screen.  You can click anywhere within the Circle to exit the application.  Notice the number of binds required to get this to work.

import javafx.stage.\*;
import javafx.scene.\*;
import javafx.scene.shape.\*;
import javafx.scene.paint.\*;
import javafx.scene.input.\*;

function run() : Void {
    var stage: Stage = Stage {
        fullScreen: true
        scene: Scene {
            content: [
                Rectangle {
                    width: bind stage.width
                    height: bind stage.height
                    fill: Color.BLUE
                }
                Circle {
                    centerX: bind stage.width / 2
                    centerY: bind stage.height / 2
                    radius: bind if (stage.width < stage.height) then
                            stage.width / 2 else stage.height / 2
                    fill: Color.RED
                    onMouseClicked: function(me: MouseEvent) {
                        FX.exit();
                    }
                }
            ]
        }
    }
}

Imagine what this would look like if you had lots of complex custom components with many more dependencies on height and width.  In addition to the potential performance impact, this could be error-prone and cumbersome to code.  To avoid the over usage of binding and the potential for a bindstorm, applications of this sort could be re-written as follows:

import javafx.stage.\*;
import javafx.scene.\*;
import javafx.scene.shape.\*;
import javafx.scene.paint.\*;
import javafx.scene.input.\*;

function run() : Void {
    var AWTtoolkit = java.awt.Toolkit.getDefaultToolkit ();
var screenSizeFromAWT = AWTtoolkit.getScreenSize (); Stage { fullScreen: true scene: Scene { content: [ Rectangle { width: screenSizeFromAWT.width height: screenSizeFromAWT.height fill: Color.BLUE } Circle { centerX: screenSizeFromAWT.width / 2 centerY: screenSizeFromAWT.height / 2 radius: if (screenSizeFromAWT.width <
screenSizeFromAWT.height) then
screenSizeFromAWT.width / 2
else screenSizeFromAWT.height / 2 fill: Color.RED onMouseClicked: function(me: MouseEvent) { FX.exit(); } } ] } } }

We achieve the same effect as the first example by first making a call to a method in the java.awt.Toolkit package.  With this information we can statically define our scenegraph without the use of binding.

There is one caveat to this solution.  As the AWT (Advanced Windowing Toolkit) is an integral part of Java SE, this code should be portable across all JavaFX desktops.  However, if you wish to deploy a JavaFX Mobile solution, the AWT calls would likely change.  Is there a mechanism that might work across both models?

As a final thought, while we're on this theme of coining terms, my compadres Jim Clarke and Eric Bruno, co-authors of the aforementioned JavaFX book, jokingly asked what word could be used to describe this scenario:

"Condition where binds lead to binds that leads back to the original bind, ending up in a Stack fault?"

BindQuake? BindTsunami? Bindless? BindSpin? BindHole (BlackHole)? BindPit?


        
    

Tuesday Feb 24, 2009

Registering Multiple Actions (or Handlers) in JavaFX

Java developers, especially those performing any type of GUI work, will ultimately encounter Java's event-driven programming paradigm.  In short, if programmers want to act upon some kind of event they bundle up a chunk of code into a Java method, typically referred to as a handler, and register the handler with that event.  Whenever that event occurs, the handler code will automatically be executed.

JavaFX provides a similar mechanism.  For a straightforward example, the code below defines a simple timer in JavaFX with a resolution of 1 second.  Each time a second expires, the function specified by the action instance variable will be executed.  Here's what it looks like:

import javafx.animation.\*;

public class SimpleTimer {
    public def timeline = Timeline {
        repeatCount: 5
        interpolate: false
        keyFrames: [
            KeyFrame {
                time: 1s
                action: function () : Void {
                    println("tick");
                }
            }
        ]
    }
}

Adding a run() function, as follows, to the bottom of this source will enable you run an instance of this timer:

function run() : Void {
    var s = SimpleTimer{};
    s.timeline.playFromStart();
} 

The output from this run looks like this:

tick
tick
tick
tict
tick 

It's all well and good if you only need a single action.  What if you wanted to perform multiple actions and/or dynamically add or subtract a number of actions?  We can enhance our previous SimpleTimer class to dynamically register and unregister handlers by taking advantage of two of JavaFX's features: sequences and function pointers.

Our new class provides more flexibility:

  • It defines an instance variable called duration, which enables the user to specify the resolution of a clock tick at object instantiation.
  • It defines two additional public functions called registerHandler() and unRegisterHandler() which take a function pointer (a handler) as an argument.  By registering a handler, the function will be included in the list of handlers to be executed each time the specified duration expires.
  • A handler is registered by inserting it's function pointer argument into an internal sequence of function pointers called handlers[].
  • A handler is similarly unregistered by deleting it's function pointer argument from the handlers[] sequence.
  • The action instance variable, which is part of the KeyFrame instance, now calls an internal function called runHandlers()runHandlers() sequentially executes the functions found in the handlers[] sequence.
Here's the new class:
import javafx.animation.\*;

public class Timer {
    /\*\*
     \* User-definable:  specifies the length of time for one cycle.
     \*/
    public var duration = 100ms;

    public def timeline = Timeline {
        repeatCount: Timeline.INDEFINITE
        interpolate: false
        keyFrames: [
            KeyFrame {
                time: duration
                action: runHandlers
            }
        ]
    }

    // Holds the list of handlers to run
    protected var handlers: function() [];

    /\*\*
     \* Add the function, represented by the handler argument, to the list
     \* of handlers.  These will run when the elapsed time, specified
     \* by the duration instance variable, expires.
     \*/
    public function registerHandler(handler : function()) : Void {
        for (func in handlers) {
            if (handler == func) {
                return;  // handler already registered -- skip
            }
        }
        insert handler into handlers;
    }

    /\*\*
     \* Remove the function, represented by the handler argument, from
     \* the list of handlers.
     \*/
    public function unRegisterHandler(handler : function()) : Void {
        delete handler from handlers;
    }

    protected function runHandlers() : Void {
        for (handler in handlers) {
            handler();
        }
    }
} 

To test this class out, we'll add a run() function at the script level.  The run() function creates a Timer instance and registers two handler functions, decrementTenthsRemaining() and processTicks().  Here's the code:

function run() : Void {
    var t = Timer {};
    var tenthsRemaining = 100;
    var decrementTenthsRemaining = function() : Void {
        tenthsRemaining -= 1;
    }
    var processTick = function() : Void {
        if (tenthsRemaining mod 10 == 0) {
            println("seconds left: {tenthsRemaining / 10}");
        }
        if (tenthsRemaining == 0) {
            t.timeline.stop();
        }
    };
    t.registerHandler(decrementTenthsRemaining);
    t.registerHandler(processTick);
    t.timeline.play();
}

And this is the output from the run:

seconds left: 9
seconds left: 8
seconds left: 7
seconds left: 6
seconds left: 5
seconds left: 4
seconds left: 3
seconds left: 2
seconds left: 1
seconds left: 0

Shameless Promotion:  Keep up to date with the latest status of our upcoming JavaFX Book entitled JavaFX: Developing Rich Internet Applications at jfxbook.com.


Tuesday Feb 17, 2009

JavaFX Book Coming to a Theatre Near You

Despite the considerable attention JavaFX has garnered, publications (i.e. books) that discuss JavaFX in any detail are few and far between, and anything that has been published, as good as it may have been, is unfortunately hopelessly out of date.  The reality is that up until recently, the JavaFX platform has been a moving target.  With the advent of JavaFX 1.1 however, the platform has stabilized to the point where you should begin to see legitimate books appearing on the subject.

Jim Clarke, Eric Bruno and I have been working steadfastly on a book entitled JavaFX: Developing Rich Internet Applications, which should be among the first -- if not the first -- of these new books.  From our standpoint the content is nearly finished.  What remains is the editing and publication process which takes a few additional months to complete.  Plans call for this book to be available in time for the JavaOne 2009 Conference, if not earlier.

We also plan on making rough cuts of certain chapters available on Safari.  As soon as these are ready, we'll let you know.  Finally, check out our website, jfxbook.com, dedicated to the book.  There you will find additional resources that accompany the book, including sample code and applications.  One such application is a JavaFX version of the popular Sudoku game, pictured below.

Visit jfxbook.com and give it a try.

Wednesday Jan 14, 2009

Overhead in Increasing the Solaris System Clock Rate

In a previous entry entitled Real-Time Java and High Resolution Timers, we discussed how Sun's Java Real-Time System requires access to timers with a resolution greater than the default 10ms to do anything really interesting.   It was also stated that most modern processors have an APIC or Advanced Programmable Interrupt Controller which supports much finer-grained clock tick rates.

Unfortunately there are many instances where a system does indeed contain an APIC, but it is not exposed by the BIOS.  Furthermore, we've found that some of the embedded, low-power x86-based processors do not contain an APIC at all.  For an example, take a look at the AMD Geode LX 800 based fit-PC Slim.

So if you wanted to utilize higher resolution timers for this class of system, you'd have to resort to alternative methods.  Solaris and OpenSolaris provide two /etc/system parameters called hires_tick and hires_hz to facilitate increasing your default system clock tick.  By adding the following line to /etc/system, you'll increase the system clock tick rate from the default of 100 per second to 1,000 per second, effectively changing the clock resolution from 10ms to 1ms.

   set hires_tick=1

If you want to further increase the clock resolution, you can do so via the hires_hz system tunable parameter.  Although formally unsupported, it does work.   In order to, for example, increase the clock tick rate to 10,000, add this to /etc/system:

    set hires_tick=1
    set hires_hz=10000

To achieve the desired effect above, you must include both the hires_tick assignment in addition to setting the hires_hz parameter.

These modifications do not come without side-effects, and depending upon the hardware in question and the granularity of the desired timer resolution, they could be significant.  In short, it takes additional CPU cycles to field all those timer interrupts.  So I thought it'd be interesting to see what effect changing the clock tick rate had on two separate systems.   Here they are:

 System  fit-PC Slim
 Panasonic Toughbook CF-30 (Revision F)
 CPU  AMD Geode LX 800 (500 Mhz)
 Intel Core 2 Duo L7500 1.60GHz
 OpenSolaris Version
 snv_98  snv_101b

The measuring tool used for this simple experiment is vmstat(1m).  Solaris aficionados will likely point out that there are much more accurate alternatives, but I think vmstat(1m) gives a decent feel for what's going on without having to expend a whole lot of extra energy.  In particular we'll look at the following fields returned by issuing a 'vmstat 5', and picking one of the interim samples:

  • in - interrupts per second
  • us - percentage usage of CPU time in user space
  • sy - percentage usage of CPU time in system space
  • id - percentage usage of CPU time idling

The sum of (us + sy + id) should approximate 100%.  The table below shows sample vmstat output on various clock tick settings for our two hardware platforms.

Clock tics/sec
 100
 1000  10000  100000
/etc/system settings
 none (default)
 set hires_tick=1
set hires_tick=1
set hires_hz=10000
set hires_tick=1
set hires_hz=100000
vmstat(5) sample fit-PC
 in: 201
 us: 0
 sy: 1
 id: 99
 in: 2001
 us: 0
 sy: 5
 id: 95
 in: 19831
 us: 0
 sy: 43
 id: 57 
n/a

vmstat(5) sample CF-30

 in: 471
 us: 0
 sy: 0
 id: 99
 in: 2278
 us: 0
 sy: 1
 id: 99
 in: 20299
 us: 0
 sy: 5
 id: 95
 in: 200307
 us: 0
 sy: 21
 id: 79

Notes/Conclusions:

  • The vmstat(5) was let run for about a minute.  The output above shows one of the typical 5 second samples.  The other 5 second samples are almost identical.
  • The user (us) CPU time numbers give us a reasonable idea that these systems were predominantly in an idle state when being sampled.
  • The number of interrupts serviced per second is directly proportional to the clock tick rate.
  • And of course, the larger the number of interrupts, the more system CPU time is required.
  • The amount of overhead taken up by increasing the clock rate is a function of system capability.  The CF-30 not only has a much faster processor, it also has two cores to share the interrupt load.  As such it could accommodate a much higher clock tick rate.  For the fit-PC, performance is impacted profoundly even at modest clock tick rates.

Thursday Dec 04, 2008

Why JavaFX is Relevant

This week marks the formal release of JavaFX 1.0.  During the interval between the early marketing blitz and now, we've heard a lot from our friends in the press and the blogosphere, and in many instances what they had to say was not very pretty.  Some think the Rich Internet Application platform battle lines are already drawn between Adobe and Microsoft, and dismiss Sun as having arrived too late to the party.  Others opine that JavaFX's underlying Java platform is so yesterday.  In fact Java is the primary reason why JavaFX will, much to the chagrin of many, receive serious consideration.  Here's why:

  • Java is ubiquitous.  It is the proven, de-facto platform for web-based deployment.  On the desktop, it is estimated that approximately 90% of PCs have Java installed. In fact the major PC OEMs have seen fit to install it for you out of the box.  In the mobile world, Java is the dominant deployment platform.  Billions (that's with a 'b') of devices run Java.
  • The Java development community is arguably the largest on the planet.  Java gained initial widespread acclaim as a productive development environment, and continues to do so.  As JavaFX is an evolution of Java and seamlessly integrates with it, human nature tells us that individuals will naturally want to work with and leverage that which they already know and are familiar with.
  • Alternatives are still no match for the Java Virtual Machine.  It has been extensively studied, vetted, scrutinized, poked, prodded, abused, cloned, and optimized more than any other virtual machine in the history of computing. And just in case you're under the impression that the Java Virtual Machine is limited only to the Java (and now JavaFX script) programming languages, think again.  At last count there were over 200 projects integrating modern dynamic languages to the Java VM.  That list includes the usual suspects like PHP, Ruby, JavaScript, Python, and [insert your favorite language here].
  • The amount of Java Standard Edition online updates is staggering.  We know.  We supply the downloads.  And once a desktop is upgraded, it will be able to take full advantage of the features JavaFX brings to the table, effectively trivializing the barriers to entry.
Many of our most talented folks have been working feverishly to reach this milestone.  That being said, there's still lot's more work to do.  But we're off to a real nice start.  Check out http://javafx.com.  Hmm.  looks like the site is a little sluggish right now.  Maybe we underestimated all the interest?

Wednesday Nov 05, 2008

OpenSolaris on the Smallest System Yet?

One of my compadres forwarded me this link from CompuLab, an Israeli company which has created this real small, very energy efficient PC.  It may just be the smallest full-featured system to date. 

I'm a sucker for these types of things, and thought it would be interesting to get a reduced footprint version of OpenSolaris up and running on it.  Haven't gotten around to playing with wi-fi, or for that matter the graphics (as the reduced footprint version of OpenSolaris has no windowing system), but every indication points to a system that doesn't require any magic incantations to get OpenSolaris up and running.  Here are some of the specs

CPU: AMD Geode LX800 500MHz     
Chipset: AMD CS5536
Display: Integrated Geode LX display controller up to 1920x1440  
Memory: 512MB DDR 333MHz soldered on-board
Hard disk: 2.5” IDE 60GB
Ports:
 RJ45 Ethernet port 100Mbps
 WLAN 802.11b/g 54Mbps
 3 x USB 2.0 HiSpeed 480Mbps
 mini RS-232 (cable available from CompuLab)
 VGA DB15
 Stereo line-out audio (headphones)
 Stereo line-in audio / Mic


The system has a serial port, and upon request, CompuLab will provide the required custom serial cable.  By instructing GRUB to redirect the console to the serial port, you can connect the serial cable to another system and communicate directly with the console.  To get a rough idea of how to accomplish this, check here.  The screenshots below show that from an OpenSolaris terminal window, you can use the tip(1) command to accomplish this.

So the question is, how minimal is this configuration?  Issuing `ps -ef' shows that only 14 processes are currently running, of which I'm sure there's an opportunity to eliminate one or two if need be.

To give you an idea of how much RAM is currently consumed, here's what issuing the 'memstat' macro under mdb(1m) yields:

Who says Solaris can't fit in small places?

Wednesday Jul 23, 2008

Fast Booting Solaris

A veteran Java ONE keynote presenter, Perrone Robotics has developed some real interesting technologies which take the concept of using autonomous (i.e. unmanned) vehicles to a whole new level.  One of their key ingredients is the MAX software platform which utilizes common commercially available components to enable Perrone to very quickly and cost-effectively retrofit nearly any vehicle in short order.


The MAX robotics platform runs on a (roughly 4" x 6") low-power PC board atop Solaris and Sun's Java Real-Time System (Java RTS).  This combination gives Perrone the ability to leverage the huge Java development community, and assures that their critical software components behave in a predictable and deterministic fashion.

During the Java ONE 2007 conference, I was speaking with Paul Perrone about the notion of creating a minimized version of Solaris over which his platform might run.  The helicopter pictured above, boots from a relatively small (4-8Gb)  IDE flash drive, where standard Solaris takes up a substantial chunk.  It leaves them precious little space to collect valuable information like telemetry or terrain data.  Paul asked to revist this idea for a future project.  That's where we left off.

Not that we've ignored them since, but it wasn't until a year later that small Solaris reared its head again.  In addition to saving space, their main interest in this environment was in seeing how much faster Solaris might boot up.  The ability to be fully functional from power-up in as short a time as possible is of critical importance.

So before investigating what advantages there might be, let's provide some background information:

Hardware

Two separate systems were used, and for argument's sake, represent two ends of the x86 compute spectrum. 


Embedded Profile
Modern Profile 
System iGologic i84810
Panasonic Toughbook CF-30 (Rev. F)
CPU 1GHz Celeron M
Core 2 Duo L7500 1.60GHz
RAM 512MB 1GB
Disk 4GB Flash IDE Drive
Solid State SATA Drive

Operating Systems

Minimal configurations were created for Solaris 10 08/07 (Update 4) and OpenSolaris Nevada build 93.  These configurations boot entirely into RAM and consume less than 100MB ramdisk space.  With a little more effort they can be may significantly smaller.  The original blog post describing the environment is here.   You can download the framework for these hardware/OS combinations here, and can get a feel for the build environment by taking a look at this README.

Definitions

Within the context of this discussion, here are the key terms along with their meanings.

Total Boot Time: This is the time it takes from power-up till a user is prompted to log in.  Typically for a full Solaris installation, the windowing system must first start up and present the user with a login screen.  In a minimal Solaris environment, there is no windowing system.  Instead, the total boot time is defined as the time it takes from power-up till a user is prompted with the console login: prompt.

POST Time: POST or Power On Self Test is the time taken by the system at pre-boot to handle things like diagnostics,  BIOS and device initialization.  For this discussion, we'll define POST time as the time it takes from power-up until the user is presented with a GRUB boot menu.  We call out this segment of the total time because in many cases we are at the mercy of the PC/BIOS manufacturer and can't overly influence how quickly or slowly this proceeds.

Solaris Boot Time: The time it takes from being presented with a GRUB boot menu till a Solaris user is prompted to log in.  Again, depending upon whether a windowing system is configured or not, this may represent the time it takes to be prompted with a login screen or console login: prompt respectively.  This represents the segment of time that we can influence.

We can conclude from these definitions that:

   Total Boot Time = POST Time + Solaris Boot Time

Results

Embedded Profile: iGoLogic i84810 system 

OS
Post Time
Solaris Boot Time
Total Boot Time
 Solaris 10 08/07
13 sec
26 sec
39 sec
 OpenSolaris Nevada Build 93
13 sec
18 sec
31 sec 

Modern Profile: Panasonic Toughbook CF-30

OS POST Time
Solaris Boot Time
 Total Boot Time
 Solaris 10 08/07
 6 sec
 18 sec
 24 sec
 OpenSolaris Nevada Build 93
 6 sec
 9 sec
 15 sec

Conclusions/Notes

1. These times were taken by hand with the stopwatch feature on my Timex.  If anything, the times might be slightly shorter than actually recorded as there is a human delay in reacting to seeing the necessary prompts.  I ran the tests several times, and the same numbers consistently appear.

2. The version of the OS appears to matter a lot, as OpenSolaris nvx_93 boots up nearly twice as fast as Solaris 10 U4 on the same hardware.

3. The type of I/O device subsystem seems to be a big factor too.  For example, by switching out the IDE Flash Drive with a 5400 RPM IDE hard disk, i84810 total boot time decreased by about 6 seconds for both Solaris 10 and OpenSolaris. 

4. The minimal Solaris environment is currently only available in 32-bit mode.

5. With relative ease, Solaris can be configured to boot in less that 10 seconds on modern x86 hardware.  My unofficial record stands at 9 seconds (or slightly less).   No doubt it could boot faster on more robust hardware (eliminating POST time).  Any takers?

Tuesday Jun 17, 2008

Real-Time Java in a Zone

As is often the case, Sun's technologies and offerings are being applied in ways which we hadn't necessarily anticipated.  Yet another example has reared its head in the govenrment/military space where customers have expressed interest in using Sun's Java Real-Time System with Solaris Trusted Extensions.  As it stands right now, Java RTS will neither operate nor install within the confines of such an environment.  Let's investigate why this is so, and see what current opportunities there are for working around this shortcoming.

So what is it that causes Trusted Extensions and Java RTS not to play together nicely?   It happens to revolve around Trusted Extension's extensive usage of Solaris zones to limit access between differing security levels.  Solaris packages must be specifically configured to accommodate zones, which has yet to formally take place with Java RTS.  As zones are a core component of Solaris, we can, for the sake of simplicity, just use standard Solaris to demonstrate how we can work around this temporary limitation.  These modifications should apply to Trusted Extensions just as well.   To get Java RTS to work within a zone, follow these steps:

1. Install the Java RTS cyclic driver (only) in the global zone.

global# pkgadd -d . SUNWrtjc

Processing package instance <SUNWrtjc> from </cyclic/src/packages_i386>
## Installing package <SUNWrtjc> in global zone

Java Real-Time System cyclic driver(i386) 2.1.0,REV=2008.06.13.16.10
Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Using </> as the package base directory.
## Processing package information.
## Processing system information.
   5 package pathnames are already properly installed.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user
permission during the process of installing this package.

Do you want to continue with the installation of <SUNWrtjc> [y,n,?] y

Installing Java Real-Time System cyclic driver as <SUNWrtjc>


2. Create a zone called 'rtjzone':
global# mkdir -p /zone
bash-3.00# zonecfg -z rtjzone
rtjzone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:rtjzone> create
zonecfg:rtjzone> set zonepath=/zone/rtjzone
zonecfg:rtjzone> verify
zonecfg:rtjzone> commit
zonecfg:rtjzone> exit
global# zoneadm list -vc
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   - rtjzone          configured /zone/rtjzone                  native   shared
global# zoneadm -z rtjzone install
Preparing to install zone <rtjzone>.
Creating list of files to copy from the global zone.
Copying <6984> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1074> packages on the zone.
Initialized <1074> packages on zone.
Zone <rtjzone> is initialized.
Installation of <1> packages was skipped.
The file </zone/rtjzone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.


3.  Modify the zone to allow access to the cyclic device, and to allow additional privileges

global# zonecfg -z rtjzone
zonecfg:rtjzone> set limitpriv=default,proc_priocntl,proc_lock_memory,proc_clock_highres
zonecfg:rtjzone> add device
zonecfg:rtjzone:device> set match=/dev/cyclic
zonecfg:rtjzone:device> end
zonecfg:rtjzone> verify
zonecfg:rtjzone> commit
zonecfg:rtjzone> exit

global# zoneadm -z rtjzone reboot
Note:  One privilege that is useful with Java RTS is sys_res_config.  This is typically used to assign a real-time process to a processor set.  Unfortunately zones cannot currently be given this privilege.  You can however, from the global zone, assign a processor set to a zone, which might be a reasonable workaround.

4.  Get a copy of the SUNWrtjv package and modify it so that it will install in a zone.  The postinstall script and postremove script must replaced with those provided by these hyperlinks just mentioned.

rtjzone# cd /scratch
rtjzone# ls
SUNWrtjv	postinstall	postremove
rtjzone# cp postinstall SUNWrtjv/install/
rtjzone# cp postremove SUNWrtjv/install/

5. Modify the SUNWrtjv pkgmap file with the appropriate sizes, checksums and last access dates.  The source code for a sample C program, called pkgmap_info, which prints out the necessary information, can be found here.

rtjzone# cd SUNWrtjv
rtjzone# grep post pkgmap 1 i postinstall 5402 42894 1194344457 1 i postremove 2966 34854 1194344457 rtjzone# cp pkgmap_info.c /tmp rtjzone# cc -o /tmp/pkgmap_info /tmp/pkgmap_info.c rtjzone# cd /scratch/SUNWrtjv/install/ rtjzone# /tmp/pkgmap_info postinstall postinstall 5820 9488 1213727841 rtjzone# /tmp/pkgmap_info postremove postremove 3092 45039 1213727538
Replace the postinstall and postremove entries in the pkgmap file with those produced by the pkgmap_info program.  You cannot simply use the example data above because the last access times will not match.  Doing so will cause the install to fail.


6. Install the Java RTS SUNWrtjv package inside the zone.

rtjzone# cd /scratch
rtjzone# pkgadd -d . SUNWrtjv

Processing package instance <SUNWrtjv> from </scratch>

Java Real-Time System runtime environment(i386) 1.5.0_13_Java-RTS-2.0_01-b08_RTSJ-1.0.2,REV=2007.11.06.12.47
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.

Where should this package be installed? (/opt): /opt

To achieve predictable temporal behavior, the Java Real-Time System
must be granted access to a number of privileged Solaris resources.
By default, access to these privileged resources is only granted to
the superuser (root). They can also be granted to additional users
by creating a rights profile, that is, a collection of authorizations,
that can later be assigned to an administrative role or directly
to a user.

As part of this package installation, a local 'Java Real-Time System User'
rights profile can be created on this machine.
This rights profile should NOT be created if such an action conflicts
with your computer security management policies. If unsure, contact
your system administrator or your computer security manager.
Also refer to the product's release notes for further details regarding
the privileges required by the Java Real-Time System.

Should a local 'Java Real-Time System User' rights profile be created? [y,n] (no): y
Using </opt> as the package base directory.
## Processing package information.
## Processing system information.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user
permission during the process of installing this package.

Do you want to continue with the installation of <SUNWrtjv> [y,n,?] y

...

## Executing postinstall script.
Creating the 'Java Real-Time System User' rights profile.

Refer to the 'System Administration Guide: Security Services'
documentation for further information regarding the way to assign the
'Java Real-Time System User' rights profile to users, groups, or
administrative roles using the Solaris Management Console, smc(1M) or
the usermod(1M), groupmod(1M) and rolemod(1M) commands.


Installation of <SUNWrtjv> was successful.


6.  Try running a real-time Java application in the zone.

rtjzone# /opt/SUNWrtjv/bin/java -version
java version "1.5.0_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13_Java-RTS-2.0_01-b08_RTSJ-1.0.2)
Java Real-Time System HotSpot(TM) Client VM (build 1.5.0_13-b08, mixed mode)


We hope to more formally support Solaris zones usage with Java RTS in the future.  In the interim this workaround can help you get started.  Many thanks to Jim Clarke, who did the lion's share of the legwork to find this solution.

Thursday May 29, 2008

Are Solaris RAM Disks Swappable?

As memory access is typically orders of magnitude faster than disk access, the idea of using a part of RAM as an in-memory storage device has been one of the earliest performance optimizations realized by the computer science community.  Even though today this type of optimization takes place transparently inside modern operating systems (via mechanisms like the disk buffer cache),  there are still circumstances where manually creating a RAM disk might still be quite useful.

A Solaris customer using the ramdiskadm(1m) utility to create a Ram disk device for a real-time application posed the following question: "Are RAM disks swappable?". From a real-time perspective, a RAM disk not only yields significantly better seek performance but also provides for more deterministic behavior compared to the traditional rotating disk platter.  The customer's concern here is that, under dire circumstances, is it possible for operating system swap out the RAM disk memory?  Needless to say, if a swap out occurred it would put a big crimp in the real-time application's predictability.

To get an idea of what's going on when a RAM disk is created, let's use the Solaris kstat(1m) kernel statitics utility to see how memory is being allocated.  First let's see what memory looks like before creating a RAM disk:

# pagesize
4096
# kstat -n system_pages | grep pagesfree
       pagesfree                       96334
# kstat -n system_pages | grep pageslocke      pageslocked                     26170

So, on this particular system, where a page is 4096 bytes,  there are currently 96334 pages free, and 26170 pages that are locked.  Now let's create a 50MB RAM disk:

# ramdiskadm -a rd 50m
/dev/ramdisk/rd
# ramdiskadm
Block Device                                                  Size 
Removable
/dev/ramdisk/rd                                           52428800    Yes 
# kstat -n system_pages | grep pagesfree
       pagesfree                       83507
# kstat -n system_pages | grep pageslocked
       pageslocked                     38988

Let's subtract the original number of pageslocked from the latest value and multiply by the pagesize:

# pagesize
4096
# bc
(38988-26170)\*4096
52502528
\^D

The increase in locked pages can be attributed to the creation of the RAM disk (50m + a small amount of overhead).  So yes, these pages are locked into memory.  But it would be nice to get a definitive statement on what pageslocked actually means.  According to McDougal, Mauro and Gregg's Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris, pageslocked is "Total number of pages locked into memory by the kernel and user processes".  Furthermore, the man page for plock(3C), a related library routine which enables programmers to lock memory segments, states that "Locked segments are immune to all routine swapping".  What's routine swapping?

Hmm.  Anybody care to shed some light on this?

Monday May 19, 2008

Working Around a Simple Directory Server Install Issue with Windows

While working with a customer evaluating Sun's Directory Server Enterprise Edition 6.3 for Windows, we came across a problem during very basic install and setup.  For posterity's sake, I thought it might make sense to document the issue and offer a few potential workarounds. 

How to replicate

1. Install the directory server.  In this case the downloadable image has been unzipped into the c:\\tmp\\dsee63\\DSEE_ZIP_DISTRIBUTION directory

  C:\\tmp\\dsee63\\DSEE_ZIP_Distribution> dsee_deploy --no-inter -i /tmp/ds63                                                                              
Unzipping sun-ldap-shared-l10n.zip ...
Unzipping sun-ldap-directory.zip ...
Unzipping sun-ldap-console-gui-help-l10n.zip ...
Configuring Cacao at /tmp/ds63/dsee6/cacao_2
...
You can now start your Directory Server Instances
You can now start your Directory Proxy Server Instances

2. Create a new Directory Server instance and start the instance

  C:\\tmp\\dsee63\\DSEE_ZIP_Distribution>cd \\tmp\\ds63
C:\\tmp\\ds63> set PATH=c:\\tmp\\ds63\\ds6\\bin;c:\\tmp\\ds63\\dsrk6\\bin;%PATH%
C:\\tmp\\ds63>dsadm create /tmp/instance
Choose the Directory Manager password:
Confirm the Directory Manager password:
Use 'dsadm start '/tmp/instance'' to start the instance
C:\\tmp\\ds63>dsadm start /tmp/instance
Waiting for Directory Server instance 'C:/tmp/instance' to start...
Directory Server instance 'C:/tmp/instance' started: pid=2144

3. Create a suffix.  This is where the installation fails:

C:\\tmp\\ds63>dsconf create-suffix -h localhost -p 1389 dc=example,dc=com
Enter "cn=Directory Manager" password:
Unable to bind securely on "localhost:1389".
The "create-suffix" operation failed on "localhost:1389".

The problem here is described in The Sun Java System Directory Server Enterprise Edition 6.3 Release Notes:

On Windows systems, Directory Server does not allow Start TLS by default.

This issue affects server instances on Windows systems only. This issue is due to performance on Windows systems when Start TLS is used.

To work around this issue, consider using the -P option with the dsconf command to connect using the SSL port directly. Alternatively, if your network connection is already secured, consider using the -e option with the dsconf command. The option lets you connect to the standard port without requesting a secure connection.

 

Two Potential Workarounds

Workaround 1: Issue the 'dsconf create-suffix' command by directly connecting to the SSL port (-P 1636)

C:\\tmp\\ds63>dsconf create-suffix -h localhost -P 1636 dc=example,dc=com
Certificate "CN=TECRA-A1, CN=1636, CN=Directory Server, O=Sun Microsystems" pres
ented by the server is not trusted.
Type "Y" to accept, "y" to accept just once, "n" to refuse, "d" for more details
: Y
Enter "cn=Directory Manager" password:

Workaround 2: Modify the ds-start-tls-enabled attribute that is stored in the directory server configuration.

a. Create a file, say c:\\tmp\\modify.ldif which looks like:

  dn: cn=config
  changetype: modify
  replace: ds-start-tls-enabled
  ds-start-tls-enabled: on

b. Issue an ldapmodify command something like this:

  C:\\tmp>ldapmodify -h localhost -p 1389 -D "cn=Directory Manager" -w password < c:\\tmp\\modify.ldif
modifying entry cn=config

c. Confirm modification via ldapsearch command:

  C:\\tmp>ldapsearch -b "cn=config" -h localhost -p 1389 -D "cn=Directory Manager" -w password "cn=config" ds-start-tls-enabled
version: 1
dn: cn=config
ds-start-tls-enabled: on

dn: cn=config,cn=chaining database,cn=plugins,cn=config

dn: cn=config,cn=ldbm database,cn=plugins,cn=config

c. Stop and restart the directory server instance

  C:\\tmp\\ds63>dsadm stop /tmp/instance
Directory Server instance 'C:/tmp/instance' stopped

C:\\tmp\\ds63>dsadm start /tmp/instance
Directory Server instance 'C:/tmp/instance' started: pid=3560

d. Try creating a suffix with the standard port (1389):

  C:\\tmp\\ds63>dsconf create-suffix -h localhost -p 1389 dc=example1,dc=com
Enter "cn=Directory Manager" password:

Note:  Directory Server Enterprise Edition 6.3 is supported on Windows Server 2003, but not for Windows XP.  Although not formally supported, it is possible to experiment with XP.

Tuesday May 06, 2008

Using Solaris to Burn the OpenSolaris 2008.05 CD Image

If you're using Solaris or OpenSolaris as a platform to download and burn the recently announced OpenSolaris 2008.05 image, the instructions found here may need a slight modification.  After downloading the Live CD, you are instructed to use the cdrw(1) command as follows to burn the image to CD:

cdrw -i /path/to/iso/image/os200805.iso

Here's what might happen on your system when you attempt to use the 'cdrw -i' command:


The cdrw(1) command fails because it thinks the ISO image is larger than the default capacity of the CD.  The solution to this problem is explained in these excerpts from the cdrw(1) man page:

"Most commercially available drives allow writing beyond 74 minutes as long as the media has the capacity (such as 80–minute media). However, the ability to write beyond 74 minutes might not be supported by the drive in use. If the drive supports this feature, then use the -C option to indicate that the tool should rely on the capacity indicated by the media.

"-C [option] Uses stated media capacity. Without this option, cdrw uses a default value for writable CD media, which is 74 minutes for an audio CD, 681984000 bytes for a data CD, or 4.7 Gbytes for a DVD." 
This time let's invoke 'cdrw' with the -C option.


Voila.


Monday Apr 21, 2008

Modifying and Respinning a Bootable Solaris ISO Image

As an adjunct to the previous blog post, a slightly customized boot environment capable of enabling serial console kernel debugging was required to diagnose Solaris install problems.  The post itself mentioned that a nice way to accomplish this was to set up PXE network boot via Solaris jumpstart.  It is indeed flexible and enables one to experiment with modifications and quickly test whether they perform as expected.  The one downside to this environment is that an additional DHCP/TFTP boot server has to be configured.  To eliminate that service, you could, once the customizations are well understood, simply create a new version of the Solaris install DVD with your customizations.  Let's run through the process for a simple example.

1. Get a Solaris DVD.  For this example, we'll use an interim build of the upcoming Solaris 10 Update 5.

2. Extract the the entire contents of the DVD.

# lofiadm -a /export/iso/s10x_u5b10_dvd.iso
/dev/lofi/1
# mkdir -p /iso
# mount -r -F hsfs /dev/lofi/1 /iso
# cd /iso
# mkdir /export/modified-s10x_u5b10
# find . -depth -print | cpio -pudm /export/modified-s10x_u5b10
4516208 blocks

3. Modify the content contained in /export/modified-s10x_u5b10.  In this case, we'll change the boot/grub/menu.lst file found in this directory to look like:

#
#pragma ident "@(#)install_menu 1.1 05/09/01 SMI"
#
default=0
timeout=30
serial --unit=0 --speed=9600
terminal serial
title Solaris Serial Console ttya
kernel /boot/multiboot kernel/unix -B install_media=cdrom,console=ttya
module /boot/x86.miniroot

4. Issue the following magic incantation to create a bootable ISO image based on the contents of the /export/modified-s10x_u5b10 directory.

# mkisofs -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -R -L -r -D -U \\
-joliet-long -max-iso9660-filenames -boot-info-table -o \\
/export/iso/modified-s10x_u5b10.iso /export/modified-s10x_u5b10

...

Size of boot image is 4 sectors -> No emulation
1.80% done, estimate finish Fri Apr 18 15:55:13 2008
2.25% done, estimate finish Fri Apr 18 15:59:07 2008
2.70% done, estimate finish Fri Apr 18 15:58:37 2008
3.15% done, estimate finish Fri Apr 18 15:58:48 2008
...

98.43% done, estimate finish Fri Apr 18 15:59:37 2008
98.88% done, estimate finish Fri Apr 18 15:59:37 2008
99.33% done, estimate finish Fri Apr 18 15:59:36 2008
99.78% done, estimate finish Fri Apr 18 15:59:36 2008
Total translation table size: 2048
Total rockridge attributes bytes: 3075577
Total directory bytes: 18921472
Path table size(bytes): 148014
Max brk space used 1b24000
1112471 extents written (2172 MB) 

Voila!  Acknowledgements to Tom Haynes and this blog post which served as an excellent guide.

Monday Apr 14, 2008

Enabling Remote Console Debugging of Solaris x86 Boot/Install

Our partners do a fair amount of business supplying ruggedized Solaris-powered Panasonic Toughbook computers to their US government/military customers.  As a regular part of the product cycle, Sun usually works with both the integrators and Panasonic to assure that as new models become available, Solaris runs on these systems properly.  Furthermore, when we can get our grubby little hands on the systems, we'll run them through our certification suite of tests and formally place them on the Solaris Hardware Compatibility List.  As an example, here's the certification report for one of the versions of the Panasonic Toughbook CF-29.

Panasonic recently introduced a new version of the Toughbook CF-30 (referred to as revision F) which tweaks some of the computer subsystems resulting in an all-too-familiar scenario: namely, these differences cause the current version of Solaris to fail to install.  Note: Solaris is not alone here, all Operating Systems must continually play this cat and mouse game to support the latest hardware/firmware advances.

Our initial hypothesis lead us to believe that the problem was related to the introduction of the Intel ICH8 SATA chipset.  So we called on some of our Solaris I/O experts, based out of Sun's Beijing office, to take a peek at what was going on.  As the laptop is currently in New York, we needed a way for folks half way around the world to have access to this system.  There are lots of mechanisms available to remotely diagnose systems, what's somewhat unique here is the following: (1) the diagnosis takes place very early in the boot stage, way before any windowing or networking is set up and (2) The system in question is a laptop, not a server, where things like Lights Out Management (LOM) service processors are non-existent.

The solution here was to utilize decades old RS-232 technology combined with some of features of the GRUB bootloader.  Here are two requirements needed:

  • A serial connection must be established between the system to be diagnosed, which will be referred to henceforth as the target, and the system which accesses the target which we'll refer to as the remote host.   Unfortunately most laptop manufacturers have seen fit to eliminate serial port connectors in lieu of using USB to serial converters as a replacement technology.  At the early stages of boot, USB/serial capability is not yet available, so these systems are not good candidates for this type of diagnosis.  Thankfully the target in question here, the Panasonic CF-30 Toughbook, still comes with a serial port.
  • A Jumpstart environment capable of installing Solaris on x86/x64 systems is strongly recommended.  As part of the process described below, we'll be modifying the target's GRUB environment  (menu.lst).  If you chose to use a DVD boot/install environment instead, you'd need to modify and burn a new DVD for each change made to the target's boot environment.  It took a bit of time to find the right incantations in the menu.lst file to get what was needed here; continually re-burning DVDs would have been excruciating.  This exercise is left to the reader, here's a good start to understanding the jumpstart setup process.

Here's how to set up the remote console environment:

1. A null modem cable must be physically connected between the remote host and target.  The most common cable required will be a DB-9 female-to-female connector.  Your configuration may vary.

2. Check the BIOS of the remote host and target and make sure serial ports are enabled.

3. Running Solaris on the remote host, we'll be using the tip(1) command to access the target via serial port.  Edit the /etc/remote file and look for the hardwire: entry.  Modify it to look like this:

hardwire:\\
:dv=/dev/term/a:br#9600:el=\^C\^S\^Q\^U\^D:ie=%$:oe=\^D:

4. As part of setting up a jumpstart install for the target, a series of files are created in the /tftpboot directory of the jumpstart server.  Under /tftpboot, there should be a custom menu.lst file for each managed install, suffixed by the unique network MAC address of the system in question.  For example, the network MAC address for the CF-30 in question is 0:b:97:db:c0:97.  The related /tftpboot file for the CF-30 turns out to be /tftpboot/menu.lst.01000B97DBC097. As your target will have a different MAC address, it's menu.lst  file will have a different suffix in the /tftpboot directory.  Edit that custom menu.lst file (for example, /tftpboot/menu.lst.01000B97DBC097) to look as follows:

default=0
timeout=30
serial --unit=0 --speed=9600
terminal serial
title Solaris_10 s10x_u5b10
kernel /I86PC.Solaris_10-1/multiboot kernel/unix -B console=ttya,install_media=192.168.1.5:/export/s10x_u5b10 -vkd
module /I86PC.Solaris_10-1/x86.miniroot

The key modifications here involve (1) inclusion of the serial --unit=0 --speed=9600 and terminal serial lines plus (2) additional arguments added to the kernel directive.  Grub is very fussy about the order and placement of arguments; playing around with these will likely change grub's behavior. 

5.  From the remote host, access the serial console of the target by issuing:

$ tip hardwire

6. Inside a terminal window, here's what the serial console looks like, after the system has been power cycled and runs through the POST sequence:


After the miniroot is loaded, you'll be presented with an mdb prompt and a screen which looks like this:

You can now issue mdb commands to diagnose.  In this scenario you should also be able to reboot the  system without any other manual intervention, like this:

 

Here's what issuing the mdb commands ':c' and '$c' look like in this environment.   From this simple  trace we can ascertain that the SATA drivers were never even loaded.  Turns out this is likely a VM problem.  Here's the filed bug report.


About

Jim Connors

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today