Test and development methodology

I plan to post this to the SST project site soon.  Any feedback or recommendation would be greatly appreciated.



Bridging the gap between SST 4.2 and SST:LV 5.0 is a matter of test, analysis, and fix – and once the project actually puts hands to keyboards, it shouldn't take long.

SST 4.2 support for Solaris 10 ended with Update 4. In fact it's the 4.2 source code that is available in the Mercurial repository:

hg clone ssh://hg.opensolaris.org/hg/sst/sst-lv

Earlier this year Glenn Brunette, one of the original SST developers, suggested a straight forward test-fix approach. Using the audit, harden, and undo modes of jass-execute we can easily identify which Audit and Finish scripts need to be updated to support later Solaris updates.

Test phase

The test-phase will identify which .aud and .fin scripts need to be updated.  It's possible that errors will crop up in the SST infrastructure functions, but most likely that the audit and finish scripts will be the sources of incompatibility with Solaris 10 versions subsequent to Update 4.  Scripts that produce errors will be added to a Fix List and posted to the SST Project page for updating.

There will be several Fix Lists, one for each platform type.  This can be expanded, but initial Fix Lists will include:

  1. SPARC

  2. SPARC, virtualized i.e., LDoms

  3. SPARC, platform-specific, M[34589]000, 25K / 15K

  4. x86

  5. x86, virtualized, VirtualBox

  6. x86, virtualized, xVM

  7. x86, virtualized, VMware, Fusion

  8. x86, virtualized, VMware, Workstation

  9. x86, virtualized, VMware, ESX / ESXi

In beta we'll try and test against as many hardware platforms as we can. SunFed employees can use the McLean lab, which is pretty well stocked.

Stand-alone context 

Our approach is to audit, execute, then audit again and inspect the results.  We'll want to test both the stand-alone and Jumpstart contexts, but since stand-alone is a little easier, we'll start there with the following steps:

  1. Audit

  2. Harden

  3. Audit

  4. Undo

  5. Audit

Step 1

The results of the first audit become a tentative baseline – tentative because the .aud scripts may be malfunctioning. But it's a good starting point. We'll start with the server-secure.driver Driver.

# /opt/SUNWjass/bin/jass-execute -V 4 -a server-secure.driver

Step 2

Next we run the driver in stand-alone hardening mode.

# /opt/SUNWjass/bin/jass-execute -V 4 -d server-secure.driver

We would expect the execution to produce no errors, nothing tagged with [ERR ].  Any .fin scrips that error out will be added to our Fix List.

Step 3

Now we run another audit.

# /opt/SUNWjass/bin/jass-execute -V 4 -a server-secure.driver

We would expect everything to pass, i.e. [PASS].  Any .aud scrips that error out will be added to our Fix List.

Step 4

Undo the last hardening.

# /opt/SUNWjass/bin/jass-execute -V 4 -u server-secure.driver

Any .fin scrips that error out will be added to our Fix List.

Step 5

Audit one more time.

# /opt/SUNWjass/bin/jass-execute -V 4 -a server-secure.driver

Since this audit is against a (supposedly) un-hardened system, we expect a lot of .aud scripts to [FAIL].  But the list of failures should be the same as the ones produced in Step 1.  Any deltas will be added to our Fix List.

Jumpstart context 

There's less to test from Jumpstart, but you'll need a working Jumpstart environment to do it. I'd recommend using JET since it really simplifies the whole process. I'll post a how-to for JET in the next several days.

(Jumpstart context methodology to be posted shortly.)

Analysis phase

After every run of SST, results can be found in:


Each step will produce various files of interest.  These files will feed our Fix List.

  • Step 1 - Audit
    • jass-script-errors.txt
    • jass-script-failures.txt
      • For comparison to the failures produced by Step 5
  • Step 2 - Harden
    • jass-script-errors.txt
    • jass-script-failures.txt
  • Step 3 - Audit
    • jass-script-errors.txt
    • jass-script-failures.txt
      • In this case we'd hope to see everything pass since we previously ran the hardening driver. Anything that filed here could be an indicator of a problematic .fin scrip from the previous step.
  • Step 4 - Undo
    • jass-script-errors.txt
  • Step 5 - Audit
    • jass-script-errors.txt
    • jass-script-failures.txt
      • By diffing this against the failures in Step 1 we can find malfunctioning .fin scripts from Step 4.

The scripts listed in these files, and the deltas from the Step 1 to Step 5 diff should produce a good (but not necessarily complete) Fix List.

I'll post the first Fix List (x86, virtualized, VirtualBox) in a few days.

Fix phase

For every script in the Fix List we'll need to trace the logic and fix the problems.  Some scripts will probably be trivial to fix, while others will require quite a bit more effort.  The SST project doesn't have its own mailing list yet; we'll just use our endorsing community's list, security-discuss@opensolaris.org.

To start I'd recommend adding a set -x to the .fin script, make a custom driver that calls only that script, and re-run.


Any recommendation or comments are welcome.



Post a Comment:
  • HTML Syntax: NOT allowed

Jason Callaway


« June 2016