Friday Aug 03, 2007

System Profiling 101 : Getting started using sys_diag v.7.04

The following entry is a variation of an article that I created for this month's Sun "Technocrat" publication (Aug. 2007).

This posting demonstrates the art of system profiling (from a high level overview) by introducing a few sample screenshots of the sys_diag  .html  report (it's header, Dashboard, and Table of Contents).. demonstrating how in a few minutes, sys_diag can present you with an accurately depicted system profile !

Note, the .html report snapshot samples presented here, match the command line output from my previous blog postings (from the same run of sys_diag).

If you haven't had the chance to try out sys_diag yet, this should give you the highlights of what you can expect in the .html report header sections.

Enjoy and let me know what you think,


Real world PROFILING ..

So.. what is Profiling.. ?? ... Well, in the real world, you can define profiling in many ways.

.. from the "profile" of the person standing next to you (what you see from your vantage point), to the personality "profiles" that we've all heard of in psychology (characterization based upon key attributes) ..

In your standard dictionary you'll find a definition such as this (from :


  •  the outline or contour of the human face, esp. the face viewed from one side.
  •  a verbal, arithmetical, or graphic summary or analysis of the history,
    status, etc., of a process, activity, relationship, or set of
    characteristics: a biochemical profile of a patient's blood; a profile of national consumer spending.
  •  a set of characteristics or qualities that identify a type or category of person or thing: a profile of a typical allergy sufferer.
  • Psychology. a description of behavioral and personality traits of a person compared with accepted norms or standards.

System Profiling .. in the World of Computing

Well, in the world of  technology, and more specifically.. Computing, "profiling" takes on it's own connotation, though similar to many of the more technical definitions noted above.

To some, system profiling simply includes a high level summary of resource utilization and bottleneck identification of a system during some period of data collection (point in time or over a duration).

"Broad Spectrum" System Profiling ..

System profiling to me is the characterization of a system as a whole, given a set of data, either for one event/point in time, or over a duration.  This characterization goes beyond workload (as you'll typically hear the term "workload characterization"), which is why I call it "Broad (or Full) Spectrum" profiling, more broadly taking into account and including :

  • Configuration characteristics (of all sub-systems/components within the "application environment" being profiled).
  • System Performance metrics captured, reflecting the variations in system/subsystem activity measurements (utilization, contention, throughput, latency, etc...).
  • Workload Characterization : Details correlating the Workload that was ongoing during the data collection (workload characteristics.. TPS/ Response Times/ Mbps/ ...), with the measurements taken.   (beyond the internal system workload identified, External sources need to be correlated)
  • A Characterized (Summarized) "Profile" of overall System Efficiency and "health" based upon Performance Analysis findings (system/sub-system Avg/Peaks.., Utilization vs. Workload, etc ...)
  • Notable Events and/or Exceptions encountered from the data available.
  • ... etc ...

"Narrow Spectrum" System Profiling ..

This would be in contrast to "Narrow Spectrum" (Focused) System Profiling, where attention to detail is focused in a very "narrow" and specific area of interest for analysis (typically in determining a Root Cause where a specific bottleneck is know within a sub-system or specific component of the system).

Note the common themes of defining Requirements, the "Application Environment", etc.. as presented in my previous postings .. (eg.  "What is Performance ? .. in the Real World", etc..).. and likely to be common themes.. in the Real World.. ;)

Look for more details on this and much more in an up-coming blog entry more thoroughly delineating the distinction between Profiling, Workload Characterization, Performance Analysis, and Capacity Planning....  

For now, enjoy the following discussion on how sys_diag can have you profiling in no time at all ... :)

Profiling with sys_diag ...

Automating Solaris Performance and Configuration Analysis

In the arena of Performance and Configuration Analysis, the freely available Solaris utility “sys_diag” offers the capability to automatically capture this information in a single .html report (also .txt and .ps) after running one easy to use ksh script. Typically, several utilities/tools need to be run separately, requiring manual collection, aggregation and correlation of the data, prior to conducting the analysis of data.

Sys_diag automates this legwork, by running over 100 Solaris built in commands/utilities (depending on the parameters used) and presenting the data as a structured report with a summarized header, a color-coded “dashboard” (broken down by high level workload characterization, sub-system findings, followed by a Table of Contents), all with links to the corresponding report sections with detailed configuration and/or performance analysis findings.

sys_diag 's  HTML Report Header :

sys_diag 's  HTML Performance "Dashboard" :

 The following sample .html system performance “Dashboard” (a portion of which is shown below) reflects the 4 key sub-systems (CPU/Kernel Profiling, Memory, IO, and Networking) as a summarized depiction of sub-system “health”, based against a list of rules / thresholds that the captured data is compared to during post-processing.

These rules and thresholds are listed in the Performance Analysis section (Section #24) and can be easily tuned / modified to offer more stringent or lenient identification of performance exceptions that contribute to the Green/ Yellow / Red (OK / Warning / Critical) color-coded status within each dashboard section. Within each section are listed the key performance metrics and a summary of exceptions, along with Average and Peak (High Water Mark) values present during the collection/sampling period. At the end of each section is a list of key “links” to the substantiating detailed data analysis within the report.  

\*Click to Enlarge\*

When run for performance data gathering (-g or -G), 2 types of performance data is captured :

\* vmstat, mpstat, iostat, netstat, .. data for a duration (-T total_secs), captured at specified sampling rates (-I interval_#secs). The default duration is 5 minutes of data capture @ 2 second intervals if -I / -T are not specified.

\* 3 Point in Time detailed snapshots (beginning, mid point, end point). If -G is used, and         Solaris 10 is the OS, then Dtrace and detailed lockstat, pmap/pfiles, cputrack, ...  snapshots will be taken (beyond the core “-g” snapshots that include  ps, netstat -i, vxstat, kstat,  ...).

Sys_diag has been run on virtually all models of Sun systems running Solaris 2.6 or > (from x86 laptops up to fully loaded E25K's), offering extensive Solaris 10 configuration and performance data, including DTrace snapshots. It creates a single .tar.Z compressed archive (including all raw, snapshot and post-processed datafiles) that can be emailed/ ftp'd.. for performing system configuration and/or performance analysis off-site.. from virtually anywhere.

This is one of the key characteristics that sys_diag offers.. to save a LOT of time.. not requiring many separate manual runs / collection / correlation of data, or the need for any 3rd party tools, libraries, or agents to be installed on a system other than downloading the "sys_diag" ksh script itself. Virtually no learning curve is required for loading, running, and reflecting basic performance profiling, including high level sub-system bottlenecks (deeper root cause correlation might require some level of advanced system administration knowledge, though virtually all the data needed will have been already captured by sys_diag).

This utility has been used extensively in the field over the last several years, run on literally hundreds of production systems as part of escalation root cause analysis, in addition to providing the basis for dozens of Architectural and/or Performance Assessments
(including formal Capacity Planning / Benchmarking). Graphing of the data captured (vmstat, netstat...) is also easy to do using StarOffice as explained in the README file that sys_diag creates.

sys_diag 's  HTML Report "Table of Contents" :

The screenshot below shows the Table of Contents and related sections available within the .html report (\* Click to Enlarge \*) :


Although this tool isn't meant to replace long-term historical Performance Trending and Capacity Planning packages (Teamquest, etc..), it provides the foundation and basis for a very robust starting point (and actually is much better at point in time workload characterization and root cause analysis of bottlenecks, where very granular detailed data correlation is required).

Over the time that sys_diag has been posted on BigAdmin, many Sun customers around the globe have downloaded and commented positively on their experiences with it. For more information, or to download and try it out for yourself , the following URL's should help you get started :




The latest release of sys_diag (v.7.04) is available from BigAdmin
(unpackaged ksh) at :

sys_diag  is also available as part of the "SunFreeware" Distribution
(packaged with the README) at :

The following recent blog postings provide an extended overview of sys_diag and it's capabilities :

Solaris Performance Analysis and Monitoring Tools... at what cost ?...

What is sys_diag ?? .. Automating Solaris Performance Profiling and Workload Characterization.
sys_diag v.7.04 command line output ...


\*\*Note, read the ksh script header pages or the README file prior to using, and ALWAYS test first on a representative non-production system.. as is the best practice when making ANY production environment changes... ;)

(Copyright 2007, Todd A. Jobson) 

Add to Technorati Favorites

Friday Jan 26, 2007

Deception.. my Reflections..

Take a close look.. Not everything in Life is as it Appears...


Deception, Trust, .. and Online Security ...

In this fast-paced, internet-based culture that we're all a part of, there are a few things that we all need to be aware of.   Among the long list.. items that we all need to be cognizant of, relates to online deception in all it's forms (MySpace/ Yahoo 360 Identity Fraud, Identity Theft, Security of our personal information, players, con-men/women, etc..). 

Unfortunately, an ever growing majority of those on-line are NOT honest.. trustworthy.. or even who they say they are.    MOST of these deceivers are fulfilling some inner need (conning others, living vicariously, boosting personal confidence, a fling.. a cheap thrill.. who knows).. most compelled to continue.. or addicted to the virtual recognition or relationships they hold (not honestly knowing the person on the other end of the wire).

While we need to protect our identities and personal information, we also need to work on being honest once we establish a friendship (granted.. we do need some level of keeping our guard up initially to protect ourselves.. take things slow.. don't be in a rush to give out too much about yourself.. be careful).

No matter what.. nearly every page online is backed by a person that can be hurt.. so I've written the following poem below to reflect on the impact of deception  :

Reflections on ....


The Players, the Liars, the Pathological pseudo-sympathizers..
they prey on the vulnerable,

the weak.. those sad and meek.

Those lonely, or alone,
feeling broken.. or trapped at home,

They're the ones they seek..
those helpless and weak.

Fakes and impostors,
posers with on-line rosters.

Perception, just a reflection..
a Relative Reality of a friendship defection.

.. but Truth and honesty is the exception,
when deception is my reflection...
..a window on a world where trust is an election.

A Disillusioned Delusion
of deceitful collusion..

.. on-line love.. is it real,
or just an illusion ?

For many... Exploitation is their Specialization,
their duality of sexuality..

.. a vacation for excitation.
Their lapse.. in realization

.. to others.. real hell under magnification.

Expectations and elation,
all relate to a perverted sensation.

Temptation or Emancipation,
just casual exploits of the internet generation.

To be used and discarded..
left mentally scarred and broken hearted..

our fellings shattered... hearts guarded..
to the abuser ... a victory, to the victim... abuse.

Elusive invaders,
"users" .. the Invasive persuaders

Some weekend offenders,
others weathered pretenders,
all on the loose.

Predators that pray
and prey that Surrenders,

they cast their veil by day
the lure, who remembers,
most free to run their course.

Lets hope Justice prevails
..criminals punished, some in jails,
infused with a sense of remorse.

But before we claim victory.. elated,
let's ask ourselves..
Is Jealousy and Deception.. anticipated ?

Could it just be ..

.. the "Invasion" of an Illusion.. ?
.. or simply, the "Illusion" of an Invasion ?

To me, they're all Reflections..

.. Deception .. Our perception of an expectation.

Todd A. Jobson    1/22/2007

Copyright   2006-2007  Todd A. Jobson

Be careful.. and choose your words wisely.. Broken Hearts and shattered dreams are just a keystroke away.


TRUE Friends are ones that won't run away on a moment's notice..
when the going gets tough.. (those are fair weather friends that we each
have encountered time and time again.. always looking to fulfill their own needs,
irrespective of the needs that others might have.. theirs always coming first).

Everyone hears what you say.. but TRUE Friends LISTEN to what you
are thinking about..and feel.. there for you during your darkest moments.

Caring and compassion is a key, but honesty and trust is a must.

True friends can share fears and hopes, tears and their deepest aspirations.

When all else in your life seems bleak or is gone .. you've lost hope and feel all alone..
True Friends will be there.

True Friends won't try to sway one another with Political or Religious debates.. or throw
a temper tantrum when they don't get their way. They will forgive and say their sorry..
they won't let bad days or mood swings get between them and those around them.

True friends will always care about more than just themselves.. shining when the clouds set in...

Few of us appreciate what we have.. until we loose it..

Think about the simple things in life that we have lost appreciation for.. and give thanks
to those around us just as they deserve.. reminding them of just how much they mean to us..
because here today.. could easily be .. gone tomorrow..
(Our Children, our spouses / significant others, parents/ family, friends, .. our health, our careers, our finances, our freedoms, the safety that we take for granted, the abundance of food / shelter, ..our lifestyle.. etc.. etc..)

It all underscores the topic at hand ... Loyalty, Honesty, Integrity.. all those Virtues and Values that each of us should uphold in any relationship, whether it's a friendship.. or something more.. we all need to Give Thanks !

Ask yourself... If I (or another friend) needed to reach out in a time of need... would you be there ? ?
.. and do I appreciate what those friends have done for me.. or do I just expect it ??..

.. am I only fulfilling MY needs ??.. or am I also recipricating and fulfilling the needs of those that have been there for me ??..

Balance in life is a must. To sustain what we have.. everything needs to be a 2 way street.. give thanks and don't take life .. (including those that care about you) for granted !!

Your Trustworthy blogger ..

Todd J.

Copyright :  Todd A. Jobson  2006-2007


This blog does not reflect the viewpoint or opinions of Oracle or Sun Microsystems. All comments are personal reflections and responsibility of Todd A. Jobson, and are copyrighted from the posted year to current year, to that effect.


« April 2014