Monday Dec 16, 2013

DSCP QoS tagging of DNS traffic in Solaris

A huge telco customer recently shared their desire to support DSCP marking of their core network infrastructure, specifically they asked if there's a way to do this in Solaris? Sure there is! Since DSCP marking is enforced by the router,  they wanted to know how to make their Solaris 10 systems conform to this. (Why Solaris 10 and not Solaris 11? Because their current roll-out reflects systems deployed before Solaris 11 came out) - AND they wanted to do this by the end of the year.  Given the opportunity and the fast-approaching deadline, I looked into this further by jointly teaming up with Krishna Yenduri of our Solaris network developer team, a team that focuses on network protocols and related cloud/virtualization technologies. Wanted to share what needs to be configured - and how easy it actually is.

So the customer inputs were:

  • Application: ISC BIND 
  • Requirement: Every DNS packet has to be tagged with value of 32, every DNS query needs to be tagged. 
  • Platform: x86 systems, Sun x4270 M2 servers running Solaris 10 

DSCP marking is supported in Solaris's implementation of support of IPQoS.  It is one of the most used features of IPQoS -- we have a number of scenarios of uses of Sun Ray servers where this has been deployed.   The way to configure IPQoS in Solaris 10 is thru the ipqosconf(1M)  interface.  Solaris 10 comes with a couple of sample config files located in /etc/inet/ipqosconf*, which show how to use the DSCP marking rules.  

If the need is to limit the marking to just the outgoing DNS queries, use the 'dport' parameter (that indicates the destination port), and set it to 53, as shown in the complete example that follows. Alternatively, if the need is to limit the marking to the outgoing DNS response, use the 'sport' parameter in the filter definition.

Contents of /etc/inet/ipqosinit.conf

fmt_version 1.0

# Mark the DSCP with code point AF32, 011100 = 28
action {
       module dscpmk
       name markAF32

       params {
               global_stats true
               next_action continue
               dscp_map {
               dscp_detailed_stats false

action {
       module ipgpc
       name ipgpc.classify

       params {
               global_stats true

       class {
               name myclass
               next_action markAF32
               enable_stats true

       class {
               name default
               next_action continue
               enable_stats false

       filter {
               name myfilter
               class myclass
               # DNS response
               sport 53
               direction { LOCAL_OUT }

The file gets saved as:   /etc/inet/ipqosinit.conf

Then the following operation instantiates the contents of the config file: 

# /usr/sbin/ipqosconf -s -a /etc/inet/ipqosinit.conf

and there is NO need to reboot.

How would you verify this?

If you're looking at the IPv4 network traffic, you would use:

#snoop -V -d <ipif-name> dst 53

This would show the IP header with the TOS value of '0x70', since the TOS value of 0x70 is the DSCP value of decimal 28).

Here's an example: 

host1 -> IP  D=XXX.XX.XXX.XX S=YY.YYY.YY.YYY LEN=74, ID=19461, TOS=0x70, TTL=255

Kindly note that 'TOS' is the same as the DSCP field - it happens to be an older RFC format and snoop(1M) does not know how to interpret it as DSCP, however ethereal/wireshark/tshark can perform such an interpretation properly (in case you'd be looking to verify the DSCP field value).

If you're handling IPv6 network traffic, beware that DSCP appears as a different field name: 'Traffic class' and if you're using wireshark, you should expect to see something like this:

Internet Protocol Version 6, Src: fe80::21b:21ff:fe88:fcd4 (fe80::21b:21ff:fe88:fcd4), Dst: fe80::21b:21ff:fe87:8d78 (fe80::21b:21ff:fe87:8d78)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0111 0000 .... .... .... .... .... = Traffic class: 0x00000070
        .... 0111 00.. .... .... .... .... .... = Differentiated Services Field: Assured Forwarding 32 (0x0000001c)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 33
    Next header: TCP (6)
    Hop limit: 60
    Source: fe80::21b:21ff:fe88:fcd4 (fe80::21b:21ff:fe88:fcd4)
    [Source SA MAC: IntelCor_88:fc:d4 (00:1b:21:88:fc:d4)]
    Destination: fe80::21b:21ff:fe87:8d78 (fe80::21b:21ff:fe87:8d78)
    [Destination SA MAC: IntelCor_87:8d:78 (00:1b:21:87:8d:78)]

For more information, please consult the System Administration Guide for IP Services - for Solaris 10 that book is located here:

For more general Oracle Solaris 11 How-To examples and content, please check this site out:

Saturday Nov 02, 2013

How to safely reboot via First Boot script

With the cost and performance benefits of the SPARC T4 and SPARC T5 systems undeniably validated, the banking sector is actively moving to Solaris 11.  I was recently asked to help a banking customer of ours look at migrating some of their Solaris 10 logic over to Solaris 11.  While we've introduced a number of holistic improvements in Solaris 11, in terms of how we ease long-term software lifecycle management, it is important to appreciate that customers may not be able to move all of their Solaris 10 scripts and procedures at once; there are years of scripts that reflect fine-tuned requirements of proprietary banking software that gets layered on top of the operating system. One of these requirements is to go through a cycle of reboots, after the system is installed, in order to ensure appropriate software dependencies and various configuration files are in-place. While Solaris 10 introduced a facility that aids here, namely SMF, many of our customers simply haven't yet taken the time to take advantage of this - proceeding with logic that, while functional, without further analysis has an appearance of not being optimal in terms of taking advantage of all the niceties bundled in Solaris 11 at no extra cost.

When looking at Solaris 11, we recognize that one of the vehicles that bridges the gap between getting the operating system image payload delivered, and the customized banking software installed, is a notion of a First Boot script.  I had a working example of this at one of the Oracle OpenWorld sessions a few years ago - we've since improved our documentation and have introduced sections where this is described in better detail.   If you're looking at this for the first time and you've not worked with IPS and SMF previously, you might get the sense that the tasks are daunting.   There is a set of technologies involved that are jointly engineered in order to make the process reliable, predictable and extensible.

As you go down the path of writing your first boot script, you'll be faced with a need to wrap it into a SMF service and then packaged into a IPS package. The IPS package would then need to be placed onto your IPS repository, in order to subsequently be made available to all of your AI (Automated Install) clients (i.e. the systems that you're installing Solaris and your software onto).    

With this blog post, I wanted to create a single place that outlines the entire process (simplistically), and provide a hint of how a good old "at" command may make the requirement of forcing an initial reboot handy. The syntax and references to commands here is based on running this on a version of Solaris 11 that has been updated since its initial release in 2011 (i.e. I am writing this on Solaris 11.1)

Assuming you've built an AI server (see this How To article for an example), you might be asking yourself: "Ok, I've got some logic that I need executed AFTER Solaris is deployed and I need my own little script that would make that happen. How do I go about hooking that script into the Solaris 11 AI framework?" 

You might start here, in Chapter 13 of the "Installing Oracle Solaris 11.1 Systems" guide, which talks about "Running a Custom Script During First Boot".  And as you do, you'll be confronted with command that might be unfamiliar to you if you're new to Solaris 11, like our dear new friend: svcbundle

svcbundle is an aide to creating manifests and profiles.  It is awesome, but don't let its awesomeness overwhelm you. (See this How To article by my colleague Glynn Foster for a nice working example).  In order to get your script's logic integrated into the Solaris 11 deployment process, you need to wrap your (shell) script into 2 manifests -  a SMF service manifest and a IPS package manifest.  ....and if you're new to XML, well then -- buckle up :-)

We have some examples of small first boot scripts shown here, as templates to build upon. Necessary structure of the script, particularly in leveraging SMF interfaces, is key. I won't go into that here as that is covered nicely in the doc link above.   

Let's say your script ends up looking like this (btw: if things appear to be cut-off in your browser, just select them, copy and paste into your editor and it'll be grabbed - the source gets captured eventhough the browser may not render it "correctly" - ah, computers).


# Load SMF shell support definitions
. /lib/svc/share/

# If nothing to do, exit with temporary disable
completed=`svcprop -p config/completed site/first-boot-script-svc:default`
[ "${completed}" = "true" ] && \
    smf_method_exit $SMF_EXIT_TEMP_DISABLE completed "Configuration completed"

# Obtain the active BE name from beadm: The active BE on reboot has an R in
# the third column of 'beadm list' output. Its name is in column one.
bename=`beadm list -Hd|nawk -F ';' '$3 ~ /R/ {print $1}'`
beadm create ${bename}.orig
echo "Original boot environment saved as ${bename}.orig"

# ---- Place your one-time configuration tasks here ----
# For example, if you have to pull some files from your own pre-existing system:
/usr/bin/chmod 755 /var/tmp/$SCRIPTS_THAT_GOT_PULLED_DOWN_IN_STEP_ABOVE
# Clearly the above 2 lines represent some logic that you'd have to customize to fit your needs.
# Perhaps additional things you may want to do here might be of use, like
# (gasp!) configuring ssh server for root login and X11 forwarding (for testing), and the like...
# Oh and by the way, after we're done executing all of our proprietary scripts we need to reboot
# the system in accordance with our operational software requirements to ensure all layered bits
# get initialized properly and pull-in their own modules and components in the right sequence,
# subsequently.
# We need to set a "time bomb" reboot, that would take place upon completion of this script.
# We already know that *this* script depends on multi-user-server SMF milestone, so it should be
# safe for us to schedule a reboot for 5 minutes from now. The "at" job get scheduled in the queue
# while our little script continues thru the rest of the logic. 
/usr/bin/at now + 5 minutes <<REBOOT
# ---- End of your customizations ----

# Record that this script's work is done
svccfg -s site/first-boot-script-svc:default setprop config/completed = true
svcadm refresh site/first-boot-script-svc:default

smf_method_exit $SMF_EXIT_TEMP_DISABLE method_completed "Configuration completed"

 ...and you're happy with it and are ready to move on. Where do you go and what do you do?

The next step is creating the IPS package for your script. Since running the logic of your script constitutes a service, you need to create a service manifest. This is described here, in the middle of Chapter 13 of "Creating an IPS package for the script and service". 

Assuming the name of your shell script is, you could end up doing the following:

$ cd some_working_directory_for_this_project

$ mkdir -p proto/lib/svc/manifest/site

$ mkdir -p proto/opt/site

$ cp proto/opt/site

 Then you would create the service manifest  file like so:

$ svcbundle -s service-name=site/first-boot-script-svc \
-s start-method=/opt/site/ \
-s instance-property=config:completed:boolean:false -o \

first-boot-script-svc-manifest.xml described here, and place it into the directory hierarchy above.

But before you place it into the directory, make sure to inspect the manifest and adjust the appropriate service dependencies.  That is to say, you want to properly specify what milestone should be reached before your service runs.  There's a <dependency> section that looks like this, before you modify it:

<dependency restart_on="none" type="service"
            name="multi_user_dependency" grouping="require_all">
            <service_fmri value="svc:/milestone/multi-user"/>         

So if you'd like to have your service run AFTER the multi-user-server milestone has been reached (i.e. later, as multi-user-server has more dependencies then multi-user and our intent to reboot the system may have significant ramifications if done prematurely), you would modify that section to read: 

<dependency restart_on="none" type="service"
            name="multi_user_server_dependency" grouping="require_all">
            <service_fmri value="svc:/milestone/multi-user-server"/> 

Save the file and validate it:

$ svccfg validate first-boot-script-svc-manifest.xml

Assuming there are no errors returned, copy the file over into the directory hierarchy:

$ cp first-boot-script-svc-manifest.xml proto/lib/svc/manifest/site

Now that we've created the service manifest (.xml), create the package manifest (.p5m) file named: first-boot-script.p5m

Populate it as follows:

set name=pkg.fmri value=first-boot-script-AT-1-DOT-0,5.11-0
set name=pkg.summary value="AI first-boot script"
set name=pkg.description value="Script that runs at first boot after AI installation"
set name=info.classification value=\
    "org.opensolaris.category.2008:System/Administration and Configuration"
file lib/svc/manifest/site/first-boot-script-svc-manifest.xml \
    path=lib/svc/manifest/site/first-boot-script-svc-manifest.xml owner=root \
    group=sys mode=0444
dir  path=opt/site owner=root group=sys mode=0755
file opt/site/ path=opt/site/ \
    owner=root group=sys mode=0555

Now we are going to publish this package into a IPS repository. If you don't have one yet, don't worry. You have 2 choices: You can either  publish this package into your mirror of the Oracle Solaris IPS repo or create your own customized repo.  The best practice is to create your own customized repo, leaving your mirror of the Oracle Solaris IPS repo untouched.  From this point, you have 2 choices as well - you can either create a repo that will be accessible by your clients via HTTP or via NFS.  Since HTTP is how the default Solaris repo is accessed, we'll go with HTTP for your own IPS repo.   This nice and comprehensive How To by Albert White describes how to create multiple internal IPS repos for Solaris 11. We'll zero in on the basic elements for our needs here:

We'll create the IPS repo directory structure hanging off a separate ZFS file system, and we'll tie it into an instance of pkg.depotd. We do this because we want our IPS repo to be accessible to our AI clients through HTTP, and the pkg.depotd SMF service bundled in Solaris 11 can help us do this. We proceed as follows:

# zfs create rpool/export/MyIPSrepo

# pkgrepo create /export/MyIPSrepo

# svccfg -s pkg/server add MyIPSrepo

# svccfg -s pkg/server:MyIPSrepo addpg pkg application
# svccfg -s pkg/server:MyIPSrepo setprop pkg/port=10081
# svccfg -s pkg/server:MyIPSrepo setprop pkg/inst_root=/export/MyIPSrepo
# svccfg -s pkg/server:MyIPSrepo addpg general framework
# svccfg -s pkg/server:MyIPSrepo addpropvalue general/complete astring: MyIPSrepo
# svccfg -s pkg/server:MyIPSrepo addpropvalue general/enabled boolean: true
# svccfg -s pkg/server:MyIPSrepo setprop pkg/readonly=true
# svccfg -s pkg/server:MyIPSrepo setprop pkg/proxy_base = astring: http://your_internal_websrvr/MyIPSrepo
# svccfg -s pkg/server:MyIPSrepo setprop pkg/threads = 200
# svcadm refresh application/pkg/server:MyIPSrepo
# svcadm enable application/pkg/server:MyIPSrepo

Now that the IPS repo is created, we need to publish our package into it:

# pkgsend publish -d ./proto -s /export/MyIPSrepo first-boot-script.p5m

If you find yourself making changes to your script, remember to up-rev the version in the .p5m file (which is your IPS package manifest), and re-publish the IPS package.

Next, you need to go to your AI install server (which might be the same machine) and modify the AI manifest to include a reference to your newly created package.  We do that by listing an additional publisher, which would look like this (replacing the IP address and port with your own, from the "svccfg" commands up above):

<publisher name="firstboot">
  <origin name=""/>

 Further down, in the  <software_data action="install">  section add:


Make sure to update your Automated Install service with the new AI manifest via installadm update-manifest command.  Don't forget to boot your client from the network to watch the entire process unfold and your script get tested.  Once the system makes the initial reboot, the first boot script will be executed and whatever logic you've specified in it should be executed, too, followed by a nice reboot. When the system comes up, your service should stay in a disabled state, as specified by the tailing lines of your SMF script - this is normal and should be left as is as it helps provide an auditing trail for you.   Because the reboot is quite a significant action for the system, you may want to add additional logic to the script that actually places and then checks for presence of certain lock files in order to avoid doing a reboot unnecessarily. You may also want to, alternatively, remove the SMF service entirely - if you're unsure of the potential for someone to try and accidentally enable that service -- eventhough its role in life is to only run once upon the system's first boot.

That is how I spent a good chunk of my pre-Halloween time this week, hope yours was just as SPARCkly^H^H^H^H fun!    8-)


Isaac Rozenfeld has been a Principal Product Manager for Oracle Solaris; responsibilities have included bettering the portfolio of networking and installation technologies - all with a focus on easing application deployments

You can follow Isaac on Twitter @izfromsun


« July 2016
Tech Reference

No bookmarks in folder