Solaris News, Views, and Real-World Experiences from the Field

Ops Center 12c - Provisioning Solaris Using a Card-Based NIC

Scott Dickson
Enterprise Systems Architect

It's been a long time since last I added something here, but having some conversations this last week, I got inspired to update things.

I've been spending a lot of time with Ops Center for managing and installing systems these days.  So, I suspect a number of my upcoming posts will be in that area.

Today, I want to look at how to provision Solaris using Ops Center when your network is not connected to one of the built-in NICs.  We'll talk about how this can work for both Solaris 10 and Solaris 11, since they are pretty similar.  In both cases, WANboot is a key piece of the story.

Here's what I want to do:  I have a Sun Fire T2000 server with a Quad-GbE nxge card installed.  The only network is connected to port 2 on that card rather than the built-in network interfaces.  I want to install Solaris on it across the network, either Solaris 10 or Solaris 11.  I have met with a lot of customers lately who have a similar architecture.  Usually, they have T4-4 servers with the network connected via 10GbE connections.

Add to this mix the fact that I use Ops Center to manage the systems in my lab, so I really would like to add this to Ops Center.  If possible, I would like this to be completely hands free.  I can't quite do that yet. Close, but not quite.

WANBoot or Old-Style NetBoot?

When a system is installed from the network, it needs some help getting the process rolling.  It has to figure out what its network configuration (IP address, gateway, etc.) ought to be.  It needs to figure out what server is going to help it boot and install, and it needs the instructions for the installation.  There are two different ways to bootstrap an installation of Solaris on SPARC across the network.   The old way uses a broadcast of RARP or more recently DHCP to obtain the IP configuration and the rest of the information needed.  The second is to explicitly configure this information in the OBP and use WANBoot for installation

WANBoot has a number of benefits over broadcast-based installation: it is not restricted to a single subnet; it does not require special DHCP configuration or DHCP helpers; it uses standard HTTP and HTTPS protocols which traverse firewalls much more easily than NFS-based package installation.  But, WANBoot is not available on really old hardware and WANBoot requires the use o Flash Archives in Solaris 10.  Still, for many people, this is a great approach.

As it turns out, WANBoot is necessary if you plan to install using a NIC on a card rather than a built-in NIC.

Identifying Which Network Interface to Use

One of the trickiest aspects to this process, and the one that actually requires manual intervention to set up, is identifying how the OBP and Solaris refer to the NIC that we want to use to boot.  The OBP already has device aliases configured for the built-in NICs called net, net0, net1, net2, net3.  The device alias net typically points to net0 so that when you issue the command  "boot net -v install", it uses net0 for the boot.  Our task is to figure out the network instance for the NIC we want to use. 

We will need to get to the OBP console of the system we want to install in order to figure out what the network should be called.  I will presume you know how to get to the ok prompt.  Once there, we have to see what networks the OBP sees and identify which one is associated with our NIC using the OBP command show-nets.

SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
{4} ok banner
Sun Fire T200, No Keyboard
Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.b, 32640 MB memory available, Serial #69057548.
Ethernet address 0:14:4f:1d:bc:c, Host ID: 841dbc0c.
{4} ok show-nets
a) /pci@7c0/pci@0/pci@2/network@0,1
b) /pci@7c0/pci@0/pci@2/network@0
c) /pci@780/pci@0/pci@8/network@0,3
d) /pci@780/pci@0/pci@8/network@0,2
e) /pci@780/pci@0/pci@8/network@0,1
f) /pci@780/pci@0/pci@8/network@0
g) /pci@780/pci@0/pci@1/network@0,1
h) /pci@780/pci@0/pci@1/network@0
Enter Selection, q to quit: d
/pci@780/pci@0/pci@8/network@0,2 has been selected.
Type ^Y ( Control-Y ) to insert it in the command line.
e.g. ok nvalias mydev ^Y
for creating devalias mydev for /pci@780/pci@0/pci@8/network@0,2
{4} ok devalias
net3 /pci@7c0/pci@0/pci@2/network@0,1
net2 /pci@7c0/pci@0/pci@2/network@0
net1 /pci@780/pci@0/pci@1/network@0,1
net0 /pci@780/pci@0/pci@1/network@0
net /pci@780/pci@0/pci@1/network@0
name aliases

By looking at the devalias and the show-nets output, we can see that our Quad-GbE card must be the device nodes starting with  /pci@780/pci@0/pci@8/network@0.  The cable for our network is plugged into the 3rd slot, so the device address for our network must be /pci@780/pci@0/pci@8/network@0,2.

With that, we can create a device alias for our network interface.  Naming the device alias may take a little bit of trial and error, especially in Solaris 11 where the device alias seems to matter more with the new virtualized network stack. So far in my testing, since this is the "next" network interface to be used, I have found success in naming it net4, even though it's a NIC in the middle of a card that might, by rights, be called net6 (assuming the 0th interface on the card is the next interface identified by Solaris and this is the 3rd interface on the card).  So, we will call it net4.  We need to assign a device alias to it:

{4} ok nvalias net4 /pci@780/pci@0/pci@8/network@0,2
{4} ok devalias
net4 /pci@780/pci@0/pci@8/network@0,2

We also may need to have the MAC for this particular interface, so let's get it, too.  To do this, we go to the device and interrogate its properties.

{4} ok cd /pci@780/pci@0/pci@8/network@0,2
{4} ok .properties
assigned-addresses 82060210 00000000 03000000 00000000 01000000
82060218 00000000 00320000 00000000 00008000
82060220 00000000 00328000 00000000 00008000
82060230 00000000 00600000 00000000 00100000
local-mac-address 00 21 28 20 42 92
phy-type mif

From this, we can see that the MAC for this interface is  00:21:28:20:42:92.  We will need this later.

This is all we need to do at the OBP.  Now, we can configure Ops Center to use this interface.

Network Boot in Solaris 10

Solaris 10 turns out to be a little simpler than Solaris 11 for this sort of a network boot.  Since WANBoot in Solaris 10 fetches a specified

In order to install the system using Ops Center, it is necessary to create a OS Provisioning profile and its corresponding plan.  I am going to presume that you already know how to do this within Ops Center 12c and I will just cover the differences between a regular profile and a profile that can use an alternate interface.

Create a OS Provisioning profile for Solaris 10 as usual.  However, when you specify the network resources for the primary network, click on the name of the NIC, probably GB_0, and rename it to GB_N/netN, where N is the instance number you used previously in creating the device alias.  This is where the trial and error may come into play.  You may need to try a few instance numbers before you, the OBP, and Solaris all agree on the instance number.  Mark this as the boot network.

For Solaris 10, you ought to be able to then apply the OS Provisioning profile to the server and it should install using that interface.  And if you put your cards in the same slots and plug the networks into the same NICs, this profile is reusable across multiple servers.

Why This Works

If you watch the console as Solaris boots during the OSP process, Ops Center is going to look for the device alias netN.  Since WANBoot requires a device alias called just net, Ops Center uses the value of your netN device alias and assigns that device to the net alias.  That means that boot net will automatically use this device.  Very cool!  Here's a trace from the console as Ops Center provisions a server:

Sun Sun Fire T200, No Keyboard
Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.b, 32640 MB memory available, Serial #69057548.
Ethernet address 0:14:4f:1d:bc:c, Host ID: 841dbc0c.

auto-boot? =            false
{0} ok  
{0} ok printenv network-boot-arguments
network-boot-arguments =  host-ip=,router-ip=,subnet-mask=,hostname=atl-sewr-52,client-id=0100144F1DBC0C,file=
{0} ok
{0} ok devalias net
net                      /pci@780/pci@0/pci@1/network@0
{0} ok devalias net4
net4                     /pci@780/pci@0/pci@8/network@0,2
{0} ok devalias net /pci@780/pci@0/pci@8/network@0,2
{0} ok setenv network-boot-arguments host-ip=,router-ip=,subnet-mask=,hostname=atl-sewr-52,client-id=0100144F1DBC0C,file=
network-boot-arguments =  host-ip=,router-ip=,subnet-mask=,hostname=atl-sewr-52,client-id=0100144F1DBC0C,file=
{0} ok
{0} ok boot net - install
Boot device: /pci@780/pci@0/pci@8/network@0,2  File and args: - install
/pci@780/pci@0/pci@8/network@0,2: 1000 Mbps link up
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot info: configuring /pci@780/pci@0/pci@8/network@0,2

See what happened?  Ops Center looked for the network device alias called net4 that we specified in the profile, took the value from it, and made it the net device alias for the boot.  Pretty cool!

WANBoot and Solaris 11

Solaris 11 requires an additional step since the Automated Installer in Solaris 11 uses the MAC address of the network to figure out which manifest to use for system installation.  In order to make sure this is available, we have to take an extra step to associate the MAC of the NIC on the card with the host.  So, in addition to creating the device alias like we did above, we also have to declare to Ops Center that the host has this new MAC.

Declaring the NIC

Start out by discovering the hardware as usual.  Once you have discovered it, take a look under the Connectivity tab to see what networks it has discovered.  In the case of this system, it shows the 4 built-in networks, but not the networks on the additional cards.  These are not directly visible to the system controller.

 In order to add the additional network interface to the hardware asset, it is necessary to Declare it.  We will declare that we have a server with this additional NIC, but we will also  specify the existing GB_0 network so that Ops Center can associate the right resources together.  The GB_0 acts as sort of a key to tie our new declaration to the old system already discovered.  Go to the Assets tab, select All Assets, and then in the Actions tab, select Add Asset.  Rather than going through a discovery this time, we will manually declare a new asset.

When we declare it, we will give the hostname, IP address, system model that match those that have already been discovered.  Then, we will declare both GB_0 with its existing MAC and the new GB_4 with its MAC.  Remember that we collected the MAC for GB_4 when we created its device alias.

After you declare the asset, you will see the new NIC in the connectivity tab for the asset.  You will notice that only the NICs you listed when you declared it are seen now.  If you want Ops Center to see all of the existing NICs as well as the additional one, declare them as well.  Add the other GB_1, GB_2, GB_3 links and their MACs just as you did GB_0 and GB_4.

 Installing the OS

 Once you have declared the asset, you can create an OS Provisioning profile for Solaris 11 in the same way that you did for Solaris 10.  The only difference from any other provisioning profile you might have created already is the network to use for installation.  Again, use GB_N/netN where N is the interface number you used for your device alias and in your declaration.

 And away you go.  When the system boots from the network, the automated installer (AI) is able to see which system manifest to use, based on the new MAC that was associated, and the system gets installed.

{0} ok
{0} ok printenv network-boot-arguments
network-boot-arguments =  host-ip=,router-ip=,subnet-mask=,hostname=atl-sewr-52,client-id=01002128204292,file=
{0} ok
{0} ok devalias net
net                      /pci@780/pci@0/pci@1/network@0
{0} ok devalias net4
net4                     /pci@780/pci@0/pci@8/network@0,2
{0} ok devalias net /pci@780/pci@0/pci@8/network@0,2
{0} ok setenv network-boot-arguments host-ip=,router-ip=,subnet-mask=,hostname=atl-sewr-52,client-id=01002128204292,file=
network-boot-arguments =  host-ip=,router-ip=,subnet-mask=,hostname=atl-sewr-52,client-id=01002128204292,file=
{0} ok
{0} ok boot net - install
Boot device: /pci@780/pci@0/pci@8/network@0,2  File and args: - install
/pci@780/pci@0/pci@8/network@0,2: 1000 Mbps link up
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot info: configuring /pci@780/pci@0/pci@8/network@0,2
SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Probing for device nodes ...
Preparing network image for use
Downloading solaris.zlib
--2012-02-17 15:10:17--
Connecting to connected.
HTTP request sent, awaiting response... 200 OK
Length: 126752256 (121M) [text/plain]
Saving to: `/tmp/solaris.zlib'

100%[======================================>] 126,752,256 28.6M/s   in 4.4s   

2012-02-17 15:10:21 (27.3 MB/s) - `/tmp/solaris.zlib' saved [126752256/126752256]


So, why go to all of this trouble?  More and more, I find that customers are wiring their data center to only use higher speed networks - 10GbE only to the hosts.  Some customers are moving aggressively toward consolidated networks combining storage and network on CNA NICs.  All of this means that network-based provisioning cannot rely exclusively on the built-in network interfaces.  So, it's important to be able to provision a system using other than the built-in networks.  Turns out, that this is pretty straight-forward for both Solaris 10 and Solaris 11 and fits into the Ops Center deployment process quite nicely.

Hopefully, you will be able to use this as you build out your own private cloud solutions with Ops Center.

Join the discussion

Comments ( 5 )
  • guest Saturday, December 15, 2012

    I get the error saying:

    Deployment failed as OC network does not have a gateway.ass appropriate gateway and continue.

    I have checked everything. i don't know what i am missing.

  • Scott Dickson Tuesday, December 18, 2012

    I have found that Ops Center does not always detect / remember the network gateway for discovered networks. To add one, go to the Networks section in the Navigation tab and select the network you need to update. Then on the right side, click on Edit Network Attributes. You can add a network gateway for the network here.

    I found that I have had to do this for networks besides my primary network.

  • guest Tuesday, December 18, 2012

    Hi Scott,

    thank you. i kinda figured that out strolling across the BUI. yes that worked. I have been facing small issues. figuring out with help of engineers too.

    Provisioningof OS had issue. It finished installing the OS but exited the job saying. Unable to Configure LDom agent. and the OS of CDOM doesn't show up on OpsCenter.

    Below log is from agent_install log in /var/tmp

    Monday, December 17, 2012 12:55:23 AM : Again checking ldom instance com.sun.

    hss.xvmserver.domainmodel is up or not will sleep for 2mts

    Monday, December 17, 2012 12:57:27 AM : ldom instance com.sun.hss.xvmserver.d

    omainmodel not up

    Monday, December 17, 2012 12:57:27 AM : Unable to configure the agent (unable


    Monday, December 17, 2012 12:57:27 AM :/usr/sfw/bin/wget -O /var/tmp/agent_in

    stall.run -t 5 -T 5 http://<PCip>:8004/scnospstat/?phase=agent&client_id


  • Di Cerbo Antonio Friday, July 5, 2013

    I've tested this procedure using a T4-2 with our SUN DUAL PORT 10GBE SFP+ PCIE 2.0 NETWORKING CARDS without success; the "net" alias on the OBP remains unchanged so the system will always boot using the default net0 alias port then fails.

    The procedure works fine using another Onboard NIC. I've not tried using a PCI Quad Gigabit Ethernet yet.

    System to be installed using OPS Center provisioning:

    - SPARC T4-2 - Sun System Firmware 8.3.0.b

    - Two SUN DUAL PORT 10GBE SFP+ PCIE 2.0 Cards.

    - Solaris 11.1 SRU 8.4 from the OPS Center system.

    OPS Center Server:

    - SunFire X3-2

    - Solaris 11.1 SRU 8.4

    - OPS Center Update 3 with latest patch(28Jun2013)

    - One SUN DUAL PORT 10GBE SFP+ PCIE 2.0 Cards.

  • guest Tuesday, August 27, 2013

    We're trying to use this to deploy T4-1Bs in a 6000 chassis with a pair of 40gbit Virtualized NEMs. I've managed to get it to work but it requires us to set the net devalias by hand. We're using link aggregation to handle network resilience.

    I've tried creating the extra interfaces as NEM0/F0, NEM0/F1, NEM1/F0 and NEM/F1 and adding them to the asset by defining the existing GB_0 interface as you have in your description, but these interfaces never seem to appear in Ops Centre, and it doesn't seem to check them at the ok> prompt.

    Do you have any advice on doing this with the 40gbit NEM interfaces?

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.