Thursday Jul 31, 2014

Neutron L3 Agent in Oracle Solaris OpenStack

The Oracle Solaris implementation of OpenStack Neutron supports the following deployment model: provider router with private networks deployment. You can find more information about this model here. In this deployment model, each tenant can have one or more private networks and all the tenant networks share the same router. This router is created, owned, and managed by the data center administrator. The router itself will not be visible in the tenant's network topology view. Because there is only a single router, tenant networks cannot use overlapping IPs. Thus, it is likely that the administrator would create the private networks on behalf of tenants.

By default, this router prevents routing between private networks that are part of the same tenant. That is, VMs within one private network cannot communicate with the VMs in another private network, even though they are all part of the same tenant. This behavior can be changed by setting allow_forwarding_between_networks to True in the /etc/neutron/l3_agent.ini configuration file and restarting the neturon-l3-agent SMF service.

This router provides connectivity to the outside world for the tenant VMs. It does this by performing bidirectional NAT on the interface that connects the router to the external network. Tenants create as many floating IPs (public IPs) as they need or as are allowed by the floating IP quota and then associate these floating IPs with the VMs that need outside connectivity.

The following figure captures the supported deployment model.


Figure 1 Provider router with private networks deployment

Tenant A has:

  • Two internal networks:
    HR (subnet:, gateway:
    ENG (subnet:, gateway:
  • Two VMs
    VM1 connected to HR with a fixed IP address of
    M2 connected to ENG with a fixed IP address of

Tenant B has:

  • Two internal networks:
    IT (subnet:, gateway:
    ACCT (subnet:, gateway:
  • Two VMs
    VM3 connected to IT with a fixed IP address of
    VM4 connected to ACCT with a fixed IP address of

All the gateway interfaces are instantiated on the node that is running neutron-l3-agent.

The external network is a provider network that is associated with the subnet that is reachable from outside. Tenants will create floating IPs from this network and associate them to their VMs. VM1 and VM2 have floating IPs and associated with them respectively. VM1 and VM2 are reachable from the outside world through these IP addresses.

Configuring neutron-l3-agent on a Network Node

Note: In this configuration, all Compute Nodes and Network Nodes in the network have been identified, and the configuration file for all the OpenStack services has been appropriately configured so that these services can communicate with each other.

The service tenant is a tenant for all the OpenStack services (nova, neutron, glance, cinder, swift, keystone, and horizon) and the users for each of the services. Services communicate with each other using these users who all have admin role. The steps below show how to use the service tenant to create a router, an external network, and an external subnet that will be used by all of the tenants in the data center. Please refer to the following table and diagram while walking through the steps.

Note: Alternatively, you could create a separate tenant (DataCenter) and a new user (datacenter) with admin role, and the DataCenter tenant could host all of the aforementioned shared resources. 


Table 1 Public IP address mapping


Figure 2 Neutron L3 agent configuration

Steps required to setup Neutron L3 agent as a data center administrator:

Note: We will need to use OpenStack CLI to configure the shared single router and associate network/subnets from different tenants with it because from OpenStack dashboard you can only manage one tenant’s resources at a time. 

1. Enable Solaris IP filter functionality.

   l3-agent# svcadm enable ipfilter
   l3-agent# svcs ipfilter
   online 10:29:04 svc:/network/ipfilter:default

2. Enable IP forwarding on the entire host.

   l3-agent# ipadm show-prop -p forwarding ipv4
   ipv4  forwarding  rw   on           on           off          on,off 

3. Ensure that the Solaris Elastic Virtual Switch feature is configured correctly and has the VLAN ID required for the external network. In our case, the external network/subnet uses VLAN 1.

   l3-agent# evsadm show-controlprop -p vlan-range,l2-type
   PROPERTY            PERM VALUE               DEFAULT             HOST
   l2-type             rw   vlan                vlan                --
   vlan-range          rw   200-300             --                  --

   l3-agent# evsadm set-controlprop -p vlan-range=1,200-300

Note: For more information on EVS please refer to Chapter 5, "About Elastic Virtual Switches" and Chapter 6, "Administering Elastic Virtual Switches" in Managing Network Virtualization and Network Resources in Oracle Solaris 11.2 ( In short, Solaris EVS forms the backend for OpenStack networking, and it facilitates inter-VM communication (on the same compute-node or across compute-node) either using VLANs or VXLANs.

4. Ensure that the service tenant is already there.

   l3-agent# keystone --os-endpoint=http://localhost:35357/v2.0 \
   --os-token=ADMIN tenant-list
   |                id                |   name  | enabled |
   | 511d4cb9ef6c40beadc3a664c20dc354 |   demo  |   True  |
   | f164220cb02465db929ce520869895fa | service |   True  |

5. Create the provider router. Note the UUID of the new router.

   l3-agent# export OS_USERNAME=neutron
   l3-agent# export OS_PASSWORD=neutron
   l3-agent# export OS_TENANT_NAME=service
   l3-agent# export OS_AUTH_URL=http://localhost:5000/v2.0
   l3-agent# neutron router-create provider_router
   Created a new router:
   | Field                 | Value                                |
   | admin_state_up        | True                                 |
   | external_gateway_info |                                      |
   | id                    | 181543df-40d1-4514-ea77-fddd78c389ff |
   | name                  | provider_router                      |
   | status                | ACTIVE                               |
   | tenant_id             | f164220cb02465db929ce520869895fa     |

6. Use the router UUID from step 5 and update /etc/neutron/l3_agent.ini file with following entry:

router_id = 181543df-40d1-4514-ea77-fddd78c389ff

7. Enable the neutron-l3-agent service.

   l3-agent# svcadm enable neutron-l3-agent
   l3-agent# svcs neutron-l3-agent
   online 11:24:08 svc:/application/openstack/neutron/neutron-l3-agent:default

8. Create an external network.

   l3-agent# neutron net-create --provider:network_type=vlan \
   --provider:segmentation_id=1 --router:external=true  external_network
   Created a new network:
   | Field                    | Value                                |
   | admin_state_up           | True                                 |
   | id                       | f67f0d72-0ddf-11e4-9d95-e1f29f417e2f |
   | name                     | external_network                     |
   | provider:network_type    | vlan                                 |
   | provider:segmentation_id | 1                                    |
   | router:external          | True                                 |
   | shared                   | False                                |
   | status                   | ACTIVE                               |
   | subnets                  |                                      |
   | tenant_id                | f164220cb02465db929ce520869895fa     |

9. Associate a subnet to external_network

   l3-agent# neutron subnet-create --enable-dhcp=False \
   --name external_subnet external_network
   Created a new subnet:
   | Field            | Value                                            |
   | allocation_pools | {"start": "", "end": ""} |
   | cidr             |                                   |
   | dns_nameservers  |                                                  |
   | enable_dhcp      | False                                            |
   | gateway_ip       |                                      |
   | host_routes      |                                                  |
   | id               | 5d9c8958-0de0-11e4-9d96-e1f29f417e2f             |
   | ip_version       | 4                                                |
   | name             | external_subnet                                  |
   | network_id       | f67f0d72-0ddf-11e4-9d95-e1f29f417e2f             |
   | tenant_id        | f164220cb02465db929ce520869895fa                 |

10. Apply the workaround for not having --allocation-pool support for subnets. Because through IP addresses are set aside for other OpenStack API services, perform the following floatingip-create steps to ensure that no tenant will assign these IP addresses to VMs:

NOTE: This workaround is not needed if you are running S11.2 SRU5 and above as the support
for allocation pool was added in that update.

   l3-agent# for i in `seq 1 6`; do neutron floatingip-create \
   external_network; done
   l3-agent# neutron floatingip-list -c id -c floating_ip_address
   | id                                   | floating_ip_address |
   | 58fbccdd-1b60-c6ba-9a51-bbc2cbcc95f8 |         |
   | ce620f79-aed4-6d1c-b5e7-c64c5f6d1f28 |         |
   | 6442eef1-b748-cb51-8a96-98b90e264bd0 |         |
   | a9792d03-f5de-cae1-fa5a-bb614720b22c |         |
   | da18a52d-73a5-4c7d-fb98-95d292d9b0e8 |         |
   | 22e02f77-5b44-402a-d369-9e6b1d831ca0 |         |

11. Add external_network to the router.

    l3-agent# neutron router-gateway-set -h
    usage: neutron router-gateway-set [-h] [--request-format {json,xml}]
     router-id external-network-id

    l3-agent# neutron router-gateway-set \
    181543df-40d1-4514-ea77-fddd78c389ff \  (provider_router UUID)
    f67f0d72-0ddf-11e4-9d95-e1f29f417e2f    (external_network UUID)
    Set gateway for router 181543df-40d1-4514-ea77-fddd78c389ff

    l3-agent# neutron router-list -c name -c external_gateway_info
| name            | external_gateway_info                                  |
| provider_router | {"network_id": "f67f0d72-0ddf-11e4-9d95-e1f29f417e2f"} |

12. Add the tenant's private networks to the router. The networks shown by neutron net-list were previously configured.

    l3-agent# keystone tenant-list
    |                id                |   name  | enabled |
    | 511d4cb9ef6c40beadc3a664c20dc354 |   demo  |   True  |
    | f164220cb02465db929ce520869895fa | service |   True  |

    l3-agent# neutron net-list --tenant-id=511d4cb9ef6c40beadc3a664c20dc354
    | id                            | name | subnets                      |
    | c0c15e0a-0def-11e4-9d9f-      | HR   | c0c53066-0def-11e4-9da0-     |
    |  e1f29f417e2f                 |      | e1f29f417e2f|   
    | ce64b430-0def-11e4-9da2-      | ENG  | ce693ac8-0def-11e4-9da3-     |
    |  e1f29f417e2f                 |      | e1f29f417e2f|

    Note: The above two networks were preconfigured 

    l3-agent# neutron router-interface-add  \
    181543df-40d1-4514-ea77-fddd78c389ff \ (provider_router UUID)
    c0c53066-0def-11e4-9da0-e1f29f417e2f   (HR subnet UUID)
    Added interface 7843841e-0e08-11e4-9da5-e1f29f417e2f to router 181543df-40d1-4514-ea77-fddd78c389ff.

    l3-agent# neutron router-interface-add \
    181543df-40d1-4514-ea77-fddd78c389ff \ (provider_router UUID)
    ce693ac8-0def-11e4-9da3-e1f29f417e2f   (ENG subnet UUID)
    Added interface 89289b8e-0e08-11e4-9da6-e1f29f417e2f to router 181543df-40d1-4514-ea77-fddd78c389ff.

13. The following figure shows how the network topology looks when you log in as a service tenant user.


Steps required to create and associate floating IPs as a tenant user

1. Log into the OpenStack Dashboard using the tenant user's credential

2. Select Project -> Access & Security -> Floating IPs

3. With external_network selected, click the Allocate IP button


4. The Floating IPs tab shows that Floating IP is allocated.


5. Click the Associate button and select the VM's port from the pull down menu.


6. The Project -> Instances window shows that the floating IP is associated with the VM.


If you had selected a keypair (SSH Public Key) while launching an instance, then that SSH key would be added into the root's authorized_keys file in the VM. With that done you can ssh into the running VM.

       [gmoodalb@thunta:~] ssh root@
       Last login: Fri Jul 18 00:37:39 2014 from Oracle Corporation SunOS 5.11 11.2 June 2014

       root@host-192-168-101-3:~# uname -a
       SunOS host-192-168-101-3 5.11 11.2 i86pc i386 i86pc
       root@host-192-168-101-3:~# zoneadm list -cv
       ID NAME              STATUS      PATH                 BRAND      IP    
        2 instance-00000001 running     /                    solaris    excl 
       root@host-192-168-101-3:~# ipadm
       NAME             CLASS/TYPE STATE        UNDER      ADDR
       lo0              loopback   ok           --         --
         lo0/v4         static     ok           --
 lo0/v6          static     ok          --         ::1/128
       net0             ip         ok           --         --
         net0/dhcp      inherited  ok           --

Under the covers:

On the node where neutron-l3-agent is running, you can use IP filter commands (ipf(1m), ippool(1m), and ipnat(1m)) and networking commands (dladm(1m) and ipadm(1m)) to observe and troubleshoot the configuration done by neturon-l3-agent.

VNICs created by neutron-l3-agent:

    l3-agent# dladm show-vnic
    LINK                OVER         SPEED  MACADDRESS        MACADDRTYPE VIDS
    l3i7843841e_0_0     net1         1000   2:8:20:42:ed:22   fixed       200
    l3i89289b8e_0_0     net1         1000   2:8:20:7d:87:12   fixed       201
    l3ed527f842_0_0     net0         100    2:8:20:9:98:3e    fixed       0

IP addresses created by neutron-l3-agent:

    l3-agent# ipadm
    NAME                  CLASS/TYPE STATE   UNDER      ADDR
    l3ed527f842_0_0       ip         ok      --         --
      l3ed527f842_0_0/v4  static     ok      --
      l3ed527f842_0_0/v4a static     ok      --
    l3i7843841e_0_0       ip         ok      --         --
      l3i7843841e_0_0/v4  static     ok      --
    l3i89289b8e_0_0       ip         ok      --         --
      l3i89289b8e_0_0/v4  static     ok      --

IP Filter rules:

   l3-agent# ipfstat -io
   empty list for ipfilter(out)
   block in quick on l3i7843841e_0_0 from to pool/4386082
   block in quick on l3i89289b8e_0_0 from to pool/8226578
   l3-agent# ippool -l
   table role = ipf type = tree number = 8226578
{; };
   table role = ipf type = tree number = 4386082
{; };

IP NAT rules:

   l3-agent# ipnat -l
   List of active MAP/Redirect filters:
   bimap l3ed527f842_0_0 ->
   List of active sessions:
   BIMAP  22  <- ->  22 [ 36405]

Known Issues:

1. The neutron-l3-agent SMF service goes into maintenance when it is restarted. This will be fixed in an SRU. The workaround is to restart the ipfilter service and clear the neutron-l3-agent.

# svcadm restart ipfilter:default
# svcadm clear neutron-l3-agent:default

2. The default gateway for the network node is removed in certain setups.

If the IP address of the Network Node is derived from the external_network address space, then if you use the neutron router-gateway-clear command to remove the external_network from the provider_router, the default gateway for the network node is deleted and the network node is inaccessible.

     l3-agent# neutron router-gateway-clear <router_UUID_goes_here>

To fix this problem, connect to the network node through the console and then add the default gateway again.

OpenStack 101 - How to get started on Oracle Solaris 11

As Eric has already mentioned with Oracle Solaris 11.2 we've included a complete, enterprise-ready distribution of OpenStack based on the "Havana" release of the upstream project. We've talked to many customers who have expressed an interest in OpenStack generally, but also being able to have Oracle Solaris participate in a heterogeneous mix of technologies that you'd typically see in a data center environment. We're absolutely thrilled to be providing this functionality to our customers as part of the core Oracle Solaris platform and support offering, so they can set up agile, self-service private clouds with Infrastructure-as-a-Service (IaaS), or develop Platform-as-a-Service (PaaS) or Software-as-a-Service (SaaS) solutions on top of this infrastructure.

If you haven't really had much experience with OpenStack, you'll almost certainly be confused by the myriad of different project names for some of the core components of an OpenStack cloud. Here's a handy table:

Component Description
Nova OpenStack Nova provides a cloud computing fabric controller that supports a wide variety of virtualization technologies. In addition to its native API, it includes compatibility with the commonly encountered Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) APIs.
Neutron OpenStack Neutron provides an API to dynamically request and configure virtual networks. These networks connect "interfaces" from other OpenStack services (for example, VNICs from Nova VMs). The Neutron API supports extensions to provide advanced network capabilities, for example, quality of service (QoS), access control lists (ACLs) and network monitoring.
Cinder OpenStack Cinder provides an infrastructure for managing block storage volumes in OpenStack. It allows block devices to be exposed and connected to compute instances for expanded storage, better performance, and integration with enterprise storage platforms.
Swift OpenStack Swift provides object storage services for projects and users in the cloud.
Glance OpenStack Glance provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image. VM images made available through Glance can be stored in a variety of locations from simple file systems to object-storage systems such as OpenStack Swift.
Keystone OpenStack Keystone is the OpenStack identity service used for authentication between the OpenStack services.
Horizon OpenStack Horizon is the canonical implementation of OpenStack's dashboard, which provides a web-based user interface to OpenStack services including Nova, Neutron, Cinder, Swift, Keystone and Glance.

So how do you get started? Due to the distributed architecture of OpenStack and being able to run different services across multiple nodes, OpenStack isn't the easiest thing in the world to configure and get running. We've made that easier for you to be able to set up a single-node pre-configured instance to evaluate initially with an OpenStack Unified Archive and an excellent getting started guide. Once you've got up to speed on a single-node set up, you can use your experience to deploy OpenStack on a multi-node setup. We've also got a bunch of other resource available:

We're just starting our journey of providing OpenStack on Oracle Solaris with this initial integration and we expect to deliver more value over time. Ready to start your journey with OpenStack in your data center?

-- Glynn Foster

OpenStack Immutable VMs

Solaris 11 brought us the ability to have Immutable non global Zones.  With Solaris 11.2 we have extended that capability so that it works with Kernel Zones, LDOMs (OVM SPARC) and bare metal global zones.

Now what about deploying Immutable Zones via OpenStack ?

The way to do this is to via the Flavors facility in Nova.

From the OpenStack Dashboard (Horizon) navigate to the Admin-> Flavor page. We can either update an existing one of the Solaris flavours or create a new one.  Lets do this by creating a new one called 'Immutable Solaris non global Zone'

Make sure you set the 'Flavor Access' to include the projects you want to use this.

Then from the 'More' menu on the entry in the table select 'View Extra Specs'

that will bring up a window like this one, since we are creating a new entry from scratch we have to also setup the type of zone this will be.

Select Create and fill in the following to set a non global zone (if you wanted a kernel zone instead then change the value to solaris-kz):

The do the same again and create a key/value pair for 'zonecfg:file-mac-profile' with the value being one of 'flexible-configuration,fixed-configuration,strict' eg:

Thats it close the flavor window and now we can select this as a type when we deploy a new instance.

If create a new VM instance using this flavor and look at the configuration of the zone that Nova is deploying for us we will see something like this:

$ zonecfg -z instance-0000000f info
zonename: instance-0000000f
zonepath: /system/zones/instance-0000000f
brand: solaris
autoboot: false
autoshutdown: shutdown
file-mac-profile: fixed-configuration

It is possible to set other zonecfg global scope settings here as well.  Currently the choice is limited to a fixed set but I'm hoping to change that to allow any of the known global scope settings.  This would allow using some of the more advanced Zone resource controls via an OpenStack Nova Flavor.

 -- Darren J Moffat


OpenStack Cinder Volume encryption with ZFS

In an OpenStack deployment the VMs is provided by the Cinder service. In the case of a Solaris instance these VMs are either Kernel Zones or non global zones configured for ZOSS (Zones On Shared Storage).  When Solaris 11.1 came out I wrote about using ZFS to encrypt zones.

The Cinder volume service for OpenStack can be provided by ZFS using ZVOLs.  So it shouldn't be surprising that we get to benefit from ZFS features such as compression, encryption and deduplication.

When deploying a simple OpenStack configuration using the 'solaris.zfs.ZFSVolumeDriver'  we  create ZVOLs in the dataset specified by the 'zfs_volume_base' variable in /etc/cinder/cinder.conf.  If the dataset specified by 'zfs_volume_base' doesn't already exist then the SMF service 'svc:/application/openstack/cinder/cinder-volume:setup' will create it for you and sets the file system permissions and zfs allow delegations for the 'cinder' user appropriately.

If we pre-create the ZFS dataset that zfs_volume_base points to all the ZVOLs that are created by cinder below that are automatically encrypted.

For example if I'm using a ZFS pool called 'cloudstore' and I set 'cloudstore/cinder' as 'zfs_volume_base' I can do this:

# zfs create -o encryption=on -o keysource=passphrase, cloudstore/cinder

In the above example I'm assuming we have an ad-hoc key manager available already that is providing keys/passphrases over https, you could also use a raw file, PKCS#11 keystore or interactively prompt; see the ZFS Encryption documentation for more guidance.

Now restart the  cinder-volume:setup service and we are ready to use our transparent encryption of Cinder volumes:

# svcadm restart cinder-volume:setup

If we look at the ZFS datasets that are created after we have launched a VM instance and the cinder volume for it was created we see this:

$ zfs get -r encryption cloudstore/cinder                   zfs-bugs
NAME                                                      PROPERTY    VALUE  SOURCE
cloudstore/cinder                                              encryption  on     local
cloudstore/cinder/volume-8ae498b7-5866-60da-85f6-d22d6bc932e9  encryption  on     inherited from cloudstore/cinder

Using the above method neither Cinder or Nova are aware of the encryption of the volumes nor are they involved in the key management. 

We are investigating what will be required to extend the Solaris ZFS drivers for Cinder so that Cinder is involved in or at least aware of ZFS encryption and then eventuall the key management since Cinder has some support for this already and a future OpenStack release will be extending this via the Barbican project.

-- Darren J Moffat

Oracle Solaris 11 - Engineered for Cloud

Today's release of Oracle Solaris 11.2 is especially meaningful for many of us in Solaris Engineering that have been hard at work over the last few years making OpenStack Cloud Infrastructure a first class Solaris technology. Today we release not only one of the most significant, complete, and solid versions of Solaris ever, with many new cloud virtualization features, but also included is the fully integrated cloud infrastructure software itself....everything needed (from a software perspective anyway ;)) to stand up a fully functional, OpenStack cloud system providing Infrastructure as a Service (IaaS), and Cloud block/object storage on both SPARC and x86 based systems.

Why is the Solaris Engineering Team tackling Cloud Infrastructure? For the Enterprise, what we consider to be the "Operating System" is shifting thanks to the rise of cloud computing. When you think about the role of an Operating System, what comes to mind? What does it do, fundamentally? Of course, it's the software that manages and allocates compute resources to users and workloads. It virtualizes those resources (CPU, memory, persistent storage) to provide applications with elasticity in their resource use. It runs workloads, hosts services, and provides APIs and interfaces for both workloads and users of those services. Operating Systems have tended to do this within the confines of single physical systems (or VMs) however.

Cloud Systems fundamentally need to provide all of these same basic OS services as well. From a pool of virtualized compute, networking, and storage, those resources need to be virtualized and allocated. Applications needs to have the illusion of resource elasticity to enable them to scale to meet the demands of the workload and users...and the Cloud System needs to run workloads and host services.

We've evolved from the time when enterprise applications were simply comprised of a number of processes/threads running on bare metal or in a VM consuming CPU, memory, storage, and talking over the network...and we see the enterprise OS evolving as well. Today's and tomorrow's enterprise applications are distributed workloads and cloud services that are hosted and run on cloud systems spanning many physical nodes. OpenStack provides a standard set of interfaces which have enabled us to evolve Solaris into a fully open, yet very differentiated platform for hosting cloud services and workloads.

That differentiation comes in part because we've built OpenStack on Solaris to seamlessly leverage many new features newly available with Solaris 11.2, including Kernel Zones based virtualization being offered up via OpenStack Nova, Unified Archive based Image deployment served up via Glance, and Elastic Virtual Switch based SDN managed by OpenStack Neutron. Solaris also provides ZFS backed cloud block and object storage (though OpenStack Cinder and Swift) over iSCSI and Fiber Channel connected storage and/or via Oracle's ZFS Storage Appliance(s).

Differentiation also comes about because Solaris based OpenStack has at its foundation the platform and technology you know and trust for running your mission critical enterprise workloads. Unparalleled reliability, scalability, efficiency and performance...both for hosting mission critical cloud services, as well as your mission critical cloud infrastructure, are all just as important as they've always been.

So what's the best way to get started? You don't need a massive sprawl of infrastructure to begin. With just a system or two, you can get create your own Solaris based OpenStack cloud providing Infrastructure As A Service (IaaS). Check out Getting Started with OpenStack on Solaris 11.2 to get started. You can also find Solaris 11.2 in the OpenStack Marketplace.

You'll find packages for the Havana version of OpenStack available in the Solaris 11 package repositories, including Nova, Neutron, Cinder, Glance, Keystone, Horizon, and Swift.

If you run into issues, or have questions, feel free to drop us a note at're happy to help! Enjoy!


Oracle OpenStack is cloud management software that provides customers an enterprise-grade solution to deploy and manage their entire IT environment. Customers can rapidly deploy Oracle and third-party applications across shared compute, network, and storage resources with ease, with end-to-end enterprise-class support. For more information, see here.


July 2014 »