Sunday Jul 19, 2015

OpenStack Juno - Flat Networking and Fixed IP Addresses

OpenStack Juno was recently released as a part of Solaris 11.3 Beta. There have been a number of very good blogs around the new capabilities of OpenStack Juno and how they relate to Solaris 11.  Networking within OpenStack is one of the areas where a number of people struggle, since it can get pretty complicated pretty quickly.  I always rely on the blogs that Girish has done as my source of truth for OpenStack networking in Solaris.

OpenStack Neutron has thus far delivered networks based either on VLANs or VXLANs.  Neutron, using the Solaris Elastic Virtual Switch, creates new internal, non-routed networks over either VLANs or VXLANs, whichever is prefered.  The internal network provides strong network isolation for tenants from each other so that traffic does not get leaked from one tenant to another.  They also provide the flexibility to create whatever kind of network infrastructure is required within a tenant and keep it private to the tenant.

Once the tenant wants packets to get out to the rest of the world, Neutron provides floating IP addresses for that purpose.  Just a quick word on nomenclature.  The address that Neutron assigns to a guest is its fixed address, even though it is usually assigned via DHCP.  This address is private to the tenant and may not even be routed beyond its internal subnet.  A floating IP address is an address assigned from a pool of externally accessible addresses that is mapped to a particular fixed address.  In Solaris, using the EVS, this is implemented with a bidirectional NAT.  The notion is that there are probably only a fraction of the guests that really need to communicate with hosts outside their tenant, and even fewer of them require inbound connections and a specific IP address.  So the pool of floating IP addresses is typically far smaller than the pool of fixed IP addresses.

The idea of floating IP addresses and fixed IP addresses and VLANs or VXLANs to separate tenants is quite powerful, especially in a multi-tenant cloud world.  However, many of the enterprise customers that I deal with have existing networking infrastructures, firewall rules external to the cloud, monitoring tools, and a host of network procedures that make this ad hoc, flexible networking somewhat of a challenge.

Flat Networks

With OpenStack Juno in Solaris 11.3 (or Solaris 11.2 SRU 10 and newer), flat networking is also available.  Flat networks, in the parlance of OpenStack and the EVS are untagged networks, typically already existing infrastructure, where you want to directly apply OpenStack guests.  Only a single flat network per Neutron environment is supported at present.  But, this provides a way to provision OpenStack guests directly to an existing network infrastructure without the requirement of using floating IP addresses and NAT to access the outside world.

This short blog will show you how to configure a flat network and provision guests to it.  Additionally, it will show you how to take back control over IP addresses and MACs.  Typically OpenStack relies on DHCP, but in many enterprise environments, both MACs and IP addresses are tightly controlled for a variety of reasons.

Initial Configuration

We will start with the assumption that you already have a working OpenStack.  This might be single-node cloud based on the Solaris 11 UAR, or based on some of the other blogs and documents in OTN.  But, we presume that you already have OpenStack installed and configured.  We will work with the command line rather than through the Horizon BUI for this exercise.  Horizon is a nice, simple BUI, but it hardly exposes all of the capabilities of OpenStack without customization.  Through the command line, we can be much more precise in what we do.

The system I have used for these examples is a single-node OpenStack running within a Virtual Box.  Expanding this to a multi-node configuration is a straight-forward exercise (once you have a multi-node configuration established).

I have already created a keypair within OpenStack called root.  This is just the public key for the root user in the global zone where the cloud has been defined.  It could be any key that would allow me to log into my guests.

I have also imported a generic Unified Archive to use to launch new guests.  Since I am in a VirtualBox, I can only use Non-Global Zones, so I did not include the install media in this UAR. I created a zone and created an archive of it.  I then used glance to import it.

Formatting note: For some reason, I cannot get Roller to deal well with vertical bar characters that OpenStack uses in its tables.  In order to display the output, I have replaced all vertical bars with exclamation points in order to maintain the tables.  Doesn't change the contents, just a little bit of the display. 

root@openstack:~# export OS_USERNAME=glance OS_PASSWORD=glance \
   OS_TENANT_NAME=service OS_AUTH_URL=http://localhost:5000/v2.0
root@openstack:~# glance image-list
+----+------+-------------+------------------+------+--------+
! ID ! Name ! Disk Format ! Container Format ! Size ! Status !
+----+------+-------------+------------------+------+--------+
+----+------+-------------+------------------+------+--------+
root@openstack:~# glance image-create --container-format bare --disk-format raw \
  --is-public true --name "basezone" --property architecture=x86_64 \
  --property hypervisor-type=solariszones \
  --property vm_mode=solariszones < /root/basezone.uar 
+----------------------------+--------------------------------------+
! Property                   ! Value                                !
+----------------------------+--------------------------------------+
! Property 'architecture'    ! x86_64                               !
! Property 'hypervisor_type' ! solariszones                         !
! Property 'vm_mode'         ! solariszones                         !
! checksum                   ! 6b4f8da111c756029026f590e4c45f75     !
! container_format           ! bare                                 !
! created_at                 ! 2015-07-16T15:39:04                  !
! deleted                    ! False                                !
! deleted_at                 ! None                                 !
! disk_format                ! raw                                  !
! id                         ! 7dea7644-7013-42d8-98b7-e778f13f1389 !
! is_public                  ! True                                 !
! min_disk                   ! 0                                    !
! min_ram                    ! 0                                    !
! name                       ! basezone                             !
! owner                      ! 57c37eb8e904495f9b94ce84290ffcc9     !
! protected                  ! False                                !
! size                       ! 382597120                            !
! status                     ! active                               !
! updated_at                 ! 2015-07-16T15:39:22                  !
! virtual_size               ! None                                 !
+----------------------------+--------------------------------------+
root@openstack:~# glance image-list
+--------------------------------------+----------+-------------+------------------+-----------+--------+
! ID                                   ! Name     ! Disk Format ! Container Format ! Size      ! Status !
+--------------------------------------+----------+-------------+------------------+-----------+--------+
! 7dea7644-7013-42d8-98b7-e778f13f1389 ! basezone ! raw         ! bare             ! 382597120 ! active !
+--------------------------------------+----------+-------------+------------------+-----------+--------+

Elastic Virtual Switch Configuration

Since we are starting with a working OpenStack, I will not go into all of the key creation and setup for the EVS. But, using a pretty standard default configuration, you can see that I have set up my cloud so that it is using VLANs and is using an etherstub that I created as its uplink port.  This allows for clean creation of VLANs on top of that etherstub, regardless of switch configuration.  But, none of that matters for the flat networking.

root@openstack:~# evsadm show-controlprop -o all
PROPERTY          PERM VALUE       DEFAULT   FLAT VLAN_RANGE  VXLAN_RANGE HOST
l2-type           rw   vlan        vlan      --   --          --          --
uplink-port       rw   l3stub0     --        no   200-300     --          --
uri-template      rw   ssh://      ssh://    --   --          --          --
uuid              r-   33d487c6-29b2-11e5-b04f-ebdabd3733c8 -- -- -- --   --
vlan-range        rw   200-300     --        --   --          --          --
vlan-range-avail  r-   200-300     --        --   --          --          --
vxlan-addr        rw   0.0.0.0     0.0.0.0   --   --          --          --
vxlan-ipvers      rw   v4          v4        --   --          --          --
vxlan-mgroup      rw   0.0.0.0     0.0.0.0   --   --          --          --
vxlan-range       rw   --          --        --   --          --          --
vxlan-range-avail r-   --          --        --   --          --          --
 

Instead, we will configure net0 as an additional uplink port and tag it to use flat networking instead of VXLANs or VLANs.  Note that the EVS assumes that the same network interface is used for the same function on each node in the environment, unless you use the -h option when you set the control properties.  In that way, you could say that net0 might be the flat uplink port on one host, but net1 or aggr0 is the flat network uplink on some other.  Likewise for configuration of which VXLANs or VLANs are presented.

root@openstack:~# evsadm set-controlprop -p uplink-port=net0,flat=yes
root@openstack:~# evsadm show-controlprop -o all
PROPERTY          PERM VALUE       DEFAULT   FLAT VLAN_RANGE  VXLAN_RANGE HOST
l2-type           rw   vlan        vlan      --   --          --          --
uplink-port       rw   l3stub0     --        no   200-300     --          --
uplink-port       rw   net0        --        yes  --          --          --
uri-template      rw   ssh://      ssh://    --   --          --          --
uuid              r-   33d487c6-29b2-11e5-b04f-ebdabd3733c8 -- -- -- --   --
vlan-range        rw   200-300     --        --   --          --          --
vlan-range-avail  r-   200-300     --        --   --          --          --
vxlan-addr        rw   0.0.0.0     0.0.0.0   --   --          --          --
vxlan-ipvers      rw   v4          v4        --   --          --          --
vxlan-mgroup      rw   0.0.0.0     0.0.0.0   --   --          --          --
vxlan-range       rw   --          --        --   --          --          --
vxlan-range-avail r-   --          --        --   --          --          --

Creating a Flat Network

Creating our flat network is pretty simple.  Just as with other networks, we configure a network and a subnet with neutron.  We will give an address range for dynamic address assignment via DHCP.  In this case, internal_network is an existing network created through Horizon in the standard way.  We will not use it for anything here.

root@openstack:~# export OS_USERNAME=admin OS_PASSWORD=secrete OS_TENANT_NAME=demo
root@openstack:~# neutron net-list
+--------------------------------------+------------------+------------------------------------------------------+
! id                                   ! name             ! subnets                                              !
+--------------------------------------+------------------+------------------------------------------------------+
! cec86f8b-9753-4d63-b56c-cafc4ee9b88d ! internal_network ! d7928b14-83f9-4e1b-a9c3-86f817028d87 192.168.33.0/24 !
+--------------------------------------+------------------+------------------------------------------------------+
root@openstack:~# neutron net-create flat_network --router:external true \
  --provider:physical_network external --provider:network_type flat
Created a new network:
+-----------------------+--------------------------------------+
! Field                 ! Value                                !
+-----------------------+--------------------------------------+
! admin_state_up        ! True                                 !
! id                    ! dc81a1dc-6f2c-42d9-bbeb-a4bcb5771dcd !
! name                  ! flat_network                         !
! provider:network_type ! flat                                 !
! router:external       ! True                                 !
! shared                ! False                                !
! status                ! ACTIVE                               !
! subnets               !                                      !
! tenant_id             ! 01059c84dd4f48f2b335c0ba70eab324     !
+-----------------------+--------------------------------------+
root@openstack:~# neutron subnet-create --name flat_subnet \
  --allocation_pool start=10.0.2.200,end=10.0.2.210 flat_network 10.0.2.0/24
Created a new subnet:
+-------------------+----------------------------------------------+
! Field             ! Value                                        !
+-------------------+----------------------------------------------+
! allocation_pools  ! {"start": "10.0.2.200", "end": "10.0.2.210"} !
! cidr              ! 10.0.2.0/24                                  !
! dns_nameservers   !                                              !
! enable_dhcp       ! True                                         !
! gateway_ip        ! 10.0.2.1                                     !
! host_routes       !                                              !
! id                ! c5286f85-b482-4fc8-9785-d6401d4c5178         !
! ip_version        ! 4                                            !
! ipv6_address_mode !                                              !
! ipv6_ra_mode      !                                              !
! name              ! flat_subnet                                  !
! network_id        ! dc81a1dc-6f2c-42d9-bbeb-a4bcb5771dcd         !
! tenant_id         ! 01059c84dd4f48f2b335c0ba70eab324             !
+-------------------+----------------------------------------------+
root@openstack:~# neutron net-list
+--------------------------------------+------------------+------------------------------------------------------+
! id                                   ! name             ! subnets                                              !
+--------------------------------------+------------------+------------------------------------------------------+
! cec86f8b-9753-4d63-b56c-cafc4ee9b88d ! internal_network ! d7928b14-83f9-4e1b-a9c3-86f817028d87 192.168.33.0/24 !
! dc81a1dc-6f2c-42d9-bbeb-a4bcb5771dcd ! flat_network     ! c5286f85-b482-4fc8-9785-d6401d4c5178 10.0.2.0/24     !
+--------------------------------------+------------------+------------------------------------------------------+

At this point, you can use regular Horizon tools to launch a guest on this network. Select the flat network as the network for the guest and Nova will do the rest. It will insert whatever key you asked for, use DHCP within that address range to set an address and create the guest.

However, a number of the enterprise customers I work with want more control than that. They want to create a guest with a specific IP address and sometimes even with a specific MAC. By default, OpenStack tends toward DHCP and dynamic assignment of MACs and IP addresses. However, there are many cases where you might want more control - firewall rules, centralized DNS and address assignment, LDOM VNET MAC assignment, etc.

In this case, we will fall back to the command line to create our guests. Before having Nova create the guests, we will create the network ports the way we want them with Neutron, and then have Nova use the ports we already create. neutron port-create is used to create the port. We can specify a MAC or an IP address or both. In fact, there are a number of other parameters you can define with neutron port-create. See neutron help port-create for more details

root@openstack:~# neutron port-create --mac-address 8:1:27:20:a0:1b \
  --fixed-ip subnet_id=c5286f85-b482-4fc8-9785-d6401d4c5178,ip_address=10.0.2.222 flat_network
Created a new port:
+----------------+-----------------------------------------------------------------------------------+
! Field          ! Value                                                                             !
+----------------+-----------------------------------------------------------------------------------+
! admin_state_up ! True                                                                              !
! device_id      !                                                                                   !
! device_owner   !                                                                                   !
! fixed_ips      ! {"subnet_id": "c5286f85-b482-4fc8-9785-d6401d4c5178", "ip_address": "10.0.2.222"} !
! id             ! 79cd3b1f-9489-4036-9590-f5278cd3d598                                              !
! mac_address    ! 8:1:27:20:a0:1b                                                                   !
! name           !                                                                                   !
! network_id     ! dc81a1dc-6f2c-42d9-bbeb-a4bcb5771dcd                                              !
! status         ! ACTIVE                                                                            !
! tenant_id      ! 01059c84dd4f48f2b335c0ba70eab324                                                  !
+----------------+-----------------------------------------------------------------------------------+

And there we have a network port created with a particular MAC and a particular IP address. Again, we don't have to specify both, but for this example, we will.

Now to use them.

Lauching a Guest with a Specific Neutron Port

Just to get a picture of our default environment, here are the list of our flavors, images, and keypairs. Pretty default.

root@openstack:~# nova flavor-list
+----+-----------------------------------------+-----------+------+-----------+------+-------+-------------+-----------+
! ID ! Name                                    ! Memory_MB ! Disk ! Ephemeral ! Swap ! VCPUs ! RXTX_Factor ! Is_Public !
+----+-----------------------------------------+-----------+------+-----------+------+-------+-------------+-----------+
! 1  ! Oracle Solaris kernel zone - tiny       ! 2048      ! 10   ! 0         !      ! 1     ! 1.0         ! True      !
! 10 ! Oracle Solaris non-global zone - xlarge ! 16384     ! 80   ! 0         !      ! 32    ! 1.0         ! True      !
! 2  ! Oracle Solaris kernel zone - small      ! 4096      ! 20   ! 0         !      ! 4     ! 1.0         ! True      !
! 3  ! Oracle Solaris kernel zone - medium     ! 8192      ! 40   ! 0         !      ! 8     ! 1.0         ! True      !
! 4  ! Oracle Solaris kernel zone - large      ! 16384     ! 40   ! 0         !      ! 16    ! 1.0         ! True      !
! 5  ! Oracle Solaris kernel zone - xlarge     ! 32768     ! 80   ! 0         !      ! 32    ! 1.0         ! True      !
! 6  ! Oracle Solaris non-global zone - tiny   ! 2048      ! 10   ! 0         !      ! 1     ! 1.0         ! True      !
! 7  ! Oracle Solaris non-global zone - small  ! 3072      ! 20   ! 0         !      ! 4     ! 1.0         ! True      !
! 8  ! Oracle Solaris non-global zone - medium ! 4096      ! 40   ! 0         !      ! 8     ! 1.0         ! True      !
! 9  ! Oracle Solaris non-global zone - large  ! 8192      ! 40   ! 0         !      ! 16    ! 1.0         ! True      !
+----+-----------------------------------------+-----------+------+-----------+------+-------+-------------+-----------+
root@openstack:~# nova image-list
+--------------------------------------+----------+--------+--------+
! ID                                   ! Name     ! Status ! Server !
+--------------------------------------+----------+--------+--------+
! 7dea7644-7013-42d8-98b7-e778f13f1389 ! basezone ! ACTIVE !        !
+--------------------------------------+----------+--------+--------+
root@openstack:~# nova keypair-list
+------+-------------------------------------------------+
! Name ! Fingerprint                                     !
+------+-------------------------------------------------+
! root ! 55:6c:ba:c5:bd:81:01:d1:cf:32:4c:cf:8d:c8:15:f5 !
+------+-------------------------------------------------+

To launch our guest, we use nova boot and specify the details of the guest we want to launch. In this case, we will create a tiny NGZ, using the basezone images, the root keypair, and the neutron port we created earlier.

root@openstack:~# nova boot --flavor 6 --image basezone --key-name root \
   --nic port-id=79cd3b1f-9489-4036-9590-f5278cd3d598 flatz2
+--------------------------------------+-------------------------------------------------+
! Property                             ! Value                                           !
+--------------------------------------+-------------------------------------------------+
! OS-DCF:diskConfig                    ! MANUAL                                          !
! OS-EXT-AZ:availability_zone          ! nova                                            !
! OS-EXT-SRV-ATTR:host                 ! -                                               !
! OS-EXT-SRV-ATTR:hypervisor_hostname  ! -                                               !
! OS-EXT-SRV-ATTR:instance_name        ! instance-00000004                               !
! OS-EXT-STS:power_state               ! 0                                               !
! OS-EXT-STS:task_state                ! scheduling                                      !
! OS-EXT-STS:vm_state                  ! building                                        !
! OS-SRV-USG:launched_at               ! -                                               !
! OS-SRV-USG:terminated_at             ! -                                               !
! accessIPv4                           !                                                 !
! accessIPv6                           !                                                 !
! adminPass                            ! gfQ6g3qXgqZf                                    !
! config_drive                         !                                                 !
! created                              ! 2015-07-17T15:47:38Z                            !
! flavor                               ! Oracle Solaris non-global zone - tiny (6)       !
! hostId                               !                                                 !
! id                                   ! ec42f82f-0399-46cd-b75b-807a5dd888bc            !
! image                                ! basezone (7dea7644-7013-42d8-98b7-e778f13f1389) !
! key_name                             ! root                                            !
! metadata                             ! {}                                              !
! name                                 ! flatz2                                          !
! os-extended-volumes:volumes_attached ! []                                              !
! progress                             ! 0                                               !
! security_groups                      ! default                                         !
! status                               ! BUILD                                           !
! tenant_id                            ! 01059c84dd4f48f2b335c0ba70eab324                !
! updated                              ! 2015-07-17T15:47:38Z                            !
! user_id                              ! f2a3c09637d546c5a230991a4df9ca44                !
+--------------------------------------+-------------------------------------------------+

And that's about all there is to that. Nova gets the guest built and uses the neutron network port we already created. We are able to specify our own MAC and IP as we go. Some colleagues of mine have done something similar using a first-boot service at install time to re-assign the DHCP IP address to one that they prefer. Their goal was to avoid the command line and avoid modifying Horizon. My customers are comfortable with command-line or API calls, since they plan to integrate this into an existing framework rather than using Horizon directly. I think this is simpler than having to do surgery after the fact. But, requirements vary.

Just to prove this all works, we will log in. Notice our keypair lets us ssh in as root. We have the IP address that we set. Since this is an NGZ, it's hard to see the MAC quite as directly. So, from the global zone of the nova compute node hosting this guest, we can see the vnic used for this guest and we see that it has the MAC we assigned. Very cool!

root@openstack:~# ssh root@10.0.2.222
Last login: Fri Jul 17 15:53:35 2015 from host-10-0-2-201
Oracle Corporation      SunOS 5.11      11.3    June 2015
root@flatz2:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128
net0              ip         ok           --         --
   net0/dhcp      inherited  ok           --         10.0.2.222/24
root@flatz2:~# exit
logout
Connection to 10.0.2.222 closed.
root@openstack:~# dladm show-vnic -m
LINK            OVER           SPEED  MACADDRESSES      MACADDRTYPES IDS
dh197a6832_2d_0 l3stub0        40000  fa:16:3e:97:5d:c5 fixed        VID:200
dh001d8202_30_0 net0           1000   fa:16:3e:c:d3:4a  fixed        VID:0
instance-00000003/net0 net0    1000   fa:16:3e:3e:57:ed fixed        VID:0
instance-00000004/net0 net0    1000   8:1:27:20:a0:1b   fixed        VID:0
instance-00000009/net0 net0    1000   fa:16:3e:8e:50:ec fixed        VID:0

Conclusion

While standard Neutron networking using VXLANs and VLANS provides the secure, flexible, dynamic networking environment that a multi-tenant cloud requires, sometimes that's not what you need. Many enterprises need are more controlled interface into legacy network infrastructures. Using flat networking, OpenStack can bridge the gap between the legacy network environment and the dynamic, elastic world of the cloud.

Wednesday Jul 09, 2014

Oracle Solaris and OpenStack Workshops

This is the summer of Oracle Solaris 11.2 and OpenStack workshops.  I'm on the road covering some, along with by team mates, Pavel Anni and Bob Netherton.

The Solaris workshops are full-day, hands-on workshops that will give you not only an introduction into Solaris 11, but a view into the new features and capabilities in Solaris 11.2.  Primarily, these will use Pavel Anni's fantastic hands-on lab and Virtualbox.

OpenStack workshops are a shorter, 2-3 hour event that will let you see what we are up to with OpenStack in Solaris and how OpenStack can help you move into a world of modern cloud computing.

These are only the events that I am participating in.  Take a look at http://oracle.com/events to see the rest of the events.  Also take a look at Bob's blog for more info on where he will be.

Here's my schedule:

Solaris Workshops

OpenStack Workshops

 Hope to see you at one of these events!

About

Interesting bits about Solaris, Virtualization, and Ops Center

Search

Archives
« September 2015
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today