Tuesday Jul 08, 2014

Network Virtualization High Availability

How to add high availability to the network infrastructure of a multitenant cloud environment using the DLMP aggregation technology introduced in Oracle Solaris 11.1.
This article is Part 1 of a two-part series. In Part 1, we will cover how to implement network HA using datalink multipathing (DLMP) aggregation technology, which was introduced in Oracle Solaris 11.1.

In Part 2 of this series, we will explore how to secure the network and perform typical network management operations for an environment that uses DLMP aggregations.

Once we virtualize a network cloud infrastructure using Oracle Solaris 11 network virtualization technologies—such as virtual network interface cards (VNICs), virtual switches, load balancers, firewalls, and routers—the network itself becomes an increasingly critical component of the cloud infrastructure.

In order to add resiliency to the network infrastructure layer, we need to implement an HA solution at this layer, such as we would do for any other mission-critical component of the data center.

A DLMP aggregation allows us to deliver resiliency to the network infrastructure by providing transparent failover and increasing throughput.
The objects that are involved in the process are VNICs, irrespective of whether they are configured inside Oracle Solaris Zones or in logical domains under Oracle VM Server for SPARC.

Using this technology, you can add HA to your current network infrastructure without the cross-organizational complexity that might often be associated with this kind of solution.

The benefits of this technology are clear and they take into account the limitations of existing technologies:

Since the IEEE 802.3ad trunking standard does not cover the case for building a trunk across multiple network switches, the network switch becomes a single point of failure (SPOF). Some vendors have added this capability to their product, but these implementations are vendor-specific and, therefore, prevent combining switches from multiple vendors when building a multi-switch trunk. Because Oracle Solaris provides resilience, DLMP aggregation can be implemented across two different network switches, thus eliminating the network switch as a SPOF. As an additional benefit, because the aggregation is implemented at the operating system layer, there is no need to set anything up on the switch.

Building a network HA solution that is based on previously available IP network multipathing (IPMP) can be a complex task. With IPMP, HA is implemented at Layer 3 (the IP layer), which needs to be configured in the global zones and within each zone, and requires multiple VNICs to be assigned to each zone or virtual machine (VM). This involves more configuration steps, requires spare IP addresses out of the address pool, and generally can be an error-prone process. In contrast, the DLMP aggregation setup is much simpler since all the configuration takes place at Layer 2 in the global zone; therefore, every non-global zone can directly benefit from the underlying technology without the need for additional configuration. In addition, every new Oracle Solaris Zone that is provisioned automatically benefits from this capability. Moreover, we can create an aggregation over four 10 Gb/sec network interfaces; combining all the interfaces together, we can achieve up to 40 Gb/sec of network bandwidth.

DLMP can provide additional benefits when employed together with other network virtualization technologies that are implemented in the Oracle Solaris 11 operating system, such as link protection and the ability to configure a bandwidth limit on a VNIC or a traffic flow to meet service-level agreements (SLAs). Combining these technologies provides for a uniquely compelling network solution in terms of HA, security, and performance in a cloud environment.

Monday Jun 24, 2013

How to Set Up a MongoDB NoSQL Cluster Using Oracle Solaris Zones

This article starts with a brief overview of MongoDB and follows with an example of setting up a MongoDB three nodes cluster using Oracle Solaris Zones.
The following are benefits of using Oracle Solaris for a MongoDB cluster:

• You can add new MongoDB hosts to the cluster in minutes instead of hours using the zone cloning feature. Using Oracle Solaris Zones, you can easily scale out your MongoDB cluster.
• In case there is a user error or software error, the Service Management Facility ensures the high availability of each cluster member and ensures that MongoDB replication failover will occur only as a last resort.
• You can discover performance issues in minutes versus days by using DTrace, which provides increased operating system observability. DTrace provides a holistic performance overview of the operating system and allows deep performance analysis through cooperation with the built-in MongoDB tools.
ZFS built-in compression provides optimized disk I/O utilization for better I/O performance.
In the example presented in this article, all the MongoDB cluster building blocks will be installed using the Oracle Solaris Zones, Service Management Facility, ZFS, and network virtualization technologies.

Figure 1 shows the architecture:

Friday Jun 14, 2013

Public Cloud Security Anti Spoofing Protection

This past week (9-Jun-2013) Oracle ISV Engineering participated in the IGT cloud meetup, the largest cloud community in Israel with 4,000 registered members.
During the meetup, ISV Engineering presented two presentations: 

Introduction to Oracle Cloud Infrastructure presented by Frederic Pariente
Use case : Cloud Security Design and Implementation presented by me 
In addition, there was a partner presentation from ECI Telecom
Using Oracle Solaris11 Technologies for Building ECI R&D and Product Private Clouds presented by Mark Markman from ECI Telecom 
The Solaris 11 feature that received the most attention from the audience was the new Solaris 11 network virtualization technology.
The Solaris 11 network virtualization allows us to build any physical network topology inside the Solaris operating system including virtual network cards (VNICs), virtual switches (vSwitch), and more sophisticated network components (e.g. load balancers, routers and fire-walls).
The benefits for using this technology are in reducing infrastructure cost since there is no need to invest in superfluous network equipment. In addition the infrastructure deployment is much faster, since all the network building blocks are based on software and not in hardware. 
One of the key features of this network virtualization technology is the Data Link Protection. With this capability we can provide the flexibility that our partners need in a cloud environment and allow them root account access from inside the Solaris zone. Although we disabled their ability to create spoofing attack  by sending outgoing packets with a different source IP or MAC address and packets which aren't types of IPv4, IPv6, and ARP.

The following example demonstrates how to enable this feature:
Create the virtual VNIC (in a further step, we will associate this VNIC with the Solaris zone):

# dladm create-vnic -l net0 vnic0

Setup the Solaris zone:
# zonecfg -z secure-zone
Use 'create' to begin configuring a new zone:
zonecfg:secure-zone> create
create: Using system default template 'SYSdefault'
zonecfg:secure-zone> set zonepath=/zones/secure-zone
zonecfg:secure-zone> add net
zonecfg:secure-zone:net> set physical=vnic0
zonecfg:secure-zone:net> end
zonecfg:secure-zone> verify
zonecfg:secure-zone> commit
zonecfg:secure-zone> exit

Install the zone:
# zoneadm -z secure-zone install
Boot the zone: # zoneadm -z secure-zone boot
Log In to the zone:
# zlogin -C secure-zone

NOTE - During the zone setup select the vnic0 network interface and assign the IP address.

From the global zone enable link protection on vnic0:

We can set different modes: ip-nospoof, dhcp-nospoof, mac-nospoof and restricted.
Any outgoing IP, ARP, or NDP packet must have an address field that matches either a DHCP-configured IP address or one of the addresses listed in the allowed-ips link property.
mac-nospoof: prevents the root user from changing the zone mac address. An outbound packet's source MAC address must match the datalink's configured MAC address.
dhcp-nospoof: prevents Client ID/DUID spoofing for DHCP.
only allows IPv4, IPv6 and ARP protocols. Using this protection type prevents the link from generating potentially harmful L2 control frames.

# dladm set-linkprop -p protection=mac-nospoof,restricted,ip-nospoof vnic0

Specify the IP address as values for the allowed-ips property for the vnic0 link:

# dladm set-linkprop -p allowed-ips= vnic0

Verify the link protection property values:
# dladm show-linkprop -p protection,allowed-ips vnic0

vnic0 protection rw mac-nospoof, -- mac-nospoof,
restricted, restricted,
ip-nospoof ip-nospoof,
vnic0 allowed-ips rw -- --

We can see that is set as allowed ip address.

Log In to the zone

# zlogin secure-zone

After we login into the zone let's try to change the zone's ip address:

root@secure-zone:~# ifconfig vnic0
ifconfig:could not create address: Permission denied

As we can see we can't change the zone's ip address!

Optional - disable the link protection from the global zone:

# dladm reset-linkprop -p protection,allowed-ips vnic0

NOTE - we don't need to reboot the zone in order to disable this property.

Verify the change

# dladm show-linkprop -p protection,allowed-ips vnic0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
vnic0 protection rw -- -- mac-nospoof,
vnic0 allowed-ips rw -- -- --

As we can see we don't have restriction on the allowed-ips property.

Conclusion In this blog I demonstrated how we can leverage the Solaris 11 Data link protection in order to prevent spoofing attacks.

Monday May 20, 2013

How To Protect Public Cloud Using Solaris 11 Technologies

When we meet with our partners, we often ask them, “ What are their main security challenges for public cloud infrastructure.? What worries them in this regard?”
This is what we've gathered from our partners regarding the security challenges:

1.    Protect data at rest in transit and in use using encryption
2.    Prevent denial of service attacks against their infrastructure
3.    Segregate network traffic between different cloud users
4.    Disable hostile code (e.g.’ rootkit’ attacks)
5.    Minimize operating system attack surface
6.    Secure data deletions once we have done with our project
7.    Enable strong authorization and authentication for non secure protocols

Based on these guidelines, we began to design our Oracle Developer Cloud. Our vision was to leverage Solaris 11 technologies in order to meet those security requirements.

First - Our partners would like to encrypt everything from disk up the layers to the application without the performance overhead which is usually associated with this type of technology.
The SPARC T4 (and lately the SPARC T5) integrated cryptographic accelerator allow us to encrypt data using ZFS encryption capability.
We can encrypt all the network traffic using SSL from the client connection to the cloud main portal using the Secure Global Desktop (SGD) technology and also encrypt the network traffic between the application tier to the database tier. In addition to that we can protect our Database tables using Oracle Transparent Data Encryption (TDE).
During our performance tests we saw that the performance impact was very low (less than 5%) when we enabled those encryption technologies.
The following example shows how we created an encrypted file system

# zfs create -o encryption=on rpool/zfs_file_system

Enter passphrase for 'rpool/zfs_file_system':
Enter again:

NOTE - In the above example, we used a passphrase that is interactively requested but we can use SSL or a key repository.
Second  - How we can mitigate denial of service attacks?
The new Solaris 11 network virtualization technology allow us to apply virtualization technologies to  our network by splitting the physical network card into multiple virtual network ‘cards’. in addition, it provides the capability to setup flow which is sophisticated quality of service mechanism.
Flows allow us to limit the network bandwidth for a specific network port on specific network interface.

In the following example we limit the SSL traffic to 100Mb on the vnic0 network interface

# dladm create-vnic vnic0 –l net0
# flowadm add-flow -l vnic0 -a transport=TCP,local_port=443 https-flow
# flowadm set-flowprop -p maxbw=100M https-flow

During any (Denial of Service) DOS attack against this web server, we can minimize the impact on the rest of the infrastructure.
Third -  How can we isolate network traffic between different tenants of the public cloud?
The new Solaris 11 network technology allow us to segregate the network traffic on multiple layers.

For example we can limit the network traffic based on the layer two using VLANs

# dladm create-vnic -l net0  -v 2 vnic1

Also we can be implement firewall rules for layer three separations using the Solaris 11 built-in firewall software.
For an example of Solaris 11 firewall see
In addition to the firewall software, Solaris 11 has built-in load balancer and routing software. In a cloud based environment it means that new functionality can be added promptly since we don't need an extra hardware in order to implement those extra functions.

Fourth - Rootkits have become a serious threat is allowing the insertion of hostile code using custom kernel modules.
The Solaris Zones technology prevents loading or unloading kernel modules (since local zones lack the sys_config privilege).
This way we can limit the attack surface and prevent this type of attack.

In the following example we can see that even the root user is unable to load custom kernel module inside a Solaris zone

# ppriv -De modload -p /tmp/systrace

modload[21174]: missing privilege "ALL" (euid = 0, syscall = 152) needed at modctl+0x52
Insufficient privileges to load a module

Fifth - the Solaris immutable zones technology allows us to minimize the operating system attack surface
For example: disable the ability to install new IPS packages and modify file systems like /etc
We can setup Solaris immutable zones using the zonecfg command.

# zonecfg -z secure-zone
Use 'create' to begin configuring a new zone.
zonecfg:secure-zone> create
create: Using system default template 'SYSdefault'
zonecfg:secure-zone> set zonepath=/zones/secure-zone
zonecfg:secure-zone> set file-mac-profile=fixed-configuration
zonecfg:secure-zone> commit
zonecfg:secure-zone> exit

# zoneadm -z secure-zone install

We can combine the ZFS encryption and immutable zones for more examples see:

Sixth - The main challenge of building secure BIG Data solution is the lack of built-in security mechanism for authorization and authentication.
The Integrated Solaris Kerberos allows us to enable strong authorization and authentication for non-secure by default distributed systems like Apache Hadoop.

The following example demonstrates how easy it is to install and setup Kerberos infrastructure on Solaris 11

# pkg install pkg://solaris/system/security/kerberos-5
# kdcmgr -a kws/admin -r EXAMPLE.COM create master

Finally - our partners want to assure that when the projects are finished and complete, all the data is erased without the ability to recover this data by looking at the disk blocks directly bypassing the file system layer.
ZFS assured delete feature allows us to implement this kind of secure deletion.
The following example shows how we can change the ZFS wrapping key to a random data (output of /dev/random) then we unmount the file system and finally destroy it.

# zfs key -c -o  keysource=raw,file:///dev/random rpool/zfs_file_system
# zfs key -u rpool/zfs_file_system
# zfs destroy rpool/zfs_file_system

In this blog entry, I covered how we can leverage the SPARC T4/T5 and the Solaris 11 features in order to build secure cloud infrastructure. Those technologies allow us to build highly protected environments without  the need to invest extra budget on special hardware. They also  allow us to protect our data and network traffic from various threats.
If you would like to hear more about those technologies, please join us at the next IGT cloud meet-up

Thursday Jan 24, 2013

How to Set Up a Hadoop Cluster Using Oracle Solaris Zones

This article starts with a brief overview of Hadoop and follows with an
example of setting up a Hadoop cluster with a NameNode, a secondary
NameNode, and three DataNodes using Oracle Solaris Zones.

The following are benefits of using Oracle Solaris Zones for a Hadoop cluster:

· Fast provision of new cluster members using the zone cloning feature

· Very high network throughput between the zones for data node replication

· Optimized disk I/O utilization for better I/O performance with ZFS built-in compression

· Secure data at rest using ZFS encryption

Hadoop use the Distributed File System (HDFS) in order to store data.
HDFS provides high-throughput access to application data and is suitable
for applications that have large data sets.

The Hadoop cluster building blocks are as follows:

· NameNode: The centerpiece of HDFS, which stores file system metadata,
directs the slave DataNode daemons to perform the low-level I/O tasks,
and also runs the JobTracker process.

· Secondary NameNode: Performs internal checks of the NameNode transaction log.

· DataNodes: Nodes that store the data in the HDFS file system, which are also known as slaves and run the TaskTracker process.

In the example presented in this article, all the Hadoop cluster
building blocks will be installed using the Oracle Solaris Zones, ZFS,
and network virtualization technologies.

Wednesday Mar 23, 2011

How Traffix Systems Optimized Its LTE Diameter Load Balancing and Routing Solutions Using Oracle Hardware and Software

Please read the following paper "How Traffix Systems Optimized Its LTE Diameter Load Balancing and Routing Solutions Using Oracle Hardware and Software"

This paper provides technical information on how Traffix Systems,
the leading Diameter protocol solutions vendor, optimized its
Long Term Evolution (LTE) Traffix Diameter Load Balancer and Traffix Diameter Router to benefit from Oracle's software and

This paper also includes brief technical descriptions of how specific
Oracle Solaris features and capabilities are implemented in the Traffix

Wednesday Nov 25, 2009

Solaris Zones migration with ZFS

In this entry I will demonstrate how to migrate a Solaris Zone running
on T5220 server to a new  T5220 server using ZFS as file system for
this Zone.
Introduction to Solaris Zones

Solaris Zones provide a new isolation primitive for the Solaris OS,
which is secure, flexible, scalable and lightweight. Virtualized OS
services  look like different Solaris instances. Together with the
existing Solaris Resource management framework, Solaris Zones forms the
basis of Solaris Containers.

Introduction to ZFS

ZFS is a new kind of file system that provides simple administration,
transactional semantics, end-to-end data integrity, and immense
Architecture layout :

Prerequisites :

The Global Zone on the target system must be running the same Solaris release as the original host.

To ensure that the zone will run properly, the target system must have
the same versions of the following required operating system packages
and patches as those installed on the original host.

Packages that deliver files under an inherit-pkg-dir resource
Packages where SUNW_PKG_ALLZONES=true
Other packages and patches, such as those for third-party products, can be different.

Note for Solaris 10 10/08: If the new host has later versions of the
zone-dependent packages and their associated patches, using zoneadm
attach with the -u option updates those packages within the zone to
match the new host. The update on attach software looks at the zone
that is being migrated and determines which packages must be updated to
match the new host. Only those packages are updated. The rest of the
packages, and their associated patches, can vary from zone to zone.
This option also enables automatic migration between machine classes,
such as from sun4u to sun4v.

Create the ZFS pool for the zone
# zpool create zones c2t5d2
# zpool list

zones   298G    94K   298G     0%  ONLINE  -

Create a ZFS file system for the zone
# zfs create zones/zone1
# zfs list

zones         130K   293G    18K  /zones
zones/zone1    18K   293G    18K  /zones/zone1

Change the file system permission
# chmod 700 /zones/zone1

Configure the zone
# zonecfg -z zone1

zone1: No such zone configured
Use 'create' to begin configuring a new zone.

zonecfg:zone1> create -b
zonecfg:zone1> set autoboot=true
zonecfg:zone1> set zonepath=/zones/zone1
zonecfg:zone1> add net
zonecfg:zone1:net> set address=
zonecfg:zone1:net> set physical=e1000g0
zonecfg:zone1:net> end
zonecfg:zone1> verify
zonecfg:zone1> commit
zonecfg:zone1> exit
Install the new Zone
# zoneadm -z zone1 install

Boot the new zone
# zoneadm -z zone1 boot

Login to the zone
# zlogin -C zone1

Answer all the setup questions

How to Validate a Zone Migration Before the Migration Is Performed

Generate the manifest on a source host named zone1 and pipe the output
to a remote command that will immediately validate the target host:
# zoneadm -z zone1 detach -n | ssh targethost zoneadm -z zone1 attach -n -

Start the migration process

Halt the zone to be moved, zone1 in this procedure.
# zoneadm -z zone1 halt

Create snapshot for this zone in order to save its original state
# zfs snapshot zones/zone1@snap
# zfs list

zones             4.13G   289G    19K  /zones
zones/zone1       4.13G   289G  4.13G  /zones/zone1
zones/zone1@snap      0      -  4.13G  -
Detach the zone.
# zoneadm -z zone1 detach

Export the ZFS pool,use the zpool export command
# zpool export zones

On the target machine
 Connect the storage to the machine and then import the ZFS pool on the target machine
# zpool import zones
# zpool list

zones   298G  4.13G   294G     1%  ONLINE  -
# zfs list

zones             4.13G   289G    19K  /zones
zones/zone1       4.13G   289G  4.13G  /zones/zone1<
zones/zone1@snap  2.94M      -  4.13G  -

On the new host, configure the zone.
# zonecfg -z zone1

You will see the following system message:

zone1: No such zone configured

Use 'create' to begin configuring a new zone.

To create the zone zone1 on the new host, use the zonecfg command with the -a option and the zonepath on the new host.

zonecfg:zone1> create -a /zones/zone1
Commit the configuration and exit.
zonecfg:zone1> commit
zonecfg:zone1> exit
Attach the zone with a validation check.
# zoneadm -z zone1 attach

The system administrator is notified of required actions to be taken if either or both of the following conditions are present:

Required packages and patches are not present on the new machine.

The software levels are different between machines.

Note for Solaris 10 10/08: Attach the zone with a validation check and
update the zone to match a host running later versions of the dependent
packages or having a different machine class upon attach.
# zoneadm -z zone1 attach -u

Solaris 10 5/09 and later: Also use the -b option to back out specified patches, either official or IDR, during the attach.
# zoneadm -z zone1 attach -u -b IDR246802-01 -b 123456-08

Note that you can use the -b option independently of the -u option.

Boot the zone
# zoneadm -z zone1 boot

Login to the new zone
# zlogin -C zone1

[Connected to zone 'zone1' console]

Hostname: zone1

All the process took approximately five minutes

For more information about Solaris ZFS and Zones


This blog covers cloud computing, big data and virtualization technologies


« March 2015