• March 1, 2012

Deployments over InfiniBand Infrastructure

I have been talking to many people who are working hard towards migrating an existing Ethernet based infrastructure or starting from grounds up to deploy their services and applications on InfiniBand based computing environments. At the end of the talk, it all looks very simple.. right ? But where to start.. that's the fun part. Lets give some insight into these type of discussions.

Recall one of my previous blog where I focused on a few keywords - Consolidation, Isolation and Virtualization. This blog is build up on the discussion I presented there.

First things first !

Go through your server sizing activity and get an idea on how many applications you can fit into a compute machine based on its resources like CPU, memory, disk space etc. When you have a number of applications to deploy or even migrate from Ethernet environments, you would probably go with a model where you group some servers to perform certain set of functions. Right ? Ok, now you want to plan how these groups communicate with each other and at a lower level, how the servers within each group communicate with each other. So, all this work will create a basis for network design and architecture.

Network Design with InfiniBand

For simplicity, I will assume that we have total eight compute machines divided equally into four groups. And by now, you have already decided what applications run where. Networks can be classified into two main categories - Internal and External. As these terms suggest, internal networks are primarily used for communications like server management, patching, updates, synchronization, health checks etc and external networks are used to advertise and offer the running services to your outside world which may even be Internet.

Question - Will all eight machines be talking to the external network ? May be, may be not. Lets assume that we have a few select machines for this and they are categorized in group A. This group could be serving the purpose of a load balancer. Sure ! The two machines in this group may also be hosting some other service applications e.g. a name service and proxy. Sure ! Now, what I am trying to highlight here is that a machine may be deployed with multiple applications, those of which may have a need to participate in multiple networks.

So, the network participation is not driven by machine as a hardware.. but its a function of each running service or application inside the machine.

External Networks

Group A faces the external world and if this is a case of Exalogic hardware, the external access is made possible through Ethernet over InfiniBand or EoIB. If we have two different functions e.g. load balancer and a name service exposed to external world, we may wish to isolate them. This can be done by creating separate Virtual NICs and assigning their own IP addresses. These may even be on unique VLANs. If isolation is not needed, we can simply use same IP address. Most of the applications have a "Listen" feature which dictates what socket (IP and port) they bind to. Use of in-line firewalls on these external networks is not uncommon and this will provide security to our edge computing group A. We can even go a step ahead and write iptable rules on the machine's inside group A for further security and hardening.

Internal Networks

Ok, so we took care of how the group A faces external world. In our example where we used load balancing as a function on the edge group A, it will have to communicate to internal groups where actual services may be running. And as I initially mentioned that we have already subdivided into three more groups - B, C and D. To build up a use case, lets assume that services to be load balanced are only deployed in group B. So that means, group A will be communicating to B on some specific sockets and we dont want any exposure from A to [C and D]. So, what do we do here ? We know that everything is connected over the same physical segment which is InfiniBand. Just like we will use VLAN in Ethernets, we will use Partition Keys here. The hosts in group A and B will share a common Partition key to allow a secured communication. Lets name this network as AB-10.

It may so happen that when machines in group B receives jobs to do from A, they need some help from machines in group C. Earlier, we excluded group C from network AB-10 for security. Hmm.. so what we do here is create a new network between group B and C just for this internal usage. Lets call it BC-20. And let me remind you that when you create a new network using VLAN or Partiton Keys, it means you have a new set of network interfaces and their associated IP addresses. This is what I discussed in my earlier blog - Networks and Virtualization.

What about group D ? Well, I left it alone on purpose just to demonstrate a use case that this group may be designated for some internal development and testing. But still, some network definitions must exist for this lone group D because there are two machines inside it and they may need to communicate to each other. Lets call it D-44. It will remain isloated from [A, B & C] as well as the external EoIB network created for A.

I have not forgotten about the NFS server.. so lets finish our conversation here. If all machines need to mount their respective shares from a common NFS server, then we can allow it to participate in all sub-networks AB-10, BC-20 and D-44. The access control policies can be enforced to allow only specific hosts to use the exported shares.

Wrap Up

What I have discussed here is that a set of machines can be grouped into functional roles which eventually drives the need of networks for communications - internal and external. Network isolation can be achieved in InfiniBand through Partition Keys just like VLANs in Ethernet. In-line firewalls can be used on EoIB network paths. Host level iptable rules can also be used if desired.

Join the discussion

Comments ( 1 )
  • martin francis kallukalam Saturday, September 8, 2012

    Neeraj, good series of posts on IB pertaining to Exalogic. couple of Qs, all pertaining to bringing DMZ web tier inside the exalogic with main focus on N/W security.

    (a): Let us say I would like to move the DMZ web tier into exalogic, without compromising security. To make it secure, I need to create a private VLAN similar to how I would do in a Cisco based DMZ .

    Private VLAN: this is to ensure that in case a DMZ node is compromised, the node can not be used to launch attack on other DMZ nodes or to app/web tier .For this in cisco world,I define a private VLAN for DMZ. In short a private VLAN will prevent DMZ web boxes to communicate between each other but at the same time will permit web boxes to communicate with the load balancer.

    I believe in IB world, this can be achieved using full and partial membership.bit 16 of the PKEY.

    Is such a configuration supported in Exalogic?

    (b):Let us say I bring DMZ web tier inside the exalogic. The web box will share an internal internal VLAN with other boxes in exalogic & with the NFS storage for mounts and such. If the web box is compromised, then its technically possible to compromise the internal network, since there is a common network element. In this case, I want to enforce security by creating a separate VLAN over a seperate IB partition just for the DMZ web box'es internal N/W. This way the web box will be on its own isolated internal Network VLAN .

    Is this supported?

    (c): Is bringing DMZ web zones inside the exalogic a recommended configuration with security in mind. Are there any better recommendations on how to deal with this.?

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.