Broadcasting and multicasting are integral parts of the Internet Protocol suite. While there are many more unicast applications, multicasting does have its uses especially in the enterprise. It is a missed opportunity that none of the public infrastructure clouds of today support it.
In Ravello, using our software-defined network, a "clean" L2 network with full support for broadcasting and multicasting is available. This functionality is available in all of the clouds we support.
This post is the second part in a two-part series on cloud networking. In the first part, I talked about layer 2 (L2) networking. We saw that in today's public infrastructure clouds, there is very limited access to layer 2. This makes networking in the cloud very different from the datacenter, where there is normally full L2 access.
In this post I will talk about IP broadcasting and multicasting. I will give an introduction to what these are, if and how they are supported in the cloud, and I will give a few examples of applications that make use of it.
In the video below, I demonstrate some of the broadcasting and multicasting capabilities inside a Ravello application. I also show a real-life use case where I've built a highly available load balancer using keepalived and HAproxy. The keepalived is used to manage a single virtual IP that is shared between two HAProxy nodes.
The most common addressing mode in an IP network is called "unicast". In this mode, two hosts on the network communicate with each other. One host is usually called the server, and the other is called the client. The vast majority of all traffic on the Internet is unicast, if only because the Transmission Control Protocol (TCP), the most widely used Internet protocol, is always unicast.
There are other addressing modes besides unicast. IP version 4 defines broadcast and multicast, while IP version 6 has subsumed broadcast into its multicast support, and added a third mode called anycast.
In a broadcast addressing mode, a packet is addressed not to a single host, but to all hosts on the local network.
Multicast can be seen as a more general case of broadcast. In multicast, a network packet is not addressed to all hosts, but instead a group of hosts called a "multicast group". Multicast groups are dynamic. Hosts can join a group and leave it again. The joining and leaving happens through a protocol called Internet Group Management Protocol (IGMP). A multicast group is identified by an IP address, just like a host. In IP version 4, the range between 18.104.22.168 and 22.214.171.124 is reserved for multicast addresses.
When a host is joined to a multicast group, it receives messages addressed to the group. The protocol that is most commonly used with multicasting is the User Datagram Protocol (UDP). UDP is a very flexible protocol that can work with any addressing mode. TCP on the other hand works with unicast only.
Broadcasting and multicasting are not new technologies. In fact, they have existed for almost 30 years. Broadcasting was first introduced in RFC919 in October 1984, while multicasting was first defined in RFC966 in December 1985. Both date just a few years after the Internet Protocol itself was created.
Anycast is a relatively new addressing mode where a packet is sent to exactly one host in a group. It is present in IP version 6 only. I won't go into detail on anycast now but maybe I will get back to it in a future blog post.
In the previous section we talked about broadcasting and multicasting at the IP level (layer 3). At the level below, on layer 2, similar concepts exist. Let's have a look at the most widely deployed L2 technology, Ethernet. A packet on an Ethernet is called a "frame", and has an embedded source and destination address. An Ethernet address, also called a MAC address, consists of 48 bits and is commonly written in hexadecimal as 00:11:22:33:44:55. Two types of addresses exist: unicast and broadcast addresses. An address is a unicast address if the low bit in the high byte is 0, and broadcast if it is a 1. So, for example, 01:00:00:00:00:00 would be a broadcast address. A unicast frame is delivered only to the host with the specified Mac address, while a broadcast frame is delivered to all hosts.
When an IP unicast packet is passed to layer 2 so that it can be sent to the next hop, it is wrapped in an unicast Ethernet frame. The MAC address of the next hop is determined using a protocol called the Address Resolution Protocol (ARP, which incidentally uses broadcast Ethernet frames to find out the Mac address for a given IP).
IP broadcast and multicast do not use ARP. IP broadcasts are always sent to the "all-ones" Ethernet address ff:ff:ff:ff:ff:ff. Since the low bit of the high byte is a 1, this is a broadcast address, and it will be delivered to all hosts on the L2 network. IP multicast instead uses a formula to convert the IP multicast group address to an Ethernet address. This formula is described in RFC1112. The group address 126.96.36.199 for example is translated to 01:00:52:01:02:03. The mapping is not unique: multiple group addresses correspond to the same broadcast address on the Ethernet.
So, with all that background on broadcasting and multicasting at the various network layers, we can finally take a deeper look at the real topic of this blog: IP broadcasting and multicasting in the cloud.
In the first part of this series we saw that the primary public infrastructure clouds support only unicast at the L2 level. And with the background from the previous chapter we know that for broadcast and multicast at the IP level, we need L2 broadcast. Based on that, we can already conclude that IP broadcasting and multicasting in the cloud is not going to work.
In fact, the above conclusion is correct. I've looked at the documentation for Amazon EC2, Google Compute Engine and Microsoft Azure and they all state that IP broadcast and multicast are not supported. An interesting fact is that in the case of Amazon, it’s also not supported in VPC, which was designed to give an environment more similar to the datacenter.
People have tried to work around the lack of multicasting using various OS level tools. A few interesting ones are using n2n to set up a peer to peer L2 VPN between virtual machines, or using various approaches to turn multicast into unicast. Some of these approaches may have valid use cases. That said, in all cases, these solutions add significant complexity, and push what is essentially a network responsibility back into the OS.
While many more applications use unicast addressing, multicasting does have a few important use cases. The two main areas seem to be infrastructure for high availability solutions, and to implement "zero config" discovery mechanisms.
Examples of high availability solutions that use multicasting are the well-known keepalived (an implementation of Cisco's Virtual Router Redundancy Protocol or VRRP), uCarp, the Red Hat Cluster Suite (based on the open source Corosync/OpenAIS projects) and JGroups. In this category, there is also the venerable Veritas Cluster Server (VCS). It should be mentioned that some of these projects have grown unicast support recently, exactly because of the lack of multicasting in the cloud. However in all cases the most optimal solution is to use multicasting. The multicast networking in this category is used to send "heartbeat" messages. All nodes listen to these messages. If, at some point, a message is not received for a certain amount of time, the nodes assume something went wrong and can start a corrective action.
Examples of discovery solutions that use multicasting include the Apple Bonjour/Zeroconf protocol (also known as multicast DNS or DNS service discovery), the Java NoSQL databases Hazelcast and EhCache, and the Oracle Grid Infrastructure. These solutions use multicast to announce a presence or a status on the network, without having to explicitly configure which other nodes exist.
The Ravello software-defined network provides a "clean" L2 network that does not filter out any L2 frames, including broadcast frames. This means that both IP broadcasting and IP multicasting work transparently and, as expected, without any special handling.