Is there room for Intelligence in the Internet?

One of the well entrenched dogmas in
the Internet is the role of the intermediate nodes has to be kept
minimal, mainly to the level of a dumb lookup 'n forward action based
on packets' destination address. The cost of adding any intelligent
processing per packet is believed to be prohibitively high.
Traditionally, the function of a router a compared to that of the
airport in the air travel industry.

The space and time considerations for
routers and airports are summarized in the following table:




High cost of real estate

Closeness to the departure gates

Buffering capacity


Short time for travelers

Marginal per packet latency

Real estate in an airport is very
limited, because all vendors are competing for the areas of high
traffic of passengers as possible. These are typically the areas
close to the departure gates, where passengers tend to spend the
longest waiting time after clearing security. A router also has a
limited fast memory, just as a buffering space to absorb the
occasional peaks and jitters in packet inflows. It is not intended
for storage, because the moment it fills up routers will have to
resort to the dreaded packets dropping.

Time is also scarce for a traveler
through an airport. Since the whole travel experience is a non
productive and not very enjoyable time, the shorter it is the better.
Even more so for packets across an intermediate node in the Internet,
the marginal per-packet latency has to be kept as short as possible.
With dozens of intermediate nodes between the source and the
destination of a packets, the time costs for each extra hop can add
up to long delays that degrade the overall quality of service. It can
also cause some transports to back off rather aggressively, in an
attempt to avoid network congestion.

Providing any service in the airport
beyond getting to the next leg in the trip is expensive, because
vendors need to charge more to recover the high fixed costs. For
similar reasons, adding any extra processing in the way for packets
has been highly discouraged for router designers and intentionally
made difficult by the routing protocols.

for routing design

Routing protocols take into account the
limitations above. The lack of buffering capacity made it very
inconvenient to keep per flow state for example. For a router, all
packets are anonymous and all traffic is stateless. For example, a
router typically does not reconstitute fragments of big datagrams
because such functionality would require setting aside memory for
storing the fragments until the last one is received. The main
reasons behind that kind of restrictions are first, the fact that the
router is a shared resource, and tying it up by any specific flow
means it is less available for other occupants of the network, with a
risk of exposure to denial of service by malignant of bogus clients.
The second reason is the unavoidable extra latency from adding non
trivial packet processing in the network.

Of course, the need for adding more
than just forwarding of packets in the network is real and has been
addressed by special purpose hardware (Network Processors). These
hardware appliances commonly run some imbedded OS, with a limited
programmability. You may find specialized appliances that add some
transformation of the payload (e.g. encoding for video or audio
streams), or of the shape of the traffic., Other appliances may be
dedicated to intercept traffic of certain type for scanning and
analysis, or for metring and charging for example.

Adding any intelligence to the packets
in flight is costly. It involves parsing of each packet's chain of
relevant headers, looking up a predefined flow table to retrieve a
matching rule, then executing the action for that rule, and finally
keeping record (or account) of that action. This is typically
implemented in a special ASIC. It is no no job for software running
in running a general purpose operating system on an off-the-shelf

Or is it?

the Door More Intelligent Networks

Looking back at the
network virtualization and resource partitioning
(aka Crossbow),
it is a collection of innovative solutions to complex networking and
resource management problems in the data center. It brings a deeper
understanding of applications' and virtual machines' footprint on the
network, and offers a more efficient and controlled use of the
networking resources. From that point of view, Crossbow is an
effective solution for reducing the deployment and operation
complexity and costs of increasingly virtualized and network dense
data centers.

There is a more exciting (to me at
least) aspect to Crossbow, which is at the heart of this blog's
topic. The Crossbow technology ushers the convergence of the
networking and computing. By interacting directly with the advanced
hardware capabilities of modern NICs (support of multiple receive and
transmit rings/lanes, hardware classification, MAC addresses
filtering, etc), and by changing the scheduling algorithms of the
networking to be host driven instead of interrupt driven, Crossbow
delivers a vertically integrated network and computing stack that can
meet a great deal of the challenges facing the router designers
mentioned above. The laborious packet parsing and classification is
off-loaded to the hardware of an intelligent NIC. Packets are steered
on the fly to the hardware lane that matches the flow it was
programmed for. From that moment, packets are taken in accordance
with the configured priority, bandwidth allocation and CPU
assignment for the flow. They can be forwarded by IP after a route
lookup, delivered to destination application, or passed along to a
kernel-level consumer function that may transform the packets and/or
re-inject them in the traffic. The right balance between what is
off-loaded to the hardware vs what runs under the main OS on the one
side, and the virtually limitless programmability for the
applications on the general purpose OS on the other side, are what
makes adding any arbitrary processing on the data in transit credibly
possible in software, and with no need for custom made expensive
dedicated hardware.

Technology: A Game Changer

There are obvious limitations to what I
just described in the previous paragraph. The hardware capabilities
of the NICs, the total number of NICs that can be added, the speed
across the I/O bus, etc, all these are limiting factors. With that
said, I do view Crossbow as a disruptive technology that changes the
parameters of the game in many aspects. Essentially:

  • It lowers the bar of entrance for
    developers who see many opportunities for more intelligent
    processing inside the network, but have been so far kept out of the
    game because of the steep walls of proprietary and closed OS'es of
    major router vendors. You can design, develop, test and debug the
    application using the same convenient IDEs as desktop applications,

    Further, the developer can build a
    Crossbow VWire that emulates the conditions (multiple hops at
    varying link speeds and delays, etc) of an arbitrarily complex
    network when the behavior of applications can be studied and
    debugged in the virtual world, before real world deployment [1].

  • It changes the economics of the
    routers market. There are definitely a low to mid-range lines or
    routing products that do not require the top performance of the
    high-end specialized routers. The needs of that market can be met by
    commodity servers built with off-the-shelf components and running
    OpenSolaris/Crossbow. A solution vendor, can use these commodity
    appliances, which cost a fraction of the price of the traditional
    closed sources router, to conveniently and quickly develop their
    solution, bundle it as an appliance and market it at a very
    competitive price.

In future blogs, I'll be sharing the
experience of the Crossbow team's cooperation with a few partners
that successfully designed solutions based on commodity hardware
running OpenSolaris/Crossbow. Some of them were featured during the
2009 Crossbow BOF
and the latest JavaOne/CommunityOne


[1] S. Tripathi, N. Droux, T.
From hardware virtualizedd. Crossbow:
NICs to virtualized networks. In Proceedings of the ACM SIGCOMM

shop VISA’09 (To appear), 2009.


Post a Comment:
  • HTML Syntax: NOT allowed



« September 2016