• October 14, 2015

Global Impacts of Recent Leaks

Recent routing leaks remind us why monitoring Internet routing and performance is important and requires effective tools.  Routing leaks are the 'benign cousin' of the malicious BGP route hijack.  They happen accidentally, but the result is the same: traffic to affected prefixes is redirected, lost, or intercepted.  And if they happen to you, your online business and brand suffers.

In this blog, we look at examples of a full-table peer leak, an origination leak, and a small peer leak and what happens to traffic when these incidents occur.  As we will see, some events can go on for years, undetected and hence, unremediated, but extremely impactful never the less.  As you read this blog, keep the following  questions in mind.  Would  you know if the events described here were happening to you?  Would you know how to identify the culprit if you did?


iTel/Peer1 routing leak

Starting on 10 October at 10:54 UTC, iTel (AS16696) leaked a full routing table (555,010 routes) to Peer 1 (AS13768).  Normally, iTel exports 49 routes to Peer 1;  however, over the course of several minutes, it leaked 436,776 routes from Hurricane Electric (AS6939) and 229,537 routes from Shaw Communications (AS6327), two of iTel's transit providers.  Peer 1 then sent these routes on to its substantial peering base, resulting in a brief but widespread disruption in global routing.

Shortly afterwards, Peer 1 announced to its customers that it had suffered a widespread network outage.  While it is difficult to ascertain whether the routing leak was a cause of the network outage or a product of it, whenever a full routing table is leaked — especially by a highly-peered network like Peer 1 — the impacts can be far reaching.


Many thousands of our continuous global traceroute measurements were redirected into iTel through Peer 1 (visible in green in the graph below).

16696Some of the more prominent companies impacted include Microsoft (AS8075) and Netflix (AS2906).  The following illustrates the propagation profile of two Microsoft routes around the time of the leak.  The leak manifests itself as a Hurricane Electric bulge, as more peers begin accepting leaked routes to Microsoft that go through Peer 1, iTel and then Hurricane Electric before reaching Microsoft.  In each case, nearly the entire Internet elected to send traffic destined for these address ranges through the leaked routes.

Interestingly, routes to Netflix (AS2906) behaved quite differently.  As the leaked routes propagated, they were subsequently withdrawn, suggesting some sort of automated leak detection and remediation process attempting to counter such events.

As an example of a more distant impact, this leak amazingly resulted in Shaw Communications of Canada becoming a major upstream of China Telecom for a number of routes for almost an hour.  As we've explained in earlier blogs, routing leaks like these can often have widespread global impacts.  The following two graphs show how our routing peers reached China Telecom for and over time.  Shaw Communications (AS6327, light blue) isn't normally in the path for these routes (for any provider other than Shaw or its customers) and only made an appearance during the leak.

Consider the following traceroute, originating in Tanzania and directed toward China, about 9,000km away as the crow flies.  We would not expect the path to traverse Canada, of all places, but in this example that's exactly what happened.  First, Tata (AS6453) takes the traffic from Dar es Salaam, though Paris, London, New York, Chicago and Seattle before arriving at iTel Networks in Vancouver.  Then the path continues on to Los Angeles and finally to China by way of Taipei, Taiwan.  The nearly 500ms required here is just under that seen for geosynchronous satellite communications, satellites which orbit over 42,000 km above earth's surface!

trace from Dar es Salaam, TZ to (China Telecom) at 11:05 Oct 10, 2015
1  *
2       core.dar-es-salaam.aptus.co.tz                    0.885
3      if-0-0-1-110.core1.41B-Dar-Es-Salaam.as6453.net   0.610
4      Pos-channel3.core1.WYN-Marseille.as6453.net     119.783
5    if-0-2-2-27.tcore1.WYN-Marseille.as6453.net     300.206
6      if-8-1600.tcore1.PYE-Paris.as6453.net           310.017
7     if-3-6.tcore1.L78-London.as6453.net             308.617
8      if-2-2.tcore2.L78-London.as6453.net             307.179
9       if-20-2.tcore2.NYY-New-York.as6453.net          300.803
10       if-12-6.tcore1.CT8-Chicago.as6453.net           304.046
11        if-22-2.tcore2.CT8-Chicago.as6453.net           290.148
12      if-15-0-0.core1.00S-Seattle.as6453.net          281.590
13      if-4-1-1.core3.VCW-Vancouver.as6453.net         287.499
14 *
15     iTel Networks Inc (Vancouver, Canada)           339.794
16 *
17 *
18 *
19 *
20     CHINANET backbone network (Los Angeles)         307.980
21     CHINANET backbone network (Taipei, TW)          385.910
22       CHINANET backbone network (Taipei, TW)          405.530
23      CHINANET backbone network (Shanghai, CN)        459.055
24      CHINANET backbone network (Shanghai, CN)        481.006
25   CHINANET jiangxi province network (Ji’an, CN)   480.762
26      CHINANET jiangxi province network (Ji’an, CN)   475.348

The impacts of the leak continue to reverberate.  Ten prefixes of Canadian provider ViaNet are still routed (at least in part) along the leaked AS Path as follows:

... 6453 13768 16696 16696 16696 16696 16696 16696 16696 16696 6939 5690


Freifunk Rheinland leak

The day after the iTel/Peer 1 leak, Freifunk Rheinland e.V. (AS201701) of Germany initiated an origination leak — one in which the guilty party masquerades as the origin of the route, unlike the previous incident which left the origin intact.  Just after 23:00 UTC on 11 October 2015, AS201701, which normally announces four prefixes, originated (leaked) 4,423 prefixes.  For the most part, the leak was brief in duration and the routes didn't propagate very far across the Internet, so the impact was fairly minimal.

The following graphs show the portion of our peering base that observed AS201701 as the origin for two prefixes (shown in red).  The graph on left shows a Venezuelan prefix ( normally originated by Venezuelan incumbent CANTV (AS8048) and minimally impacted by the leak.  On the right, we see a Vietnamese prefix originated by Netnam (AS24173) that was greatly affected.  Why drives the dramatic difference in route propagation in these two cases?

As we have described in previous blog posts about routing leaks such as the Indosat leak of 2014 and the Telekom Malaysia leak of 2015, there are prefixes in the global routing table that are heavily prepended to everyone and hence are at extreme risk in these situations.  For what is most certainly a very bad oversight, the prefix is prepended six(!) times to all of its upstreams and is routed along the following AS Path.

... 6453 45903 24173 24173 24173 24173 24173 24173 24173

In these situations, the routes from the leaker will almost assuredly have a shorter AS Path and will always be selected when path length is the deciding criteria for BGP route selection.


Korea Telecom and Broadband Systems Corporation of Rwanda

Unlike the previous two examples, routing mishaps can also occur on a smaller scale.  Take for example the case of Broadband Systems Corporation of Rwanda.  For whatever reason, Broadband Systems Corporation (AS37288) peers with Korea Telecom (AS4766).  (How much traffic really flows between Rwanda and South Korea?)  However, Korea Telecom alternates how it routes the three prefixes it receives from Broadband Systems throughout the day.  It transits them for several hours, then it treats them as peer routes and doesn't send them to its peers and transit providers.  This oscillation has been going since 14 September 2013 when first came into existence.  Many important Rwandan government web resources appear to be hosted in this address space including the following.  mail.presidency.gov.rw www.presidency.gov.rw www.parliament.gov.rw www.usa.embassy.gov.rw www.russia.embassy.gov.rw www.rdfcsc.mil.gov.rw

Next, we see a graph of the propagation profile of each of the three routes exported to Korea Telecom over the past two weeks.  In the case of, it is a more-specific of a /22 and only is in circulation when Korea Telecom is routing it — guaranteeing all Internet traffic to this address space will have to pass through South Korea before making its way into Rwanda. The other two routes are at least partially available through normal Africa transit paths when Korea Telecom is transiting them.

The result is that for Internet users around the world, including those in East Africa, when these routes are being propagated, traffic to these sites is directed through South Korea before reaching Rwanda.  Consider the following traceroute from Kenya to Rwanda, via South Korea!   The straight line distance between the origin and destination is only 755 km, but these neighboring countries might as well be communicating via satellite, as the round trip times exceed 500ms.

trace from Nairobi, Kenya to at 21:13 Oct 11, 2015
1  *
2     RFC 1918 (private use)                           1.192
3   Safaricom Kenya                                  8.094
4  Safaricom Kenya                                  7.558
5    if-0-0-1.core1.N71-Fujairah.as6453.net          57.756
6   if-7-0-2-0.tcore1.WYN-Marseille.as6453.net     160.797
7     if-8-1600.tcore1.PYE-Paris.as6453.net          259.128
8    Tata (Paris, FR)                               368.336
9   prs-bb2-link.telia.net (Paris)                 360.714
10    nyk-bb2-link.telia.net (New York)              343.712
11   chi-b21-link.telia.net (Chicago)               259.029
12    sea-b1-link.telia.net  (Seattle)               415.526
13     ktc-ic-310746-sea-b1.c.telia.net               307.934
14   Korea Telecom (Seongnam-si, KR)                428.873
15    Korea Telecom (Seongnam-si, KR)                428.804
16   Korea Telecom (Seongnam-si, KR)                430.371
17    Korea Telecom (Seoul, KR)                      521.208
18   Content Network (Kigali, Rwanda)               521.676

Quartz ran a story in August entitled "Why your internet connection is slow wherever you are in Africa" based largely on a presentation made by Dyn Chief Scientist Jim Cowie at Africa DNS Forum.  The story makes the case that lack of local content hosting, and not constrained international links, is the leading cause of slow Internet experience in East Africa.  (See video posted at the bottom of this post.)  In the case we illustrated here, the locally hosted content is being severely undermined by poor BGP routing, even though it is readily available in the region.


It is a wonder that the Internet works as well as it does despite the continuous noise of routing goof-ups and the like.  For tools to help monitor for these types of incidents, please take a look at Dyn's Internet Intelligence family of products.  For further discussion about dangers of peer leaks, see our blog post from last year entitled Use Protection if Peering Promiscuously.


Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha