Recent routing leaks remind us why monitoring Internet routing and performance is important and requires effective tools. Routing leaks are the 'benign cousin' of the malicious BGP route hijack. They happen accidentally, but the result is the same: traffic to affected prefixes is redirected, lost, or intercepted. And if they happen to you, your online business and brand suffers.
In this blog, we look at examples of a full-table peer leak, an origination leak, and a small peer leak and what happens to traffic when these incidents occur. As we will see, some events can go on for years, undetected and hence, unremediated, but extremely impactful never the less. As you read this blog, keep the following questions in mind. Would you know if the events described here were happening to you? Would you know how to identify the culprit if you did?
iTel/Peer1 routing leak
Starting on 10 October at 10:54 UTC, iTel (AS16696) leaked a full routing table (555,010 routes) to Peer 1 (AS13768). Normally, iTel exports 49 routes to Peer 1; however, over the course of several minutes, it leaked 436,776 routes from Hurricane Electric (AS6939) and 229,537 routes from Shaw Communications (AS6327), two of iTel's transit providers. Peer 1 then sent these routes on to its substantial peering base, resulting in a brief but widespread disruption in global routing.
Shortly afterwards, Peer 1 announced to its customers that it had suffered a widespread network outage. While it is difficult to ascertain whether the routing leak was a cause of the network outage or a product of it, whenever a full routing table is leaked — especially by a highly-peered network like Peer 1 — the impacts can be far reaching.
We're currently receiving notifications of an outage. We are investigating and will provide an update as soon as we have more information
— Peer 1 Hosting (@PEER1) October 10, 2015
Many thousands of our continuous global traceroute measurements were redirected into iTel through Peer 1 (visible in green in the graph below).
Some of the more prominent companies impacted include Microsoft (AS8075) and Netflix (AS2906). The following illustrates the propagation profile of two Microsoft routes around the time of the leak. The leak manifests itself as a Hurricane Electric bulge, as more peers begin accepting leaked routes to Microsoft that go through Peer 1, iTel and then Hurricane Electric before reaching Microsoft. In each case, nearly the entire Internet elected to send traffic destined for these address ranges through the leaked routes.
Interestingly, routes to Netflix (AS2906) behaved quite differently. As the leaked routes propagated, they were subsequently withdrawn, suggesting some sort of automated leak detection and remediation process attempting to counter such events.
As an example of a more distant impact, this leak amazingly resulted in Shaw Communications of Canada becoming a major upstream of China Telecom for a number of routes for almost an hour. As we've explained in earlier blogs, routing leaks like these can often have widespread global impacts. The following two graphs show how our routing peers reached China Telecom for 22.214.171.124/24 and 126.96.36.199/22 over time. Shaw Communications (AS6327, light blue) isn't normally in the path for these routes (for any provider other than Shaw or its customers) and only made an appearance during the leak.
Consider the following traceroute, originating in Tanzania and directed toward China, about 9,000km away as the crow flies. We would not expect the path to traverse Canada, of all places, but in this example that's exactly what happened. First, Tata (AS6453) takes the traffic from Dar es Salaam, though Paris, London, New York, Chicago and Seattle before arriving at iTel Networks in Vancouver. Then the path continues on to Los Angeles and finally to China by way of Taipei, Taiwan. The nearly 500ms required here is just under that seen for geosynchronous satellite communications, satellites which orbit over 42,000 km above earth's surface!
trace from Dar es Salaam, TZ to 188.8.131.52 (China Telecom) at 11:05 Oct 10, 2015
2 184.108.40.206 core.dar-es-salaam.aptus.co.tz 0.885
3 220.127.116.11 if-0-0-1-110.core1.41B-Dar-Es-Salaam.as6453.net 0.610
4 18.104.22.168 Pos-channel3.core1.WYN-Marseille.as6453.net 119.783
5 22.214.171.124 if-0-2-2-27.tcore1.WYN-Marseille.as6453.net 300.206
6 126.96.36.199 if-8-1600.tcore1.PYE-Paris.as6453.net 310.017
7 188.8.131.52 if-3-6.tcore1.L78-London.as6453.net 308.617
8 184.108.40.206 if-2-2.tcore2.L78-London.as6453.net 307.179
9 220.127.116.11 if-20-2.tcore2.NYY-New-York.as6453.net 300.803
10 18.104.22.168 if-12-6.tcore1.CT8-Chicago.as6453.net 304.046
11 22.214.171.124 if-22-2.tcore2.CT8-Chicago.as6453.net 290.148
12 126.96.36.199 if-15-0-0.core1.00S-Seattle.as6453.net 281.590
13 188.8.131.52 if-4-1-1.core3.VCW-Vancouver.as6453.net 287.499
15 184.108.40.206 iTel Networks Inc (Vancouver, Canada) 339.794
20 220.127.116.11 CHINANET backbone network (Los Angeles) 307.980
21 18.104.22.168 CHINANET backbone network (Taipei, TW) 385.910
22 22.214.171.124 CHINANET backbone network (Taipei, TW) 405.530
23 126.96.36.199 CHINANET backbone network (Shanghai, CN) 459.055
24 188.8.131.52 CHINANET backbone network (Shanghai, CN) 481.006
25 184.108.40.206 CHINANET jiangxi province network (Ji’an, CN) 480.762
26 220.127.116.11 CHINANET jiangxi province network (Ji’an, CN) 475.348
The impacts of the leak continue to reverberate. Ten prefixes of Canadian provider ViaNet are still routed (at least in part) along the leaked AS Path as follows:
... 6453 13768 16696 16696 16696 16696 16696 16696 16696 16696 6939 5690 18.104.22.168/22
Freifunk Rheinland leak
The day after the iTel/Peer 1 leak, Freifunk Rheinland e.V. (AS201701) of Germany initiated an origination leak — one in which the guilty party masquerades as the origin of the route, unlike the previous incident which left the origin intact. Just after 23:00 UTC on 11 October 2015, AS201701, which normally announces four prefixes, originated (leaked) 4,423 prefixes. For the most part, the leak was brief in duration and the routes didn't propagate very far across the Internet, so the impact was fairly minimal.
The following graphs show the portion of our peering base that observed AS201701 as the origin for two prefixes (shown in red). The graph on left shows a Venezuelan prefix (22.214.171.124/16) normally originated by Venezuelan incumbent CANTV (AS8048) and minimally impacted by the leak. On the right, we see a Vietnamese prefix originated by Netnam (AS24173) that was greatly affected. Why drives the dramatic difference in route propagation in these two cases?
As we have described in previous blog posts about routing leaks such as the Indosat leak of 2014 and the Telekom Malaysia leak of 2015, there are prefixes in the global routing table that are heavily prepended to everyone and hence are at extreme risk in these situations. For what is most certainly a very bad oversight, the prefix 126.96.36.199/23 is prepended six(!) times to all of its upstreams and is routed along the following AS Path.
... 6453 45903 24173 24173 24173 24173 24173 24173 24173
In these situations, the routes from the leaker will almost assuredly have a shorter AS Path and will always be selected when path length is the deciding criteria for BGP route selection.
Korea Telecom and Broadband Systems Corporation of Rwanda
Unlike the previous two examples, routing mishaps can also occur on a smaller scale. Take for example the case of Broadband Systems Corporation of Rwanda. For whatever reason, Broadband Systems Corporation (AS37288) peers with Korea Telecom (AS4766). (How much traffic really flows between Rwanda and South Korea?) However, Korea Telecom alternates how it routes the three prefixes it receives from Broadband Systems throughout the day. It transits them for several hours, then it treats them as peer routes and doesn't send them to its peers and transit providers. This oscillation has been going since 14 September 2013 when 188.8.131.52/24 first came into existence. Many important Rwandan government web resources appear to be hosted in this address space including the following.
Next, we see a graph of the propagation profile of each of the three routes exported to Korea Telecom over the past two weeks. In the case of 184.108.40.206/24, it is a more-specific of a /22 and only is in circulation when Korea Telecom is routing it — guaranteeing all Internet traffic to this address space will have to pass through South Korea before making its way into Rwanda. The other two routes are at least partially available through normal Africa transit paths when Korea Telecom is transiting them.
The result is that for Internet users around the world, including those in East Africa, when these routes are being propagated, traffic to these sites is directed through South Korea before reaching Rwanda. Consider the following traceroute from Kenya to Rwanda, via South Korea! The straight line distance between the origin and destination is only 755 km, but these neighboring countries might as well be communicating via satellite, as the round trip times exceed 500ms.
trace from Nairobi, Kenya to 220.127.116.11 at 21:13 Oct 11, 2015
2 172.17.28.10 RFC 1918 (private use) 1.192
3 18.104.22.168 Safaricom Kenya 8.094
4 22.214.171.124 Safaricom Kenya 7.558
5 126.96.36.199 if-0-0-1.core1.N71-Fujairah.as6453.net 57.756
6 188.8.131.52 if-7-0-2-0.tcore1.WYN-Marseille.as6453.net 160.797
7 184.108.40.206 if-8-1600.tcore1.PYE-Paris.as6453.net 259.128
8 220.127.116.11 Tata (Paris, FR) 368.336
9 18.104.22.168 prs-bb2-link.telia.net (Paris) 360.714
10 22.214.171.124 nyk-bb2-link.telia.net (New York) 343.712
11 126.96.36.199 chi-b21-link.telia.net (Chicago) 259.029
12 188.8.131.52 sea-b1-link.telia.net (Seattle) 415.526
13 184.108.40.206 ktc-ic-310746-sea-b1.c.telia.net 307.934
14 220.127.116.11 Korea Telecom (Seongnam-si, KR) 428.873
15 18.104.22.168 Korea Telecom (Seongnam-si, KR) 428.804
16 22.214.171.124 Korea Telecom (Seongnam-si, KR) 430.371
17 126.96.36.199 Korea Telecom (Seoul, KR) 521.208
18 188.8.131.52 Content Network (Kigali, Rwanda) 521.676
Quartz ran a story in August entitled "Why your internet connection is slow wherever you are in Africa" based largely on a presentation made by Dyn Chief Scientist Jim Cowie at Africa DNS Forum. The story makes the case that lack of local content hosting, and not constrained international links, is the leading cause of slow Internet experience in East Africa. (See video posted at the bottom of this post.) In the case we illustrated here, the locally hosted content is being severely undermined by poor BGP routing, even though it is readily available in the region.
It is a wonder that the Internet works as well as it does despite the continuous noise of routing goof-ups and the like. For tools to help monitor for these types of incidents, please take a look at Dyn's Internet Intelligence family of products. For further discussion about dangers of peer leaks, see our blog post from last year entitled Use Protection if Peering Promiscuously.