The Washington Post recently published a great piece about the development and current weaknesses of the Border Gateway Protocol (BGP, which is used to route all Internet traffic). This morning Telekom Malaysia (a.k.a TMnet) helped to illustrate the points made in the article by leaking almost half of the global routing table via Level 3 at 08:44 UTC.
Some of the most affected companies were those peering with Telekom Malaysia. The following graphics illustrate the impact to routes from Amazon and Cloudflare.
Google's extensive peering likely insulated it from some of the effects of having its routes leaked. However, it didn't escape the incident completely unscathed. Here is an example of a normal traceroute to Google's data center in Council Bluffs, Iowa from Prague, which goes via Frankfurt and London before crossing the Atlantic Ocean.
trace from Prague to Google, Council Bluffs, IA at 02:45 Jun 11, 2015
2 184.108.40.206 ge-6-14.car2.Prague1.Level3.net 16.583
3 220.127.116.11 ae-3-80.edge3.Frankfurt1.Level3.net 22.934
4 18.104.22.168 Level 3 (Frankfurt, DE) 23.101
5 22.214.171.124 Google (Frankfurt, DE) 23.796
6 126.96.36.199 Google (Frankfurt, DE) 24.086
7 188.8.131.52 Google (London, GB) 32.709
8 184.108.40.206 Google (New York City) 103.091
9 220.127.116.11 Google (Council Bluffs) 133.098
10 18.104.22.168 Google (Council Bluffs) 133.245
11 22.214.171.124 Google (Council Bluffs) 133.536
13 126.96.36.199 Google (Council Bluffs) 132.643
During the routing leak, traces were redirected to Hong Kong (where Telekom Malaysia gets Level 3 transit) and across the Pacific Ocean for a performance hit of almost 400ms.
trace from Prague to Google, Council Bluffs, IA at 09:04 Jun 12, 2015
2 188.8.131.52 ge-6-14.car2.Prague1.Level3.net 41.213
4 184.108.40.206 telekom-malaysia-berhad.xe-0-2-0.ar2.clk1.gblx.net 451.264
6 220.127.116.11 Google (Mountain View) 509.481
7 18.104.22.168 Google (Mountain View) 482.303
8 22.214.171.124 Google (Mountain View) 459.441
9 126.96.36.199 Google (Council Bluffs) 457.846
10 188.8.131.52 Google (Council Bluffs) 468.626
11 184.108.40.206 Google (Council Bluffs) 456.841
13 220.127.116.11 Google (Council Bluffs) 509.298
One of the most severely impacted entities out there was Free of France. Free (AS12322) provides service just in France but peers with Telekom Malaysia. They also pre-pend their AS paths for routes transiting Cogent as follows:
… 174 12322 12322 12322 12322 18.104.22.168/12
The repeated '12322' in the AS path is a commonly used technique to de-prioritize a particular route — in this case: transit from Cogent. This is because in BGP, all other things being equal, the shortest AS path wins. The risk of prepending is that by artificially lengthening an AS path, the routes from any leaker will often be shorter and more likely to be chosen, as illustrated in the following path.
… 3549 4788 12322 22.214.171.124/12
This example highlights an issue I've raised in the past year about the risks of promiscuous peering.
Of course, a routing change like this is bound to have an impact on performance. Note the higher latencies in tan in the graphic below as traffic bound to Free in France is routed through Telekom Malaysia:
In addition to leaking globally routed prefixes, Telekom Malaysia's leak introduced thousands of new routes from its private peering sessions. The graphics below show the propagation footprint of two routes (Facebook in Malaysia and Layer 2 Broadband of Australia) over a period of four hours.
This isn't the first routing leak involving the former Global Crossing network (AS3549) in recent months. In April, a small provider in Brazil (Tribunal Regional Federal da 1a Regiao, AS61569), that normally announces a single prefix to AS3549, leaked over 12,000 routes from another Brazilian ISP (Global Village Telecom, AS18881), including the routes of several CDNs that peer with AS18881. As in the case of this morning's incident, AS3549 clearly hasn't bothered with a MAXPREF setting as it couldn't recognize that it shouldn't be propagating 12,000 routes from AS61569 — or 260,000 prefixes from AS4788.
— Dyn Research (@DynResearch) April 7, 2015
//platform.twitter.com/widgets.jsUnlike Indosat's leak from last year in which Indosat's announced routes asserting itself as the origin, this leak preserved the origins of the routes, but inserted itself in the AS path. While the big routing leaks get a lot of attention, be aware that these types of incidents occur on a smaller scale nearly every day. Just yesterday Filanco of Russia leaked around 3,000 routes to MTS, briefly impacting networks around the world. Below is the visualization of the upstreams of an Akamai route over a one hour period — Filanco shouldn't be here.
With a tool like Dyn's Internet Intelligence, companies can monitor and help mitigate the impact of routing snafus on their community of users.
And a Happy Friday to you, too!