• June 12, 2015

Global Collateral Damage of TMnet leak

The Washington Post recently published a great piece about the development and current weaknesses of the Border Gateway Protocol (BGP, which is used to route all Internet traffic). This morning Telekom Malaysia (a.k.a TMnet) helped to illustrate the points made in the article by leaking almost half of the global routing table via Level 3 at 08:44 UTC.

Some of the most affected companies were those peering with Telekom Malaysia.  The following graphics illustrate the impact to routes from Amazon and Cloudflare.

Google's extensive peering likely insulated it from some of the effects of having its routes leaked.  However, it didn't escape the incident completely unscathed.  Here is an example of a normal traceroute to Google's data center in Council Bluffs, Iowa from Prague, which goes via Frankfurt and London before crossing the Atlantic Ocean.

trace from Prague to Google, Council Bluffs, IA at 02:45 Jun 11, 2015
1  *
2    ge-6-14.car2.Prague1.Level3.net     16.583
3     ae-3-80.edge3.Frankfurt1.Level3.net 22.934
4      Level 3 (Frankfurt, DE)             23.101
5   Google (Frankfurt, DE)              23.796
6   Google (Frankfurt, DE)              24.086
7     Google (London, GB)                 32.709
8   Google (New York City)             103.091
9   Google (Council Bluffs)            133.098
10     Google (Council Bluffs)            133.245
11   Google (Council Bluffs)            133.536
12 *
13   Google (Council Bluffs)            132.643

During the routing leak, traces were redirected to Hong Kong (where Telekom Malaysia gets Level 3 transit) and across the Pacific Ocean for a performance hit of almost 400ms.

trace from Prague to Google, Council Bluffs, IA at 09:04 Jun 12, 2015
1  *
2    ge-6-14.car2.Prague1.Level3.net                     41.213
3  *
4    telekom-malaysia-berhad.xe-0-2-0.ar2.clk1.gblx.net 451.264
5  *
6   Google (Mountain View)                             509.481
7    Google (Mountain View)                             482.303
8   Google (Mountain View)                             459.441
9   Google (Council Bluffs)                            457.846
10     Google (Council Bluffs)                            468.626
11   Google (Council Bluffs)                            456.841
12 *
13   Google (Council Bluffs)                            509.298


One of the most severely impacted entities out there was Free of France.  Free (AS12322) provides service just in France but peers with Telekom Malaysia.  They also pre-pend their AS paths for routes transiting Cogent as follows:

… 174 12322 12322 12322 12322

The repeated '12322' in the AS path is a commonly used technique to de-prioritize a particular route — in this case: transit from Cogent. This is because in BGP, all other things being equal, the shortest AS path wins.  The risk of prepending is that by artificially lengthening an AS path, the routes from any leaker will often be shorter and more likely to be chosen, as illustrated in the following path.

… 3549 4788 12322

This example highlights an issue I've raised in the past year about the risks of promiscuous peering.

Of course, a routing change like this is bound to have an impact on performance.  Note the higher latencies in tan in the graphic below as traffic bound to Free in France is routed through Telekom Malaysia:

12322b_All_Sources-4In addition to leaking globally routed prefixes, Telekom Malaysia's leak introduced thousands of new routes from its private peering sessions.  The graphics below show the propagation footprint of two routes (Facebook in Malaysia and Layer 2 Broadband of Australia) over a period of four hours.

This isn't the first routing leak involving the former Global Crossing network (AS3549) in recent months.  In April, a small provider in Brazil (Tribunal Regional Federal da 1a Regiao, AS61569), that normally announces a single prefix to AS3549, leaked over 12,000 routes from another Brazilian ISP (Global Village Telecom, AS18881), including the routes of several CDNs that peer with AS18881.  As in the case of this morning's incident, AS3549 clearly hasn't bothered with a MAXPREF setting as it couldn't recognize that it shouldn't be propagating 12,000 routes from AS61569 — or 260,000 prefixes from AS4788.

//platform.twitter.com/widgets.jsUnlike Indosat's leak from last year in which Indosat's announced routes asserting itself as the origin, this leak preserved the origins of the routes, but inserted itself in the AS path.  While the big routing leaks get a lot of attention, be aware that these types of incidents occur on a smaller scale nearly every day.  Just yesterday Filanco of Russia leaked around 3,000 routes to MTS, briefly impacting networks around the world.  Below is the visualization of the upstreams of an Akamai route over a one hour period — Filanco shouldn't be here.
With a tool like Dyn's Internet Intelligence, companies can monitor and help mitigate the impact of routing snafus on their community of users.

And a Happy Friday to you, too!



Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha