• November 6, 2014

Chinese Routing Errors Redirect Russian Traffic

In recent weeks, Russian President Vladimir Putin announced a plan to enact measures to protect the Internet of Russia. In a speech to the Russian National Security Council he said, “we need to greatly improve the security of domestic communications networks and information resources.” Perhaps he should add Internet routing security to his list because, on a number of occasions in the past year, Russian Internet traffic (including domestic traffic) was re-routed out of the country due to routing errors by China Telecom. When international partners carry a country's domestic traffic out of the country, only to ultimately return it, there are inevitable  security and performance implications.

Last year, Russian mobile provider Vimpelcom and China Telecom signed a network sharing agreement and established a BGP peering relationship. However, as can often happen with these relationships, one party can leak the routes received from the other and effectively insert itself into the path of the other party's Internet communications. This happened over a dozen times in the past year between these two providers. This is a general phenomenon that occurs with some regularity but isn't often discussed in BGP security literature. In this blog post, we'll explore the issue via the lens of this single example. In a follow-on blog, we'll explore several additional examples.


How peering links can go wrong

In the world of BGP parlance, the term peering is overloaded. The same term can refer to a simple BGP adjacency between two autonomous systems (AS) as well as a particular type of relationship between two ASes where traffic is exchanged without money changing hands. This is also referred to as a "settlement-free" relationship and is the definition of peering used in this blog post.

Peering relationships between ISPs are quite common. By exchanging traffic directly without using a transit provider, each ISP saves on transit costs — although as transit prices fall the overhead of maintaining these peering relationships becomes less attractive. Regardless, from a routing standpoint, routes exchanged between two peering ISPs typically stay within the customer cone of each ISP. In the diagram below, ISP A sends routes from its customers to its peer ISP B. As a result, traffic from ISP B's customers following those routes flows over the peering link to ISP A's customers.


This arrangement can go wrong in a couple of different ways. ISP B could take the routes learned from ISP A and send them on to its providers and/or peers as illustrated in the graphic below. By doing this, ISP B effectively inserts itself into the path of traffic destined for ISP A from the outside world. We occasionally field questions from customers of Dyn IP Transit Intelligence product (formerly called Renesys Market Intelligence) who want to know why we reclassified one of their peers as a transit provider.

Typically after a little investigation, we determine that the peer is mistakenly passing their company's routes on to the peer's transit providers (or its peers) — effectively providing the company with free transit. In fact, one popular use case for IPTI is for ISP operators to monitor how their network's adjacencies get classified. When there is a difference between what IPTI shows and how the operator thought his network was connected, someone isn't propagating routes correctly.


The other way a peering relationship can go wrong is if ISP B announces routes learned from its providers and/or peers to ISP A. In this scenario (illustrated below), ISP A will then redirect traffic following these routes through ISP B, even if these routes are not normally handled by ISP B.


Unlike other routing leak scenarios, such as Indosat originating the entire global routing table or VolumeDrive leaking nearly the entire BGP table from one transit provider to another, the leaks described above occur with much greater frequency and with little fanfare. In fact, typically the parties involved are unaware of the glitch and, as a result, these more limited problems can persist much longer than the larger catastrophic incidents. Nonetheless, the results typically include traffic going where it wasn't intended with a corresponding immediate hit to performance and the potential for security implications.

The Vimpelcom - China Telecom connection

200px-VimpelCom_logo     505px-China_Telecom_Logo.svg

In September 2013, China Telecom and Vimpelcom (one of Russia’s “big three” mobile operators) announced an agreement to establish a direct network interconnection. We would have expected to see this relationship appear in BGP routing, either as a peering relationship or possibly a limited transit relationship to allow Vimpelcom to reach the Far East. However, on several occasions in the past year, we observed China Telecom mishandling routes it received from (and sent to) Vimpelcom, effectively inserting itself into the path of Internet traffic to and from Vimpelcom.

One of those occasions was the 5th of August of this year, when China Telecom (AS4134) announced routes from Vimpelcom (AS3216) to all of its peers for several hours (leak scenario 1 from above) which forced inbound traffic to Vimpelcom to go through China Telecom routers. During this incident, over 7,000 routes from Vimpelcom's customer cone were globally announced by China Telecom with AS paths in the following form:

   … {1299,3257,1273,3320,6461,174,…} 4134 3216 …

The following charts illustrate the impact of these incidents by looking at how traceroute measurements into Vimpelcom were affected. As the charts show, latencies spike as traffic is dragged out to China and through the molasses of China Telecom's under-provisioned international links.

NA_3216_vps01.xrs1 NA_3216_vps01.hnl1
NA_3216_vps01.sjc1 NA_3216_vps01.gdl1

The following example traceroute illustrates an errant path taken by traffic during this routing leak. In this example, the traceroute begins in Reston, Virginia where NTT takes it to California and hands it off to China Telecom. China Telecom then takes it to Shanghai, China and then to its routers in Frankfurt, Germany before entering Russia en-route to Omsk.

trace from Reston, VA to Omsk, Russia at 12:00 Aug 05, 2014
1  *
2 (NTT America, Ashburn, US)                1.010ms
3    (NTT America, Ashburn, US)                0.934ms
4  *
5    (NTT America, San Jose, US)              74.649ms
6     xe-0.chinanet.snjsca04.us.bb.gin.ntt.net 70.909ms
7    (China Telecom, San Jose, US)            71.451ms
8   (China Telecom, Shanghai, CN)           341.819ms
9    (China Telecom, Frankfurt, DE)          641.424ms
10   (China Telecom, Frankfurt, DE)          608.965ms
11  pe-r.Omsk.gldn.net                      632.054ms
12 (Vimpelcom, Omsk, RU)                   687.536ms
13    vpn2-gi0-0.omsk.corbina.net             682.153ms
14  128-73-155-213.broadband.corbina.ru     707.525ms

As seen from within China, latencies to Vimpelcom via the direct link from China Telecom got lower and more stable, but did not improve dramatically. In fact, these traces were still getting directed to China Telecom's routers in Frankfurt, Germany, and given the fact that overland round-trip latencies from Russia to China are around 120ms, the observed latencies suggest that such traffic may still go across submarine cables to reach Europe.

CN_3216_vps01.pek1 CN_3216_vps01.pvg1

If this routing arrangement is intended to provide Vimpelcom low-latency access to the Far East, it isn't working that well. The examples below show latencies increasing during a routing leak from locations in the Far East.

3216_Aug5_vps01.sin3 3216_Aug5_bgp02.tyo1

The August 5th event was one of the times that China Telecom briefly announced nearly a full BGP table (326,622 routes) to Vimpelcom, placing itself in the path of outbound traffic from Vimpelcom to the outside world — including Russian routes. (Leak scenario 2 from above.)

Below is a traceroute illustrating the new path to a New Hampshire ISP. Vimpelcom takes the traffic to Frankfurt, hands it over to China Telecom which takes it to Shanghai before handing it over to Cogent in Los Angeles. Cogent takes it across the country to Fairpoint Communications in New Hampshire.

trace from Moscow to Manchester, NH at 12:09 Aug 05, 2014
1  *
2 (Vimpelcom, Moscow, RU)                     0.743ms
3  mx01.Frankfurt.gldn.net                    40.574ms
4  beeline-gw3.china-telecom.net              43.198ms
5   (China Telecom, Shanghai, CN)             302.433ms
6  (China Telecom, Los Angeles, US)          479.642ms
7   (China Telecom, Los Angeles, US)          487.225ms
8  te0-7-0-24.ccr21.sjc03.atlas.cogentco.com 380.087ms
9   be2000.ccr21.sjc01.atlas.cogentco.com     375.079ms
10   be2164.ccr21.sfo01.atlas.cogentco.com     371.727ms
11   be2132.ccr21.mci01.atlas.cogentco.com     372.585ms
12    be2156.ccr41.ord01.atlas.cogentco.com     370.596ms
13   be2351.ccr21.cle04.atlas.cogentco.com     367.498ms
14   be2009.ccr21.alb02.atlas.cogentco.com     371.972ms
15   (Cogent, Albany, US)                      367.334ms
16 burl-lnk.ngn.east.myfairpoint.net         321.980ms
17 (Fairpoint Communications, Concord, US)   315.036ms
18  static.man.east.myfairpoint.net           321.682ms

But the path from Moscow to New Hampshire was always going to be long. Who's to say that Los Angeles (or even Shanghai) might not be a reasonable waypoint under certain conditions? Alternatively, we can pick something closer to Moscow to illustrate how crazy this routing is. The traceroute below shows Vimpelcom taking traffic to Frankfurt, handing it over to China Telecom which takes it to Shanghai before handing it over to Chello Broadband, which peers with China Telecom in Los Angeles. Chello then takes it from New York to Frankfurt (again!) and then into the German countryside.

trace from Moscow, Russia to Marburg an der Lahn, Germany at 14:29 Aug 05, 2014
1  *
2   (Vimpelcom, Moscow, RU)                  0.908ms
3    mx01.Frankfurt.gldn.net                 40.695ms
4    beeline-gw2.china-telecom.net           48.799ms
5     (China Telecom, Shanghai, CN)          290.810ms
6    (China Telecom, Los Angeles, US)       514.115ms
7     (China Telecom, Los Angeles, US)       537.933ms
8   213-46-190-217.aorta.net (New York)    414.365ms
9   (UPC Austria GmbH, Vienna, AT)         420.591ms
10   uk-lon02a-rd1-pos-4-0-0.aorta.net      421.530ms
11    (UPC Austria GmbH, Frankfurt, DE)      421.557ms
12   7111a-mx960-01-ae1.fra.unity-media.net 420.867ms
13    7111a-mx960-02-ae0.fra.unity-media.net 418.427ms
14    7211a-mx960-01-ae1.gie.unity-media.net 420.454ms
15     (Unitymedia, Marburg an der Lahn, DE)  423.105ms

Even Internet paths from Moscow to other parts of Russia were forced through China Telecom's routers. In the following example, a traceroute from Moscow is taken by Vimpelcom to Frankfurt, handed over to China Telecom's routers in Frankfurt and, (mercifully) redirected back into Russia via Megafon without getting directed out to Shanghai.  (This diversion of domestic Russian traffic is illustrated in the graphic at the beginning of this blog.)

trace from Moscow, Russia to Yaroslavl, Russia at 13:13 Aug 05, 2014
1  *
2  (Vimpelcom, Moscow, RU)             0.542ms
3   mx01.Frankfurt.gldn.net            37.006ms
4   beeline-gw4.china-telecom.net      39.505ms
5  ffm-b10-link.telia.net             41.481ms
6  ffm-bb2-link.telia.net             42.227ms
7   s-bb4-link.telia.net               42.894ms
8 s-b2-link.telia.net                41.528ms
9  megafon-ic-151430-s-b2.c.telia.net 42.707ms
10 *
11     (MegaFon, Volga,RU)                49.992ms
12  ysu1-ccr1036-1.yar.ru              50.301ms
13 (NETIS Telecom, Yaroslavl’, RU)    54.769ms


In September, ACM Magazine published a terrific piece by Boston University Professor Sharon Goldberg which asked "Why is it taking so long to secure Internet routing?". She does a great job covering the issues surrounding routing security and the efforts thus far to address its vulnerabilities. I would add peering leaks like those described in this blog to her list of unresolved issues, problems that are much less well known or even discussed. At present, the best defense is to monitor how your routes traverse the Internet (such as with Dyn Internet Intelligence) and to carefully filter the routes you receive.  Without both measures, your traffic could be easily misdirected, potentially hurting both the performance and security of your Internet communications.

Investors also weren't too excited about Putin's Internet security plan, causing Vimpelcom's stock to drop to a record low, but maybe they were disappointed that it didn't address routing security.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha