Beginning at 09:43 UTC today (6 June 2019), Swiss data center colocation company AS21217 leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany. China Telecom then announced these routes on to the global internet redirecting large amounts of internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Impacts were seen by some of Europe's largest networks in Switzerland, Holland, and France among other countries.
Often routing incidents like this only last for a few minutes, but in this case many of the leaked routes in this incident were in circulation for over two hours. In addition, numerous leaked routes were more-specifics of routed prefixes, suggesting the use of route optimizers or similar technology.
At 09:57 UTC, over 1,300 Dutch prefixes were announced in this leak. For 470 routes of AS1136, the leak took the form:
… 4134 21217 21217 21217 21217 21217 21217 13237 1136
If someone thought the prepending of AS21217 would keep these routes from leaking out, they were mistaken. Below is a visualization of the propagation profile of a prefix (18.104.22.168/16) announced by AS1136.
Beginning at 09:44 UTC, over 200 Swiss prefixes were announced by this leak. For 64 routes of AS3303, the leak took the form:
… 4134 21217 21217 21217 21217 21217 21217 6830 3303
Below is a visualization of the propagation profile of a prefix (22.214.171.124/17).
Each of these plots depicts peaks and valleys in the spread of propagation over time. These “features” are caused by changes in how China Telecom exported these routes to Tier-1 telecoms during the leak. Below we can see the preceding graphics side-by-side with equivalent views from the upstream perspective of AS4134.
Beginning at 10:10 UTC, 150 AS5410 prefixes were in this leak – 127 of which were more-specifics of existing routes. For example, 126.96.36.199/24 is not normally in the global routing table and is a more-specific of 188.8.131.52/10, announced by AS5410. These leaked routes took the form below:
… 4134 21217 21217 21217 21217 21217 21217 25091 5410.
Users of these services began noticing problems and reporting seeing traceroutes to their networks traversing China, as evidenced by user reactions on Twitter.
Many of our traceroute measurements were also sucked into this routing leak. The traceroute below begins in Google in Ashburn, Virginia and is destined for Vienna, Austria, but is detoured through China Telecom (hops 5-8) en-route to its destination.
Today’s incident shows that the internet has not yet eradicated the problem of BGP route leaks. It also reveals that China Telecom, a major international carrier, has still implemented neither the basic routing safeguards necessary both to prevent propagation of routing leaks nor the processes and procedures necessary to detect and remediate them in a timely manner when they inevitably occur. Two hours is a long time for a routing leak of this magnitude to stay in circulation, degrading global communications.
A great place for any telecom to start improving their routing hygiene is to join the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project.