On Wednesday, March 21, a massive power failure impacted large parts of northern Brazil, leaving tens of millions of people without electricity. Beginning at about 3:40pm local time (18:40 UTC), the outage was reportedly due to the failure of a transmission line near the Belo Monte hydroelectric station.
As occurred in a major power outage in Brazil in 2009, this power failure had a measureable impact on the country’s Internet. This is illustrated below through graphs from Oracle Dyn’s Internet Intelligence team based on BGP and traceroute data, as well as graphs from Akamai’s mPulse service, based on end user Web traffic.
The graphic below depicts the counts of available networks (lower graph) and unstable networks (upper graph) for Brazil in the latter half of March 21. The number of unstable networks spikes around 18:40 UTC as routers of ISPs in Brazil began re-routing traffic away from disabled connections, while the lower graph shows that the corresponding drop in available networks (i.e. routed prefixes) was minor when compared to the total number routes that define the Internet of Brazil.
In addition to aggregating BGP routing information from around the globe, the Internet Intelligence team also performs millions of traceroutes each day to endpoints in ISPs and enterprises across the Internet, providing us with insight into changes in Internet traffic paths and latencies. Based on the analysis of data from these traceroute measurements, we observed impacts to thirty networks (autonomous systems) from the power outage. Of those thirty, graphs highlighting six networks are shown in the figure below.
The impact of the power outage on March 21 is clearly evident within each of these graphs, with the number of completing traceroutes dropping significantly in each case.
The six graphs also show that the power failure had a mixed impact on the median latency of traceroutes to endpoints in the selected networks. The upper graphs for Eletrodata Ltda (AS262728), Softcomp Telecomunicacoes (AS52873), and Tascom Telecomunicaes (AS52871) show noticeable drops in median latency during the outage. These drops are likely attributable to a decrease in successfully completed traceroutes, with only traceroutes from nearby networks with short geographic paths reaching the endpoints. The upper graphs for FSF Tecnologia (AS61568) and Noroestecom Telecom (AS52579) show the opposite - a spike in latency during the outage. The increased median latency could be due to increased congestion, which would increase latency, or to the decrease in completing traceroutes, where the last measurements recorded were from networks far from the target. Interestingly, although the number of completed traceroutes to Electrodata Ltda (AS262740) dropped significantly, there appeared to be no meaningful impact on the median latency.
It is also interesting to look at the impact of the power failure from a ‘last-mile’ perspective – how did it impact an end-user’s ability to consume Web content? The reported tens of millions of consumers impacted by the power outage account for at least 10% of Internet users in Brazil – losing power to their homes means that they were unable to get online for several hours.
Akamai mPulse is a real user monitoring service that thousands of Web sites use to analyze their performance as experienced by actual end users. Analysis of the data aggregated across customers can also highlight trends in traffic to those customer sites. It is clear in the figure below that the number of Web pages loaded by end users in Brazil dropped significantly at 18:47 UTC, gradually recovering over the next several hours. Brief traffic spikes at 23:21 UTC and 00:23 UTC may be related to power being restored to impacted areas.
As noted above, the power failure affected large parts of northern Brazil, and using the mPulse data, we can drill down to examine the impact on traffic patterns across selected cities in the affected area. The figure below illustrates the relative page views from three cities in northern Brazil – Fortaleza, Salvador, and Recife. It is clear that in each of these cities, there was a sudden drop in traffic at the time of the power failure.
Looking more closely at Fortaleza in the figure below, the immediate drop in traffic is clear, followed by a gradual recovery over the next several hours. Interestingly, it appears that during the power failure, the percentage of users accessing the Web via mobile devices increased significantly.
Events like widespread power failures have an obvious impact on Web content consumption, as end users are unable to access the Internet from traditional wired last mile connections to their homes, apparently driving them to use mobile/cellular connectivity instead. These events also affect core network infrastructure, impacting latency and Internet traffic paths as traffic is routed away from disabled connections.
(Thank you to Paul Calvano, Senior Web Performance Architect at Akamai Technologies for the mPulse insights and graphs.)