In June, we launched the Internet Intelligence microsite, including the new Internet Intelligence Map. In July, we published the inaugural “Last Month in Internet Intelligence” overview, covering Internet disruptions observed during the prior month. The first summary included insights into exam-related outages and problems caused by fiber cuts. In this month’s summary, covering July, we saw power outages and fiber cuts, as well as exam-related and government-directed shutdowns, disrupt Internet connectivity. In addition, we observed Internet disruptions in several countries where we were unable to ascertain a definitive cause.
It is no surprise that power outages can wreak havoc on Internet connectivity – not every data center or router is connected to backup power, and last mile access often becomes impossible as well.
At approximately 20:00 GMT on July 2, the Internet Intelligence Map Country Statistics view showed a decline in the traceroute completion ratio and DNS query rate for Azerbaijan, related to a widespread blackout. These metrics gradually recovered over the next day. Published reports (Reuters, Washington Post) noted that the blackout was due to an explosion at a hydropower station, following an overload of the electrical system due to increased use of air conditioners, driven by a heat wave that saw temperatures exceed 100° F. Power was restored after several hours, but reportedly failed again, causing a second blackout, which again impacted the traceroute and DNS metrics as seen around 15:00 GMT on July 3.
Just a day later, Tropical Storm Maria caused an islandwide power outage in Guam, which disrupted Internet service on the island for several hours. However, Guam Power Authority (GPA) responded quickly once the storm had passed, with the Guam Daily Post noting that the GPA expected “to have substantial load for power restoration around 11 am”. In looking at the graphs shown below, they appear to have hit that target as the traceroute completion ratio and BGP routes count returned to prior levels around that time (Guam is GMT+10).
At the end of the month, Venezuela experienced a large power failure that left most of the capital city of Caracas without electricity, which caused a disruption in Internet connectivity as well. As shown in the figure below, both the traceroute and DNS metrics saw minor declines at around 13:00 GMT. Approximately two hours later, a Tweet from the country’s Energy Minister stated that 90% of the service had been restored in Caracas, and a subsequent Tweet several hours later explained that the initial fault in Caracas originated from voltage transformer control cables being cut. It appears that the measured metrics for Venezuela returned to regular levels several hours after power was restored.
On July 4, Twitter user @ADMIRAL12 posted the following Tweet:
أكثر من 80% من سرعة الإنترنت في #اليمن تم فقدانها نتيجة سبب غير معروف!#يمن_نت ماذا تفعلون؟!!— Rashad H. AL-Khmisy (@ADMIRAL12) July 4, 2018
More than 80% of Internet connection in #Yemen lost as a result of unknown reason!! Speed connection is very slowly .. Maximum speed I have 10 KB/s !! What are you doing #YemenNet ?
Oracle Director of Internet Analysis Doug Madory responded, noting “DNS query rate is down. Otherwise BGP routes and completing traceroutes are unaffected.” Minutes later, Madory also commented, “Both YemenNet ASNs lost transit from @GlobalCloudX and @etisalat (AS15412 and AS8966) at that time.” YemenNet’s issues can be seen in the Traffic Shifts graphs below.
A published report indicated that Houthi rebels disrupted Internet service to nearly 80% of Yemen by damaging a fiber optic cable in the port city of Hodeidah. The publication quoted a source from the Public Telecommunication Corporation, who explained "The cable that connects the country to the Internet was cut in three places in the districts of Al Kanawes and Al Marawya in Hodeidah as the Houthi militia continues to dig trenches in the area."
Just days later, Internet connectivity in Haiti was disrupted for more than a day, including a complete outage for local telecommunications provider Digicel Haïti. The Internet disruptions occurred in the midst of widespread protests over government plans to raise gas prices. Several Digicel fiber optic lines were cut, and the U.S. Embassy in the country stated “Telecommunications services, including Internet and phone lines, have been affected throughout Haiti.” The disruption to Haiti’s Internet connectivity, as well as Digicel’s outage, can be seen in the graphs below.
Following this damage to Digicel’s infrastructure, the company's Chairman took to Twitter to provide a status update on repairs:
Update on Digicel internet: fibre connection from PaP to the north of Haiti repaired at 5am today. Now configuring this connection to take intl. voice traffic. Other 2 fibre cuts towards St Marc fibre landing station being repaired but facing unrest during repairs. #haiti— Maarten Boute (@mboute) July 8, 2018
As would be expected, Digicel’s outage impacted connectivity for downstream customers. As seen in the graph below, traceroutes to targets in AS263685 (Sogebank, one of Haiti's three largest commercial banks) passed through Digicel ahead of the fiber cuts, as seen in the yellow area on the left side of the graph. Concurrent with the fiber cut, traceroutes fail to reach Sogebank for several hours, before they shift to using Télécommunications de Haití as an upstream provider, as seen in the green area on the graph. They maintained this connectivity arrangement for approximately three days before shifting back to Digicel.
On July 9, incumbent provider Telecommunication Services of Trinidad and Tobago (TSTT) was down for over three hours, causing a partial disruption to Internet connectivity in Trinidad and Tobago, as seen in the graphs below. A published report quoted a TSTT executive as stating that a major break in a fiber optic cable in the Chaguaramas area had caused a temporary disruption in all mobile data and Internet services.
Similar to the Sogebank discussion above, the Traffic Shifts graph below shows the impact of this cable cut on AS26317 (Lisa Communications), which uses TSTT as an upstream provider. As the graph shows, the vast majority of traceroutes to Lisa Communications passed through TSTT in the days prior to the cut, as see in the yellow area on the left side of the graph. Concurrent with the cut, the number of completed traceroutes briefly declines to approximately half of its average rate, although provider Columbus Communications Trinidad Limited quickly picked up the slack, as seen in the blue area. After approximately half a day, the majority of the Lisa Communications-bound traceroutes begin passing through TSTT once again, as seen in the return of the yellow area on the right side of the graph.
A fiber cut caused a multi-hour Internet disruption in Kenya on July 22, starting at approximately 06:30 GMT. A published report indicated that service was restored by 11:00 GMT, which aligns with the traceroute completion ratio and DNS query rates shown in the figure below.
Safaricom, Kenya’s largest telecommunications provider, issued a statement to customers that noted “We wish to apologize to our customers and partners that are currently experiencing voice and data outage, caused by multiple fiber link cuts affecting critical transmission equipment”. To that end, the Traffic Shifts graph below shows the impact of the cut on One Communications. A subsidiary of Safaricom since 2008, it relies on its parent for Internet connectivity. The cut caused a complete loss of completed traceroutes to targets in One Communications for several hours, until service was restored.
While not explicitly a cable cut, Internet connectivity in Bangladesh was significantly impacted at the end of July as the SeaMeWe-4 (SMW4) submarine cable was taken down from July 25-30 for maintenance, resulting in a loss of almost half of the country’s international Internet capacity. Repairs to the SMW4 cable also impacted the Internet in Bangladesh in May 2018, October 2017, and August 2011, as did cuts to the cable in June 2012. Taking down SMW4 for repairs resulted in a significant shift in how traffic reaches Bangladesh, as shown in the Traffic Shifts graph below for AS17494 (Bangladesh Telecommunications Company Limited). The biggest impact appeared to occur during the first few days of the repair period, stabilizing by July 27.
On the heels of exam-related Internet shutdowns on June 21 and 27 (covered in last month’s post), similar disruptions were observed in Iraq on July 1, 4, 7, and 11 as seen in the figures below. A table published by media advocacy and development organization SMEX listed high school diploma exams as taking place between June 21 and July 12, which aligns with the shutdowns discussed here. In addition, the issues observed in June and July also fit the profile of similar past actions – a significant, but not complete, outage lasting two to three hours.
As seen in the figure below, similar Internet disruptions in Iraq were also observed in the Internet Intelligence Map on July 17 and 19. While they appear to be similar in profile to the exam-related outages discussed above, there was no available information that could be found regarding exams taking place on these two days.
Heading into the end of July, Syria closed out the month with three multi-hour outages where the Internet was shut down nationwide to prevent cheating on high school exams. As seen in the figure below, the number of completed traceroutes into Syrian endpoints dropped to near zero, and the number of routed networks in Syria also dropped to near zero. However, as we have seen with similar prior shutdowns in Syria, the number of DNS requests from resolvers within the country jumps sharply during the shutdown. We believe that this indicates that the shutdown was implemented asymmetrically – that is, traffic from within Syria can reach the global Internet, but traffic from outside the country can’t get in. These spikes in DNS traffic are likely related to local DNS resolvers retrying when they don’t receive the response from Oracle Dyn authoritative nameservers – normally, the client traffic they are making requests on behalf of would be served from the resolver's cache.
Sandwiched between the exam-related outages referenced above, Iraq experienced a nationwide Internet blackout that lasted nearly two days, stemming from a shutdown ordered in response to a week of widespread protests. The disruption, shown in the figure below, lasted from July 14-16, and had a significant impact on all three measured metrics.
Unfortunately, as noted in a blog post on the disruption, “Government-directed Internet outages have become a part of regular life in Iraq.” The first such outage documented by the Internet Intelligence team occurred in 2013 and revolved around a pricing dispute between the Iraqi Ministry of Communications and various telecommunications companies operating there. Over the subsequent five years, we have seen several more such Internet disruptions.
The Internet Intelligence blog post referenced above highlighted that not all of Iraq was taken offline during the weekend disruption, with about 400 BGP routes (out of a total of 1,300 for Iraq) staying online. Some telecommunications providers with independent Internet connections through the north of Iraq stayed online, as did those with independent satellite links.
ITC operates the Iraqi fiber backbone, and the impact of the government-directed disruption is clearly evident in the Traffic Shifts graph above over the July 14-15 weekend period. Iraqi provider Earthlink is based in Baghdad and is one of Iraq’s largest ISPs. It was also down during the same period, as seen in the Traffic Shifts graph below.
On July 9, Twitter user @Abdalla_Salmi posted the following Tweet:
It appears that there has been severe internet disruptions in #Eritrea, over the past two days, coinciding with the visit by #Ethiopia PM Abiy Ahmed (Source: @InternetIntel) pic.twitter.com/Ii37j3zQro— Abdalla S (@Abdalla_Salmi) July 9, 2018
The Country Statistics graphs below show that there was no change in the number of routed networks geolocated to Eritrea, but there were significant declines in the traceroute completion ratio and DNS query rate metrics during the time period highlighted in the above Tweet. As @Abdalla_Salmi noted, the Internet disruption in Eritrea was coincident with a visit from the Ethiopian Prime Minister. (The visit marked a shift in relations between Ethiopia and Eritrea, which have been locked in two decades of conflict.) In some instances in the past, we have observed state-ordered disruptions to a country’s Internet connectivity as a means of limiting their citizens from being able to organize protests around political events of this type. However, such government involvement in an Internet shutdown is often reported in the press and/or on social media; in this case, no such reports have been found.
The Eritrea Telecommunication Service Corporation (EriTel) is the national telecommunications service provider, and is the state-owned monopoly for fixed and mobile connectivity. As the graph below shows, the number of completed traceroutes into EriTel dropped to approximately 10% of their previous rate during the period of disruption. While no publicly available information on a root cause has been found for the issues observed at a country level and with EriTel, the disruptions were corroborated by colleagues at Akamai and CAIDA through data they collect and analyze.
On July 12/13 and again on July 17/18, the Internet Intelligence Map highlighted Internet disruptions in Bhutan, as shown in the figure below. Although the observed issues appeared to last less than a day in each case, they left artifacts across all three metrics. Unfortunately, the root cause of these disruptions is unknown, as there were no published reports found on state involvement, fiber cuts, power outages, or the like.
Just after midnight GMT on July 23, the Internet Intelligence Map Country Statistics view for Syria showed an approximately 30% decline in the traceroute completion ratio metric, as seen in the graph below. This reduced ratio persisted through the end of the month and may represent the “new normal”, although the reduced rate of DNS queries from Syrian resolvers returned to previous levels after a few days; the number of routed networks from Syria remained unchanged. This type of profile is often indicative of last mile access issues or catastrophic technical failure closer to the edge of the network. However, in this case we believe that this observed disruption may have been due to a change in network configuration at Syrian Telecom.
The impact of this possible network configuration change on traceroutes into AS29256 (Syrian Telecom) can be seen in the Traffic Shifts graph below. In this case, the number of completed traceroutes into Syrian Telecom appears to drop right before midnight GMT on July 23 – just ahead of the significant drop in the country-level traceroute completion ratio graph above.
July was a busy month for Internet disruptions around the world as observed within Oracle’s Internet Intelligence Map. For better or worse, the disruptions were largely due to familiar causes, with related information found in local or international press coverage, on Twitter, or on telecommunications provider Web sites. However, some had impacts large enough to leave artifacts in the Internet Intelligence graphs, but without correlated press coverage, provider apologies, or user complaints on Twitter. Although root cause information can be hard to find, we feel that it is valuable to highlight all significant Internet disruptions in support of #keepiton efforts around the world.