In November, we saw fewer significant Internet disruptions in the Oracle Internet Intelligence Map as compared to prior months. As usual, there were hundreds of brief issues with limited impact and generally unknown causes, but the most notable issues last month were due to reported DDoS attacks, problems with terrestrial and submarine cables, and general network issues.
On November 4 and 5, several Cambodian ISPs were targeted by DDoS attacks described as the “biggest attacks in Cambodian history.” Published reports indicated that ISPs including EZECOM, SINET, Telcotech, and Digi were targeted by DDoS attacks totaling nearly 150 Gbps, causing subscriber downtime lasting as much as half a day. Disruption from the attacks was visible in the Country Statistics view for Cambodia in the Internet Intelligence Map, as shown in the figure below. However, because Internet connectivity remained generally available (albeit impaired) across the country, the impact appears nominal in the graphs.
However, when viewed at a network level, the impact of the attacks appears to be more significant. SINET, one of the ISPs targeted by the DDoS attacks, posted a Tweet on November 5 letting users know that they were under attack, and followed up the next day with a longer explanation of what occurred, and the mitigation steps that were taken.
We are experiencing network attacks from external sources. Our engineers are hard at work to restore the service as soon as we humanly can. We sincerely apologize for inconveniences. Thanks for your patience and understandings.— SINET ISP (@sinetKH) November 5, 2018
Official Statement on DDOS Incident on November 5, 2018. pic.twitter.com/6bvjSoNltp— SINET ISP (@sinetKH) November 6, 2018
The Traffic Shifts figure below for SINET shows the DDoS attacks beginning to impact the network late in the day (GMT) on November 4, resulting in a significant decline in the number of completed traceroutes to endpoints in the network, along with a corresponding increase in latency. The problems lasted for approximately half a day before the metrics stabilized, though with fewer completed traceroutes at slightly higher combined latency.
On November 13, a brief disruption in Internet connectivity in the U.S. Virgin Islands was visible in the Oracle Internet Intelligence Map, as shown in the figure below. Subsequent research revealed that the observed issue was likely related to a fiber cut experienced by local ISP Viya. According to updates posted on the company’s website and Facebook page, “Viya’s fiber on St. Croix was accidentally cut by the Virgin Islands Water and Power Authority. As a result of the damage to the network, customers throughout St. Croix do not have internet, long distance and Cable TV service.” The impact of this fiber cut on Viya’s network can be seen in the Traffic Shifts figure below, with a clear drop in completed traceroutes to endpoints in the network and an associated increase in combined latency.
In April 2018, we saw damage to the ACE (Africa Coast to Europe) Submarine Cable impact Internet connectivity for ten connected countries. On November 18, another issue with the cable once again disrupted connectivity across a number of countries connected to it. A Sierra Leone network provider reported that the issue originated at the landing station in Lisbon, Portugal.
Sierra Leone goes without internet connectivity for about 7 hours today. The Sierra Leone Cable Limited (SALCAB) says they are "experiencing a downtime on the Africa Coast to Europe submarine cable due to technical failure at our landing partner station in Lisbon."— TheReporterSL (@TheReporterSL) November 18, 2018
The figure below shows a matrix of eight countries connected to the ACE cable for which the Country Statistics views exhibited impacts from the reported technical failure. (The graphs are organized by distance from the Lisbon landing station where the problem reportedly occurred, with the closest appearing first.)
It is interesting to note that there is a range of impacts seen in the measurements, and that the observed disruptions don’t strictly align with (cable) distance from Lisbon or the number of submarine cable connections a given country has. The first six countries shown count the ACE cable as their sole submarine cable provider, while Benin connects to two cables, and Equatorial Guinea three, according to Telegeography’s Submarine Cable Map.
It is also interesting to observe how the ACE cable issue impacted individual network providers in the affected countries. While multiple examples were captured, there was a noticeable difference seen between two network providers in The Gambia. For AS25250 (Gamtel), the cable disruption caused a clear decline in the number of completed traceroutes to endpoints in the network, while combined latency spiked to approximately 3x normal levels. However, for AS37309 (QCell Limited) during the disruption period, it appears that traceroutes took alternate paths through Telecom Italia Sparkle and Orange when the path through Portugal Telecom became unavailable, with no significant impact to latency. While both networks have multiple upstream providers/peers, only one appeared to benefit from the redundancy.
In addition to the disruption discussed above, planned maintenance on the ACE cable was scheduled to take place between November 23-28, according to a Tweet from @Gamtel. The Traffic Shifts graphs in the figure below for AS25250 (Gamtel) show increases in combined latency on multiple days during the maintenance period, as well as an increase in the number of traceroutes reaching Gamtel through Sonatel when the path through Portugal Telecom became unavailable. In this case, the redundant connections appeared to minimize the disruption to Gamtel’s Internet connectivity.
Early in the morning (GMT) of November 2, the Cogent Communications Status Page reported an issue on their network due to a double fiber cut, impacting regions including México City, Querétaro, and Guadalajara. While this outage was barely evident in the metrics shown for Mexico in the Country Statistics view of the Internet Intelligence Map, it was much more obvious in the Traffic Shifts graphs for Mexican network providers that have upstream connectivity from Cogent, as shown in the figures below. For AS14178 (MCM Telecom), traceroutes to endpoints within the network arrived via Level 3 (and PCCW Global to a lesser extent) during the Cogent outage, while at AS6503 (Axtel), Level 3 also picked up the slack, although the drop in traceroutes through Cogent was not as severe as was seen at MCM Telecom. The Cogent outage also resulted in nominal increases in latency for traceroutes to both networks.
Late in November, Internet disruptions were observed in both Burkina Faso and Somalia, as shown in the Country Statistics graphs in the figures below. While neither was a complete disruption in connectivity, further research showed a common point of failure.
In reviewing Traffic Shifts graphs for network providers in the two countries, we observed that the West Indian Ocean Cable Company (AS37662) [WIOCC] was the common chokepoint – that is, the issues seen were in networks downstream of WIOCC. As shown in the figures below, a disruption is clearly visible in AS37073 (Sahel Finance Burkina) on November 25, which is downstream of AS328316 (Point d'Atterissement Virtuel - Burkina Faso). The latter network relies on connectivity from WIOCC, and a coincident disruption is visible here as well. However, no disruption is evident when looking at the Traffic Shifts graph for WIOCC, indicating that the issue likely occurred in the connectivity between AS37662 and AS328316 and not upstream from WIOCC, although it isn’t clear if the root cause is related to some sort of configuration issue, or if it was due to physical infrastructure failure.
Similarly, the majority of traceroutes to Somali networks AS37371 (Hormuud Telecom Somalia) and AS327828 (Somali Optical Networks) (among others) also pass through WIOCC. As noted above, no disruptions were observed upstream of WIOCC, indicating that the issue likely occurred in the connectivity between it and these downstream networks.
Internet connectivity relies on a multi-tiered architecture: the control plane, the data plane, and the physical plane. We passively observe the control plane (collecting BGP updates), and actively probe/measure the data plane (through traceroutes and DNS usage data), but the physical plane (cables, satellites, power grid, etc.) remains largely invisible as it cannot be probed from afar. For those Internet disruptions that are not explained by “ground truth” (such as a Tweet, Facebook post, published article, status page update, etc.), we are forced to make educated inferences as to their cause, based on historical experience, factors including local geopolitical or weather activity, and other disruptions seen nearby and/or at the same time.