After looking at the countries most impacted by the cable cut in our first blog on this topic, we now turn our attention to the Internet service providers in the region and how they fared. Due to differences in network architecture, cable ownership, and transit purchasing, carriers in the same country may not all experience the same degree of outage. For all of the following, we consider a network to be "outaged" when it is unreachable from the perspective of the broader Internet—as represented by Renesys's 250 peering sessions.
The following two tables provide the top 15 providers with the largest number of outaged networks. We list the provider's name, the country in which most of their unreachable networks are located and their autonomous system number (ASN), an assigned number that uniquely identifies their organization on the Internet.
In the first table, we list the providers in decreasing order by total number of outaged networks. In the second table, we list them by decreasing order of the percentage of their networks that are unreachable.Not surprisingly, the hardest hit providers are located primarily in the hardest hit countries: Egypt, Kuwait, India and Pakistan. One local provider in each of Egypt and Kuwait lost essentially all of their Internet connectivity.
| Provider | Country | ASN | Num |
|---|---|---|---|
| LINKdotNET | Egypt | 24863 | 268 |
| Pakistan Telecom | Pakistan | 17557 | 234 |
| Wataniya Telecom | Kuwait | 29357 | 199 |
| TEDATA | Egypt | 8452 | 191 |
| Sify | India | 9583 | 184 |
| EgyNet | Egypt | 20858 | 169 |
| Dancom | Pakistan | 23966 | 147 |
| Micronet | Pakistan | 23674 | 125 |
| Hathway | India | 17488 | 111 |
| Internet Egypt | Egypt | 5536 | 95 |
| QualityNet | Kuwait | 9155 | 90 |
| Nile Online | Egypt | 15475 | 90 |
| Data Network | Pakistan | 9260 | 87 |
| IDM | Lebanon | 9051 | 80 |
| Provider | Country | ASN | % |
|---|---|---|---|
| Wataniya Telecom | Kuwait | 29357 | 100 |
| Yalla Online | Egypt | 20484 | 99 |
| Dancom | Pakistan | 23966 | 80 |
| EgyNet | Egypt | 20858 | 77 |
| Internet Egypt | Egypt | 5536 | 77 |
| Micronet | Pakistan | 23674 | 73 |
| Data Network | Pakistan | 9260 | 71 |
| Pakistan Telecom | Pakistan | 17557 | 64 |
| TEDATA | Egypt | 8452 | 60 |
| IDM | Lebanon | 9051 | 56 |
| Exatt | India | 18231 | 53 |
| Nile Online | Egypt | 15475 | 49 |
| Emirates Internet | UAE | 5384 | 46 |
| Eepad | Algeria | 33783 | 43 |
| QualityNet | Kuwait | 9155 | 35 |
After totaling up the damage to the local providers, we wondered if any of harder hit ones managed to regain connectivity for some of their networks via alternate paths. We often hear statements like "the Internet is good at routing around damage". Well, that can be true, but only when there are available alternatives. Looking at hard-hit Egypt and Kuwait, we plotted the number of outaged networks per provider over the past day in the following stacked graph, where the width of each color represents the number of unreachable networks for a given provider. If any of the providers had choices, we would expect to see the width of their color decrease over time as they shifted traffic to alternative paths. Except in one case, that didn't happen. The exception was the Egyptian provider, LINKdotNET (ASN 24863), which did regain connectivity for most of their unavailable networks for about one hour at 20:00 UTC, only to lose more than twice as many after that time. Whatever backup routes they used obviously didn't hold up. The others had essentially the same number of networks out all day, the typical shape of a prolonged catastrophic failure.
To make this last point perhaps more forcefully, we next plot the total number of outaged networks for the region as a whole for 24 hours, excluding the Indian subcontinent. As shown, there was no immediate relief for the large swath of the Internet cut off by this disaster.
Next up, we'll look at the global Internet providers and who won and lost in the battle to retain their regional customers or acquire new ones.
A number of providers could have had protection on SMW3, although given how unreliable that has been in the recent past maybe they gave up on it, which would be pretty ironic.
Buying bw on cables with protection on another seems to be a thing of the past, if the number of links on alternate paths (either using smw3 or going eastwards on smw4) that we are turning up "on the rush" means anything. Maybe a prearranged protected path is just plain too expensive an people prefer to take the risk.
Aside from the Internet outages caused by these, now three!!, cable cuts, I wonder about the private/VPN networks outaged by these. I know that one global provider is both a consortium member on SEA-ME-WE-4 and a large customer for their 'Private IP" network on SEA-ME-WE-4. This has to hurt their private services even more as they are often less meshy and have less redundancy route-wise (physical and logical route) than the Internet.
Thanks for the analysis so far!
VSNL (AS6453) restored Internet Services within 24 hours following the dual cable breaks. Our network and operations teams across three continents united to execute an ambitious recovery plan. Our Internet backbone around the world combined with our cable assets in Middle East (SMW4 Eastward), India (national), Asia (TIC) and Pacific (TGN-P) allowed us to survive this double cable failure by deploying enough capacity from the Middle East eastward to reach the Internet in North America and Europe.
Obviously going around the world via this route increases the latency to Europe and North America but all customer packets are routed.
VSNL - having over one second delay and more than 10% packet loss between LHX - London and JSD - Jeddah, Saudi Arabia does not mean that "all customer packets are routed".
Over 11% packet loss TCP connections get very slow, if they can be initiated.
Does anyone know at what depth these two cables lie?