In January, we reported the news that the ALBA-1 submarine cable connecting Cuba to Venezuela had started carrying Internet traffic two years after its construction, answering the question of what happened to the mystery cable to Cuba. In the last week, we have observed a second non-satellite connection established for Cuban state telecom, ETECSA. This time a different segment of the ALBA-1 submarine cable is being used to connect Cuba to the neighboring island nation of Jamaica. At 15:04 UTC on 13 May 2013, we observed ETECSA beginning to receive international Internet service through Cable & Wireless Jamaica.

The graphic on the right shows the make up of Internet transit providers used by ETECSA to reach the global Internet since 1 January 2013. Telefonica showed up on 10 January when ALBA-1 began carrying Internet traffic. C&W Jamaica appears in yellow on 13 May. There is a brief dip in the Telefonica plot as ALBA-1 was offline for about eight hours on 15 March and for two hours on 17 March, something we tweeted. This past March, C&W Caribbean listed Cuba as one of the markets they planned to enter and fulfill the relationship signed three years prior when C&W Jamaica signed a partnership with the company established to oversee the ALBA-1 cable project, Cuban-Venezuelan joint venture Telecomunicaciones Gran Caribe (TGC).

Two weeks ago, I had the opportunity to present at LACNIC 19 in Medellin, Colombia. While there, I was able to speak to a manager from ETECSA. He said he and his staff had seen our blogs from January and confirmed that we were right about the initial asymmetric traffic misconfiguration of ALBA-1, which they fixed after a couple of days. It was pleasure to meet some of the people involved with this historic activation.

When we look at our recent traceroute measurement data into Cuba, we can see that the 13 May activation of ETECSA's link to Jamaica followed some connectivity problems on 10 May, as there were brief decreases in the rate of completing measurements on that day, regardless of upstream provider. This was followed by a reduction in connectivity through Telefonica (purple) and then the C&W Jamaica (green) activation.

On the Telecom Venezuela website describing the ALBA-1 cable project, it mentions that the link to Jamaica is for restoration purposes ("El otro segmento para fines de restauración será entre Cuba y Jamaica"). So perhaps this activation is to help alleviate some minor connectivity issues experienced recently by ETECSA. Regardless, it is great news to see another submarine cable connection get activated for Cuba.

Update (15:26 UTC, 15 May): Routes to Syrian networks have been restored, at 18:26 Damascus time. Outage duration: 8h25m

 

 

 

 

Update (14:20 UTC, 15 May): Plot of latency measurements to Syrian hosts from various locations, indicating that replies stopped returning shortly after 7am UTC, aligned with the withdrawal of routes to Syrian networks. (Click image for details)

Update (07:30 UTC, 15 May): Syrian Internet down again since 07:01 UTC (10:00 Damascus time), Wednesday, 15 May 2013. Syrian news agency reports that they're working to fix. Potentially related to forthcoming UN decision today?

 

 

 

 

 

---

Older Update: Syrian Internet has returned. Outage lasted 19.5 hours, from 18:45 UTC May 7th to 14:13 UTC May 8th.

As we write, the Syrian people are still disconnected from the global Internet at the most fundamental level, nearly all of their paths withdrawn from the global routing table. Since 18:45 UTC on May 7th, Renesys hasn't seen a flicker of activity. We haven't been able to successfully send a ping or a traceroute to any host inside Syria. Government websites, universities, domain name servers, core infrastructure routers, banks, businesses, DSL customers, smartphones: all silent.

As I look back at what we've written about Internet outage over the years, I see a sort of evolution in our perspective. We've covered Internet failures due to war, politics, censorship, central planning, earthquakes, hurricanes, cable cuts, business disputes, terrorism, undersea mud volcanoes, and (perhaps) cyberwarfare.

In the early days, we reported each outage breathlessly, shocked that the Internet could fail in such spectacular ways. If you look around the web this morning, you'll see a lot of that same shock-and-awe reporting from companies who are just discovering the fragilities visible in Internet data.

In this case, however, what strikes me is the depressing sameness of the sequence of Syrian Internet disconnections. Just as in June 2011, July 2012, August 2012, and November 2012, the entire nation disappeared from the Internet in 30 seconds, as if a switch had been thrown. Everyone in the Twittersphere seems to share the same strange lack of perspective about these events — in the middle of the chaos and tragedy of civil war, why is anyone surprised when the Internet stops working? Isn't it actually more shocking and noteworthy that the Internet in Syria actually functions pretty well 360 days out of the year?

If you're looking for a different way to think about events like these, you might enjoy reading Nassim Nicholas Taleb's book, Antifragile (ironically, published just days before the last major Syrian Internet disconnection).

Antifragility is a property of systems that get stronger when they are challenged by stresses. It's not merely robustness, or resilience (systems that return to stable state after stresses have passed) — an antifragile system like the Internet actually gets better when you try to break it. Systems of systems attain antifragility because their individual elements are so fragile. For example, Taleb writes:

"Restaurants are fragile; they compete with each other, but the collective of local restaurants is antifragile for that very reason. Had restaurants been individually robust, hence immortal, the overall business would be either stagnant or weak, and would deliver nothing better than cafeteria food — and I mean Soviet-style cafeteria food. Further, it would be marred with systemic shortages, with, once in a while, a complete crisis and government bailout. All that quality, stability, and reliability are owed to the fragility of the restaurant itself."

If you've been reading all of our Internet outage case study blogs, you might see a familiar pattern. The Internet fails in some way, taking individual institutions (or even entire countries or geographic regions) offline. The outage exposes fundamental undetected weaknesses — lack of submarine or terrestrial cable diversity, lack of Internet provider diversity, underinvestment in alternative forms of transport, political control over Internet chokepoints. If the Internet damage is painful enough, engineers (or revolutionaries) are forced to take creative steps to address those weaknesses. Repeat... forever.

Take Pakistan, for example. This cycle of insult-and-repair is what made Pakistan Telecom serious about buying eastbound backup transit to Asia in the wake of the first round of Mediterranean cable cuts in 2008. It's what led (indirectly or directly) to the establishment of competing providers in Pakistan, with foreign investment and additional cable resources connecting Pakistan to the Gulf States.

And because Internet is easily exportable, Pakistan's improved Internet connectivity spilled over to improve the situation in their historically fragile neighbor, Afghanistan. Here's a picture of one slice of Afghanistan's diverse connectivity over the last few weeks (click to zoom):

Note three paths from Afghan providers to Cable and Wireless in Europe. Borrowing from Pakistan's diversity, one route goes through Pakistan's competitive provider, Transworld Associates, then to Omantel and over the EPEG cable through Iran and the Russian Caucasus to Europe. Another goes by satellite to Bahrain's Batelco, then over to Omantel and out through Iran on the same EPEG route. A third goes north through Uzbekistan's Intal Telecom, then through two Russian routes to Europe. The Iranian EPEG route is itself an improbably antifragile innovation, inspired by multiple years of Mediterranean cable cuts and avoidance of the demonstrably fragile Egyptian transit corridor.

So what does all this mean for Syria and its Internet neighborhood? One has to look past short-term dysfunction and think about what comes next. Every significant Internet disconnection, and the local and global reaction of outrage and dismay, sends an important signal about the fragility of the underlying system. It makes single points of failure and control visible, so that those fragilities can be found and fixed, and the Internet as a whole can continue to gain strength from disorder.

Sometimes, it takes a real disaster to create something genuinely new. March 2013 was a month of disasters in the Middle Eastern, South Asian, and East African Internet, with major submarine cable cuts affecting SMW3, SMW4, IMEWE, EIG, SEACOM, and TE-North.

One of the "genuinely new" Internet traffic paths that emerged in response is a counterintuitive terrestrial route, linking the ancient Indian Ocean trade empire of Oman with the Internet markets of Western Europe, by way of Iran, Azerbaijan, and the Russian Caucasus. As we'll see, its effects are now being felt across the region, from Pakistan, to Gulf states like Bahrain and Oman, to Kenya.

The EPEG (Europe-Persia Express Gateway) consortium was actually born in June 2011, as an alternative to the congested, politically uncertain Suez transit corridor. EPEG links together existing fiber routes from the Iranian, Azeri, and Russian incumbents, connected to Cable and Wireless's network to approximate a Great Circle route to Frankfurt. With the aid of one final submarine hop across the Strait of Hormuz to Muscat, EPEG promised to deliver a major new low-latency, high-capacity terrestrial route to carry the Gulf states' traffic to Europe.

EPEG was scheduled to begin service in the summer of 2012, but delays on the final submarine links between Iran and Oman caused the start date to be pushed, and pushed again. When Vodafone acquired Cable and Wireless in July 2012, one of the attractions of the deal may have been the prospect of participating in EPEG — when the project reached completion, Vodafone/C&W would sit astride the first low-latency terrestrial link connecting the Gulf region to the financial and content markets of Western Europe.

We reported in February 2013 that in the wake of Russian service interruptions in the Black Sea, Iran's DCI had begun receiving Internet transit from Omantel, indicating that the missing link in the southern path was alive at last. We began watching intently for signs of end-to-end service activation on EPEG.

Last month, we saw them.

EPEG in Action in Oman

To some within the GCC, EPEG may seem geopolitically implausible, but as a lower-latency alternative to Egyptian transit, it makes a lot of sense.

Here are some screenshots from our new Internet Business Intelligence suite, due out later this year, showing the emergence of Cable and Wireless transit as the fastest transit path into Oman (click on image for higher resolution). The new transit path from Omantel to Cable and Wireless emerges on March 31st (in green), some 15 milliseconds faster than existing transit paths from Frankfurt to Oman. Note the impact of the cable cuts before March 31st. By comparison, latency variance on the new path is tight, suggesting that the Iranian path has ample capacity and is still lightly loaded.

Bahrain taking advantage of EPEG

Batelco, Bahrain's incumbent provider, wasted no time in taking advantage of the new, lower-latency path to Europe for selected customers.

As this screenshot illustrates, we estimate that transit through Omantel and on to Cable and Wireless through Iran reduces the round trip time for Bahraini traffic to Frankfurt by as much as 30ms. Despite the ongoing political tensions between Bahrain and Iran, that's a pretty compelling argument.

Pakistan's Transworld Transiting EPEG

Pakistani competitive provider Transworld (AS38193) partners with Omantel to operate the TWA-1 submarine cable, so it wasn't a big surprise to see Omantel appear at the end of March as a remedy for the transit confusion caused by the SMW4 cut off Alexandria, Egypt. In short order, a significant portion of all the traffic to Transworld's 2,000+ routed prefixes has begun to traverse the EPEG link through Oman and Iran.

Faster Transit for East Africa

We even see EPEG's benefits reaching into East Africa, where AccessKenya (AS15808) picked up Omantel transit, creating a path that's more than 20 milliseconds faster than the previous alternatives to Frankfurt.

As we've seen before, East African providers are very quick to adapt to changes in the transit environment, and Omantel may do very well if other providers in Kenya, Uganda, and Tanzania get on board.

A lot will depend on the price — Omantel is probably able to command a significant premium for the uniqueness and low latency of the EPEG route. When competing cables return to service, we'll see how that pricing affects the new market equilibrium.

Conclusion

If you'd told me five years ago that we would one day see Iranian and Russian terrestrial Internet transit serving the countries of the Indian Ocean, from Pakistan to East Africa, I wouldn't have believed it.

Today, I'm a believer. Middle Eastern and East African network operators are using the Renesys Internet Intelligence tools to sniff out new transit opportunities in the market, and finding creative ways to recover from almost unimaginable connectivity challenges.

We've seen all kinds of strategies emerge in the wake of the SMW4 cuts, from Telecom Egypt transit (as far away as Afghanistan), to Israeli transit, to Iraqi transit, to Syrian transit through a war zone. In the hands of Omantel, EPEG is the first solution that appears to be delivering globally visible terrestrial routes to a diverse set of regional providers.

It remains to be seen whether an Internet path through Iran and the Russian Caucasus region will have the kind of stability that enterprises require. But frankly, compared to the submarine cable competition, they're already looking pretty good.

It has been a rough few weeks for the global Internet, given numerous submarine cable failures and the largest DDOS attack ever reported. While we're hard-pressed to find evidence of the purported global Internet slowdown due to the DDOS attack, the dramatic impacts of yesterday's SMW4 submarine cable cut were profound. Recent reports that the cable break was the result of sabotage make the incident even more intriguing. In this blog, we detail what happened to some of the providers in four countries along the route of the cable: Egypt, Saudi Arabia, Pakistan and India.