Renesys Blog

'Crecimiento' in Latin America

2013-05-23T20:17:09Z

Two weeks ago, I had the great pleasure of visiting Medellin, Colombia to present at the Latin American and Caribbean Network Operators Group (LACNOG) portion of LACNIC 19. Medellin is a vibrant place, recently recognized as the world's most innovative city by the Urban Land Institute due to the city's investments in public infrastructure and civic spaces.

Perhaps equally innovative is Colombia's Internexa, which in recent years has been building the region's first international terrestrial telecommunications network. Meanwhile, another remarkable regional story is the exponential growth of the domestic Internet in Brazil — especially when contrasted with the stagnation in Mexico. While government initiatives in Brazil, the region's largest economy, were able to foster much of its recent growth, the current regulatory overhaul in Mexico hopes to achieve something similar in the region's second largest economy.

Internet Growth in Latin America

In the presentation, I discussed where we did and did not see Internet growth (crecimiento in Spanish) in the Latin American region. To measure growth, I borrowed a metric, domestic ASN growth, from my colleague Jim Cowie's recent presentation on the growth of the Internet of the Middle East at MENOG 12 in Dubai.

For each country, we counted up the number of domestic Autonomous Systems (ASNs), where an ASN is considered domestic in a country when at least 70% of its customer base is in that country. Of course, ASNs can vary dramatically in size, from large companies to small non-profits. But regardless of size, our metric accurately captures the "number of Internet players" in a country. It is the relative change in this count over time that is most informative, as shown in the following table for a subset of Latin American countries:

Our data illustrates very robust growth in Brazil, Argentina and Costa Rica since January 2010. During this period, Brazil grew by 340% and now exceeds the size of the rest of the region combined. By this metric, Brazil adds the equivalent of two Mexicos to its domestic Internet each year.

Costa Rica

Dramatic growth of the Internet in Costa Rica is a result of the telecommunications regulatory reform of 2008. The reform was a requirement of the CAFTA-DR trade agreement, which mandated that the government of Costa Rica end the monopoly of ICE, set up a telecoms regulator Sutel, and allow new competition. Although Costa Rica was one of the smaller countries in our analysis, it had the largest relative growth in the region. New players in Costa Rica include new telecommunications providers as well as new enterprises operating in the country.

Brazil

Brazil's tremendous growth has been a result of a national program of infrastructure investment and government incentives, which was described at LACNIC 19 by Artur Coimbra of the Brazilian Ministry of Communications. In addition to upgrading and expanding the domestic backbone of Telebras, Brazil has established 22 municipal Internet exchange points called PTTs (Pontos de Troca de Tráfego) to facilitate domestic exchange of Internet traffic. In his presentation, Wardner Maia, President of the Association of Brazilian Internet Providers (ABRINT) discussed the recent explosion of small regional Internet service providers serving parts of Brazil that hadn't previously had access to the Internet. On average, he said, the Brazilian government was issuing over 700 licenses per year to these new providers, 78% percent of which are considered small companies. When we compared our data, it was clear that the growth of domestic ASNs matched the growth in licensing that his organization was witnessing.

We're not the only ones witnessing Brazil's growth. In their most recent State of the Internet report, content provider Akamai noted the largest year-to-year increase in unique IP addresses reaching their servers was from Brazil (a 33% increase).

Argentina

Like Brazil, Argentina's growth was due to infrastructure development. A relatively competitive market like Brazil, Argentina also had a program of developing regional IXPs to foster domestic connectivity. Argentina's CABASE program has established 10 municipal IXPs and is facilitating the growth we're seeing in that country.

This growth has vaulted Argentina past Mexico in the number of players in the domestic Internet. Argentina has the third largest economy in Latin America, behind Mexico and Brazil.

Mexico

On the other end of the spectrum is Mexico, where we have observed negligible growth in its domestic Internet.

One of the least competitive markets in Latin America, Mexico is dominated by its incumbent Telmex. Last year, the OECD concluded that, "inefficient telecommunication markets impose a significant cost on the Mexican economy and the welfare of its population" to the tune of US$129Bn per year or 1.8% of the GDP of Mexico.

The lack of competition is such a major problem that newly elected Mexican President Enrique Peña Nieto has made telecom reform one of the key features of his Pacto por México. Among other measures, the legislation would cap market share at 50% and remove the limit on foreign ownership of telecommunications companies. Once instituted, the market share cap may require the break-up of Telmex and pit Peña Nieto against Telmex's owner and CEO, billionaire Carlos Slim, the richest man in the world.

As of this writing, the Mexican telecom reform has passed both houses of congress and has been ratified by a majority of Mexican states. America Movil, the parent company of Telmex, has already warned shareholders that the new legislation could have a material impact on its Mexican business.

In separate development, several mid-sized Mexican telecommunications companies have united to establish the first domestic Internet Exchange Point (IXP) in Mexico, the only country in the OECD not to currently have one. However, without Telmex on board, the impact on Mexican domestic connectivity as a whole may be limited.

The regulatory reform in Mexico won't take effect until 2014. If Costa Rica can serve as a model of what to expect in Mexico, it may take a couple of years until we observe a meaningful increase in growth. However, given the size and richness of the Mexican economy, perhaps there is reason to believe that a long-awaited expansion could occur at a much faster rate once facilitated by this reform.

Regional Terrestrial Connectivity

In many parts of the world, the promise of regional terrestrial connectivity that directly connects neighboring countries has proven elusive. The Program for International Development in Africa (PIDA) has been pushing for this in Africa, while LIRNEasia has been advocating for the same thing for South and Southeast Asia. The GCCIX project has begun to address this in the Middle East. Without terrestrial connectivity, countries in these regions are reliant on submarine cables to carry their traffic to neighboring countries. This implies that Internet traffic must often traverse long distances to reach geographically close locations, resulting in longer latencies and poorer performance. In addition, submarine cables occasionally suffer faults and undersea repairs take significantly more time than those on land.

Until recently, South America could also be defined much in the same way, but that is changing. Miami is still the common international connection point for Internet providers in Latin America (Los Angeles and Dallas for Mexico). However, Colombian company Internexa is leveraging the international network of power lines owned by its parent company ISA to build the region's first multi-country terrestrial network (pictured at right). At LACNIC 19, Internexa presented statistics about how much regional Internet traffic they are currently carrying that no longer needs to leave the continent in order to reach its destination. They currently have direct terrestrial connections between Brazil, Argentina and Chile as well as between Venezuela, Colombia, Ecuador and Peru. Ultimately, their goal is to build a terrestrial network spanning all the countries of South America to provide the fastest and most direct service in the region.

In my talk, I used the screenshots below from our upcoming product, Internet Business Intelligence, to illustrate which Internet provider combinations provided the lowest latency paths from the city hosting the conference (Medellin, Colombia) to nearby Quito, Ecuador. By eliminating the need to send Internet traffic out of the region, Internexa was the fastest option to Ecuador. When looking at performance over time in the graphic below and to the right, we can see that the Ecuadorian government-owned provider Corporacion Nacional de Telecomunicaciones (CNT) experienced some brief performance issues, while Internexa's subsidiary in Ecuador, Transnexa, provided the most stable low-latency path between these two cities during this time period.

Looking to the Future

Developments such as Internexa's terrestrial network and Brazil's amazing Internet boom have been fascinating to observe. However, the most important story for the region in the coming years may be whether or not Peña Nieto is successful at breaking up Carlos Slim's Telmex and its grip on the Mexican telecom market. Solo el tiempo lo dirá — only time will tell.

Cuba-Jamaica Link Activated

2013-05-20T21:33:38Z

In January, we reported the news that the ALBA-1 submarine cable connecting Cuba to Venezuela had started carrying Internet traffic two years after its construction, answering the question of what happened to the mystery cable to Cuba.

In the last week, we have observed a second non-satellite connection established for Cuban state telecom, ETECSA. This time a different segment of the ALBA-1 submarine cable is being used to connect Cuba to the neighboring island nation of Jamaica. At 15:04 UTC on 13 May 2013, we observed ETECSA beginning to receive international Internet service through Cable & Wireless Jamaica.

]]>

The graphic on the right shows the make up of Internet transit providers used by ETECSA to reach the global Internet since 1 January 2013. Telefonica showed up on 10 January when ALBA-1 began carrying Internet traffic. C&W Jamaica appears in yellow on 13 May. There is a brief dip in the Telefonica plot as ALBA-1 was offline for about eight hours on 15 March and for two hours on 17 March, something we tweeted.

This past March, C&W Caribbean listed Cuba as one of the markets they planned to enter and fulfill the relationship signed three years prior when C&W Jamaica signed a partnership with the company established to oversee the ALBA-1 cable project, Cuban-Venezuelan joint venture Telecomunicaciones Gran Caribe (TGC).

Two weeks ago, I had the opportunity to present at LACNIC 19 in Medellin, Colombia. While there, I was able to speak to a manager from ETECSA. He said he and his staff had seen our blogs from January and confirmed that we were right about the initial asymmetric traffic misconfiguration of ALBA-1, which they fixed after a couple of days. It was pleasure to meet some of the people involved with this historic activation.

When we look at our recent traceroute measurement data into Cuba, we can see that the 13 May activation of ETECSA's link to Jamaica followed some connectivity problems on 10 May, as there were brief decreases in the rate of completing measurements on that day, regardless of upstream provider. This was followed by a reduction in connectivity through Telefonica (purple) and then the C&W Jamaica (green) activation.

On the Telecom Venezuela website describing the ALBA-1 cable project, it mentions that the link to Jamaica is for restoration purposes ("El otro segmento para fines de restauración será entre Cuba y Jamaica"). So perhaps this activation is to help alleviate some minor connectivity issues experienced recently by ETECSA. Regardless, it is great news to see another submarine cable connection get activated for Cuba.

Syrian Internet.. Fragility?

2013-05-08T15:03:00Z

Update (15:26 UTC, 15 May): Routes to Syrian networks have been restored, at 18:26 Damascus time. Outage duration: 8h25m

Update (14:20 UTC, 15 May): Plot of latency measurements to Syrian hosts from various locations, indicating that replies stopped returning shortly after 7am UTC, aligned with the withdrawal of routes to Syrian networks. (Click image for details)

Update (07:30 UTC, 15 May): Syrian Internet down again since 07:01 UTC (10:00 Damascus time), Wednesday, 15 May 2013. Syrian news agency reports that they're working to fix. Potentially related to forthcoming UN decision today?

Older Update: Syrian Internet has returned. Outage lasted 19.5 hours, from 18:45 UTC May 7th to 14:13 UTC May 8th.

As we write, the Syrian people are still disconnected from the global Internet at the most fundamental level, nearly all of their paths withdrawn from the global routing table. Since 18:45 UTC on May 7th, Renesys hasn't seen a flicker of activity. We haven't been able to successfully send a ping or a traceroute to any host inside Syria. Government websites, universities, domain name servers, core infrastructure routers, banks, businesses, DSL customers, smartphones: all silent.
]]> As I look back at what we've written about Internet outage over the years, I see a sort of evolution in our perspective. We've covered Internet failures due to war, politics, censorship, central planning, earthquakes, hurricanes, cable cuts, business disputes, terrorism, undersea mud volcanoes, and (perhaps) cyberwarfare.

In the early days, we reported each outage breathlessly, shocked that the Internet could fail in such spectacular ways. If you look around the web this morning, you'll see a lot of that same shock-and-awe reporting from companies who are just discovering the fragilities visible in Internet data.

In this case, however, what strikes me is the depressing sameness of the sequence of Syrian Internet disconnections. Just as in June 2011, July 2012, August 2012, and November 2012, the entire nation disappeared from the Internet in 30 seconds, as if a switch had been thrown. Everyone in the Twittersphere seems to share the same strange lack of perspective about these events — in the middle of the chaos and tragedy of civil war, why is anyone surprised when the Internet stops working? Isn't it actually more shocking and noteworthy that the Internet in Syria actually functions pretty well 360 days out of the year?

If you're looking for a different way to think about events like these, you might enjoy reading Nassim Nicholas Taleb's book, Antifragile (ironically, published just days before the last major Syrian Internet disconnection).

Antifragility is a property of systems that get stronger when they are challenged by stresses. It's not merely robustness, or resilience (systems that return to stable state after stresses have passed) — an antifragile system like the Internet actually gets better when you try to break it. Systems of systems attain antifragility because their individual elements are so fragile. For example, Taleb writes:

"Restaurants are fragile; they compete with each other, but the collective of local restaurants is antifragile for that very reason. Had restaurants been individually robust, hence immortal, the overall business would be either stagnant or weak, and would deliver nothing better than cafeteria food — and I mean Soviet-style cafeteria food. Further, it would be marred with systemic shortages, with, once in a while, a complete crisis and government bailout. All that quality, stability, and reliability are owed to the fragility of the restaurant itself."

If you've been reading all of our Internet outage case study blogs, you might see a familiar pattern. The Internet fails in some way, taking individual institutions (or even entire countries or geographic regions) offline. The outage exposes fundamental undetected weaknesses — lack of submarine or terrestrial cable diversity, lack of Internet provider diversity, underinvestment in alternative forms of transport, political control over Internet chokepoints. If the Internet damage is painful enough, engineers (or revolutionaries) are forced to take creative steps to address those weaknesses. Repeat... forever.

Take Pakistan, for example. This cycle of insult-and-repair is what made Pakistan Telecom serious about buying eastbound backup transit to Asia in the wake of the first round of Mediterranean cable cuts in 2008. It's what led (indirectly or directly) to the establishment of competing providers in Pakistan, with foreign investment and additional cable resources connecting Pakistan to the Gulf States.

And because Internet is easily exportable, Pakistan's improved Internet connectivity spilled over to improve the situation in their historically fragile neighbor, Afghanistan. Here's a picture of one slice of Afghanistan's diverse connectivity over the last few weeks (click to zoom):
Note three paths from Afghan providers to Cable and Wireless in Europe. Borrowing from Pakistan's diversity, one route goes through Pakistan's competitive provider, Transworld Associates, then to Omantel and over the EPEG cable through Iran and the Russian Caucasus to Europe. Another goes by satellite to Bahrain's Batelco, then over to Omantel and out through Iran on the same EPEG route. A third goes north through Uzbekistan's Intal Telecom, then through two Russian routes to Europe. The Iranian EPEG route is itself an improbably antifragile innovation, inspired by multiple years of Mediterranean cable cuts and avoidance of the demonstrably fragile Egyptian transit corridor.

So what does all this mean for Syria and its Internet neighborhood? One has to look past short-term dysfunction and think about what comes next. Every significant Internet disconnection, and the local and global reaction of outrage and dismay, sends an important signal about the fragility of the underlying system. It makes single points of failure and control visible, so that those fragilities can be found and fixed, and the Internet as a whole can continue to gain strength from disorder.

Gulf States Turn to Iran, Russia for Internet

2013-04-22T21:00:00Z

Sometimes, it takes a real disaster to create something genuinely new. March 2013 was a month of disasters in the Middle Eastern, South Asian, and East African Internet, with major submarine cable cuts affecting SMW3, SMW4, IMEWE, EIG, SEACOM, and TE-North.

One of the "genuinely new" Internet traffic paths that emerged in response is a counterintuitive terrestrial route, linking the ancient Indian Ocean trade empire of Oman with the Internet markets of Western Europe, by way of Iran, Azerbaijan, and the Russian Caucasus. As we'll see, its effects are now being felt across the region, from Pakistan, to Gulf states like Bahrain and Oman, to Kenya.

The EPEG (Europe-Persia Express Gateway) consortium was actually born in June 2011, as an alternative to the congested, politically uncertain Suez transit corridor. EPEG links together existing fiber routes from the Iranian, Azeri, and Russian incumbents, connected to Cable and Wireless's network to approximate a Great Circle route to Frankfurt. With the aid of one final submarine hop across the Strait of Hormuz to Muscat, EPEG promised to deliver a major new low-latency, high-capacity terrestrial route to carry the Gulf states' traffic to Europe.

EPEG was scheduled to begin service in the summer of 2012, but delays on the final submarine links between Iran and Oman caused the start date to be pushed, and pushed again. When Vodafone acquired Cable and Wireless in July 2012, one of the attractions of the deal may have been the prospect of participating in EPEG — when the project reached completion, Vodafone/C&W would sit astride the first low-latency terrestrial link connecting the Gulf region to the financial and content markets of Western Europe.

We reported in February 2013 that in the wake of Russian service interruptions in the Black Sea, Iran's DCI had begun receiving Internet transit from Omantel, indicating that the missing link in the southern path was alive at last. We began watching intently for signs of end-to-end service activation on EPEG.

Last month, we saw them.

]]> EPEG in Action in Oman

To some within the GCC, EPEG may seem geopolitically implausible, but as a lower-latency alternative to Egyptian transit, it makes a lot of sense.

Here are some screenshots from our new Internet Business Intelligence suite, due out later this year, showing the emergence of Cable and Wireless transit as the fastest transit path into Oman (click on image for higher resolution). The new transit path from Omantel to Cable and Wireless emerges on March 31st (in green), some 15 milliseconds faster than existing transit paths from Frankfurt to Oman. Note the impact of the cable cuts before March 31st. By comparison, latency variance on the new path is tight, suggesting that the Iranian path has ample capacity and is still lightly loaded.

Bahrain taking advantage of EPEG

Batelco, Bahrain's incumbent provider, wasted no time in taking advantage of the new, lower-latency path to Europe for selected customers.

As this screenshot illustrates, we estimate that transit through Omantel and on to Cable and Wireless through Iran reduces the round trip time for Bahraini traffic to Frankfurt by as much as 30ms. Despite the ongoing political tensions between Bahrain and Iran, that's a pretty compelling argument.

Pakistan's Transworld Transiting EPEG

Pakistani competitive provider Transworld (AS38193) partners with Omantel to operate the TWA-1 submarine cable, so it wasn't a big surprise to see Omantel appear at the end of March as a remedy for the transit confusion caused by the SMW4 cut off Alexandria, Egypt. In short order, a significant portion of all the traffic to Transworld's 2,000+ routed prefixes has begun to traverse the EPEG link through Oman and Iran.

Faster Transit for East Africa

We even see EPEG's benefits reaching into East Africa, where AccessKenya (AS15808) picked up Omantel transit, creating a path that's more than 20 milliseconds faster than the previous alternatives to Frankfurt.

As we've seen before, East African providers are very quick to adapt to changes in the transit environment, and Omantel may do very well if other providers in Kenya, Uganda, and Tanzania get on board.

A lot will depend on the price — Omantel is probably able to command a significant premium for the uniqueness and low latency of the EPEG route. When competing cables return to service, we'll see how that pricing affects the new market equilibrium.

Conclusion

If you'd told me five years ago that we would one day see Iranian and Russian terrestrial Internet transit serving the countries of the Indian Ocean, from Pakistan to East Africa, I wouldn't have believed it.

Today, I'm a believer. Middle Eastern and East African network operators are using the Renesys Internet Intelligence tools to sniff out new transit opportunities in the market, and finding creative ways to recover from almost unimaginable connectivity challenges.

We've seen all kinds of strategies emerge in the wake of the SMW4 cuts, from Telecom Egypt transit (as far away as Afghanistan), to Israeli transit, to Iraqi transit, to Syrian transit through a war zone. In the hands of Omantel, EPEG is the first solution that appears to be delivering globally visible terrestrial routes to a diverse set of regional providers.

It remains to be seen whether an Internet path through Iran and the Russian Caucasus region will have the kind of stability that enterprises require. But frankly, compared to the submarine cable competition, they're already looking pretty good.

Intrigue Surrounds SMW4 Cut

2013-03-28T15:04:54Z

It has been a rough few weeks for the global Internet, given numerous submarine cable failures and the largest DDOS attack ever reported. While we're hard-pressed to find evidence of the purported global Internet slowdown due to the DDOS attack, the dramatic impacts of yesterday's SMW4 submarine cable cut were profound. Recent reports that the cable break was the result of sabotage make the incident even more intriguing. In this blog, we detail what happened to some of the providers in four countries along the route of the cable: Egypt, Saudi Arabia, Pakistan and India.

]]>

Impacts of the loss of SeaMeWe-4

This month's submarine cable outages have profoundly degraded connectivity to the Middle East, Asia and Africa. In particular, last week's EASSy and SEACOM outages wiped out connectivity in parts of East Africa from Djibouti to South Africa. As if that weren't bad enough, the biggest submarine cable connecting Europe and Asia, SeaMeWe-4, suffered a failure at 6:20 UTC, 27 March.

Later in the day, the Egyptian Naval forces claimed that they had caught divers sabotaging a submarine cable off the coast of Alexandria, Egypt. In any event, SMW4 clearly suffered a major failure, and as we will see below, caused widespread disruption of Internet services from Egypt to Pakistan.

These plots of latency measurements below show that westbound transit to Europe for many of these providers was either degraded or unavailable due to the SMW4 cut off the coast of Alexandria, Egypt. These providers fell back on eastbound transit through the Far East, but these backup paths weren't always up to carrying the load.

Egypt

Immediately following the cut (marked with a red line), Egyptian incumbent, Telecom Egypt, experienced a brief disruption in Internet transit followed by a reshuffling of its transit providers as it lost service from Level3 and NTT.

The loss of service from Level3 and NTT and gain of eastbound providers Bharti and Singapore Telecom (SingTel) is much clearer when the same data is viewed as a stacked graph of counts of traceroutes crossing through various providers into TE's network:

Saudi Arabia

Saudi Arabian incumbent, Saudi Telecom (STC), experienced dramatic increases in latencies to Europe as traffic shifted away from the severed submarine cable.

Headed the other direction from Saudi Arabia to Hong Kong, the cut caused STC to reshuffle its transit, but without the dramatic rises in latencies.

UAE (added 2 April)

As was the case with other Middle Eastern providers, Etisalat experienced a dramatic shift in traffic from Europe and the west, but less so from the east.

Emirates Integrated Telecommunications Company also known by its retail name, "Du", appears to have weathered the cable cut without significant disruptions to either the west or the east.

Pakistan

The impact of the SMW4 cut on the Pakistani Internet has been severe, as reported by the Pakistani media.

Pakistani incumbent, PTCL, was already struggling due to the recent loss of another major submarine cable. The graph on the left shows that latencies from Frankfurt to PTCL went through the roof as traffic was re-routed around the world to get to Pakistan. From Singapore, (on the right) there was a minor disturbance, but not nearly as crippling as for westbound traffic.

For Pakistan's other international gateway, Transworld, the impacts were severe when connecting to both the west and the east.

India

With the loss of SMW4, India-based Bharti lost its main route to the west. Delays for Internet traffic coming from western Europe spiked. From the east, things were fine.

Like Bharti, Vodafone India experienced dramatic increases in latencies for connections to and through Europe.

When connecting through the East, Vodafone India was largely stable. In fact, from Los Angeles, latencies improved slightly as traffic from LA was allowed to go a slightly shorter (and presumably more expensive) path across the Pacific Ocean instead of the Atlantic to get to India.

Conclusions

For me, one of the takeaways from the above analysis is that many of these providers have carefully engineered geographic diversity into their submarine cable strategy. However, the backup paths don't always deliver relief. In Pakistan, for example, the loss of westbound transit has virtually crippled the Internet despite the fact that eastbound routes stayed up. It is almost the mirror image of the impact on Bangladesh in last year's SMW4 cable cut near Singapore. In that case, Bangladesh lost eastbound transit due to the cable cut and the westbound backup just didn't have the bandwidth to handle the load, leaving the country essentially cut off.

Finally, I'll be giving a talk next month in Paris about visualizing the impacts of submarine cable breaks such as yesterday's SMW4 cut at Suboptic 2013, the world's largest conference about the submarine cable business. We obviously have a lot more to discuss now as an industry. If you are attending, please stop by and introduce yourself.

Cyber Escalation in Korea?

2013-03-20T15:39:00Z

On Friday, we published a blog confirming a disruption in Internet connectivity affecting the Internet of North Korea.

This morning, South Korean authorities reported that they have been the victims of a cyber attack which impacted TV News organizations as well as banking institutions.

Renesys can confirm that at least some of today's incidents escalated to the point of global visibility, as both South and North Koreans networks experienced actual disconnections. We note similarly timed outages affecting South Korea's largest natural gas company.

]]> South Korea News Media Outages

We observed 5 routed networks of Korea Broadcasting System go down at 05:54:18 UTC this morning (20 March). At the time of this writing they are still down. The Yonhap News Network (YTN) also experienced outages of two of its networks today at 05:54:30 UTC and 06:29:26 UTC.

South Korea Banking and Energy Outages

About thirty minutes after the South Korean media outages, we observed Korea Gas Corporation (Kogas, AS9857), the largest liquefied natural gas import company in the world, go completely offline as its 10 routed networks went down at 06:26:30 UTC and stayed down for over two hours. At the same time, we saw three routed networks of Shinhan Bank taken down as well.

For context, remember that South Korea normally routes over 15,000 networks. On a typical day, 40-50 of these networks are down temporarily for various technical reasons. However, networks from these sectors (Media, Energy, and Banking) are typically some of the most stable, and the timing of their simultaneous outages seems suspicious.

Additional outages in North Korea

Since last week's disruption in connectivity in North Korea, we have observed additional brief routing outages for the four routed networks of North Korea. On Monday (18 March) and this morning (20 March), we observed outages lasting for just a few minutes in North Korea. It should be noted that although North Korea's Internet is small, it is very stable. Until last week, North Korean outages had been very rare.

Summary

It is impossible to know from connectivity measurements alone whether these outages were the direct result of cyber attacks. However, given the recent rhetoric between these two nations, it is hard not to see these as ominous developments on the Korean peninsula.

North Korea Suffers Outage

2013-03-15T19:47:28Z

Earlier this morning, North Korea accused the United States of conducting a cyber attack that disrupted their Internet connectivity. While the details remain unknown, we can confirm that, in the last two days, North Korea's sole Internet provider has had ongoing problems staying connected to the global Internet. We'll summarize some of our evidence in this blog entry.

]]> Internet in North Korea

North Korea has an extremely small Internet for a country of 24 million people. Not counting the network involved in the recent Pirate Bay hoax, the four networks of North Korea are routed by a single Internet service provider, Star JV (AS 131279), which has two international Internet service providers: China Unicom (AS 4837) and Intelsat (AS 22351). Star began service on 18 November 2010 and gained Intelsat as a provider on 8 April 2012.

Recent Disruptions

We observed disruptions in North Korean Internet connectivity beginning at 00:59:30 UTC on 13 March 2013. At this time, North Korea's four networks were very briefly removed from the global routing table (chart lower left). When the routes were restored, one of the four networks was routed over Intelsat, while the other three were routed over China Unicom. After a few hours, all networks were once again routed over China Unicom. For about two hours starting at 22:40 UTC on 13 March, all four networks disappeared for a second time from the global routing table. Later on 14 March, we saw Intelsat again appear as a provider for one of the networks for several hours.

Despite such routing instabilities, North Korean networks were generally available in the global routing table. However, when we look at our active measurements (i.e., traceroutes) into North Korea during this time, we see a significant drop-off in successful responses, suggesting a loss of connectivity not visible in routing data alone (chart lower right).

The following graphic presents another view of latencies into North Korea, constructed from recent measurements using all of our worldwide servers. This perspective helps illustrate the disruption of connectivity. Traceroutes through Intelsat appear for some North Korean hosts as they failover to satellite service.

Rest easy everyone. The Pirate Bay was unharmed in this incident.

Faraway Fallout from Black Sea Cut

2013-02-14T15:19:00Z

Last month, Wired.com's fascinating geological sciences blog, Eruptions, cast doubt on the purported cause of the December 23, 2012 failure of the Georgia-Russia submarine cable. That is, the author of the Eruptions blog post thought it unlikely to have been due to an undersea volcanic eruption. Without weighing in on the likelihood of active volcanoes in the Black Sea, we tweeted about some of the Internet impacts of this incident, although in 140 characters, we could only scratch the surface. We'll take a more in-depth look in this blog, noting shifts in traffic as far away as Oman, more than 3,000 kilometers distant!

]]> Fiber Cuts in the Caucasus

But first, let's consider some recent history. The last time a fiber optic cable cut in the Caucasus grabbed international attention was almost two years ago (March 28, 2011) when a 75 year old Georgian woman reportedly severed the major terrestrial fiber optic cable serving the region while collecting scrap metal.

After her arrest, she told the press she had never heard of "the Internet" and that she did not break the cable, which knocked out western Internet transit for providers in Georgia and Armenia at exactly 13:00 UTC. I emphasize western because Russian incumbent, Rostelecom, was still serving the region from Baku, Azerbaijan during the fiber outage. Headlines suggesting Armenia was completely offline were incorrect, the cut just shifted the remaining Georgian and Armenian traffic through Rostelecom until the fiber was fixed less than 8 hours later at 20:32 UTC.

Although Armenian incumbent Armentel was entirely dependent on the severed cable, other Internet providers in Armenia such as Fibernet, which had Russian transit, were able to survive the incident relatively unscathed. In the graphics below, we observe traceroute measurements into both Armentel (AS 12297) and Fibernet (AS 41965) on March 28, 2011.

In upper left, Armentel's service from global provider TeliaSonera and Armentel's parent company, Russian provider Vimpelcom, were lost during the incident. In the upper right, Fibernet lost Level 3 service from the West, but not Internet transit from Rostelecom.

As a result of this incident, we saw an uptick in western and Russian transit diversity in the region as presented by my colleague Jim Cowie in June 2011 at the Eurasia Network Operators Group conference in Moscow.

Impacts of the Georgia-Russia cable cut

After this fiber cut and repair in 2011, Internet access to the region was relatively stable until recently. As we tweeted last month, service into Georgia from Azeri provider Delta and service in Armenia via Armentel were impacted by the Black Sea submarine fiber cut. Armentel lost its service from parent company, Vimpelcom, initially replacing it with transit through Caucasus Online and later TeliaSonera. Delta lost its primary transit provider for Georgia, Russian provider TransTelecom (TTK) (AS 20485), and made up for the loss with additional transit from Tata (AS 6453) and Level 3 (AS 3356) to the west and Rostelecom to the east.

On Tuesday of this week, headlines in Armenia carried the news that Armentel had just days prior expanded their international bandwidth capacity by again acquiring service from TeliaSonera. Prior to the headlines, users of Renesys's Market Intelligence had spotted the change in service in its news feed.

Impacts on Iranian Transit

Providers in Georgia and Armenia were not the only ones to lose Russian transit when the Georgia-Russia cable failed. Iran also uses Russian transit to access the Internet, some of which was dependent on that cable. On January 7, Iranian news carried a story (cached English translation) with the headline "Internet slow over the last week due to an earthquake reported in Georgia".

Iranian incumbent TIC, previously known as DCI, uses two ASNs for its International gateways, AS 12880 and since last year, AS 48159. As can be seen from the transit shift graphics below, the fiber break had no impact on upstreams of AS 12880. However, AS 48159 lost its service from Rostelecom when the cable was cut. What is more interesting is that the loss of Rostelecom for AS 48159 appears to have immediately triggered the activation of a dormant backup transit link from Omantel. This is the first time we have seen Omantel providing Internet service in Iran (although we did observe Omantel and TIC testing this connection from November 3 to 7, 2012). For good measure, TIC also added Delta as a provider in early January for both of its ASNs — representing another completely new business relationship. Of course, all of these changes were also identified in Market Intelligence.

The activation of this connection could be evidence that the final leg of the Europe-Persia Express Gateway (EPEG) fiber optic cable connecting Iran to Oman has been completed. It was this segment of the cable system that was blamed for the delay of the launch of EPEG last year. Two weeks ago, Azeri press announced EPEG would be activated this month. If activated, EPEG could serve as a vital terrestrial alternative to the dozen or so submarine fiber optic cables through the Suez canal which connect Europe and Asia — thereby mitigating this potential single point of failure for global communications.

So was it an undersea Volcano?

The most likely cause of the cable break in the Black Sea was the magnitude 5.7 earthquake, which struck a little more than an hour before traffic stopped flowing and appears to have occurred under the path of the cable. Regardless of the cause, it is fascinating to see how an earthquake in the Black Sea can immediately impact Internet pathways as far away as Oman. The Black Sea and the Caucasus are the Internet crossroads where Russian, Turkish, Gulf State, and Central Asian traffic mingle on their way to the datacenters of Western Europe and the Americas. Small impacts to particular cables can have large impacts on the flow of traffic regionally. Such is the Internet and our increasingly interconnected world. If you want to learn more from the operators involved, come join us at MENOG 12 (March 6, Dubai) and ENOG 5 (May 27-28, St Petersburg).

Bangladesh Connects via India

2013-01-29T14:02:31Z

The Internet of Bangladesh has been connected to the world by a single submarine cable, Sea-Me-We 4 (SMW4), since this 18,800 kilometer-long optical-fiber system made its landing at Cox's Bazar in 2006. However, in the nearly seven years since SMW4's activation, national Internet outages have plagued Bangladesh with some regularity. When their portion of this system is sabotaged, suffers a failure or is down for maintenance, virtually all Internet bandwidth for the 7th most populous country in the world disappears, forcing local providers to fall back to slow and expensive satellite services or to simply wait for restoration.

However, recent national outages due to planned SMW4 maintenance have revealed that some Bangladeshi providers have now activated a long-awaited second connection to the Internet via a terrestrial link to India. We'll examine this new development here and highlight those providers who can now offer fault-tolerant Internet service for the first time in Bangladesh.

]]>

The Need for a Backup Link

As far back as 2007, only one year after activating the nation's sole submarine cable connection, the telecommunications community in Bangladesh began advocating for the construction of a secondary physical connection to the Internet. However, despite years of plans and proposals, a second operational submarine cable hookup still seems to be years away.

In the meantime, major disruptions of Bangladesh's Internet service continue to occur. A recent example is the SMW4 fault from last summer, which we examined here, a failure that underscored how uniquely vulnerable Bangladesh's connectivity really is. Such events have prompted the Bangladesh Telecommunications Regulatory Commission (BTRC) to act. In January 2012, the BTRC issued licenses to six companies to connect to India via the International Terrestrial Cable (ITC). The six are:

First evidence of Terrestrial Link to India

In November, we tweeted the first hard evidence of the ITC in use. When Bangladesh's connection to SMW4 was once again taken down for maintenance, Fiber@Home was able to stay online via India, while the incumbent BTCL and others were either disconnected or had to rely on satellite back-ups. Using our latency measurements into Bangladesh during the outage, we were able to identify the impact of SMW4's unavailability on each provider. Of the four providers illustrated below, only Fiber@Home stayed online (through India) and they even managed to attract additional traffic from dual-homed customers who then had to rely solely on them to maintain connectivity.

The two SMW4 planned outages for Bangladesh on the 19th and 23rd of this month have helped to reveal additional providers which are now receiving Internet transit from India-based Tata via the ITC and no longer solely dependent on SMW4 for Internet access. The graphics below show traceroutes going into providers in Bangladesh during the SMW4 outage from 20:14 to 21:53 UTC on 23 January, 2013. While incumbent BTCL and Mango are completely offline during the downtime, 1Asia, Aamra Technologies, NovoCom and Fiber@Home experienced increases in their traffic as customers automatically shift traffic to the surviving connection. It should be noted that the increase in latencies to Aamra is due to Tata hairpinning traffic to Aamra via Los Angeles, not because it is providing satellite service.

Greater Terrestrial Connectivity in Asia

Abu Saeed Khan, the Senior Policy Fellow of LIRNEasia, helped persuade the government of Bangladesh to join the SMW4 consortium in 2002 and has been working ever since to increase Internet inter-connectivity all across South and Southeast Asia. For several years now, he has been advocating his concept called the Longest International Open-access Network (LION) which seeks to have Asian countries migrate from the submarine cables which currently connect them together to terrestrial links, leveraging the extensive international highway system already in existence. The Bangladesh-India link may be a first step towards that reality.

For Bangladesh, the recent establishment of an operational terrestrial connection to India to serve as an alternative to SMW4 is a tremendous accomplishment. As this service becomes more widespread in the country, the recurring national Internet outages should become a thing of the past.

Cuban Fiber: Completo?

2013-01-22T19:00:01Z

For more latest information, please follow us on Twitter: @Renesys

Cuban Internet connectivity continues to evolve by the hour, with a new, faster mode of operation in evidence as of this morning. Our measurements from around the world suggest that Cuban technicians may have completed the work they began a week ago, creating the first bidirectional Internet paths that are free of satellite connectivity.

]]> In our previous blog, we described how we first detected the signs of change in Cuba's Internet connectivity on Monday, 14 January 2013. Before that date, everything sent to Cuba experienced a delay of at least half a second, because of the speed of light: every packet sent and received had to travel around 70,000 km (44,000 miles) through space, bouncing off of geostationary satellites and returning to Earth.

On January 14th, we began to see routed paths for data flowing inbound to Cuba through a new service provider, Telefonica. At the same time, we saw a second "mode" of latency for Internet traffic emerge, with reduced delays in the 400ms range. That's fast enough that at least half the path must be terrestrial; we inferred that we were seeing signs of the activation of ALBA-1 from Venezuela.

For the next week, however, nothing changed. People on the ground in Havana reported no changes in Internet performance. When we wrote our blog on Sunday, it wasn't clear whether this curious one-way connectivity was intentional, or the result of misconfiguration.

Today, that all changed. At exactly 14:01 UTC Tuesday (09:01 local time), we saw yet another mode emerge in the latency diagrams. In this plot, you can see the original pure-satellite mode (A), the new asymmetric satellite mode (B), and a third, lower mode (C) that excludes the possibility of geosynchronous satellite service altogether. At 180-220ms, these paths suggest a pure terrestrial solution, based on subsea and overland cables — the traditional Internet that nearly everyone else on earth enjoys. Almost immediately, we started getting reports from Havana that delays for Internet traffic were dropping perceptibly, as the new routing policy kicked in.

What happened here? We speculate that Cuban network operators changed their routing policy to make the ALBA-1 cable the default path for all outbound traffic from certain Cuban networks. That would align with what we see in the data: some satellite providers, like Intelsat, move from mode A to faster mode B (becoming asymmetric: cable outbound, satellite inbound), while some prefixes move from mode B to still faster mode C (becoming symmetric terrestrial: cable outbound, cable inbound).

We'd like to hear confirmation from the Cuban network operators themselves, and we hope they'll comment below.

Mystery Cable Activated in Cuba

2013-01-20T14:00:03Z

Recent developments captured in a new blog here. (January 22, 2013)

In February 2011, the first submarine cable connecting the island nation of Cuba to the global internet (by way of Venezuela) landed on Siboney beach, Santiago de Cuba. In the two years since, the fate of the cable has been a mystery for Cuba observers. In the past week, our global monitoring system has picked up indications that this cable has finally been activated, although in a rather curious way, as we explain below.

]]> Connecting Cuba to the Internet

In 2007, state-owned telecommunications companies from Cuba and Venezuela joined forces to build a submarine cable between the two Caribbean nations, linking Cuba directly to the global Internet and allowing it to end its reliance on satellite-based Internet services. At least that was the hope. The cable was named the "Alternativa Bolivariana para los Pueblos de nuestra América" or ALBA-1 for short.

Originally planned to be completed in 2009, the project hit delay after delay, until construction was finally completed in early 2011. However, despite the announcement of its completion, Cuba's Internet has still limped along on high-latency satellite service via three different Internet service providers. That is, until last Monday when we noticed that Spanish telecom giant Telefonica began service to Empresa de Telecomunicaciones de Cuba S.A. (ETECSA), the state telecom of Cuba.

To underscore the significance of this development, we look back at how ETECSA obtained Internet service over the past 6 years. During this period, we see the same three satellite providers, although the routes carried over each has varied considerably over the years (below left). However, zooming in on the last couple of months, we see the entrance of Telefonica in our routing data as of last week (below right).

In addition to our routing data, we observe traceroutes into Cuba following a new path via Telefonica and recording significantly lower latencies. Latency measurements from four cities are illustrated in the graphic on the right.

We must emphasize lower latencies because, despite the drop, these aren't exactly low latencies. Our measured latencies to Cuba are still quite high, albeit improved. The fact that the latencies to Cuba from many locations around the world have dropped below 480ms means that the new Telefonica service cannot be entirely via satellite. However, if it were solely via submarine cables, we would expect latencies from many nearby countries to be less than 50ms. (Note: Round trip latencies for crossing the Atlantic and Pacific oceans are approximately 60ms and 110ms, respectively.)

Conclusions

We believe it is likely that Telefonica's service to ETECSA is, either by design or misconfiguration, using its new cable asymmetrically (i.e., for traffic in only one direction), similar to the situation we observed in Lebanon in 2011. In such a configuration, ETECSA enjoys greater bandwidth and lower latencies (along the submarine cable) when receiving Internet traffic but continues to use satellite services for sending traffic. While the activation of the ALBA-1 cable may be a good first step to providing ETECSA a better link to the Internet, the lack of widespread public access to Internet service throughout the island will likely continue.

On the same day last week that we saw the first evidence of the ALBA-1 cable, Cuba eliminated the requirement of an exit visa for its people to travel outside the country. Could these two developments be part of a greater trend towards a freer and more open Cuba?

A Baker's Dozen, 2012 Edition

2013-01-17T17:58:43Z

It is an annual tradition at Renesys to provide a year-end review of how the Internet providers at the top of our Market Intelligence global rankings fared over the previous year. The Internet remains a huge blind spot for many organizations that are buying Internet access. Market Intelligence provides the insight into who the leaders in the Internet transit marketplace are today and how they have changed over time. Back in 2008, we chose to look at the 13 providers that spent at least some time in the Top Ten that year, hence the name "Baker's Dozen". We looked at the top players again in 2009, 2010 and 2011. A lot has changed over the years and for 2012, we welcome two new members to this exclusive club, PCCW and XO. As predicted last year, we also say good-bye to a declining AT&T and Savvis. While AT&T's departure from the top of the global stage may be surprising to some, Savvis really hasn't left as it is now part of CenturyLink, which also owns Qwest. And while Qwest did leave our top global rankings in 2011, they have now returned as part of a reinvigorated CenturyLink.

As you read this blog, keep in mind that all of the rankings we discuss are relative to IPv4, the Internet protocol carrying over 99% of all Internet traffic. (For example, compare total traffic to IPv6 traffic at the very busy Amsterdam Internet Exchange.) While we did also review IPv6 rankings last year, so little has changed that we'll just refer you to that blog or, for more current information, our Market Intelligence product offering which covers both IPv4 and IPv6 in detail. So let's dive in and highlight a few of the trends and changes we observed in 2012.

]]>

And the Winners are …

This graph shows our global scores for the Baker's Dozen over the past year. As always, the absolute scores are not meaningful in this context, so we omit the scale. Five years ago, at the beginning of 2008, the top five in our rankings were Sprint, Level 3, Verizon, NTT and Global Crossing, in that order. Sprint has been steadily declining ever since and, as 2013 begins, was just surpassed by Tata, leaving it at 7th globally. Level 3 has acquired number-2 Global Crossing to become the undisputed number 1, and NTT and TeliaSonera are in a tight race for number 2. Verizon has dropped to 8th overall. In other words, the deck has been thoroughly reshuffled in a few short years. To make sense of the tangled graph above, we'll divide up the players into three tiers and zoom in on each.

On Top of the World …

Level 3 (AS 3356) completed their acquisition of Global Crossing (AS 3549) in October of 2011. We show them as separate entities here, since their networks have not yet been merged. In previous acquisitions, Level 3 has quickly absorbed the other organization. Thus we expected to see the old Global Crossing AS fall off dramatically in 2012, while the Level 3 AS soared. That was not the case. Instead, both entities declined steadily in the first half of the year before leveling out. The decline is best explained by several small changes, rather than any one large customer loss. Perhaps the market is adjusting to the merger by buying other transit in an attempt to preserve diversity? A wide range of customers reduced the number of networks that they had previously routed via Level 3, such as North America's nLayer (AS 25973, and recently acquired by Global Telecom & Technology, Inc. (GTT)), Japan's Softbank (AS 4725), Britain's Easynet Global Services (AS 4589) and Brazil's Embratel (AS 4230). But one of the more significant reductions was due to a drop-off by Cable and Wireless Worldwide (AS 1273), starting in the spring of 2012.

Despite these losses, the combined North American-based Level 3/Global Crossing score is still more than twice that of its nearest competitors, East Asian carrier NTT (AS 2914) and Eurasian carrier TeliaSonera (AS 1299), both of which have vied for the number-2 spot throughout the year. But with Level 3 declining or at best stagnating, the competition is gaining ground as the Internet continues to expand. When we first reported on the proposed Level 3/Global Crossing merger in April of 2011, we stated that the next five global providers would have to combine to rival the score of new mega-Level 3. Now, "only" the next three would have to do so, namely, NTT, TeliaSonera and Inteliquent. It will be interesting to see if this trend continues in 2013.

But absent dramatic consolidation, it's safe to say the global race is now for the number-2 position. While TeliaSonera was briefly ahead of NTT at the end of the summer, NTT has more recently reasserted itself and appears to be gaining on its nearest competitor. NTT saw increases in transit from Japan's KDDI (AS 2516), South Korea's LG DACOM (AS 3786) and a surging China Mobile (AS 58453) to name just a few. TeliaSonera saw dramatic increases in transit from SingTel (AS 7473). Clearly Asia, with most of the world's population and still relatively low Internet penetration, will remain a key battleground for years to come. This is a region of the world where we rank Level 3 second as AS 3356 and fifth as AS 3549.

Closest Thing to Heaven …

After leaving the rarefied air of the top three providers, we encounter our next group of five very strong global carriers. Here we see Inteliquent (AS 3257), formerly Neutral Tandem and Tinet, Cogent (AS 174) and Tata (AS 6453) keeping up with the overall growth of the Internet. Sprint (AS 1239), which was about to be passed by TeliaSonera at last year this time, has now just been passed by Tata. If they continue on this trajectory, Sprint may end up in our third group by next year. Finally, Verizon (AS 701) saw an interesting jump at the end of year to get within striking distance of Tata.

So where were the big wins? In Asia, of course. The main contributor to Inteliquent's rise was a substantial increase in transit from the 16th-ranked Asian carrier, Flag (AS 15412). Cogent's largest gains came from Korea Telecom (AS 4766), while Tata's came from Russia's Vimpelcom (AS 3216), Vietnam's VNPT Corporation (AS 45898), Bharti Airtel (AS 9498) and a number of smaller players. Even Verizon got into the act with a year-end surge largely due to increased transit from China Telecom (AS 4134). Now China Telecom roughly equally splits its transit needs between Verizon and Sprint. Without this single enormous customer, these two US-based carriers would tumble into our third group, shown next.

The Rest of the Best …

This brings us to the final five of our Baker's Dozen, namely, Telecom Italia Sparkle (AS 6762), PCCW (AS 3491), China Telecom (AS 4134), XO (AS 2828) and CenturyLink (AS 209). CenturyLink acquired both Qwest and Savvis in 2011. PCCW entered our Baker's Dozen for the first time this year after solid and consistent gains throughout 2011 and 2012. XO, after a strong 2011, ended up treading water in 2012, but still made the cut for the first time, thanks to the departure of AT&T and Savvis. Both Sparkle and CenturyLink stumbled a bit later in the year, which kept CenturyLink from passing XO and put PCCW within striking distance of Sparkle.

Again, we are seeing the influence of Asia within movement in this third group. Sparkle's biggest increases came from India's Bharti Airtel and Russia's Vimpelcom. The decreases later in the year are predominantly due to losing transit from Russia's Rostelecom (AS 12389) and losing Russia's MegaFon (AS 31133) as a customer entirely. Meanwhile, PCCW gained significantly from China Mobile's increased transit. The one exception to Asian influence in our rankings concerns CenturyLink, whose rise was dominated by increased transit from GTT/nLayer.

Conclusions

Westerners view the Internet as a commodity and ubiquitous access is taken for granted. Unsurprisingly, we continue to see a wave of consolidation in these markets as companies can only "grow" by acquisition. Organic growth can now only come in underserved markets, and today we see the battle for global Internet dominance playing out in Asia. Tomorrow, as its physical infrastructure improves, we may see more attention shift to Africa. Although Asia continues to be the growth engine for many of the Baker's Dozen NSPs, we are starting to see commoditization and market consolidation on its western edge. During the year a number of Russian NSPs successfully integrated their 2011 acquisitions and are now looking for a compelling competitive advantage in a tight IP transit market.

But, nearly everywhere you look, enterprises have a lot more good choices for Internet transit than they have in years past. That means that they are becoming more sophisticated consumers of Internet transit, which is why we find ourselves spending more time in 2013 studying Internet path performance, in addition to the providers' relative market share. Enterprises are starting to managing the Internet as part of their overall networking strategy and demanding explanations for poor performance from their providers. Any provider who can serve an increasingly interconnected world, and do so reliably with low latencies in good times and bad, will undoubtedly gain customers and grow in our rankings. Stay tuned.

Restoration in Syria

2012-12-01T16:24:57Z

Renesys confirms a largely complete restoration of the Syrian Internet this morning, starting at 14:32:10 UTC (16:32 local time in Damascus).

Transit providers for the full prefix set do not appear to be significantly changed, with Internet service being provided post-restoration by Telecom Italia, Tata Communications, Turk Telecom, and PCCW.

Here's a view of live Syrian prefix counts during the outage and restoration, from 29 November to 1 December:

]]>

And here's a zoom into the same data, during the period of the restoration:
Syrian upstream providers post-restoration look much as they did before the outage; Turk Telekom's restoration appears to have been delayed by an hour relative to the other providers.

The restoration was achieved just as quickly and neatly as the outage: like a switch being thrown. Does that mean that we believe the government (or the opposition) threw the switch? Frankly, the data available just don't support attribution at this point, despite all the speculation.

The only way we're going to know for sure will be to wait for a resolution to the conflict, at which point we will hear from the people who know for sure: the network engineers in Syria. We hope they're safe and we look forward to hearing their story firsthand.

Could It Happen In Your Country?

2012-11-30T16:32:57Z

{Updated; see below for lists of countries for easier reference, and a special offer for WCIT attendees. —jim}

How hard is it to disconnect a country from the Internet, really?

That's the number one question we've received about our analysis of the Egyptian and Syrian Internet blackouts, and it's a reasonable question. If the Internet is so famously resilient, designed to survive wars and calamities, how can it fail so abruptly and completely at the national level?

The key to the Internet's survival is the Internet's decentralization — and it's not uniform across the world. In some countries, international access to data and telecommunications services is heavily regulated. There may be only one or two companies who hold official licenses to carry voice and Internet traffic to and from the outside world, and they are required by law to mediate access for everyone else.

Under those circumstances, it's almost trivial for a government to issue an order that would take down the Internet. Make a few phone calls, or turn off power in a couple of central facilities, and you've (legally) disconnected the domestic Internet from the global Internet. Of course, this level of centralization also makes it much harder for the government to defend the nation's Internet infrastructure against a determined opponent, who knows they can do a lot of damage by hitting just a few targets.

]]> With good reason, most countries have gradually moved towards more diversity in their Internet infrastructure over the last decade. Sometimes that happens all by itself, as a side effect of economic growth and market forces, as many different companies move into the market and compete to provide the cheapest international Internet access to the citizenry.

Even then, though, there's often a government regulator standing by, allowing (or better yet, encouraging) the formation of a diverse web of direct connections to international providers. Here's the problem: increased diversity at the international frontier often spells less money for the national incumbent provider (typically the old telephone company, often owned by the government itself). Without some strong legal prodding and guidance from the telecoms regulator, significant diversification in smaller markets with a strong incumbent can take a long, long time.

Here's a map of the world, with countries colored according to the Internet diversity at the international frontier. We did a census, from our own view of the global Internet routing table, of all the domestic providers in each country who have direct connections (visible in routing) to foreign providers.

As a first cut at a diversity metric, this makes a lot of sense; it's easy to compute, and fairly objective (an NSP either has a foreign transit provider visible in the routing tables, or it doesn't). You can think of this, to first approximation, as the number of phone calls (or legal writs, or infrastructure attacks) that would have to be performed in order to decouple the domestic Internet from the global Internet.

If you have only 1 or 2 companies at your international frontier, we classify your country as being at severe risk of Internet disconnection. Those 61 countries include places like Syria, Tunisia, Turkmenistan, Libya, Ethiopia, Uzbekistan, Myanmar, and Yemen.
If you have fewer than 10 service providers at your international frontier, your country is probably exposed to some significant risk of Internet disconnection. Ten providers also seems to be the threshold below which one finds significant additional risks from infrastructure sharing — there may be a single cable, or a single physical-layer provider who actually owns most of the infrastructure on which the various providers offer their services. In this category, we place 72 countries, including Oman, Benin, Botswana, Rwanda, Pakistan, Kyrgyzstan, Uganda, Armenia, and Iran. Disconnection wouldn't be trivial, but it wouldn't be all that difficult. Egypt falls into this category as well; it took the Mubarak government several days to hunt down and kill the last connections, but in the end, the blackout succeeded.
If you have at least 10 internationally-connected service providers, but no more than 40, your risk of disconnection is fairly low. Given a determined effort, it's plausible that the Internet could be shut down over a period of days or weeks, but it would be hard to implement and even harder to maintain that state of blackout. There are 58 countries in this situation, ranging from Bahrain (at the small end) to Mexico (at the largest end). India, Israel, Ecuador, Chile, Vietnam, and (perhaps surprisingly) China are all in this category.
So is Afghanistan, reminding us that sometimes national Internet diversity is the product of regional fragmentation and severe technical challenges. It's true; the government in Kabul is powerless to turn off the national Internet, because it's built out of diverse service from various satellite providers, as well as Uzbek, Iranian, and Pakistani terrestrial transit.
Finally, if you have more than 40 providers at your frontier, your country is likely to be extremely resistant to Internet disconnection. There are just too many paths into and out of the country, too many independent providers who would have to be coerced or damaged, to make a rapid countrywide shutdown plausible to execute. A government might significantly impair Internet connectivity by shutting down large providers, but there would still be a deep pool of persistent paths to the global Internet. In this category are the big Internet economies: Canada, the USA, the Netherlands, etc., about 32 countries in all.

So, could what happened to Egypt and Syria happen in your country? Hopefully not. But it's an important question that companies ask Renesys about all the time, as they decide which countries might reasonably host their new data centers.

Governments that want to encourage direct foreign investment in ICT should have this in mind as they head to Dubai next week for the World Conference on International Telecommunications. Next to Internet performance and stability, the political risks of Internet disconnection are starting to appear on due diligence checklists, as companies consider where to make their investments in global cloud infrastructure.

Postscript (3 Dec 2012)

Wow, a lot of interest in this subject so far. Thanks for the comments and questions. Keep in mind that we're starting with a simple maturity model of foreign/domestic interconnection diversity — factoring in physical interconnection and the 'hidden entanglements' among the logical relationships is the next step. It's interesting that most people who are suggesting modifications to this model believe that their country is much more vulnerable to disconnection. Nobody has claimed that their country is more resilient than we give them credit for!

I'm collecting a list of most-commented countries, and we'll revisit these in some more detail in a future post.

Also, many have asked for a browsable list of the countries that appear in the map. Here's a table of the ISO country codes for each of the four categories. If you'd like more detail on these countries, the relationships among autonomous system providers in them, the size of the incumbents, their international transit arrangements, and the like, I have to recommend that you check out our Market Intelligence service. If you're trying to understand how the online world really fits together at the provider-customer level, region by region and market by market, Market Intelligence provides the best roadmap of who's connected to whom in the service provider industry, from San Francisco to Sao Paolo, from Benin to Vladivostok.

If you're a registered attendee at the WCIT conference in Dubai this week, send me your accreditation and we'll set you up with a free evaluation account.

Severe Risk	Significant Risk	Low Risk	Resistant
AD AI AN AW AX BB BQ BT CF CI CK CU DJ DM ET FO GD GF GL GM GN GP GY IO JE KM KN KP LS LY MC MF MH ML MM MR NF NR PF PM PW RE SB SO SR SS ST SX SY SZ TC TD TK TL TM TN TO UZ VA WF YE	AE AG AM AZ BF BI BJ BM BN BO BS BW BY BZ CD CG CM CW DO EG FJ FM GA GE GG GI GQ GU HT IM IR JM JO KG KY LA LK LR MA ME MG MN MO MQ MT MU MV MW NA NC NE OM PG PK PY QA RW SA SC SD SL SM SV TG TJ TT UG UY VG VI VU WS	AF AL AO BA BD BG BH CL CN CO CR CY DZ EC EE GH GR GT HN HR HU IL IN IQ IS KE KH KR KW KZ LB LI LT LU LV MD MK MX MZ NG NI NP NZ PA PE PR PS PT RS SI SK TH TR TW TZ VE VN ZA ZM ZW	AR AT AU BE BR CA CH CZ DE DK ES FI FR GB HK ID IE IT JP MY NL NO PH PL RO RU SE SG UA US

Syrian Internet Is Off The Air

2012-11-29T13:31:09Z

Click for latest update: 01:00 GMT Friday.

Starting at 10:26 UTC on Thursday, 29 November (12:26pm in Damascus), Syria's international Internet connectivity shut down. In the global routing table, all 84 of Syria's IP address blocks have become unreachable, effectively removing the country from the Internet.

We are investigating the dynamics of the outage and will post updates as they become available.

]]>

Update (15:45 UTC)

Looking closely at the continuing Internet blackout in Syria, we can see that traceroutes into Syria are failing, exactly as one would expect for a major outage. The primary autonomous system for Syria is the Syrian Telecommunications Establishment; all of their customer networks are currently unreachable.

Now, there are a few Syrian networks that are still connected to the Internet, still reachable by traceroutes, and indeed still hosting Syrian content. These are five networks that use Syrian-registered IP space, but the originator of the routes is actually Tata Communications. These are potentially offshore, rather than domestic, and perhaps not subject to whatever killswitch was thrown today within Syria.

These five offshore survivors include the webservers that were implicated in the delivery of malware targeting Syrian activists in May of this year.

It has been a crazy year for Syrian Internet transit arrangements, most recently with the loss of Deutsche Telekom as a transit provider earlier this month.

Update (20:30 UTC):

Still no signs of life from the affected prefixes. Looking back over the last week, you can see that the routing of the Syrian Internet has actually been pretty stable until today's wholesale shutdown.

There have been some brief up-and-down flickers affecting the reachability of a particular 12 networks, and there was one brief whole-country outage of less than ten minutes on 25 November. By the time that one was confirmed, the outage was over. It would be reaching to call that a "precursor event" or "practice run," but that's a possibility.

Also, some have asked about the submarine connectivity into Syria. Here's a map showing the three principal routes. There's also terrestrial connectivity into Turkey to the north, but those paths have not been reliable in recent months.

Update (01:00 GMT, 30 Nov):

The last 5 networks belonging to Syria, a set of smaller netblocks previously advertised by Tata Communications, have been torn down and are no longer routed. These blocks survived today's Internet blackout in Syria, but 12 hours after the onset, they, too are off the air. Traceroutes to these blocks now die on Tata's network in New Jersey, and websites hosted in these blocks are no longer responding.