Submarine Cables Add Resilience But Paths Still Matter

Last month, I had the honor of opening the second day of the Submarine Networks World 2013 conference at the Marina Bay Sands in Singapore. Using a handful of recent examples, I demonstrated that, while new submarine cables certainly contribute to physical diversity and hence increased Internet resiliency, cables alone do not necessarily reduce traffic latency. As we explained in our recent blog, when trying to understand performance across the global Internet, paths matter. 0534

We say that paths matter because while there are many tools for monitoring end-to-end latencies of Internet connections, understanding the path that traffic takes through the global Internet is vital to understanding the root cause of performance issues and in designing an effective Internet strategy for your target markets.

As I told the audience in Singapore, Renesys is not in the submarine cable industry. We are an Internet measurement company. Our unique ability to detail submarine cable events (failures and activations) comes as a fortunate side-effect of our extensive data sets and the capabilities we have built on top of them for observing structural changes in the global Internet. These changes can dramatically impact performance, sometimes for the better, sometimes for the worse. The most significant events are often those involving submarine cables.

Submarine cables are carefully engineered to minimize cost and latency. However, Internet routing–which determines the actual paths taken by traffic–plays by a different set of rules. Though the shortest physical path may be across a submarine cable, routing policy may steer the traffic a different way. When routing policy does align with physical infrastructure, latencies can drop dramatically, as in the following two examples.

Tonga

In August, Tonga, an island nation of 100,000 people, got its first submarine cable, allowing it to forgo expensive and slow Internet satellite service. How expensive? In the telecom business, prices are typically quoted in terms of Mbps per Month (megabits per second per month or MM). In the competitive US market, this cost can be below $3/MM. For satellite service in the Pacific, the cost is around $3,000/MM — 1,000 times more expensive. How slow? Take a look at the following graphic that uses our latency measurements to illustrate Tonga’s Internet connectivity transition from satellite (600ms minimum) to submarine cable (40ms minimum), capturing the cable activation that occurred at 05:56 UTC on 5 August 2013.

tonga
I can only imagine that August 6th must have been a new day for Internet users in Tonga. While the business case of future cable projects faces distinct challenges (discussed later), Pacific islands are in the category the submarine cable industry calls “thin routes” — submarine cable connections that provide vital connectivity to people on remote islands but can only deliver a “thin” amount of demand, and hence, are unlikely to make much (if any) money. The World Bank and the Asia Development Bank funded the Tonga cable and other similar submarine cable construction on humanitarian grounds.

ACE cable activation in Sierra Leone

Funded in part by the World Bank, another recent cable activation was France Telecom’s ACE (“Africa Coast to Europe”) cable serving West Africa. With an additional US$15 million contributed by the government of China, the ACE cable came ashore in Sierra Leone and was activated in February in a ceremony attended by the country’s president. His Minister of Information and Communications called it a “landmark day in Sierra Leone.” Ace

This summer, an online publication in Sierra Leone questioned why Internet service from SierraTel, the country’s incumbent, to various government offices, including the Ministry of Information and Communications, was still slow. While there can be many reasons for slow Internet access, we can see that SierraTel may not be getting the maximum benefit of the new cable. In the diagram below, we compare the activations of SierraTel and Afcom, the “largest Internet and IP service provider in Sierra Leone” according to their website.

sierratel afcom

The downward shifts in both latency distributions mark the moments SierraTel (13:00 UTC, 21 February 2013) and Afcom (19:06 UTC, 20 February 2013) began routing traffic over the ACE cable. However, while parts of Western Europe can reach Afcom in less than 100ms, it can take over three times longer to reach SierraTel. Unfortunately, the in-bound path to SierraTel is still via the same satellite services in use prior to the ACE cable activiation.

This scenario suggests SierraTel is potentially misconfigured; asymmetrically using satellite for inbound traffic and submarine cable for outbound traffic. This is similar to the misconfiguration we saw in Cuba when ETECSA activated the ALBA-1 submarine cable in January, but this error was corrected a couple days later. We suspect a similar situation with SierraTel, which would explain the reports of continued slow Internet access.

Submarine Cable Industry Feeling the Pinch

Advances in the state of the art of submarine cable technologies, such as Coherent technology, are enabling upgrades to existing cables that can increase capacity by orders of magnitude, such as the upgrade of NTT’s Pacific Crossing-1 to 100G this summer. PacificCrossing_RGB_thumbnail

At the same time that submarine cables are increasing in capacity, the growth of international bandwidth demand has been slowing due to advancements in distributed content delivery networks. These networks position content closer to users for faster delivery, reducing the need to reach overseas hosting centers via submarine cables.

An increase in supply combined with a decrease in demand makes it much more difficult to build a business case for a new submarine cable project. However, with this year’s revelations concerning National Security Agency (NSA) interception, a new motivating factor for future cables may be emerging: national sovereignty.

Following the news that her email correspondence had been intercepted by the NSA, Brazilian President Dilma Rousseff has announced plans to bypass United States for Brazilian traffic destined for other countries (Brazil, like most Latin American countries, connects to the global Internet through submarine cables that land in Miami, Florida).

Screen Shot 2013-10-08 at 2.26.22 PM
This crisis couldn’t have come at a better time for Angola Cables, which is one of four efforts working to build a cable (pictured above) across the south Atlantic Ocean between Brazil and Africa. When complete, Angola Cables’ South Atlantic Cable System (SACS) will connect the two Portuguese speaking countries, offering Brazil another path to reach the global Internet by way of Africa, side-stepping the United States.

National sovereignty may be the trump card that overcomes the one-two punch of increasing capacity and decreasing demand, and justify the installation of submarine cables to serve political interests. Brazil has come to realize that Internet paths matter, albeit for security as opposed to performance issues.

Hair-pinning

I concluded my talk in Singapore with a few examples of Internet hair-pinning in the Far East. Hair-pinning is the phenomenon in which Internet traffic gets unnecessarily routed through a faraway city, a common occurrence that clearly illustrates that the Internet is not designed for efficiency or performance. In other words, you cannot simply use Telegeography’s excellent Submarine Cable Map in the same way you might use a street map for determining likely Internet traffic paths. The following example traceroute illustrates an extreme case of hair-pinning, where traffic from Singapore to another part of Singapore first traveled through Los Angeles, California, over 13,000 kilometers away!


singapore_hairpin2
This seemingly strange behavior does not reflect just routing inefficiencies or misconfigurations of minor players of the Internet. Rather, it is the direct result of the fact that Tata and NTT, two of the largest global networks in the world, do not interconnect in Singapore (presumably for business reasons) and instead elect to exchange traffic for that part of the world in California. This arrangement impacts the customers of both providers, dramatically increasing latencies for traffic between locations in the Far East.

Nearly every provider in the world has idiosyncrasies like these. They can make global Internet performance very difficult to understand, let alone monitor. It’s Renesys’ business to help our customers understand these issues so they can effectively plan their Internet strategy and monitor the delivery of their services, and it is why we say paths matter.

2 comments
routingbear
routingbear

event correlation by BGP is such a difficult task, if BGP4 can carry Lat/Lon information of each hop(even ibgp box) in announcement(1 time for each prefix per path/day i.e.),  things can be much easier.

Todd
Todd

Excellent article, even by your high standards!

Trackbacks

  1. […] of all, anyone who understands BGP, will tell you about scores of hair-pinning instances where even if the physical path (submarine cables) bypass the US, the traffic itself may very well […]