Recently in Engineering Category

The end of the year is approaching which seems to be a harbinger of Internet disasters. Four years ago (on 24 Dec. 2004), TTNet significantly disrupted Internet traffic by leaking over 100,000 networks that were globally routed for about an hour. Two years ago (on 26 Dec. 2006), large earthquakes hit the Luzon Strait, south of Taiwan, severing several underwater cables and wreaking havoc on communications in the region. Last year there was a small delay. On 30 Jan. 2008, more underwater cables were severed in the Mediterranean, severely disrupting communications in the Middle East, Africa, and the Indian subcontinent.

Calamity returned to its customary end-of-year schedule this year, when early today (19 Dec. 2008) several communications cables were severed, affecting traffic in the Middle East and Indian subcontinent. According to a press release by France Telecom three major cables were damaged: Sea-Me-We 4 at 7:28 UTC, Sea-Me-We 3 at 7:33 UTC, and FLAG FEA at 8:06 UTC. It appears that the SMW3 cable was only partially cut, the SMW4 cable was completely cut, while the FLAG cable was "observed down" with no other information given. The location of the cut appears to be between Sicily and Tunisia in a section which is the responsibility of Egypt Telecom. The causes of the cut remained unclear. It seems that ships were deployed to repair the damaged cables, but no ETA was given.

In a few weeks, I will be leaving Renesys, a company I have been associated with for over five years. I moved from New Hampshire (where Renesys is headquartered) to Pittsburgh, PA, over the summer, and I've decided to work a bit closer to my new home.

Before I go, there is work yet to be done. The Renesys blog has become an important place for Internet engineers, managers, developers and salespeople to seek unbiased information about what is happening on the backbones. I have enjoyed contributing to it over the years, and I have enjoyed watching some of my colleagues (most actively Earl Zmijewski and Martin Brown) take the helm more recently. Before I ride off into the sunset, there are at least two things I'd like to contribute to this forum:

1. A clear assessment of where we are with this whole Internet project
2. A good guess about where we're going

At the end of the next series of posts by me, you should either be very, very worried or convinced that I'm very, very wrong. The Internet is facing a confluence of engineering, financial and policy storms that have some small potential to completely derail it. These tempests have a high likelihood of marking a sharp departure from several characteristics once considered fundamental to the the Internet.

If we get through the next five years, I'm sure everything will be fine. Today, I'll tackle the technology and engineering issues. In my next post, I'll address financial issues, followed by policy issues. At the end of this torrent of pessimism, I'll try to point to some plausible ways out of the mess that we have gotten ourselves into.

Sprint re-enabled the connection between Sprint and Cogent at 21:00 UTC (16:00 EST) on Sunday, 2 Nov, 2008. Sprint issued a hastily prepared statement about the reconnection (the HTML is a cut-and-paste job from "IP/MPLS Products from Sprint"), explaining their position. Cogent hasn't commented yet.

The connection appears to be routed much as it was before Oct 30. Previously, we saw Sprint selecting 2700-2900 prefixes from Cogent (that is, picking Cogent as the best path for that many network prefixes). We saw Cogent selecting about 7500-8000 prefixes from Sprint. Now that they have reconnected, Sprint is selecting 2538 prefixes from Cogent and Cogent is selecting 7016 from Sprint. So down slightly, but not appreciably. The link is up.

The fact that Sprint has reconnected this indicates clearly that they intend to fight this battle in court rather than in the routing tables or in the court of public opinion. This fact alone makes this likely to be one of the more interesting peering disputes of the last few years. But the resolution may take months or years, given the speed with which the courts move.

A special Halloween edition of the Renesys Blog: That which was whole is now torn asunder, and cries of grief ring out throughout the land. Cogent (AS174) and Sprint (AS1239) are no longer connected to each other. Customers of each network who do not have other providers—namely single-homed customers—cannot reach each other. Two large portions of the Internet are separated.

Cogent is frequently involved in peering disputes. In the last three years, the only significant peering dispute (one that caused a temporary partition of the Internet) that did not involve Cogent was between Level 3 and XO. That one was settled very quickly. All of the others (Cogent depeering Telia, Level 3 depeers Cogent, and further disputes going back years involving Teleglobe (now Tata, AS6453), France Telecom (AS5511)) involved Cogent.

But in this case, Cogent may have picked the wrong sparring partner. In the past, Cogent won peering disputes simply because their customer base was less sensitive to the outage than the other party in the dispute. Ultimately, the one whose customers complain the loudest loses. This time it may be very different. Sprint hasn't paid any particular attention to its IP product and network at a senior management level for a very long time. They are clearly focused on wireline and wireless telecom services and Overland Park management seem to remain mostly unaware that they even operate an IP network. In other words, Cogent has picked a fight with a zombie here. They may even rip off a limb or two, but that doesn't mean the zombie will notice.

Sprint and Cogent only starting peering recently, back in November of 2006. Prior to that the two networks reached each other via NTT Communications (AS2914). Now, almost exactly two years later, it appears that Sprint has disconnected Cogent and chosen to divide the Internet. Cogent has stated that they will litigate this issue so this one is unlikely to get resolved quickly. In the mean time, over 200 downstream autonomous system customers of each organization cannot reach the networks in the other. This is ugly and will remain so.

Let's take a quick look at what we know so far and set the stage for a story that will likely continue for several days, if not weeks. I'll also try to set this in a larger context about the evolution of each of these networks and the evolution of Internet interconnection on the whole.

As the world waited for Gustav to hit the US, at Renesys we wondered how the Internet would fare this time around. Would we see the large scale, long term outages we observed during Katrina? Or would the critical communications infrastructure of the region stand fast? As of 19:00 UTC on date of Gustav's landfall, the score so far is Internet 1, Gustav 0. Connectivity in the region is very good and outages are sporadic. Either we got lucky or we've learned some valuable lessons.

We've been keeping an eye on Georgia all week. It's rather hard not to as the media keeps calling, looking for a juicy story. (It's amusing how the questions can seem designed solely to confirm a story that has already been partially written.) Not being schooled in this "art", we haven't been able to invent any interesting "facts", as the network infrastructure of Georgia has been relatively stable all week. But then today, we did see about one third of the country go away again for an extended period. Since it wasn't the entire country, we didn't rush out and buy oil futures. And since the outaged networks did come back, we're assuming this event was due to a temporary (although perhaps extensive) power outage.

As the world watches events unfold in Georgia, all eyes are on the Baku-Tbilisi-Ceyhan (BTC) pipeline, a major source of European oil that is not under Russian control and is projected to carry 1 million barrels a day by 2009. (See this link for a map of oil pipelines in the area.) What many people don't realize is that the cyber world is often built alongside the physical one. That is, those fiber optic cables that carry Internet traffic tend to follow the world's pipelines, bridges, and railroad tracks. Loss of Internet connectivity can therefore imply the physical destruction of vital pathways for trade. And so it is with some interest that we monitored Georgian Internet connectivity over the weekend as hostilities with Russia escalated. This blog takes a quick look at how Georgia connects to the 'net and what has been happening over the last three days.