February 2008 Archives

Late in the (UTC) day on 24 February 2008, Pakistan Telecom (AS 17557) began advertising a small part of YouTube's (AS 36561) assigned network. This story is almost as old as BGP. Old hands will recognize this as, fundamentally, the same problem as the infamous AS 7007 from 1997, a more recent ConEd mistake of early 2006 and even TTNet's Christmas Eve gift 2004.

Just before 18:48 UTC, Pakistan Telecom, in response to government order to block access to YouTube (see news item) started advertising a route for 208.65.153.0/24 to its provider, PCCW (AS 3491). For those unfamiliar with BGP, this is a more specific route than the ones used by YouTube (208.65.152.0/22), and therefore most routers would choose to send traffic to Pakistan Telecom for this slice of YouTube's network.

Bob, the sales guy.

Ditched my #@!%$! cell in Stockholm. Verizon CDMA does not work in Europe! Upside: I now have a shiny, new World Edition Blackberry GSM/CDMA. I call it Trixie.

Road Tip: Just say NO! to mouth-searing kimchi or Indian curry for breakfast. No matter how polite you're trying to be.

With barely enough time to recharge Trixie after calls in Denver, Albuquerque, Stockholm and Bonn, I hopped a jet for Tokyo, Hong Kong and Kuala Lumpur. (Trixie and I barely made it out of KL alive. Cab driver must have been conserving gas; tried to piggyback car in front.) Beginning to feel like Marco Polo on 'roids, but I gotta check out LA and DC before catching a shuttle back home to Boston (close enough) .

Trixie is overloaded with commentary, observations, insider scoops, and . . . new NSP sales and marketing contacts! (Hey, I'm a sales guy.) Time to download and see what comes out . . .

In the past 14 months, the world has seen two catastrophic failures of its global telecommunications systems: the Taiwan quakes, which snapped 7 of 9 important cables in Asia in December 2006, and a series of mishaps in the Mediterranean and the Gulf, damaging several others. In a world increasingly dependent on global trade and communications, what lessons can we learn from all of this and what measures should we take?

I'll discuss these questions in what follows, but let me warn you in advance. There is nothing earth-shattering here. In fact, I can save you time and sum up the entire discussion with three bullet points:

• You get what you pay for.
• Entropy happens.
• Geography matters.

We've seen a lot of comments and discussion that fail to take into account one or more of these basics truths. Let's look at each point in detail.

We started this blog thread last week, when we only had two broken cables to consider, but since that time there have been reports of several more failures and they seem to keep coming in. As far as this thread is concerned, the first two parts (here and here) focused on the countries and local providers most impacted on the day of the first two cable failures. We then looked at the providers of some of the harder-hit countries and how they were able to restore connectivity (or not) during the subsequent 48 hours. And along the way, we felt obliged to counter some nonsense circulating on the Internet claiming that Iran had been cut off. It's been a busy week and we've barely scratched the surface. But plowing ahead, we will take an extended look at two local providers, Bharti in India and DCI in Iran, and how they weathered the storm. One week later, how are these two local providers gaining access to the global Internet? What has changed? We will use these examples to provide a glimpse into what can be discovered by collecting up enough public routing data from enough carefully selected places, combining it with geo-location information and then doing an enormous amount of processing.

Let me repeat, Iran is not disconnected from the Internet!

We have gotten a few queries about why we did not highlight Iran in our review of the network outages that resulted from the cable breaks. (See here, here and here.) Like most countries in the region, the outages in Iran were very significant, but for the most part they did not exceed 20% of their total number of networks. Now 20% is a significant loss, but in the context of an event where countries lost almost all of their connectivity, such a loss did not place Iran into the top 10 of impacted countries. So we focused most of our attention where the losses where the highest.

Our first two blog entries on this topic focused on the events of 30 January 2008, when two submarine cables systems were damaged. These systems provided much of the capacity into the Middle East and the Indian subcontinent from the west. Although some countries were hurt more than others, the loss of connectivity was extensive and very widespread. Some countries and a few providers were almost completely knocked off the Internet. As Day 1 came to a close, it was clear that the damaged cables were not going to be repaired anytime soon and the impacted parties would have to look for alternatives to waiting it out.

Day 2 and 3 saw a frenzy of activity as local providers in the region tried to broker agreements with anyone who still had capacity. They were under intense pressure to restore service to local governments and businesses. In turn, global and regional providers with surviving capacity into the region were busy hunting for new customers. We definitely had a seller's market. At Renesys, we watched all of the activity with great interest and decided to wait until the end of Day 3 to report on the winners and losers, after the initial deals were made and things had settled down to some degree.