American Pie was previously parodied at a RIPE meeting and now is practically a tradition, much to Mike Hughes's chagrin, as he thinks it's overdone already. The great thing about the original song is that it's choc full of references in the music industry. I tried to pepper several more into my version (and I have a few additional verses in progress that I just didn't finish).

What links would you provide to these references? What additional references do you think are important and missing (given the history of the Internet theme)?

The Day the YouTube Died

A long long time ago
I can still remember how the videos used to make me smile.
And I knew if I had my chance,
I'd watch the prison thriller dance
and maybe I'd be happy for a while.
But February made me shiver with every packet I'd deliver
bad routes in the tables, the paths they were not stable.
I can't remember if I cried when I saw my request was denied
but boredom welled up deep inside
the day the YouTube died.

Chorus:

Bye bye ye old YouTube of mine
Sent my packet to the prefix but then I was denied
Them old mullahs were hiding filth from our eyes
Singing this'll be the day YouTube dies!
This'll be the day YouTube dies.

Did you write the streaming code
And do you think videos should load
If the prefix takes you there?
Now do you believe in full control
When you end up at a rickroll
Man that guy sure had great hair

Well I know you loved that silly site
Cause you posted crap day and night.
Just leave Britney alone
Until she's fully grown.

I was a lonely late-night emo star
Singing offkey, strumming guitar
But my demise came from afar
The day the YouTubes died.

I started singing...

Chorus

Now for 20 years we've been in these tubes
Clogged with lolcats and mostly boobs
But that's not how it used to be.
When Verizon was still ALTERnet
And their peering was something you could get
On the network that we called FDDI.

Oh and while Genuity's stock was down
The Three stole their dubious crown.
AS1 was shuttered
The tables all got cluttered.
And while CDNs copied Akamai
The court cases they flew on by
And prices rose up to the sky
The day the YouTubes died.

We were singing...

Chorus

Napster drove serious traffic
Tunes for the younger demographic
But then one day it went away.

I went down to the Comcast store
Where I'd streamed the downloads years before
But the kids there, said BitTorrent wouldn't play

And on the net the Google rules
We're just eyeballs for AdSense tools
They know when we've clicked it
My Adblock Plus just nixed it.

I tried to squeeze bucks from NANOG
But y'all are broke so I must blog
and now I'm stuck with Babbledog
Ever since the YouTube died...

While this is good for the Internet, Cogent claimed that this dispute was about capacity issues and no one orders and installs new high capacity circuits in a week, especially during a contract dispute. So if there was a capacity issue, there is still a capacity issue. As a result, the situation is bound to be very fluid for the next few weeks. We'll update this blog as we analyze the resulting shifts in routing.

• At the end of the day on March 13th, the peering link between Cogent and Telia went away, cutting off some of their respective customers (those without other providers) from each other.
• Cogent took responsibility for pulling the plug.
• Despite this, Telia traffic found its way to Cogent via Verizon for 12 hours after the initial event.
• Something then happened to cut off this alternate route.

Why did the Verizon routes disappear?

So what exactly was that "something"? It is difficult to say with certainty, but we have only three choices:

• Cogent stopped accepting routes to Telia via Verizon.
• Telia stopped accepting routes to Cogent via Verizon.
• Verizon stopped transiting traffic between the two estranged parties.

In our last blog, we gave reasons why we thought the first option might be true. After sifting through more data, we will present possible reasons for the other two options, either of which may be more likely than the first. We'll let you decide, but first we're going to have to digress a bit and provide some background for those less familiar with the workings of Internet routing.

Background

While it's an overused term, Tier-1 providers are thought of as those organizations like Sprint and AT&T who are at the top of the Internet food chain. They pay no one for service — people pay them directly or pay their customers. The only reason all customers of Sprint can reach all customers of AT&T, for example, is because both companies agree to exchange traffic with one another without charging. At a routing level, Sprint tells AT&T which networks it is ultimately responsible for so that AT&T will send traffic for those networks to Sprint. AT&T does likewise for Sprint. Such an arrangement is known as peering, and since no money changes hands it is more preciously known as settlement-free peering. There are only about nine Tier-1 providers, and for everything to work, they all need to peer with one another. They also have no economic interest in letting anyone else join the club. But of course that doesn't stop others from trying to force their way in, and there are certainly marketing advantages to claiming you are at the top of the heap.

Cogent and Telia have long been Tier-1 wannabes. To attain this status, it helps if you have a global network and lots of customers and peers yourself. Then you can start to play the existing Tier-1s off one another for your traffic, and with a lot of effort and good fortune, you might get settlement-free peering with some of them. At worst, you might have to pay for peering from those with whom you don't have enough leverage. From a routing perspective, paid peering looks exactly like settlement-free peering. At Renesys, our Market Intelligence offering denotes both relationships as simply "peering", since we have no way of knowing if money is changing hands.

Analysis

For a long time, we have watched Cogent and Telia get closer and closer to Tier-1 status, at least as far as looking like they have no providers, whether or not they pay for some of their peerings. Cogent is almost there, but is still a customer of NTT (AS 2914) for the sole purpose of reaching AOL (AS 1668). (Interestingly enough, Cogent could become Tier-1 by default before long, as AOL continues to shrink and sell off its network.) As of a few weeks ago, Telia also looked to have one provider (namely Verizon) for reaching certain networks. But we stopped seeing evidence of that on February 27th. That is, after this date, Telia had no known provider to reach any markets anywhere, using only their own network or links routed as peering (paid or not) to reach everyone.

Then Cogent de-peered Telia and suddenly Verizon and others started providing a path between the two and their respective customers. At the time of my last blog, I was still thinking Verizon was a provider for Telia, so I was not surprised. (The other providers we saw were the result of leaks (mistakes) or because of customer dual homing.) But it would make no sense for Verizon to suddenly go away if they were still a transit provider for Telia, hence my conjecture about Cogent stopping this path. As mentioned above, there are two other possibilities, so let's now examine these.

Suppose Telia really did stop being a customer of Verizon a few weeks ago and got settlement-free peering. Verizon's routing announcements might not have fully reflected that change in their business relationship. That is, perhaps Verizon was announcing more than their customer routes to Telia, routes that might only show up if Telia lost access to parts of the Internet via other means. And so when the Telia-Cogent link was severed, Verizon inadvertently started providing free transit to Telia to reach Cogent. If true, it would have been in Verizon's interest to correct this mistake as quickly as possible. They would have no reason to connect Telia to Cogent without getting paid by either party. This scenario seems very unlikely to us as it would involve a substantial misconfiguration on the part of Verizon of a magnitude we have not seen in the past.

Finally, suppose that Telia has paid peering with Verizon and so Verizon still views them as a customer and perhaps is sending them more routes than they would a settlement-free peer. Again, such routes might have only become visible when the Telia-Cogent link went away and Telia started using them instead. If Telia has paid peering with Verizon, then they have to pay according to the amount of traffic they send them. Telia might not be very interested in paying for something (access to Cogent) that was formerly free. Such a situation could have inspired Telia to block this path themselves.

Conclusions

But no matter what happened here at a technical level and what the corresponding motivations were, the Internet has been partitioned for more than one week after the start of this spat. This seems quite odd to people and I keep getting asked "OK, but why isn't this a bigger deal? Give me just one juicy example, e.g., Norway can't reach CNN or something like that." Unfortunately and fortunately, I don't have any such juicy examples. Unfortunately, since if there were such highly visible examples, there would probably be enough external pressure on the warring parties to bring about a quick peace agreement. Fortunately, since this partitioning is not impacting most of us.

But for those of you who have asked for concrete examples, here is one. Martha Stewart Living is single-homed behind Cogent and announces one network, namely, 38.96.143.0/24. If you go to Telia's looking glass as of the time of this posting, you cannot get to Martha's network. As far as Telia is concerned, Martha doesn't exist. Does this mean that the Swedes are deprived of the pleasure of buying Martha's wares and sending her email? Not at all. Her web site is hosted by Savvis and a customer service email address points to AOL. But if Martha's parole officer allows her to visit Scandinavia any time soon, she won't be able to reach her corporate network. (By the way, I found this network by looking up all the single-homed customers of Cogent in Renesys' Market Intelligence.) This is not to make light of the situation, but to point out that the disruptions are largely personal in nature. And while this rift in the Internet is seriously impacting a wide range of individuals, it's unlikely to be resolved anytime soon without a lot more yelling and screaming.

It also points out the dangers of being a Tier-1 provider or a near Tier-1. Sure you can claim you've made it to the big time and don't need to buy from anyone, but you do need cooperation from the rest of the cartel and not one of them is going to do you any favors if you are perceived to be in a position of weakness relative to them. After all, this is all about money, and outside of the US Federal Reserve, no one else seems to be giving that away.

So let's look at the facts. On 13 March 2008 at 22:03 UTC, we saw the link between Cogent (AS 174) and Telia (AS 1299) disappear. The way this looks in the routing announcements is that all AS paths with "174 1299" or "1299 174" on them vanished, meaning there is no longer a physical link between these two providers. At this point, Telia lost direct access to 4474 prefixes (networks) transited via Cogent, whereas, Cogent lost 1633 Telia networks. Now, you might be thinking, "So what? Won't the Internet just route around the problem, finding alternate paths?" Well, yes if there are alternate paths to be found and if the players actually allow traffic to flow via them. For around 12 hours, most Telia customers did access Cogent via Verizon (56% of the 4474 networks), Level 3 (16%), AT&T (6%) and others, but then that abruptly stopped. We're guessing it's because Cogent eventually slammed the door shut on these alternate paths to their network from Telia, since none of Cogent's customers accessed Telia via alternate routes during this time. Like divorce court, depeering is supposed to be painful, otherwise you might not get what you want. You only hurt the ones you love.

So now we have a bit of a problem. Customers who ultimately rely solely on Cogent for transit cannot get to Telia, likewise for customers downstream of Telia. So where are these customers exactly? The following tables show the geo-location of those networks downstream of Cogent that cannot reach Telia and vice versa for Telia.

Telia cannot reach Cogent

Country	# Prefixes
US	1868
Canada	232
France	98
Spain	41
Germany	31
UK	27
Others	86

Cogent cannot reach Telia

Country	# Prefixes
Sweden	444
Finland	322
Russia	153
Poland	113
US	73
Latvia	62
Bulgaria	52
Spain	40
Denmark	35
Norway	30
Others	249

Given the markets Cogent and Telia operate in, these lists are not too surprising. What is surprising is that networks in the US are actually cut off from each another, since a largely US provider is playing hardball with a largely European one. For the Internet to be whole again, Cogent and Telia need to kiss and make up. No one can force either one to carry traffic destined for the other. But my guess is that Telia is hearing more grief from Scandinavian customers not being able to reach US content than Cogent is hearing from US customers cut off from Northern Europe.

Of course, the list of impacted networks is too long to be included here, but they include a wide range of commercial, educational and government clients. On the Telia side, the victims include the Swedish Defense Data Agency, the Finnish State Computer Center, and broadband customers in St. Petersburg. With regard to Cogent, Blue Cross and Blue Shield of Delaware, Kansas State University and Reuters America were all collateral damage.

But people can and do de-peer all the time for business reasons without blowing holes in the Internet along the way. Early on 14 March 2008, Flag (AS 15412) and SingTel (AS 7473) parted ways, but almost all of the few thousand networks carried between the two managed to find alternate paths via their peers or providers. That's because neither tried to "stick it" to the other and allowed Internet routing to do what it does best, find another way to get from here to there.

Just before 18:48 UTC, Pakistan Telecom, in response to government order to block access to YouTube (see news item) started advertising a route for 208.65.153.0/24 to its provider, PCCW (AS 3491). For those unfamiliar with BGP, this is a more specific route than the ones used by YouTube (208.65.152.0/22), and therefore most routers would choose to send traffic to Pakistan Telecom for this slice of YouTube's network.

The data points identified below are culled from over 250 peering sessions with 170 unique ASNs. While it is hard to describe exactly how widely this hijacked prefix was seen, we estimate that it was seen by a bit more than two-thirds of the Internet.

This table shows the timing of the event and how quickly the route propagated (this is actually a fairly normal propagation pattern). The ASNs seeing the prefix were mostly transit ASNs below, so this means that these routes were distributed broadly across the Internet. Almost all of the default free zone (DFZ) carried the hijacked route at least briefly.

18:47:00	uninterrupted videos of exploding jello
18:47:45	first evidence of hijacked route propagating in Asia, AS path 3491 17557
18:48:00	several big trans-Pacific providers carrying hijacked route (9 ASNs)
18:48:30	several DFZ providers now carrying the bad route (and 47 ASNs)
18:49:00	most of the DFZ now carrying the bad route (and 93 ASNs)
18:49:30	all providers who will carry the hijacked route have it (total 97 ASNs)
20:07:25	YouTube, AS 36561 advertises the /24 that has been hijacked to its providers
20:07:30	several DFZ providers stop carrying the erroneous route
20:08:00	many downstream providers also drop the bad route
20:08:30	and a total of 40 some-odd providers have stopped using the hijacked route
20:18:43	and now, two more specific /25 routes are first seen from 36561
20:19:37	25 more providers prefer the /25 routes from 36561
20:28:12	peers of 36561 start seeing the routes that were advertised to transit at 20:07
20:50:59	evidence of attempted prepending, AS path was 3491 17557 17557
20:59:39	hijacked prefix is withdrawn by 3491, who disconnect 17557
21:00:00	the world rejoices; Leeroy Jenkins online again.

Since BGP relies on a transitive trust model, validation between customer and provider is important. In this case, PCCW (3491) did not validate Pakistan Telecom's (17557) advertisement for 208.65.153.0/24. By accepting this advertisement and readvertising to its peers and providers PCCW was propagating the wrong route. Those who saw this route from PCCW selected it since it was a more specific route. YouTube was advertising 208.65.152.0/22 before the event started and the /24 was a smaller (and more specific) advertisement. According to usual BGP route selection process, the /24 was then chosen, effectively completing the hijack.

Because of the fast detection and reaction of the YouTube staff and cooperation with other providers, service for their (sub-) prefix was interrupted for about an hour and forty minutes for some lucky customers and, at most, a bit more than two hours. The exact duration of the outage depends on your vantage point on the Internet.

When these sorts of events occur, there is renewed interest in a variety of solutions to this problem. BGP is fundamental to provider relationships and will not be going away anytime soon. Cryptographic extensions to BGP have been suggested (Pretty Good BGP, Secure Origin BGP and SBGP). These may be too taxing for router CPUs. Of course, after any sort of hijacking event (whether inadvertent or malicious) prefix and AS monitoring is suggested (e.g., the Internet Alert Registry, the Prefix Hijack Alert System, RIPE's MyASN and Renesys' Routing Intelligence).

Ultimately, though, the problem remains one of transitive trust. A provider can and should limit the advertisements it will accept from a customer. The mechanics can be arranged manually or can be configured using Routing Policy Specification Language (RPSL) to communicate the policy and drive configuration. In the case of Pakistan Telecom, they originate or transit fewer than 1000 prefixes.

So, it's heartwarming to know that two things are still true. It is still trivially possible to hijack prefixes (whether maliciously or inadvertently). I can go to sleep knowing that my neighbors are happily watching their LOLCATS.

Ditched my #@!%$! cell in Stockholm. Verizon CDMA does not work in Europe! Upside: I now have a shiny, new World Edition Blackberry GSM/CDMA. I call it Trixie.

Road Tip: Just say NO! to mouth-searing kimchi or Indian curry for breakfast. No matter how polite you're trying to be.

With barely enough time to recharge Trixie after calls in Denver, Albuquerque, Stockholm and Bonn, I hopped a jet for Tokyo, Hong Kong and Kuala Lumpur. (Trixie and I barely made it out of KL alive. Cab driver must have been conserving gas; tried to piggyback car in front.) Beginning to feel like Marco Polo on 'roids, but I gotta check out LA and DC before catching a shuttle back home to Boston (close enough) .

Trixie is overloaded with commentary, observations, insider scoops, and . . . new NSP sales and marketing contacts! (Hey, I'm a sales guy.) Time to download and see what comes out . . .

Action out of Africa. Early NSP missionary work (infrastructure, key relationships, etc.) is starting to make converts, mostly along the tips of the continent. South Africa and the balmy Mediterranean shore are lapping up most of the attention.

Road Tip: Need to track a package from anywhere to anywhere? Don't use anyone but FedEx. (They didn't pay me to say this.)

Middle East incumbent NSPs (grandfathered from the good old government-run land line days) are eager to establish Internet transit hub for traffic between Europe and Asia. They're investing heavily in technology and talent.

Construction cranes dot the dunes of Dubai.

Last year's Taiwan fiber outages caused by the Boxing Day (I grew up in England) earthquake of 2006 shook up the IP transit market. Back-up routing out of Asia is now all the rage and new undersea cables are planned. Russian comrades pushing a land-based fiber path across Asia can't take orders fast enough. Margins are enviable.

Road Tip: Re Asian hotel Internet connections. VoIP operational only in Japan.

Localization, localization, localization. Watch it grow — if you can watch that fast. Seems like just a year or two ago (a half-life in Internet years) most Internet content came from the US. Now, language-based contours are increasing exponentially. It's like the Wild West (or East) out there!

My finely-tuned sales guy antenna is vibrating: Japan now generates 50% of its own content, China 24%, and 5% for Malaysia! Spain and Latin America have their own action. Then there's Poland. Who needs Google and MySpace? They've got their own.

Nightlife in Tokyo's Roppongi district.

Now that Asia's generating so much of its own content, major home-grown NSPs expect to have very different (read: smaller) relationships with the current Tier-1 NSPs. If China Telecom's relationship with Sprint should, shall we say, dwindle, Sprint could be knocked down a peg or two in Renesys' Global Customer Base index.

Scuttlebutt, afterthoughts, noise.

Want to make a profit? Then forget about selling Internet transit to Wal-Mart . . . I mean, Google.

Take a walk on the wild wide, otherwise known as Internet transit peering. Relationships are often sealed with the equivalent of a handshake in a back room. Not so with their not-so-distant relatives in the more orthodox voice market. They negotiate contracts! On paper! At the likes of Intelsat GTM in Washington, DC each spring. I'm not saying that people don't peer closely at their peering ratios; they do. And make important decisions based on them, too. It's just kind of like, oh, living together instead of getting married.

Internet transit pricing varies. Duh. Here's why: it depends on local competition of course, but also on just plain geography. It's priciest in Australia and the Middle East. One exec told me that they changed plans to locate their Asian office in Sydney, and decided to go with Hong Kong — based solely on projected cost of Internet transit in Sydney! Sure didn't seem cheap to me last time I was in Hong Kong. Sydney pricing must be surreal.

Can't help but notice this correlation: NSP growth equals (or surpasses) how much they've outsourced. Reminds me of the light and nimble British Navy taking pot shots at a lumbering Spanish Armada. Large NSPs lug around large organizations. They don't respond to change and opportunities as quickly as leaner organizations that outsourced everything but their crack sales staff after the dot com crash. Disclaimer: Exclude from preceding characterization all nimble, responsive NSPs who subscribe to Renesys Market Intelligence. They're doing great. (Hey, I'm a sales guy.)

Contact me at bobf@renesys.com to chat (or buy licenses :-)

I'll discuss these questions in what follows, but let me warn you in advance. There is nothing earth-shattering here. In fact, I can save you time and sum up the entire discussion with three bullet points:

• You get what you pay for.
• Entropy happens.
• Geography matters.

We've seen a lot of comments and discussion that fail to take into account one or more of these basics truths. Let's look at each point in detail.

The segment of humanity that relies on the Internet can be roughly broken into two camps: Internet Consumers and Internet Providers. Internet Consumers are individuals, governments or businesses who use the Internet for their own purposes. Internet Providers are in the business of providing Internet access to Internet Consumers. While it is possible to be in both camps, most groups are largely in one or the other. Each has different lessons to learn.

To protect themselves, Internet Consumers need to do at most three things.

1. Determine the importance of their connectivity.
2. Make the business case for change, if it is warranted.
3. Become an informed consumer and buy appropriately.

Can your business survive a complete loss of connectivity or severely degraded connectivity for a day, a week, or a month? If your answer is "no" for anything less than a month, you need to seriously consider your disaster plans. Undersea cables can easily take a month to repair. And in times of distress, there may be many more organizations (with deeper pockets) in line ahead of you. You need more than multiple providers or backup lines. You need to concern yourself with their physical independence. In other words, having 3 or 10 providers is of no help if they all use the same submarine cable. Don't think of the expense of multiple providers as insurance against an unlikely event, such as your house catching on fire. Your house almost certainly is not going to burn to the ground. However, the Internet cables your business runs on are going to fail; it is only a matter of when and to what degree. Figure out what reliable, fault-tolerant connectivity is worth to you and then buy what makes sense. No one else can do this for you.

To sell to informed Internet Consumers and stay competitive, Internet Providers need to do at least three things.

1. Plan for disasters and have a response plan in place.
2. Build redundancy.
3. Sell your great network to your customers.

When planning for disasters, consider your response to individual cable breaks. If you plan to just "wait it out", consider what losing all your customers in a region will mean to your bottom line. They won't be waiting for the repair ships to arrive. As we saw with the latest round of cable cuts, reaction time is everything in gaining new business. Your former loyal customers will be gone in a heartbeat if you can't deliver in an emergency. Of course, in order to have any contingency plan at all, you need some redundancy in your network, which then gives you a great sales and marketing tool, especially after events like these. Remind your potential customers of the dangers of a focus on price alone. Even if they don't buy from you today, they will still be around when the next cable breaks. Of course, to be effective and stay one step ahead of your competitors, you need up-to-the-minute Internet Intelligence which is where Renesys comes in. It is of no surprise to us to see our customers doing well both during and after such outages.

Entropy happens.

While I'll admit that multiple cable breaks in a week does seem somewhat unlikely, I learned long ago to never ascribe to malice that which can be explained by incompetence or simply chance. Given the laws of nature and the fact that there are over 6 billion of us bumbling along, I can't say it is all that surprising either. Individual cable outages occur all the time and seldom get much attention - the sea floor is not the most hospitable environment for fiber optics. The Economist reports that there were over 50 repair operations in the Atlantic last year alone. Wired claims an average of one cable cut every three days. What these outages teach us is that chance events do not have to be uniformly distributed by geography or by time. And the only reason we even noticed these is because of the extreme lack of redundancy in the region.

Geography matters.

As does weather, economics, politics and other annoyances of a non-virtual world. You can see these forces in action on cable maps. If you look at the Middle East in particular, where would you lay the cables?

• West? That takes you through the Sahara.
• South? It's a long way around the horn of Africa.
• East? That's the really long way to reach your business partners in Europe.
• North? Better, but avoid the war zones.

The point is simply that your options are going to be influenced by where you live.

I don't mean to imply that any of the problems we've seen are unsolvable. If Internet Consumers demand cheap access above all else, they will get it and, as a consequence, it will be unreliable. If they demand reliable access above all else and are willing to pay for it, the Internet Providers will be more than happy to accommodate them. The ones who don't won't be around for long. And in the overall scheme of things, we are not talking about a lot of money. A mere $125M USD gets you a brand new cable from Egypt to France, but only about one third of a plane. Internet consumers, which include governments, need to decide which is more useful.

With this background, let's take a closer look at Bharti and the major carriers who connect them to the rest of the global Internet. Before the cable cuts, Bharti was receiving service from several carriers including British Telecom (BT), Deutsche Telekom (DTAG), Cable & Wireless (C&W) and Sprint. We observed these four particular carriers until the cable breaks, and then each of these simply went away. Only Sprint eventually recovered to some degree on 2 February, but ended up carrying far fewer networks. It is not surprising that certain carriers went completely off-line, but why did Sprint come back after two days? No cables were repaired during that time and no new ones were suddenly brought into service.

Sprint has a strong global network and has considerable capacity heading from Asia to the west coast of the US. If their outage could have been corrected by a configuration change, you would think that that would not have taken two days. Are they selling service in India on routers without capacity in both directions? Were they preferring their more expensive MPLS service over IP and had no available bandwidth for IP? If so, what happened after two days to restore IP service? Looking at Sprint's network maps (see page 9), they claim to have capacity on SEA-ME-WEA 3, which was not impacted by the outages. What exactly was Sprint doing for those two days in India?

As for the providers who gained new traffic, AT&T, SingTel and Level 3 initially picked up new networks from Bharti. However, all of them subsequently fell, perhaps due to another cable cut, with only Level 3 managing to preserve some of their gains. This answers one of our questions from an earlier blog about exactly how Level 3 managed to gain business in India. It was due almost entirely to Bharti, a very large local provider.

Now, let's consider DCI in Iran. DCI is the only provider in Iran with connections to the outside world. Most of their traffic flowed via TTNet, SingTel or Flag before the breaks, and not surprisingly, Flag lost many of the networks it carried earlier.

But this graph might seem to contradict our previous blog, where we said that the outaged networks in Iran had been quickly recovered. The graph shows a drop in networks carried by Flag, but no corresponding rise in networks for TTNet and/or SingTel. This is explained by the fact that networks can be carried by more than one provider. For example, I might reach a network in Iran via Flag, but you might reach that very same network via TTNet. This is why Iran was able to recover so quickly. DCI could use any one of their three primary providers and, in fact, were using more than one of them for many of their networks. When Flag failed, traffic could easily move to one of the surviving providers. So although total bandwidth into the country was reduced, there was little in the way of a long-term outage for many networks.

From this discussion, we can see that graphing the number of networks over time does always tell the whole story. Here SingTel and TTNet both could have picked up a lot of new traffic because of the failure of Flag, but not necessarily any new networks. How can we observe such situations in the routing data? Well, when Iran had three main providers, the rest of the world would pick these three in some proportion based on various routing attributes, which are beyond the scope of this discussion. However, when Flag went away, there were only two primary Iranian providers left standing. Renesys' worldwide assortment of routing peers (i.e., data collection points) would have been forced to pick one of the two survivors for traffic into Iran. We can capture this with a metric we call PPT, peer-prefix-time. Basically, for each of the Renesys peers worldwide, we count up the total amount of time this peer routes a network (prefix) in a particular way. Thus, for each network in Iran and each peer, we'll know how long it was routed via TTNet or Flag or SingTel. Adding up these times for all networks in Iran tells us how popular these providers are for gaining access to Iran on any given day. We show this in the graph below.

Immediately after the cable cuts, TTNet was preferred for access to Iran by a significant majority of the world over SingTel. But then as time went on, the two providers achieved rough parity. This could have been because of routing decisions made by DCI to balance traffic between their remaining providers. This example is to show that simply counting up routed networks, while useful, only gets you so far. You also need to know how the rest of the world chooses between the available options and in what proportion. Flag, which still routed networks to Iran after the cuts, was selected by almost no one.

I want to thank you for getting to this point in my blogs and for all the thoughtful comments I have been receiving both publicly and privately. I wish I had the time to answer all the questions, but I guess if I did we wouldn't have much of a business. Renesys makes its money by selling such Internet Intelligence to its customers. So with this blog, I am going to close out our discussion of cable breaks for now, except that I'll soon follow up with some non-technical concluding remarks and lessons learned.

We have gotten a few queries about why we did not highlight Iran in our review of the network outages that resulted from the cable breaks. (See here, here and here.) Like most countries in the region, the outages in Iran were very significant, but for the most part they did not exceed 20% of their total number of networks. Now 20% is a significant loss, but in the context of an event where countries lost almost all of their connectivity, such a loss did not place Iran into the top 10 of impacted countries. So we focused most of our attention where the losses where the highest.

The Slashdot claim was made since a web page at the Internet Traffic Report was reporting that the country was down. This report seems to be based on pings to a single router in Iran from multiple places around the world, which at best only indicates that one router in Iran is unavailable, not that the entire Internet has ceased to function there. Of course, once something ends up "in print", it tends to gain credibility and then be referenced by others. And before long, large numbers of people think it is actually true. (For a detailed ping analysis to the region during the outage, see this article.)

To understand what happened in Iran after the fiber cuts, we looked at actual routing data for the country, collected from around the globe. You can say with absolute certainty that if a provider does not have a route to any network in Iran, then no traffic will flow from that provider or its customers to Iran. But that is all you can say. The problem could be with the provider. That is why Renesys collects routes from a carefully selected set of peers around the world. If none of them know how to get to Iran, then you can be assured that Iran is truly off the air. Note that you have to be careful here with your selection of peers. If all of them end up traversing the same cable to get to Iran, even when other options exist, then the problem could be only with that cable and nothing more. To make a definitive statement about the worldwide reachability of any geography, you need to collect data from a diverse and at least somewhat independent set of peers so that you'll see all paths into the area. When the overwhelming majority of them have the same view of a situation, then you can conclude that the view is almost certainly correct for the entire world.

So back to Iran. In the following graph, we plotted the availability of Iranian networks for four entire days, 30 January 00:00 UTC until 3 February 00:00 UTC. The first day is the day of the cable cuts. Of the 695 networks that geo-locate to Iran, at no time were more than 199 unavailable, as observed by large number of Renesys peers. A few peers here and there might not have been able to reach Iran for local reasons, but the vast majority of the world could get to most of the networks in Iran for this entire time period. Note also that around 64 networks were unavailable before the event even started. These networks could be simply unused at this time. In other words, at most 135 networks that were active before the cable cuts disappeared for at least a short while during the outages.

Global Reachability of Iranian Networks

So much for Iran being off the Internet. Again, this is not to imply that Iran was not impacted by this event. A lot of networks were unavailable and some of them continue to be so. The end users of those networks are certainly noticing the problem and everyone in the country might be experiencing a slowdown due to the decrease in bandwidth to the region. Still, Iran fared much better than most.

Day 2 and 3 saw a frenzy of activity as local providers in the region tried to broker agreements with anyone who still had capacity. They were under intense pressure to restore service to local governments and businesses. In turn, global and regional providers with surviving capacity into the region were busy hunting for new customers. We definitely had a seller's market. At Renesys, we watched all of the activity with great interest and decided to wait until the end of Day 3 to report on the winners and losers, after the initial deals were made and things had settled down to some degree.

We start off by looking at Egypt, one of the harder-hit countries. At the end of Day 3, the big winner was Telecom Italia, which clearly picked up a lot of new business, restoring service to almost 300 more networks than they had originally lost. France Telecom was the big loser, with no down networks restored. Flag, who maintains one of the broken cables, managed to restore service to a few hundred networks, presumably by sending that traffic east, rather than west. Although not shown here, VSNL seized the opportunity and entered the Egyptian market for the first time, acquiring a local customer.

The situation in Kuwait was even more interesting with Global Voicecom, Telecom Italia, Flag and Verizon unable to restore a single down network. VSNL did what the others could not and restored service for all of their networks, as well as gained some new business. They clearly exploited the fact that they have capacity to both the east and west from this region. PCCW was the big winner, managing to pick up almost 70 new networks. And when all else fails, there is always satellite. Horizon Satellite entered the market, providing service to four of the fallen.

Saudia Arabia saw AT&T, Flag and Sprint largely unable to act to restore service. The big winners here were Deutsche Telekom, gaining 13 networks, and VSNL, gaining 11.

We now turn our attention to the Indian subcontinent and the two big players, Pakistan and India. First up is Pakistan. The big loser here was Verizon, with over 700 networks that they failed to restore. BT was the big winner, picking up almost 500 networks. In a country with under 1400 routed networks, this was a huge gain. PCCW also did very well, gaining over 200 new networks, while Stixlite started servicing an additional 166 networks.

And finally, we look at the huge market of India. Sprint, Cable & Wireless, Deutsche Telekom, BT and Verizon all took it hard here, largely failing to restore connectivity to their networks. AT&T and Flag also lost a lot of networks. But the big winner was SingTel, gaining over 200 networks, followed by Level 3, picking up over 100. One wonders why AT&T, with lots of fiber assets in the continent, had a net loss among these networks whereas Level 3, with no fiber in the region, saw a net increase.

Having reviewed these five countries in detail, we wondered how many networks were still unavailable? That is, how much money was still on the table for those nimble providers with additional capacity? For any country, there will always be a small number of unavailable networks at any given time. These could be down for entirely local reasons: scheduled maintenance, power outages, etc. So we graphed the number of networks unreachable both before the cable cuts and at the end of Day 3. As you can see, there is still much to do. But keep in mind that even when reachability is eventually restored to all down networks, overall capacity to the region will be severely curtailed until those cables are repaired. And latencies could be very high if, for example, Europeans now need to reach the Middle East by way of the US and Asia. Of course, some connectivity is always better than none at all.

So what will happen next, when service is fully restored? Flag is claiming this work will be complete by mid-February, but there was yet another cable cut yesterday. Who will restore service to those networks still off the air? Who were the large customers that shifted service to other providers? What were the lessons learned by this event and what can be done to guard against future cable cuts? These are questions we will leave to subsequent blogs. Stay tuned.

The following two tables provide the top 15 providers with the largest number of outaged networks. We list the provider's name, the country in which most of their unreachable networks are located and their autonomous system number (ASN), an assigned number that uniquely identifies their organization on the Internet.

Not surprisingly, the hardest hit providers are located primarily in the hardest hit countries: Egypt, Kuwait, India and Pakistan. One local provider in each of Egypt and Kuwait lost essentially all of their Internet connectivity.

Provider	Country	ASN	%
Wataniya Telecom	Kuwait	29357	100
Yalla Online	Egypt	20484	99
Dancom	Pakistan	23966	80
EgyNet	Egypt	20858	77
Internet Egypt	Egypt	5536	77
Micronet	Pakistan	23674	73
Data Network	Pakistan	9260	71
Pakistan Telecom	Pakistan	17557	64
TEDATA	Egypt	8452	60
IDM	Lebanon	9051	56
Exatt	India	18231	53
Nile Online	Egypt	15475	49
Emirates Internet	UAE	5384	46
Eepad	Algeria	33783	43
QualityNet	Kuwait	9155	35

After totaling up the damage to the local providers, we wondered if any of harder hit ones managed to regain connectivity for some of their networks via alternate paths. We often hear statements like "the Internet is good at routing around damage". Well, that can be true, but only when there are available alternatives. Looking at hard-hit Egypt and Kuwait, we plotted the number of outaged networks per provider over the past day in the following stacked graph, where the width of each color represents the number of unreachable networks for a given provider. If any of the providers had choices, we would expect to see the width of their color decrease over time as they shifted traffic to alternative paths. Except in one case, that didn't happen. The exception was the Egyptian provider, LINKdotNET (ASN 24863), which did regain connectivity for most of their unavailable networks for about one hour at 20:00 UTC, only to lose more than twice as many after that time. Whatever backup routes they used obviously didn't hold up. The others had essentially the same number of networks out all day, the typical shape of a prolonged catastrophic failure.

To make this last point perhaps more forcefully, we next plot the total number of outaged networks for the region as a whole for 24 hours, excluding the Indian subcontinent. As shown, there was no immediate relief for the large swath of the Internet cut off by this disaster.

Next up, we'll look at the global Internet providers and who won and lost in the battle to retain their regional customers or acquire new ones.

Most Impacted Countries

As you can see from the above map, there are several cable systems that connect Europe, the Middle East and Asia, via the Suez Canal. The countries highlighted in red are those whose Internet connectivity is being disrupted the most by this event. At Renesys, we geo-locate all routed networks and observe their reachability from over 250 locations around the globe. In the case of disasters like this, we will suddenly see a large percentage and/or a large number of country-specific networks disappear from the Internet. As the following charts show, Egypt and Pakistan lost the highest percentage of their networks, while India lost the least. However, India had the third highest total number of networks disappear. Looking at the cable map, it is not surprising that the Indian subcontinent was impacted by events off the coast of Egypt. There are essentially two ways to get to this part of the world: via the Suez Canal or via Southeast Asia.

The next graph show the outages over time for the four countries who lost the most number of networks, namely, Egypt, Pakistan, Kuwait and India. You can observe a sharp loss of connectivity for these countries at 04:30 UTC, followed by another event at 08:00 UTC.

Most Impacted Countries by Total Number of Networks

Our final graph shows the total number of networks lost for the region, excluding the Indian subcontinent, in order to more clearly illustrate the timing of these events. Notice that there are two long term events starting at 04:30 UTC and 08:00 UTC, presumably the two cable breaks. Then there are shorter lived events at around 06:00 UTC and 13:00 UTC, which may reflect measures taken in an attempt to route around the problem.

Total Number of Outaged Regional Networks

Stay tuned to this blog for more information as we continue our analysis.

Since I sometimes find myself hopelessly lost, I tend to wonder about global navigation in the days before GPSes or even accurate maps. I imagine you started off with just a general idea of where you wanted to go (e.g., "The New World"), crude navigational aids (the stars, Sun and Moon when you could see them), and hearsay from your fellow travelers or the locals about your proposed course. In addition, you only had a view of the world from your current location, limited by the curvature of the earth.

Thinking about it, this sounds a lot like modern day routing. You have a general idea of where you want some packets to go (a network prefix) and very crude navigational tools (BGP updates), most of which are hearsay from peers, providers and random strangers you have no reason to trust. Plus your world view (routing table) is limited to the place where you collect your routes and from whom you collect them. There is no comprehensive map of the world (global routing table) or any reason to believe that your packets won't fall off the end of the earth (be blackholed) when you send them on their way. The lack of a global routing table is the reason Renesys has to collect routes from all over the world (over 250 places at present) and the reason we see around 330,000 routes, compared to what is currently considered a "full table" at just under 240,000 routes. Compared to modern navigation in the physical world, it seems like we would have a better way to route Internet traffic by now.

Single homed

Conveniently, Renesys is located in New Hampshire where real life navigation can remind you more of the ancient world than the modern one. (Maybe that's why we're good at Internet routing?) Our physical maps are often wrong, showing roads that no longer exist or landmarks in the wrong locations. Cell service comes and goes and doesn't even work at my house, in a neighborhood less than 2km from Dartmouth College. And it is often so cold that battery life is limited to minutes; so even if your GPS did work, it would be dead when you pulled it from your pack.

Happy not to have been blackholed

With this in mind, we ventured out on a Renesys team-building exercise on Sunday, 20 January 2008. We picked this day to go above tree line since the weather forecast was particularly nasty. A storm was coming and the highs for the day were supposed to drop to -26C on the peaks with wind speeds exceeding 100kph. Luckily, we got to the top ahead of the storm and the day was absolutely brilliant and comfortable: -22C with a cooling breeze of 50kph. Our routes were clearly visible and no one was blackholed - a very successful day of navigating the mountain.

In high risk activities under adverse conditions, it's not hard to make poor decisions that you would never contemplate from the comfort of your favorite living room chair. But while there is little risk to life and limb on the Internet, its very connectedness means that the blunders of pretty much anyone can impact you. What is important in this environment is the half-life and the reach of the mistakes. Those that are local and die out quickly have little chance of resulting in global mayhem. Others compound with all the other endless screw-ups regularly going on and eventually become a giant avalanche careening down hill, collecting mass and bearing down on the sleeping village below. This is one of those stories. It might be true or it might not. Your opinion depends on how much imagination you think we have!

Mistake #1: Ignore scaling issues and take the easy way out.

To compound this error, Hotel-A elects to take the defaults, updating all PCs at exactly the same time.

Mistake #2: Ignore scheduling issues. Who cares what happens at 3am anyway?

Hotel-A staff now have a ticking time bomb in place, set to go off on a particular day of the month. Every one of their PCs (probably thousands) will start to suck down a large and identical set of updates at that time. This will go on for many hours and effectively saturate all of their Internet links. ( Like this.) Sure enough, Hotel-A's self-DOS proceeds on schedule and their network operations staff identifies it as such, failing only to prepend "self" to the problem report. Network operations in turn calls their transit provider, Provider-A, another global carrier.

Mistake #3: Blame the Boogeymen on the Internet before looking in your own backyard.

We've all been in similar situations. Provider-A greatly values and implicitly trusts Hotel-A. They are a good customer and pay serious money for services. The time to act is now!

Mistake #4: Accept the reported problem at face value, rather than investigate yourself.

Now, how hard would it have been to notice that the source of the "attack" was a major CDN? Provider-A decides to blackhole all traffic from the source network. But they do more than that - they blackhole all traffic from this network to all of their customers, not just Hotel-A.

Mistake #5: Overreact in a time of crisis.

Next, Provider-A compounds the problem by announcing the black-holed network as their very own. That is, they start originating a network that in fact belongs to Provider-C.

Mistake #6: Carelessly inject your IGP routes into BGP.

The folks at Provider-C now start to get reports about the inaccessibility of the CDN they host. These folks are smart and run a tight ship. They carefully check out the complaints from various points on their worldwide network, but to no avail. Everything looks great from inside their network and from various external points as well. Provider-A ultimately figures out what they have done and withdraws the route, leaving Provider-C scratching their collective heads about exactly what happened, as their traffic suddenly returned to normal. They give Renesys a call to see if we have noticed "anything odd" about the problem network. One quick look shows Provider-A and Provider-C both originating the same network for a short period of time one nice summer morning. Given the reach of both carriers, the Renesys global peering set was roughly evenly divided between the two. So half the world had access to the CDN and half did not. The CDN and Provider-C were both collateral damage from a series of mistakes made by others.

You might think that such a sequence of events is highly unlikely and so probably didn't happen. Or you might think that we aren't clever enough to have actually made up such a story. Regardless, I happen to believe that cascading failures are common, although underreported. With billions of mistake prone humans connected to the global Internet, how could this not be the case?

Epilogue: In conclusion, there are really two important morals to take away from this story. First, it is fairly trivial to economically harm even the best run networks. Second, you can't effectively monitor your network from within your network. While it's true all the protocols on the Internet are from a happier day and are now seriously broken and in need of replacement, your only real alternative to awaiting for nirvana might be to have someone else watch your (high-value) back and to keep a list of those NOC phone numbers handy.