Renesys Blog http://www.renesys.com/blog/ en Copyright 2008 Tue, 15 Apr 2008 13:17:47 -0500 http://www.sixapart.com/movabletype/?v=3.2 http://blogs.law.harvard.edu/tech/rss The Day the Youtube Died: The Video Randy Epstein of Host.net and WVFiber graciously (or perhaps maliciously, given the quality of the performance) filmed and did the post-production on the recent performance at the Global Peering Forum. If I had a virtual tip jar, I would set it out. Enjoy:

]]> http://www.renesys.com/blog/2008/04/the_day_the_youtube_died_video.shtml http://www.renesys.com/blog/2008/04/the_day_the_youtube_died_video.shtml Society Tue, 15 Apr 2008 13:17:47 -0500 The Day the YouTube Died At the recent Global Peering Forum I performed a spoof song based on the recent YouTube hijacking. (I'm told that video will eventually be available, at which point I'm sure I'll have to go into hiding at an undisclosed location.)

American Pie was previously parodied at a RIPE meeting and now is practically a tradition, much to Mike Hughes's chagrin, as he thinks it's overdone already. The great thing about the original song is that it's choc full of references in the music industry. I tried to pepper several more into my version (and I have a few additional verses in progress that I just didn't finish).

What links would you provide to these references? What additional references do you think are important and missing (given the history of the Internet theme)?

The Day the YouTube Died


A long long time ago
I can still remember how the videos used to make me smile.
And I knew if I had my chance,
I'd watch the prison thriller dance
and maybe I'd be happy for a while.
But February made me shiver with every packet I'd deliver
bad routes in the tables, the paths they were not stable.
I can't remember if I cried when I saw my request was denied
but boredom welled up deep inside
the day the YouTube died.

]]> http://www.renesys.com/blog/2008/04/the_day_the_youtube_died_1.shtml http://www.renesys.com/blog/2008/04/the_day_the_youtube_died_1.shtml Society Mon, 14 Apr 2008 15:34:00 -0500 Telia and Cogent Kiss and Make Up On March 28th at 17:52 UTC, we saw the peering link between Telia and Cogent come back up. Recently, peering disputes, especially with Cogent, tend to be all about traffic ratios: as long as both parties send roughly the same amount of traffic to each other, life is good. But when the ratios get out of whack, someone's feelings get hurt (more specifically someone's business model is threatened). Before the de-peering, we would typically see Cogent using Telia to reach around 2700 networks (prefixes). Now that count has dropped to just about 1450 networks. On the other hand, Telia used to reach approximately 7000 networks via Cogent and that number has now increased to almost 8600. So was Cogent sending too much traffic to Telia before? Did Telia then do something to provoke Cogent to turn them off (like send a bill)? We'll never know definitively, but someone blinked and the Internet is now whole again.

While this is good for the Internet, Cogent claimed that this dispute was about capacity issues and no one orders and installs new high capacity circuits in a week, especially during a contract dispute. So if there was a capacity issue, there is still a capacity issue. As a result, the situation is bound to be very fluid for the next few weeks. We'll update this blog as we analyze the resulting shifts in routing.

]]> http://www.renesys.com/blog/2008/03/telia_and_cogent_kiss_and_make_1.shtml http://www.renesys.com/blog/2008/03/telia_and_cogent_kiss_and_make_1.shtml Engineering Fri, 28 Mar 2008 16:09:36 -0500 He said, she said: Cogent vs. Telia As in most lovers' quarrels, it is difficult to objectively evaluate the claims of the combatants. Naturally, we tend to side with the person we know best, as it's their viewpoint we hear most often and are inclined to be sympathetic towards. Both Cogent and Telia are claiming to be the aggrieved party in their peering dispute and are now making their case in the court of public opinion. We will almost certainly never know the details of their private business relationship, but we can make a few more inferences from the data. Let me state up front that, like many major ISPs, Telia and Cogent are customers of Renesys and we love them both equally. Everything we report in our blogs is based on objective analysis of our global data, independent of our own business relationships.

]]> http://www.renesys.com/blog/2008/03/he_said_she_said_cogent_vs_tel.shtml http://www.renesys.com/blog/2008/03/he_said_she_said_cogent_vs_tel.shtml Engineering Fri, 21 Mar 2008 11:10:31 -0500 You can't get there from here Cogent and Telia are having a lover's quarrel and, as a result, the Internet is partitioned. That means customers of Cogent and Telia cannot necessarily reach one another. This was not due to a configuration error or a physical cable break. This is the way the Internet works and sometimes doesn't work. If the businesses that run the show don't play nice with one another, their customers can pay the price of being cut off from parts of the 'net. At least when Pakistan mistakenly hijacked YouTube, the matter was sorted out in hours and did not require the cooperation of Pakistan. The Cogent/Telia tiff has been going on for 4 days now and only they can resolve their differences. The rest of the world can only hope for full connectivity to be restored.

]]> http://www.renesys.com/blog/2008/03/you_cant_get_there_from_here_1.shtml http://www.renesys.com/blog/2008/03/you_cant_get_there_from_here_1.shtml Engineering Mon, 17 Mar 2008 23:01:53 -0500 Pakistan hijacks YouTube Late in the (UTC) day on 24 February 2008, Pakistan Telecom (AS 17557) began advertising a small part of YouTube's (AS 36561) assigned network. This story is almost as old as BGP. Old hands will recognize this as, fundamentally, the same problem as the infamous AS 7007 from 1997, a more recent ConEd mistake of early 2006 and even TTNet's Christmas Eve gift 2004.

Just before 18:48 UTC, Pakistan Telecom, in response to government order to block access to YouTube (see news item) started advertising a route for 208.65.153.0/24 to its provider, PCCW (AS 3491). For those unfamiliar with BGP, this is a more specific route than the ones used by YouTube (208.65.152.0/22), and therefore most routers would choose to send traffic to Pakistan Telecom for this slice of YouTube's network.

]]> http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml Engineering Sun, 24 Feb 2008 19:50:21 -0500 On the road again: Diary of an itinerant Internet transit sales guy Bob, the sales guy.

Ditched my #@!%$! cell in Stockholm. Verizon CDMA does not work in Europe! Upside: I now have a shiny, new World Edition Blackberry GSM/CDMA. I call it Trixie.

Road Tip: Just say NO! to mouth-searing kimchi or Indian curry for breakfast. No matter how polite you're trying to be.

With barely enough time to recharge Trixie after calls in Denver, Albuquerque, Stockholm and Bonn, I hopped a jet for Tokyo, Hong Kong and Kuala Lumpur. (Trixie and I barely made it out of KL alive. Cab driver must have been conserving gas; tried to piggyback car in front.) Beginning to feel like Marco Polo on 'roids, but I gotta check out LA and DC before catching a shuttle back home to Boston (close enough) .

Trixie is overloaded with commentary, observations, insider scoops, and . . . new NSP sales and marketing contacts! (Hey, I'm a sales guy.) Time to download and see what comes out . . .

]]> http://www.renesys.com/blog/2008/02/on_the_road_again_diary_of_an_1.shtml http://www.renesys.com/blog/2008/02/on_the_road_again_diary_of_an_1.shtml Business Thu, 21 Feb 2008 07:45:45 -0500 Cable Breaks: Lessons Learned In the past 14 months, the world has seen two catastrophic failures of its global telecommunications systems: the Taiwan quakes, which snapped 7 of 9 important cables in Asia in December 2006, and a series of mishaps in the Mediterranean and the Gulf, damaging several others. In a world increasingly dependent on global trade and communications, what lessons can we learn from all of this and what measures should we take?

I'll discuss these questions in what follows, but let me warn you in advance. There is nothing earth-shattering here. In fact, I can save you time and sum up the entire discussion with three bullet points:

  • You get what you pay for.
  • Entropy happens.
  • Geography matters.

We've seen a lot of comments and discussion that fail to take into account one or more of these basics truths. Let's look at each point in detail.

]]> http://www.renesys.com/blog/2008/02/cable_breaks_lessons_learned_1.shtml http://www.renesys.com/blog/2008/02/cable_breaks_lessons_learned_1.shtml Business Mon, 11 Feb 2008 13:30:36 -0500 Mediterranean Cable Break - Part IV We started this blog thread last week, when we only had two broken cables to consider, but since that time there have been reports of several more failures and they seem to keep coming in. As far as this thread is concerned, the first two parts (here and here) focused on the countries and local providers most impacted on the day of the first two cable failures. We then looked at the providers of some of the harder-hit countries and how they were able to restore connectivity (or not) during the subsequent 48 hours. And along the way, we felt obliged to counter some nonsense circulating on the Internet claiming that Iran had been cut off. It's been a busy week and we've barely scratched the surface. But plowing ahead, we will take an extended look at two local providers, Bharti in India and DCI in Iran, and how they weathered the storm. One week later, how are these two local providers gaining access to the global Internet? What has changed? We will use these examples to provide a glimpse into what can be discovered by collecting up enough public routing data from enough carefully selected places, combining it with geo-location information and then doing an enormous amount of processing.

]]> http://www.renesys.com/blog/2008/02/mediterranean_cable_break_part_3.shtml http://www.renesys.com/blog/2008/02/mediterranean_cable_break_part_3.shtml Engineering Thu, 07 Feb 2008 14:58:00 -0500 ATTENTION: Iran is not disconnected! Let me repeat, Iran is not disconnected from the Internet!

We have gotten a few queries about why we did not highlight Iran in our review of the network outages that resulted from the cable breaks. (See here, here and here.) Like most countries in the region, the outages in Iran were very significant, but for the most part they did not exceed 20% of their total number of networks. Now 20% is a significant loss, but in the context of an event where countries lost almost all of their connectivity, such a loss did not place Iran into the top 10 of impacted countries. So we focused most of our attention where the losses where the highest.

]]> http://www.renesys.com/blog/2008/02/attention_iran_is_not_disconne_1.shtml http://www.renesys.com/blog/2008/02/attention_iran_is_not_disconne_1.shtml Engineering Sun, 03 Feb 2008 18:15:55 -0500 Mediterranean Cable Break - Part III Our first two blog entries on this topic focused on the events of 30 January 2008, when two submarine cables systems were damaged. These systems provided much of the capacity into the Middle East and the Indian subcontinent from the west. Although some countries were hurt more than others, the loss of connectivity was extensive and very widespread. Some countries and a few providers were almost completely knocked off the Internet. As Day 1 came to a close, it was clear that the damaged cables were not going to be repaired anytime soon and the impacted parties would have to look for alternatives to waiting it out.

Day 2 and 3 saw a frenzy of activity as local providers in the region tried to broker agreements with anyone who still had capacity. They were under intense pressure to restore service to local governments and businesses. In turn, global and regional providers with surviving capacity into the region were busy hunting for new customers. We definitely had a seller's market. At Renesys, we watched all of the activity with great interest and decided to wait until the end of Day 3 to report on the winners and losers, after the initial deals were made and things had settled down to some degree.

]]> http://www.renesys.com/blog/2008/02/mediterranean_cable_break_part.shtml http://www.renesys.com/blog/2008/02/mediterranean_cable_break_part.shtml Engineering Sat, 02 Feb 2008 06:17:24 -0500 Mediterranean Cable Break - Part II After looking at the countries most impacted by the cable cut in our first blog on this topic, we now turn our attention to the Internet service providers in the region and how they fared. Due to differences in network architecture, cable ownership, and transit purchasing, carriers in the same country may not all experience the same degree of outage. For all of the following, we consider a network to be "outaged" when it is unreachable from the perspective of the broader Internet—as represented by Renesys's 250 peering sessions.

The following two tables provide the top 15 providers with the largest number of outaged networks. We list the provider's name, the country in which most of their unreachable networks are located and their autonomous system number (ASN), an assigned number that uniquely identifies their organization on the Internet.

In the first table, we list the providers in decreasing order by total number of outaged networks. In the second table, we list them by decreasing order of the percentage of their networks that are unreachable.

Not surprisingly, the hardest hit providers are located primarily in the hardest hit countries: Egypt, Kuwait, India and Pakistan. One local provider in each of Egypt and Kuwait lost essentially all of their Internet connectivity.

]]> http://www.renesys.com/blog/2008/01/mediterranean_cable_break_part_1.shtml http://www.renesys.com/blog/2008/01/mediterranean_cable_break_part_1.shtml Engineering Thu, 31 Jan 2008 19:20:05 -0500 Mediterranean Cable Break Early this morning local time, two cable systems north of Alexandria, Egypt were severed, greatly impacting both Internet and voice traffic to the region. The broken cables are operated by Flag Telecom and SEA-ME-WEA 4, and if past undersea cable cuts are any measure, repair time will be measured in weeks, not days. This is a preliminary report on the countries most impacted by this failure, as seen from the perspective of Internet routing.

]]> http://www.renesys.com/blog/2008/01/mediterranean_cable_break.shtml http://www.renesys.com/blog/2008/01/mediterranean_cable_break.shtml Engineering Wed, 30 Jan 2008 18:53:22 -0500 15th Century Routing Which way is up?

Since I sometimes find myself hopelessly lost, I tend to wonder about global navigation in the days before GPSes or even accurate maps. I imagine you started off with just a general idea of where you wanted to go (e.g., "The New World"), crude navigational aids (the stars, Sun and Moon when you could see them), and hearsay from your fellow travelers or the locals about your proposed course. In addition, you only had a view of the world from your current location, limited by the curvature of the earth.

]]> http://www.renesys.com/blog/2008/01/15th_century_routing.shtml http://www.renesys.com/blog/2008/01/15th_century_routing.shtml Engineering Wed, 23 Jan 2008 14:14:05 -0500 Cascading Failures: Believe It or Not! Maybe it speaks to a risk-averse nature, but I've always been interested in failure and in learning from the mistakes of others - obviously so I don't have to learn such lessons first hand. This is particularly important when you engage in activities where bad decisions can kill you. But generally, as any book on mountaineering mishaps demonstrates, it takes a series of errors in the "correct order" and at the wrong times to cause you serious harm.

In high risk activities under adverse conditions, it's not hard to make poor decisions that you would never contemplate from the comfort of your favorite living room chair. But while there is little risk to life and limb on the Internet, its very connectedness means that the blunders of pretty much anyone can impact you. What is important in this environment is the half-life and the reach of the mistakes. Those that are local and die out quickly have little chance of resulting in global mayhem. Others compound with all the other endless screw-ups regularly going on and eventually become a giant avalanche careening down hill, collecting mass and bearing down on the sleeping village below. This is one of those stories. It might be true or it might not. Your opinion depends on how much imagination you think we have!

]]> http://www.renesys.com/blog/2007/09/cascading_failures_believe_it.shtml http://www.renesys.com/blog/2007/09/cascading_failures_believe_it.shtml Engineering Tue, 25 Sep 2007 17:00:00 -0500