Renesys Blog

Satellite Service Sets in Lebanon

2011-11-16T20:16:28Z

Early last month, my blog "Pinning Down Latency" included this prediction:

In the coming weeks we expect to see a dramatic shift in transit as Lebanese providers move away from expensive and high-latency satellite service to IMEWE-based service.

Well, it didn't take long for this to play out. ]]> Cyberia (AS24634) and TerraNet (AS39010) wasted no time dropping satellite provider SatGate (AS30721) as a means of primary Internet transit - colored green in the charts below. In these stacked charts, a taller band of color at a given point in time corresponds to a larger share of transit.

Looking at the graph on the right, IncoNet Data (AS9051) has greatly reduced their transit through SatGate (in green in the graph), but not completely eliminated it.

Virtual ISP (AS35197) which serves high-profile domains such as the website of the Lebanese President has shifted from exclusively using satellite via InSat (AS20535), in green, to exclusively using submarine cable via Liban Telecom (AS42020), in blue.

Finally, we observe the ongoing reduction in latencies to Lebanon due to the move away from satellite and onto the IMEWE submarine cable. This improvement has held steady since we first reported on it blog last month. In the visualization to the right, latencies into Lebanon separate into two modes based on delivery medium: asymmetric satellite (primarily via SatGate, AS30710, in green) and submarine (primarily via Level3, AS3356, in blue).

These dramatic changes in Lebanon mimic shifts we have observed in other parts of the world when submarine cables are activated in markets primarily served by satellite. As one of many examples, consider the situation in the landlocked country of Uganda in 2009. Prior to the arrival of the Seacom cable via an extension through Kenya, Uganda was exclusively dependent on satellite Internet. Once the cable arrived, incumbent Uganda Telecom (AS21491) abruptly shifted transit away from SkyVision (AS8513 and AS25228), Intelsat (AS22351), and Gilat Satcom (AS12491), and almost entirely onto Interoute (AS8928).

For Lebanon, capitalizing on inexpensive low-latency connectivity should pay dividends for an economy previously handicapped by the limitations of satellite. In what could be seen as a sign of such progress, current and former Lebanese prime ministers confronted each other over Twitter in recent days. More broadly the moral of the story is that for any market where satellite providers are still dominant, these providers are always one cable activation away from relegation to a back-up service — at best.

Cyber Attack in Palestine?

2011-11-01T23:08:52Z

We can confirm reports of significant but sporadic Internet outages in the Palestinian Territories today. As many as half of the routed networks of the Palestinian Territories were unreachable (withdrawn from the global routing table), possibly as a result of reported cyber attacks. These outages are the largest we have observed all year for this country, which normally has a fairly stable Internet. Impacted networks are located in both the West Bank and Gaza Strip.

]]> The vast majority of the 182 affected networks were customers of incumbent provider Paltel (AS12975). Fusion Services (AS42314) and Call U Communications (AS51440), which have independent international Internet transit, were also impacted. Additionally, some networks in Israel disappeared and recovered at the exact same times as Palestinian ones, suggesting problems with a common infrastructure.

Finally, we have observed a pattern of reduced connectivity in our traceroute data, which depicts a partial loss of connectivity via active measurements. In the graph below, we plot the number of completed traceroutes from our servers around the world to Paltel networks. Each color represent a different Paltel Internet provider used to reach local networks. The largest drop off occurred via Israeli provider Bezeqint (AS8551).

At the time of this writing, these networks have been restored. We will update this blog post if further major outages occur or as new information becomes available.

Large Outage in Pakistan

2011-10-27T00:02:07Z

For about 3.5 hours today, almost half of the Internet in Pakistan was down. Renesys observed 46% of the routed networks of Pakistan withdrawn from the global routing table between 19:37 and 23:02 UTC making this the largest Internet outage we have observed in the country in recent years.

]]> All 1253 networks involved in the outage were routed through Transworld Associates (AS38193), suggesting the outage was likely the result of a problem on the Transworld network serving Pakistan (TW1).

At the time of this writing, these networks have been restored.

Asia on My Mind and on the Move

2011-10-20T16:39:17Z

Undersea cables are expensive to install. But if you're an Asian Internet hub trying to connect to other Asian Internet hubs across un-cabled waters, what else can you do?

Well, one alternative we see is Internet Providers heading to California, as many Asian providers opt for Internet paths out of Asia to the west coast of the US, and then back to Asia. These tortuous routes, aptly called hair-pinning (observe their supple shapes), may be cost-effective initially, but generate latency, which can be a problem for some businesses (and their end users). ]]>
For more than a year, Renesys has been collecting daily latency data from 30+ locations to every prefix on the Internet. As you can imagine, we've learned a lot.

For example, here's how Australian carrier Vocus routes from Sydney to California and then back through Asia to reach India:

Who's Number One?

Such nimble routing schemes can produce surprising effects on our Market Intelligence analysis of Asia. Some US-based carriers with little infrastructure in Asia have higher Renesys Wholesale and Backbone Rankings because they have lots of Asian networks interconnecting via the USA. Level 3, for example, has a modest Asian infrastructure, but due to its Asia/US network connections, topped our Asian rankings even before they acquired Global Crossing.

Asian carriers are beefing up their intra-Asia networks to satisfy increasing demand for local Asian content (and a corresponding decrease in demand for US content.) Renesys is developing a model to determine the most influential NSPs in this intra-Asian market. However, since BGP routing tables don't tag routes by geography, it can be tricky distinguishing between wholesale business based on Asian customers within Asia, and Asian customers connecting via the US. So we adapted our Asian Retail Rankings algorithms to differentiate US and Asian wholesale NSPs. TelecomAsia recently published our initial rankings as the first quarterly insight into the intra-Asia market.

Jockeying for Position

Several distinctive characteristics excluded PCCW, one of the largest Asian networks, from our initial intra-Asian rankings. PCCW's success in Asia comes mostly from their backbone business rather than retail and wholesale endeavors. Their network is nearly settlement free (6% of their weighted routes go to their provider Global Crossing, while Pacnet routes 57% to their providers), which made it resemble large US Tier-1 providers - and why it didn't appear in our previous rankings. Rapid growth should catapult PCCW into Tier-1 status soon.

PCCW's backbone bias got us thinking about how Renesys evaluates Asian providers. We analyze three rankings to calculate total customer size: Retail, Wholesale and Backbone Rankings are correlated to approximate the corresponding business models of IP transit providers.

Depending on which ranking we examine, very different pictures of Asia emerge. For example, Chinese and Japanese NSPs with retail business divisions dominate Retail Rankings because their huge market size influences their Total Customer Ranking. US providers dominate the Backbone Ranking because they generally connect large regional or national incumbent providers' wholesale customers to each other and also to US content.

It's enlightening to throw a few extraordinary cases into the mix. China Telecom is by far the largest retail provider in the world (not just in Asia), but is ranked 4th in Asian wholesale and 26th (including non-Asian NSPs) in Asian backbone. NTT's international backbone network is ranked 24th in the Asian Retail index, 1st in the Wholesale Index and 4th in the Backbone Index (including non-Asian NSPs). When you combine these three rankings, NTT (AS 2914) is about 20% bigger than China Telecom (AS 4134)!

Here's the distribution of a representative sampling of providers based on their Wholesale to Retail bias:

While not a quantitative comparison, by clustering providers according to their relative rank in the Retail and Wholesale markets, we can group them into functional categories. Providers in the green segment are strong in both retail and wholesale rankings. Yellow-segment providers are strong in wholesale (typically in multiple countries). Providers in the salmon-colored segment are strongest in backbone and wholesale.

As you can see, selecting criteria to produce the most accurate representation of the Asian market is pretty complicated. The Internet continues to grow and change at Moore's Law pace, and Renesys is constantly monitoring how every switch, shift, transmutation or transfiguration affects the global Internet market. As a subset of our Market Intelligence Asian Ranking, the Intra-Asian Wholesale Ranking (at right) provides an additional view of a dynamic marketplace.

Pinning Down Latency

2011-10-07T17:35:13Z

Packet latency is a big issue in Internet-based applications (i.e. the stuff in the cloud). In conducting analysis on Internet infrastructure over the years, we have seen many patterns of connectivity. One such pattern that can wreak havoc on latency is "hair-pinning", a phenomenon where traffic takes an unnecessarily long physical path between two points on the Internet due to suboptimal routing. The increased distance results in increased latency, and the "lag" or "sluggishness" that users experience as a result can hinder latency-sensitive online applications whether they are financial trading applications or MS SharePoint.

]]>

Geographic Locality does not translate into Internet Locality

A couple of weeks ago (September 23^rd to be exact), I noticed a small routing outage (150 globally routed prefixes) in Sakhalinskaya Oblast, the most Eastern part of Russia. We observe these types of outages every day — the Internet is a big messy place behind the scenes.

The obscurity of this location piqued my curiosity and I looked into how this area gets Internet connectivity — almost exclusively through Russian provider, TransTelecom (AS20485). One can see that this province connects to Japan via Hokkaido-Sakhalin Cable System (HSCS) built by NTT in 2008 to provide a shorter path from Japan to Europe over Russian soil. However, despite the proximity to Japan, a traceroute from NTT's Tokyo Looking Glass to an IP address in Sakhalinskaya looks like this:

Query Results: Router: Tokyo - JP Command: traceroute 217.148.202.106 traceroute to 217.148.202.106 (217.148.202.106) 1 ae-0.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.2.116) 0.592 ms 2 ae-3.r22.osakjp01.jp.bb.gin.ntt.net (129.250.3.220) 9.322 ms 3 as-0.r23.londen03.uk.bb.gin.ntt.net (129.250.5.34) 329.864 ms 4 po-1.r00.londen03.uk.bb.gin.ntt.net (129.250.4.134) 325.062 ms 5 83.231.221.38 (83.231.221.38) 283.933 ms 6 kbk15.skh21.transtelecom.net (217.150.52.254) 456.641 ms 7 ip.217-148-198-18.sakhalintelecom.ru (217.148.198.18) 467.577 ms 8 kentzrussia.com (217.148.202.106) 481.945 ms

Which roughly follows a path like this:

Circuitous routes like this are called "hair-pinning" in the business - this traceroute travels from Japan to London just so it can make its way back to a location just off the coast of Japan. On the Internet, geographic locality does not translate into Internet locality, and the distinction can result in unexpected latency, frustrating users of real-time applications.

Latency in the Middle East

Last week I was in the fascinating city of Muscat, Oman for MENOG 9 giving a talk about latencies we have measured into each Middle Eastern country. While regional traffic in the Middle East often suffers from hair-pinning via European Internet exchanges, latencies to and from Europe appear to be steadily improving.

In the past, when Renesys has analyzed Middle East connectivity it was with regard to complete outages such as the cable cuts in 2008. This year, connectivity into the Middle East has been much more stable (obvious exceptions notwithstanding) and the fact that regional providers can focus energy on reducing latency is a good sign for the region. For example, OmanTel's strategy (AS8529) to reduce latency to Europe by gaining presence at AMSIX (AS1200) is paying dividends in 2011 by reducing median latency by 41ms or 20% in 2011. This is illustrated in the chart below, which is a plot of overall latencies from London to Oman in 2011 colored by OmanTel's upstream provider or exchange as observed by traceroute. Higher latency bands (PCCW, AS3491, in Grey and LINX, AS5459, in blue) disappear midway through the year and are replaced by a lower latency path through AMSIX (green).

Until a regional Internet Exchange emerges for the Middle East, the best regional providers can hope for is to reduce latency on the path of the hair-pinning through Europe for regional traffic!

Impact of IMEWE Activation in Lebanon

The most recent (and intriguing) development in the area of latencies to the Middle East is the reported activation of the India-Middle East-Western Europe (IMEWE) cable system at Tripoli, Lebanon. Despite speculation that the activation would be delayed, there is evidence that the cable is now being actively used. Certainly something has happened in Lebanon to reduce latencies in recent days.

Thus far, development of the Internet market in Lebanon has been stifled by the country's extensive dependence on satellite Internet service — the ultimate form of hair-pinning. In the visualization below, latencies into Lebanon separate into two modes based on delivery medium: satellite (primarily via SatGate, AS30710) and submarine (primarily via Level3, AS3356). Further examination of the two latency modes clearly indicates asymmetric routing. First, latency measurements should be at least 480ms for symmetric round-trip delay across a satellite connection due to the speed of light and altitude of orbit, but we observe latencies clustering around 400ms. Second, the dramatic 70ms drop in median submarine latencies starting September 22 is shadowed by a corresponding effect in the latencies inbound through the satellite provider. Packets are arriving inbound via satellite but returning via submarine cable. Note also the further 15ms reduction on September 29th, which has held through today. (Note: Dates referenced appear at small ticks along x-axis)

Additionally, we see a reconfiguration on August 24^th visible as a slight change in this latency distribution. Here there was an address change in the IP-IP hop in Level3's network (AS3356) that experiences the vast majority of the overall latency to Lebanon. It is a reduction of the latency across this hop that contributes the most to the downward shift in latencies to Lebanon in recent weeks.

In the coming weeks we expect to see a dramatic shift in transit as Lebanese providers move away from expensive and high-latency satellite service to IMEWE-based service. This is likely to resemble the collapsing of satellite Internet markets which we have observed occurring in African countries almost immediately after new cable landings are established nearby. For example, the chart below is the relative mix of Internet transit providers for Lebanese provider TerraNet (AS39010) in 2011. In this stacked chart, a taller band of color indicates greater amount of Internet transit through that provider for a given point in time. Except for Liban (AS42020), the vast majority of their transit is satellite.

This chart is almost certain to look different in a couple of months if the new cable is here to stay. We'll update this story as it develops. With any luck, latencies to Lebanon may begin to resemble those of other countries of the Eastern Mediterranean and Lebanon's inexplicable dependence on satellite will be a thing of the past.

What to do about hair-pinning?

There are many causes of latency in network performance, such as congestion and router overutilization, but hair-pinning is often avoidable. Businesses need Internet intelligence to know their high latency is a result of hair-pinning and then press their providers to do something about it! This is particularly important for mobile providers because it adds another potentially severe performance penalty to the Internet service they provide.

Irene Wallops US Internet

2011-08-30T16:49:23Z

Hurricane Irene knocked out power to millions of homes and businesses as it travelled up the US East Coast this weekend. Even as the winds subsided, torrential rains triggered savage flooding throughout Eastern New York state and Vermont, tearing up roads and exposing the telecommunications infrastructure to further risks. The storm's impacts were clearly visible in the Internet's global routing table, as tens of thousands of networks were cut off from the rest of the world.

Here are a couple screenshots from our Internet Health Portal, which we provide to the US Computer Emergency Readiness Team (US-CERT). During an emergency like Hurricane Irene, this tool provides the US-CERT with critical information about the availability of Internet services across America. Working from lists of impacted customers in each state and county, and lists of correlated outage events, we can supply a lot of useful information about the problems being experienced by enterprises in the affected area. That information can be passed along to state and local governments to aid in prioritization of disaster relief.

]]> State-By-State Impacts

During Irene, this feedback was particularly important because the storm's progress affected different regions at different times. To the left below, you can see a sequence of state-by-state plots of withdrawn routes (representing networks that have become unreachable due to power loss or infrastructure damage) along the US East Coast, from south to north. (Click on individual state plots to see details.)

As time marches on, you can see the peak window of Internet damage move north along the track of the storm. Outage counts rise with the local onset of storm winds, and fall again as restoration efforts are successful (significant outages continue in the last states affected: Vermont, New Hampshire, and Maine).

Animation: East Coast Internet Outages

Finally, here's an animation that combines the network outage information in each zip code with the satellite imagery from Irene. It gives you a good sense of the massive scope and impact of this event. Short-duration outages are green; longer ones that linger (due to sustained power outages or infrastructure damage) are red.

Staying Connected, Preparing to Rebuild

Overall, it seems that the East Coast's power and Internet infrastructure fared pretty well during this storm, with good evidence of restoration after the storm had passed. This is good news, given the important role Twitter now plays in ad hoc rescue coordination, and the importance of the Web for keeping people informed about what they're facing in an emergency situation. I suspect that always-on, ubiquitous Internet access is going to fundamentally change the way people on the ground manage their affairs in the wake of disasters like Irene.

I spent 12 hours picking my way across the ruined roads and bridges of Eastern New York State yesterday, trying to get back to New Hampshire, and I can attest to the fact that the transportation network is now far more vulnerable to disruption by an event of this scale than is the cyber-infrastructure.

As we drove past legions of idle 18-wheeler trucks full of food and fuel, unable to reach their destinations, 3G mobile connectivity kept us connected to the Internet and in touch with the tweets of local emergency management officials and people back home. At one point we were even part of a stream of vehicles heading urgently for higher ground, following a report that the Gilboa Dam had failed. Thanks to Google Maps we knew where to climb to, and thanks to Twitter we knew when it was safe to come down again. You can't eat the Internet, or burn it to keep warm, but compared to the days of the transistor radio and EBS alerts, we've come a long way.

The Battle for Tripoli's Internet

2011-08-21T12:15:00Z

As dawn broke in Libya on the morning of Sunday 21 August, it appeared that the battle for control of Tripoli was underway. Throughout the night, a steady stream of tweets and retweets emerged from Libyan sources, painting a confusing, often contradictory picture of the evolving situation.

We're still piecing together the data that can confirm or deny much of what's been reported overnight, but one thing is clear: something very strange was going on with Tripoli residents' Internet access. Service was restored suddenly in Tripoli, flickered on and off for a couple of hours, and then died, with the majority of the country's international BGP routes withdrawn from service for good measure. Today the routes are back in Tripoli, but ADSL service isn't. This morning we're looking back at this curious overnight episode, and speculate about what might have happened.

]]> Background

For several months, our picture of the Libyan Internet has been essentally static. The national connection to the Internet consists of 16 blocks of IP addresses, each routed to the outside world through Libyan Telecom and Technology. That basic routing footprint has been advertised to the world, with few interruptions, since the end of the March Internet blackout.

But average people in Tripoli still haven't had much access to the Internet, because local DSL service has been largely unavailable for the last three to five months, depending on where you live. The few people who did retain their official Internet service continued to access Google and YouTube, as measured by Google's excellent Transparency Report. When LTT's international Internet connection started to show sporadic signs of failure a couple weeks ago, it only affected 11 of 16 blocks, leaving intact the neighborhoods who appear to have been generating the majority of the country's surviving Google traffic.

With so much of Libya's Internet disconnected in the last mile, there are few alternatives, other than outlawed satellite phones or international dial-up. These are dangerous, slow, expensive ways to get the news out.

Operation "Mermaid Dawn"

Nonetheless, it became apparent from the Libyan Twitterstream over the last couple days that things were about to heat up in Tripoli. It seemed likely that mobile networks, and perhaps the entire phone system, could be shut down within the capital, as the government attempted to prevent the Tripoli uprising from self-organizing. There were sporadic tweets about phone calls not completing, but the expected telecoms shutdown never came.

Instead, Al Jazeera began to report that Tripoli residents were receiving mobile phone text messages, urging them to take to the streets (typically, in the fog of war, it seems unclear whether they were being exhorted to support the government or the rebels). And early Sunday morning, the Twitterstream suddenly began reporting something that seemed, on the face of it, totally improbable: the Internet had been turned back on.

Why would the government turn the Internet back on in the middle of an armed uprising? To get people to stay at home and catch up on five months of email? It seemed preposterous. But clearly, as more and more people realized, it had happened. Bandwidth was scarce, but DSL service was back. People started Skypeing with friends and relatives, some reporting hearing live gunfire in the background as their VoIP calls began to connect.

And then, as suddenly as it had come, Tripoli's Internet access stopped working again. For a total of perhaps an hour and a half of uptime, spread out in bursts between the hours of 2:00am and 4:30am, local time, the Internet had been functional again. Who was responsible? Would it come back?

Routes Withdrawn

Forty-five minutes later, at 5:14am local time, we may have received an answer of sorts. Eleven of LTT's 16 international routes to the Internet were withdrawn from the global routing table: the same eleven routes that had been sporadically unstable two weeks ago. (Note the drop at the right edge of the plot, below.) They stayed down for the next four hours, and came back up at 9:22 local time. While these routes were down, the up-or-down fate of the corresponding local DSL or Wimax services became essentially irrelevant; their plug had been pulled at the international border, and nobody in those local networks could exchange traffic with the world outside Libya, until LTT chose to reestablish routing for them.

Now, even though the routes are back in place, local Internet service appears to be down again -- the status quo ante for the last five months. We note that LTT's website is still down. Did the brief Tripoli Internet flicker represent a sign of conflict within the phone company itself, with someone struggling to reactivate service at the neighborhood level, only to have it switched off again at the national level? Or was the overnight routing failure just another in a sequence of (probably power-related) outages for LTT's outlier networks? The people without Internet access in Libya have a lot of questions at this point, and we don't have enough data yet to give them a satisfactory answer.

Update (12:00am Monday in Tripoli): LTT website (http://ltt.ly) is back online, and the Arabic crawl at the bottom says "Congratulations, Libya, on emancipation from the rule of the tyrant."

Libyan Internet Instability

2011-08-12T15:10:01Z

There hasn't been much to say about the Internet in Libya this summer, as their patterns of connectivity have been fairly stable. It was interesting, therefore, to observe that much of the country's Internet routing has started to show evidence of sporadic failures this week, which have gone unreported in the media.

The following plot shows the number of Libyan networks (blocks of Libyan IP addresses) that appear in the global routing table. There are typically 16 of these, all routed by Libyan Telecom and Technology (LTT) via Telecom Italia. This week they have suffered some impairment, in groups of 6 or 10, in episodes that typically last no more than a few hours.

There didn't seem to be any pattern to these outages, which took place at all times of the day and night. It seems to suggest power outages, rather than permanent facilities damage, or deliberate action by the government (for suppressing communications, say).

Is it possible that LTT is suffering power outages, and having trouble finding fuel for their generators due to NATO's unofficial fuel blockade of Tripoli?

One reason why the world at large may not have noticed much in the way of Internet impairment: the affected networks don't seem to be the same ones used to access the majority of Internet content from inside Libya. Looking at Google's plots of inbound traffic from Libya, these substantial network outages seem to have had very little impact on the daily traffic curves:

In other words, the handful of Libyan networks that aren't affected by these outages seem to be the same ones that are consistently generating the Google traffic. If you're lucky enough to be an LTT customer in one of these Internet neighborhoods (presumably in Tripoli), your connectivity stays up. Everyone else, well, you're on your own. Time for Internet in a suitcase?.

Tracing the Syrian Blackout

2011-06-10T23:00:23Z

Updated Monday morning to include detailed Syrian network map, and include one-second BGP plots during the day of outage. --jim

Thanks to everyone who's stopped by this week to read about the Internet blackout that affected Syria last Friday. We're always glad to hear your comments, especially when you fill in some of the missing parts of the story that aren't obvious from the data.

A lot of the questions we've received about Syria this week have speculated about a repeat performance of last week's Internet blackout. Would the Syrian Internet get shut off again this Friday, as it was last Friday? Would it be visible in the BGP routing table? Would this become a recurring tactic, just another mechanism for crowd control?

Initial data from Google's transparency report suggest that total traffic was down, even compared to a normally quiet Friday.

But looking at the broader picture, a couple things are clear. First, there was no repeat of last week's event, in which two-thirds of all Syrian networks became flatly unreachable from around the world, an Egyptian-style disconnection at a very fundamental level. This week, while traffic levels were reduced (perhaps throttled or rate-limited, as in Iran), the routes themselves remained intact.

If you wanted to reach a Syrian website Friday, or if a Syrian browser wanted to reach a European website, the paths were known and the lines were open. Whether you could actually get enough bandwidth to upload videos, or make a Skype call, was another story. We've heard some anecdotal evidence that there were connectivity problems. But anecdotes aren't data.

Speaking of data, that's another question we get all the time. How do we examine a market like the Syrian Internet, and make our call about the structural availability of routes to its networks? Let me take a moment and describe some of the technology behind these reports.

]]> Finding All The Paths

Many people assume that we somehow have direct access to the world's traffic data, measuring the rates of traffic on millions of network links as they ebb and flow with the rising and setting of the sun — a poetic vision, but obviously impossible (what a measurement challenge, to be everywhere all at once!)

Instead, our information comes from two indirect sources: changes in the BGP routing table (all the myriad ways that global ISPs believe they can reach Syria at any given moment) and active traceroute measurement.

First, BGP. In this plot, you can see the percentage of Internet transit (top) and total Internet transit of customer base (bottom) that Syrian Telecom, AS29386, sends to each of their international ISPs. This plot displays changes as small as one second in duration, meaning that we should see any differential fluctuations in Syria Telecom's transit spectrum, no matter how brief. In Egypt, which has much more complex provider structure, we saw a rocky ride to the bottom, before the outage was complete.

In fact, the Syrian transitions are clean and sharp, with a single outage, followed at 19:00 the next day by a partial restoration, followed by the remaining restoration Saturday morning. PCCW was relatively most affected, going to zero for the duration (no service to Syria). Deutsche Telecom, Turk Telekom, and Tata's relative standings are basically preserved.

Active Measurement of the Syrian Outage

Having looked at the routes, we can look at what active measurement has to say. For that, we use multipoint traceroute.

Traceroute is an old, old Internet protocol. You send small probe packets towards a destination, with different expiration times, and when they come back from intermediate points along the route, you see where they've been and how long they've been travelling. We do that 24 hours a day, from computers all over the world, building up a coherent picture of all the paths traffic is actually taking into networks in Chicago, or China, or Syria.

To show you what I mean, here's a picture that summarizes our traceroutes into the 60-odd networks that make up what we call "the Syrian Internet." Time flows from left to right, covering about 2 weeks. The vertical lines-and-boxes show you the latency distribution of packets going to Syria and back to one of our collectors worldwide.

The small red line is the median value, normally about 230 milliseconds for a packet to travel into Syria and back out again. When networks get really busy, and traffic exceeds the capacity of the Internet connection, they will sit in router buffers for a long time, and this latency measurement will edge upward, sometimes by a factor of 2 or 3 times more delay. Here, you don't really see that in evidence — there's some fluctuation, but no consistent upward surge in delay at any point over the two-week period.

In fact, median latencies drop during some of the June 3rd event, suggesting that the routers that remained reachable were better-connected than average: close to the Syrian backbone, fewer hops away. That would be consistent with the theory that access networks around the country (DSL, 3G mobile) were down, and government/core telecoms resources remained up.

Of course, last Friday's event is clearly visible. That's the second dataset you see plotted as a blue background: the total number of traces inbound to Syria that successfully completed. What happened during last Friday's day of rage was a real disconnection: the withdrawal of routes. Many of our traceroutes into Syria that day simply couldn't get started, because they had no idea how to find their way into Syrian Telecom. After the Syrian blackout was lifted, things basically returned to "normal," just as we saw after the (much longer) Egyptian blackout in January.

Looking through this particular lens, we see that Friday June 10th, looks nothing like Friday June 3rd. There's some reduction overall in the number of successful traceroute completions this week, compared to the same days last week. But there's nothing visible here to distinguish Friday 10th in particular as a day of rage. Both completion counts and measured latencies look the same Wednesday, Thursday, and Friday.

Another Perspective: Provider Choice

Okay, let's look through one more lens at the same event, showing how paths changed within the traceroute dataset. Remember, traceroutes don't just show you the delays for traffic into and out of Syria, they show you which providers' routers were encountered on the way in and out. That can tell us something about what those providers are up to, when there are big transit changes.

Again, this plot covers 2 weeks from left to right. The Friday June 3rd event is clearly visible as a drop in the total number of traces into Syria through all international providers. Each color band indicates the relative proportion of traces that chose a given international provider as a way of reaching Syrian Telecom's routers. From this, assuming that our traceroute set is large and well-distributed globally (and it is), you can draw some conclusions about the relative weight Syrian Telecom places on each of its provider routes.

Primary transit here was from Tata, Deutsche Telecom, and Turk Telekom, with tiny numbers of paths through other providers. During the first half of the event, all providers' Syrian services were reduced. Tata's and Turk Telekom's shares dropped by about 2/3, proportional to the total number of outaged routes. Deutsche Telekom's service drops by a larger amount, nearly disappearing in the first 12 hours of the outage, despite still being a large part of the inbound available routing.

Halfway through the event, however, more traces start to succeed through Deutsche Telekom, as some routes are partially restored. By 5th June, the trace proportions are basically restored to their relative blend from before the event. It's hard to read, but it suggests to me that this blackout was actively managed by Syrian Telecom, with different groups of networks and different routes coming and going in blocks through the course of the day on Friday, not a simple lights-out event.

You can see that Friday June 10th was nothing like Friday June 3rd. Whatever happened there on the 10th, it did not affect Syria Telecom's provider blend or overall reachability in visible ways. On the 3rd, the government's goal may have been to disrupt online organization of coordinated protests — there aren't that many other explanations for a one-day blackout (videos that couldn't go out to YouTube on Friday, simply went out on Saturday).

And so it goes. The Syrian Internet is, by and large, back online, and has stayed up for a week. This map shows its basic structure and international transit, midway in complexity between Egypt (much larger) and Libya (much smaller). Just a few autonomous systems controlling the transit, just a few dozen networks in all, and yet, it's critical infrastructure for the Syrian people.

Did a day without Internet dampen the protests, or amplify the fury in the streets? One senses that a line has been crossed, and whatever the outcome, the status quo ante solution is now off the table. We can only wait for information to trickle out of Syria, and try to infer the course of events from the available data.

World IPv6 Day

2011-06-08T20:01:01Z

Today is World IPv6 Day, a day when major content providers have agreed to furnish service over IPv6 for a 24-hour test period. Hopefully, you didn't notice anything different about your Internet experience today, but providers will have gained valuable experience with the technology and any technical hurdles that remain to be overcome. In this blog, we'll report on how far into the IPv4-to-IPv6 transition we actually are and, more importantly, just how far we have to go. There is no denying that there has been a tremendous amount of progress in the last decade or so, but much remains to be done and we are only at the very beginning of a long process.

]]> Background

As has been reported by Renesys and endlessly in the press, the world is running out of usable IPv4 addresses, which are 32-bit numbers and hence limited to 4 billion distinct values. With every mobile device, desktop, server and (maybe soon) light bulb needing an IP address for connectivity, we simply don't have enough. The only long term game in town is IPv6, a new system with 128-bit addresses or 340,282,366,920,938,463,463,374,607,431,768,211,456 distinct values. (We aren't going to make that mistake again!) Unfortunately, these different addressing schemes are not compatible. If you want to provide content on both "Internets", you need IPv4 and IPv6 addresses for your servers and you need all the infrastructure that goes along with both approaches.

What is World IPv6 Day anyway?

When it comes to upgrading the Internet in place, there are a lot of moving parts to consider: end-user operating systems, home networks, routers, firewalls, servers, Internet service providers, applications, and so on. Despite all the transition planning that has been carried out to date, a lot can go wrong. If a content provider suddenly started supporting both IPv4 and IPv6 on the same domain, that action alone could prevent some customers from gaining access. World IPv6 Day gives the major providers cover to make this change at the same time to see what happens. In other words, if you couldn't reach Google today since they turned on IPv6 support, well you couldn't reach Yahoo or Microsoft either. For domains participating in this experiment, DNS should return IPv4 and IPv6 addresses. What your computer ends up doing with the very different answers depends on all the factors mentioned above. And from this experience, the providers hopefully gain valuable insight into what needs to happen next.

Are the Internet service providers ready for IPv6?

Only somewhat. A lot has been accomplished, but much of it has been limited to the largest global and regional players on the Internet and the most developed countries. On either "Internet", connectivity requires blocks of IP addresses (prefixes) and Autonomous Systems (ASes) to handle the routing of traffic. So it's natural to compare both Internets relative to these quantities. In IPv4, we currently see over 360,000 routed prefixes, distributed (unevenly) to pretty much every country on earth. In IPv6, we only have around 6,500 prefixes with a concentration in the developed world as shown in the following map.

(Note: IP address geo-location is a tricky business and only more so with respect to IPv6, where tunnels are often employed. We've made our best approximations here, but if you feel there are inaccuracies, please let us know.)

As for ASes that originate IPv6 prefixes, only 13% of those seen in the IPv4 world are also seen in IPv6, although there has been a recent slight up-tick in interest as shown in the following graph. And it is clear that the IPv4 Internet is still growing at consistent rate.

Another way to visualize IPv6 adoption is to consider those ASes that transit only IPv4 compared to those that transit both IPv4 and IPv6. In the next image, ASes transiting IPv4 only are shown as red dots and those transiting both as blue dots. Dots in the outer ring correspond to ASes with no customers. As you move toward the center, inner rings depict ASes with more and more customers. The barely visible dots near the center correspond to the so-call Tier-1 providers. (Each dot representing an AS is placed at a distance from the center that is inversely log-proportional to the number of downstream customers the AS has.) As you can see, the largest providers are ready for IPv6, as it is becoming a competitive differentiation, however, much of the rest of the world is sitting on the sidelines.

Are the content providers ready for IPv6?

No. As expected, the major players are there, but keep in mind that there are over 108 million .com and .net domains alone. From our estimates, much fewer than 0.1% of these domains also have some sort of IPv6 presence. So even if you do have IPv6 connectivity today, the IPv6 Internet is quite barren compared to the IPv4 Internet. Basically, there isn't much to see on the IPv6 Internet and, whatever is there, is almost certainly replicated in the IPv4 world.

Are consumers ready for IPv6?

In general, consumers simply want Internet access and don't think in terms of IP addresses or the underlying technology. Fortunately, modern operating systems generally have mature IPv6 support. Where consumers might run into difficultly is with their access devices (e.g., DSL/cable modems) or the fact that their provider is one of the many lacking IPv6 support. If you want to test out your own IPv6 connectivity, you can go here and, if all is well, you might see output of the following form.

Presumably, the end user equipment will take care of itself as technology is refreshed. Currently, busy DNS servers worldwide show that about 20-25% of clients are making IPv6 queries. This adoption rate is slowly increasing over time.

Impediments to Adoption

The IPv6 Internet lacks two critical ingredients for success: users and content. In this chicken-and-egg dilemma, many organizations have no interest preparing for a largely barren IPv6 Internet and content providers aren't going to waste resources on an Internet with no users. The economic driver is currently absent for many.

But there is another large barrier to progress that is not getting much publicity. The IPv6 Internet is not fully connected. That's right, even if you have IPv6 connectivity today, you might not be able to reach the entire IPv6 Internet, as small as it might be. The IPv6 Internet lacks a fully connected mesh of "Tier-1" providers. Our data supports the information about this partitioning found here. As one example, if you have IPv6 connectivity solely from Level 3, you cannot reach those who are solely dependent on Hurricane Electric. This is very different than the IPv4 world, where any single provider will generally take you everywhere you want to go (ignoring issues of government censorship).

What's next?

We'll close out this blog with more questions than answers. Future growth in IPv6 deployments could very well depend on how the following are answered.

Will the growth of mobile devices and their need for address space finally spur businesses to seriously invest in an IPv6 presence?
Will developing world demands on IP space boost global IPv6 adoption rates?
Will a robust secondary market develop for IPv4 addresses that will further delay IPv6 adoption? With or without the blessing of the registrars?
Will scarcity of IPv4 addresses erode cooperation on the Internet?
Will an alternative technology, such as carrier grade NAT, reduce the need for routable IPv4 addresses?
Will a fully connected set of IPv6 Tier-1s finally emerge, allowing end users to reach the entire IPv6 Internet via any single IPv6 carrier?
Will any future government mandates encourage IPv6 adoption?
Will interoperable and ubiquitous support for IPv6 eventually exist in all devices?

How these questions are answered and in which order could seriously impact the ultimate fate of IPv6. In the near term, it is probably safe to say that a market will develop for IPv4 addresses once the shortage becomes more acute and further band-aids (e.g., layers of NAT) will be applied to conserve IPv4 addresses, while IPv6 deployments will continue at a steady, and hopefully increasing, pace. We'll continue to follow and comment on these developments in our blog. In addition, the next release of our popular Market Intelligence product, due out later this year, will enable customers to explore the currently sparse IPv6 Internet.

Syrian Internet Shutdown

2011-06-03T14:09:26Z

(Updates on the restoration of Syria's internet at the bottom of this page. --jim)

Starting at 3:35 UTC today (6:35am local time), approximately two-thirds of all Syrian networks became unreachable from the global Internet. Over the course of roughly half an hour, the routes to 40 of 59 networks were withdrawn from the global routing table.

This image shows the current state (green: reachable, red: unreachable) of each network prefix in the Middle East this morning, visualized as a packed Hilbert-curve representation. The size of the colored area is proportional to each country's Internet presence, so you can see that Syria's Internet (red block near the top center) is a little smaller than that of Kuwait.

]]>
The Internet in Syria basically depends on one domestic provider, state-owned Syrian Telecom Establishment (AS29256 and AS29386). They buy most of their Internet transit from Turk Telekom and Deutsche Telekom, with some contribution from PCCW, Tata, and Telecom Italia. Connectivity has historically come in over submarine cable from Cyprus; activation of new terrestrial fiber connections to Turkey have been delayed by this year's political unrest.

The network prefixes that remain reachable include those belonging to the Syrian government, although many government websites are slow to respond or down. The Oil Ministry is up, for example, and Syrian Telecom's official page, but the Ministry of Education is down, as is the Damascus city government page, and the Syrian Customs website.

The networks that are not reachable include, substantially, all of the prefixes reserved for SyriaTel's 3G mobile data networks, and smaller downstream ISPs including Sawa, INET, and Runnet.

We'll update when we have more information. We don't know yet how the outage was coordinated, or what specific regions or cities may be affected more than others. News is filtering out of Syria very slowly. If Egypt and Libya's Internet outages are any guide, one might conclude that events on the street in Syria are reaching a tipping point.

Edit: clarified that Syrian Telecom Establishment is the state-owned Internet service provider, as distinguished from SyriaTel, the mobile provider that is "not quite" government-owned.

Update (Saturday 4 June 2011):

The Syrian Internet is back up. Seven of the 40 networks returned around 19:00 UTC (22:00 local time Friday night). The rest came back shortly after 04:00 UTC (07:00 local time Saturday morning). With connectivity restored, the Google Transparency Report confirms that traffic has resumed, at levels that look provisionally similar to those before the blackout. Will Friday Internet blackouts become a regular feature of the Syrian protests?

Qwavvis: The Battle for Second

2011-04-29T16:56:48Z

About two weeks ago, Level 3 announced plans to acquire Global Crossing and we blogged on the enormous size and scope of the new entity, which we called Level Crossing. This week, CenturyLink, a regional US phone company, agreed to acquire Savvis. Since CenturyLink also owns Qwest, we are seeing another merger of two Tier-1 Internet providers, a pairing which we'll label Qwavvis. In what follows, we examine the possible business considerations behind the move, as well as the impact on Internet transit customers and Renesys Market Intelligence rankings.

]]> Renesys rankings of Internet service providers are based entirely on the quantity of IP space ultimately transited by each provider. While a crude measure, it is an objective one, and it is the ranking trends, rather than any absolute number, that are the most revealing. As noted in our last blog, if #1-ranked Level 3 succeeded in merging with #2-ranked Global Crossing today, the rankings of the top global providers would have the following form. The newly combined entity would dominate the global provider market.

Assuming regulatory approval of the merger, the battle is now for second place. It would seem that NTT is in a relatively comfortable position here, as Sprint has done nothing but decline in our rankings over the years, after aggressively competing for the #1 spot with Level 3 throughout 2008. However, if Qwest and Savvis succeed in merging, NTT will find the combined entity nipping at their heels as shown below. Assuming the traditional US carriers continue to prioritize product lines other than IP transit, another very tight race will ensue for the #4 position, between Tinet, Telia and Tata.

Investor reaction to the Qwavvis deal has been generally positive and discussed mainly in terms of EBITDA and return on investment. But there are other ways to look at this. First, Qwest is focused on traditional telecommunications services, while Savvis is more aligned with data center services — cloud computing if you will. In a buyout, it makes little sense to spend money to acquire your same customers again. And since Qwest and Savvis are so different, they have almost no customer overlap — one of the more dubious aspects of the Level Crossing deal. In fact, only 12 networks (i.e., prefixes) will suddenly find themselves single-homed behind Qwavvis and might need to go shopping to regain provider redundancy. These include such "powerhouses" as Allen's TV Cable Service with three locations in South Central Louisiana and AAA Internet, a dial-up Internet provider.

As a result, with respect to our rankings, Qwavvis is essentially the sum of its two parts, i.e., the Renesys score for Qwavvis is over 99% of the sum of the individual scores for Qwest and Savvis. Compare this to Level Crossing. As noted in our last blog there is considerable overlap between Global Crossing and Level 3. While they have tended to focus on different geographic areas, their merger will still leave around 3,500 networks (prefixes) relying on a single provider. This overlap is reflected in our scoring. Level Crossing's score is only 75% of the sum of the individual scores for Level 3 and Global Crossing. Such a dilution of value is largely unavoidable for competitors with a global footprint and maybe the best you can expect in a such a situation, although we haven't run the numbers for all such possible pairings.

Rather than look at existing customers, another way to consider such mergers is by the potential for future sales. While divining the future is never easy, there is one certainty in this market: the raw material of the current Internet is extremely limited and becoming more so all the time. I'm talking about IPv4 addresses. Before much longer, there will be only one place to get them, namely, existing service providers. The lack of this raw material will undoubtedly curtail the growth of some providers and limit new entrants into the market.

In fact, if the Microsoft purchase of Nortel address space for $7.5 million is any indication, corporations are starting to wake up to the impending resource scarcity. While we will have a lot more to say on the value of IPv4 space in future blogs, we'll briefly consider the issue here for the proposed mergers. Taking into account the various Level 3 and Global Crossing acquisitions over the years, the newly formed Level Crossing will control nearly 50 million IPv4 addresses or over 1% of all available address space. In contrast, Qwavvis will control about half as much space. For both companies, much of this space is currently unused. As consolidation in the industry continues, not only will customers have fewer choices, they might also need to consider if their potential vendors will have the IP resources available to accommodate their future growth. Stay tuned for more on this important topic!

Level Crossing

2011-04-14T19:53:53Z

On Monday, 11 April 2011, Level 3 announced they had entered a definitive agreement to acquire Global Crossing. According to the Renesys Market Intelligence rankings, this merger would bring together the world's #1 and #2 global providers, with over half the Internet market on earth dependent on the combined entity. If the deal gained regulatory approval in the US and elsewhere today, how would the Internet provider landscape change? We'll answer that question in this blog, giving the proposed union a fictional name of Level Crossing for the purposes of our discussion.

]]> It's become a tradition at Renesys to provide a periodic review of how the Internet providers at the top of our Market Intelligence global rankings are faring. We took our most recent look at the end of 2010, at which point Sprint had just been overtaken by Global Crossing for the #2 position and was on the verge of being passed again by NTT. Note that our rankings are a rather crude measure of size, as they are based entirely on the quantity of IP space ultimately transited by each provider. However, it's the ranking trends that are more revealing than any absolute number. Who is adding customers? Who is losing them or just standing still? Changes in IP transit answer these questions and provide an objective measure by which to rate providers.

Looking at the top five global providers today, we have the following breakdown by global market share. The percentages add up to more than 100%, since any organization serious about its Internet presence is multi-homed, i.e., has more than one service provider for redundancy. By our metric, Level 3 has just over 40% of the market and Global Crossing just over 30%.

If Level 3 and Global Crossing merged today, the top global rankings would change as shown below. Note that the combined entity, Level Crossing, would have a 55% market share, marking the first time we've seen one company in such a dominant global position. However, this is lower than what the sum of the two parts might suggest, due to the fact that some businesses buy from both providers already. That is, Level 3 already has some of the same customers as Global Crossing.

Regardless of the overlap, Level Crossing would certainly be a global colossus. Another way to see this is to consider all of the currently routed IPv4 address space. Using a Hilbert curve representation and software from The Measurement Factory, we next show how pervasive and critical the two merging entities are to the operation of the Internet. Networks transited by Level 3 and not by Global Crossing are shown in yellow, those by Global Crossing and not Level 3 in blue, and by both in green. All other routed networks are displayed in gray, while unrouted networks are in black.

All IPv4 address space, showing the impact of the proposed merger

By our measure, Level Crossing would tower above well-known national incumbents. The larger the entity, the more on-net ("internal") traffic they have and the more revenue they garner from customer-to-customer communications.

In fact, the next five global providers would have to merge to rival Level Crossing's score!

Conclusions

Internet transit remains a very tough business as prices continue to erode while demand only escalates. As with any commodity, customers are fickle and often base decisions solely on price and availability. For anyone in this business, it makes sense to acquire the largest possible global footprint and the greatest pricing power. In fact, we suggested that Level 3 acquire Global Crossing back in 2006, as it certainly made sense from an engineering perspective.

From a customer's point of view, there will be one fewer choice in the market — probably not a big deal unless you reside in an already poorly served area. The customers that will be the most impacted are those that now suddenly find themselves single-homed behind Level Crossing. If they want to maintain provider diversity, they'll have to go shopping. By our reckoning, almost 3,500 networks (prefixes) could soon find themselves in this position, including those from Thomson Financial, Bank of America, CBS, Reuters, and even Major League Baseball. The list includes both global and regional entities, such as Vodafone Italy and Turk Telekom. Ukraine is one of the most impacted countries, with 7% of their prefixes suddenly becoming single-homed.

This isn't necessarily a bad thing and the industry has certainly had its share of consolidation in the past, an inevitable feature of a maturing market. It's the size and scope of this deal that makes it truly noteworthy.

Japan Quake

2011-03-12T00:20:43Z

Today's 8.9 magnitude earthquake in Japan has had surprisingly limited impacts on the structure and routing dynamics of the regional Internet. Of roughly 6,000 Japanese network prefixes in the global routing table, only about 100 were temporarily withdrawn from service — and that number has actually decreased in the hours since the event. Other carriers around the region have reported congestion and drops in traffic due to follow-on effects of the quake, but most websites are up and operational, and the Internet is available to support critical communications.

]]> Those who have been following our blogs on Libya will be familiar with the excellent Google Transparency Report, which summarizes the rate of queries coming from each country over time. Despite terrible fires, floods, and power outages, traffic from Japanese clients just keeps going. It's quite a remarkable plot.

Why have we not seen more impact on international Internet traffic from this incredibly devastating quake? We don't know yet, but we'll keep studying the situation. Compared to the 2006 Taiwan earthquake, which resulted in a larger number of major cable breaks, it appears that the majority of the region's submarine cables have escaped the worst damage, and diverse capacity remains to carry traffic around the points of damage.

In- and out-bound traffic at the Japan Internet Exchange dropped by some 25 gigabits per second after the quake .. and then climbed back to robust levels by the end of the day.

Traffic at the JPNAP also seems to be down by only about 10% over its historical rates from the last two weeks.

The primary effects seen during the hours after the quake seem to have been related to breaks in 2 segments of Pacnet's EAC cable system. The plot at right shows prompt increases in unreachable networks in Japan and the Philippines, with follow-on events several hours later in Hong Kong and the Philippines. Various Philippine companies (BellTel, Eastern Telecoms, and Bayan) experienced outages that correlated in time with the initial Japan and subsequent Hong Kong events, suggesting common routing on the affected cable. But again, it's important to note that these are very small numbers of affected networks, relative to the total Internet presence of these countries.

Since the initial event, the Pacific Crossing system has also gone down. Based on experience from the Taiwan quake, it's possible that lingering damage to fibers, repeaters, and landing station equipment may continue to generate new problems over the coming days and weeks, even in cable systems that survived the initial event.

Still, it's clear that Internet connectivity has survived this event better than anyone would have expected. The engineers who built Japan's Internet created a dense web of domestic and international connectivity that is among the richest and most diverse on earth, as befits a critical gateway for global connectivity in and out of East Asia. At this point, it looks like their work may have allowed the Internet to do what it does best: route around catastrophic damage and keep the packets flowing, despite terrible chaos and uncertainty.

[Saturday 23:30 UTC] Added JPNAP traffic graph per comments. Thanks!

What Libya Learned from Egypt

2011-03-05T10:58:04Z

Libya's nationwide Internet blackout is entering its second full day. From a technical standpoint, it's clear that this is a very different strategy than the one used by Egypt in the last days of the Mubarak regime. The ultimate outcome is probably going to be the same. Let's take a few minutes to compare the two, and think about the implications for future Internet engagements in the Jasmine Revolution.

]]>

First, the facts as we know them. We observed nearly every host inside Libya becoming unresponsive on the afternoon of Thursday, March 2nd. You could attempt to "ping" them, send a traceroute along the path to them, try to retrieve pages, try to look up domain names ... but in nearly every case, there was no response. Simultaneously, we heard reports that all of the classic Internet communication services like Skype were down, and external websites were unreachable. To top it off, the Google Transparency Report showed query traffic from within Libya flatlining, and not recovering.

So far, these symptoms match what was experienced during the Egyptian Internet blackout pretty closely. But the underlying technical implementation couldn't have been more different. Look very closely at that Google plot again, and observe the floor. It's not perfectly flat, is it? That's because the Libyan Internet is actually still alive, even though almost all traffic is blocked from traversing it. The BGP routes to Libya are still intact, which means that the Libyan ISP's border routers are powered on and the fiberoptics are lit. In fact, we've identified a handful of isolated live IP addresses inside Libya, responding to ping and traceroute, and presumably passing traffic just fine. Someone in Libya is still watching YouTube, even though the rest of the country is dark.

Libya vs Egypt: A Different Strategy

Why did Libya put its Internet in 'warm standby mode' instead of just taking it down, as Egypt did? Perhaps because they're learning from Mubarak's experience. Cutting off the Internet at the routing level (powering down the Internet exchange point, going after the remaining providers with secret police to enact a low-level shutdown) was a technically unsophisticated desperation move on Egypt's part. It signalled to the world that the Egyptian government considered itself out of options, ready to cut off internal communications and external dialogue, looking for a last chance to turn off all the cameras and clean out the Square.

We expected to see something similar happen in Libya as the crisis came to a head, and on Thursday afternoon, the government appears to have taken action ahead of Friday's Day of Rage. Implementation was straightforward because of centralized control of the Internet economy: Libya doesn't have five independent Internet Service Providers with international connectivity, as Egypt did. They have just one, Libya Telecom and Technology (LT&T). Founded in 1997 and run by the Gaddafi family, LTT was folded into the state-owned GPTC (General Post and Telecommunications Company) in 2004. Each Internet route to Libya, and therefore all of the traffic to Libya, flows through this one provider's infrastructure. So on Thursday afternoon, like turning off a tap, the stream of traffic was slowed to a trickle, and then to a few drips.

This tactic makes all kinds of sense from the government's perspective. The Internet is a valuable wartime resource, like a critical bridge over which supplies can flow. As long as you can deny it to your enemy, you don't blow it up — you keep it intact for your own use.

Throttling the Internet to the point of uselessness, instead of killing it outright, also delayed International recognition of the fact that the Internet was down during the most critical period. Most international media didn't clue into the fact that the Libyan Internet had gone silent until after the sun had gone down in Tripoli on Friday. By taking a softer route to shutdown, the government deprived the opposition of much of the international "flash crowd" of attention and outrage that an unambiguous "kill switch" tactic might have garnered.

Conclusions

Using denial of Internet access as a political weapon during crisis events is all about timing and messaging. Mubarak waited too long to implement his blackout, and then let it run past the point where the damage to the Egyptian economy and the cost of international outrage exceeded the dwindling benefits to the regime. In the end, all the Egyptian government accomplished was to attract the sort of sympathetic attention and message support from the Internet community that is pure oxygen to a democratic opposition movement. You can't buy that kind of press!

Libya faced this same decision in the runup to civil war, and each time, perhaps learning from the Egyptian example, they backed down from implementing a multiday all-routes blackout. On 19 and 20 February there were two consecutive nights of routing-based blackout, but in each case, service was restored the next morning, at reduced levels. Through the course of the following week, traffic continued to grow, not only from Tripoli, but from eastern provinces where the government was no longer in control.

That message couldn't go unanswered. The current blackout, which probably signals the onset of the endgame, comes too late to contain the message. Together with restrictions on journalist movement, it will provide temporary cover for some of the endgame brutality, and for that reason, it's deeply sad.

When some future government faces this decision, backed into a corner by a popular uprising supported by Internet communication, they will probably reach the same conclusions that Libya and Egypt did: reestablish control over national communications at any cost, and pick up the pieces later. That's why the Internet is too vital to be left in the hands of centralized authority, and it's why you should press for more diverse Internet connectivity wherever you happen to live.