Articles By James Cowie

Hurricane Irene knocked out power to millions of homes and businesses as it travelled up the US East Coast this weekend. Even as the winds subsided, torrential rains triggered savage flooding throughout Eastern New York state and Vermont, tearing up roads and exposing the telecommunications infrastructure to further risks. The storm's impacts were clearly visible in the Internet's global routing table, as tens of thousands of networks were cut off from the rest of the world.

Here are a couple screenshots from our Internet Health Portal, which we provide to the US Computer Emergency Readiness Team (US-CERT). During an emergency like Hurricane Irene, this tool provides the US-CERT with critical information about the availability of Internet services across America. Working from lists of impacted customers in each state and county, and lists of correlated outage events, we can supply a lot of useful information about the problems being experienced by enterprises in the affected area. That information can be passed along to state and local governments to aid in prioritization of disaster relief.

We're still piecing together the data that can confirm or deny much of what's been reported overnight, but one thing is clear: something very strange was going on with Tripoli residents' Internet access. Service was restored suddenly in Tripoli, flickered on and off for a couple of hours, and then died, with the majority of the country's international BGP routes withdrawn from service for good measure. Today the routes are back in Tripoli, but ADSL service isn't. This morning we're looking back at this curious overnight episode, and speculate about what might have happened.

The following plot shows the number of Libyan networks (blocks of Libyan IP addresses) that appear in the global routing table. There are typically 16 of these, all routed by Libyan Telecom and Technology (LTT) via Telecom Italia. This week they have suffered some impairment, in groups of 6 or 10, in episodes that typically last no more than a few hours.

There didn't seem to be any pattern to these outages, which took place at all times of the day and night. It seems to suggest power outages, rather than permanent facilities damage, or deliberate action by the government (for suppressing communications, say).

Is it possible that LTT is suffering power outages, and having trouble finding fuel for their generators due to NATO's unofficial fuel blockade of Tripoli?

One reason why the world at large may not have noticed much in the way of Internet impairment: the affected networks don't seem to be the same ones used to access the majority of Internet content from inside Libya. Looking at Google's plots of inbound traffic from Libya, these substantial network outages seem to have had very little impact on the daily traffic curves:

In other words, the handful of Libyan networks that aren't affected by these outages seem to be the same ones that are consistently generating the Google traffic. If you're lucky enough to be an LTT customer in one of these Internet neighborhoods (presumably in Tripoli), your connectivity stays up. Everyone else, well, you're on your own. Time for Internet in a suitcase?.

Updated Monday morning to include detailed Syrian network map, and include one-second BGP plots during the day of outage. --jim

Thanks to everyone who's stopped by this week to read about the Internet blackout that affected Syria last Friday. We're always glad to hear your comments, especially when you fill in some of the missing parts of the story that aren't obvious from the data.

A lot of the questions we've received about Syria this week have speculated about a repeat performance of last week's Internet blackout. Would the Syrian Internet get shut off again this Friday, as it was last Friday? Would it be visible in the BGP routing table? Would this become a recurring tactic, just another mechanism for crowd control?

Initial data from Google's transparency report suggest that total traffic was down, even compared to a normally quiet Friday.

But looking at the broader picture, a couple things are clear. First, there was no repeat of last week's event, in which two-thirds of all Syrian networks became flatly unreachable from around the world, an Egyptian-style disconnection at a very fundamental level. This week, while traffic levels were reduced (perhaps throttled or rate-limited, as in Iran), the routes themselves remained intact.

If you wanted to reach a Syrian website Friday, or if a Syrian browser wanted to reach a European website, the paths were known and the lines were open. Whether you could actually get enough bandwidth to upload videos, or make a Skype call, was another story. We've heard some anecdotal evidence that there were connectivity problems. But anecdotes aren't data.

Speaking of data, that's another question we get all the time. How do we examine a market like the Syrian Internet, and make our call about the structural availability of routes to its networks? Let me take a moment and describe some of the technology behind these reports.

(Updates on the restoration of Syria's internet at the bottom of this page. --jim)

Starting at 3:35 UTC today (6:35am local time), approximately two-thirds of all Syrian networks became unreachable from the global Internet. Over the course of roughly half an hour, the routes to 40 of 59 networks were withdrawn from the global routing table.

This image shows the current state (green: reachable, red: unreachable) of each network prefix in the Middle East this morning, visualized as a packed Hilbert-curve representation. The size of the colored area is proportional to each country's Internet presence, so you can see that Syria's Internet (red block near the top center) is a little smaller than that of Kuwait.

Today's 8.9 magnitude earthquake in Japan has had surprisingly limited impacts on the structure and routing dynamics of the regional Internet. Of roughly 6,000 Japanese network prefixes in the global routing table, only about 100 were temporarily withdrawn from service — and that number has actually decreased in the hours since the event. Other carriers around the region have reported congestion and drops in traffic due to follow-on effects of the quake, but most websites are up and operational, and the Internet is available to support critical communications.

Libya's nationwide Internet blackout is entering its second full day. From a technical standpoint, it's clear that this is a very different strategy than the one used by Egypt in the last days of the Mubarak regime. The ultimate outcome is probably going to be the same. Let's take a few minutes to compare the two, and think about the implications for future Internet engagements in the Jasmine Revolution.

Final updates on this thread added at the bottom. New analysis here. --jim

Renesys confirms that the 13 globally routed Libyan network prefixes were withdrawn at 23:18 GMT (Friday night, 1:18am Saturday local time), and Libya is off the Internet. One Libyan route originated by Telecom Italia directly is still BGP-reachable, but inbound traceroutes appear to die in Palermo. A minority of our peers report some surviving paths through the peering connection between Level3 and Telecom Italia, but traceroutes into those prefixes fail, suggesting that the Libyan cutoff is complete.

We wondered whether anyone would repeat Egypt's strategy. Tonight, it appears that we have our answer.

Early reports from Algeria tonight suggested that another Internet takedown may be underway, similar to the one that affected Egypt. So far, however, we don't see confirming evidence for it.

Algerian providers get their international connectivity via submarine cables from Europe, with diverse transit from a long list of providers: Level3, Cogent, Telecom Italia, Telefonica, France Telecom, and Tinet. A pretty wide range of Algerian providers (Telecom Algeria, Wataniya Telecom Algeria, SPA Anwarnet, Smart Link, Orascom/Djezzy, etc.) have direct international connectivity, as seen in the BGP routing table. There's still no Internet exchange, or at least none that's widely advertised. All of that presumably makes a "kill switch" strategy somewhat more difficult to implement.

Algeria typically has about 135 routed network prefixes in the global routing table, and our data show that they are all still routed and relatively stable. Traceroutes inbound confirm that sites hosted in these prefixes are still alive, and spot checks of websites hosted in Algeria show that most are up and functioning normally. A few that we checked were unreachable, including the telecommunications regulatory authority (http://www.arpt.dz), the Prime Minister's office (http://www.cg.gov.dz), and other sites hosted at Djaweb (Telecom Algeria's hosting brand).

It's possible that new Internet blocks have been put in place that would not be visible from outside, such as Iran-style throttling or shutdowns of residential connections. We'll have to wait for some assessment of drops in inbound/outbound traffic levels to spot those subtler internal changes.

Latest updates on Wednesday's restoration of Internet service in Egypt can be found at the bottom of this page. We'll update through the day. cheers, --jim

Egyptian Internet providers returned to the Internet at 09:29:31 UTC (11:29am Cairo time). Websites such as the Egyptian Stock Exchange, Commercial International Bank of Egypt, MCDR, and the US Embassy in Cairo, are once again reachable.

Even before their communications blackout, Egypt really was a small part of the Internet in absolute terms, just a few thousand routable networks out of nearly 400,000 making up the global IPv4 address space.

To illustrate the point, we put together these images, which use a Hilbert curve representation of the Internet. The world's routed networks are in translucent grey, the unrouted networks are in black, and Egypt's networks are in orange. Look closely and you can see Egypt's Internet presence embedded in Africa and Europe's address space.

Thanks to all for great comments and questions. Please see below for latest updates on the ongoing Egyptian Internet blackout, including some trace-based analysis and a few words about neighboring countries. After this morning we'll be closing this post out, and looking for the restoration. Hopefully sooner than later. --jim

Confirming what a few have reported this evening: in an action unprecedented in Internet history, the Egyptian government appears to have ordered service providers to shut down all international connections to the Internet. Critical European-Asian fiber-optic routes through Egypt appear to be unaffected for now. But every Egyptian provider, every business, bank, Internet cafe, website, school, embassy, and government office that relied on the big four Egyptian ISPs for their Internet connectivity is now cut off from the rest of the world. Link Egypt, Vodafone/Raya, Telecom Egypt, Etisalat Misr, and all their customers and partners are, for the moment, off the air.

This has been an exciting month for those of us who study the Internet's infrastructure and think about ways to keep it running (and growing). Did I say exciting? Maybe "exhausting" would be more accurate. From China, to Iran, to the US Congress, everyone seems to be wondering how best to control the Internet and bring it in line with local law.

And then came the latest iteration of the WikiLeaks drama.

Did China's government really divert 15% of the Internet's traffic for eighteen minutes in April, effortlessly intercepting sensitive traffic in flight, and generally creating a massively embarrassing man-in-the-middle attack on vulnerable global communications?

Well, yes and no. Mostly no.

Yes, this event really happened. No, it probably wasn't a deliberate attack, or if it was, it wasn't a very effective one, compared to what might have happened, but that's where the story tends to bog down in technical detail and lose most readers.

Bear with me for a few minutes. There are several fragments of truth floating around this story, but they get more confused with each retelling. Let's lay out what's known, and then you can draw your own conclusions.

Afghans headed to the polls today for parliamentary elections in a tense but hopeful atmosphere. If the Internet has a role to play this year in helping Afghanistan develop a peaceful civil society, it will probably turn on two key developments: cheap GPRS Internet delivered over mobile phones, and strong relationships with neighboring states to provide Internet transit.

In today's followup to last week's blog, we present the evidence we see in the global Internet routing tables for a strengthening technical relationship between the Tehran and Kabul governments. In Afghanistan, as in Iraq, Iran now sees an opportunity to export influence by exporting its technological infrastructure.

Until this year, Iranian companies participated in the Internet primarily as consumers of international bandwidth. In 2010, however, they have expanded their scope. Earlier this year the Iranian state telecommunications company began providing Internet transit services in Afghanistan and Iraq, acting as a carrier for both commercial and government traffic. Over the next few days, we'll take a look at this interesting evolution, and speculate a bit about what it might mean for the growth of the Eurasian Internet.

The Internet infrastructure has been having a bad month. Not as bad as, say, the world's aviation infrastructure, but bad enough.

First, Chinese Internet censorship leaked out to a few massively unlucky users of the I root server. Then China Telecom failed to filter someone who leaked thousands of hijacked routes to other people's networks through them, probably by accident.

And then, inexplicably, Forbes went where no one had gone before (with a wink to Wired), and asked whether China might actually be testing a "cybernuke".

At first, this irritated me. Journalists and bloggers and blogger-journalists are fanning the flames of US unease about the growing role of China in world affairs. But then I realized that I could probably make tens of thousands of people read my blog, too, by jumping on the bandwagon. By all means, then, grab an MRE and hunker down in your Internet bomb shelter while I try to answer some of the obvious questions that came our way in the wake of the Forbes article:

• How would anyone build a cybernuke? What is that?
• Could a single actor, state-sponsored or otherwise, actually take down the global or regional Internet infrastructure of 2010, disrupt financial markets, throw civilization into chaos?
• How do I get my cybernuke movie screenplay optioned by Jerry Bruckheimer? His people won't return my calls.

I'm writing this blog entry from the campground at Vermont's beautiful Quechee Gorge, where I took the kids after work. Yes, Renesys is located smack in the middle of some of the nicest hiking, camping, and climbing on earth. No, you shouldn't move here, Northern New England has enough out-of-staters already, thanks. Unless, that is, you are an unusually talented web developer, have worked as a peering coordinator, or know the Internet transit industry inside-out, in which case you should send me your CV, posthaste. thanks, --jim

Here We Go Again.

Imagine an innocent BGP message, sent from a random small network service provider's border router somewhere in the world. It contains a payload that is unusual, but strictly speaking, conformant to protocol. Most of the routers in the world, when faced with such a message, pass it along. But a few have a bug that makes them drop sessions abruptly and reopen them, flooding their neighbors with full-table session resets every time they hear the offending message. The miracle of global BGP ensures that every vulnerable router on earth gets a peek at the offending message in under 30 seconds. The global routing infrastructure rings like a bell, as BGP update rates spike by orders of magnitude in the blink of an eye. Links congest. Small routing hardware falls over and dies. It takes hours for things to return to normal.

If you put 65 million people in a locked room, they're going to find all the exits pretty quickly, and maybe make a few of their own. In the case of Iran's crippled-but-still-connected Internet, that means finding a continuous supply of proxy servers that allow continued access to unfiltered international web content like Twitter, Gmail, and the BBC.

We've received enough interest about our previous notes on Iranian Internet connectivity that I wanted to give a brief update, and some reflections.

Seventy-five years ago today, on May 29th, 1934, Egyptian private radio stations fell silent, as the government shut them down in favor of a state monopoly on broadcast communication. Egyptian radio "hackers" (as we would style them today) had, over the course of about fifteen years, developed a burgeoning network of unofficial radio stations. They offered listeners an unfiltered, continuous mix of news, gossip, and live entertainment from low-powered transmitters located in private houses and businesses throughout Cairo.

It couldn't last. After two days of official radio silence, on May 31st, official state-sponsored radio stations (run by the Marconi company under special contract) began transmitting a clean slate of government-sanctioned programming, and the brief era of grass-roots Egyptian radio was over.

New York.--Senator Robert Bulkley, of Ohio, has made a proposal which is certainly worth considering.

It is as clear as daylight that, to bring about any sort of recovery, somebody must start some new sort of business or some extension of an old business.

It is also clear that nobody is in sight right now who has any notion of doing that — at least not in time to do this country any good as a depression cure.

There is one business which is a public business but is also a private one. This is the road-building business. The Government pays for the roads and hires the contractors. But the roads are built usually by private contractors and with materials furnished by private manufacturers.

If there is one thing needed in this country now, in view of the development of the automobile, it is express highways running east and west and north and south. Why, therefore, cannot the Government go into the business of building these highways?

Tough Times for Local Exchange Carriers

This week, the headlines seem to be full of fresh doom and gloom for wireline carriers, who employ people in every congressional district across America. Sooner or later, someone is going to call for Congress to tap some of the hundreds of billions in 2009 economic stimulus to help the LECs through troubled times, save lots of jobs, and preserve the way we do business in our critical last-mile communications infrastructure.

Is this wise? Is there a better way?