Renesys Blog

The Proxy Fight for Iranian Democracy

2009-06-22T11:30:00Z

If you put 65 million people in a locked room, they're going to find all the exits pretty quickly, and maybe make a few of their own. In the case of Iran's crippled-but-still-connected Internet, that means finding a continuous supply of proxy servers that allow continued access to unfiltered international web content like Twitter, Gmail, and the BBC.

]]> A proxy server is a simple bit of software that you run on your computer. It effectively lets you share your computer with anonymous strangers as a "repeater" for content that they aren't allowed to fetch themselves. For example, an Iranian web browser might be manually configured to use your computer (identified by an IP address and a port number) as a Web proxy. When your anonymous friend reads twitter.com, or posts a tweet, the request goes via your computer, instead of to Twitter's web server directly. Except for a little delay, and the fact that your friend gets to see what the uncensored Internet looks like from New York or London or São Paolo instead of Tabriz or Qom, surfing through a proxy is pretty much like surfing without one.

As you might imagine, open web proxies are valuable commodities in places where it's forbidden, possibly dangerous, to surf the Internet. Iran's opposition movement has been vigorously trading lists of open proxies over the past week. And as you might further imagine, the Iranian government censors have worked overtime to identify these proxies and add them to the daily blacklists.

As an experiment, we geolocated a list of about 2,000 web proxies (unique IP addresses and port numbers) that were shared on Twitter and other web sites over the course of the last week, to see if we could discern patterns in the places that are hosting them. Most of these are no longer reachable from inside Iran, of course, precisely because they were made public. The following map shows the distribution of those proxies worldwide.

The USA and Western Europe were well-represented, but so were China, India, Russia, Romania, Bulgaria, Vietnam, ... 87 countries in all, a pretty impressive breadth of representation, considering the relatively small size of this sample. (You can also see about a dozen Iranian IP addresses represented in the set. Not surprisingly, all but one of these belong to networks originated by DCI, the government-run service provider who operates the modern-day Internet equivalent of the Alamūt Castle.)

Here's a geographic visualization of the proxies, drawn in Google Earth. In the first one, we've drawn Iran in green, with some of their domestic network sketched in white, and their major international connections drawn in red. Each of the colored arcs represents a single open web proxy; they are "fountaining" out of a cable landing or Internet traffic exchange point that makes approximate sense for their Iranian Internet routing. For example, all of the web proxies in Europe are drawn from the Marseilles termination of the Sea-Me-We-4 cable. The web proxies in Turkey are drawn in light blue, radiating from Ankara, where the Iran-Turkey gas pipeline passes through on its way from Bazargan. Those unusual Iranian proxies emerge from Tehran, and so forth.

If we rotate the globe, you can see how the countries of Asia are doing their part to keep the bits flowing in Iran. India, China, South Korea, Taiwan, Vietnam, and Japan are all visible sources of web proxy activity.

I'd like to be able to say that these maps are a measure of the strength of the democratic impulse and volunteer spirit in all the countries of the world. But that might be a stretch. You see, looked at another way, an open proxy is a security hole, something you might find in a machine that's been compromised, or at the very least, badly administered. Security purists think of them as the "unlocked gun cabinet" of the Internet — a resource for anyone who wants to abuse a website, commit fraud, cover their tracks.

Some of the proxies in this dataset are undoubtedly fresh, created by people who want to keep the Internet alive for the Iranian people. But many of these proxies have probably been around for months or years, mapped out by those that map out such things.

We did see a few organizers try to explain the concept of an ACL (Access Control List) to all the new proud parents of open proxies. If you are diligent, it is possible to restrict the anonymous users of your new proxy to just the Iranians, or even just the Iranian non-government networks, if you have a good enough list of the IP address blocks (network prefixes) in question. But I expect that the complexity of configuring anything tighter than an "open access" proxy is going to prove too high a barrier to entry for most people who might volunteer to run one.

For one thing, we know how hard this is. Renesys has pretty good lists of per-country networks and their transit patterns, based on our analysis of the global routing tables, and trust me, they take some work to maintain. And even given good maps of Iran's address space to work from, ACLs are notoriously hard to test, if you don't have Iranian friends who can try your server from inside the protest zone and report back to you with problems. Most people aren't going to bother, and that's probably okay. Freedom is messy. There'll be time for security later.

Perhaps the strangest thing of all, given how diverse and active and vocal the proxy server farmers have been, is that by and large, it isn't working. The rate with which new proxies are being created has slumped over the last few days. It's getting harder and harder to propagate new proxies to the people who need them, as the government consolidates its hold on the filtering mechanisms. Any new proxy addresses that are posted to Twitter, or emailed, will be blocked very quickly.

People we talk to inside Iran say that almost no proxies are usable any more. Freegate, a Chinese anti-censorship application that makes use of networks of open proxies, has proven popular in Iran. But this week, it, too, has been experiencing problems. Many popular applications, like Yahoo! Messenger, have stopped working. The authorities are said to be using power interruptions as a cyberweapon, causing brief outages during rallies that cause computers to reboot, just as people are trying to upload images and video. The net result, as Arbor's excellent analysis shows, has been a drastic reduction in inbound traffic on filtered ports since the election.

If there's a lesson here for the rest of the world, perhaps it's this: Install a few proxy instances on machines you control. Learn how to lock them down properly. Swap them with your friends overseas who live in places where the Internet is fragile. Set up your tunnels and test them. And don't wait until the tanks are in the streets to figure this out, because by that point, you may have already lost the proxy war.

Iran and the Internet: Uneasy Standoff

2009-06-16T21:21:25Z

We've received enough interest about our previous notes on Iranian Internet connectivity that I wanted to give a brief update, and some reflections.

]]> In short: Iran is still on the Internet. As the crisis deepens, people are literally risking their lives by continuing to use the Internet for coordination and communication. Iran's physical connectivity to the Internet is so centralized, and so fragile, that it's within the power of the government to simply "turn it off" if they so desired. And yet, they have not done so.

Except for a brief period of outage over the weekend, the routes into Iran from the rest of the world have been basically intact, if a bit congested and unstable. Most of that congestion and instability is probably the result of six billion people who are freshly interested in Iranian politics, all reading (and in some cases, yes, attacking) Iranian websites. We aren't making things easier for the people inside Iran, who need that same bandwidth to get out their images and observations and tweets.

To show that nothing much has changed structurally, consider the fact that the same lineup of six international carriers are still carrying the megabits back and forth to the government's monopoly points-of-presence at the Iranian border. (See the following chart, which shows how the relative percentages of routes to Iranian networks carried by those providers has changed over the last few days.)

What happens inside Iran to those bits is anyone's guess (censorship, site blocking, traffic interception, and harassment, from all accounts). But the pipes are open and the traffic is flowing. In a few cases (which I will not detail, for obvious reasons), there are actually direct paths to international carriers, in defiance of government monopoly, that are now getting good use.

I've talked to a lot of people in the last few days about this puzzle. In a crisis that verges on revolution, the first thing a government typically does is take control of the media that can be controlled, and shut down the media that cannot.

Why is it different this time? There seem to be three basic theories.

The cynics. Perhaps the government has left the Internet intact so that they can use it to surveil and round up dissidents. Perhaps they even put bandwidth constraints in place to make it easier to cope with the volumes of traffic that need to be captured and filtered.
The optimists. Perhaps the government has realized that a modern economy relies on the Internet to such an extent that it cannot be turned off, for fear of disrupting financial transactions and business communications. Iran's Internet ecosystem is relatively rich, and the impact on their economy of a sustained Internet shutdown would be significant. Why make it harder for companies to do business in Iran at a time when oil revenues are cratering and foreign investment is looking for reasons to take a walk?
The realists. Perhaps the government is too busy with other things to worry about the Internet. Governments aren't well-suited to run the Internet, and they don't completely understand how it works. The Internet has never been "turned off" before, and it would take creativity and thoughtful action to figure out who to ask in order to get it done. So it simply hasn't happened, and probably won't. Good thing, too, because they might not be able to turn it on again.

You can pick the theory you like. I guess I'm a realist, but I'd like to be an optimist. If you wait long enough, something good can come out of something bad. If fertility rates hadn't tripled during the long Iran-Iraq war in the 80s, Iran wouldn't be faced with this demographic bulge of restless 20-somethings who have grown up with the Internet and expect it to keep working.

You can fight people, but you can't fight human nature, and you can't fight demographics. Iran is going to follow the same long-term trend as the rest of the developing world: building a civil society based on free expression, global communication, free mobility of human capital, and cross-border investment. That's what the Internet symbolizes, and yes, if you ask these folks, it's worth fighting for.

Strange Changes in Iranian Transit

2009-06-14T12:33:22Z

Many media sources have reported outages in Iranian mobile networks and Internet services in the wake of Friday's controversial elections. We took a look at the state of Iranian Internet transit, as seen in the aggregated global routing tables, and found that the story is not as clear-cut as has been reported. ]]> There's no question that something large happened in the Iranian telecom space, and that the timing aligns with the close of voting and the emerging controversy. Iran typically has a fairly high baseline level of sporadic route instability, due to the country's highly centralized incumbent transit through DCI (Data Communications Iran, AS12880) and DCI's somewhat peripheral connectivity to the main east-west conduits for data. Even so, we started seeing spikes of route instability (changes in the paths to Iranian IP space) starting around 08:05 UTC on Saturday (just after noon in Tehran) that were significantly larger than normally expected. These bursts affected as many as 400 prefixes (blocks of IP addresses) — the majority of Iran's Internet presence.

At 17:48 UTC, instability turned into outage, as more than 180 Iranian networks were withdrawn from the global routing tables, indicating that there were no remaining paths into DCI for that portion of Iranian traffic. Contrary to media reports, however, the outages were fairly short-lived. Within a few minutes, half of the outaged population were restored to alternative transit; over the course of an hour, outage levels returned to their normal baseline. Route instability continued to be fairly high, and that pattern has continued through the night and into Sunday.

What can we say for sure? Not much, except that Iran remains well-connected to the Internet from a routing perspective. If I had to guess, I'd say that there are probably a lot more people around the world pulling local content from Iran's providers right now, and that surge of demand is probably contributing to increased congestion and (perhaps) some of the route instability we see. It wouldn't be unusual for there to be some inbound cyber-mischief as well, from supporters of one or the other side, but so far we only have rumors on that front.

It is interesting to note that the changes in routing that took place were very specific in their impact on DCI's various transit providers, who keep the country connected to the world. There are six of them: Turk Telecom (TTNet, AS9121), FLAG (AS15412), Singapore Telecom (AS7473), PCCW (AS3491), Telia (AS1299), and Telecom Italia Sparkle (AS6762). As the following plot shows, five of them lost Iran's transit, and one of them (Turkish Telecom) was a big gainer. (Red arrows indicate loss of transit preference from the outside world; green indicates a gain in transit via the given provider.)

A transit shift of this magnitude may indicate that something (administrative, or physical) has affected Iran's connection to the submarine cables running east and west — not a total outage, but some kind of significant impairment. Turkey has their own, interesting arrangements with Iran for transit, and those are still in good shape (perhaps somewhat congested, having presumably doubled or tripled in transit volume). It wasn't unusual to see 300ms traceroutes from North America and Europe in this timeframe to many Iranian sites.

Of course, you have to remember that globally visible routes are the signposts for inbound traffic to and through DCI to the local providers; from the outside, there's no telling what the Internet experience of the average person inside Iran is like today. It sounds as if a lot of content is being blocked within the country. For now, it's a good sign that information continues to flow, and Iran is still connected to the world at large. Let's hope they stay connected.

How a Resilient Society Defends Cyberspace

2009-05-30T02:45:48Z

Seventy-five years ago today, on May 29th, 1934, Egyptian private radio stations fell silent, as the government shut them down in favor of a state monopoly on broadcast communication. Egyptian radio "hackers" (as we would style them today) had, over the course of about fifteen years, developed a burgeoning network of unofficial radio stations. They offered listeners an unfiltered, continuous mix of news, gossip, and live entertainment from low-powered transmitters located in private houses and businesses throughout Cairo.

It couldn't last. After two days of official radio silence, on May 31st, official state-sponsored radio stations (run by the Marconi company under special contract) began transmitting a clean slate of government-sanctioned programming, and the brief era of grass-roots Egyptian radio was over.

]]> Was there something in the air in the early summer of 1934? Two weeks later, the United States Congress passed the Communications Act of 1934, which established the Federal Communications Commission, and gave it regulatory authority over not only the US airwaves, but interstate telephone services and telegraphy -- in short, all forms of over-the-air and wireline communications among private citizens. In the United States, as in Egypt, the government's oversight of public and private communications has survived, basically intact, for the last seventy-five years. Telephony replaced telegraphy, and television supplanted radio, and the Internet threatened to absorb all of them in a wave of convergence. But throughout all of the technological change, the US government's role as regulator and defender of the public interest seemed to have survived in recognizable form.

That's why it's refreshing to see at least some evidence, in today's release of the long-awaited Cyberspace Policy Review, that the Obama administration is taking a long, careful look at how to update the Government's role in "protecting the Internet." The report nods to regulatory precedent, but makes it clear that new regulations are not the first order of business. Instead, the themes of the day are those recognizable to any digital native: bottom-up coordination, transparency, sharing, establishment of trust, resiliency in the face of unknown threats. Leadership comes from above, but it takes the form of continuous measurement and accountability and consensus-building.

Sounds kind of "kumbaya," right? Well, yes. But I suspect there's actually some substance behind the rhetorical styling.

For one thing, the President has consistently, though subtly, identified "resiliency" as a key virtue in the various complex systems that make up the critical infrastructure behind American society. Having just merged the Homeland Security Council and National Security Council, the President is building an integrated National Security Staff that will include both a senior cybersecurity advisor (yet to be named), and a more general "resiliency policy" advisor, who will focus on survivability and flexible response. The emphasis on resilience, on building complex systems that are less brittle and more intelligently flexible and responsive in the face of attacks, are the kinds of wonky concepts that light up the eyes of computer scientists and mathematicians who study complex systems. They aren't there to provide sound bites for the Sunday talk shows (and, indeed, concepts like "resiliency" tend to get a lot less play in the popular press than firmer fare, like "eliminating vulnerabilities" and "shutting out attackers" and even, God help us, "turning off the Internet in case of attack").

The very fact that the report has been some time in arriving, and that it does not contain a laundry list of quick fixes, suggests that the President intends to be appropriately deliberate in building a picture of what, exactly, the government can do to improve its own cybersecurity posture. Of course, it also signals how thickly political the whole cybersecurity issue has become, as various factions battle it out for control and presidential access.

I think we can take some cheer in the fact that the President isn't rushing to action, that the balance between safety and civil liberties appears front and center in the terms of debate, and that there's a general awareness of how much economic and strategic value is created for the American people by the wild global Internet, even as it creates some novel (but managable) risks for continuation of government and critical infrastructure.

1934, thankfully, is a long time gone.

Reaching Google via Asia?

2009-05-15T01:42:17Z

Across the Internet, yesterday, Google users twittered, blogged and emailed that Google search and mail were not usable. And, yesterday afternoon, on Google's official blog, Urs Hoelzle reported that Google "direct[ed] some [...] web traffic through Asia".

]]> Renesys observes the topology of the Internet by collecting routing data from a wide variety of vantage points (peers). Different peers may choose different routes to the same destination (e.g. Google) depending on their location in the Internet. Analogously, a driver starting from Los Angeles will choose a different highway to reach New York City than a driver starting in Montreal, Canada. On the Internet, each peer may choose a different best path or route to a destination. By recording the choices each peer makes, we can gain some insight into the relative importance of a given route to a destination.

The Internet is carved up into IP blocks (called prefixes) which make management of the network easier. Each provider learns about the prefixes of other providers by passing the knowledge around (like a game of 'telephone'). Google directly advertises or provides service to approximately 190 different prefixes, so the Internet learns how to reach Google based on the relationships Google has.

When trying to reach Google, then, any inbound traffic to them needs to pass across other providers with whom Google has a BGP routing relationship. We'll look at a very obvious shift in the number of other Internet providers who started reaching Google differently during yesterday's event.

Even though Google is very well-connected to a diverse set of other networks, two of these relationships predominate. It is through these relationships that much of the Internet reaches Google. If we look at Google's primary providers, Level 3 (ASN 3356) and AT&T (ASN 7018), we see that these are very important relationships to Google, since in the case of Level 3, just under 80% of peers choose to reach Google via Level 3 for just over half of all of Google's prefixes.

During yesterday's routing instability, Google's prefixes shifted to different relationships. When automatic, this sort of change is a key feature of redundancy. Sometimes such changes are motivated from a business change. Over the course of yesterday's instability, virtually all of our peers, at one time or another, chose to reach many of Google prefixes through NTT (ASN 2914), what had been a far less important relationship.

Note that the bar chart above is showing the total percentage of Google's approximately 190 prefixes and the total percentage of peers (different vantage points on the Internet) choosing to reach Google through NTT. At no single time did the entire Internet choose to reach Google through NTT, but over the course of the event, almost everybody tried to reach Google on this path.

Before yesterday's event, NTT was not so important an inbound path to Google. In fact, NTT carried very few of Google's prefixes (only 16) and was the preferred path to Google for only 10% of our peers. During several different bursts of announcements, this relationship became a much more important inbound path to Google.

In one particular series of routing changes, which started at around 14:45 UTC, several major providers, e.g. Deutsche Telekom (ASN 3320), Teleglobe (ASN 6453), Telia (ASN 1299), and even, peculiarly, AT&T (ASN 7018) started choosing to reach Google through NTT.

While yesterday's instability was disruptive to Google users all over the Internet, we are happy to report that it's back to business as usual with one of the most robust services on the Internet.

AfNOG Takes Byte Out of Internet

2009-05-05T20:47:11Z

A couple of months ago, we discussed how a small Czech provider ended up causing global Internet mayhem by tickling a Cisco bug via a rather ridiculous routing announcement. While it's easy to fault the instigator of this meltdown, ultimate responsibility belongs with the vendors of poorly tested code. If we've learned anything in decades of software engineering, it is that you can't assume anything about user input. If you don't check that input for validity, you are not just being careless, you are creating a time bomb that will eventually go off. Another such bomb went off on Sunday, 3 May 2009, taking out a large swath of the Internet. We recount the sorry tale here.

]]> Sundays are usually pretty quiet from an engineering perspective on the Internet, so we were surprised by a sudden and prolonged influx of routing updates at our collection sites. Thankfully, we have some very nifty event correlation software, written by Renesys über-hacker Alin Popescu, to which I turned to see what was going on. Alin's code sorts through all the routing activity on the Internet in real time and correlates activity via the attributes they have in common. We can then say things like "this spike in outages came from networks in Kyrgyzstan as transited via Golden Telecom in Russia", potentially providing some insight into the root cause of the problem. It's really very slick. However, the only problem with pinpointing Sunday's event was that it occurred all over the place. We saw sharp spikes in outages and instabilities in Indonesia, Bulgaria, Germany, Romania, Brazil, Russia, the United States and many other places. Upwards of a couple thousand prefixes (networks) were impacted during a period of about 6 hours. Given the wide reach of the event, we immediately thought of the Czech situation and wondered if this was another live Internet QA test. Turns out we were right.

To understand the problem, let's first provide some background. As readers of this blog will know, every organization responsible for Internet routing has two things under their control: block(s) of IP addresses (prefixes) and an identifying number, or ASN. Like IP addresses themselves, ASNs are also running out, at least the old style ones. The original Internet designers allocated only 2 bytes for storing ASNs, which means at most 65,536 of them. Since then, ASNs have been largely allocated consecutively starting at 1, with Phosworks Drift & Services getting the most recent allocation of ASN 49232. Reserved (private) ASNs are at the high end of the range, starting at 64512. So this means that at present we only have about 15 thousand 2-byte ASNs remaining unassigned.

If there weren't an alternative, the 2-byte limitation for ASNs would mean that before too much longer, there would be no new members of the Internet routing club: no new ISPs and no companies with multiple providers and direct control of their routing. Fortunately, there is an alternative available today, and that is 4-byte ASNs. RIPE has been assigning them by default since the start of 2009. But unlike IPv6, the proposed alternative to dwindling IPv4 addresses, 4-byte ASNs can seamlessly and safely co-exist with 2-byte ASNs. No complete overhaul of the Internet is required.

Returning to our story, if 4-byte ASNs are available and interoperate with 2-byte ASNs, there shouldn't be any problems with announcing them anyway we want today. Yeah right. The African Network Operators' Group (AfNOG) will soon be meeting in Cairo, Egypt, and plans to be multi-homed. They acquired the 4-byte ASN 327686, which equals 5*(2¹⁶) + 6 and hence is also written as 5.6, and planned to use it as stated here:

One of the aims of this setup is to demonstrate that 32-bit ASNs do work and people should not steer away from them, especially since the pool of 16-bit ASNs is shrinking fast. A showcase network goes a long way in this regard.

Now, AfNOG does have an IPv4 prefix, 196.200.208.0/20, and a traditional 2-byte ASN, namely 37095, at their disposal. On 30 April at 17:21:49 UTC, we started seeing the 2-byte ASN announcing this prefix, but with ridiculously long AS paths like

24863 37095 37095 37095 37095 37095 37095 37095 37095 37095 37095 37095

AS 24863 is LINKdotNET, a local Egyptian provider. So far, so good. Then on 3 May at 12:00:30 UTC, the above paths get replaced with 4-byte AS prepends, namely,

24863 327686 327686 327686 327686 327686 327686 327686 327686 327686 327686 37095

and the global impact is felt immediately, as shown in the following graph.

So what happened? Well, not everyone can afford a Cisco or Juniper router. Some folks make do with cheap Linux boxes running the free routing software called Quagga. And you guessed it! Older versions of Quagga have a buffer overflow problem, dropping the associated BGP sessions like a brick when they saw this announcement. The latest Quagga release, 0.99.11, came out on 2 October 2008. The prior version had the following gem of a comment.

  /* ASN takes 5 chars at least, plus seperator, see below.
   * If there is one differing segment type, we need an additional
   * 2 chars for segment delimiters, and the final '\0'.
   * Hopefully this is large enough to avoid hitting the realloc
   * code below for most common sequences.
   *
   * With 32bit ASNs, this range will increase, but only worth changing
   * once there are significant numbers of ASN >= 100000
   */

Or until someone prepends a 4-byte ASN a few times. Talk about lame. So this latest experiment in real-time Internet QA identified everyone running old versions of Quagga, thanks to the time bomb left behind by one very lazy programmer. The latest version of Quagga appears to handle this condition a little more carefully, but I can't help but wonder about any software version that starts with a zero.

Looking for the positive, it was nice to see some diversity on the Internet with respect to routing. Imagine if this had been a Cisco bug? Still we found it surprising to see the number and variety of organizations running Quagga in a production environment. The following graph gives the per-country distribution of prefixes for every country with at least 10 outages due to this bug.

The lessons are obvious. If you don't check user input and don't allocate sufficient memory, bad things will ultimately happen. Every possible permutation that you can think of, and perhaps some that you can't, will eventually be tried by someone, either accidentally or on purpose. So far, we're mainly seeing the former, which is good news. Ultimately though, bugs need to be found in lab, not in the production Internet.

The Blind Routing the Blind

2009-05-04T11:00:00Z

In our last blog entry, we talked about measuring the state of routing anarchy that exists on the Internet on a per-country basis. We looked at every routed network (prefix) by country of origin and tried to answer the question: do folks do what they say and say what they do, as articulated via routing registries? Although many manage to administer their routes with care, the overall results are quite varied. And without some way of verifying routes via some authoritative source, we are left only with the current system of believing everything we're told and hoping for the best. The dangers of such a system are demonstrated dramatically from time to time.

Although they certainly could, countries typically don't exercise any control over the routing hygiene of the companies operating within their borders. Countries might tax those companies, filter their traffic for objectionable content, mandate the types of software or equipment they can use and even spy on them, but if a company wants to screw up routing on the global Internet, well that's their business. As we've noted in the past, no driver's license is required on the Information Superhighway, as there are essentially no rules, regulations or enforcement. So in this blog entry, we'll apply our scoring idea to those who can easily effect change, namely, those organizations who are ultimately responsible for how traffic flows on the Internet.

]]> At first, this exercise might seem pretty straightforward: pick your favorite company or organization, compare their stated routing policies to reality, and give them a score based on the difference. But on the Internet, nothing is ever so simple, especially when it involves any sort of attribution. Simply identifying a company's Internet presence is complicated by mergers, buyouts, spin-offs and other aspects of modern capitalism. How can you hope to sort it all out? Well, there is only one thing you can really count on: anyone managing routing on the Internet must have one or more blocks of IP addresses (prefixes) and at least one "identifying" number, an Autonomous System Number (ASN). At some point in time, these things must have been doled out by some authority of sorts. That's about it. The person who requested these values (essentially a bunch of integers) might be long dead as might be the original company. And the associated administrative records might no longer be accurate or even maintained, but some IP addresses and an ASN are all you need to route traffic on the Internet. Once you've acquired these, you are free to proudly assert your presence to humanity and thereby consume a little bit of memory in every router on the planet.

In fairness, the overwhelming number of organizations we see on the Internet have only one or a handful of prefixes. These organizations are easy to identify, as they have little to begin with and hence little to have gone stale. But large companies and old timers to the Internet are rarely so fortunate. Let's take AT&T as one example. In the US, AT&T was the phone company for a very long time, until they were broken up by the courts and a newly formed AT&T became just a much smaller part of the original whole. This greatly diminished AT&T was ultimately acquired by SBC Communications, as was Ameritech, Bell South and others, and eventually the entire collection was renamed to simply "AT&T".

So what does AT&T look like on the Internet today? Old hands will think of it as ASN 7018, but we can actually associate almost 100 different ASNs with this single organization, about half of which are announcing prefixes. These are registered under a variety of names such as AT&T Network Systems, AT&T Internet Services, AT&T Global Network Services and almost 40 others. We even still see names containing SBC (and no mention of AT&T) and the now extinct Rogers AT&T Wireless (AS 20453). However, AS 20453 continues to announce Rogers' prefixes (over 70 of them) on the Internet, but AT&T is not one of their providers. So while Rogers ended their partnership with AT&T, no one updated the associated ASN registration. Talk about confusing.

The situation looks grim until you realize that there are actually three sources of (potentially imperfect) information that can be combined to help ferret out the truth:

registration information for ASNs and prefixes
current routing information
open source intelligence, e.g., news, press releases, Wikipedia, etc

I won't go into the details here, but we employ all three sources appropriately in an attempt to categorically define each organization on the Internet. While registration information is often incomplete and outdated, strings like AT&T and Verizon are unlikely to appear outside of those organizations, whereas others like Internet or Inc. have essentially no informational content. Other terms can be associated because of known business relationship or brand names, such as Microsoft and Hotmail or Sprint and Nextel. But since registration information can't be trusted, the associations that are uncovered need to be checked via routing data. This is how we discovered the Rogers AT&T Wireless problem. It just didn't make sense that an "AT&T entity" was not routing through AT&T and a quick Internet search confirmed that Rogers AT&T Wireless did not belong to the AT&T organization.

Intelligently analyzing these different inputs, we can finally list out all the organizations on the Internet: their ASNs (if any) and associated prefixes. And once we have that, we can give them a score based on their routing hygiene, exactly as we did in our blog entry for individual countries. Despite the fact that there are only around 31,000 unique ASNs seen on the Internet at present, we manage to find around 75,000 organizations, as not everyone with a registered prefix also has an ASN. (In these cases, their provider typically handles the routing of their prefix.)

We found over 13,000 organizations who managed a perfect score of 100, although all but 218 of them had fewer than 10 prefixes. While a small number of prefixes makes their administration easier, it is not a guarantee of success. One organization with 2 prefixes scored an embarrassing 9 out of a 100 on our scale. For those with at least 10 prefixes to manage, the following scatter plot gives the distribution. As we saw with countries, the more prefixes you have, the harder it is to keep everything straight and attain a good score. And while many manage to do a very good job, the overall results are quite varied, especially when you recall that doing essentially nothing will land you around 25 on our scale, and you have to do less than nothing to rate below that. Still almost 5,000 organizations manage to score below 25.

The overall winner in this exercise was Kazakh Telecom with a perfect score of 100 for over 150 prefixes. Airtel Broadband and Telephone Services (ABTS) came within a quarter of a point of a perfect score for their couple of hundred prefixes. At the other end of the spectrum, China Enterprise Communications Ltd. (CEC), in which Tata Communications recently acquired a 50% stake, barely managed a score of 20. CEC is around one point behind our second to last place finisher, the AARP, a US non-profit organization for people over 50. Overall, it's clear that there is a lot of room for improvement for most organizations, but if the Internet is ever going to have any measure of accountability, it is going to have to start with those who control the IP space and orchestrate global routing. Without attribution, there can be no accountability and no hope of improving upon the current situation.

Update: The earlier version of this blog inadvertently misclassified a number of registered networks due to a software bug in a script used for displaying the data. The scoring algorithm remains unchanged.

Route Hygiene: The Dirt on the Internet

2009-03-25T20:00:00Z

Since Renesys maintains large quantities of data on the Internet going back many years, we sometimes get the question: If you guys are watching the entire 'net, why don't you just warn people when things break? My response is generally along the lines of: Sure we can do that. Simply tell us the correct state of the Internet at each moment in time and we'll alert you to any operational differences we observe. This is generally met with silence.

Renesys can tell you a lot about the current state of the Internet, but absolutely no one can tell you the correct state. And that is because no one is in charge, and so there is no central authoritative source of information. Think of the Internet as a highway system where anyone can buy a car and simply start driving: no need to register the car, attach a license plate, buy insurance or get a driver's license. You don't even have to show an id or be sober. Just pay some fees, buy some equipment, hook up and go. The barrier to entry really is that low.

Obviously, this arrangement can cause some problems. When Pakistan hijacked YouTube last year by announcing YouTube IP space, out of the hundreds of thousands of routing announcements seen on Internet, how was anyone to know this particular one was incorrect? Okay sure, you couldn't get your videos, but maybe YouTube had just opened a data center in Karachi and the problem was internal to them? Without some way of checking the authenticity of routes, the routers that direct traffic on the Internet simply believe what they are told. And if the best route to YouTube appears to be via Pakistan, then they are all going to use it, no questions asked. This is not a new problem, and this blog explores an old and largely failed attempt to address it. We then compare the differences between countries with respect to their routing hygiene.

]]>

The intended solution to this problem was developed way back in the early 1990s when the concept of a routing registry was first proposed. The idea was to construct a common database of routing information maintained by the appropriate parties, which could then be used to debug operational problems and filter out incorrect routes. So for example, YouTube might have registered its blocks of IP addresses (prefixes) with a registry saying that these prefixes belonged to them. They might further declare that their prefixes should only be seen originating from the YouTube Autonomous System (AS 36561), which in turn has the following list of providers. And so forth. The typical language for expressing your routing policy, known as RPSL, is very rich, if somewhat arcane, and you can fully describe your Internet presence using it. This allows other 'net citizens to figure out if what they observe about you on the Internet matches your stated policy.

The only problem with registries is that, like most things on the Internet, no one is required to use them or, if they do use them, is required to keep them up-to-date. In addition, there are dozens to choose from. Which one do you pick? Some are regional, others are run by individual countries, nonprofits, or even Internet service providers. Of the 39 registries that Renesys monitors, the nonprofit Merit Networks Inc maintains the registry with the largest amount of routing information (the RADb). In fact, it has more than double the second largest such service, run by RIPE NCC. Others of note are run by Level 3, Savvis, NTT, and ARIN. So as with most things on the Internet, there are potentially many sources of information and there is no telling how much of it is incomplete, outdated or otherwise erroneous. And while registry data is widely viewed as unreliable at best, we set out to answer the question more definitively. For a moment in time (20 March 2009 at 00:00:00 UTC), we compared the true operational state of the Internet to the explicitly listed "correct" state as articulated in any one of the aforementioned 39 registries.

To conduct this comparison, we first had to come up with a scoring mechanism. We used prefixes as the basic building blocks for our scoring approach because these can then be rolled up into more meaningful higher level scores, such as by country or by provider. For each prefix, we looked at only two things: how is the prefix originated and who are the upstream providers for the originator? We compared the registry entries (if any) to what we were actually seeing on the Internet and gave each prefix a score based on the variance. The best score is 100 and each prefix starts there — innocent until proven guilty. Then points are deducted for each difference found between registry data and reality. The exact algorithm is not terribly important and competing alternatives are easy to suggest, but what is important is that we tried to set the bar very low, making a good score easy to obtain. For example, correct originations are worth much more than correct upstream providers. If a prefix has a correct origination and only one upstream, it will still get a score of 80 even if the upstream is incorrectly registered or missing altogether. For two incorrect or missing upstreams, the score drops by only a few points. Thus, correctly originated prefixes tend to have scores in excess of 70. And if you register nothing at all for a prefix, its score will be around 25 or slightly less, depending on the number of providers. To get lower scores, you have to do less than nothing, namely, make lots of mistakes in your registration, enough to outweigh anything you managed to handle correctly.

Once each prefix is scored in this way, we compute the average scores for all prefixes in certain classes, such as those that geolocate to a particular region or are ultimately transited by a given provider. We can then compare countries, providers and organizations for their level of routing hygiene, i.e., how close does their presence on the Internet match their stated policy? My expectation was that there would be a few folks who got things completely right, but by and large, most would either not have registered their routes or not have done so correctly. In other words, I expected most of the world to map to a score of around 25, with a few outliers around 100 and very little in between. That turns out not to be the case. The average score for all of planet Earth was fairly respectable 65.6 for around 280,000 routed prefixes. And there was quite a wide range of scores and surprising differences between countries.

The above scatter plot displays our per-country scoring results. Each point on the graph represents one of the 228 countries or territories that we see originating prefixes on the Internet. Since it is much easier for countries with few prefixes to get a high score, we plot the prefix count versus the total score for each country. Thus, we'd expect countries lower on the graph (fewer prefixes) to have higher scores, but that is generally not the case. Five tiny countries manage a perfect score of 100, namely, San Marino, Djibouti, Faroe Islands, Åland Islands, and Cape Verde. The bottom three, all with scores below 25, are also quite small: Saint Helena, Equatorial Guinea, and Senegal.

It is important to compare countries with a similar number of prefixes, i.e., those that appear in the same horizontal band of the graph. For example, Ukraine is in first place in their band (1,000 — 10,000 prefixes), while Chile is last. The US is in its own band as the only country with over 100,000 prefixes, but comes in with a below-average score of 50.5. Of the major powers, Russia is way ahead of the others with an 88.4, while China manages only a miserable 45.4. And finally, we note that Korea (84.9) nudges out India (84.0) to win their band.

If you ignore prefix count entirely, the US ranks 191^st out of 228 countries. If instead you consider only countries with at least 1000 prefixes, i.e., those with a rich Internet infrastructure, the US ranks 34^th out of 39 countries, just behind Mexico and just ahead of Colombia. The top 39 such Internet-rich countries are listed below by decreasing score.

Ukraine
Thailand
Poland
Austria
Switzerland
Romania
Russia
Pakistan
Sweden
Germany
Korea
Japan
India
Indonesia
Bulgaria
Italy
Egypt
United Kingdom
Israel
Netherlands
Australia
South Africa
Philippines
Hong Kong
Singapore
Canada
Taiwan
Spain
New Zealand
France
Turkey
Brazil
Mexico
United States
Colombia
Ecuador
China
Argentina
Chile

In my view, if the Internet is ever going to avoid degenerating into nothing more than a den of thieves and predators, we have to introduce some accountability somewhere. Knowing the assigned IP addresses of each organization and their authorized providers is a good first step, and it doesn't seem to be asking all that much. And while a lot of the world falls short of accurately providing even this minimal information to some "authority", I couldn't help but notice that over 7,300 autonomous systems managed to get a perfect score of 100 in our system, although 63% of them had only 1 prefix to worry about. Of those with a perfect score, Daewoo Information Systems (AS 4961) manages the most prefixes at just under 100. The remaining 24,000 or so ASes could easily do a better job, and if they did, the registry information could be used as was originally intended, to both solve and avoid problems.

Notes:

Renesys computes these scores and others on a daily basis for the purposes of analysis, trending and product offerings.
For the most part, individual countries listed here have little control over prefixes registered in their country. That is, the scores are more a reflection of the businesses found in each country, rather than their governments.
Richard A. Steenbergen presented work on registry scoring of select providers at NANOG 44.
While writing this blog, I was alerted to similar work being done by the US National Institute of Standards and Technology and described here.

Longer is not always better

2009-02-21T19:30:30Z

This post is a follow-up to our blog last week about a small Czech provider briefly causing global Internet mayhem via a single errant routing announcement. In this incident, SuproNet (AS 47868) announced its one prefix, 94.125.216.0/21, to its backup provider, Sloane Park Property Trust (AS 29113), with an extremely long AS path. We've gotten more feedback about this entry than any other in recent memory, so we thought we'd try to answer some of the questions that were posed both here and elsewhere, as well as provide some clarification about exactly what went on. The questions we try to address include:

How could anyone be this dumb?
Why did this cascade throughout the planet?
Can you provide more details about the impact and its spread?
How do we prevent this from happening again?

]]> How could anyone be this dumb?

I'll admit that this was my first thought. And since this incident interrupted my lunch, I was only too happy to join the mob. In hindsight, my reaction was due to the fact that my router experience is largely limited to Cisco gear and their router software, known as IOS. For example, suppose SuproNet was using a Cisco and wanted to prepend their ASN (47868) an additional four times to announcements to a particular provider. They could use something like the following, where the string of x's refers to the IP address of the provider's router. Notice that I had to explicitly list 47868 four times.

     neighbor xx.xx.xx.xx route-map longerisbetter out
     route-map longerisbetter permit 10
       set as-path prepend 47868 47868 47868 47868

The is a common way of prepending in Cisco IOS, so I naturally thought, who would be so dumb as to type (or cut-and-paste or whatever) their own ASN hundreds of times into a configuration for a router? Who? The only problem with this line of reasoning is that SuproNet wasn't using a Cisco. They were apparently using a router from MikroTik, a router vendor from Latvia, as first reported in this Czech blog. MikroTik obviously targets the Czech market since they have a local language web page and domain.

So how do you prepend on a MikroTik? According to their on-line manual, you set the following variable in an appropriate configuration mode.

bgp-prepend (integer: 0..16) - number which indicates how many times to prepend AS_NAME to AS_PATH

So if SuproNet was thinking "Cisco IOS", they might have typed "bgp-prepend 47868" to prepend 47868 once. However, this would be a mistake as this router is expecting a count, not an ASN. So at this point, it would be reasonable to expect the MikroTik to report something like "value out of range". Let's assume they didn't do any range checking on the input value and let's assume they devoted one byte (8 bits) to store this value. One byte can represent all integers from 0 to 255. So what happens when you try to stuff something larger, like 47868, into one byte? You get 47868 modulo 256 (i.e., the remainder after dividing of 47868 by 256), which equals 252. As Mikael Abrahamsson first noticed, this was the exact number of prepends of 47868 he was seeing. So I went back and looked at the copious number of announcements we saw of SuproNet's prefix and guess what? Every single one had 252 prepends of 47868, leading me to conclude that this was the exact number sent out by SuproNet. Originally I was thinking the number of prepends probably varied based how these long paths were being truncated and that it was this random truncation that was causing part of the problem.

And using this clue, Ivan Pepelnjak was able to spell out exactly what happened in his blog. As it turns out, the reason for all those routing resets and general instability was due to a previously unknown Cisco bug involving AS paths close to 255 in length. If you try to prepend to a long path that you receive and by doing so, create a path longer than 255, you are toast. So the maps we gave in our our last blog were more of an indication of Cisco market share (at least among prependers), rather than the propensity of outdated routers. Kudos to Ivan for figuring this out.

In summary, we have a situation where a single careless operator in the Czech Republic tickled one bug (i.e., lack of bounds checking in the MikroTik router) that in turn tickled another bug (i.e., a problem with long AS paths on a Cisco). And the result was global Internet instability due to prevalence of Cisco gear in the market. But in fairness to MikroTik we note that Mathias Sundman observes that bounds checking does now exist in version 3.20 of their router software.

Why did this cascade throughout the planet?

Short answer: There is a bug in Cisco IOS with regard to long AS paths and lots of folks use Cisco gear. Longer answer: Most ISPs apparently do not filter out announcements with long AS paths. As we noted in our previous blog on this topic, we are all fairly close to one another on the Internet and there is really no reason to be seeing excessively long AS paths. Such paths only indicate a problem or a clueless operator or both, and can be safely discarded. The fact that they were not dropped allowed them to tickle this bug on many Cisco boxes along the way.

Since we are all just a few AS hops away from each other, the problem only occurred because the paths originated from SuproNet were so close to 255. This allowed them to reach the core of the Internet and continue onto other edge networks before exceeding the 255 path length boundary. It was only when they did that all hell broke loose, far away from the original source of the problem. As Andree Toonk provides on this page, there are apparently others who have made the same mistake on MikroTik routers. AS 20912, Panservice of Italy, is doing it as of this writing, but 20912 modulo 256 is only 176 and these announcements are apparently not causing a problem.

Can you provide more details about the impact and its spread?

This was an easy one, as Renesys monitors every prefix (network) seen on the Internet and computes their stability over time. We also geo-locate them as accurately as possible. Thus we can see events like this propagate through the planet and Google Earth provides an excellent way of performing the visualization. We used it to show every newly unstable prefix in during the hour before and the hour of the incident. Here are a few composite images taken from Google Earth of a few regions and an indication of all the unstable prefixes seen during the 2-hour period. We start with the US where the impact was the greatest.

Next up is the heart of South America, where Cisco obviously needs to send some sales folks. (Before someone points out the population density of South America relative to the US, we noted in our last blog that South America was the least impacted continent on a percentage basis.)

Finally, we take a look at Europe, where all the trouble started.

How do we prevent this from happening again?

This one is really about assigning blame and there is plenty to go around. But before we get too caught up with that, keep in mind that this was really the perfect storm. As of today, Renesys has observed 31,188 unique non-private ASNs on the Internet over the last few weeks. If you compute modulo 256 of each of them, you get 731 with associated values ≥ 250 or 2.3% of the total. There is nothing special about 250. However, the likelihood of a problem decreases significantly as the values get lower, and 250 seems like a reasonable cutoff, given typical path lengths in the Internet. And there are still only 1,919 ASNs whose modulo 256 value is ≥ 240, or 6.2% of the total. Thus for this event to have occurred at all, besides the bugs in the router software of two vendors, only a few percent of the ASes on the Internet could have possibly initiated the meltdown, but only if they had a careless operator and an obscure Latvian router with outdated software. How likely was that?

As for the blame, network operators (SuproNet) should obviously read their router documentation and test any proposed changes in a lab environment to see if they get the results they expect. Router vendors should check bounds on input parameters (MikroTik) and on boundary conditions (Cisco). ISPs should filter out obvious useless garbage, like ridiculously long AS paths and unrouteable (private) IP addresses. They obviously don't, given the scope of the event. And who designed this BGP routing protocol anyway? What were they thinking?

Seriously, the reason for the success of the Internet is because it is not under the control of any one government or company. Because of this fact, it is both cheap and ubiquitous. But because there is no centralized control or authority, we are largely at the mercy of the weakest link. Sure there is plenty we can do to prevent things like this from happening again, but there will always be the next perfect storm. Who could have guessed something like this could have happened? You won't be able to guess the next one either. The happy ending to this story is that the community quickly rallied and worked together to both identify and mitigate the problem. No meetings were held, no bailouts were requested and not a single lawyer was needed to draft an agreement. The Internet was back to normal in short order.

To Catch a Thief

2009-02-19T19:45:00Z

Last August at DEFCON, Alex Pilosov and Tony Kapela presented a talk entitled Stealing the Internet: An Internet Scale Man-In-The-Middle Attack, which illustrated a technique for misdirecting specific Internet traffic via carefully constructed BGP routing messages. Using this approach, an attacker can redirect the incoming traffic of any victim through his own site for further inspection or alteration before ultimately passing it on to the victim. Furthermore, the attack can be carried out in a way that is largely transparent to the victim. Since this talk, Renesys staff have been repeatedly asked "So are people using this technique today?" That is, are people currently "stealing the Internet", and if so, who is attacking whom? Given the volume of routing data that Renesys has at our disposal and the number of tools we have to slice and dice it, we thought this would be a relatively straightforward question to answer. We were wrong.

Although we ultimately succeeded in answering the question and in developing a general Man-In-The-Middle (MITM) detection algorithm for the global Internet, we ended up writing a lot of code over the course of several months and burning through endless CPU cycles looking for attack evidence. Our results were presented this week at Black Hat and the complete presentation can be found here. In this blog, we'll hit on some of the highlights from the presentation.

]]> Stealing the Internet

As in the infamous example of Pakistan hijacking YouTube, the MITM attack relies on the attacker announcing a more-specific of a prefix announced by the victim. Since the route to the most specific prefix wins, the attacker gets all traffic intended for that more-specific, rather than the rightful owner (victim). The new trick is to then pass the misdirected traffic to the victim in a way that is transparent to him. This is accomplished by creating a viable path from the attacker to the victim, one that is blind to the hijack and hence serves as an avenue for returning the hijacked traffic. This is accomplished by manipulating the Autonomous System (AS) paths that get passed around with each prefix announcement. Relative to the more-specific prefix used in the hijack, the attacker simply includes the ASes found on the viable path between himself and the victim, which serves to blind these ASes to his bogus announcement. This works because in order to prevent the endless circulation of routes, routers will not accept any routes that they think they've already seen. So while most of the Internet will see the attacker's hijack and act accordingly, those ASes on the path between the attacker and his victim will not, providing a pathway to ultimately return the stolen traffic on to the victim. That is a pretty fast summary, but all the details can be found in the aforementioned talks.

Looking for Clues

So the question is: Is someone snooping on your incoming traffic? How about on the traffic you send to your business partners? It turns out the first question is relatively easy to answer, while the second is extremely difficult. The reason the former is easy is because you presumably know your own network. Since the attack relies on injecting a more-specific of one of your prefixes into the global routing table, this is something you can have a route monitoring service watch for you. You cannot observe this yourself, since you are blind to the attack. However, most of the Internet is not blinded, so if a more-specific of one of your prefixes appears and you didn't authorize it, then someone else is messing with you.

But knowing if this is going on in general in the Internet is quite another matter, since no one can tell you exactly how every prefix should be appearing at any given point in time. There are over 270,000 of them and new more-specifics are constantly being announced. Which ones are legitimate and which ones are hijacks? At a glance, no one can say, since there is no central authoritative source to consult. So, it might seem that this question is impossible to answer, but a closer look gives us the clues we need.

Conveniently, the MITM attack was demonstrated at the DEFCON conference, so we had at least one known example to study. In this case, the victim was Sparkplug Las Vegas, Inc. (AS 20195), who had SWITCH Communications Group (AS 23005) as their sole provider. The hijacker was Pilosoft (AS 26627). The clues we need to automatically detect such hijacks can be found in the associated AS paths. Looking at the tail end of the relevant AS paths moments before the hijack took place, we see that they all have the following form. Namely, AS 20195 must go through its one provider, AS 23005, but from there the paths branch out to any of the several providers of AS 23005.

19151 23005 20195
22822 23005 20195
3356 23005 20195

Now let's look at the tails of a few relevant paths at the start of the hijack. Notice any difference?

3356 3561 26627 4436 22822 23005 20195
3549 3561 26627 4436 22822 23005 20195
1239 3561 26627 4436 22822 23005 20195

These paths all end with a long globally unvarying segment that was purposely injected by the hijacker to blind ASes on the path from him to the victim. The artificially constructed segment of the path is 4436 22822 23005 20195, which is peculiar for several reasons. First, AS 23005 had 4 providers at the time of the attack and AS 22822 (Limelight) had 8 providers and 147 peers. Yet from all of these possible routes, the entire world appears to have selected the same path through them to the victim. Second, none of the ASNs in the artificial segment of the path were seen to have actually announced anything themselves — they were silent ASNs (with respect to this prefix). They were seen only in this longer path, suggesting that they were added by a third party in order to blind them.

And there is even more. Renesys computes the business relationships between all the ordered pairs of adjacent ASes we observe. These fall into one of a handful of possible categories: a customer to a provider relationship, a provider to a customer, two peers, and a few other, less common and more complex arrangements. If we draw the first artificial path given above with peers side by side and customers below their providers, we get the following depiction.

And this path makes no sense, as it violates the valley-free property of Internet routing. An AS in the "valley" (e.g., 26627) is transiting traffic for free, i.e., throwing money out. And while mistakes do occur, one business does not typically provide some other unrelated business with free Internet service. Similarly, multiple peering links (e.g., 4436_22822 and 3356_3561) in one path means that someone is providing free transit and thus is another red flag.

Catching the Thief

To alert on MITM attacks on the entire global Internet, we start by examining all the new more-specifics seen in a day. There will be hundreds of these to consider, and most days all of them will be perfectly legitimate. We then carefully examine the associated AS paths, looking for monkey business of the type described above. Full details can be found in our presentation. But the short answer is that we take great care not to miss any hijacked more-specifics. We assume our attacker does everything possible to cover his tracks, so we must be equally cautious in avoiding false negatives.

We ran our algorithm on all of our historical global BGP data from 1 July 2008 until 31 January 2009, sifting through tens of thousands of more-specifics and their associated millions of AS paths. After all of this, we found only three cases that passed all of our tests and looked like actual MITM attacks. One of these was the actual MITM demonstration conducted at DEFCON. Of the other two, upon visual inspection, one was almost certainly simple traffic engineering using BGP manipulation. The other was confirmed by email as a fail-over test gone awry.

Conclusions

The good news is that we can say with a great deal of certainty that the MITM attack is not yet appearing in the wild. In addition, we now have a comprehensive global approach for detecting it when it does ultimately make an appearance, one that rarely produces a false positive and studiously avoids false negatives. Only "lucky" corner cases on the edge of the Internet where the attacker is very close to the victim could escape all efforts at detection, but these could easily be thwarted by the victim himself.

The bad news is that we've built a global Internet that allows for such an attack in the first place. And even if you defend your incoming traffic by monitoring for suspicious BGP announcements related to your networks, you've done nothing to defend the data you send to others. To guard against interception of your outgoing traffic, the Internet would require a comprehensive global alarming system, one that alerts on any attempt to divert traffic through unusual third parties. But constructing such a system is non-trivial to get right, requiring a rich source of global BGP data and sophisticated real-time analytics. Unfortunately, short of getting everyone on the Internet to rigorously police their own networks, there simply is no other alternative. The problems outlined here are inherent in how we currently route traffic on the Internet and until we come up with a comprehensive, incremental replacement for BGP and get it widely deployed, we're going to be living with the consequences. We can either ignore the problem entirely or work toward a solution, while keeping a vigilant eye on our current increasingly wobbly trust-based system.

Reckless Driving on the Internet

2009-02-17T03:40:00Z

This weekend, John Markoff wrote an interesting piece for the New York Times entitled Do We Need a New Internet? While his emphasis was largely on security, or rather the lack thereof, the central point Markoff makes is that the Internet may be so hopelessly broken that it could be better to start over, rather than continue to apply band-aids. As if to emphasize this point, SuproNet, a local Czech provider, single-handedly caused a global Internet meltdown for upwards of an hour today. SuproNet accomplished this feat by sending out a rather unusual routing update, one which a lot of routers did not handle very well. The result was Internet bedlam.

]]> Some Preliminaries

Routing on the Internet is strictly a cooperative affair. Neighboring routers tell each other what they know and that information ultimately propagates globally. Eventually everyone figures out how to reach everyone else. And what routers know are prefixes, i.e., blocks of IP addresses, that are routed in the same way. Since there is often more than one way to reach any given prefix, routing announcements include various attributes so that everyone can decide on their preferred path to each prefix. One such attribute is the Autonomous System (AS) path, i.e., the list of organizations that have to be traversed to reach the prefix. For example, I'm typing this blog from my home Verizon DSL connection. If I wanted to reach an IP address in Qatar served by Qtel, Verizon might hand that traffic off to Tata/Teleglobe (AS 6543), which in turn hands it off to Qtel (AS 8781). The prefix in question and its associated AS path are depicted graphically below.

AS path length is one important factor in route selection, with shorter paths favored over longer ones. Suppose that Qtel wanted to used Tata/Teleglobe as a backup provider for this prefix, only to be used when other alternatives failed. They could effect this by making the announced path artificially long. Instead of

701 6453 8781

we might see paths like

701 6453 8781 8781 8781

In this example, Qtel would have prepended its own AS to the path several times so that this particular route to this particular prefix would tend not to be selected by others.

Now the average path length on the Internet is only around 4. That is, we are all fairly close to one another. So if I make any path seem just a little bit longer, one or two ASes, it generally will not get selected and will accomplish the objective of being the path of last resort. Nothing stops you from prepending your own AS a dozen or even a hundred times, but it is not going to accomplish anything and will only pointlessly consume everyone else's router memory. It's also an indication that you don't know what you are doing. Which brings us to a central problem, you don't need a driver's license on the Information Superhighway.

Bedlam on the Internet

Now suppose you just got your Internet learner's permit yesterday and you really don't want your backup provider being used unless your main provider is down. You could prepend your AS a few times in the route announcements you make to your backup provider and that would do the trick, but to make really sure you go for a few hundred instead. In a perfect Internet, that wouldn't matter, but we don't have one of those. What we think happened next is the Internet equivalent of a massive buffer overflow. While most of the core routers run by major ISPs fared just fine, processing the ridiculous path and sending it on, others choked. Perhaps they weren't as well maintained or were running buggy software. These routers viewed the update as malformed and so tore down their session with whoever sent them the update. In other words, two routers that were happily exchanging traffic with each other just moments before suddenly stopped all communication. Traffic was lost, alternative paths were explored, and maybe the former cooperating routers recovered and re-established contact. Multiply this by thousands of routers around the world and you can begin to appreciate the ensuing pandemonium. At Renesys, we experienced an almost 100-fold increase in the rate of routing updates from our worldwide array of sensors

The Details

SuproNet (AS 47868) normally announces a single prefix, 94.125.216.0/21, to a single provider, CD-Telematika (AS 25512). On February 16^th at 16:23:30 UTC, we saw this same prefix via a different provider, Sloane Park Property Trust (AS 29113), but with an AS path exceeding 255 ASNs. Such messages continued for almost exactly one hour or until 17:23:00 UTC. We observed Level 3 (AS 3356), Tiscali (AS 3257) and TeliaSonera (AS 1299) propagating most of these routes globally, with a total of 230 unique ASes ultimately sending us the problematic announcements.

This single Czech provider announcing a single prefix caused a huge increase in the global rate of updates, peaking at 107,780 updates per-second. This peak occurred at 16:30:54 UTC, less than 8 minutes after the first announcement.

At Renesys, we call a prefix impacted in a given hour if either suffers an outage or has a non-trivial amount of instability. In the hour before this event, there were 1215 impacted prefixes globally out of a total of 271,175. During the event, that number surged to 12,920 or 4.8% of all prefixes on earth. One announcement from one provider and we have a 10-fold increase in planetary routing instability for an hour. North America suffered the most, increasing from 0.35% to 4.76%, while South America suffered the least, increasing from 0.52% to 1.75%.

Mapping the Damage

It's always the middle of the night somewhere, a time when ISPs perform their maintenance. And bad weather and backhoes roam the earth. So routing instability on the Internet is always present, as networks come and go. The following map shows instability levels by country in the hour before this event starting at 15:00 UTC and computed as a percentage of all prefixes geo-locating to the country.

Global Instability by Country - Before

The next map show instability levels by country during the hour of the event, starting at 16:00 UTC. Can you spot the outdated routers?

Global Instability by Country - During

Now what?

We were heartened to see that most of Internet's core survived a single odd announcement, but this does speak to a lot of outdated equipment or software at the edge. And if you manage to get all of edge routers to reset, you aren't going to have many people to talk to no matter what the core is doing. While it might be tempting to bash SuproNet, can anyone really defend a system where a failure in probably one of the weaker links can cause the entire system to unravel? Maybe we really do need a new Internet and for more reasons than better security. The next one needs to come with an operating permit too.

"The Adventurous Parts of the Internet"

2009-01-28T16:40:26Z

I just spent a very pleasant 3 days attending NANOG 45 in the Dominican Republic. The whole thing was a whirlwind of peering, technical presentations, and catching up with the people who keep the North American parts of the internet backbone alive. What can I say? The DR is overflowing with friendly people, great food, warm breezes (82F in Santo Domingo, versus 0F at my house in New Hampshire), and very decent Presidente beer. Very conducive to thinking the big thoughts. The trick is to write them down ...

]]> The presentation is here if you'd like to read it.

I'll give you the gist of it, though. The vast majority of the internet's "control traffic" (the BGP protocol messages that keep every router informed about how to reach the 250,000 or so individual networks on earth) is actually generated by, or on behalf of, a really small number of networks. How small? On any given day, perhaps 1% of the world's networks generate, directly, or indirectly, 50% to 70% of the total traffic being exchanged by routers all over the world. If you're a glass-half-full kind of network engineer, that means that 99% of the world's networks are good citizens — exemplary stability, quietly hanging out in the routing table, very rarely changed, going about their business and contributing very little to the traffic load that the world's routers have to process.

I'm not one of those glass-half-full guys! I wanted to find out who the worst-behaved 1% were, and more to the point, who was making money by selling them transit without keeping an eye on the mess they were generating.

Pretty straightforward, I thought: there's a tragedy-of-the-commons thing going on here, and people who provide transit to unstable networks are propagating their "table pollution" beyond their borders, around the world, making us all pay more to keep the Internet alive by their carelessness.

Perhaps, I suggested innocently, customers would seek out transit providers with better stability report cards, and offer them more money, because stability of the neighborhood could be a source of positive differentiation in an ever-more-cutthroat, low-margin industry. Perhaps, the audience responded realistically, transit providers could instead charge unstable downstream customers more, to penalize them for their tendency to flap their routes incessantly (the internet engineering equivalent of blasting your stereo at a red light with the top down). We were in basic agreement, though: someone should be pricing and paying for this pollution. (Overcoming the industry's history of abject failure at billing for anything more subtle than commodity megabits is a discussion best left for another day.)

As usual, when you put a big idea in front of a bunch of intelligent people, it got more and more interesting. By the time I sat down, I had a few new observations.

First of all, the technical voices in the operator community understand these issues quite well. They know firsthand about CPU loads, traffic trends, and how hard it is to keep your network one step ahead of resource constraints as the Internet continues to grow like a weed. Since every large provider serves a broad range of customers, pretty much everyone has at least a few dozen autonomous systems downstream who exhibit serious instability, and don't clean it up, or even know that they're making a mess. But because the offending parties may be two or three signed contracts downstream (customers of customers of customers...), the sales and business development groups at big providers have zero incentive to take an interest. In this economy, if it doesn't turn up big new revenue, or melt the network outright, it's off the radar.

The second observation was that instability is really only one of three big routing "externalities" that people inflict on the planet's infrastructure without significant penalty, the others being deaggregation (littering the routing table with redundant more-specific fragments, rather than advertising a few larger tidy networks) and failure to maintain clean, up-to-date, accurate entries in the routing registries. Being unstable chews up other people's CPU cycles all over the world. Wanton deaggregation chews up other people's scarce router memory resources all over the world. Failing to advertise your routing intentions, and stick to them, makes it nearly impossible (barring some future cryptographic miracle in global coordination) to decide whether a given routing advertisement is legitimate or suspicious, when you hear it on the other side of the world.

But these are all regarded as venial sins, not even worthy of confession. Kind of like recharging your car's air conditioner with CFCs, or burning all your kitchen garbage out behind your house in an oil drum.

The final observation was one that caused me to moderate my thirst for backbone street justice, at least a little bit. Many of the providers who I gave poor instability scores to had something in common: they serve what I call the "adventurous parts of the internet" — regions of the planet that are underserved by internet transit infrastructure because of their political instability, their poverty, their unfortunate accidents of geography or choice of neighbors, or all three.

What I found was that unstable networks tend, more often than not, to be in places like Mexico, or Armenia, or Iran, or Vietnam, or Georgia, or Egypt, or Cambodia. It's not that all providers in these places are unstable — far from it — but substantial populations are served by providers who just can't seem to keep their routes under control. Often these are the old national incumbent telephone companies, trying to bring people into the 21st century, hobbled by a regulatory environment that doesn't discover efficient solutions.

By constructing a metric that ranks providers according to the aggregate stability of their customer base, wasn't I just penalizing carriers who chose to serve the adventurous parts of the internet? Isn't there a "positive externality" being created by these providers, in that they bring the internet within reach of everyone on earth?

I took a question from an audience member who suggested, partly tongue-in-cheek, that the instability we had documented was a sign that fewer people should be "speaking BGP" at the edge of the internet. That is, that these unstable edge providers should simply pick the single best provider they can afford, put all their eggs in that basket, and delegate the difficult challenges of maintaining a stable Internet presence to that one provider (be it Sprint, or Tata Teleglobe, or Verizon Business).

To be sure, that's a conclusion you could draw from the data. But I have the feeling that in the long run, the benefits of keeping the adventurous parts of the internet well-connected via diverse, redundant transit will outweigh the extra noise they generate. I hope that we can use instability scoring as a tool to identify providers who need a little extra help, a few words of advice from the global networking community, maybe even an invitation to participate in global community events like NANOG, so that they, too, can find and fix their internet pollution problems.

Internet Year in Review 2008

2008-12-29T13:15:00Z

It's been an interesting year in many ways, not least of which for the Internet. This year, I started to contribute in earnest to the Renesys blog and back in January I was wondering "How am I going to find anything interesting to talk about on a regular basis? Nothing much happens on the Internet, right?" Well, it certainly did this year and now I've got many more ideas than I have time to research and write about. In hindsight, I guess it isn't too surprising. As the world becomes more interconnected and more Internet-dependent, we're bound to bump into each other more and expose the limitations of the current system. So let's review what 2008 brought us and take a guess at what is in store for the new year.

]]> January

The year started quietly enough and here in New Hampshire, we were largely trying to stay warm. Then the Mediterranean cables starting snapping and suddenly things got much more interesting. Entire countries and providers were cut off from the Internet and there was a mad scramble to restore connectivity to the area. As always, there were winners and losers, but the main lessons countries and businesses in the area learned were: "stuff happens", don't put all your eggs in one basket, and you get what you pay for. Of course, these cliches are not very sexy, so the conspiracy theorists, apparently having never seen bad weather or a boat anchor, were only too quick to claim that all these cable breaks were a prelude to war, only they weren't. No new wars were started in January, the cables were ultimately repaired, and a lot of folks in the region started planning for failure by building some redundancy.

February

Just after the Mediterranean cables were repaired and we were starting to wonder if it would ever stop snowing, Pakistan had another bad hair day and their local incumbent (Pakistan Telecom) inadvertently knocked YouTube off the global Internet for about 2 hours. This incident got a lot of press attention, and there was considerable hand-wringing about this grave new threat to the Internet, despite the fact that such incidents have been taking place for over a decade and are both inevitable and unavoidable in a trust-based system with no central authority. So unfortunately, no one fixed the Internet in 2008.

March

It's still snowing and the locals are starting to come a bit unglued. New Englanders are buying seeds and starting to walk around in shorts in a vain attempt to defy the weather gods. And our cooperative trust-based Internet dramatically showed another shortcoming when TeliaSonera and Cogent depeered, partitioning the Internet for those dependent on one or the other of these providers. To name just two of the many victims, this event affected both Martha Stewart's corporate headquarters, single-homed behind Cogent, and Blizzard Entertainment's World of Warcraft servers, single-homed behind TeliaSonera. In other words, the Internet was broken in lots of little ways: Martha wouldn't have been able to play a computer game, Europeans might not have been able to reach Cogent-hosted content, and so forth. But then after two weeks, this particular episode of Internet chicken ended when the two carriers reestablished connectivity, and the Internet was once again whole.

April

Mud season. The snow is melting fast now and those quaint dirt roads we are famous for are turning to rivers of mud. It was a slow Internet month, but thankfully the start of Todd Underwood's musical career gave us something to write about. Mercifully, both the season and the career were short-lived.

May

The ice is breaking up on the lakes and Joe's pond had their official iceout at the end of April. And while that happens every year, we did report something truly bizarre on the Internet, brought to our attention by ICANN. ICANN, responsible for one of the 13 DNS root name servers, had recently changed the IP address of the server under its stewardship, a switch undoubtedly missed by most of humanity. Then bogus root name servers started appearing on the old IP and were happily providing answers for up to six months before anyone noticed. Although we are far from certain, it appears that no harm was done in this case, but the potential for mischief was considerable. Another contest of egos? As Rodney King once said "Why can't we all just get along?"

June

All the traces of snow and ice are now gone, so it's time for black flies to make their annual appearance. These pests feed on human blood and like to rip out chunks of your flesh near a hairline, which you notice only after blood is streaming down your face. Having the ground covered up for the half the year suddenly doesn't seem so bad. We're still wondering what exactly happened with the ICANN root name server, but unfortunately have more questions than answers. At least I start to monitor all of the root name servers from my Renesys Routing Intelligence account, as I might need new blog material one day! Besides, someone should be watching over our critical infrastructure and it might as well be me. The year isn't even half over and we've seen a pretty incredible assortment of major headline-grabbing events, all pointing to serious flaws in how the Internet actually works. On the business front, Cogent became transit-free this month, meaning all the big boys really do need to peer with them to prevent a partitioning of the Internet, like it or not.

July

Ahh, summer. We get exactly one month of summer in New Hampshire and we aren't going to spend it writing blogs. Besides, not much happened. Oh, wait, there was Dan Kaminsky's revelation of a DNS implementation vulnerability so bad that DNS servers all over the planet were trivially subject to DNS cache poisoning. Despite the frantic vendor response and software upgrades, the vulnerability is not resolved, nor resolvable given the current domain name system. Maybe we can finally get widespread interest in DNSSEC. Otherwise, it was a quiet month and we worked on our tans.

August

Our long "summer" is finally winding down and by the end of the month we could come close to the freezing mark again. While no one was looking, Tony Kapela told the Defcon 16 crowd how to steal the Internet, a technique for transparently intercepting anyone's incoming Internet traffic before eventually passing it on to the rightful owner, and doing so in a way that can be largely invisible to the victim. As if that wasn't bad enough, Russia also invaded Georgia. We expected this to have a rather negative impact on Georgian Internet connectivity. Only it didn't. Sure there were website defacements and some DoS attacks, but that pretty much happens every day in every country. Despite constant calls from the media, we really couldn't find any changes to Georgia's Internet presence or any significant or prolonged losses of connectivity. Georgia's meager Internet infrastructure simply wasn't blown up.

September

The days are getting noticeably shorter and much cooler. Thankfully the bugs will all soon be dead, as this was a particularly bad year for winged pests. But as New England cools, the Gulf coast is just heating up with one hurricane after another. Gustav started out looking like a replay of Katrina, but in the end, Louisiana wasn't hit as hard and was much better prepared for Katrina's little brother. The real loser in the hurricane lottery for 2008 was Texas, which was hit very hard by Ike. After rolling through Texas, Ike still packed enough punch afterward to cause extensive Internet outages throughout the US. We ended the destructive month with a good old-fashioned public stoning: the notorious Intercage was knocked off the Internet by their irate providers, only to briefly rise from the dead before being taken down for good.

October

It's leaf peeper time and while the bugs might be dead, a new scourge has arrived on the land: tour buses. As if these flatlanders weren't entertaining enough, Sprint depeered Cogent at the end of the month, resulting in a partitioned Internet for the second time this year and impacting a few thousand customers of each carrier. Perhaps Sprint upset the wrong customer, since the peering link was quickly restored over the weekend, when upper management and their lawyers were unlikely to be working. I can't help but wonder where the phone call came from and to whom in Sprint's organization, but Sprint clearly didn't hesitate long to blink. I guess some customers are always right. Also this month, Outpost24 claims to have discovered a way to trivially DoS anyone by exploiting some TCP vulnerabilities, but details remain murky.

November

No leaves, No snow, No-vember. At least the tourists are gone and the growing darkness gives us time for reflection. Even if you ignore all of the security concerns, the Internet is still heading for a train wreck. We're running out of usable IP space and the providers are all going broke. If you follow any of the links in this blog, it should be the preceding three for Todd Underwood's insightful comments on the industry and difficulties that lie ahead. After reading these, you won't be surprised when the Internet doesn't work. You'll be thankful every day that it does. Speaking of which, we almost had another Internet-sized blowout when Companhia de Telecomunicacoes do Brasil Central leaked a "full table". Had this spread, Brazil would have ended up on the receiving end of much of the traffic on the global Internet, which would not have been a good thing for anyone. As it turned out, we dodged the bullet this time and the damage was very contained. And the month saw another public flogging of an Internet bad boy; this time it was McColo's turn.

December

Darkness. The earliest sunset is now at 4:12pm, but when it's cloudy, it's more like 3:12pm or even earlier as the sun doesn't get very high in the sky now. We spend way too much time indoors, looking at computer screens. This problem is partially solved for us when a ferocious ice storm knocks out power to more than half of New Hampshire. Three Renesys employees were without power over a week later. Jim Cowie, was one of them, but he refused to despair over his own homelessness, not to mention the growing global economic gloom, and wrote a hopeful and interesting blog about a possible economic stimulus that does not involve bailing out dying industries. Since the year was drawing to a close, those of us with power also took the opportunity to review how the top global Internet service providers fared in 2008. In short, if you figured out that more than half the planet lives in underserved Asia and built your business plan around that part of the world, you probably did alright, at least in terms of gaining market share. (And we were happy to see that most of the winners were also users of Renesys' Market Intelligence product.) Finally, it was Deja Vu All Over Again when more Mediterranean cables broke to provide a nice symmetry to the year.

The Year Ahead

I predict that New England will have a banner ski season this winter. Although it is only December as I write this blog, I've observed three leading indicators: I can reach my mailbox from above, the picnic table in my backyard is a ski hazard, and the local firefighters have been observed patrolling the neighborhood looking for lost fire hydrants. In other words, we've had a lot of snow for this early in the season.

As far as the Internet goes, it's a different story. If you managed to get through all of the above and maintain your sense of optimism, I applaud you. It's really not a very pretty picture, nor am I able to find many silver linings. So what will 2009 bring for the Internet? It's probably pretty safe to say that things are going to get worse before they get better. With the world economy in turmoil, don't expect your service to improve. And with all the incentive for ill-gotten gain in a deteriorating economic situation, don't expect much relief on the security front either. Major new vulnerabilities will continue to surface and older ones will be more fully exploited, as the inherent trust-based model of the Internet continues to crumble. Nation-states will beef up their cyber-warfare capabilities. And I wouldn't be surprised to see a major telco declare bankruptcy and at least another major de-peering event. In other words, I don't expect 2009 to look much different than 2008. To get out of this mess, we're going to need some global leadership and a game-changing shift in both the economic and security models surrounding the Internet. If things get bad enough, maybe we can at least start a serious discussion around that in 2009.

Deja Vu All Over Again: Cables Cut in the Mediterranean

2008-12-20T02:23:00Z

The end of the year is approaching which seems to be a harbinger of Internet disasters. Four years ago (on 24 Dec. 2004), TTNet significantly disrupted Internet traffic by leaking over 100,000 networks that were globally routed for about an hour. Two years ago (on 26 Dec. 2006), large earthquakes hit the Luzon Strait, south of Taiwan, severing several underwater cables and wreaking havoc on communications in the region. Last year there was a small delay. On 30 Jan. 2008, more underwater cables were severed in the Mediterranean, severely disrupting communications in the Middle East, Africa, and the Indian subcontinent.

Calamity returned to its customary end-of-year schedule this year, when early today (19 Dec. 2008) several communications cables were severed, affecting traffic in the Middle East and Indian subcontinent. According to a press release by France Telecom three major cables were damaged: Sea-Me-We 4 at 7:28 UTC, Sea-Me-We 3 at 7:33 UTC, and FLAG FEA at 8:06 UTC. It appears that the SMW3 cable was only partially cut, the SMW4 cable was completely cut, while the FLAG cable was "observed down" with no other information given. The location of the cut appears to be between Sicily and Tunisia in a section which is the responsibility of Egypt Telecom. The causes of the cut remained unclear. It seems that ships were deployed to repair the damaged cables, but no ETA was given.

]]> In this preliminary analysis we considered prefixes (networks) that suffered outages between 07:00 UTC and 17:00 UTC on 19 Dec. 2008. At least 2840 prefixes suffered outages during this interval. The most seriously affected countries are shown in the map below. Darker shades of red indicate that a larger percentage of prefixes in the country suffered outages.

The impact for each affected country is illustrated in the bar graphs below. Shown in the first graph is the distribution per country of the 2840 prefixes that suffered outages. In the second graph, we show the percentage of prefixes in each country that suffered outages.

This event appears very similar to the late January 2008 cable breaks, with Egypt taking most of the impact: over 1400 of Egypt's globally routed prefixes suffered outages, about 81% of their prefix count. The second most impacted country was Saudi Arabia with 374 prefixes, 42% of its total. While India and Pakistan had large numbers of prefixes experiencing outage, 456 and 169 respectively, these outages represented a relatively small percentage of their globally routed prefixes: 4% and 10%. The other countries fell somewhere in between with approximately 25% of their total prefixes suffering as a result of these cable cuts.

The plot below details the total number of network outages in the above countries between 07:00 UTC and 09:00 UTC. The times when the three cables were damaged (SMW3 7:28 UTC, SMW4 7:33 UTC, and FLAG 8:06 UTC) can be clearly seen.

It is not clear if any lessons were learned from the previous cable cut event. There are at least a couple of initiatives to build new cables in the Mediterranean: one led by a consortium of 9 companies, and another led by Telecom Egypt who signed a contract with Alcatel-Lucent to build the cable. The new cables however seem designed only for increased capacity rather than for redundancy, as they appear to follow the same geographic route as the current cables: from Alexandria, Egypt, going by Malta, south of Sicily and eventually landing in Marseille, France. If that is the case, the new cables will likely be susceptible to the same problems as the current ones. It is interesting to notice that both in January and now all the cables in the region were damaged at the same time. It is possible that building new cables to follow a different geographic route is either impractical or too expensive. The geography is what it is and there is not much one can do about that. The providers and consumers in the region will, however, have to accept the risk that occasional outages like these can and will happen.

For now we can only hope that the repairs will be quicker than in the past, although with the holiday season in Western Europe coming up it may not be very realistic to expect it. We will come back with updates as the situation develops.

Rising to the Top: A Baker's Dozen

2008-12-17T20:00:00Z

As readers of this blog will know, Renesys collects Internet routing data — a lot of it. We use this data in a variety of ways: in determining the impact of cable breaks, natural disasters and deliberate partitionings; in uncovering the source of hijacks or other questionable activity; in analyzing Internet business relationships; and in exploring "what-if" scenarios.

All of our reports and products are based on hard facts and objective analysis. Perhaps the only controversial thing we do with our data is to rank all the service providers in the world: globally, by geography, and by market segment. The rankings are a rather crude measure of size, as they are based entirely on the quantity of IP space ultimately transited by each provider. Although there are obvious shortcomings in this approach, it is certainly objective and the process is fully automated. It also happens to be derived from data that is readily available for all providers. Routing data, unlike most other metrics we could consider using, is inherently public.

While everyone wants to be #1 (hence the controversy around rankings), changes in rank can be far more revealing than the actual rank itself. In other words, while there are surely big differences between #1 and #50 in our rankings, the differences between #5 and #6 are much less clear given the nature of the metric. What we tend to look for are abrupt changes and long-term trends. Did a provider just jump in the rankings? Maybe they picked up a large customer or a nearby rival lost one? Who was it? Is another provider showing steady gains in the rankings? Maybe they are consistently taking market share with an aggressive, well-executed business plan in a particular part of the world? This is why changes in rankings matter: they capture some of the dynamics of the business of providing Internet service. With this in mind, we will take a look at the top 13 providers in the world for 2008 and how they have jockeyed for position throughout the year. We will also highlight some of the more interesting changes.

]]> To rank providers, Renesys starts out by determining the business relationships between adjacent Autonomous Systems (ASes), as seen via BGP announcements from our over 300 peers. This is a complex, computation-intensive task, consisting of processing over 10 million AS paths and 100,000 unique AS pairs that make up these paths. By combining the computed business relationships with IP prefix geo-location information, we can determine exactly which prefixes are ultimately transited by a given provider in a given location. Rankings are then based on the scores we assign to the relevant prefixes, scores which depend on prefix size and date of assignment. (More details can be found here.) In this blog, we'll concentrate on the top 13 providers according to our global rankings and how their score changed over 2008. Since the exact numeric score is not that meaningful in this context, we will not provide the scale in our graphs. Rather, we will show how the scores changed over the year and consider some of the reasons behind these changes.

The above graph plots global scores over time for the top 13 providers. While this might not look very informative, if you squint carefully enough, you can see three distinct clusters of providers, namely, the top two, the middle six, and the final five, each operating within roughly the same range of scores over time. We will consider each cluster in turn, enlarging the scale so we can more clearly see the changes, and then making note of some of the more interesting ones.

In 2006, when Renesys first started ranking providers, Sprint (AS 1239) seemed to have the #1 spot locked up with a score of almost 25% more than second-place Level 3 (AS 3356). Since that time, Level 3 has been gradually and consistently closing the gap and took over the #1 spot for the first time this year. Despite a few long, relatively flat periods, Level 3 had two huge surges, one in July and another in November. In July, Level 3 picked up Korea Telecom (AS 4766) and ReTN.net (AS 9002) as customers. (Previously we had them classified as peers.) And Asia Netcom (AS 10026) greatly increased the number of prefixes they were transiting via Level 3. November's rise was almost entirely due to Japan's Softbank Telecom (AS 4725) increasing the number of transited prefixes from 2% to just over 11%. Although they seem to have tailed off recently, Sprint's growth was more consistent over the year, but not enough to counter the big wins by Level 3. The average score in this cluster increased by 14% over the year.

Looking at the second cluster of providers, a few things are readily apparent. First, Verizon (AS 701) and NTT (AS 2914) both suffered sharp declines over short periods. Second, Global Crossing (AS 3549) and TeliaSonera (AS 1299) grew consistently during the year and were the only ones in their peer group to show appreciable gains. Verizon lost big after TeliaSonera stopped using them for transit. (Peers, whether paid or not, do not get credit for each others customers.) Likewise, Cogent became transit-free mid-year after they stopped using NTT to reach AOL ATDN (AS 1668), which accounted for much of NTT's decline. Global Crossing's gains consisted of numerous big wins, many in Asia, picking up increased transit from Hurricane Electric (AS 6939), PCCW (AS 3491), ReTN.net (AS 9002), Asia Netcom (10026) and Golden Telecom (AS 3216) to name a few. As a result, Global Crossing catapulted from #5 to #3 in the world, quite an accomplishment, especially considering they were #25 back in 2006. TeliaSonera also picked up increased transit from Hurricane Electric and Asia Netcom, briefly passing AT&T to capture the #7 spot for a time. But as of a few days ago, AT&T sneaked ahead again by picking up more transit from Uninet (AS 8151), a large retail provider in Mexico. Still, given the flat or declining fortunes of most of this group, I would not be surprised to see TeliaSonera ranked #4 or #5 by the end of 2009. While quite small, the average score increase for this cluster is not very meaningful, given the very different trends observed.

The members of our final cluster largely tracked one other with similar growth rates. One exception is Tiscali (AS3257), which moved from the bottom of its peer group to near the top, within striking range of Tata/VSNL/Teleglobe (AS 6543). In fact, they surpassed them briefly during the year on a couple of occasions. Tiscali picked up increased transit from Asia Netcom and gained Interoute (AS 8928) and KPN (AS 286) as customers. China Telecom (AS 4134) moved to #11 all the way from #21 back in 2006. While China Telecom is by no means a global provider, they are by far the dominant player in the huge Chinese market, and also provide transit to South Korea and Vietnam. These large and emerging markets allowed China Telecom to move up two places in our global rankings this year and to within easy striking range of the top 10. The average score in this cluster increased by over 40%.

Conclusions

It's pretty clear from this narrative that the providers who are moving up in the rankings are doing so thanks in large measure to Asia. This should come as no surprise. With more than half the world's population and only 15% online, there is enormous potential here. But market share does not imply profitability or even a well-run network. So we would never suggest you pick your providers based on Renesys Market share rankings alone. While declining market share might indicate a problem, a growing company is worth your business only if they meet your service needs and are going to be around tomorrow. Rankings, by their very nature, cannot take all possible considerations into account, nor can they be tailored for each consumer's needs, and thus they should only be one factor used in making an informed business decision.