Renesys Blog

The Geopolitics of Iranian Connectivity

2010-02-11T12:59:18Z

As Iran celebrates the anniversary of the 1979 Islamic Revolution, it seems like an opportune time to look in on the evolving state of their Internet connectivity. When we last looked, after the disputed elections in June 2009, the picture was one of uneasy stability: logically diverse but physically constrained transit via the United Arab Emirates, backup transit via Turkey. Today, a third way out of the bottle is visible in the routing table: substantial amounts of Internet transit have materialized through a Russian provider. And there, in those obscure entries in the global Internet routing table, may lie echoes of Iran's larger geopolitical strategy.

]]> Examining the Global Routing Table

Let me back up for a moment and review what's visible today, and how we interpret it. Renesys collects independent routing table perspectives from Internet service providers all over the world, and fuses them to get a daily picture of the evolving economic relationships (provider-customer, peer-to-peer) that allow Internet traffic to flow. Our Market Intelligence product uses this evolving roadmap to give Internet transit wholesalers critical visibility into the commercial relationships in the markets where they make their money.

As you might guess, when we look at this aggregate global routing table, we learn a lot about the size and structure of the Internet transit marketplace in a particular region. In this case, as we observed back in June, it's clear that Iran has a very rich internal Internet, with dozens of service providers and content providers. It's one of the oldest and most advanced domestic Internet ecosystems in the region, with a very sizable and sophisticated domestic audience.

However, nearly all of Iran's international traffic (packets bound for Twitter, Google Mail, Yahoo, etc.) must squeeze painfully through a state-imposed bottleneck. One organization, DCI (Data Communications Iran), the Internet arm of the state telecommunications company, serves as a common transit gateway. This gives the Iranian government a centralized locus of control from which to monitor and filter international Internet traffic.

Getting Out of Iran

So, the obvious question: where does the Iranian government purchase its international Internet transit? Think for a moment about the constraints that they have to satisfy. They need enough capacity to sustain a 21st century information economy. They want to maintain centralized control over all that information. They have the challenge of maintaining adequate logical and physical diversity, so that a single point of failure can't take down the whole country's Internet access (unless they choose to do so themselves!). And they have the additional headache of choosing providers that are geopolitically diverse, to route around sanctions and military threats.

Take a look at the map of Iran and its neighbors. The good choices are few and far between. In the south, they can access the UAE over a submarine fiberoptic link from Jask to Fujairah, built by Alcatel-Lucent. This is the obvious way out, as it gives access to the primary submarine fiber paths headed east (to Asia) and west (to Europe). You get your choice of service providers once you reach the Emirates, and Iran has historically made good use of this diversity. But in the end, it's logical diversity, not physical diversity. A thin piece of glass strung across one of the world's busiest and tensest energy shipping lanes is not what you want to hang the nation's Internet future on.

So they established a connection to Turkey along a terrestrial fiber route to the northwest, from Tabriz to the frontier and on to Ankara and Istanbul along the gas pipeline. Turkey is an important regional conduit for Internet transit to Europe, and the Turkish government has quietly signalled their support for Iran's right to nuclear self-determination in recent months, as part of a warming relationship. Still, one can imagine that the Iranian government might want to seek a third way out, to further improve their diversity in times of political crisis.

Where else can Iran turn for transit?

The southeastern frontier goes to Pakistan through divided Baluchistan; this would be a fine path to reach Asian transit, but would be politically complicated. The TAE cable system also goes to Turkmenistan along the northeastern frontier, but it's unlikely to offer the kind of capacity or stability that you'd want for a primary backup route. Afghanistan and Iraq are out, for obvious reasons. Armenia will be an interesting choice someday, when the railroad goes in, but lacks the capacity today.

That leaves the Caspian coastal route to Azerbaijan, following existing pipelines to Baku. We've seen DCI use the Azeri route before, showing brief evidence of limited transit through Azerbaijan's Delta Telecom in 2008-2009. Baku has been a nexus for gas and oil transport for many, many years: north through Russia, south through Iran, west through Georgia. The Internet follows these same routes, with copper and glass fiber laid alongside existing rail and pipeline resources. And indeed, this is the route that Iran seems to have chosen to complete their Internet diversity play in the weeks leading up to the commemoration of the 1979 Revolution.

Enter the Russians

Two weeks ago, on January 26th, Russia's Rostelecom suddenly stepped in as another major transit provider to DCI, visible in the global Internet routing tables. This plot shows the relative stability of DCI's transit relationships over the preceding 16 months. Each international transit provider's percentage of Iran's transit is rendered in a different color; they all add up to 100% each day, but the mixture changes over time. DCI seems to change its provider mix just before significant events: TeliaSonera (brown) joins the provider mix in the weeks before last year's election. Rostelecom (yellow) shows up two weeks before the commemoration of the Revolution.

If we zoom into the last four months, we can see how abruptly DCI's transit preferences have shifted. Rostelecom has suddenly and decisively become the third-most important provider for Iran's international transit, behind Turk Telekom (TTnet) and TeliaSonera.

For Iran, it's a smart choice in two ways.

First, buying Russian transit actually makes good technical sense. Rostelecom and other Russian providers have made huge progress building out their domestic networks and their connectivity to European and Asian markets. Their terrestrial east-west capacity is ample, diverse, and poses an increasingly attractive alternative to the southern submarine cables. Would you rather wait two weeks for a fix to a cable on the ocean floor, or a few hours for a fix to a fiber along a rail line?

Secondly, of course, from Iran's perspective, Russian transit makes for great geopolitical diversity. In the last year, Iran and Russia have made a lot of visible progress on strategic military and energy coordination. The appearance of northbound Internet transit provides additional evidence that beyond the tactical concerns of censoring its citizens' access to Western Internet content, the Iranian government is thinking strategically about cyberspace as a national resource to be defended.

Much Ado About Baidu

2010-01-13T10:58:00Z

As our faithful readers know, Renesys monitors routing on the global Internet in real time and uses that information in a variety of ways. For example, we can instantly let you know which networks a hurricane has disabled or even tell you when a war has left things pretty much as they were. In short, we keep an eye on the Internet, the entire Internet, but this is all done at the level of IP addresses and the paths they follow.

The recent attack on Twitter got us thinking. Maybe we should be keeping an eye on a few more things? While your IP addresses and routes to them might be completely stable, the average user doesn't know about those. In other words, when was the last time you typed ...
http://216.239.59.104
instead of ...
http://www.google.com
into your browser?

What if someone manages to point your domain name to some other IP addresses? You would still be operational as far as the Internet routers were concerned, but no humans would probably be reaching you. And that's the problem we'll briefly consider in this blog.

]]> Of course, none of this is new. DNS cache poisoning has been with us a long time, but security researchers are devising increasingly clever ideas to defend DNS from these sorts of attacks. Unfortunately, securing one part of the Internet, simply sends the miscreants to less well defended avenues. With respect to state-of-the-art DNS servers (admittedly probably a small set), the weakest link may now be via some of the many domain name registrars. These are the folks who are ultimately responsible for how your domains get associated with IP addresses and you generally keep this information current yourself via an online account at the registrar. But what if hackers compromised your login credentials, say by guessing your password or using a poorly designed password reset functionality or maybe by a little social engineering? They could then point your domain anywhere they wanted. No need for a sophisticated attack against a hardened DNS server when a simple username and password will do.

So do we really need to worry about this or was the Twitter attack a one-time event? We didn't have to wait long for the answer. According to Alexa, baidu.com, the "Google" of China, is the 8th most popular site in the world and, not surprisingly, the top site within China. And it too seems to have had its registration compromised. From our queries around this time, www.baidu.com most commonly resolved to 220.181.6.175 and a few other IPs in the 220.181.6.0/24 prefix. These IPs are actually routed via the larger 220.181.0.0/19 prefix, originated by AS 23724, a China Telecom "Internet Data Center". From there, these IPs reach the rest of the Internet via any one of five global providers, as depicted below.

But then something funny happened, on January 11th at 23:09 UTC, we received our first answer with a very different IP address, namely, 188.95.49.6, which resolves to pink2.warez-host.com. This IP is routed via the prefix 188.95.49.0/24, originating from the hosting provider, Interactive3D (AS 49544) in the Netherlands. Interactive3D, which we first observed in the routing tables on 24 October 2009, owes its connectivity to the providers shown below. And this new IP did not host the Baidu web site, but rather a page for a group claiming to be the Iranian Cyber Army. (Interestingly, there are several other domains with "warez" in the title hosted at Interactive 3D.)

To reduce the load on DNS servers, every DNS response comes with a TTL or "Time to live". This value tells the receiver how long to cache the response before troubling the server again with the same request. Small TTLs allow the site administrator to change servers quickly at the expense of more frequent queries. For the operators of baidu.com, they seem to have been typically using a TTL of 14400 seconds or 4 hours, a perfectly reasonable value. Whoever took over their domain left that value unchanged when they misdirected the site. If they had increased it, servers around the world could have retained the bogus baidu.com IP address for this domain for a very long time.

Within about 2 hours, we started seeing the return of the 220.181.6.175 address. As of this writing, all of the DNS servers for this baidu.com domain are reporting a TTL 1200, or 20 minutes. Better safe than sorry, I guess. But before everything had settled down, at 02:35:59 UTC on January 12th, about two and half hours after we saw the first misdirected response, we starting seeing responses of 127.0.0.1 with a TTL of 300, which went on for about 20 minutes. This is the the loopback IP address, or the IP address every computer on the Internet uses for local communications. Clearly this was an error or perhaps a misguided attempt to clear out caches.

The moral of the story here seems to be that regardless of how well run and secure your network may be, you depend on the goodwill of others to be reachable on the Internet. If someone hijacks your IP space altogether, as happened to YouTube in 2008, or your registrar is compromised and your domain is pointed elsewhere, as with Twitter and now Baidu, the best you can sometimes do is watch and wait. And then react to the attacks when they occur, as quickly as your monitoring system allows.

A Baker's Dozen in 2009

2010-01-01T04:59:59Z

As our regular readers know, Renesys collects a lot of Internet routing data, using it to create reports and products based on hard facts and objective analysis. Perhaps the only controversial thing we do with our data is to rank all the service providers in the world: globally, by geography, and by market segment. The rankings are a rather crude measure of size, as they are based entirely on the quantity of IP space ultimately transited by each provider. However, it's the ranking trends that are more revealing than any absolute number. Who is adding customers? Who is losing them or just standing still? Changes in IP transit answer these questions and more. Although there are obvious shortcomings in this approach, it is certainly objective and the process is fully automated. All of our rankings are updated daily and available via our Market Intelligence offering. In this posting, we will take a look at the top 13 providers in the world for 2009 and how they have jockeyed for position throughout the year, similar in spirit to our December 2008 blog, which provides more details about our methodology. We will see what a difference a year has made and highlight some of the more interesting changes.

]]>

The above graph plots global scores over time for the top 13 providers in 2009. Since the exact numeric score is not that meaningful in this context, we will not provide the scale in our graphs. Rather, we will show how the scores changed over the year and consider some of the reasons behind these changes. The first thing to note is that membership in this elite set of top global providers did not change in the last two years. The next thing to note is that within this set, there are distinct tiers, where members of each tier vie for position. These details will emerge as we enlarge the scale and break the group into three distinct clusters, namely, the top two, the middle seven, and the final four. In these expanded graphs, the changes will be more obvious and we'll make note of some of the more interesting ones.

In 2006, when Renesys first started ranking providers, Sprint (AS 1239) had a commanding lead with a global score of almost 25% more than second-place Level 3 (AS 3356). In 2009, Level 3 pulled away from largely stagnant Sprint, opening up a roughly 12% lead. In the spring of 2009, Level 3 gained nLayer (AS 4436) as a customer, which Renesys ranks as #12 in the US, thereby giving Level 3 a substantial boost in score. They also picked up KDDI (AS 2516) as a customer, the #2 ranked provider in Japan. Finally, Level 3 won increased transit from Japan's Softbank Telecom (AS 4725), Europe's Interoute Communications (AS 8928), Turkey's TTnet (AS 9121) and a wide variety of other strong regional players. In short, Level 3 is aggressively gaining Internet transit business around the globe in both established and emerging markets, while Sprint, still solidly in the #2 position, appears to be largely treading water.

Looking at the second cluster of providers, it is readily apparent that Global Crossing (AS 3549) is in firm control of the #3 spot, but their growth slowed considerably in 2009 over 2008. In 2008, Global Crossing's score increased by almost 50%, as they vaulted from #5 to #3. Their score increased 11% in 2009, leaving them safely in #3 with no clear challengers, but still a long way from the #2 position at three quarters of the size of Sprint.

Savvis (AS 3561) surged upward by over 25% in the summer, in large measure by winning Asia Netcom's (AS 10026) business. But then they fell back considerably by losing Singapore Telecom (AS 7473) and Comcast (AS 7922) in the fall, ending the year with a score of around 8% higher than at the start.

TeliaSonera (AS 1299) rose by over 20% during the year, around a quarter of the increase due to gaining Russia's Rostelecom (AS 12389) as a customer. Tata (AS 6453) entered this middle tier of providers in 2009 from the bottom tier in 2008 by consistently winning business throughout the world, including Serbia's Telekom Srbija (AS 8400), UAE's Emirates Telecommunications (AS 8966), the US's Road Runner (AS 7843) and Comcast (AS 7922), South Korea's Hanaro (AS 9318) and many others. As a result, Tata has now passed AT&T (AS 7018) to capture the #8 global ranking.

The remaining members of this tier, AT&T (AS 7018), Verizon (AS 701) and NTT (AS 2914), like Sprint in the top tier, largely treaded water in 2009.

Within the final set of providers under consideration, both Tinet (AS 3257) and China Telecom (AS 4134) showed significant spikes in score in 2009. In April, Tinet saw increased transit from Interoute (AS 8928) and, in November, they picked up Singapore Telecom (AS 7473) as a customer. China Telecom's growth was of a very different nature: they grew by routing more address space on behalf of themselves, not by entering new markets. Rounding out this final group, we see Cogent (AS 174) with slow and steady growth throughout the year, while Qwest's (AS 209) score remained essentially constant.

Finally, note that large global providers are not necessarily transit-free, sometimes called Tier 1, providers. Tinet and China Telecom, who made our rankings, are not transit-free, whereas, XO (AS 2828) and Abovenet (AS 6461), who did not make our rankings, are.

Conclusions
Internet transit is an increasingly tough business, as prices continue to collapse. The emerging markets in Asia and the Middle East with their relativity low Internet penetration and higher margins remain a bright spot. These markets still offer enormous potential and providers willing and able to service them are growing and rising in our rankings. Those that do not service these areas are being left behind. For example, the US old guard of Sprint, AT&T and Verizon are at a virtual standstill, China is growing but without diversity, and Level 3 and Tata, two companies to keep your eye on, are growing thanks to strong global diversity.

But we hasten to point out that market share does not imply profitability or even a well-run, resilient and stable network. So we would never suggest you pick your providers based on Renesys market share rankings alone. While declining market share might indicate a problem, a growing company is worth your business only if they meet your service needs and are going to be around tomorrow. Objective rankings, by their very nature, cannot be tailored for each consumer's needs, but they can contribute meaningfully to informed business decisions.

Bonjour, Y'all! ASN Split Personalities

2009-12-09T01:59:42Z

Remember when the telephone company came to your house to hook up your phone and gave you a new phone number? This new number was how your friends and family were going to contact you. You counted on the telephone company to ensure that someone hadn't already been issued that number, because if they had, various problems would ensue. What would happen when your mom tried to call your number if it was also assigned to someone else? Could you directly call the other party to work out the problem? Well, in the BGP realm, something similar has been happening with autonomous system numbers (ASNs).

Organizations need an ASN to run BGP and route on the Internet. They are each assigned globally unique ASN(s) by their local Regional Internet Registry (RIR), who get them from IANA. A few weeks ago, the NANOG folks noticed that AS1712 had been registered by two different organizations (in France and Texas) that were both using the number to announce their separate network prefixes. ARIN issued a statement conveying that they were aware of the problem and were working to resolve it. We took a look at the data and found that AS1712 isn't the only dually-assigned ASN out there. In fact, even a root server didn't escape unscathed.

]]> First let me introduce myself: I'm Doug Madory, the newest member of the Renesys team. I came to Hanover, NH to do grad school and after a two-year stint as a medical center ISO, I jumped at the chance to be part of the cool work that goes on at Renesys. As part of my initiation here, I was handed the task to see if I could find any other examples of dually-assigned ASNs in the various RIRs — particularly ones that are currently in use. This isn't as easy as it sounds, but we will get to that in a moment.

Knowing how to direct packets from an Internet café in Singapore to a network in Buenos Aires is what the Internet routers do. BGP is the protocol Internet routers use to communicate changes in routes from one autonomous system (an organization on the Internet) to another. Each autonomous system is assigned a unique number, its ASN.

These ASNs are primarily used to avoid routing loops (i.e. you shouldn't be learning about your networks from your AS neighbors!). This is called loop detection and operates correctly only if ASNs are unique. No two RIRs are supposed to give out the same IP addresses or the same ASNs, otherwise we'll have bedlam — but this has happened!

In November 2009, it was noted in a NANOG list discussion, AS1712 has been doubly assigned to both a Texas-based organization by ARIN and an organization in Paris, France by RIPE. They were simultaneously using their ASN to announce their respective network prefixes to the world.

$ whois -h whois.ripe.net AS1712
...
% Information related to 'AS1712'

aut-num: AS1712
as-name: FR-RENATER-ENST
descr:   Ecole Nationale Superieure 
         des Telecommunications,
descr:   Paris, France.

$ whois -h whois.arin.net AS1712
...
OrgName:    Twilight Communications
OrgID:      TWILI
Address:    1674 Kosik Ln
City:       Wallis
StateProv:  TX
PostalCode: 77485
Country:    US

ASNumber:   1712
ASName:     TWLT
ASHandle:   AS1712

According to IANA, AS1712 is to be assigned by ARIN, not RIPE. However, according to RIPE's records it appears that many AS numbers in the 1700's were assigned by RIPE in 1993 and therefore AS1712 could have been assigned by RIPE in the swamp days before ARIN even existed.

AS 1712 isn't the only case of this...

Other doubly-assigned ASNs in the 17XX range include:

AS1708 (RIPE: Renater, ARIN: Abacus, San Francisco, CA)
AS1715 (RIPE: Renater, ARIN: Harrier Hawk Management LLC, NY, NY)
AS1716 (RIPE: Renater, ARIN: Critical Data Network, San Diego, CA)
AS1723 (RIPE: Renater, ARIN: Twilight Communications, TX)

ARIN believes that they have worked to fix the problem on the 17XX range, but doubly-assigned ASNs are not limited to the 1700's as evidenced by:

AS3745 (RIPE: Volkswagen, ARIN: Perot Systems, Auburn Hills, MI)
AS35868 (RIPE: Internet Software Consortium, ARIN: Logix3).

In each of these examples, ASNs are both registered and are actively announcing prefixes for different organizations. We have alerted those responsible for these networks of the mix-up. What is fascinating is: why didn't someone check when the duplicate assignment was made? Anyone can query the databases of the five RIRs from the Linux command line in the following way:

$ whois -h whois.arin.net AS1712
$ whois -h whois.ripe.net AS1712
$ whois -h whois.afrinic.net AS1712
$ whois -h whois.apnic.net AS1712
$ whois -h whois.lacnic.net AS1712

In the last example from above, AS35868 is a particularly interesting case:

Logix3, a Florida company, was assigned AS35868 by ARIN in 2005
Logix3 originates their own prefix, globally visible via AS35868, since 24 June 2006.
AS38568 was assigned by APNIC to ISC ("ISC-SUV1") in 2006.
ISC started advertising 203.119.51.0/24 via AS35868 on 23 May 2007 via Fiji's University of the South Pacific, in order to host Fiji's local copy of the F-root server.
AS35868 first appeared in RIPE as ISC on 29 September 2009.

However this gets cleared up, we strongly suspect that the local transit for the Fiji F-root will have to change. (Its address won't, fortunately.) Since we don't have a peer in Fiji, so we can't confirm that 192.5.5.0/24 (F-root block) is anycasted there. Anybody in Fiji want to peer with us?

Despite the fact that verification services are readily available, neither the RIRs, the companies who received the duplicated ASNs, nor their providers seems to have checked if the ASN was assigned before making and accepting the ASN assignment. When an organization is assigned an ASN, it still needs to check to verify that this ASN hasn't already been assigned. One can't assume that the ASN handed to you by an RIR is unique because their databases aren't perfect.

But don't blame the RIRs, rooting out duplicate ASNs across RIRs is non-trivial. To identify duplicate ASNs, one cannot simply look for the same ASN in two RIRs. Some RIRs, like ARIN, try to list many of the ASNs in other RIRs as a pointer to them. Also, some large organizations may legitimately have the same ASN in two RIRs announcing different networks, for example, AS33771 (Safaricom) is listed in both AFRINIC and RIPE, but it probably does so because it announces network prefixes in the two continents (196.201.208.0/20 in Kenya and 41.90.0.0/16 in England). It doesn't help that the RIRs are not consistent in their naming conventions, for example, AS109 is "ciscosystems" in ARIN and "cisco-eu-109" in RIPE. So an effective automated ASN duplication detection tool is unlikely.

Lastly, businesses merge, acquire others or sometimes change their names without updating the records in the RIRs, for example AS3955 is Wang in ARIN and Getronics in RIPE, but Wang and Getronics are the same company. In fact, AS3955 also routes network prefixes (such as 150.124.0.0/16) for a third organization named, Compucom Systems. Compucom bought Getronics in 2007. AS32528 is Ross Labs in ARIN and Abbott Labs in RIPE. Ross and Abbott merged in 1964. Who has the capacity to keep up with every change to a business's name?

Like everything on the Internet, nothing is ever easy, even something seemingly so trivial as assigning unique ASNs. The moral of the story is that, going forward, all parties involved in the assignment process would do well take some simple steps to verify the uniqueness of their new ASN because it gets very difficult to identify duplicates and fix the error after the fact. And we want to avoid more ASN split personalities who think they are in Paris -- both Texas and France!!

À bientôt, pardner!

IP Backbone: Hard sell, not so much

2009-11-20T20:14:55Z

Think you're too busy to blog? Think again. Or just ask your boss. After more than 100,000 miles in coach class this year (so far), my backbone may be aching, but the IP backbone market is as agile and dynamic as ever. Sales opportunities abound, but to take advantage, you'd better be savvy, and just a little cagey.

So, as our gleaming 777 departs Kuala Lumpur, I'll just relax in my fully-reclined, ultra-deluxe coach seat and tell you what this globetrotting sales guy has seen, heard and figured out.

Two new trends
As if the global financial crisis weren't enough, beleaguered NSPs have to rejigger their business plans (yet again) to accommodate encroachment from brazen usurpers and ever more competitive pricing:

Large eyeball networks (5 million+ subscribers) are selling paid peering to the largest content providers.
There are big price reductions in IP transit all over eastern Europe - now close to parity with western Europe.

]]> Just two years ago many large European and some North American NSPs thought eastern Europe would be their salvation, the best way to sustain growth and maintain margins. And why not -- IP transit in eastern Europe was selling for an extravagant €12/Mb/month back then. During 2008, several NSPs took advantage of this seductive pricing to increase their customer base in the region:

Global Crossing
TiNet
Cogent
Deutsche Telekom
TeliaSonea
Interoute
Etc.

Cogent does it again

Not for the first time, most NSPs are blaming Cogent for shaking up the system (spoiling everybody's fun with tough competition). Early this year, they started providing eastern Europe with IP transit for just €4/Mb/month. Despite this, a few providers successfully entered the eastern Europe transit market this year:

TiNet
TeliaSonera
Global Crossing
Cogent

Their success came mainly at the expense of Verizon and Sprint. Level 3 and Telecom Italia Sparkle managed to maintain their positions.

An interesting result of lower pricing

Regional NSPs have abandoned the IP transit market and are using their networks to offer connectivity to major IXPs. And they just might reach out to Russian IXPs next year. I'll keep you posted.

IP sales teams react

So, how are IP transit sales teams coping with this market evolution? It varies. A few have cashed in their IP transit chips and are focusing on enterprise MPLS (multiprotocol label switching) opportunities. Others are reconfiguring their IP networks to gain revenue from peering partners that were previously settlement-free (paid no fees). Some are building their "on-net" business by seizing every available IP transit opportunity. While the most savvy NSPs are relying on Renesys Market Intelligence® to give them a competitive edge. :-)

Road Tip: Get cozy with the crew.

I always chat up flight attendants and/or pilots when I'm going someplace I haven't been to before; after all, they go there regularly and can let you in on stuff you'd never find in a guide book.

On a recent visit to a not-so-well-developed part of South America, I got to talking with an American Airlines captain in the departure lounge at Miami airport. He and the crew would be staying at the same hotel I was, so I asked the usual questions about shopping and restaurants. During his enlightening response he casually mentioned that I shouldn't casually wear my fancy watch outside the hotel. Whoa. I asked him to elaborate. Turns out, just a few weeks ago his copilot stood outside our hotel waiting for a crew bus, all his possessions crammed into a backpack. A motorcyclist zoomed out of traffic, slid impressively to stop right in front of him, produced a gun and demanded the backpack. It was over in a split nanosecond.

Flight crews wait inside the hotel now. So do I.

Paid peering has its effects

Now that large eye-balls have seen their way to paid peering, the traditional wholesale IP transit market has split into two groups. Some have eyeball networks, the rest are wholesale pure players.

In this scenario, incumbents and cable operators with large domestic retail broadband networks have the advantage. So far, only retail-focused NSPs offer paid peering, but only because they don't have large legacy wholesale businesses to cannibalize. I'll be keeping my own eyeballs on IP transit wholesalers with large retail networks (e.g., ATT, Deutsche Telekom, BT) to see if they join the fray. My money's on no. Pure players such as Cogent, Level 3 and TInet have the most to lose if this trend spreads.

On-net business is a sweet deal for NSPs. They get revenue coming and going because both parties pay for exchanged traffic. (If traffic on their network is from a peering link or an upstream provider, they can only charge at one end.) It's not just giants like Level 3 cashing in on this, regional NSPs are doing it at home and abroad.

NSPs are very secretive about their on-net percentages, but insiders believe the largest NSPs' on-net routing is in the 30% range.

Asian tigers hungry for more

The Capacity Asia NSP networking event in Kuala Lumpur was as vibrant as ever this fall. Asian NSPs shook off the financial crisis and are growing again, while their European and North American counterparts continue to struggle. Prices haven't plunged in Asia, and many populous locations are still eager and ready for broadband connectivity.

Asian-generated Internet content is growing rapidly, so they don't want or need more connectivity to the USA. But there was a lot of buzz about the pressing need for a fail-safe mechanism - additional subsea communication cables in key areas. Sites to consider: just offshore from the Singapore landing station and emerging low-cost manufacturing locations such as Bangladesh.

BTW, I heard a rumor that a single cable from Singapore to Hong Kong had four breaks already this year. NSPs with limited fiber diversity were not amused.

Road Tip: The laid-back dichotomy.

I really enjoyed my first visit to Sao Paolo, Brazil last March. As promised, the steak was fabulous, the people friendly and easygoing. This charmingly relaxed attitude extends to the Brazilian consulate in Boston where I applied for a visa about a month before I was scheduled to travel. The application process was effortless, and I anticipated the return of my passport and the requested visa in a few days. That's what usually happens.

Ten days pass. No mail. No contact. I begin to leave increasingly urgent voice mail messages at the consulate. Another ten days go by. I'm running out of time and options, so, I dive into their IVR system maze hoping for contact with a sentient being, and finally get connected to . . . voice mail. OK, I'm supposed to leave in three days now. It's visa hotline time. To my surprise, a very amiable fellow instantly responds . . . oh yeah, he seems to remember something like that, and he's pretty sure he put the paperwork somewhere.

Brazilian visas are good for five years. So, if you're planning to go to Rio for the 2016 Olympics, apply for your visa in 2011. On the other hand, if you're planning a Russian excursion, not to worry - nothing escapes their attention. You'll have a gorgeous, foil-embossed document within days. Sunny, relaxed Rio, or frigid, efficient Russia. You decide.

A kind of twisted unity

Google and a group of NSPs (Bharti Airtel, Global Transit, KDDI Corp., Pacnet, and SingTel) recently formed a transpacific cable consortium. They call it Unity. Cable deployment is ahead of schedule and should be activated soon, but with an interesting twist. Members pay a share of the infrastructure cost, and get to install their portion of the cable in their own optical equipment at both Chikura and Los Angeles. So, some members may deploy 10G waves, while others might fancy 40G or even 100G.

A few NSPs quietly mentioned that Google has already been offering to sell bandwidth on their part of the cable in the wholesale market. This puts some NSPs between a rock and a hard competitive place -- with one of their biggest customers!

This creates an intriguing dilemma for Google, according to a financial markets expert I chatted with at the Capacity show. Google's P/E valuation has been driven by its reputation for disruptive technologies since the beginning of Google-time. Although it's unlikely, if they also start operating as a telecom utility (offering bandwidth and all sorts of cellular technology/services), that could change. P/E values for NSPs, wireless equipment manufacturers and wireless operators are rather modest to say the least. So even if Google has spectacular success selling bandwidth, I can't imagine it would contribute to even a modest change in their P/E expectations.

Will the bumpy IP transit ride continue in 2010? I'll be watching and reporting from my cushy coach seat.

Cheers!
Bob Fletcher
Sales guy

Glossary:

An Eyeball Network is a content provider, such as a cable TV provider or DSL access network, which connects its content viewers (eyeballs) to the Internet through an NSP.

An Internet Exchange Point (IXP) is a physical infrastructure that network service providers (NSPs) use to exchange Internet traffic between their networks. It enables networks to interconnect directly, rather than through third-party networks.

Peering is a mutually beneficial interconnection of separate Internet service providers to exchange traffic between the downstream customers of each network. With paid peering, fees are accessed by the NSP that receives significantly more downstream IP traffic than its peering partner.

Lights Out in Rio

2009-11-11T06:43:25Z

When the power goes out to a large part of Brazil, as happened last night shortly after 10pm, it's going to have an impact on telecommunications.]]> When power drops, there's an immediate shock to the Internet routing system. You can see it clearly in this plot, which shows the number of South American networks affected by serious instability or unreachability in the previous 15-minute window.

By compiling the routing traffic we collected from all over the world during this period of time, it's possible to pinpoint the time of the failure in the Brazilian electric grid (10:26pm, or 00:26 UTC). From an essentially negligible background level, the number of affected networks vaults to more than 150 within seconds. Following the classic pattern of widespread power failures, the number of freshly affected networks drops over time as backup power kicks in, and then there are 'echoes' of new failures as the backup power suffers its own problems.

These are some screenshots from our KML-based regional outage visualizer, and our Internet Health Portal tool. As you can see, Brazil took the largest hit, but we also saw Paraguayan and Uruguayan networks out as a result of the largest regional power outage to hit Brazil and its neighbors in several years.

Inevitably, speculation will fly fast and furious about the cause. Media reports this week suggested that hackers may have taken down the Brazilian grid in 2005 and 2007. Brazil says it simply wasn't so. Was tonight a sardonic repeat performance, in response to Sunday's 60 Minutes coverage?

Staring Into The Gorge: Router Exploits

2009-08-19T10:32:31Z

I'm writing this blog entry from the campground at Vermont's beautiful Quechee Gorge, where I took the kids after work. Yes, Renesys is located smack in the middle of some of the nicest hiking, camping, and climbing on earth. No, you shouldn't move here, Northern New England has enough out-of-staters already, thanks. Unless, that is, you are an unusually talented web developer, have worked as a peering coordinator, or know the Internet transit industry inside-out, in which case you should send me your CV, posthaste. thanks, --jim

Here We Go Again.

Imagine an innocent BGP message, sent from a random small network service provider's border router somewhere in the world. It contains a payload that is unusual, but strictly speaking, conformant to protocol. Most of the routers in the world, when faced with such a message, pass it along. But a few have a bug that makes them drop sessions abruptly and reopen them, flooding their neighbors with full-table session resets every time they hear the offending message. The miracle of global BGP ensures that every vulnerable router on earth gets a peek at the offending message in under 30 seconds. The global routing infrastructure rings like a bell, as BGP update rates spike by orders of magnitude in the blink of an eye. Links congest. Small routing hardware falls over and dies. It takes hours for things to return to normal.

]]> Last time, it was the African Network Operators' Group, who had the temerity to actually use those fancy new 4-byte autonomous system numbers in a heavily prepended production context, causing certain Quagga routers to reset sessions.

The time before that, it was a small Czech ISP, who tickled a missed input range check in the configuration screen for the relatively obscure Mikrotel routers, which led them to send out updates containing massively bloated AS paths, which caused certain Cisco routers to reset sessions. Cisco promptly issued a security advisory and patched the problem.

And This Week ....

At 17:07:26 UTC on Monday, CNCI (AS9354), a small network service provider in Nagoya, Japan, advertised a handful of BGP updates containing an empty AS4PATH attribute. Yes, a BGP update whose path attribute contained just three bytes: one byte for flags, one byte containing the AS4_PATH opcode, and a size byte equal to zero.

If you were writing this code, would you do something sensible if the ASPATH contained zero autonomous systems en route to the prefix in question? It's not terribly meaningful, but it's just a range check. It doesn't seem like an outrageous corner case to anticipate and handle correctly. Especially if you're writing the operating system for a big carrier-class router, where this logic is on one of the most important code paths for performing the mission of a very expensive piece of mission-critical hardware.

This one seems to have bitten Cisco's IOS XR, a relatively newborn "from scratch" rewrite of the venerable IOS, destined to run on big iron, like the CSR-1 or the 12000 series. If you were trying to cause a meltdown of the global Internet, this would be the kind of platform you'd target. Cisco promptly released a security advisory and patched the problem. (We refrained from blogging about this for a couple days in order to give everyone time to get fixed, but I expect that the people who run the kinds of big iron that use IOS XR are probably well-cared-for by Cisco's support team and didn't have to wait long.)

Musing at METRICON

Coincidentally, I was at USENIX last week, and sat in on a great workshop on security metrics, where Sandy Clark gave a somewhat controversial presentation on the interaction between software quality and the timing of exploit appearances. (Sandy is a member of Matt Blaze's secure systems research group at UPenn.)

In a nutshell (and Sandy, please correct me if I mangle your argument), one of the strongest predictors of a significantly large time to the emergence of the first zero-day exploit for a new version of software is the degree to which the release represents a substantial rewrite of the code. Doing a rewrite seems to start a "honeymoon period, " during which time the system in question is safer from exploitation than it has been in a long time. In fact, the magnitude of the protective effect is so significant, that you might ask yourself whether a dollar spent in pursuit of higher quality code is actually better spent rewriting the code periodically, to whatever quality standard you can achieve. Maybe using robots. Seriously.

I had a chance to chat with Matt and Sandy briefly afterward, and we mused about the implications for the security of the OSes that Internet routers run -- Cisco IOS, Juniper's JunOS, Quagga, etc. To be fair, the threat model is completely different. You typically can't speak BGP to someone's router, for example, outside the context of a preapproved trust relationship. But once you're talking, you're talking (through them) to every other router on the planet. The stakes are a lot higher, but the value to an attacker is less clear-cut — crashing the Internet doesn't give you control of tens of millions of valuable computing platforms, the way a zero-day exploit in Internet Explorer does.

So, here we have a serious vulnerability (the first I know of) in a substantially rewritten version of a critical operating system, one which has been out there in the market for at least a few years without making major news in a bad way.

The honeymoon is over, in other words.

How Long Before This Happens Again?

The good news is that the Cisco TAC is at the top of its game, getting in touch with affected customers and rolling upgrades. The bad news is that all of us (not just Cisco) are evidently locked in this completely reactive mode of finding and fixing problems in the world's critical communications infrastructure. Again and again, we have to wait to discover problems in the real world that should have been caught through automated techniques during development, or during testing in the lab.

The global mesh of BGP-speaking routers that we call the Internet has inherent vulnerabilities that stem from the software quality and policy weaknesses of its weakest participants, and the amplification potential of its best-connected participants. Running sloppy software at the edge of the routing mesh (in enterprises, say) is unlikely to give anyone the ability to propagate large amounts of instability or partition the Internet. But closer to the core, I think we have a serious problem to contemplate.

Remember, if you can get just one provider to listen to you, and not filter your announcements, you can get your message into the ear of just about every BGP-speaking router on the planet within about thirty seconds. And if some subpopulation of those routers can be reset, they act as amplifiers for your instability. Power law outage-size distributions are not a myth — they are a logical consequence of the structure of the Internet, the importance of a few key participants in carrying global traffic, and their reliance for interconnection on technologies that are clearly still in the shaking-out-the-obvious-bugs mode.

The honeymoon is over, folks. We're staring into the gorge, and it's a long way down.

Routing Redundancy: How much is enough?

2009-08-15T17:17:30Z

Internet connectivity is a good thing. Many of us depend on it for everything from our livelihoods to our entertainment. However, the Internet is very fragile and even the The New York Times is worried about it. But they're primarily concerned with overloads that can occur when everyone on the planet does the same thing at roughly the same time, such as surfing for news about Michael Jackson. Unfortunately, we will never avoid all such scenarios. Physical systems are designed around average and typical peak loads, not around extremely high loads associated with very unlikely events. Who would pay for that?

And this applies to other complex systems besides the Internet. I was in India during 9/11 and, for two days, I could not make a traditional phone call to the US. Why? Everyone in India knows someone in NYC, and they all picked up the phone at the same time to check in on them. The circuits were so overloaded, I couldn't even get the friendly "Your call cannot be completed as dialed" message.

No system is ever going to be engineered for insanely high loads. If everyone in your town decides to take a shortcut through your neighborhood to avoid an accident on the highway, you are going to have trouble getting out of your driveway. But rather than give up and wait it out, there is something you can do in advance and at reasonable cost: build a second driveway to a different street on the other side of your house, one that isn't fed by the same access roads from the highway. This blog is about building such redundancy into your Internet connectivity, so you aren't disconnected by a single failure. And while it's good that the New York Times and various governments are watching the problem, if your business depends on the Internet, you're largely on your own to audit and verify that you are buying a sufficient level of redundancy for your budget. A lot of fragility problems could be solved by more informed consumers performing the necessary due diligence.

]]> To plan for inevitable disruptions in service, you must find and then eliminate single points of failure. A small business with a single provider, for example, has a single point of failure because if that provider goes offline, so does the business. Larger organizations often combat this problem by having multiple providers, but it still may not be enough. In particular, if all of their providers ultimately rely on the same still-further-upstream provider. This kind of indirect single point of failure, illustrated in the first graphic below, is usually more difficult to identify. Other kinds of hidden single points of failure are also important, such as when one's two providers both run their cables through the same physical conduit. If one errant backhoe severs the conduit, this "redundancy" is rendered useless. For this discussion, though, we'll focus on organization-level redundancy.

Single points of failure: direct (left) and indirect (center and right)

Planning for direct provider redundancy is generally straightforward. But avoiding indirect upstream single points of failure requires an awareness of the ever-evolving business relationships found on the Internet. To understand these relationships, we first have to introduce some terminology. As readers of this blog will know, every organization with multiple Internet providers will tend to have two things under their control: block(s) of IP addresses (prefixes) and an identifying number, or ASN. Renesys currently observes over 110,000 relationships between pairs of ASes on the Internet that have decided to interconnect. By carefully studying routing data from numerous vantage points around the globe, we are able to classify these relationships and determine both the direct and indirect dependencies between providers. Once you map out these relationships, you can then look for the single points of failure.

True diversity — complete freedom from single points of failure — is illustrated in the following graph, where AS 1 has three providers, each of which is richly connected to the rest of the Internet. One or even two of AS 1's providers can have technical problems and AS 1 will still be able to maintain its Internet presence. This organization has planned well for eventual failures.

Diversity: no single points of failure

So securing direct redundancy is fairly easy, but beyond that it gets complicated. Large ASes typically encompass many geographic locations and might even have offices all over the world. If AS 1 has three providers, does it have them at all of its worldwide locations? If not, maybe they have an internal network that connects their offices. Hence, an office whose local Internet connection fails might still be able to reach the Internet via one of the other offices. In other words, you cannot view an AS as a single entity, as it may in fact consist of many independent locations with their own providers and private connectivity to other parts of the organization. As if sorting through all this wasn't enough, there is a final complication. Mergers, buyouts, spin-offs and other aspects of modern capitalism can result in a single organization consisting of many ASes. Finding them all is quite complex, as we discussed in a previous blog.

To truly measure diversity, we cannot merely look at ASes and their relationships. We have to look at organizations as a whole and how their associated prefixes are routed. Does this particular prefix show diversity in how it is routed or is it solely dependent on one provider? Nor is it sufficient to look at one instant in time. Rather we have to track the prefix over longer periods in order to uncover seldom-used backup routes. If the prefix really has multiple ways to reach it, you want to account for that and give credit where it is due.

With these ideas in mind, we set about trying to measure Internet diversity for each organization. Every associated prefix is given a score based on the amount of diversity observed over a period of two months. Then the individual prefix scores are combined to give an overall score to the organization in the range of 0 — 100. A score of zero represents no diversity at all (like the left side of the first graphic), and a score of 100 represents very rich diversity (as in the second graphic). We leave the details of the algorithm to a follow-up piece, which will also include data for the Internet as a whole. For now let's look at a few examples of individual organizations.

Amazon is a well-known and hugely successful commercial retailer with a worldwide presence. They have around a half dozen ASes and almost three dozen prefixes. At an AS level, the Renesys Market Intelligence tool shows Amazon with a rich set of top-tier providers, such as Level 3, NTT, Tinet (formally Tiscali), Qwest and others. But are all Amazon prefixes seen via such a diverse set of providers, at least once in a while? In fairness, Amazon might only care about diversity for some of its prefixes, but since we can't know which ones matter, we treat all prefixes the same and score them all based on their observed diversity. Then we combine the results to get an overall Amazon score.

And the answer is … (drum roll please) … 99.16 for Amazon's total score. They miss out on a perfect score of 100 only because of a few less diverse prefixes, one of which geo-locates to Hong Kong and is only seen via NTT. As you might expect, Amazon has very rich global Internet diversity and generally has no single points of failure with respect to its logical connectivity.

In total, out of over 75,000 organizations that we have identified, not all of which have their own ASNs, over 9,500 score a perfect 100. But most of these are rather small or geographically constrained, so they perhaps have an easier time of implementing diversity once they realize the importance. An example of a larger organization achieving perfection is Vietnam Posts and Telecommunications (AS 7643). Their hundreds of prefixes are all seen via numerous large providers ranging from AT&T to China Telecom.

At the other end of the spectrum, we have the United States Social Security Administration. Their 18 prefixes show no diversity at all, as they are all seen only via Verizon (AS 701). No matter how many physical connections Social Security has to Verizon and no matter how reliable Verizon has been for them in the past, no service provider can guarantee their customers uninterrupted service to the entire Internet. Through no fault of Social Security or even necessarily Verizon, if another top-tier Internet provider were to de-peer Verizon, then Social Security would lose access to parts of the Internet. Since we saw two such top-tier de-peerings in 2008 alone, between Cogent and TeliaSonera and between Cogent and Sprint, this is not a far-fetched concern. To ensure their connectivity and improve their score, Social Security would need to pick up additional providers, ones not solely dependent on Verizon. But they are not alone: over 26,000 other organizations also have zero diversity.

Now let's return to the New York Times, whose article motivated us to write up this work in the first place. Should the Times worry about its own Internet diversity? Looking at its approximately half-dozen prefixes, the company receives a respectable score of 85.37. But if we disregard two of its smaller prefixes, keeping just the larger ones and the one in which www.nytimes.com lives, its score increases to 93.5. So while the Times has some room for improvement, someone in the organization is clearly paying attention to the issue of redundancy when provisioning for its most important networks.

For our more technically inclined readers, a subsequent blog will provide more of the details around our diversity algorithm and give results for the Internet as a whole. Not surprisingly, as our examples suggest, the results vary widely.

The Proxy Fight for Iranian Democracy

2009-06-22T11:30:00Z

If you put 65 million people in a locked room, they're going to find all the exits pretty quickly, and maybe make a few of their own. In the case of Iran's crippled-but-still-connected Internet, that means finding a continuous supply of proxy servers that allow continued access to unfiltered international web content like Twitter, Gmail, and the BBC.

]]> A proxy server is a simple bit of software that you run on your computer. It effectively lets you share your computer with anonymous strangers as a "repeater" for content that they aren't allowed to fetch themselves. For example, an Iranian web browser might be manually configured to use your computer (identified by an IP address and a port number) as a Web proxy. When your anonymous friend reads twitter.com, or posts a tweet, the request goes via your computer, instead of to Twitter's web server directly. Except for a little delay, and the fact that your friend gets to see what the uncensored Internet looks like from New York or London or São Paolo instead of Tabriz or Qom, surfing through a proxy is pretty much like surfing without one.

As you might imagine, open web proxies are valuable commodities in places where it's forbidden, possibly dangerous, to surf the Internet. Iran's opposition movement has been vigorously trading lists of open proxies over the past week. And as you might further imagine, the Iranian government censors have worked overtime to identify these proxies and add them to the daily blacklists.

As an experiment, we geolocated a list of about 2,000 web proxies (unique IP addresses and port numbers) that were shared on Twitter and other web sites over the course of the last week, to see if we could discern patterns in the places that are hosting them. Most of these are no longer reachable from inside Iran, of course, precisely because they were made public. The following map shows the distribution of those proxies worldwide.

The USA and Western Europe were well-represented, but so were China, India, Russia, Romania, Bulgaria, Vietnam, ... 87 countries in all, a pretty impressive breadth of representation, considering the relatively small size of this sample. (You can also see about a dozen Iranian IP addresses represented in the set. Not surprisingly, all but one of these belong to networks originated by DCI, the government-run service provider who operates the modern-day Internet equivalent of the Alamūt Castle.)

Here's a geographic visualization of the proxies, drawn in Google Earth. In the first one, we've drawn Iran in green, with some of their domestic network sketched in white, and their major international connections drawn in red. Each of the colored arcs represents a single open web proxy; they are "fountaining" out of a cable landing or Internet traffic exchange point that makes approximate sense for their Iranian Internet routing. For example, all of the web proxies in Europe are drawn from the Marseilles termination of the Sea-Me-We-4 cable. The web proxies in Turkey are drawn in light blue, radiating from Ankara, where the Iran-Turkey gas pipeline passes through on its way from Bazargan. Those unusual Iranian proxies emerge from Tehran, and so forth.

If we rotate the globe, you can see how the countries of Asia are doing their part to keep the bits flowing in Iran. India, China, South Korea, Taiwan, Vietnam, and Japan are all visible sources of web proxy activity.

I'd like to be able to say that these maps are a measure of the strength of the democratic impulse and volunteer spirit in all the countries of the world. But that might be a stretch. You see, looked at another way, an open proxy is a security hole, something you might find in a machine that's been compromised, or at the very least, badly administered. Security purists think of them as the "unlocked gun cabinet" of the Internet — a resource for anyone who wants to abuse a website, commit fraud, cover their tracks.

Some of the proxies in this dataset are undoubtedly fresh, created by people who want to keep the Internet alive for the Iranian people. But many of these proxies have probably been around for months or years, mapped out by those that map out such things.

We did see a few organizers try to explain the concept of an ACL (Access Control List) to all the new proud parents of open proxies. If you are diligent, it is possible to restrict the anonymous users of your new proxy to just the Iranians, or even just the Iranian non-government networks, if you have a good enough list of the IP address blocks (network prefixes) in question. But I expect that the complexity of configuring anything tighter than an "open access" proxy is going to prove too high a barrier to entry for most people who might volunteer to run one.

For one thing, we know how hard this is. Renesys has pretty good lists of per-country networks and their transit patterns, based on our analysis of the global routing tables, and trust me, they take some work to maintain. And even given good maps of Iran's address space to work from, ACLs are notoriously hard to test, if you don't have Iranian friends who can try your server from inside the protest zone and report back to you with problems. Most people aren't going to bother, and that's probably okay. Freedom is messy. There'll be time for security later.

Perhaps the strangest thing of all, given how diverse and active and vocal the proxy server farmers have been, is that by and large, it isn't working. The rate with which new proxies are being created has slumped over the last few days. It's getting harder and harder to propagate new proxies to the people who need them, as the government consolidates its hold on the filtering mechanisms. Any new proxy addresses that are posted to Twitter, or emailed, will be blocked very quickly.

People we talk to inside Iran say that almost no proxies are usable any more. Freegate, a Chinese anti-censorship application that makes use of networks of open proxies, has proven popular in Iran. But this week, it, too, has been experiencing problems. Many popular applications, like Yahoo! Messenger, have stopped working. The authorities are said to be using power interruptions as a cyberweapon, causing brief outages during rallies that cause computers to reboot, just as people are trying to upload images and video. The net result, as Arbor's excellent analysis shows, has been a drastic reduction in inbound traffic on filtered ports since the election.

If there's a lesson here for the rest of the world, perhaps it's this: Install a few proxy instances on machines you control. Learn how to lock them down properly. Swap them with your friends overseas who live in places where the Internet is fragile. Set up your tunnels and test them. And don't wait until the tanks are in the streets to figure this out, because by that point, you may have already lost the proxy war.

Iran and the Internet: Uneasy Standoff

2009-06-16T21:21:25Z

We've received enough interest about our previous notes on Iranian Internet connectivity that I wanted to give a brief update, and some reflections.

]]> In short: Iran is still on the Internet. As the crisis deepens, people are literally risking their lives by continuing to use the Internet for coordination and communication. Iran's physical connectivity to the Internet is so centralized, and so fragile, that it's within the power of the government to simply "turn it off" if they so desired. And yet, they have not done so.

Except for a brief period of outage over the weekend, the routes into Iran from the rest of the world have been basically intact, if a bit congested and unstable. Most of that congestion and instability is probably the result of six billion people who are freshly interested in Iranian politics, all reading (and in some cases, yes, attacking) Iranian websites. We aren't making things easier for the people inside Iran, who need that same bandwidth to get out their images and observations and tweets.

To show that nothing much has changed structurally, consider the fact that the same lineup of six international carriers are still carrying the megabits back and forth to the government's monopoly points-of-presence at the Iranian border. (See the following chart, which shows how the relative percentages of routes to Iranian networks carried by those providers has changed over the last few days.)

What happens inside Iran to those bits is anyone's guess (censorship, site blocking, traffic interception, and harassment, from all accounts). But the pipes are open and the traffic is flowing. In a few cases (which I will not detail, for obvious reasons), there are actually direct paths to international carriers, in defiance of government monopoly, that are now getting good use.

I've talked to a lot of people in the last few days about this puzzle. In a crisis that verges on revolution, the first thing a government typically does is take control of the media that can be controlled, and shut down the media that cannot.

Why is it different this time? There seem to be three basic theories.

The cynics. Perhaps the government has left the Internet intact so that they can use it to surveil and round up dissidents. Perhaps they even put bandwidth constraints in place to make it easier to cope with the volumes of traffic that need to be captured and filtered.
The optimists. Perhaps the government has realized that a modern economy relies on the Internet to such an extent that it cannot be turned off, for fear of disrupting financial transactions and business communications. Iran's Internet ecosystem is relatively rich, and the impact on their economy of a sustained Internet shutdown would be significant. Why make it harder for companies to do business in Iran at a time when oil revenues are cratering and foreign investment is looking for reasons to take a walk?
The realists. Perhaps the government is too busy with other things to worry about the Internet. Governments aren't well-suited to run the Internet, and they don't completely understand how it works. The Internet has never been "turned off" before, and it would take creativity and thoughtful action to figure out who to ask in order to get it done. So it simply hasn't happened, and probably won't. Good thing, too, because they might not be able to turn it on again.

You can pick the theory you like. I guess I'm a realist, but I'd like to be an optimist. If you wait long enough, something good can come out of something bad. If fertility rates hadn't tripled during the long Iran-Iraq war in the 80s, Iran wouldn't be faced with this demographic bulge of restless 20-somethings who have grown up with the Internet and expect it to keep working.

You can fight people, but you can't fight human nature, and you can't fight demographics. Iran is going to follow the same long-term trend as the rest of the developing world: building a civil society based on free expression, global communication, free mobility of human capital, and cross-border investment. That's what the Internet symbolizes, and yes, if you ask these folks, it's worth fighting for.

Strange Changes in Iranian Transit

2009-06-14T12:33:22Z

Many media sources have reported outages in Iranian mobile networks and Internet services in the wake of Friday's controversial elections. We took a look at the state of Iranian Internet transit, as seen in the aggregated global routing tables, and found that the story is not as clear-cut as has been reported. ]]> There's no question that something large happened in the Iranian telecom space, and that the timing aligns with the close of voting and the emerging controversy. Iran typically has a fairly high baseline level of sporadic route instability, due to the country's highly centralized incumbent transit through DCI (Data Communications Iran, AS12880) and DCI's somewhat peripheral connectivity to the main east-west conduits for data. Even so, we started seeing spikes of route instability (changes in the paths to Iranian IP space) starting around 08:05 UTC on Saturday (just after noon in Tehran) that were significantly larger than normally expected. These bursts affected as many as 400 prefixes (blocks of IP addresses) — the majority of Iran's Internet presence.

At 17:48 UTC, instability turned into outage, as more than 180 Iranian networks were withdrawn from the global routing tables, indicating that there were no remaining paths into DCI for that portion of Iranian traffic. Contrary to media reports, however, the outages were fairly short-lived. Within a few minutes, half of the outaged population were restored to alternative transit; over the course of an hour, outage levels returned to their normal baseline. Route instability continued to be fairly high, and that pattern has continued through the night and into Sunday.

What can we say for sure? Not much, except that Iran remains well-connected to the Internet from a routing perspective. If I had to guess, I'd say that there are probably a lot more people around the world pulling local content from Iran's providers right now, and that surge of demand is probably contributing to increased congestion and (perhaps) some of the route instability we see. It wouldn't be unusual for there to be some inbound cyber-mischief as well, from supporters of one or the other side, but so far we only have rumors on that front.

It is interesting to note that the changes in routing that took place were very specific in their impact on DCI's various transit providers, who keep the country connected to the world. There are six of them: Turk Telecom (TTNet, AS9121), FLAG (AS15412), Singapore Telecom (AS7473), PCCW (AS3491), Telia (AS1299), and Telecom Italia Sparkle (AS6762). As the following plot shows, five of them lost Iran's transit, and one of them (Turkish Telecom) was a big gainer. (Red arrows indicate loss of transit preference from the outside world; green indicates a gain in transit via the given provider.)

A transit shift of this magnitude may indicate that something (administrative, or physical) has affected Iran's connection to the submarine cables running east and west — not a total outage, but some kind of significant impairment. Turkey has their own, interesting arrangements with Iran for transit, and those are still in good shape (perhaps somewhat congested, having presumably doubled or tripled in transit volume). It wasn't unusual to see 300ms traceroutes from North America and Europe in this timeframe to many Iranian sites.

Of course, you have to remember that globally visible routes are the signposts for inbound traffic to and through DCI to the local providers; from the outside, there's no telling what the Internet experience of the average person inside Iran is like today. It sounds as if a lot of content is being blocked within the country. For now, it's a good sign that information continues to flow, and Iran is still connected to the world at large. Let's hope they stay connected.

How a Resilient Society Defends Cyberspace

2009-05-30T02:45:48Z

Seventy-five years ago today, on May 29th, 1934, Egyptian private radio stations fell silent, as the government shut them down in favor of a state monopoly on broadcast communication. Egyptian radio "hackers" (as we would style them today) had, over the course of about fifteen years, developed a burgeoning network of unofficial radio stations. They offered listeners an unfiltered, continuous mix of news, gossip, and live entertainment from low-powered transmitters located in private houses and businesses throughout Cairo.

It couldn't last. After two days of official radio silence, on May 31st, official state-sponsored radio stations (run by the Marconi company under special contract) began transmitting a clean slate of government-sanctioned programming, and the brief era of grass-roots Egyptian radio was over.

]]> Was there something in the air in the early summer of 1934? Two weeks later, the United States Congress passed the Communications Act of 1934, which established the Federal Communications Commission, and gave it regulatory authority over not only the US airwaves, but interstate telephone services and telegraphy -- in short, all forms of over-the-air and wireline communications among private citizens. In the United States, as in Egypt, the government's oversight of public and private communications has survived, basically intact, for the last seventy-five years. Telephony replaced telegraphy, and television supplanted radio, and the Internet threatened to absorb all of them in a wave of convergence. But throughout all of the technological change, the US government's role as regulator and defender of the public interest seemed to have survived in recognizable form.

That's why it's refreshing to see at least some evidence, in today's release of the long-awaited Cyberspace Policy Review, that the Obama administration is taking a long, careful look at how to update the Government's role in "protecting the Internet." The report nods to regulatory precedent, but makes it clear that new regulations are not the first order of business. Instead, the themes of the day are those recognizable to any digital native: bottom-up coordination, transparency, sharing, establishment of trust, resiliency in the face of unknown threats. Leadership comes from above, but it takes the form of continuous measurement and accountability and consensus-building.

Sounds kind of "kumbaya," right? Well, yes. But I suspect there's actually some substance behind the rhetorical styling.

For one thing, the President has consistently, though subtly, identified "resiliency" as a key virtue in the various complex systems that make up the critical infrastructure behind American society. Having just merged the Homeland Security Council and National Security Council, the President is building an integrated National Security Staff that will include both a senior cybersecurity advisor (yet to be named), and a more general "resiliency policy" advisor, who will focus on survivability and flexible response. The emphasis on resilience, on building complex systems that are less brittle and more intelligently flexible and responsive in the face of attacks, are the kinds of wonky concepts that light up the eyes of computer scientists and mathematicians who study complex systems. They aren't there to provide sound bites for the Sunday talk shows (and, indeed, concepts like "resiliency" tend to get a lot less play in the popular press than firmer fare, like "eliminating vulnerabilities" and "shutting out attackers" and even, God help us, "turning off the Internet in case of attack").

The very fact that the report has been some time in arriving, and that it does not contain a laundry list of quick fixes, suggests that the President intends to be appropriately deliberate in building a picture of what, exactly, the government can do to improve its own cybersecurity posture. Of course, it also signals how thickly political the whole cybersecurity issue has become, as various factions battle it out for control and presidential access.

I think we can take some cheer in the fact that the President isn't rushing to action, that the balance between safety and civil liberties appears front and center in the terms of debate, and that there's a general awareness of how much economic and strategic value is created for the American people by the wild global Internet, even as it creates some novel (but managable) risks for continuation of government and critical infrastructure.

1934, thankfully, is a long time gone.

Reaching Google via Asia?

2009-05-15T01:42:17Z

Across the Internet, yesterday, Google users twittered, blogged and emailed that Google search and mail were not usable. And, yesterday afternoon, on Google's official blog, Urs Hoelzle reported that Google "direct[ed] some [...] web traffic through Asia".

]]> Renesys observes the topology of the Internet by collecting routing data from a wide variety of vantage points (peers). Different peers may choose different routes to the same destination (e.g. Google) depending on their location in the Internet. Analogously, a driver starting from Los Angeles will choose a different highway to reach New York City than a driver starting in Montreal, Canada. On the Internet, each peer may choose a different best path or route to a destination. By recording the choices each peer makes, we can gain some insight into the relative importance of a given route to a destination.

The Internet is carved up into IP blocks (called prefixes) which make management of the network easier. Each provider learns about the prefixes of other providers by passing the knowledge around (like a game of 'telephone'). Google directly advertises or provides service to approximately 190 different prefixes, so the Internet learns how to reach Google based on the relationships Google has.

When trying to reach Google, then, any inbound traffic to them needs to pass across other providers with whom Google has a BGP routing relationship. We'll look at a very obvious shift in the number of other Internet providers who started reaching Google differently during yesterday's event.

Even though Google is very well-connected to a diverse set of other networks, two of these relationships predominate. It is through these relationships that much of the Internet reaches Google. If we look at Google's primary providers, Level 3 (ASN 3356) and AT&T (ASN 7018), we see that these are very important relationships to Google, since in the case of Level 3, just under 80% of peers choose to reach Google via Level 3 for just over half of all of Google's prefixes.

During yesterday's routing instability, Google's prefixes shifted to different relationships. When automatic, this sort of change is a key feature of redundancy. Sometimes such changes are motivated from a business change. Over the course of yesterday's instability, virtually all of our peers, at one time or another, chose to reach many of Google prefixes through NTT (ASN 2914), what had been a far less important relationship.

Note that the bar chart above is showing the total percentage of Google's approximately 190 prefixes and the total percentage of peers (different vantage points on the Internet) choosing to reach Google through NTT. At no single time did the entire Internet choose to reach Google through NTT, but over the course of the event, almost everybody tried to reach Google on this path.

Before yesterday's event, NTT was not so important an inbound path to Google. In fact, NTT carried very few of Google's prefixes (only 16) and was the preferred path to Google for only 10% of our peers. During several different bursts of announcements, this relationship became a much more important inbound path to Google.

In one particular series of routing changes, which started at around 14:45 UTC, several major providers, e.g. Deutsche Telekom (ASN 3320), Teleglobe (ASN 6453), Telia (ASN 1299), and even, peculiarly, AT&T (ASN 7018) started choosing to reach Google through NTT.

While yesterday's instability was disruptive to Google users all over the Internet, we are happy to report that it's back to business as usual with one of the most robust services on the Internet.

AfNOG Takes Byte Out of Internet

2009-05-05T20:47:11Z

A couple of months ago, we discussed how a small Czech provider ended up causing global Internet mayhem by tickling a Cisco bug via a rather ridiculous routing announcement. While it's easy to fault the instigator of this meltdown, ultimate responsibility belongs with the vendors of poorly tested code. If we've learned anything in decades of software engineering, it is that you can't assume anything about user input. If you don't check that input for validity, you are not just being careless, you are creating a time bomb that will eventually go off. Another such bomb went off on Sunday, 3 May 2009, taking out a large swath of the Internet. We recount the sorry tale here.

]]> Sundays are usually pretty quiet from an engineering perspective on the Internet, so we were surprised by a sudden and prolonged influx of routing updates at our collection sites. Thankfully, we have some very nifty event correlation software, written by Renesys über-hacker Alin Popescu, to which I turned to see what was going on. Alin's code sorts through all the routing activity on the Internet in real time and correlates activity via the attributes they have in common. We can then say things like "this spike in outages came from networks in Kyrgyzstan as transited via Golden Telecom in Russia", potentially providing some insight into the root cause of the problem. It's really very slick. However, the only problem with pinpointing Sunday's event was that it occurred all over the place. We saw sharp spikes in outages and instabilities in Indonesia, Bulgaria, Germany, Romania, Brazil, Russia, the United States and many other places. Upwards of a couple thousand prefixes (networks) were impacted during a period of about 6 hours. Given the wide reach of the event, we immediately thought of the Czech situation and wondered if this was another live Internet QA test. Turns out we were right.

To understand the problem, let's first provide some background. As readers of this blog will know, every organization responsible for Internet routing has two things under their control: block(s) of IP addresses (prefixes) and an identifying number, or ASN. Like IP addresses themselves, ASNs are also running out, at least the old style ones. The original Internet designers allocated only 2 bytes for storing ASNs, which means at most 65,536 of them. Since then, ASNs have been largely allocated consecutively starting at 1, with Phosworks Drift & Services getting the most recent allocation of ASN 49232. Reserved (private) ASNs are at the high end of the range, starting at 64512. So this means that at present we only have about 15 thousand 2-byte ASNs remaining unassigned.

If there weren't an alternative, the 2-byte limitation for ASNs would mean that before too much longer, there would be no new members of the Internet routing club: no new ISPs and no companies with multiple providers and direct control of their routing. Fortunately, there is an alternative available today, and that is 4-byte ASNs. RIPE has been assigning them by default since the start of 2009. But unlike IPv6, the proposed alternative to dwindling IPv4 addresses, 4-byte ASNs can seamlessly and safely co-exist with 2-byte ASNs. No complete overhaul of the Internet is required.

Returning to our story, if 4-byte ASNs are available and interoperate with 2-byte ASNs, there shouldn't be any problems with announcing them anyway we want today. Yeah right. The African Network Operators' Group (AfNOG) will soon be meeting in Cairo, Egypt, and plans to be multi-homed. They acquired the 4-byte ASN 327686, which equals 5*(2¹⁶) + 6 and hence is also written as 5.6, and planned to use it as stated here:

One of the aims of this setup is to demonstrate that 32-bit ASNs do work and people should not steer away from them, especially since the pool of 16-bit ASNs is shrinking fast. A showcase network goes a long way in this regard.

Now, AfNOG does have an IPv4 prefix, 196.200.208.0/20, and a traditional 2-byte ASN, namely 37095, at their disposal. On 30 April at 17:21:49 UTC, we started seeing the 2-byte ASN announcing this prefix, but with ridiculously long AS paths like

24863 37095 37095 37095 37095 37095 37095 37095 37095 37095 37095 37095

AS 24863 is LINKdotNET, a local Egyptian provider. So far, so good. Then on 3 May at 12:00:30 UTC, the above paths get replaced with 4-byte AS prepends, namely,

24863 327686 327686 327686 327686 327686 327686 327686 327686 327686 327686 37095

and the global impact is felt immediately, as shown in the following graph.

So what happened? Well, not everyone can afford a Cisco or Juniper router. Some folks make do with cheap Linux boxes running the free routing software called Quagga. And you guessed it! Older versions of Quagga have a buffer overflow problem, dropping the associated BGP sessions like a brick when they saw this announcement. The latest Quagga release, 0.99.11, came out on 2 October 2008. The prior version had the following gem of a comment.

  /* ASN takes 5 chars at least, plus seperator, see below.
   * If there is one differing segment type, we need an additional
   * 2 chars for segment delimiters, and the final '\0'.
   * Hopefully this is large enough to avoid hitting the realloc
   * code below for most common sequences.
   *
   * With 32bit ASNs, this range will increase, but only worth changing
   * once there are significant numbers of ASN >= 100000
   */

Or until someone prepends a 4-byte ASN a few times. Talk about lame. So this latest experiment in real-time Internet QA identified everyone running old versions of Quagga, thanks to the time bomb left behind by one very lazy programmer. The latest version of Quagga appears to handle this condition a little more carefully, but I can't help but wonder about any software version that starts with a zero.

Looking for the positive, it was nice to see some diversity on the Internet with respect to routing. Imagine if this had been a Cisco bug? Still we found it surprising to see the number and variety of organizations running Quagga in a production environment. The following graph gives the per-country distribution of prefixes for every country with at least 10 outages due to this bug.

The lessons are obvious. If you don't check user input and don't allocate sufficient memory, bad things will ultimately happen. Every possible permutation that you can think of, and perhaps some that you can't, will eventually be tried by someone, either accidentally or on purpose. So far, we're mainly seeing the former, which is good news. Ultimately though, bugs need to be found in lab, not in the production Internet.

The Blind Routing the Blind

2009-05-04T11:00:00Z

In our last blog entry, we talked about measuring the state of routing anarchy that exists on the Internet on a per-country basis. We looked at every routed network (prefix) by country of origin and tried to answer the question: do folks do what they say and say what they do, as articulated via routing registries? Although many manage to administer their routes with care, the overall results are quite varied. And without some way of verifying routes via some authoritative source, we are left only with the current system of believing everything we're told and hoping for the best. The dangers of such a system are demonstrated dramatically from time to time.

Although they certainly could, countries typically don't exercise any control over the routing hygiene of the companies operating within their borders. Countries might tax those companies, filter their traffic for objectionable content, mandate the types of software or equipment they can use and even spy on them, but if a company wants to screw up routing on the global Internet, well that's their business. As we've noted in the past, no driver's license is required on the Information Superhighway, as there are essentially no rules, regulations or enforcement. So in this blog entry, we'll apply our scoring idea to those who can easily effect change, namely, those organizations who are ultimately responsible for how traffic flows on the Internet.

]]> At first, this exercise might seem pretty straightforward: pick your favorite company or organization, compare their stated routing policies to reality, and give them a score based on the difference. But on the Internet, nothing is ever so simple, especially when it involves any sort of attribution. Simply identifying a company's Internet presence is complicated by mergers, buyouts, spin-offs and other aspects of modern capitalism. How can you hope to sort it all out? Well, there is only one thing you can really count on: anyone managing routing on the Internet must have one or more blocks of IP addresses (prefixes) and at least one "identifying" number, an Autonomous System Number (ASN). At some point in time, these things must have been doled out by some authority of sorts. That's about it. The person who requested these values (essentially a bunch of integers) might be long dead as might be the original company. And the associated administrative records might no longer be accurate or even maintained, but some IP addresses and an ASN are all you need to route traffic on the Internet. Once you've acquired these, you are free to proudly assert your presence to humanity and thereby consume a little bit of memory in every router on the planet.

In fairness, the overwhelming number of organizations we see on the Internet have only one or a handful of prefixes. These organizations are easy to identify, as they have little to begin with and hence little to have gone stale. But large companies and old timers to the Internet are rarely so fortunate. Let's take AT&T as one example. In the US, AT&T was the phone company for a very long time, until they were broken up by the courts and a newly formed AT&T became just a much smaller part of the original whole. This greatly diminished AT&T was ultimately acquired by SBC Communications, as was Ameritech, Bell South and others, and eventually the entire collection was renamed to simply "AT&T".

So what does AT&T look like on the Internet today? Old hands will think of it as ASN 7018, but we can actually associate almost 100 different ASNs with this single organization, about half of which are announcing prefixes. These are registered under a variety of names such as AT&T Network Systems, AT&T Internet Services, AT&T Global Network Services and almost 40 others. We even still see names containing SBC (and no mention of AT&T) and the now extinct Rogers AT&T Wireless (AS 20453). However, AS 20453 continues to announce Rogers' prefixes (over 70 of them) on the Internet, but AT&T is not one of their providers. So while Rogers ended their partnership with AT&T, no one updated the associated ASN registration. Talk about confusing.

The situation looks grim until you realize that there are actually three sources of (potentially imperfect) information that can be combined to help ferret out the truth:

registration information for ASNs and prefixes
current routing information
open source intelligence, e.g., news, press releases, Wikipedia, etc

I won't go into the details here, but we employ all three sources appropriately in an attempt to categorically define each organization on the Internet. While registration information is often incomplete and outdated, strings like AT&T and Verizon are unlikely to appear outside of those organizations, whereas others like Internet or Inc. have essentially no informational content. Other terms can be associated because of known business relationship or brand names, such as Microsoft and Hotmail or Sprint and Nextel. But since registration information can't be trusted, the associations that are uncovered need to be checked via routing data. This is how we discovered the Rogers AT&T Wireless problem. It just didn't make sense that an "AT&T entity" was not routing through AT&T and a quick Internet search confirmed that Rogers AT&T Wireless did not belong to the AT&T organization.

Intelligently analyzing these different inputs, we can finally list out all the organizations on the Internet: their ASNs (if any) and associated prefixes. And once we have that, we can give them a score based on their routing hygiene, exactly as we did in our blog entry for individual countries. Despite the fact that there are only around 31,000 unique ASNs seen on the Internet at present, we manage to find around 75,000 organizations, as not everyone with a registered prefix also has an ASN. (In these cases, their provider typically handles the routing of their prefix.)

We found over 13,000 organizations who managed a perfect score of 100, although all but 218 of them had fewer than 10 prefixes. While a small number of prefixes makes their administration easier, it is not a guarantee of success. One organization with 2 prefixes scored an embarrassing 9 out of a 100 on our scale. For those with at least 10 prefixes to manage, the following scatter plot gives the distribution. As we saw with countries, the more prefixes you have, the harder it is to keep everything straight and attain a good score. And while many manage to do a very good job, the overall results are quite varied, especially when you recall that doing essentially nothing will land you around 25 on our scale, and you have to do less than nothing to rate below that. Still almost 5,000 organizations manage to score below 25.

The overall winner in this exercise was Kazakh Telecom with a perfect score of 100 for over 150 prefixes. Airtel Broadband and Telephone Services (ABTS) came within a quarter of a point of a perfect score for their couple of hundred prefixes. At the other end of the spectrum, China Enterprise Communications Ltd. (CEC), in which Tata Communications recently acquired a 50% stake, barely managed a score of 20. CEC is around one point behind our second to last place finisher, the AARP, a US non-profit organization for people over 50. Overall, it's clear that there is a lot of room for improvement for most organizations, but if the Internet is ever going to have any measure of accountability, it is going to have to start with those who control the IP space and orchestrate global routing. Without attribution, there can be no accountability and no hope of improving upon the current situation.

Update: The earlier version of this blog inadvertently misclassified a number of registered networks due to a software bug in a script used for displaying the data. The scoring algorithm remains unchanged.