<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>Renesys Blog</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/" />
    <link rel="self" type="application/atom+xml" href="http://www.renesys.com/blog/atom.xml" />
    <id>tag:www.renesys.com,2008-10-18:/blog//1</id>
    <updated>2010-07-21T12:10:15Z</updated>
    
    <generator uri="http://www.sixapart.com/movabletype/">Movable Type 4.25</generator>

<entry>
    <title>What happened to Sprint?</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/07/what-happened-to-sprint.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.172</id>

    <published>2010-07-08T12:00:00Z</published>
    <updated>2010-07-21T12:10:15Z</updated>

    <summary> As our regular readers know, Renesys computes daily rankings of all the service providers in the world: globally, by geography, and by market segment. The rankings are a rather crude measure of size, as they are based entirely on...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Business" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="internettransit" label="internet transit" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="sprint" label="sprint" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="tinet" label="tinet" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
As our regular readers know, 
Renesys computes daily rankings of all the service providers in the world: 
globally, by geography, and by market segment.  
The rankings are a rather crude measure of <em>size</em>, 
as they are based entirely on the quantity of IP space ultimately transited by each provider.
However, it's the ranking <em>trends</em> that are more revealing than any absolute number. Who is adding customers?  Who is losing them or just standing still?
All of the rankings and the reasons for any changes are updated daily and available via our 
<a href="http://www.renesys.com/products_services/market_intel/">Market Intelligence</a>
offering.
For the past couple of Decembers
(<a href="http://www.renesys.com/blog/2009/12/a-bakers-dozen-in-2009.shtml">2009</a>,
<a href="http://www.renesys.com/blog/2008/12/winners-and-losers-for-2008.shtml">2008</a>),
we've also provided a glimpse into some of this data via year-end blogs devoted to the top global providers.
Halfway through 2010,
we decided to revisit this topic and highlight some recent changes:
the fall of Sprint and rise of Tinet being perhaps the most interesting.
</p>]]>
        <![CDATA[<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/top-13.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/top-13.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
The above graph plots global scores over time for the top 13 providers through the first half of 2010.
Since the exact numeric score is not that meaningful in this context, 
we will not provide the scale in our graphs. 
Rather, we will show how the scores changed over time and consider some of the reasons behind these changes.
To add clarity to the trends and changes,
we'll enlarge the scale of the graph and break this group of thirteen into three distinct clusters,
namely, the top three, the middle six, and the final four.
In these expanded graphs,
the changes will be more obvious and we'll make note of some of the more interesting ones.
</p>
 
<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/top-tier.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/top-tier.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
In 2006, when Renesys first started ranking providers, 
Sprint (AS 1239) had a commanding lead with a global score of almost 25% more than second-place Level 3 (AS 3356). 
In 2009,
Level 3 pulled away from a largely stagnant Sprint, 
opening up a roughly 12% lead.
Now in mid-2010,
Sprint is almost 40% below Level 3 and has just been passed by Global Crossing.
Sprint's precipitous decline was largely the result of losing Tinet (AS 3257) as a customer.
We now see Tinet as 
<a href="http://en.wikipedia.org/wiki/Tier_1_network">transit-free</a>
&mdash; a status sometimes confused with the much misused "Tier-1" designation.
As a result, Sprint lost credit in our ranking system for all of Tinet's downstream customers,
a substantial loss as Tinet is currently ranked 10<sup>th</sup> in the world.
</p>
<p>
More recently, Sprint also lost Russian provider EDN Sovintel (AS 3216),
who is now favoring Global Crossing (AS 3549), Verizon (AS 702) and Cable and Wireless (AS 1273) for transit.
Sovintel, a top-20 European provider in our rankings,
was
<a href="http://en.wikipedia.org/wiki/Golden_Telecom">acquired by Vimpelcom (Beeline)</a> in 2008 and owns the Golden Telecom trademark.
One possible explanation for these Sprint losses is the continued 
<a href="http://www.telegeography.com/cu/article.php?article_id=33320&email=html">decline in Internet transit pricing</a>.
Every provider has their own internal costs for providing access to their network. 
If Sprint's costs exceed or even approach current marketing pricing,
they might simply have made the rational business decision to stop competing in an activity with limited potential return.
Sprint's own web pages suggest a decreased emphasis on IP transit,
as their current offerings are buried 
<a href="http://shop.sprint.com/en/solutions/ip_networking/index.shtml">several levels</a>
below their mobile services-oriented main page.
</p>
</p>

<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/mid-tier.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/mid-tier.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
Our next group is decidedly more mixed.
Verizon's (AS 701) big increase resulted from temporarily picking up transit from Tinet, 
only to quickly lose it.
Otherwise, NTT (AS 2914), Savvis (AS 3561), Telia (AS 1299) and Verizon remained relatively flat during the first half of 2010.
In contrast,
Tata (AS 6453) and AT&T (AS 7018) continue to chalk up customer wins and trade places in our rankings.
Overall, the six providers in the group appear to be converging to roughly the same level and it might be another year or so before a clear winner or winners emerge.
</p>

<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/bottom-tier.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2010/06/bottom-tier.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
In our final group,
Tinet has been on an absolute tear in 2010 with wide-ranging customer wins including India's Bharti (AS 9498), Europe's LambdaNet (AS 13237), Venezuela's CANTV (AS 8048), Vietnam's Saigon Postel Corporation (AS 7602) and many others.
As a result, 
Tinet has secured the number 10 position in our global rankings,
after an early surge by China Telecom (AS 4134) briefly vaulted the Chinese into the top 10.
In addition, if Tinet can continue at this pace, 
they could easily find themselves in the middle of the second grouping by year end, 
as they currently sit at only 6% below AT&T.
</p>

<p>
<b>Conclusions</b>
<br>
Internet transit is an extremely 
<a href="http://www.renesys.com/blog/2009/11/ip-backbone-hard-sell-not-so-m.shtml">tough business</a>,
one with ever falling profit margins.
With lower Internet penetration, fewer competitors and higher margins,
the Middle East and Asia have provided somewhat of a refuge for providers who can operate effectively in these geographies.
As a result,
you can expect many traditional US-centric carriers,
such as AT&T, Sprint and Verizon,
to either grow very slowly or decline,
while those with strong global diversity, such as Level 3, Global Crossing, Tinet and Tata, 
should continue expand proportionally to the markets they serve.
And if older, less nimble players "leave the field",
such departures might just relieve some of the extreme pricing pressure found in the industry today, 
allowing the rest of us to continue to enjoy all that great Internet 
<a href="http://www.youtube.com/watch?v=oHg5SJYRHA0">"content"</a>, 
but at slightly higher (and more sustainable) pricing levels.
</p>]]>
    </content>
</entry>

<entry>
    <title>Two Strikes For the I-root</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/06/two-strikes-i-root.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.173</id>

    <published>2010-06-09T20:04:04Z</published>
    <updated>2010-07-08T18:46:47Z</updated>

    <summary> Here we go again. In March we wrote a blog entitled Accidentally Importing Censorship which described how incorrect DNS answers were returned in response to certain queries to the I-root. The problem was tracked down to a single instance...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Engineering" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="china" label="China" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="iroot" label="I-root" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="censorship" label="censorship" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
Here we go again.
In March we wrote a blog entitled
<a href="http://www.renesys.com/blog/2010/03/fouling-the-global-nest.shtml">Accidentally Importing Censorship</a>
which described how incorrect DNS answers were returned in response to certain queries to the I-root.
The problem was tracked down to a single instance of the I-root located in China.
Queries to this server for domains blocked in China, such as Facebook,
would return seemingly arbitrary answers.
As we noted, countries, and even companies, 
can impose their own standards on the Internet and block anything they want.  
This story was only noteworthy because those blocks (via bad DNS answers) became
visible <em>outside of China</em>.
Well, guess what?
We are once again seeing the Beijing I-root from outside of China.
</p>]]>
        <![CDATA[<p><b>Background</b></p>

<p>
Let's start with a few disclaimers and some background.
First and foremost, the sky is not falling.
Getting the wrong DNS answer, 
even when querying the Chinese I-root instance is an 
<em>extremely</em> rare event.
Go back and read our earlier
<a href="http://www.renesys.com/blog/2010/03/fouling-the-global-nest.shtml">blog</a>
to see the exact alignment of the stars that would be necessary.
The fact that it is so rare is what kept the problem from being detected for weeks.
However, as we noted in that earlier blog,
given the broad swath of the Internet potentially querying the Chinese I-root instance, 
someone was bound to stumble on a bad DNS answer and,
as a result, not be able to <em>friend</em> their pals.
This is exactly what happened and is what brought the problem to light.
</p>

<p>
Second, the fine folks at 
<a href="http://www.netnod.se/">Netnod</a>, 
who provide the exceptional and free I-root service,
vigorously defended their  
<a href="https://lists.dns-oarc.net/pipermail/dns-operations/2010-March/005343.html">services in China</a>, 
asserting they provide the same DNS answers regardless of location.
We have no reason to think otherwise.
</p>

<p>
Third, it's quite easy to see incorrect answers from DNS servers in China yourself,
whether or not you happen to live there.  
This has nothing to do with any of the root name servers.
Just pick your favorite DNS server based in China and ask it about Facebook.
Here is an example of repeated queries from the Linux command line from a US-based machine to a China Telecom DNS server.
</p>

<pre>
   dig @dns1.chinatelecom.com.cn. www.facebook.com.
   ...
   www.facebook.com.       11556   IN      A       37.61.54.158
   www.facebook.com.       24055   IN      A       78.16.49.15
   www.facebook.com.       38730   IN      A       203.98.7.65
</pre>

<p>
None of these IP addresses has anything to do with Facebook.
In fact, addresses starting with <b>37</b> haven't even been 
<a href="http://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml">allocated by IANA</a> as of this writing.
</p>

<p>
Of course, if you don't live in China,
you probably don't use a Chinese DNS server <em>directly</em>.
The problem is that we all use the root name servers and they are spread throughout the world.
Thanks to the vagaries of Internet routing,
you may end up querying any of them,
regardless of where you live and where they are hosted.
Thus, if you live outside of China and just happen to query a root name server hosted in China,
your queries will pass through what is known as the
<a href="http://en.wikipedia.org/wiki/Internet_censorship_in_the_People%27s_Republic_of_China">The Great Firewall</a>, 
and hence will be subject to any restrictions it imposes.
</p>

<p><b>Details, Details</b></p>

<p>
While doing some research for next week's 
<a href="http://www.nanog.org/meetings/nanog49/">NANOG</a> meeting in San Francisco,
we revisited the time line for the March I-root announcements from China and couldn't help but notice the problem resurfacing on June 3<sup>rd</sup>.
The I-root resolves to 192.36.148.17, 
which is announced by AS 29216 (which is dedicated to the I-root)
as both 192.36.148.0/23 and 192.36.148.0/24.
From there, these prefixes travel via Netnod's AS 8674 and then onto the general Internet.
Since Netnod 
<a href="http://en.wikipedia.org/wiki/Anycast">anycasts</a> these prefixes from dozens of locations around the world,
we expect to see them via any number of BGP adjacencies to AS 8674 and, in fact, we do.
Around 80 different ASes adjacent to Netnod's AS 8674 see the two I-root prefixes and, in turn, propagate them onward.
</p>

<p>
What we do not expect to see are mainland Chinese ASes adjacent to AS 8674 propagating these prefixes outside of China.
This is what we did see in March 2010 and it implies Internet users <em>outside</em> of China could be directed to the I-root instance <em>inside</em> of China.
Unfortunately, 
this problem has returned.
We see AS 8674 passing just 192.36.148.0/24 off to AS 24151 and then AS 7497, 
both of which are associated with the China Internet Network Information Center.
From there,
the prefix travels via Pacnet (AS 10026), formerly Asia Netcom, and PCCW (AS 3491) out to the general Internet.
This started just before 10:20 UTC on June 3<sup>rd</sup> and is still ongoing as of the date of this blog.
</p>

<p>
As we noted last time,
to get a bogus DNS response outside of China,
you not only have to query the I-root, 
you have to query the Chinese instance of it.
To measure <em>potential</em> impact,
we looked at the originating country of all prefixes downstream of any provider selecting the Chinese I-root.
We then computed the percentage of these relative to the total number of prefixes in the country.
A graph of the top dozen from the March incident is shown below,
followed by those from this current (and ongoing) incident.
</p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/03/China-Iroot-86.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/03/China-Iroot-86.shtml','popup','width=800,height=600,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/China-Iroot.png" width="500" height="375" alt="China-Iroot.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/06/China-Iroot-105.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/06/China-Iroot-105.shtml','popup','width=800,height=600,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/China-Iroot-new.png" width="500" height="375" alt="China-Iroot-new.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>

<p>
Not surprisingly, 
most of the affected countries are in Asia, as before,
but there are important differences from the last event.
Russia, India and Taiwan all entered the top twelve,
while Pakistan, New Zealand and Bangladesh have dropped out.
The impact on the countries in both lists is roughly similar, 
except that US impact went up by a factor of 10.
Potentially impacted US states include Florida and California, 
making up approximately half of the US total.
In addition, Singapore increased from 73% to 96%.
</p>

<p><b>Conclusions</b></p>
<p>
Censorship is a fact of life on the Internet today.
But unfortunately, given the open, trust-based nature of the network,
such censorship can easily spread beyond its intended boundaries.
While individuals can do little to avoid such issues,
there are actions network and system administrators can take.
Filtering root name server announcements with Chinese ASes on the path is one approach.
Never querying the I-root is another.
Such actions would guard against this particular problem,
but probably not the next one &mdash; whatever it might be.
Ultimately, 
<a href="http://www.ruexruex.com/4%20week%20bskt%20of%20puppies.jpg">we are all in this together</a>.
We depend on each country or organization not to inadvertently or intentionally interfere with any other.  
All other paths lead down a very slippery slope.
</p>]]>
    </content>
</entry>

<entry>
    <title>IPv6: Are we there yet?</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/04/ipv6-are-we-there-yet.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.170</id>

    <published>2010-04-28T10:00:00Z</published>
    <updated>2010-06-10T09:12:24Z</updated>

    <summary> For an advanced technology that we all depend upon, it sure seems that the Internet has more than its fair share of problems: spam, viruses, malware, spyware, phishing, worms, trojans, DDoS attacks, hijacks, DNS cache poisoning, botnets, keystroke loggers,...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Engineering" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
For an advanced technology that we all depend upon,
it sure seems that the Internet has more than its fair share of problems:
spam, viruses, malware, spyware, phishing, worms, trojans, DDoS attacks, hijacks, DNS cache poisoning, botnets, keystroke loggers, etc.
We need an entirely new vocabulary just to talk about this stuff.
Most of it appears to come out of the blue,
forcing the rest of the world to react.
But the good news is that there is at least one problem we can do something about 
<em>in advance</em>.
Unfortunately, not everyone has been taking the problem seriously enough and we are 
about to <a href="http://en.wikipedia.org/wiki/Hitting_the_wall">hit the wall</a>.
</p>

<p>
I'm talking about the impending exhaustion of IP addresses, 
IPv4 addresses to be exact. 
Every computer on the Internet needs access to at least one unique address in order to be connected.
Around the dawn of the 
<a href="http://en.wikipedia.org/wiki/Internet">Internet</a>,
32-bit IPv4 addresses, 
which allow for 4,294,967,296 different possibilities,
seemed like more than enough.
This was a simpler time when computers cost millions and no one imagined a phone you could put in your pocket.
As the Internet grew,
it soon became obvious that the seemingly inexhaustible supply of 4 billion addresses wasn't quite enough.
And so, a 128-bit IPv6-based Internet was proposed, 
this one with 340,282,366,920,938,463,463,374,607,431,768,211,456 different addresses.
(We're not going to make that mistake again!)
The only problem was that the new Internet wasn't interoperable with the old one we already knew and loved.
Without a 
<a href="http://en.wikipedia.org/wiki/Y2K">Y2K-type</a> hard deadline to focus on,
we kept barreling along toward the edge of the IPv4 cliff.
Now that the edge is clearly in sight,
this blog looks at how far we have come in adopting the not-so-new-anymore IPv6 Internet and, perhaps more importantly,
how much further we need to go.
</p>]]>
        <![CDATA[<p><b>Background</b></p>

<p>
The <a href="http://en.wikipedia.org/wiki/Internet_Engineering_Task_Force">
Internet Engineering Task Force</a> 
(IETF) started looking into creating an 
<a href="http://en.wikipedia.org/wiki/Ipv6">IP Next Generation</a> (IPng) protocol in 1992.  After a six-year gestation period, 
IPv6 was born with the publication of 
<a href="http://tools.ietf.org/html/rfc2460">RFC 2460</a> in December 1998. 
Twelve years on, the various 
<a href="http://en.wikipedia.org/wiki/Regional_Internet_registry">Regional Internet Registries</a> (RIRs) are projected to exhaust their IPv4 allocations from 
<a href="http://en.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority">IANA</a> 
within two years.  
After that, there will still be plenty of unused IPv4 addresses, 
but they will all reside with <em>existing</em> companies, governments, schools and organizations.
New entities are going to face a serious problem acquiring IPv4 addresses.
And what about the over 4 billion mobile phones in use globally today, 
all of them potentially Internet-capable devices?
The need for new addresses is only going to increase dramatically.
</p>

<p><b>IPv6 Routing and DNS</b></p>

<p>
Among other things, 
Renesys collects both IPv4 and IPv6 routing data from 
<a href="http://www.renesys.com/tech/peering.shtml">various exchange points</a> worldwide.  
From this data source, 
we have seen a marked increase in allocated, registered and routed IPv6 prefixes 
(blocks of consecutive IP addresses) over the past year.  
In addition, the number of Autonomous Systems (ASes) observed originating IPv6 prefixes has been on the rise.  
The graphs on this page show the growth rates in these metrics over the past 18 months.  
A view into the IPv6 marketplace can also be found in Renesys' latest version of 
<a href="http://www.renesys.com/products_services/market_intel/">Market Intelligence</a>.
</p>

<p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/04/IPv6_counts_4_cmyk-89.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/04/IPv6_counts_4_cmyk-89.shtml','popup','width=1468,height=1385,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/04/IPv6_counts_4_cmyk-thumb-500x471-89.jpg" width="550" height="519" alt="IPv6_counts_4_cmyk.jpg" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>
</p>

<p>
IPv6 is supported in all major modern operating systems, including Windows, Mac OS and Linux, and is enabled by default or easily activated.   
In DNS,
IPv6 addresses are looked up via 
<a href="http://en.wikipedia.org/wiki/IPv6_address#IPv6_addresses_in_the_Domain_Name_System">AAAA</a>-type queries, 
also known as quad-A.
The ratio of quad-A queries to total DNS query volume, now 15 - 20% on busy servers, provides solid evidence of the increasing prevalence of IPv6-capable clients.
</p>

<p>
While Google has supported IPv6 access to its search engine since 
<a href="http://en.wikipedia.org/wiki/IPv6#Major_announcements_and_availability">March 2008</a>,
this year it turned on 
<a href="http://www.pcworld.com/article/188276/youtube_turns_on_ipv6_support_net_traffic_spikes.html">IPv6 support for YouTube</a>,
resulting in an immediate and sustained jump in global IPv6 traffic.  
And with over 15 million Internet subscribers, 
Comcast has 
<a href="http://blog.comcast.com/2010/01/preparing-for-the-ipv6-transition.html">announced extensive IPv6 trials</a> starting in the second quarter of 2010.  
The biggest players on the Internet cannot afford to have their businesses constrained by lack of IP space and have been at the forefront of IPv6 adoption.  
With major content providers and ISPs enabling IPv6 transit and popular operating systems turning on support by default, 
is an IPv6 Internet now a reality?
</p>

<p><b>Drum roll, please &#8230;</b></p>

<p>
Not really.  We still have a very long way to go.   
Of the over 34,000 ASes in use on the Internet, 
only around 6% (just over 2,000) handle IPv6 prefixes 
(i.e., the ASN is seen in the path of an IPv6 announcement).  
Of the top 100 global providers according to Renesys' 
<a href="http://www.renesys.com/products_services/market_intel/">Market Intelligence</a>
rankings, 
only 28% originate more than one IPv6 prefix and 36% originate none.  
These are very low bars for measuring success.  
It does <em>not</em> imply that the providers have IPv6 content, 
only that they lay claim to some IPv6 space or can pass along an IPv6 routing announcement.
</p>

<p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/04/ASN_IPv6_6_cmyk-thumb-550x353-92-93.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/04/ASN_IPv6_6_cmyk-thumb-550x353-92-93.shtml','popup','width=550,height=353,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/04/ASN_IPv6_6_cmyk-thumb-550x353-92-thumb-550x353-93.jpg" width="550" height="353" alt="Thumbnail image for ASN_IPv6_6_cmyk.jpg" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>
</p>

<p><b>Conclusions</b></p>

<p>
In summary,
if you possessed an IPv6-only device and had no way to 
<a href="http://en.wikipedia.org/wiki/NAT">NAT</a> your way to the "real" Internet, 
you'd probably be rather lonely.  
Even the rather small IPv6 world is not completely connected.  
At the end of March, 
we randomly selected two global providers who supply us with their IPv6 routing tables, 
call them A and B.  
Considering all possible covering routes, 
provider A had 30 IPv6 prefixes not reachable from B, 
while B had 19 prefixes not reachable from A
(2,643 prefixes could be reached from both). 
While much of the machinery necessary for IPv6 is currently in place, 
the content, the availability and even the basic connectivity of the network continues to be lacking.
But this much is certain,
we <em>will</em> run out of IPv4 address space and IPv6 is the only other game in town.
The fine folks at 
<a href="http://www.he.net/">Hurricane Electric</a>
even provide a 
<a href="http://ipv6.he.net/statistics/">nice app</a> to let you count down the days until IPv4 exhaustion.
So even though the 
<a href="http://www.fix6.net/archives/2010/03/10/ipv6-query-app-for-the-iphone/">iPhone
still doesn't support IPv6 natively</a>,
if you want to reach the billions of mobile Internet users yet to come online, 
you are going to have to support both IPv4 and IPv6 before long.
Time is running out for those still on the sidelines,
as the growth of the Internet shows no signs of slowing down.
</p>]]>
    </content>
</entry>

<entry>
    <title>How To Build A Cybernuke</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/04/how-to-build-a-cybernuke.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.171</id>

    <published>2010-04-22T10:30:00Z</published>
    <updated>2010-06-10T09:12:51Z</updated>

    <summary>The Internet infrastructure has been having a bad month. Not as bad as, say, the world&apos;s aviation infrastructure, but bad enough. First, Chinese Internet censorship leaked out to a few massively unlucky users of the I root server. Then China...</summary>
    <author>
        <name>James Cowie</name>
        
    </author>
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="cybernuke" label="cybernuke" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="media" label="media" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="resilience" label="resilience" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="sensationalism" label="sensationalism" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="threatmodels" label="threat models" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>The Internet infrastructure has been having a bad month.  Not as bad as, say, the world's <a href="http://www.bloomberg.com/apps/news?pid=newsarchive&sid=ar4Bv0kc8XgE">aviation infrastructure</a>, but bad enough.    </p>
<p>First, Chinese Internet censorship leaked out to a few massively unlucky users of the <a href="http://www.renesys.com/blog/2010/03/fouling-the-global-nest.shtml">I root server</a>.    Then China Telecom failed to filter someone who <a href="http://www.theregister.co.uk/2010/04/09/china_bgp_interweb_snafu">leaked thousands of hijacked routes</a> to other people's networks through them, probably by accident.</p>
<p> And then, inexplicably, Forbes went where no one had gone before (with a wink to <a href="http://www.wired.com/threatlevel/2010/03/cyber-hype/">Wired</a>), and asked whether China might actually be testing a "<a href="http://blogs.forbes.com/firewall/2010/04/09/is-china-testing-cybernukes/"><strong>cybernuke</strong></a>".</p>

<p>At first, this irritated me.   Journalists and bloggers and blogger-journalists are fanning the flames of US unease about the growing role of China in world affairs.   But then I realized that I could probably make tens of thousands of people read my blog, too, by jumping on the bandwagon.   By all means, then, grab an <a href="http://en.wikipedia.org/wiki/Meal,_Ready-to-Eat">MRE</a> and hunker down in your Internet bomb shelter while I try to answer some of the obvious questions that came our way in the wake of the Forbes article:</p>

<ul>
<li>How would anyone build a cybernuke?    What is that? 
<li>Could a single actor, state-sponsored or otherwise, actually take down the global or regional Internet infrastructure of 2010, disrupt financial markets, throw civilization into chaos?  
<li>How do I get my cybernuke movie screenplay optioned by <a href="http://www.hollywood.com/celebrity/195797/Jerry_Bruckheimer">Jerry Bruckheimer</a>?   His people won't return my calls.
</ul></p>]]>
        <![CDATA[<br>
<p><b><big>What is a cybernuke?</big></b></p>

<p>Let's start by distinguishing carefully between an attack designed to take a single organization off the Internet, and a cybernuke. The former takes place every day, and nobody is entirely immune.   If you get the wrong kind of people angry, you can be thrown off the Internet in very short order, using unsubtle distributed denial of service attacks that you can readily rent for cash at the prevailing <a href="http://blog.damballa.com/?p=330">market rates</a>.  No controversy there.  </p>

<p>We're talking instead about designing a <em>cybernuke</em>:  an infrastructure attack that would allow you to shut down (or centrally subvert, or control) a large part of the Internet, not a single organization.   That's a different class of beast.  </p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/04/705px-Nuclear_fireball-97.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/04/705px-Nuclear_fireball-97.shtml','popup','width=705,height=599,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/04/705px-Nuclear_fireball-thumb-300x254-97.jpg" width="300" height="254" alt="705px-Nuclear_fireball.jpg" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>
<p>There are three broad schools of thought here.</p>

<p><b><big>Option 1: Hijack Everything</big></b></p>

<p>This is the cybernuke that Forbes saw lurking in the dark shadows.    Using the Border Gateway Protocol, inject just the right kinds of false traffic into the global routing network, so that all the packets go to the wrong places.   Game over, man! </p>

<p>There are a few problems with this scenario.   To make it work, you have to inject your false routes in such a way that a substantial part of the planet will hear them and believe them.  Because of the way the BGP routing protocol works, that means that your ersatz paths to other people's networks have to look <em>more attractive than the real thing </em>(meaning: short and direct, or more specific).</p>
<p> Many of the routes you'll be attacking are already about as specific as they can get and still be globally propagated, so you have to compete on directness; for the rest, you're going to have to advertise more than one more-specific network for every network you're trying to attack (300,000 make up the whole Internet, more or less).   That's a lot of routes.  If you're injecting enough different paths to take down large swaths of the Internet, you'll therefore need to enlist a partner who already advertises tens of thousands of routes, so that the massive increase in routes they propagate on your behalf won't raise an eyebrow.</p> 

<p>Together, those requirements mean that you almost certainly need to convince one of the dozen-or-so largest worldwide Internet carriers to act as your agent.   Anyone smaller is too far from the Internet's core, and since the average packet on the Internet only changes hands three or four times en route, even one extra handoff is going to make your fake routes look sleazy and unattractive.</p><p>Moreover, most of these carriers are very clueful about the possibility of being used as an unwitting agent of evil.  They have procedures and filters and circuit-breakers in place to prevent exactly such an embarrassment.   That doesn't mean it can't happen, although it happens more and more rarely as the years go by. </p>
<p>(It did just happen to <a href="http://www.theregister.co.uk/2010/04/09/china_bgp_interweb_snafu">China Telecom</a>; however, because of the bad press it received, I would wager that this is the <b>last time</b> it will happen to China Telecom.)</p>
<p>Even if you manage to get your fifty thousand fake network routes announced by a major carrier, <strong>and </strong>the rest of the world believes them, <strong>and</strong> your routes are selected as "best" by some significant percentage of the Internet, will the world's traffic actually be impacted?   Not yet. </p>
<p>It's almost certain that the immediate neighborhood of each victim network (all of their Internet service provider's customers, and all of <em>their</em> providers' customers, and so forth) will blithely ignore the cybernuke, and continue sending traffic to the correct destination, as usual. The parts of the Internet that are close to the attacker's point of injection may change their mind, so the victims may well lose visibility to the attacker's networks.  Do you care?   It becomes more of a local censorship issue, an attack on the attacker's network, if you will, rather than a major irritation to the supposed victims.</p>
<p>The final indignity, of course, is that an attacker who deploys such a cybernuke will probably blow themselves off the Internet by accident.   If you successfully manage to subvert BGP to publish a large number of attractive routes to places that matter, you will shortly be on the receiving end of many, many gigabits per second of traffic that are trying in vain to find their rightful home.   This flood of misrouted traffic will crush your network, and your launching zone will disappear beneath the waves, like Atlantis.  Look on my works, ye Mighty, and despair! </p>

<p><b><big>Option 2: Cut the Cord</big></b></p>

<p>I hope I've convinced you that option 1, while a perfectly plausible way of wreaking <a href="http://www.renesys.com/blog/2008/02/pakistan-hijacks-youtube-1.shtml">mysterious small-scale damage</a>, isn't going to move you down the road toward cyber-world domination.</p>

<p>Option 2 is physical damage to the infrastructure:  classically, cutting the cables that hold the Internet together.    There have actually been <a href="http://www.reuters.com/article/idUSTRE63J3LV20100420">very decent studies</a> of this recently, highlighting both the vulnerability of the infrastructure, and the frightening dependence of the world economy on good communications.   </p>

<p>The hard part about this cybernuke option is that it's precisely the scenario that the Internet has evolved to avoid.   With every month that passes, the Internet becomes better and better connected to itself.   Remember, the Internet consists of tens of thousands of independent infrastructure service providers and content delivery companies, all working to keep the traffic flowing to billions of paying customers. </p>

<p>When submarine cable cuts happen, as they have over and over in history, these providers take notice.  New cables get laid and lit.  New contracts get signed that create alternative paths for traffic to take.   Companies that have a global footprint no longer trust the Internet to "just work" -- they take provider diversity seriously as a core element of a due diligence strategy.   They seek out providers who are well-connected and can speak the language of risk management.</p>

<p>There's no doubt that physical damage at one of the Internet's pinch points, whether that be in the Red Sea or in the Straits of Malacca or at one of a number of windowless buildings throughout the world, would cause some serious disruption.    But our data and experience suggest that each cable cut causes less serious impact than comparable ones that preceded it.    Human networks are more resistant to point-source damage than you'd think, and the Internet is a human network. </p>

<p><b><big>Option 3: Inject and Amplify</big></b></p>

<p>That leaves option 3, which is more of a "Cyber-Bioweapon" than a Cybernuke.    The challenge would be to design an injectible routing message that would cause a large fraction of the world's routing infrastructure to fail, while going unnoticed by the rest.  The idea is to have the immune population propagate your attack to all the vulnerable machines, which fail and take down the Internet with them (at least until they can be individually patched, and brought back into service).</p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/04/500px-Biohazard_symbol_(red).svg-100.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/04/500px-Biohazard_symbol_(red).svg-100.shtml','popup','width=500,height=473,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/04/500px-Biohazard_symbol_(red).svg-thumb-300x283-100.png" width="300" height="283" alt="500px-Biohazard_symbol_(red).svg.png" class="mt-image-left" style="float: left; margin: 0 20px 20px 0;" /></a></span>
<p>Who would think of such a thing?   The scary part is that <em>no one had to think of it</em> -- it has happened naturally more than once as a byproduct of the complexity of the Internet ecosystem.  A design defect in one kind of router will create malformed routing traffic, which gets passed obliviously to the four corners of the earth, where vulnerable routers encounter the bad messages, and die noisy deaths.   We've documented mild outbreaks of this sort before -- three times just last year, in fact:  in <a href="http://www.renesys.com/blog/2009/02/the-flap-heard-around-the-worl.shtml">February</a> and  <a href="http://www.renesys.com/blog/2009/05/byte-me.shtml">May</a> and <a href="http://www.renesys.com/blog/2009/08/staring-into-the-gorge.shtml">August 2009</a>.</p>

<p>Unfortunately, unlike the previous two options, this one actually scares me.    Known threats, like bogus routes that exploit trust relationships within the definition of the routing protocols, we can defend against.  Point source damage to buildings and cables, we can defend against.   But the Internet's routing hardware and software diversity is actually pretty poor, compared to its topological richness. </p>

<p>Think of it this way: two hardware vendors (Cisco and Juniper) probably represent something like 60%+ of all the infrastructure routers in the world.    Craft a vulnerability that passes one and crashes the other, and you could do some serious damage.   The size and decentralization of the Internet work against us here:  you can't just go out and patch hundreds of thousands of routers on demand, even in the face of a material threat.    Vulnerable machines will litter the ecosystem for months or years to come.   The Internet can catch the same flu over and over.  </p>
<p>The good news here, if there is any, is that stiff price competition, particularly in <a href="http://www.trefis.com/articles/14958/huawei-can-put-pressure-on-ciscos-router-market-share-and-margins/2010-04-12">emerging markets</a>, is driving a healthy trend toward hardware diversification: as recently as 2005, vendors C and J probably controlled more than 90% of the market, instead of 60%. Welcome to the neighborhood, Huawei! </p>

<p><b><big>Conclusion: Busted .. or Plausible?</big></b></p>

<p>As much as I'd like to say that the myth of the Cybernuke is busted .. option 3 gives me pause and makes me reluctantly conclude that it's just barely  <strong>Plausible</strong> (although not along the lines everyone expects, and not necessarily in a form that could be targeted at anyone smaller than The Whole Planet). </p>

<p>Just to get the last of the sensationalist metaphorical Bruckheimer-bait out of the way, what stands between us and the impact of an Internet Dinosaur-Killer?    The same three advantages that have stood the Internet in good stead throughout its incredible 40-year evolution:
<ul>
	<li><b>Awareness</b> of the dangers of centralized controls and hardware/software monocultures that can be exploited by bad guys</li>
        <li><b>Acceptance</b> of the frustrations that come with decentralization and diversity, and a willingness to do whatever it takes to buy an acceptable level of redundancy for key services</li>
        <li><b>Communities</b> like <a href="http://www.nanog.org">NANOG </a>and <a href="http://www.ripe.net/meetings/">RIPE</a> and <a href="http://www.apricot.net/about.html">APRICOT </a>and <a href="http://www.menog.net/meetings">MENOG</a>, where network operators reach rough consensus about the kinds of security and operational best practices that keep everything from blowing up.   Have you thanked a network engineer today?
</ul></p>


<p>And Jerry, if you're reading this:  Call my people.  Seriously.  This cybernuke screenplay will be MONSTER BOX OFFICE.</p>]]>
    </content>
</entry>

<entry>
    <title>Accidentally Importing Censorship</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/03/fouling-the-global-nest.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.169</id>

    <published>2010-03-30T21:00:00Z</published>
    <updated>2010-06-09T16:30:24Z</updated>

    <summary> With advancements in hardware and software, sophisticated filtering technologies are increasingly being applied to restrict access to the Internet. This happens at the level of both governments and corporations. Renesys is headquartered in the &quot;Live Free or Die&quot; US...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Engineering" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="china" label="China" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="iroot" label="I-root" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="censorship" label="censorship" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
With advancements in hardware and software,
sophisticated filtering technologies are increasingly being applied to restrict access to the Internet.
This happens at the level of both governments and corporations.
Renesys is headquartered in the 
<a href="http://en.wikipedia.org/wiki/Live_Free_or_Die">"Live Free or Die"</a> US state of 
New Hampshire.
In our 
<a href="http://en.wikipedia.org/wiki/Hanover,_New_Hampshire"</a>small town of roughly 10,000 folks,
we know of a local company who tries to restrict <em>non-work related</em> (e.g., shopping) websites from their employees.  
Unfortunately, someone who works there can't read about Amazon's cloud computing as a result &mdash; a small bit of collateral damage.
Entire countries act in much the same way.
The <a href="http://map.opennet.net/">OpenNet Initiative</a> keeps track of such state-sponsored restrictions and publishes interesting maps based on the level of filtering by topic.
But given the open nature of the trust-based Internet,
one country's restrictions, if not handled very carefully, 
can easily foul the global Internet nest we all live in.
This blog is about one such story of Internet restrictions in China becoming visible (seemingly at random) from other parts of the world and <b>going undetected for 3 weeks</b>.
Given the increasing complexity of this technology, 
the difficulty in controlling a very open Internet, 
and the strong desire of some to do just that,
this could be a harbinger of things to come.
</p>]]>
        <![CDATA[<p><b>Background</b></p>

<p>
To understand this problem,
one needs to consider Internet routing, the behavior of DNS and the root nameservers, and the economics of Internet transit.  
So let's briefly review a few basics before we describe the incident.
When you type
<blockquote>
www.facebook.com
</blockquote>
into your browser, your computer must first contact a DNS server to convert this name into an IP address in order to contact the host serving this content.
Answers to DNS requests are cached on both your machine and the various servers involved to reduce the load of subsequent identical queries.
Now suppose for the moment that the caches on your machine and your DNS server are both empty and you make the above query.
Your DNS server first contacts a 
<a href="http://en.wikipedia.org/wiki/Root_nameserver">root nameserver</a> 
with your request.
If configured according to convention,
the root nameserver will not provide the answer to your query, 
but instead direct your DNS server to the .com nameservers.
In turn, those will direct your DNS server to the Facebook nameserver, 
which will ultimately provide an IP address to one of Facebook's web servers.
At least that is how it is supposed to work.
</p>

<p>
</p>

<p>
Now suppose organization C runs a root nameserver and C doesn't want you going 
to Facebook.  
Nothing requires C to direct you to the .com nameservers.
Since C sees your complete request, it could just answer you directly.
If C gave you the wrong answer, 
it would effectively block your access to Facebook,
assuming you didn't know enough to pick a different server.
Since the Internet runs on trust,
you'll also end up caching C's invalid response 
(known as <a href="http://en.wikipedia.org/wiki/DNS_cache_poisoning">cache poisoning</a>)
with C being the one who tells you how long to cache the result!
(Alternatively, C could actually provide the correct answer, 
but a firewall in front of C could alter the result on its way back to you.  Either way, you lose.)
</p>

<p><b>Incident Details</b></p>

<p>
While this scenario might seem very unlikely,
it in fact just happened with the 
<a href="http://en.wikipedia.org/wiki/Autonomica">I-root</a> instance run out of China,
as first reported by 
<a href="https://lists.dns-oarc.net/pipermail/dns-operations/2010-March/005260.html">Mauricio Vergara Ereche</a> in Chile
and subsequently blogged on 
<a href="http://www.neowin.net/news/china039s-censorship-firewall-invades-foreign-systems">here</a>.
In fact, 
the problem existed from March 3<sup>rd</sup> until March 25<sup>th</sup>, 
before it was reported on and corrected.
Despite the fact that a lot of people <em>could have</em> been impacted,
the chances of any one of them having gotten the incorrect DNS response are extremely remote,
as we'll explain later.
This is thanks again to the way DNS operates and the overall resiliency of the Internet.
</p>

<p>
So let's review exactly what happened here.
First, as is well known,
China censors the Internet in a variety of ways.
One way is to return invalid answers to DNS requests to Chinese users.
For example,
at this moment,
a Chinese DNS server is returning 46.82.174.68 as an IP address for www.facebook.com,
when in fact, all legitimate Facebook IPs are of the form 66.220.x.y or 69.63.x.y.
Seemingly random, but often unrouted, IPs are also returned for www.twitter.com, www.youtube.com and many other domains.
This is normal and expected behavior <em>inside</em> of China.
</p>

<p>
However, China also happens to house an instance of a root nameserver, the I-root, and when this server became visible outside of China on March 3<sup>rd</sup>,
anyone who happened to query it could have gotten bogus responses.
Keep in mind that there are 13 different root nameserver IP addresses 
(associated with the A-root, B-root, &#8230;, M-root) 
and the I-root is just one of these, namely, 192.36.148.17.
In addition, there are dozens of instances of the I-root housed in many locations around the world.  
To get a bogus DNS response outside of China,
you not only had to query the I-root, you had to query the Chinese version of it.
The countries with the most number of network prefixes that <em>could have queried</em> the Chinese I-root during this time are given in the following graphic.
Not surprisingly,
the most exposed countries were all in Asia, 
but some some prefixes in the US were also vulnerable,
more than half of which geo-locate to California.
</p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/03/China-Iroot-86.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/03/China-Iroot-86.shtml','popup','width=800,height=600,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/China-Iroot.png" width="500" height="375" alt="China-Iroot.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>

<p>
So let's review the unlikely series of events that would have been required to observe a bogus answer to www.facebook.com.
<ol>
  <li>You attempt to go to www.facebook.com.</li>
  <li>You don't have this entry in your DNS cache, nor does your DNS server.</li>
  <li>Your DNS server does not have the .com servers cached either.</li>
  <li>Your DNS server happens to choose the I-root (as opposed to A-root, B-root, C-root, etc).</li>
  <li>Due to current Internet routing in place at your location, your DNS server happens to be directed to China's instance.</li>
</ol>
Since Facebook is blocked in China, your DNS server does not get the expected list of .com servers, but rather a bogus response to your original request, 
either from the I-root itself or a firewall in between.  
You cache the result and can't "friend" anyone for a while.
</ul>
</p>

<p></p>
<p>
Think for a moment about how unlikely all of this is.
Responses for .com nameservers are set to expire only after 48 hours.
So to have <em>any chance</em> of seeing a bogus response,
the <b>first request</b> into your DNS server after its cached .com entries expire 
must be for a blocked domain in China <em>and</em> 
your DNS server has to query the I-root, 
one of 13 possibilities.
(Once the .com servers are cached, there is no need to query to the root servers for them.)
In addition, of the many instances of the I-root, 
you have to somehow end up selecting the Chinese one.
But with 
<a href="http://www.internetworldstats.com/stats.htm">1.7 billion Internet users</a> surfing the web for 3 weeks, 
it was bound to happen and be reported.
</p>

<p>
Of course, 
you don't really have any control over which I-root instance you see from your location.  
That is determined by Internet routing.
Many of the root nameservers are 
<a href="http://en.wikipedia.org/wiki/Anycast">anycast</a> from multiple locations around the world. 
This means that the associated IP prefixes are announced from multiple locations,
all of which house servers with copies of the appropriate data.
<a href=http://en.wikipedia.org/wiki/BGP">BGP</a>,
the Internet routing protocol,
is then used to sort out who sees which instance of the root servers from which locations.
In general,
the Chinese I-root instance is only visible from within China, 
but for 3 weeks these routes leaked out to the global Internet.
This announcement exited China when it was leaked by the China Internet Network Information Center (AS 24151).
</p>

<p>
The careful reader might wonder how anyone in the US could have selected the Chinese I-root.  
Isn't there a closer one?
Well, that's the thing about Internet routing.
It's driven more by economics than by physical distance, although the two are often related.
For example,
suppose two smaller providers, call them A and B,
agree to exchange traffic with each other for free.
This common arrangement on the Internet is known as <em>peering</em> 
and allows A and B to save money,
specifically, transit costs to larger providers.
Suppose further that A (or one of its customers) is running the I-root.
If B needs to get to the I-root, 
it should pick its settlement-free peering link with A,
rather than its link to a larger carrier for whom they have to pay.
China Telecom, by far the largest carrier in China,
peers with nearly 100 other providers.
If those providers or their customers aren't running an instance of the I-root themselves,
they might use their peering link to China Telecom to reach their instance.
This is how countries far from China could end up selecting the Chinese I-root as the "best" of many possibilities.
</p>

<p><b>Conclusions</b></p>
<p>
This story illustrates both the fragility and the resiliency of the global Internet.
It's fragile because it is ultimately trust-based and almost anyone can violate that trust, 
deliberately or by accident
(and there is no reason to think this wasn't an innocent mistake).
It's resilient because there are often many alternatives or workarounds for any screw-ups or attempts at control. 
The Internet is a big place full of very clever people &mdash; making it tough to wall off.
</p>]]>
    </content>
</entry>

<entry>
    <title>The Geopolitics of Iranian Connectivity</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/02/irans-internet-the-geopolitics.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.168</id>

    <published>2010-02-11T12:59:18Z</published>
    <updated>2010-03-13T22:53:10Z</updated>

    <summary> As Iran celebrates the anniversary of the 1979 Islamic Revolution, it seems like an opportune time to look in on the evolving state of their Internet connectivity. When we last looked, after the disputed elections in June 2009, the...</summary>
    <author>
        <name>James Cowie</name>
        
    </author>
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Politics" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="azerbaijan" label="azerbaijan" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="internettransit" label="internet transit" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="iran" label="iran" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="russia" label="russia" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[
As Iran celebrates the anniversary of the 1979 Islamic Revolution, it seems like an opportune time to look in on the evolving state of their Internet connectivity.   When we last looked, after the <a href="http://www.renesys.com/blog/2009/06/strange-changes-in-iranian-int.shtml">disputed elections in June 2009</a>, the picture was one of uneasy stability: logically diverse but physically constrained transit via the United Arab Emirates, backup transit via Turkey.   Today, a third way out of the bottle is visible in the routing table: <em>substantial amounts of Internet transit have materialized through a Russian provider.</em>  And there, in those obscure entries in the global Internet routing table, may lie echoes of Iran's larger geopolitical strategy.  <p>]]>
        <![CDATA[

<big><strong>Examining the Global Routing Table</strong></big><p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.crossed-flag-pins.com/Friendship-Pins/Russia/Flag-Pins-Russia-Iran.html"><img alt="flags.jpg" src="http://www.renesys.com/blog/assets_c/2010/02/flags-thumb-300x240-81.jpg" width="300" height="240" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>
  
Let me back up for a moment and review what's visible today, and how we interpret it.  Renesys collects independent routing table perspectives from Internet service providers all over the world, and fuses them to get a daily picture of the evolving economic relationships (provider-customer, peer-to-peer) that allow Internet traffic to flow.   Our <a href="http://renesys.com/products_services/market_intel/">Market Intelligence</a> product uses this evolving roadmap to give Internet transit wholesalers critical visibility into the commercial relationships in the markets where they make their money.<p>

As you might guess, when we look at this aggregate global routing table, we learn a lot about the size and structure of the Internet transit marketplace in a particular region.  In this case, <a href="http://www.renesys.com/blog/2009/06/the-proxy-fight-for-iranian-de.shtml">as we observed back in June</a>, it's clear that Iran has a very rich internal Internet, with dozens of service providers and content providers.  It's one of the oldest and most advanced <em>domestic</em> Internet ecosystems in the region, with a very sizable and sophisticated domestic audience.  <p>

However, nearly all of Iran's<em> international </em>traffic (packets bound for Twitter, Google Mail, Yahoo, etc.) must squeeze painfully through a state-imposed bottleneck.  One organization,  DCI (Data Communications Iran), the Internet arm of the state telecommunications company, serves as a common transit gateway.   This gives the Iranian government a centralized locus of control from which to monitor and filter international Internet traffic. <p>

<big><strong>Getting Out of Iran</strong></big><p>

So, the obvious question: where does the Iranian government purchase its international Internet transit?   Think for a moment about the constraints that they have to satisfy.   They need enough capacity to sustain a 21st century information economy.   They want to maintain centralized control over all that information.  They have the challenge of maintaining adequate logical and physical diversity, so that a single point of failure can't take down the whole country's Internet access (unless they choose to do so themselves!).   And they have the additional headache of choosing providers that are <em>geopolitically diverse</em>, to route around sanctions and military threats.<p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/02/blog_feb_2-83.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/02/blog_feb_2-83.shtml','popup','width=767,height=536,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/02/blog_feb_2-thumb-600x419-83.jpg" width="600" height="419" alt="blog_feb_2.jpg" class="mt-image-none" style="" /></a></span><p>

Take a look at the map of Iran and its neighbors.   The good choices are few and far between.  In the south, they can access the UAE over a submarine fiberoptic link from Jask to Fujairah, built by Alcatel-Lucent.   This is the obvious way out, as it gives access to the primary submarine fiber paths headed east (to Asia) and west (to Europe).   You get your choice of service providers once you reach the Emirates, and Iran has historically made good use of this diversity.  But in the end, it's logical diversity, not physical diversity.   A thin piece of glass strung across one of the world's busiest and tensest energy shipping lanes is not what you want to hang the nation's Internet future on.<p>

So they established a connection to Turkey along a terrestrial fiber route to the northwest, from Tabriz to the frontier and on to Ankara and Istanbul along the gas pipeline.   Turkey is an <a href="http://www.turktelekom.com.tr">important regional conduit</a> for Internet transit to Europe, and the Turkish government has quietly <a href="http://news.xinhuanet.com/english/2009-10/27/content_12342380.htm">signalled</a> their support for Iran's right to nuclear self-determination in recent months, as part of a <a href="http://www.presstv.ir/detail.aspx?id=116124&sectionid=351020204">warming relationship.</a>  Still, one can imagine that the Iranian government might want to seek a third way out, to further improve their diversity in times of political crisis.<p>

<big><strong>Where else can Iran turn for transit?</strong></big><p>

The southeastern frontier goes to Pakistan through divided Baluchistan; this would be a fine path to reach Asian transit, but would be <a href="http://pakistan.foreignpolicyblogs.com/tag/pakistan-iran-relations/">politically complicated.</a>  The <a href="http://www.taeint.net/en/network/middle">TAE cable system</a> also goes to Turkmenistan along the northeastern frontier, but it's unlikely to offer the kind of capacity or stability that you'd want for a primary backup route.  Afghanistan and Iraq are out, for obvious reasons.   Armenia will be an interesting choice someday, when the <a href="http://en.rian.ru/world/20090415/121147361.html">railroad goes in</a>, but lacks the capacity today.<p>

That leaves the Caspian coastal route to Azerbaijan, following existing pipelines to Baku.   We've seen DCI use the Azeri route before, showing brief evidence of limited transit through Azerbaijan's <a href="http://www.delta-telecom.net/">Delta Telecom</a> in 2008-2009.   Baku has been a nexus for gas and oil transport for many, many years: north through Russia, south through Iran, west through Georgia.  The Internet follows these same routes, with copper and glass fiber laid alongside existing rail and pipeline resources.   And indeed, this is the route that Iran seems to have chosen to complete their Internet diversity play in the weeks leading up to the commemoration of the 1979 Revolution.<p>

<big><strong>Enter the Russians</strong></big><p>

Two weeks ago, on January 26th, Russia's Rostelecom suddenly stepped in as another major transit provider to DCI, visible in the global Internet routing tables.    This plot shows the relative stability of DCI's transit relationships over the preceding 16 months.   Each international transit provider's percentage of Iran's transit is rendered in a different color; they all add up to 100% each day, but the mixture changes over time.   DCI seems to change its provider mix just before significant events: <a href="http://www.teliasonera.com/about_teliasonera">TeliaSonera</a> (brown) joins the provider mix in the weeks before last year's election.    <a href="http://www.rt.ru/en/about/info/">Rostelecom</a> (yellow) shows up two weeks before the commemoration of the Revolution. <p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/02/iran-transit-72.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/02/iran-transit-72.shtml','popup','width=800,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/02/iran-transit-thumb-600x360-72.png" width="600" height="360" alt="iran-transit.png" class="mt-image-none" style="" /></a></span><p>

If we zoom into the last four months, we can see how abruptly DCI's transit preferences have shifted.   Rostelecom has suddenly and decisively become the third-most important provider for Iran's international transit, behind Turk Telekom (TTnet) and TeliaSonera.  <p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2010/02/iran-transit-zoom-75.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2010/02/iran-transit-zoom-75.shtml','popup','width=800,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2010/02/iran-transit-zoom-thumb-600x360-75.png" width="600" height="360" alt="iran-transit-zoom.png" class="mt-image-none" style="" /></a></span><p>


For Iran, it's a smart choice in two ways.<p>

First, buying Russian transit actually makes good technical sense.   <a href="http://www.rt.ru/en/about/info/">Rostelecom</a> and other Russian providers have made huge progress building out their domestic networks and their connectivity to European and Asian markets.   Their terrestrial east-west capacity is ample, diverse, and poses an increasingly attractive alternative to the southern submarine cables.  Would you rather wait two weeks for a fix to a cable on the ocean floor, or a few hours for a fix to a fiber along a rail line?<p>

Secondly, of course, from Iran's perspective, Russian transit makes for great geopolitical diversity.   In the last year, Iran and Russia have made a lot of <a href="http://www.atimes.com/atimes/Middle_East/KG30Ak01.html">visible progress</a> on strategic <a href="http://www.upi.com/Top_News/Special/2009/07/30/Iran-Russia-conduct-naval-exercise/UPI-81271248998193/">military</a> and <a href="http://www.tehrantimes.com/index_View.asp?code=208936">energy</a> <a href="http://www.businessweek.com/news/2010-02-11/iran-urges-russia-to-keep-strategic-link-set-oil-price-floor.html">coordination. </a>  The appearance of northbound Internet transit provides additional evidence that beyond the <a href="http://www.google.com/support/forum/p/gmail/thread?tid=7b50afe70c40ab3b&hl=en">tactical concerns</a> of censoring its citizens' access to <a href="http://www.itiran.com/?type=news&id=11590">Western Internet content</a>, the Iranian government is thinking strategically about cyberspace as a national resource to be defended. ]]>
    </content>
</entry>

<entry>
    <title>Much Ado About Baidu</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2010/01/baidu.shtml" />
    <id>tag:www.renesys.com,2010:/blog//1.167</id>

    <published>2010-01-13T10:58:00Z</published>
    <updated>2010-02-11T20:24:55Z</updated>

    <summary> As our faithful readers know, Renesys monitors routing on the global Internet in real time and uses that information in a variety of ways. For example, we can instantly let you know which networks a hurricane has disabled or...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
As our faithful readers know,
Renesys monitors routing on the global Internet in real time and uses that information in a variety of ways.
For example, we can instantly let you know which networks a 
<a href="http://www.renesys.com/blog/2008/09/ike-brings-biggest-multistate.shtml">hurricane</a> has disabled or even
tell you when a 
<a href="http://www.renesys.com/blog/2008/08/georgia-clings-to-the-net.shtml">war</a> has left things pretty much as they were.
In short, we keep an eye on the Internet, the entire Internet,
but this is all done at the level of IP addresses and the paths they follow.
</p>

<p>
The recent 
<a href="http://www.theregister.co.uk/2009/12/18/dns_twitter_hijack/">attack</a> on Twitter got us thinking.   
Maybe we should be keeping an eye on a few more things?
While your IP addresses and routes to them might be completely stable, 
the average user doesn't know about those.
In other words, when was the last time you typed ...
<br>
&nbsp;&nbsp;&nbsp;&nbsp;<a href="http://216.239.59.104">http://216.239.59.104</a>
<br>
instead of ...
<br>
&nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.google.com">http://www.google.com</a>
<br>
into your browser?
</p>

<p>
What if someone manages to point your domain name to some other IP addresses?
You would still be operational as far as the Internet routers were concerned,
but no humans would probably be reaching you. 
And that's the problem we'll briefly consider in this blog.
</p>
]]>
        <![CDATA[<p>
Of course, none of this is new.
<a href="http://en.wikipedia.org/wiki/DNS_cache_poisoning">DNS cache poisoning</a> has been with us a long time, 
but security researchers are devising increasingly
<a href="http://portal.acm.org/citation.cfm?id=1455770.1455798">clever ideas</a> to defend
DNS from these sorts of attacks.
Unfortunately,
securing one part of the Internet, 
simply sends the miscreants to less well defended avenues.
With respect to state-of-the-art DNS servers (admittedly probably a small set), 
the weakest link may now be via some of the many <a href="http://en.wikipedia.org/wiki/Domain_name_registrar">domain name registrars</a>.
These are the folks who are ultimately responsible for how your domains get associated with IP addresses and you generally keep this information current yourself via an online account at the registrar.
But what if hackers compromised your login credentials,
say by guessing your password or using a poorly designed password reset functionality or maybe by a little social engineering?
They could then point your domain anywhere they wanted.
No need for a sophisticated attack against a hardened DNS server when a simple username and password will do.
</p>

<p>
So do we really need to worry about this or was the Twitter attack a one-time event?
We didn't have to wait long for the answer.
According to <a href="http://www.alexa.com">Alexa</a>, 
baidu.com, the "Google" of China, is the 8th most popular site in the world and,
not surprisingly,
the top site within China.
And it too seems to have had its 
<a href="http://www.computerworld.com/s/article/9144039/Baidu.com_probably_attacked_from_U.S._domain_registrar_says_researcher">registration compromised</a>.
From our queries around this time,
www.baidu.com most commonly resolved to 220.181.6.175 and a few other IPs in the
220.181.6.0/24 prefix.
These IPs are actually routed via the larger 220.181.0.0/19 prefix, 
originated by AS 23724, a China Telecom "Internet Data Center".
From there, 
these IPs reach the rest of the Internet via any one of five global providers,
as depicted below.
</p>

<p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><img alt="220.181.6.175.1263254400.png" src="http://www.renesys.com/blog/2010/01/12/220.181.6.175.1263254400.png" width="500" height="265" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></span>
</p>

<p>
But then something funny happened, 
on January 11th at 23:09 UTC, 
we received our first answer with a very different IP address, 
namely, 188.95.49.6, which resolves to pink2.warez-host.com.
This IP is routed via the prefix 188.95.49.0/24,
originating from the hosting provider,
<a href="http://www.i3d.net/">Interactive3D</a> (AS 49544) in the Netherlands.
Interactive3D, which we first observed in the routing tables on 24 October 2009,  
owes its connectivity to the providers shown below.
And this new IP did not host the Baidu web site,  
but rather a page for a group claiming to be the 
<a href="http://thelede.blogs.nytimes.com/2010/01/12/iranian-cyber-army-strikes-chinese-site/">Iranian Cyber Army</a>.
(Interestingly, there are several other domains with "warez" in the title hosted at Interactive 3D.)
</p>

<p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><img alt="188.95.49.6.1263254400.png" src="http://www.renesys.com/blog/2010/01/12/188.95.49.6.1263254400.png" width="288" height="251" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></span>
</p>

<p>
To reduce the load on DNS servers,
every DNS response comes with a
<a href="http://en.wikipedia.org/wiki/Time_to_live">TTL</a> or "Time to live".
This value tells the receiver how long to cache the response before troubling the server again with the same request.
Small TTLs allow the site administrator to change servers quickly at the expense of more frequent queries.
For the operators of baidu.com,
they seem to have been typically using a TTL of 14400 seconds or 4 hours, 
a perfectly reasonable value.
Whoever took over their domain left that value unchanged when they misdirected the site.
If they had increased it,
servers around the world could have retained the bogus baidu.com IP address for this domain for a very long time.
</p>

<p>
Within about 2 hours, 
we started seeing the return of the 220.181.6.175 address.
As of this writing, all of the DNS servers for this baidu.com domain are reporting a TTL 1200,
or 20 minutes.
Better safe than sorry, I guess.
But before everything had settled down,
at 02:35:59 UTC on January 12th, about two and half hours after we saw the first misdirected response,
we starting seeing responses of 127.0.0.1 with a TTL of 300,
which went on for about 20 minutes.
This is the 
<a href="http://en.wikipedia.org/wiki/Localhost">the loopback IP address</a>,
or the IP address every computer on the Internet uses for local communications.
Clearly this was an error or perhaps a misguided attempt to clear out caches.
</p>

<p>
The moral of the story here seems to be that regardless of how well run and secure your network may be, 
you depend on the goodwill of others to be reachable on the Internet.
If someone hijacks your IP space altogether,
as happened to 
<a href="http://www.renesys.com/blog/2008/02/pakistan-hijacks-youtube-1.shtml">YouTube</a> in 2008,
or your registrar is compromised and your domain is pointed elsewhere,
as with Twitter and now Baidu,
the best you can sometimes do is watch and wait.
And then react to the attacks when they occur,
as quickly as your monitoring system allows.
</p>
]]>
    </content>
</entry>

<entry>
    <title>A Baker&apos;s Dozen in 2009</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/12/a-bakers-dozen-in-2009.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.166</id>

    <published>2010-01-01T04:59:59Z</published>
    <updated>2010-01-13T19:07:07Z</updated>

    <summary> As our regular readers know, Renesys collects a lot of Internet routing data, using it to create reports and products based on hard facts and objective analysis. Perhaps the only controversial thing we do with our data is to...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Business" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
As our regular readers know, 
Renesys collects a lot of Internet routing data,
using it to create reports and products based on hard facts and objective analysis.
Perhaps the only controversial thing we do with our data is to <em>rank</em> all the service providers in the world: globally, by geography, and by market segment.  
The rankings are a rather crude measure of <em>size</em>, 
as they are based entirely on the quantity of IP space ultimately transited by each provider.
However, it's the ranking <em>trends</em> that are more revealing than any absolute number. Who is adding customers?  Who is losing them or just standing still?
Changes in IP transit answer these questions and more.
Although there are obvious shortcomings in this approach, 
it is certainly objective and the process is fully automated.
All of our rankings are updated daily and available via our 
<a href="http://www.renesys.com/products_services/market_intel/">Market Intelligence</a>
offering.
In this posting,
we will take a look at the top 13 providers in the world for 2009 and
how they have jockeyed for position throughout the year,
similar in spirit to our 
<a href="http://www.renesys.com/blog/2008/12/winners-and-losers-for-2008.shtml">December 2008 blog,</a>
which provides more details about our methodology.
We will see what a difference a year has made and highlight some of the more interesting changes.
</p>
]]>
        <![CDATA[<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/top-13.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/top-13.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
The above graph plots global scores over time for the top 13 providers in 2009.
Since the exact numeric score is not that meaningful in this context, 
we will not provide the scale in our graphs. 
Rather, we will show how the scores changed over the year and consider some of the reasons behind these changes.
The first thing to note is that membership in this elite set of top global providers did not change in the last two years.
The next thing to note is that within this set, 
there are distinct tiers,
where members of each tier vie for position.
These details will emerge as we enlarge the scale and break the group into three distinct clusters,
namely, the top two, the middle seven, and the final four.
In these expanded graphs,
the changes will be more obvious and we'll make note of some of the more interesting ones.
</p>
 
<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/top-tier.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/top-tier.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
In 2006, when Renesys first started ranking providers, 
Sprint (AS 1239) had a commanding lead with a global score of almost 25% more than second-place Level 3 (AS 3356). 
In 2009,
Level 3 pulled away from largely stagnant Sprint, 
opening up a roughly 12% lead.
In the spring of 2009,
Level 3 gained nLayer (AS 4436) as a customer, 
which Renesys ranks as #12 in the US, 
thereby giving Level 3 a substantial boost in score.
They also picked up KDDI (AS 2516) as a customer, 
the #2 ranked provider in Japan.
Finally,
Level 3 won increased transit from Japan's Softbank Telecom (AS 4725),
Europe's Interoute Communications (AS 8928),
Turkey's TTnet (AS 9121) and a wide variety of other strong regional players.
In short, Level 3 is aggressively gaining Internet transit business around the globe in both established and emerging markets,
while Sprint, still solidly in the #2 position,
appears to be largely treading water.
</p>

<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/mid-tier.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/mid-tier.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
Looking at the second cluster of providers, 
it is readily apparent that Global Crossing (AS 3549) is in firm control of the #3 spot, 
but their growth slowed considerably in 2009 over 2008.
In 2008,
Global Crossing's score increased by almost 50%, 
as they vaulted from #5 to #3.
Their score increased 11% in 2009,
leaving them safely in #3 with no clear challengers, 
but still a long way from the #2 position at three quarters of the size of Sprint.
</p>

<p>
Savvis (AS 3561) surged upward by over 25% in the summer, 
in large measure by winning Asia Netcom's (AS 10026) business.
But then they fell back considerably by losing Singapore Telecom (AS 7473) and Comcast (AS 7922) in the fall,
ending the year with a score of around 8% higher than at the start.
</p>

<p>
TeliaSonera (AS 1299) rose by over 20% during the year, 
around a quarter of the increase due to gaining Russia's Rostelecom (AS 12389) as a customer.
Tata (AS 6453) entered this middle tier of providers in 2009 from the bottom tier in 2008 
by consistently winning business throughout the  world,
including Serbia's Telekom Srbija (AS 8400), 
UAE's Emirates Telecommunications (AS 8966), the US's Road Runner (AS 7843) and Comcast (AS 7922), South Korea's Hanaro (AS 9318) and many others.
As a result, Tata has now passed AT&T (AS 7018) to capture the #8 global ranking.
</p>

<p>
The remaining members of this tier, AT&T (AS 7018), Verizon (AS 701) and NTT (AS 2914), like Sprint in the top tier, largely treaded water in 2009.
</p>

<p>
<body bgcolor="#ffffff">
   <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
           codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
           width="690" height="400"
           id="Line" >
      <param name="movie" value="/fusioncharts/charts/ScrollLine2D.swf" />
      <param name="FlashVars" value="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/bottom-tier.xml" />
      <param name="quality" value="high" />
      <embed src="/fusioncharts/charts/ScrollLine2D.swf"
             flashVars="&dataURL=http://www.renesys.com/blog/assets_c/2009/12/bottom-tier.xml"
             quality="high"
             width="690" height="400"
             name="graphs"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer" />
   </object>
</body>
</p>

<p>
Within the final set of providers under consideration,
both Tinet (AS 3257) and China Telecom (AS 4134) showed significant spikes in score in 2009.
In April, 
Tinet saw increased transit from Interoute (AS 8928) and,
in November, 
they picked up Singapore Telecom (AS 7473) as a customer.
China Telecom's growth was of a very different nature:  
they grew by routing more address space on behalf of themselves, 
not by entering new markets.
Rounding out this final group,
we see Cogent (AS 174) with slow and steady growth throughout the year,
while Qwest's (AS 209) score remained essentially constant.
</p>

<p>
Finally, note that large global providers are not necessarily transit-free, sometimes 
called Tier 1, providers.
Tinet and China Telecom, who made our rankings, are not transit-free,
whereas, XO (AS 2828) and Abovenet (AS 6461), who did not make our rankings, are.
</p>

<p>
<b>Conclusions</b>
<br>
Internet transit is an increasingly 
<a href="http://www.renesys.com/blog/2009/11/ip-backbone-hard-sell-not-so-m.shtml">tough business</a>,
as prices continue to collapse.
The emerging markets in Asia and the Middle East with their relativity 
<a href="http://www.internetworldstats.com/stats.htm">low Internet penetration</a> and higher margins remain a bright spot.
These markets still offer enormous potential and providers willing and able to service them are growing and rising in our rankings.
Those that do not service these areas are being left behind.
For example,
the US old guard of Sprint, AT&T and Verizon are at a virtual standstill,
China is growing but without diversity,
and Level 3 and Tata,
two companies to keep your eye on,
are growing thanks to strong global diversity.
</p>

<p> 
But we hasten to point out that market share does not imply profitability or even a well-run,
<a href="http://www.renesys.com/blog/2009/08/internet-diversity.shtml">resilient</a> and
<a href="http://www.renesys.com/blog/2009/01/the-adventurous-parts-of-the-i.shtml">stable</a> network.
So we would never suggest you pick your providers based on Renesys market share rankings alone. 
While declining market share might indicate a problem, 
a growing company is worth your business only if they meet your service needs and are going to be around tomorrow. 
Objective rankings, by their very nature, cannot be tailored for each consumer's needs, but they can contribute meaningfully to informed business decisions. 
</p>]]>
    </content>
</entry>

<entry>
    <title>Bonjour, Y&apos;all! ASN Split Personalities</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/12/bonjour-yall-asn-split-persona.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.165</id>

    <published>2009-12-09T01:59:42Z</published>
    <updated>2009-12-27T01:47:54Z</updated>

    <summary>Remember when the telephone company came to your house to hook up your phone and gave you a new phone number? This new number was how your friends and family were going to contact you. You counted on the telephone...</summary>
    <author>
        <name>Doug Madory</name>
        <uri>http://www.renesys.com</uri>
    </author>
    
        <category term="Governance" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="afrinic" label="AFRINIC" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="apnic" label="APNIC" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="arin" label="ARIN" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="bgp" label="BGP" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="iana" label="IANA" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="lacnic" label="LACNIC" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="ripe" label="RIPE" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="rir" label="RIR" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>Remember when the telephone company came to your house to hook up your phone and gave you a new phone number? This new number was how your friends and family were going to contact you. You counted on the telephone company to ensure that someone hadn't already been issued that number, because if they had, various problems would ensue. What would happen when your mom tried to call your number if it was also assigned to someone else? Could you directly call the other party to work out the problem? Well, in the <a href="http://en.wikipedia.org/wiki/Bgp">BGP</a> realm, something similar has been happening with <a href="http://en.wikipedia.org/wiki/Autonomous_system_%28Internet%29">autonomous system</a> numbers (ASNs).</p>

<p>Organizations need an ASN to run BGP and route on the Internet. They are each assigned globally unique ASN(s) by their local <a href="http://en.wikipedia.org/wiki/Regional_Internet_registry">Regional Internet Registry (RIR)</a>, who get them from <a href="http://www.iana.org/">IANA</a>. A few weeks ago, the <a href="http://nanog.org/">NANOG </a>folks <a href="http://mailman.nanog.org/pipermail/nanog/2009-November/015433.html">noticed </a>that AS1712 had been registered by two different organizations (in France and Texas) that were both using the number to announce their separate <a href="http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing">network prefixes</a>. ARIN issued a <a href="http://mailman.nanog.org/pipermail/nanog/2009-November/015498.html">statement </a>conveying that they were aware of the problem and were working to resolve it. We took a look at the data and found that AS1712 isn't the only dually-assigned ASN out there. In fact, even a root server didn't escape unscathed.</p>]]>
        <![CDATA[<p>First let me introduce myself: I'm Doug Madory, the newest member of the Renesys team. I came to Hanover, NH to do <a href="http://engineering.dartmouth.edu/">grad school</a> and after a two-year stint as a <a href="http://www.dhmc.org/">medical center</a> <a href="http://en.wikipedia.org/wiki/Chief_information_security_officer">ISO</a>, I jumped at the chance to be part of the cool work that goes on at Renesys. As part of my initiation here, I was handed the task to see if I could find any other examples of dually-assigned ASNs in the various RIRs &mdash; particularly ones that are currently in use. This isn't as easy as it sounds, but we will get to that in a moment. </p>

<p>Knowing how to direct packets from an Internet café in Singapore to a network in Buenos Aires is what the Internet routers do. BGP is the protocol Internet routers use to communicate changes in routes from one autonomous system (an organization on the Internet) to another. Each autonomous system is assigned a unique number, its ASN.</p>

<p>These ASNs are primarily used to avoid <a href="http://en.wikipedia.org/wiki/Routing_loop">routing loops</a> (i.e. you shouldn't be learning about your networks from your AS neighbors!). This is called loop detection and operates correctly only if ASNs are unique. No two RIRs are supposed to give out the same IP addresses or the same ASNs, otherwise we'll have bedlam &mdash; but this has happened!</p>

<p>In November 2009, it was <a href="http://mailman.nanog.org/pipermail/nanog/2009-November/015433.html">noted</a> in a NANOG list discussion, AS1712 has been doubly assigned to both a Texas-based organization by <a href="https://www.arin.net/">ARIN </a>and an organization in Paris, France by <a href="http://www.ripe.net/">RIPE</a>. They were simultaneously using their ASN to announce their respective network prefixes to the world.</p>

<table class=MsoTableGrid border=0 cellspacing=0 cellpadding=0
 style='border-collapse:collapse;border:none;mso-yfti-tbllook:1184;mso-padding-alt:
 0in 5.4pt 0in 5.4pt;mso-border-insideh:none;mso-border-insidev:none'>
 <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;mso-yfti-lastrow:yes'>
  <td width=319 valign=top style='width:239.4pt;background:#EAF1DD;mso-background-themecolor:  accent3;mso-background-themetint:51;padding:0in 5.4pt 0in 5.4pt'>
<pre>
$ whois -h whois.ripe.net AS1712
...
% Information related to 'AS1712'

aut-num: AS1712
as-name: FR-RENATER-ENST
descr:   Ecole Nationale Superieure 
         des Telecommunications,
descr:   Paris, France.</pre>
  </td>
  <td width=319 valign=top style='width:239.4pt;background:#F2DBDB;mso-background-themecolor:  accent2;mso-background-themetint:51;padding:0in 5.4pt 0in 5.4pt'>
<pre>
$ whois -h whois.arin.net AS1712
...
OrgName:    Twilight Communications
OrgID:      TWILI
Address:    1674 Kosik Ln
City:       Wallis
StateProv:  TX
PostalCode: 77485
Country:    US

ASNumber:   1712
ASName:     TWLT
ASHandle:   AS1712</pre>
  </td>
 </tr>
</table>

<br>
<p><a href="http://www.iana.org/assignments/as-numbers/">According to IANA</a>, AS1712 is to be assigned by ARIN, not RIPE. However, according to <a href="ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-latest">RIPE's records</a> it appears that many AS numbers in the 1700's were assigned by RIPE in 1993 and therefore AS1712 could have been assigned by RIPE in the swamp days before ARIN even existed.</p>

<p>AS 1712 isn't the only case of this...</p>

<p>Other doubly-assigned ASNs in the 17XX range include:</p>
<ul>
	<li><strong>AS1708 </strong>(RIPE: Renater, ARIN: Abacus, San Francisco, CA)</li>
	<li><strong>AS1715 </strong>(RIPE: Renater, ARIN: Harrier Hawk Management LLC, NY, NY)</li>
	<li><strong>AS1716 </strong>(RIPE: Renater, ARIN: Critical Data Network, San Diego, CA)</li>
	<li><strong>AS1723 </strong>(RIPE: Renater, ARIN: Twilight Communications, TX)</li>
</ul>
<p>ARIN believes that they have <a href="http://mailman.nanog.org/pipermail/nanog/2009-December/015999.html">worked to fix the problem</a> on the 17XX range, but doubly-assigned ASNs are not limited to the 1700's as evidenced by:</p>
<ul>
	<li><strong>AS3745 </strong>(RIPE: Volkswagen, ARIN: Perot Systems, Auburn Hills, MI)</li>
	<li><strong>AS35868 </strong>(RIPE: Internet Software Consortium, ARIN: Logix3).</li>

</ul>

<p>In each of these examples, ASNs are both registered and are actively announcing prefixes for different organizations. We have alerted those responsible for these networks of the mix-up. What is fascinating is: why didn't someone check when the duplicate assignment was made?  Anyone can query the databases of the five RIRs from the Linux command line in the following way:</p>
<pre>
$ whois -h whois.arin.net AS1712
$ whois -h whois.ripe.net AS1712
$ whois -h whois.afrinic.net AS1712
$ whois -h whois.apnic.net AS1712
$ whois -h whois.lacnic.net AS1712<br>
</pre>

<p>In the last example from above, <strong>AS35868</strong> is a particularly interesting case:</p>

<ul>
	<li>Logix3, a Florida company, was assigned <strong>AS3<FONT style="BACKGROUND-COLOR: yellow">58</font>68</strong> by ARIN in 2005</li>
	<li>Logix3 originates their own prefix, globally visible via <strong>AS3<FONT style="BACKGROUND-COLOR: yellow">58</font>68</strong>, since 24 June 2006. </li>
	<li><strong>AS3<FONT style="BACKGROUND-COLOR: yellow">85</font>68</strong> was assigned by APNIC to ISC ("ISC-SUV1") in 2006.</li>
	<li>ISC started advertising 203.119.51.0/24 via <strong>AS3<FONT style="BACKGROUND-COLOR: yellow">58</font>68</strong> on 23 May 2007 via Fiji's University of the South Pacific, in order to host Fiji's local copy of the <a href="http://www.apnic.net/community/investing-in-our-community/root-servers">F-root server</a>.</li>
	<li><strong>AS3<FONT style="BACKGROUND-COLOR: yellow">58</font>68</strong> first appeared in RIPE as ISC on 29 September 2009.</li>
</ul>
<p>However this gets cleared up, we strongly suspect that the local transit for the Fiji F-root will have to change. (Its address won't, fortunately.) Since we don't have a peer in Fiji, so we can't confirm that 192.5.5.0/24 (F-root block) is <a href="http://en.wikipedia.org/wiki/Anycast">anycasted</a> there.  Anybody in Fiji want to <a href="http://www.renesys.com/tech/peering.shtml">peer</a> with us? </p>

<p>Despite the fact that verification services are readily available, neither the RIRs, the companies who received the duplicated ASNs, nor their providers seems to have checked if the ASN was assigned before making and accepting the ASN assignment. When an organization is assigned an ASN, it still needs to check to verify that this ASN hasn't already been assigned. One can't assume that the ASN handed to you by an RIR is unique because their databases aren't perfect.</p>

<p>But don't blame the RIRs, rooting out duplicate ASNs across RIRs is non-trivial. To identify duplicate ASNs, one cannot simply look for the same ASN in two RIRs. Some RIRs, like ARIN, try to list many of the ASNs in other RIRs as a pointer to them. Also, some large organizations may legitimately have the same ASN in two RIRs announcing different networks, for example, AS33771 (Safaricom) is listed in both AFRINIC and RIPE, but it probably does so because it announces network prefixes in the two continents (196.201.208.0/20 in Kenya and 41.90.0.0/16 in England). It doesn't help that the RIRs are not consistent in their naming conventions, for example, AS109 is "ciscosystems" in ARIN and "cisco-eu-109" in RIPE. So an effective automated ASN duplication detection tool is unlikely.</p>

<p>Lastly, businesses merge, acquire others or sometimes change their names without updating the records in the RIRs, for example AS3955 is Wang in ARIN and Getronics in RIPE, but Wang and Getronics are the <a href="http://www.computermuseum.li/Testpage/WangGetronics.htm">same company</a>. In fact, AS3955 also routes network prefixes (such as 150.124.0.0/16) for a third organization named, Compucom Systems. Compucom <a href="http://en.wikipedia.org/wiki/Wang_Laboratories#Getronics_North_America_sold_to_Compucom">bought Getronics in 2007</a>. AS32528 is Ross Labs in ARIN and Abbott Labs in RIPE. Ross and Abbott <a href="http://abbottnutrition.com/About-Us/Our-History.aspx">merged in 1964</a>. Who has the capacity to keep up with every change to a business's name?</p>

<p>Like everything on the Internet, nothing is ever easy, even something seemingly so trivial as assigning unique ASNs. The moral of the story is that, going forward, all parties involved in the assignment process would do well take some simple steps to verify the uniqueness of their new ASN because it gets very difficult to identify duplicates and fix the error after the fact. And we want to avoid more ASN split personalities who think they are in Paris -- both Texas and France!!</p>

<p>À bientôt, pardner!</p>
]]>
    </content>
</entry>

<entry>
    <title>IP Backbone: Hard sell, not so much</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/11/ip-backbone-hard-sell-not-so-m.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.164</id>

    <published>2009-11-20T20:14:55Z</published>
    <updated>2009-12-09T21:55:02Z</updated>

    <summary> Think you&apos;re too busy to blog? Think again. Or just ask your boss. After more than 100,000 miles in coach class this year (so far), my backbone may be aching, but the IP backbone market is as agile and...</summary>
    <author>
        <name>Bob Fletcher</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Business" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Economics" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/Kuala_Window-65.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/Kuala_Window-65.shtml','popup','width=450,height=582,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/Kuala_Window-thumb-300x388-65.gif" width="200" height="259" alt="Kuala_Window.gif" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>

<p>Think you're too busy to blog? Think again. Or just ask your boss. After more than 100,000 miles in coach class this year (so far), my backbone may be aching, but the IP backbone market is as agile and dynamic as ever. Sales opportunities abound, but to take advantage, you'd better be savvy, and just a little cagey.</p>

<p>So, as our gleaming 777 departs Kuala Lumpur, I'll just relax in my fully-reclined, ultra-deluxe coach seat and tell you what this globetrotting sales guy has seen, heard and figured out. </p>


<strong>Two new trends<br></strong>
As if the global financial crisis weren't enough, beleaguered NSPs have to rejigger their business plans (yet again) to accommodate encroachment from brazen usurpers and ever more competitive pricing:<p>

<ol>
	<li>Large eyeball networks (5 million+ subscribers) are selling paid peering to the largest content providers. </li>
	<li>There are big price reductions in IP transit all over eastern Europe - now close to parity with western Europe. </li>
</ol>]]>
        <![CDATA[<p>Just two years ago many large European and some North American NSPs thought eastern Europe would be their salvation, the best way to sustain growth and maintain margins. And why not -- IP transit in eastern Europe was selling for an extravagant €12/Mb/month back then. During 2008, several NSPs took advantage of this seductive pricing to increase their customer base in the region:</p>

<p><ol>
	<li>Global Crossing</li>
	<li>TiNet</li>
	<li>Cogent</li>
	<li>Deutsche Telekom</li>
	<li>TeliaSonea</li>
	<li>Interoute</li>
	<li>Etc.</li>
</ol></p>

<strong>Cogent does it again</strong>
<p>Not for the first time, most NSPs are blaming Cogent for shaking up the system (spoiling everybody's fun with tough competition). Early this year, they started providing eastern Europe with IP transit for just €4/Mb/month. Despite this, a few providers successfully entered the eastern Europe transit market this year: </p>

<p>
<ol>
	<li>TiNet</li>
	<li>TeliaSonera </li>
	<li>Global Crossing </li>
	<li>Cogent</li>
</ol></p>

<p>Their success came mainly at the expense of Verizon and Sprint. Level 3 and Telecom Italia Sparkle managed to maintain their positions.</p>

<strong>An interesting result of lower pricing </strong>
<p>Regional NSPs have abandoned the IP transit market and are using their networks to offer connectivity to major IXPs. And they just might reach out to Russian IXPs next year. I'll keep you posted.</p>

<strong>IP sales teams react</strong>
<p>So, how are IP transit sales teams coping with this market evolution? It varies. A few have cashed in their IP transit chips and are focusing on enterprise MPLS (multiprotocol label switching) opportunities. Others are reconfiguring their IP networks to gain revenue from peering partners that were previously settlement-free (paid no fees). Some are building their "on-net" business by seizing every available IP transit opportunity. While the most savvy NSPs are relying on Renesys Market Intelligence® to give them a competitive edge. :-)</p>


<blockquote>
<table><tr><td bgcolor=#99FFFF>
<strong><em>Road Tip: Get cozy with the crew.</em><br></strong>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/Bobf_On_Motorcycle-59.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/Bobf_On_Motorcycle-59.shtml','popup','width=350,height=275,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/Bobf_On_Motorcycle-thumb-300x235-59.gif" width="300" height="235" alt="Bobf_On_Motorcycle.gif" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>

<p>I always chat up flight attendants and/or pilots when I'm going someplace I haven't been to before; after all, they go there regularly and can let you in on stuff you'd never find in a guide book.</p>
<p>On a recent visit to a not-so-well-developed part of South America, I got to talking with an American Airlines captain in the departure lounge at Miami airport. He and the crew would be staying at the same hotel I was, so I asked the usual questions about shopping and restaurants. During his enlightening response he casually mentioned that I shouldn't casually wear my fancy watch outside the hotel.  Whoa. I asked him to elaborate. Turns out, just a few weeks ago his copilot stood outside our hotel waiting for a crew bus, all his possessions crammed into a backpack.  A motorcyclist zoomed out of traffic, slid impressively to stop right in front of him, produced a gun and demanded the backpack.  It was over in a split nanosecond. </p>

<p>Flight crews wait inside the hotel now. So do I.</p>
</td></tr></table>
</blockquote>

<strong>Paid peering has its effects</strong>
<p>Now that large eye-balls have seen their way to paid peering, the traditional wholesale IP transit market has split into two groups. Some have eyeball networks, the rest are wholesale pure players. </p>

<p>In this scenario, incumbents and cable operators with large domestic retail broadband networks have the advantage. So far, only retail-focused NSPs offer paid peering, but only because they don't have large legacy wholesale businesses to cannibalize.  I'll be keeping my own eyeballs on IP transit wholesalers with large retail networks (e.g., ATT, Deutsche Telekom, BT) to see if they join the fray. My money's on no. Pure players such as Cogent, Level 3 and TInet have the most to lose if this trend spreads.</p>


<blockquote>
<table><tr><td bgcolor=#C0C0C0>
<strong><em><p>On-net business </em> </strong>is a sweet deal for NSPs.  They get revenue coming and going because both parties pay for exchanged traffic. (If traffic on their network is from a peering link or an upstream provider, they can only charge at one end.) It's not just giants like Level 3 cashing in on this, regional NSPs are doing it at home and abroad.</p>

<p>NSPs are very secretive about their on-net percentages, but insiders believe the largest NSPs' on-net routing is in the 30% range.  </p>
</td></tr></table>
</blockquote>

<strong>Asian tigers hungry for more</strong>

<p>The Capacity Asia NSP networking event in Kuala Lumpur was as vibrant as ever this fall. Asian NSPs shook off the financial crisis and are growing again, while their European and North American counterparts continue to struggle. Prices haven't plunged in Asia, and many populous locations are still eager and ready for broadband connectivity.</p>

<p>Asian-generated Internet content is growing rapidly, so they don't want or need more connectivity to the USA. But there was a lot of buzz about the pressing need for a fail-safe mechanism - additional subsea communication cables in key areas. Sites to consider: just offshore from the Singapore landing station and emerging low-cost manufacturing locations such as Bangladesh. </p>

<p>BTW, I heard a rumor that a single cable from Singapore to Hong Kong had four breaks already this year. NSPs with limited fiber diversity were not amused.</p>

<blockquote>
<table><tr><td bgcolor=#99FFFF>
<em><strong>Road Tip: The laid-back dichotomy.<br></strong></em>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/Bobf_Visa-56.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/Bobf_Visa-56.shtml','popup','width=718,height=385,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/Bobf_Visa-thumb-600x321-56.gif" width="200" height="127" alt="Bobf_Visa.gif" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>

<p>I really enjoyed my first visit to Sao Paolo, Brazil last March. As promised, the steak was fabulous, the people friendly and easygoing. This charmingly relaxed attitude extends to the Brazilian consulate in Boston where I applied for a visa about a month before I was scheduled to travel. The application process was effortless, and I anticipated the return of my passport and the requested visa in a few days. That's what usually happens. </p>

<p>Ten days pass. No mail. No contact. I begin to leave increasingly urgent voice mail messages at the consulate. Another ten days go by. I'm running out of time and options, so, I dive into their IVR system maze hoping for contact with a sentient being, and finally get connected to . . . voice mail.  OK, I'm supposed to leave in three days now. It's visa hotline time. To my surprise, a very amiable fellow instantly responds . . . oh yeah, he seems to remember something like that, and he's pretty sure he put the paperwork somewhere.</p>

<p>Brazilian visas are good for five years. So, if you're planning to go to Rio for the 2016 Olympics, apply for your visa in 2011.  On the other hand, if you're planning a Russian excursion, not to worry - nothing escapes their attention. You'll have a gorgeous, foil-embossed document within days.  Sunny, relaxed Rio, or frigid, efficient Russia. You decide.</p>
</td></tr></table>
</blockquote>

<strong>A kind of twisted unity</strong>

<p>Google and a group of NSPs (Bharti Airtel, Global Transit, KDDI Corp., Pacnet, and SingTel) recently formed a transpacific cable consortium. They call it Unity. Cable deployment is ahead of schedule and should be activated soon, but with an interesting twist. Members pay a share of the infrastructure cost, and get to install their portion of the cable in their own optical equipment at both Chikura and Los Angeles. So, some members may deploy 10G waves, while others might fancy 40G or even 100G.</p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/Chikura_LA-62.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/Chikura_LA-62.shtml','popup','width=540,height=383,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/Chikura_LA-thumb-600x425-62.gif" width="400" height="282" alt="Chikura_LA.gif" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>

<p>A few NSPs quietly mentioned that Google has already been offering to sell bandwidth on their part of the cable in the wholesale market. This puts some NSPs between a rock and a hard competitive place -- with one of their biggest customers! </p>

<p>This creates an intriguing dilemma for Google, according to a financial markets expert I chatted with at the Capacity show. Google's P/E valuation has been driven by its reputation for disruptive technologies since the beginning of Google-time. Although it's unlikely, if they also start operating as a telecom utility (offering bandwidth and all sorts of cellular technology/services), that could change. P/E values for NSPs, wireless equipment manufacturers and wireless operators are rather modest to say the least. So even if Google has spectacular success selling bandwidth, I can't imagine it would contribute to even a modest change in their P/E expectations.</p>

<p>Will the bumpy IP transit ride continue in 2010? I'll be watching and reporting from my cushy coach seat.</p>

Cheers!<br>
Bob Fletcher<br>
<em>Sales guy<br></em>

<br><br>
<hr><br>
<strong><p><u>Glossary:</u></p></strong>

<p>An <strong>Eyeball Network </strong>is a content provider, such as a cable TV provider or DSL access network, which connects its content viewers (eyeballs) to the Internet through an NSP.</p>

<p>An <strong>Internet Exchange Point (IXP) </strong>is a physical infrastructure that network service providers (NSPs) use to exchange Internet traffic between their networks. It enables networks to interconnect directly, rather than through third-party networks.</p>

<p><strong>Peering </strong>is a mutually beneficial interconnection of separate Internet service providers to exchange traffic between the downstream customers of each network. With <strong>paid peering, </strong>fees are accessed by the NSP that receives significantly more downstream IP traffic than its peering partner.</p>
]]>
    </content>
</entry>

<entry>
    <title>Lights Out in Rio</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/11/lights-out-in-rio.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.163</id>

    <published>2009-11-11T06:43:25Z</published>
    <updated>2009-12-13T23:05:57Z</updated>

    <summary>When the power goes out to a large part of Brazil, as happened last night shortly after 10pm, it&apos;s going to have an impact on telecommunications....</summary>
    <author>
        <name>James Cowie</name>
        
    </author>
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[When the <a href="http://www.bloomberg.com/apps/news?pid=20601087&sid=a5IwDD5qXbkk&pos=8">power goes out</a> to a large part of Brazil, <a href="http://online.wsj.com/article/SB125790382947542827.html?mod=WSJ_hpp_sections_world">as happened last night shortly after 10pm, </a>it's going to have an impact on telecommunications.]]>
        <![CDATA[<p>When power drops, there's an immediate shock to the Internet routing system.   You can see it clearly in this plot, which shows the number of South American networks affected by serious instability or unreachability in the previous 15-minute window. </p>   
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/sa2-ren_wm-47.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/sa2-ren_wm-47.shtml','popup','width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/sa2-ren_wm-thumb-400x300-47.jpg" width="400" height="300" alt="sa2-ren_wm.jpg" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>
<p>By compiling the routing traffic we collected from all over the world during this period of time,  it's possible to pinpoint the time of the failure in the Brazilian electric grid (10:26pm, or 00:26 UTC).  From an essentially negligible background level, the number of affected networks vaults to more than 150 within seconds.   Following the classic pattern of widespread power failures, the number of freshly affected networks drops over time as backup power kicks in, and then there are 'echoes' of new failures as the backup power suffers its own problems.  </p><br><br>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/br-earth-ren_wm-50.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/br-earth-ren_wm-50.shtml','popup','width=937,height=870,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/br-earth-ren_wm-thumb-300x278-50.jpg" width="300" height="278" alt="br-earth-ren_wm.jpg" class="mt-image-none" style="" /></a></span>&nbsp;              &nbsp;<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/11/ihp-sa-ren-names_wm-53.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/11/ihp-sa-ren-names_wm-53.shtml','popup','width=915,height=874,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/11/ihp-sa-ren-names_wm-thumb-300x286-53.jpg" width="300" height="286" alt="ihp-sa-ren-names_wm.jpg" class="mt-image-none" style="" /></a></span>

<br><br><p>These are some screenshots from our KML-based regional outage visualizer, and our Internet Health Portal tool.   As you can see, Brazil took the largest hit, but we also saw Paraguayan and Uruguayan networks out as a result of the largest regional power outage to hit Brazil and its neighbors in several years.     </p>

<p>Inevitably, speculation will fly fast and furious about the cause.   Media reports this week suggested that <a href="http://www.cbsnews.com/stories/2009/11/06/60minutes/main5555565.shtml">hackers may have taken down the Brazilian grid in 2005 and 2007</a>.   Brazil says<a href="http://politics.theatlantic.com/2009/11/brazil_to_60_minutes_it_wasnt_a_hacker.php"> it simply wasn't so. </a>  Was tonight a sardonic repeat performance, in response to Sunday's 60 Minutes coverage?    </p>]]>
    </content>
</entry>

<entry>
    <title>Staring Into The Gorge: Router Exploits</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/08/staring-into-the-gorge.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.162</id>

    <published>2009-08-19T10:32:31Z</published>
    <updated>2009-09-13T19:11:21Z</updated>

    <summary>I&apos;m writing this blog entry from the campground at Vermont&apos;s beautiful Quechee Gorge, where I took the kids after work. Yes, Renesys is located smack in the middle of some of the nicest hiking, camping, and climbing on earth. No,...</summary>
    <author>
        <name>James Cowie</name>
        
    </author>
    
        <category term="Engineering" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/08/gorge-20.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/08/gorge-20.shtml','popup','width=300,height=400,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/08/gorge-thumb-600x800-20.jpg" width="150" height="200" alt="gorge.jpg" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span><i>I'm writing this blog entry from the campground at <a href="http://en.wikipedia.org/wiki/Quechee,_Vermont">Vermont's beautiful Quechee Gorge</a>, where I took the kids after work.  <strong>Yes,</strong> Renesys is located smack in the middle of some of the nicest hiking, camping, and climbing on earth.  <strong>No,</strong> you shouldn't move here, Northern New England has enough out-of-staters already, thanks.  <strong>Unless, </strong>that is, you are an unusually talented web developer, have worked as a peering coordinator, or know the Internet transit industry inside-out, in which case you should <a href="http://renesys.com/about/careers.shtml">send me your CV</a>, posthaste.  thanks,  --jim</i></p>

<br/><hr/><br/><br/>
<p><strong>Here We Go Again.</strong></p>
<p> Imagine an innocent BGP message, sent from a random small network service provider's border router somewhere in the world.  It contains a payload that is unusual, but strictly speaking, conformant to protocol.  Most of the routers in the world, when faced with such a message, pass it along.  But a few have a bug that makes them drop sessions abruptly and reopen them, flooding their neighbors with full-table session resets every time they hear the offending message.   The miracle of global BGP ensures that<em> every vulnerable router on earth </em>gets a peek at the offending message in under 30 seconds.  The global routing infrastructure rings like a bell, as BGP update rates spike by orders of magnitude in the blink of an eye.  Links congest. Small routing hardware falls over and dies.  It takes hours for things to return to normal.</p>]]>
        <![CDATA[
<p><a href="http://www.renesys.com/blog/2009/05/byte-me.shtml">Last time,</a> it was the African Network Operators' Group, who had the temerity to actually use those fancy new 4-byte autonomous system numbers in a heavily prepended production context, causing certain Quagga routers to reset sessions.</p>

<p><a href="http://www.renesys.com/blog/2009/02/the-flap-heard-around-the-worl.shtml">The time before that,</a> it was a small Czech ISP, who tickled a missed input range check in the configuration screen for the relatively obscure Mikrotel routers, which led them to send out updates containing massively bloated AS paths, which caused certain Cisco routers to reset sessions. Cisco promptly issued a <a href="http://www.cisco.com/warp/public/707/cisco-sa-20090729-bgp.shtml">security advisory</a> and patched the problem. </p>

<br/>
<p><b>And This Week ....</b></p>

<p>At 17:07:26 UTC on Monday, CNCI (AS9354), a small network service provider in Nagoya, Japan, advertised a handful of BGP updates containing an empty AS4PATH attribute.   Yes, a BGP update whose path attribute contained just three bytes: one byte for flags, one byte containing the AS4_PATH opcode, and a size byte equal to <em>zero.</em> </p>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/08/zeropath-17.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/08/zeropath-17.shtml','popup','width=448,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/08/zeropath-thumb-432x462-17.png" width="432" height="462" alt="zeropath.png" class="mt-image-right" style="float: right; margin: 0 0 20px 20px;" /></a></span>
<p>If you were writing this code, would you do something sensible if the ASPATH contained zero autonomous systems en route to the prefix in question? It's not terribly meaningful, but it's just a range check.  It doesn't seem like an outrageous corner case to anticipate and handle correctly.   Especially if you're writing the operating system for a big carrier-class router, where this logic is on one of the most important code paths for performing the mission of a very expensive piece of mission-critical hardware. </p>  

<p>This one seems to have bitten Cisco's IOS XR, a relatively newborn "from scratch" rewrite of the venerable IOS, destined to run on big iron, like the CSR-1 or the 12000 series.  If you were trying to cause a <a href="http://en.wikipedia.org/wiki/Scale-free_network">meltdown of the global Internet</a>, this would be the kind of platform you'd target.  Cisco promptly released a <a href="http://www.cisco.com/warp/public/707/cisco-sa-20090818-bgp.shtml">security advisory</a> and patched the problem.   (We refrained from blogging about this for a couple days in order to give everyone time to get fixed, but I expect that the people who run the kinds of big iron that use IOS XR are probably well-cared-for by Cisco's support team and didn't have to wait long.)</p>

<p><b>Musing at METRICON</b></p>

<p>Coincidentally, I was at USENIX last week, and sat in on <a href="http://www.securitymetrics.org/content/Wiki.jsp?page=Metricon4.0">a great workshop</a> on security metrics, where Sandy Clark gave a somewhat controversial presentation on the interaction between software quality and the timing of exploit appearances. (Sandy is a member of Matt Blaze's secure systems research group at UPenn.)  </p>

<p>In a nutshell (and Sandy, please correct me if I mangle your argument), one of the strongest predictors of a significantly large time to the emergence of the first zero-day exploit for a new version of software is the degree to which the release represents a substantial rewrite of the code.  Doing a rewrite seems to start a "honeymoon period, " during which time the system in question is safer from exploitation than it has been in a long time.  In fact, the magnitude of the protective effect is so significant, that you might ask yourself whether a dollar spent in pursuit of higher quality code is actually better spent rewriting the code periodically, to whatever quality standard you can achieve.  Maybe using robots.  Seriously. </p>

<p>I had a chance to chat with Matt and Sandy briefly afterward, and we mused about the implications for the security of the OSes that Internet routers run -- Cisco IOS, Juniper's JunOS,  Quagga, etc.   To be fair, the threat model is completely different.  You typically can't speak BGP to someone's router, for example, outside the context of a preapproved trust relationship.  But once you're talking, you're talking (through them) to <em>every other router on the planet.</em>    The stakes are a lot higher, but the value to an attacker is less clear-cut &mdash; crashing the Internet doesn't give you control of tens of millions of valuable computing platforms, the way a zero-day exploit in Internet Explorer does.</p>

<p>So, here we have a serious vulnerability (the first I know of) in a substantially rewritten version of a critical operating system, one which has been out there in the market for at least a few years without making major news in a bad way. </p>

<p>The honeymoon is over, in other words.</p>  

<br/>
<p><b>How Long Before This Happens Again?</b></p>

<p>The good news is that the <a href="http://www.cisco.com/cisco/web/support/index.html">Cisco TAC</a> is at the top of its game, getting in touch with affected customers and rolling upgrades.  The bad news is that all of us (not just Cisco) are evidently locked in this completely reactive mode of finding and fixing problems in the world's critical communications infrastructure.   Again and again, we have to wait to discover problems in the real world that should have been caught through automated techniques during development, or during testing in the lab.</p>
<p>The global mesh of BGP-speaking routers that we call the Internet has inherent vulnerabilities that stem from the software quality and policy weaknesses of its weakest participants, and the amplification potential of its best-connected participants.  Running sloppy software at the edge of the routing mesh (in enterprises, say) is unlikely to give anyone the ability to propagate large amounts of instability or partition the Internet.  But closer to the core, I think we have a serious problem to contemplate.</p>

<p>Remember, if you can get just <em>one provider </em>to listen to you, and not filter your announcements, you can get your message into the ear of just about every BGP-speaking router on the planet within about thirty seconds.  And if some subpopulation of those routers can be reset, they act as amplifiers for your instability.    Power law outage-size distributions are not a myth &mdash; they are a logical consequence of the structure of the Internet, the importance of a few key participants in carrying global traffic, and their reliance for interconnection on technologies that are clearly still in the shaking-out-the-obvious-bugs mode.</p>

<p>The honeymoon is over, folks.  We're staring into the gorge, and it's a long way down.</p>]]>
    </content>
</entry>

<entry>
    <title>Routing Redundancy: How much is enough?</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/08/internet-diversity.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.161</id>

    <published>2009-08-15T17:17:30Z</published>
    <updated>2010-01-04T13:46:51Z</updated>

    <summary> Internet connectivity is a good thing. Many of us depend on it for everything from our livelihoods to our entertainment. However, the Internet is very fragile and even the The New York Times is worried about it. But they&apos;re...</summary>
    <author>
        <name>Earl Zmijewski</name>
        <uri>http://www.renesys.com/blog/</uri>
    </author>
    
        <category term="Engineering" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>
Internet connectivity is a good thing.  
Many of us depend on it for everything from our livelihoods to our entertainment.  
However, the Internet is very fragile and even the 
<a href="http://www.nytimes.com/2009/08/04/business/04road.html?_r=2&partner=rss&emc=rss">
The New York Times</a> is worried about it.  
But they're primarily concerned with overloads that can occur when everyone on the 
planet does the same thing at roughly the same time, 
such as surfing for news about <a href="http://edition.cnn.com/2009/TECH/06/26/michael.jackson.internet/">Michael Jackson</a>.  
Unfortunately, we will never avoid all such scenarios.
Physical systems are designed around average and typical peak loads, 
not around extremely high loads associated with very unlikely events.  
Who would pay for that?
</p>

<p>
And this applies to other complex systems besides the Internet.  
I was in India during <a href="http://en.wikipedia.org/wiki/9/11">9/11</a> and, 
for two days, 
I could not make a traditional phone call to the US.  Why?  
Everyone in India knows <em>someone</em> in NYC, 
and they all picked up the phone at the same time to check in on them.  
The circuits were so overloaded, 
I couldn't even get the friendly "Your call cannot be completed as dialed" message.
</p>

<p>
No system is ever going to be engineered for insanely high loads.  
If everyone in your town decides to take a shortcut through your
neighborhood to avoid an accident on the highway, 
you are going to have trouble getting out of your driveway.  
But rather than give up and wait it out, 
there is something you can do <em>in advance</em> and at
reasonable cost:
build a second driveway to a different street on the
other side of your house, one that isn't fed by the same access roads
from the highway.  
This blog is about building such redundancy into your
Internet connectivity, 
so you aren't disconnected by a single failure.
And while it's good that the New York Times and various governments are watching the problem, 
if your business depends on the Internet, 
you're largely on your own to audit and verify that you are buying a sufficient level of redundancy for your budget. 
A lot of fragility problems could be solved by more informed consumers performing the necessary due diligence.
</p>
]]>
        <![CDATA[<p>
To plan for inevitable disruptions in service,
you must find and then eliminate single points of failure. 
A small business with a single provider, for example, has a single point
of failure because if that provider goes offline, so does the business.
Larger organizations often combat this problem by having multiple
providers, but it still may not be enough.  
In particular, 
if all of their providers ultimately rely on the same still-further-upstream
provider.  
This kind of indirect single point of failure, illustrated in the first graphic below, 
is usually more difficult to identify.  
Other kinds of hidden single points of failure are also important, 
such as when one's two providers both run their cables through the same physical conduit.  
If one errant backhoe severs the conduit, this "redundancy" is rendered useless.  
For this discussion, though, we'll focus on organization-level redundancy.
</p>

<p>
<a href="http://www.renesys.com/blog-support/2009/08/no-diversity.jpg" 
onclick="window.open('http://www.renesys.com/blog-support/2009/08/no-diversity.jpg','popup',scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0); return false">
<img style="border:1px solid black;" src="http://www.renesys.com/blog-support/2009/08/no-diversity.jpg" width="690">
</a>
<br>
<center>
<b>Single points of failure: direct (left) and indirect (center and right)</b>
</center>
<br>
</p>

<p>
Planning for direct provider redundancy is generally straightforward.
But avoiding indirect upstream single points of failure requires an
awareness of the ever-evolving business relationships found on the Internet.
To understand these relationships, we first have to introduce some terminology.
As readers of this blog will know, 
every organization with multiple Internet providers will tend to have two things under their control: block(s) of IP addresses (prefixes) and an identifying number, or 
<a href="http://en.wikipedia.org/wiki/Autonomous_System_Number">ASN</a>.
Renesys currently observes over 110,000 relationships between pairs of ASes on the Internet  that have decided to interconnect.
By carefully studying routing data from numerous vantage points around the globe,
we are able to classify these relationships and determine both the
direct <em>and</em> indirect dependencies between providers.
Once you map out these relationships,
you can then look for the single points of failure.
</p>

<p>
True diversity
&mdash; complete freedom from single points of failure &mdash;
is illustrated in the following graph,
where AS 1 has three providers, 
each of which is richly connected to the rest of the Internet.
One or even two of AS 1's providers can have technical problems and AS 1 will still be able to maintain its Internet presence.
This organization has planned well for eventual failures.
</p>

<p>
<a href="http://www.renesys.com/blog-support/2009/08/good-diversity.jpg" 
onclick="window.open('http://www.renesys.com/blog-support/2009/08/good-diversity.jpg','popup',scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0); return false">
<img style="border:1px solid black;" src="http://www.renesys.com/blog-support/2009/08/good-diversity.jpg" width="690">
</a>
<br>
<center>
<b>Diversity: no single points of failure</b>
</center>
<br>
</p>

<p>
So securing direct redundancy is fairly easy, but beyond that it gets complicated.
Large ASes typically encompass many geographic locations and might even have offices all over the world.
If AS 1 has three providers, does it have them at <em>all</em> of its worldwide locations? 
If not, maybe they have an internal network that connects their offices.
Hence, an office whose local Internet connection fails might still be able to reach the Internet via one of the other offices.
In other words, you cannot view an AS as a single entity, 
as it may in fact consist of many independent locations with their own providers and private connectivity to other parts of the organization.
As if sorting through all this wasn't enough, there is a final complication.
Mergers, buyouts, spin-offs and other aspects of modern capitalism can
result in a single organization consisting of many ASes.  Finding them all
is quite complex, as we discussed in 
<a href="http://www.renesys.com/blog/2009/05/keeping-score.shtml">a previous blog</a>.
</p>

<p>
To truly measure diversity, 
we cannot merely look at ASes and their relationships.
We have to look at organizations as a whole and how their associated prefixes are routed.
Does this particular prefix show diversity in how it is routed or is it solely dependent on one provider?
Nor is it sufficient to look at one instant in time.
Rather we have to track the prefix over longer periods in order to uncover seldom-used backup routes.
If the prefix really has multiple ways to reach it, 
you want to account for that and give credit where it is due.
</p>

<p>
With these ideas in mind,
we set about trying to measure Internet diversity for each organization.
Every associated prefix is given a score based on the amount of diversity observed over a period of two months.
Then the individual prefix scores are combined to give an overall score to the organization in the range of 0 &mdash; 100.
A score of zero represents no diversity at all (like the left side of
the first graphic), and a score of 100 represents very rich diversity
(as in the second graphic).
We leave the details of the algorithm to a follow-up piece, which will
also include data for the Internet as a whole.
For now let's look at a few examples of individual organizations.
</p>

<p>
<a href=http://www.amazon.com/>Amazon</a> is a well-known and hugely successful commercial retailer with a worldwide presence.
They have around a half dozen ASes and almost three dozen prefixes.
At an AS level, the Renesys 
<a href="http://www.renesys.com/products_services/market_intel/">Market Intelligence</a> tool shows Amazon with a rich set of top-tier providers, 
such as Level 3, NTT, Tinet (formally Tiscali), Qwest and others.
But are all Amazon prefixes seen via such a diverse set of providers, 
at least once in a while?
In fairness, Amazon might only care about diversity for some of its prefixes, 
but since we can't know which ones matter, 
we treat all prefixes the same and score them all based on their observed diversity.
Then we combine the results to get an overall Amazon score.
</p>

<p>
And the answer is &#8230; (drum roll please) &#8230; 99.16 for Amazon's total score.
They miss out on a perfect score of 100 only because of a few less diverse prefixes,
one of which geo-locates to Hong Kong and is only seen via NTT.
As you might expect, 
Amazon has very rich global Internet diversity and generally has no single points of failure with respect to its logical connectivity.
</p>

<p>
In total, 
out of over 75,000 organizations that we have identified, 
not all of which have their own ASNs,
over 9,500 score a perfect 100.
But most of these are rather small or geographically constrained,
so they perhaps have an easier time of implementing diversity
once they realize the importance.
An example of a larger organization achieving perfection is 
Vietnam Posts and Telecommunications (AS 7643).
Their hundreds of prefixes are all seen via numerous large providers ranging from AT&T to China Telecom.
</p>

</p>

<p>
At the other end of the spectrum, 
we have the 
<a href="http://en.wikipedia.org/wiki/Social_Security_(United_States)">
United States Social Security Administration</a>.
Their 18 prefixes show no diversity at all, 
as they are all seen only via Verizon (AS 701).
No matter how many physical connections Social Security has to Verizon and no matter how reliable Verizon has been for them in the past,
no service provider can guarantee their customers uninterrupted service to the entire Internet.
Through no fault of Social Security or even necessarily Verizon,
if another top-tier Internet provider were to de-peer Verizon,
then Social Security would lose access to parts of the Internet.
Since we saw two such top-tier de-peerings in 2008 alone,
between 
<a href="http://www.renesys.com/blog/2008/03/you-cant-get-there-from-here-1.shtml">Cogent and TeliaSonera</a> and between
<a href="http://www.renesys.com/blog/2008/10/wrestling-with-the-zombie-spri.shtml">Cogent and Sprint</a>,
this is not a far-fetched concern.
To ensure their connectivity and improve their score,
Social Security would need to pick up additional providers,
ones not solely dependent on Verizon.
But they are not alone: over 26,000 other organizations also have zero diversity.
</a>

<p>
Now let's return to the New York Times, 
whose <a href="http://www.nytimes.com/2009/08/04/business/04road.html?_r=1&partner=rss&emc=rss">article</a> 
motivated us to write up this work in the first place.  
Should the Times worry about its own Internet diversity?
Looking at its approximately half-dozen prefixes, 
the company receives a respectable score of 85.37. 
But if we disregard two of its smaller prefixes, 
keeping just the larger ones and the one in which 
<a href="http://www.nytimes.com/">www.nytimes.com</a> lives, 
its score increases to 93.5. 
So while the Times has some room for improvement, 
someone in the organization is clearly paying attention to the issue of redundancy when provisioning for its most important networks.
</p>

<p>
For our more technically inclined readers,
a subsequent blog will provide more of the details around our diversity algorithm and 
give results for the Internet as a whole.
Not surprisingly, as our examples suggest, 
the results vary widely.
</p>]]>
    </content>
</entry>

<entry>
    <title>The Proxy Fight for Iranian Democracy</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/06/the-proxy-fight-for-iranian-de.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.160</id>

    <published>2009-06-22T11:30:00Z</published>
    <updated>2009-08-13T17:38:27Z</updated>

    <summary>If you put 65 million people in a locked room, they&apos;re going to find all the exits pretty quickly, and maybe make a few of their own. In the case of Iran&apos;s crippled-but-still-connected Internet, that means finding a continuous supply...</summary>
    <author>
        <name>James Cowie</name>
        
    </author>
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Security" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="iran" label="iran" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="webproxies" label="web proxies" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[<p>If you put 65 million people in a locked room, they're going to find all the exits pretty quickly, and maybe make a few of their own.  In the case of Iran's <a href="http://www.renesys.com/blog/2009/06/iran-and-the-internet-uneasy-s.shtml">crippled-but-still-connected Internet,</a> that means finding a continuous supply of <a href="http://en.wikipedia.org/wiki/Squid_cache"><strong>proxy servers</strong></a> that allow continued access to unfiltered international web content like Twitter, Gmail, and the BBC. </p>]]>
        <![CDATA[
<p>A proxy server is a simple bit of software that you run on your computer.   It effectively lets you share your computer with anonymous strangers as a "repeater" for content that they aren't allowed to fetch themselves.  For example, an Iranian web browser might be manually configured to use your computer (identified by an IP address and a port number) as a Web proxy.  When your anonymous friend reads twitter.com, or posts a tweet, the request goes via your computer, instead of to Twitter's web server directly.  Except for a little delay, and the fact that your friend gets to see what the uncensored Internet looks like from New York or London or S&atilde;o Paolo instead of Tabriz or Qom, surfing through a proxy is pretty much like surfing without one. </p>

<p>As you might imagine, open web proxies are valuable commodities in places where it's forbidden, possibly dangerous, to surf the Internet.  Iran's opposition movement has been vigorously trading lists of open proxies over the past week.   And as you might further imagine, the Iranian government censors have worked overtime to identify these proxies and add them to the daily blacklists.</p>

As an experiment, we geolocated a list of about 2,000 web proxies (unique IP addresses and port numbers) that were shared on Twitter and other web sites over the course of the last week, to see if we could discern patterns in the places that are hosting them.   Most of these are no longer reachable from inside Iran, of course, precisely because they were made public.  The following map shows the distribution of those proxies worldwide.</p><p>
 
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/06/proxycount-5.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/06/proxycount-5.shtml','popup','width=812,height=643,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/06/proxycount-thumb-600x475-5.png" width="600" height="475" alt="proxycount.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span></p>


<p>The USA and Western Europe were well-represented, but so were China, India, Russia, Romania, Bulgaria, Vietnam, ... <strong>87 countries in all,</strong> a pretty impressive breadth of representation, considering the relatively small size of this sample.  (You can also see about a dozen <strong>Iranian </strong> IP addresses represented in the set.  Not surprisingly, all but one of these belong to networks originated by DCI, the <a href="http://www.renesys.com/blog/2009/06/strange-changes-in-iranian-int.shtml#more">government-run service provider</a> who operates the modern-day Internet equivalent of the Alam&#x016B;t Castle.)</p>

<p>Here's a geographic visualization of the proxies, drawn in Google Earth.  In the first one, we've drawn Iran in green, with some of their domestic network sketched in white, and their major international connections drawn in red.   Each of the colored arcs represents a single open web proxy; they are "fountaining" out of a cable landing or Internet traffic exchange point that makes approximate sense for their Iranian Internet routing.   For example, all of the web proxies in Europe are drawn from the Marseilles termination of the <a href="http://www.seamewe4.com/">Sea-Me-We-4 cable.</a>   The web proxies in Turkey are drawn in light blue, radiating from Ankara, where the <a href="http://en.wikipedia.org/wiki/Iran-Turkey_pipeline">Iran-Turkey gas pipeline</a> passes through on its way from Bazargan.   Those unusual Iranian proxies emerge from Tehran, and so forth. </p>
<p><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/06/third_2-8.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/06/third_2-8.shtml','popup','width=1329,height=760,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/06/third_2-thumb-600x343-8.png" width="600" height="343" alt="third_2.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span></p>
If we rotate the globe, you can see how the countries of Asia are doing their part to keep the bits flowing in Iran.  India, China, South Korea, Taiwan, Vietnam, and Japan are all visible sources of web proxy activity.</p>
<p><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/06/third_3-11.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/06/third_3-11.shtml','popup','width=1333,height=760,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/06/third_3-thumb-600x342-11.png" width="600" height="342" alt="third_3.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span></p>
<hr><p> I'd like to be able to say that these maps are a measure of the strength of the democratic impulse and volunteer spirit in all the countries of the world. But that might be a stretch.  You see, looked at another way, an open proxy is a security hole, something you might find in a machine that's been compromised, or at the very least, badly administered.  Security purists think of them as the "unlocked gun cabinet" of the Internet &mdash; a resource for anyone who wants to abuse a website, commit fraud, cover their tracks.</p>
<p>  Some of the proxies in this dataset are undoubtedly fresh, created by people who want to keep the Internet alive for the Iranian people.  But many of these proxies have probably been around for months or years, mapped out by those that map out such things. </p>
<p>We did see <a href="http://proxysetupforiran.blogspot.com/">a few</a> <a href="http://blog.austinheap.com/2009/06/15/how-to-setup-a-proxy-for-iran-citizens/">organizers</a> <a href="http://extrafuture.com/2009/06/15/how-to-set-up-an-anonymous-proxy-for-iranians-using-squid-on-mac-os-x/"> try to explain </a><a href="http://belsec.skynetblogs.be/post/7072317/setup-a-proxy-for-iran--overpower-their-censo">the concept of an ACL</a> (Access Control List) to all the new proud parents of open proxies.  If you are diligent, it is possible to restrict the anonymous users of your new proxy to just the Iranians, or even just the Iranian non-government networks, if you have a good enough list of the IP address blocks (network prefixes) in question.  But I expect that the complexity of configuring anything tighter than an "open access" proxy is going to prove too high a barrier to entry for most people who might volunteer to run one.</p>
<p> For one thing, we know how hard this is.  Renesys has pretty good lists of per-country networks and their transit patterns, based on our analysis of the global routing tables, and trust me, they take some work to maintain.  And even given good maps of Iran's address space to work from, ACLs are notoriously hard to test, if you don't have Iranian friends who can try your server from inside the protest zone and report back to you with problems.   Most people aren't going to bother, and that's probably okay.   Freedom is messy.   There'll be time for security later. </p>
<p><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/06/world_proxies-14.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/06/world_proxies-14.shtml','popup','width=766,height=655,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/06/world_proxies-thumb-600x513-14.png" width="600" height="513" alt="world_proxies.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span></p>
<hr>
<p>Perhaps the strangest thing of all, given how diverse and active and vocal the proxy server farmers have been, is that by and large, <em>it isn't working. </em>   The rate with which new proxies are being created has slumped over the last few days.   It's getting harder and harder to propagate new proxies to the people who need them, as the government consolidates its hold on the filtering mechanisms. Any new proxy addresses that are posted to Twitter, or emailed, will be blocked very quickly.  </p>
<p>  People we talk to inside Iran say that almost no proxies are usable any more.  Freegate, a Chinese anti-censorship application that makes use of networks of open proxies, has proven popular in Iran.  But this week, it, too, has been experiencing problems.   Many popular applications, like Yahoo! Messenger,  have stopped working.  The authorities are said to be using power interruptions as a cyberweapon, causing brief outages during rallies that cause computers to reboot, just as people are trying to upload images and video.  The net result, as <a href="http://asert.arbornetworks.com/2009/06/a-deeper-look-at-the-iranian-firewall/">Arbor's excellent analysis shows</a>, has been a drastic reduction in inbound traffic on filtered ports since the election.</p>
<p>If there's a lesson here for the rest of the world, perhaps it's this: Install a few proxy instances on machines you control.  Learn how to lock them down properly.  Swap them with your friends overseas who live in places where the Internet is fragile.  Set up your tunnels and test them.  And <em>don't wait until the tanks are in the streets to figure this out,</em> because by that point, you may have already lost the proxy war.</p>]]>
    </content>
</entry>

<entry>
    <title>Iran and the Internet: Uneasy Standoff</title>
    <link rel="alternate" type="text/html" href="http://www.renesys.com/blog/2009/06/iran-and-the-internet-uneasy-s.shtml" />
    <id>tag:www.renesys.com,2009:/blog//1.159</id>

    <published>2009-06-16T21:21:25Z</published>
    <updated>2009-08-13T17:37:57Z</updated>

    <summary> We&apos;ve received enough interest about our previous notes on Iranian Internet connectivity that I wanted to give a brief update, and some reflections....</summary>
    <author>
        <name>James Cowie</name>
        
    </author>
    
        <category term="Internet" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Politics" scheme="http://www.sixapart.com/ns/types#category" />
    
        <category term="Society" scheme="http://www.sixapart.com/ns/types#category" />
    
    <category term="internet" label="internet" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="iran" label="iran" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en" xml:base="http://www.renesys.com/blog/">
        <![CDATA[
<p>We've received enough interest about our <a href="http://www.renesys.com/blog/2009/06/strange-changes-in-iranian-int.shtml">previous notes on Iranian Internet connectivity</a> that I wanted to give a brief update, and some reflections.</p>]]>
        <![CDATA[
<p>In short: Iran is still on the Internet.  As the crisis deepens, people are literally <a href="http://www.reddit.com/r/worldnews/comments/8sz12/comprehensive_breakdown_of_the_current_situation/">risking their lives</a> by continuing to use the Internet for coordination and communication.   Iran's <a href="http://www.science-arts.org/internet/node38.html">physical</a> <a href="http://taeint.net/en/network/middle/">connectivity</a> to the Internet is so centralized, and so fragile, that it's within the power of the government to simply "turn it off" if they so desired.  And yet, they have not done so.</p>  

<p>Except for a brief period of outage over the weekend, the routes into Iran from the rest of the world have been basically intact, if a bit congested and unstable.   Most of that congestion and instability is probably the result of six billion people who are freshly interested in Iranian politics, all reading (and in some cases, yes, <a href="http://www.circleid.com/posts/20060615_iranian_protesters_urged_not_to_ddos_government_websites/">attacking</a>) Iranian websites.   We aren't making things easier for the people inside Iran, who need that same bandwidth to get out their images and observations and <a href="http://twitter.com/#search?q=%23IranElection">tweets.</a>  </p>

<p>To show that nothing much has changed structurally, consider the fact that the same lineup of six international carriers are still carrying the megabits back and forth to the government's monopoly points-of-presence at the Iranian border.  (See the following chart, which shows how the relative percentages of routes to Iranian networks carried by those providers has changed over the last few days.)   </p>

<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://www.renesys.com/blog/assets_c/2009/06/iran-providers-2.shtml" onclick="window.open('http://www.renesys.com/blog/assets_c/2009/06/iran-providers-2.shtml','popup','width=1500,height=450,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false"><img src="http://www.renesys.com/blog/assets_c/2009/06/iran-providers-thumb-800x240-2.png" width="640" height="192" alt="iran-providers.png" class="mt-image-center" style="text-align: center; display: block; margin: 0 auto 20px;" /></a></span>

<p>What happens inside Iran to those bits is anyone's guess (censorship, site blocking, traffic interception, and harassment, from all accounts).   But the pipes are <strong>open</strong> and the traffic is <strong>flowing.</strong>    In a few cases (which I will not detail, for obvious reasons), there are actually direct paths to international carriers, in defiance of government monopoly, that are now getting good use.</p>

<p>I've talked to a lot of people in the last few days about this puzzle.   In a crisis that verges on revolution, the first thing a government typically does is take control of the media that can be controlled, and shut down the media that cannot.</p>

<p><b>Why is it different this time?</b>   There seem to be three basic theories.</p>

<p>
<ul>
	<li><b>The cynics.</b>   Perhaps the government has left the Internet intact so that they can use it to surveil and round up dissidents.  Perhaps they even put bandwidth constraints in place to make it easier to cope with the volumes of traffic that need to be captured and filtered.</li>
       <li><b>The optimists.</b>  Perhaps the government has realized that a modern economy relies on the Internet to such an extent that it cannot be turned off, for fear of disrupting financial transactions and business communications.   Iran's Internet ecosystem is relatively rich, and the impact on their economy of a sustained Internet shutdown would be significant.  Why make it harder for companies to do business in Iran at a time when oil revenues are cratering and foreign investment is looking for reasons to take a walk?
       <li><b>The realists.</b>  Perhaps the government is too busy with other things to worry about the Internet.   Governments aren't well-suited to run the Internet, and they don't completely understand how it works.   The Internet has never been "turned off" before, and it would take creativity and thoughtful action to figure out who to ask in order to get it done.  So it simply hasn't happened, and probably won't.   Good thing, too, because they might not be able to turn it on again.  
</ul></p>

<p>You can pick the theory you like.    I guess I'm a realist, but I'd like to be an optimist.   If you wait long enough, something good can come out of something bad.   If fertility rates hadn't tripled during the long Iran-Iraq war in the 80s,  Iran wouldn't be faced with this demographic bulge of restless 20-somethings who have grown up with the Internet and expect it to keep working. </p>   

<p>You can fight people, but you can't fight human nature, and you can't fight demographics.  Iran is going to follow the same long-term trend as the rest of the developing world: building a civil society based on free expression, global communication, free mobility of human capital, and cross-border investment.   That's what the Internet symbolizes, and yes, if you ask these folks, it's worth fighting for.</p>]]>
    </content>
</entry>

</feed>
