Across the Internet, yesterday, Google users twittered, blogged and emailed that Google search and mail were not usable. And, yesterday afternoon, on Google's official blog, Urs Hoelzle reported that Google "direct[ed] some [...] web traffic through Asia".
Renesys observes the topology of the Internet by collecting routing data from a wide variety of vantage points (peers). Different peers may choose different routes to the same destination (e.g. Google) depending on their location in the Internet. Analogously, a driver starting from Los Angeles will choose a different highway to reach New York City than a driver starting in Montreal, Canada. On the Internet, each peer may choose a different best path or route to a destination. By recording the choices each peer makes, we can gain some insight into the relative importance of a given route to a destination.
The Internet is carved up into IP blocks (called prefixes) which make management of the network easier. Each provider learns about the prefixes of other providers by passing the knowledge around (like a game of 'telephone'). Google directly advertises or provides service to approximately 190 different prefixes, so the Internet learns how to reach Google based on the relationships Google has.
When trying to reach Google, then, any inbound traffic to them needs to pass across other providers with whom Google has a BGP routing relationship. We'll look at a very obvious shift in the number of other Internet providers who started reaching Google differently during yesterday's event.
Even though Google is very well-connected to a diverse set of other networks, two of these relationships predominate. It is through these relationships that much of the Internet reaches Google. If we look at Google's primary providers, Level 3 (ASN 3356) and AT&T (ASN 7018), we see that these are very important relationships to Google, since in the case of Level 3, just under 80% of peers choose to reach Google via Level 3 for just over half of all of Google's prefixes.
During yesterday's routing instability, Google's prefixes shifted to different relationships. When automatic, this sort of change is a key feature of redundancy. Sometimes such changes are motivated from a business change. Over the course of yesterday's instability, virtually all of our peers, at one time or another, chose to reach many of Google prefixes through NTT (ASN 2914), what had been a far less important relationship.
Note that the bar chart above is showing the total percentage of Google's approximately 190 prefixes and the total percentage of peers (different vantage points on the Internet) choosing to reach Google through NTT. At no single time did the entire Internet choose to reach Google through NTT, but over the course of the event, almost everybody tried to reach Google on this path.
Before yesterday's event, NTT was not so important an inbound path to Google. In fact, NTT carried very few of Google's prefixes (only 16) and was the preferred path to Google for only 10% of our peers. During several different bursts of announcements, this relationship became a much more important inbound path to Google.
In one particular series of routing changes, which started at around 14:45 UTC, several major providers, e.g. Deutsche Telekom (ASN 3320), Teleglobe (ASN 6453), Telia (ASN 1299), and even, peculiarly, AT&T (ASN 7018) started choosing to reach Google through NTT.
While yesterday's instability was disruptive to Google users all over the Internet, we are happy to report that it's back to business as usual with one of the most robust services on the Internet.
Google does not really say, in their official blog what the problem was. Do you know? Was this just an error on their part?
Editor's Note: We assume Google was responsible given that they accepted responsibility. The exact cause is somewhat inconclusive from existing external evidence.