The Internet is in for interesting times. Previously, I wrote about the engineering issues and about the policy issues facing us over the next five years. But there is at least one large issue still lurking. Most of you will not be surprised to learn that almost all of these issues are outgrowths of a single factor: money. The core of the Internet still doesn’t have a sustainable business model.
Many people are getting rich on the Internet, and almost none of them are spending money to keep the interconnection infrastructure (the “Inter” in “Internet”) growing and expanding. Look at it from a massively oversimplified perspective: Google make their money from the advertising they sell to search audiences. Comcast make their money by offering TV and Internet access on their local cable infrastructure. Amazon make money selling books and other stuff (including servers and storage space). Most datacenter companies make their money selling space and power inside of their buildings. Spammers make money filling up your inbox with useless crap. Organized crime makes money by launching attacks against profitable companies if they don’t pay extortion. DNS squatters make money registering thousands (or millions) of domain names and sitting on them until someone else is willing to pay. And almost none of this helps the core of the Internet.
Look to the wholesale carriers if you want to see an income statement wasteland. Level 3 lost $1.1b last year. They lost $120m in the most recent quarter alone. Cogent is thrilled because they reported a tiny, tiny positive net income last quarter on top of a yearly loss of $30m in 2007. Global Crossing lost $300m in 2007 and $88m in the last quarter they’re reporting, which doesn’t include much of the recent downturn. Other wholesale networks are in the same boat. Dan Golding suggested that it’s more important to look at net cash flows rather tha income, but the result is pretty much the same: almost no one is making any money. The only wholesalers who do make money make it on other service offerings: wireless service, metro Ethernet services, VPNs, local phone service, video services and so on. Are there sustainable Internet backbone business models? Does anyone have one?
An open market for buying and selling IPv4 Addresses is coming. Soon.
As I wrote previously, IANA is running out of unallocated IPv4 addresses. Estimates vary, but by 2010 (or 2012 at the latest) the world will be out of unallocated IPv4 addresses.
Sometimes it is hard for the general public to understand what this might mean. Essentially, after 2010 or so, if you want to start a new company and get connected to the Internet or just are growing and have more devices that need to have IP addresses, things won’t be the same as they are now. Right now what happens is that you go to ARIN, if you’re in North America and document your need for IP addresses, you pay a modest administrative fee, and then they allocate them to you. If you grow and you need more, you document how you’ve used up the ones that you have, and they give you more of them.
All of this assumes that you want your own IP addresses that are not tied to any particular provider (this is an important point that we’ll get back to). But even if you get your IP addresses from some provider, they have to get them from somewhere. If you want to be reachable from the Internet, you need an IP address—an IPv4 IP address in particular. And very shortly those are going to get much harder to get.
So let’s talk about what happens after the IPv4 addresses are all “used up.”
In a few weeks, I will be leaving Renesys, a company I have been associated with for over five years. I moved from New Hampshire (where Renesys is headquartered) to Pittsburgh, PA, over the summer, and I’ve decided to work a bit closer to my new home.
Before I go, there is work yet to be done. The Renesys blog has become an important place for Internet engineers, managers, developers and salespeople to seek unbiased information about what is happening on the backbones. I have enjoyed contributing to it over the years, and I have enjoyed watching some of my colleagues (most actively Earl Zmijewski and Martin Brown) take the helm more recently. Before I ride off into the sunset, there are at least two things I’d like to contribute to this forum:
- A clear assessment of where we are with this whole Internet project
- A good guess about where we’re going
At the end of the next series of posts by me, you should either be very, very worried or convinced that I’m very, very wrong. The Internet is facing a confluence of engineering, financial and policy storms that have some small potential to completely derail it. These tempests have a high likelihood of marking a sharp departure from several characteristics once considered fundamental to the the Internet.
If we get through the next five years, I’m sure everything will be fine. Today, I’ll tackle the technology and engineering issues. In my next post, I’ll address financial issues, followed by policy issues. At the end of this torrent of pessimism, I’ll try to point to some plausible ways out of the mess that we have gotten ourselves into.
Sprint re-enabled the connection between Sprint and Cogent at 21:00 UTC (16:00 EST) on Sunday, 2 Nov, 2008. Sprint issued a hastily prepared statement about the reconnection (the HTML is a cut-and-paste job from “IP/MPLS Products from Sprint”), explaining their position. Cogent hasn’t commented yet. The connection appears to be routed much as it was […]
A special Halloween edition of the Renesys Blog: That which was whole is now torn asunder, and cries of grief ring out throughout the land. Cogent (AS174) and Sprint (AS1239) are no longer connected to each other. Customers of each network who do not have other providers—namely single-homed customers—cannot reach each other. Two large portions of the Internet are separated.
Cogent is frequently involved in peering disputes. In the last three years, the only significant peering dispute (one that caused a temporary partition of the Internet) that did not involve Cogent was between Level 3 and XO. That one was settled very quickly. All of the others (Cogent depeering Telia, Level 3 depeers Cogent, and further disputes going back years involving Teleglobe (now Tata, AS6453), France Telecom (AS5511)) involved Cogent.
But in this case, Cogent may have picked the wrong sparring partner. In the past, Cogent won peering disputes simply because their customer base was less sensitive to the outage than the other party in the dispute. Ultimately, the one whose customers complain the loudest loses. This time it may be very different. Sprint hasn’t paid any particular attention to its IP product and network at a senior management level for a very long time. They are clearly focused on wireline and wireless telecom services and Overland Park management seem to remain mostly unaware that they even operate an IP network. In other words, Cogent has picked a fight with a zombie here. They may even rip off a limb or two, but that doesn’t mean the zombie will notice.
Sprint and Cogent only starting peering recently, back in November of 2006. Prior to that the two networks reached each other via NTT Communications (AS2914). Now, almost exactly two years later, it appears that Sprint has disconnected Cogent and chosen to divide the Internet. Cogent has stated that they will litigate this issue so this one is unlikely to get resolved quickly. In the mean time, over 200 downstream autonomous system customers of each organization cannot reach the networks in the other. This is ugly and will remain so.
Let’s take a quick look at what we know so far and set the stage for a story that will likely continue for several days, if not weeks. I’ll also try to set this in a larger context about the evolution of each of these networks and the evolution of Internet interconnection on the whole.
Cogent (AS174) has established a direct connection to the America Online Transit Data Network (ATDN) (AS1668). This long-awaited connection completes Cogent’s effort to directly connect with every transit-free network in the world and qualifies them, for the first time, as being transit-free.
In one sense, this is an unsurprising event. ATDN has been shrinking its transit network for some time in order to focus on their revenue-producing ad business. AOL/Time Warner has been selling off their European access networks since 2006. At the same time, Cogent has been adding customers and growing and peering. So that these two networks would eventually connect (re-connect) is unsurprising.
But the history between these two organizations is textured and murky. This connection is particularly interesting in part because of this history. It’s also interesting because of how different these two networks are, in almost every respect: history, revenues, business model, culture, brand. I’ll take a look at where Cogent is, the history between Cogent and AOL, and what this all might mean for the Internet.
Randy Epstein of Host.net and WVFiber graciously (or perhaps maliciously, given the quality of the performance) filmed and did the post-production on the recent performance at the Global Peering Forum. If I had a virtual tip jar, I would set it out. Enjoy:
At the recent Global Peering Forum I performed a spoof song based on the recent YouTube hijacking. (I’m told that video will eventually be available, at which point I’m sure I’ll have to go into hiding at an undisclosed location.)
American Pie was previously parodied at a RIPE meeting and now is practically a tradition, much to Mike Hughes’s chagrin, as he thinks it’s overdone already. The great thing about the original song is that it’s choc full of references in the music industry. I tried to pepper several more into my version (and I have a few additional verses in progress that I just didn’t finish).
What links would you provide to these references? What additional references do you think are important and missing (given the history of the Internet theme)?
The Day the YouTube Died
A long long time ago
I can still remember how the videos used to make me smile.
And I knew if I had my chance,
I’d watch the prison thriller dance
and maybe I’d be happy for a while.
But February made me shiver with every packet I’d deliver
bad routes in the tables, the paths they were not stable.
I can’t remember if I cried when I saw my request was denied
but boredom welled up deep inside
the day the YouTube died.
We’re doing something dangerous, here. We’re messing with the order of the universe. We’re changing things up on you. I think it’ll work out fine, though.
Many of you have been reading this blog since 2005 have become used to my acerbic wit and charming observations on the state of the Internet. OK, maybe you’ve just slogged through some boring posts for the few gems buried in among all of the garbage. I hope it has been worth it.
Cogent (AS174) sells IP transit and they sell it cheap. Everyone knows this. It is how they position themselves. It’s their competitive advantage. It’s why they think they will take over the world. They plan to undercut the prices and margins of all of the bigger carriers, combine that with strategic (cut-rate) acquisitions, and wait for everyone else to go broke. So far they’re doing a pretty good job of executing on that plan.
However, I recently learned that “Cheap” isn’t actually Cogent’s secret market advantage at all. I’m not denying price matters to people who buy from Cogent. Heck, that’s why Renesys decided to buy from Cogent for a development installation for Babbledog. (There will be more about Babbledog here shortly, as I’m sure it’s something that many of Renesys’s network-centric customers will have some questions about.)
This is a tale of a network-clueful small company trying to get connectivity at a well-connected building in Boston at a reasonable price. This is a tale of sorrow and woe, misery and despair. I’d like to say it has a happy ending, but on review, I believe that many of you will conclude, at least for the IP transit industry, that the ending is not happy at all.