Presentations
Blackout Results in Widespread Network Outages
Thursday, August 14, 2003
Summary
Widespread power outages in the northeastern U.S. starting on August 14 resulted in corresponding loss of reachability to several portions of the Internet. Backbone networks remained intact using emergency backup power supplies, but many regional and enterprise networks went offline. The event is can be clearly observed in global routing table size changes, and an evolving geographical timeline of network outages highlights the coverage of the blackout. The route withdrawal timeseries themselves also reveal the effects that the event had on global routing.
Routing Table Sizes
The majority of the power failures began at about 16:10 EDT. Immediately thereafter, the number of routes in global routing tables dropped rapidly, falling by nearly 1000 within five minutes. This likely corresponded to the loss of reachability of networks which did not have alternative backup power sources (or whose Internet providers didn't). Table size then continued to drop, though at a slightly more gradual pace. We suspect that losses during this time correspond to networks with limited backup power, which were able to stay online temporarily until those power supplies were exhausted. By 19:00 EDT, routing table sizes had reached their low point, a full 2500 networks fewer than the current baseline size. Any network that was to be affected by the outage was offline by that time. By midnight EDT, route totals were back on the rise again, though very slowly. They grew gradually at a constant rate until leveling off at their original sizes again around midnight Friday night.
Timeline of Events
Though a few networks did go offline earlier, the first burst of offline networks due to the power outage was indicated by BGP at about 16:11 EDT. A progressive increase, as shown by the sequence below, took place over the next few hours. The highest concentration of outages occured in the greater New York City area, as indicated by the inset. A representative 5% sample was used for these snapshots of U.S.-based outages, which were taken from a longer animation. Note that network outages are not limited to the northeast--there are always some network outages going on somewhere in the U.S. (and the world) at any given time. Each network outage blip corresponds to an IP prefix (such as 192.168.0.0/16), each of which typically correspond to anywhere from 50 to 16k or more hosts.
 |
1 Thu Aug 14 2003 |
 |
5 Thu Aug 14 2003 |
 |
2
Thu Aug 14 2003 |
 |
6 Thu Aug 14 2003 |
 | 3 Thu Aug 14 2003 |
 | 7 Thu Aug 14 2003 |
 |
4
Thu Aug 14 2003 |
 | 8 Thu Aug 14 2003 |
A geographical outage animation is also available
Route Withdrawal Timeseries
The blackout resulted in an Internet-wide burst of BGP withdrawal messages. The first plot below, though complicated, contains a lot of information about the time process of the withdrawal of routes to various networks. Each row in the plot describes a timeseries of the number of withdrawn routes sent from one of dozens of routers located in the U.S., Europe, Africa, Australia, and Asia. On the horizontal axis is time, and the vertical axis represents the number of withdrawn routes every 3600 seconds.
The highest spikes are transient, uncorrelated events affecting single routers. The alignment of high withdrawal peaks sent from every router around the number-3 blue label (16:11 EDT) is the surge in withdrawals due to the blackout. The second plot below shows a zoom-in on the onset of the blackout, where the timeseries shows the number of withdrawn routes at a finer resolution (every 300 seconds).
Note that the baseline level of withdrawals both before and after the blackout event is quite high and very noisy. Many routers withdraw on the order of 1000 routes per hour even before the blackout. Throughout the period starting August 11th, MS Blaster scanning activity caused higher than normal levels of network outages due to maintenance, edge router failures, and other collateral damage.
The first 24 hours of the plot show the relatively calm baseline withdrawal behavior that existed before the worm. The hourly withdrawal level rises and becomes consistently more noisy starting on Monday the 11th.