Brazil Leak: If a tree falls in the rainforest….
There’s been quite a lot of talk this morning on NANOG and elsewhere about AS16735 (Companhia de Telecomunicacoes do Brasil Central) leaking a “full table” of everyone else’s routes. Many people wrote in, affirming that yes, some subset of their networks had been hijacked by CTBC in the middle of the night, and they saw it in a hijacking alert from BGPMon.
So we looked. It does look like CTBC advertised a nearly-full set of prefixes to two of their upstreams (174,213 routes via AS27664, and 111,231 routes via AS22548) over a period of about 5 minutes, starting at 02:00 UTC. As luck would have it, one of those upstream providers was supplying a direct stream of route updates to RIPE RIS’s rrc15 route collector in Sao Paolo.
That route collector is one of the sources of data that feed the (excellent, publically available) RIPE RIS dataset, and BGPMon is one of the free volunteer-based projects that use RIPE’s data. BGPMon doesn’t use minimum-peer thresholding before deciding to report the existence of a hijacking, so they dutifully sent out emails to all their subscribers, alerting them to this hijacking.
So, if we consider the literal truth, they were right. CTBC hijacked most of the address space on earth, at least from the perspective of their providers’ customers in Brazil. But critically, few traces of this contagion escaped the local “customer cone.” (Some peers of their upstreams who didn’t filter, like RIPE RIS, may have propagated the routes to their customers.) But nobody on earth who was not a customer or a peer of their providers actually received any of these bad routes, and nobody got confused where the real origin of these prefixes was. In other words, there was no real global impact.
In this case, the system worked: nobody upstream of CTBC believed their claims of origination, so they didn’t propagate them. The only real “problem” was all those people who got email and misinterpreted the significance of this localized leak.
The lesson here is that alarming is hard. We’ve faced the thresholding issue many times before when providing route alarming services to our customers, and there are no simple solutions. Every hijack is potentially a serious problem — how serious it is, from a business perspective, depends on the percentage and geographic distribution of your audience who are likely to be affected by a leak from a given source. In practice, most leaks tend to fall into two categories, depending on whether they propagate: well-localized to a single regional customer space, or planetwide. Fortunately, this event was one of the former.
Over time, we’ve learned that it’s not enough to believe what your peers report at face value. You have to put their reports in context and, at the very least, apply some thresholding to determine how serious the context is. If CTBC’s routes had been picked up by fifty, or a hundred, or a few hundred, of our peers, then we would have been pretty certain that a good portion of the internet was confused and the hijacking was real. In this case, it was “real” .. for very small values of “real.”
None of this should detract from the really significant benefits that BGPMon and similar services provide the internet service provider community (we provide one such service to the broader commercial world, as well). Prefix hijacking is very serious business, and the more eyes we have watching the global tables for signs of trouble, the better.