Thanks for clarifying. Hoped to see that detailed data inside of your report rather than in the comments later on though. ;)
Two Strikes For the I-root
Here we go again. In March we wrote a blog entitled Accidentally Importing Censorship which described how incorrect DNS answers were returned in response to certain queries to the I-root. The problem was tracked down to a single instance of the I-root located in China. Queries to this server for domains blocked in China, such as Facebook, would return seemingly arbitrary answers. As we noted, countries, and even companies, can impose their own standards on the Internet and block anything they want. This story was only noteworthy because those blocks (via bad DNS answers) became visible outside of China. Well, guess what? We are once again seeing the Beijing I-root from outside of China.
Let’s start with a few disclaimers and some background. First and foremost, the sky is not falling. Getting the wrong DNS answer, even when querying the Chinese I-root instance is an extremely rare event. Go back and read our earlier blog to see the exact alignment of the stars that would be necessary. The fact that it is so rare is what kept the problem from being detected for weeks. However, as we noted in that earlier blog, given the broad swath of the Internet potentially querying the Chinese I-root instance, someone was bound to stumble on a bad DNS answer and, as a result, not be able to friend their pals. This is exactly what happened and is what brought the problem to light.
Second, the fine folks at Netnod, who provide the exceptional and free I-root service, vigorously defended their services in China, asserting they provide the same DNS answers regardless of location. We have no reason to think otherwise.
Third, it’s quite easy to see incorrect answers from DNS servers in China yourself, whether or not you happen to live there. This has nothing to do with any of the root name servers. Just pick your favorite DNS server based in China and ask it about Facebook. Here is an example of repeated queries from the Linux command line from a US-based machine to a China Telecom DNS server.
dig @dns1.chinatelecom.com.cn. www.facebook.com. ... www.facebook.com. 11556 IN A 184.108.40.206 www.facebook.com. 24055 IN A 220.127.116.11 www.facebook.com. 38730 IN A 18.104.22.168
None of these IP addresses has anything to do with Facebook. In fact, addresses starting with 37 haven’t even been allocated by IANA as of this writing.
Of course, if you don’t live in China, you probably don’t use a Chinese DNS server directly. The problem is that we all use the root name servers and they are spread throughout the world. Thanks to the vagaries of Internet routing, you may end up querying any of them, regardless of where you live and where they are hosted. Thus, if you live outside of China and just happen to query a root name server hosted in China, your queries will pass through what is known as the The Great Firewall, and hence will be subject to any restrictions it imposes.
While doing some research for next week’s NANOG meeting in San Francisco, we revisited the time line for the March I-root announcements from China and couldn’t help but notice the problem resurfacing on June 3rd. The I-root resolves to 22.214.171.124, which is announced by AS 29216 (which is dedicated to the I-root) as both 126.96.36.199/23 and 188.8.131.52/24. From there, these prefixes travel via Netnod’s AS 8674 and then onto the general Internet. Since Netnod anycasts these prefixes from dozens of locations around the world, we expect to see them via any number of BGP adjacencies to AS 8674 and, in fact, we do. Around 80 different ASes adjacent to Netnod’s AS 8674 see the two I-root prefixes and, in turn, propagate them onward.
What we do not expect to see are mainland Chinese ASes adjacent to AS 8674 propagating these prefixes outside of China. This is what we did see in March 2010 and it implies Internet users outside of China could be directed to the I-root instance inside of China. Unfortunately, this problem has returned. We see AS 8674 passing just 184.108.40.206/24 off to AS 24151 and then AS 7497, both of which are associated with the China Internet Network Information Center. From there, the prefix travels via Pacnet (AS 10026), formerly Asia Netcom, and PCCW (AS 3491) out to the general Internet. This started just before 10:20 UTC on June 3rd and is still ongoing as of the date of this blog.
As we noted last time, to get a bogus DNS response outside of China, you not only have to query the I-root, you have to query the Chinese instance of it. To measure potential impact, we looked at the originating country of all prefixes downstream of any provider selecting the Chinese I-root. We then computed the percentage of these relative to the total number of prefixes in the country. A graph of the top dozen from the March incident is shown below, followed by those from this current (and ongoing) incident.
Not surprisingly, most of the affected countries are in Asia, as before, but there are important differences from the last event. Russia, India and Taiwan all entered the top twelve, while Pakistan, New Zealand and Bangladesh have dropped out. The impact on the countries in both lists is roughly similar, except that US impact went up by a factor of 10. Potentially impacted US states include Florida and California, making up approximately half of the US total. In addition, Singapore increased from 73% to 96%.
Censorship is a fact of life on the Internet today. But unfortunately, given the open, trust-based nature of the network, such censorship can easily spread beyond its intended boundaries. While individuals can do little to avoid such issues, there are actions network and system administrators can take. Filtering root name server announcements with Chinese ASes on the path is one approach. Never querying the I-root is another. Such actions would guard against this particular problem, but probably not the next one — whatever it might be. Ultimately, we are all in this together. We depend on each country or organization not to inadvertently or intentionally interfere with any other. All other paths lead down a very slippery slope.
That one sees the China node from I-root outside china shouldn't be a suprise to anyone. Things like that happen old the time. For quite a while I used to see the J-node from Korea instead of the much closer one from Frankfurt or here in NL. I fully agree that DNSSEC is the proper way to find out that there has been tampered with DNS reponses.
To answer my question: There is no evidence that responses from I-root in Beijing are tampered. Editor's note: Here is one from the day after our blog posting, where www.facebook.com returns an invalid IP address from Korea Telecom. We will be presenting our complete results at NANOG49 in San Francisco. See you there! # dig @i.root-servers.net. www.facebook.com. A ; >> DiG X.X.X >> @i.root-servers.net. www.facebook.com. A ;; global options: printcmd ;; Got answer: ;; ->>HEADER
Does the F-root and J-root have the same issues? If not, how do they avoid it? Editor's note: Great question. According to http://root-servers.org/, instances of the F, I and J root servers are hosted in China. However, for F and J, we see no evidence of their routing announcements leaving China. For the I-root, such announcements are leaving China.
So, I tend to believe Kurt statement that they run validated data. So, the great firewall of china likes to molest DNS queries. This is clearly an issue --- and the i-root should be moved or turned off if that is occuring. The really, really, really, concerning part is: > None of these IP addresses has anything to do with Facebook. In fact, addresses starting with 37 haven't even been allocated by IANA as of this writing. Is the great firewall of china then using unallocated space? What does 220.127.116.11 route to in China? That's way more interesting a question.
To address some of the statements and comments above. To the best of my knowledge the i.root-servers.net instance in Beijing has always been reachable outside China. All packets that traverse the Internet are vulnerable to spoofing, modification or blockage by any intermediate party anywhere along the path. For DNS the solution to this problem is DNSSEC, and has nothing to do with any routing paths you see. Instead we should all make sure we enable DNSSEC validation. As more parts of the DNS hierarchy get's signed, the more protection we will have. This will give you the chance to notice all attempts to spoof DNS replies anywhere from a signed zone. Any form of RPKI will not protect against incorrect DNS data, only DNSSEC will. I would also like to publicly deny the claim that we have been paid or asked to modify DNS zone data by China or anyone else. Nor have we ever served any other root zone data than as we have received it from IANA. Kurt Erik Lindqvist CEO Netnod
@Gen - I've known Kurtis a long time, and I know for a fact that he would never participate in any kind of censorship-related activities. I think it's a reasonable inference that there are most probably middleboxes within the Chinese networks which are intercepting the DNS queries in question and are giving out bogus answers for purposes of censorship and/or surveillance. But the one thing I know *for a fact* is that Kurtis is not the type of person to participate in nor countenance such shenanigans within his span of control.
The lesson is: use DNSSEC. DNSSEC prevents tampering with DNS responses, even by the great firewall. ICANN will finally make a DNSSEC signed root zone available shortly and all root name servers will immediately publish it. Make use of that feature!
This is an important post that highlights both the censorship that is ongoing in China as well as the Swedish company that is being paid by the Chinese government to implement this censorship. However, the post is pretty difficult to understand for anyone who is not in the DNS world. If you can, in the future, append a summary in plain English at the beginning of these posts, they would have a great reach and impact.
Actually, you can send DNS requests to IP's in China (it doesn't matter if there's even a DNS server on the host) and get back spoofed DNS responses. Actually, you can even set the TTL so that the packet won't even reach the destination (simply enter China's IP space) and it will send back spoofed DNS responses. This post is a bit old, but still valid last time I checked a few weeks ago. http://www.nartv.org/2008/09/03/dns-and-the-gfw/
California being potentially impacted makes sense, but Florida seems odd. Shouldn't Florida be topologically far enough from China to not hit the Chinese I-root instance? Editor's Note: The cyber world doesn't map very well to the physical one. Who sees this depends more on the peers of China Telecom, the dominate in-country provider.
I think it is important to note that seeing the route of the I-root server in China reappearing in BGP does _NOT_ tell if the responses from there are being tampered with. It is perfectly plausible that Netnod operates the I-root server from out of Beijing on a global scale and not just locally. Did you see any tampered responses from that instance in Beijing? Editor's Note: See response to last comment.
I would first like to note that none from Renesys has made contact with us or tried to work with us on analyzing what Renesys claims to observe. Netnod has since some days activated the global node in Beijing again, in line with what we have communicated as our goals in the past. The node is a global node so our prefixes will propagate as far as our peers advertise them. This is true for all global anycast nodes. All our instances (including Beijing) serve exactly the root zone data, as we receive it from the IANA through the agreed distribution channels at Verisign, and in no case do we make distinctions between clients - neither based on network topology, nor on source address. We have not observed, nor received any reports on, incorrect DNS replies from any of our anycast nodes since re-enabling the site in Beijing. The report above does not seem to claim this either, nor does it provide any such data. Given the lack of such data we do not see any issue with us re-enabling the Beijing node, but we would appreciate direct reports to firstname.lastname@example.org if such are indeed seen. Kurt Erik Lindqvist CEO Netnod email@example.com Editor's Note: The blog states that the Beijing I-root instance is clearly reachable from outside of China and hence, queries to it from outside the country cannot be considered reliable since they pass through the Great Firewall. The same applies to any DNS server located in China as the Facebook example shows. As noted, this is not a Netnod issue, nor are there any significant routing differences between the two I-root incidents. If there are routes, you can get there. The lesson is to not query DNS servers operating across an infrastructure has been shown to give unreliable results.