| Packet latency is a big issue in Internet-based applications (i.e. the stuff in the cloud). In conducting analysis on Internet infrastructure over the years, we have seen many patterns of connectivity. One such pattern that can wreak havoc on latency is "hair-pinning", a phenomenon where traffic takes an unnecessarily long physical path between two points on the Internet due to suboptimal routing. The increased distance results in increased latency, and the "lag" or "sluggishness" that users experience as a result can hinder latency-sensitive online applications whether they are financial trading applications or MS SharePoint. |  |
Geographic Locality does not translate into Internet Locality
A couple of weeks ago (September 23rd to be exact), I noticed a small routing outage (150 globally routed prefixes) in Sakhalinskaya Oblast, the most Eastern part of Russia. We observe these types of outages every day — the Internet is a big messy place behind the scenes.
The obscurity of this location piqued my curiosity and I
looked into how this area gets Internet connectivity — almost exclusively
through Russian provider, TransTelecom
(AS20485). One can see that
this province connects to Japan via Hokkaido-Sakhalin
Cable System (HSCS) built by NTT in 2008 to provide a shorter
path from Japan to Europe over Russian soil. However, despite the proximity
to Japan, a traceroute from NTT's Tokyo Looking Glass to an IP
address in Sakhalinskaya looks like this:
Query Results:
|
 |
Which roughly follows a path like this:
Circuitous routes like this are called "hair-pinning" in the business - this traceroute travels from Japan to London just so it can make its way back to a location just off the coast of Japan. On the Internet, geographic locality does not translate into Internet locality, and the distinction can result in unexpected latency, frustrating users of real-time applications.
Latency in the Middle East
Last week I was in the fascinating city of Muscat, Oman for MENOG 9 giving a talk about latencies we have measured into each Middle Eastern country. While regional traffic in the Middle East often suffers from hair-pinning via European Internet exchanges, latencies to and from Europe appear to be steadily improving.
In the past, when Renesys has analyzed Middle East connectivity it was with regard to complete outages such as the cable cuts in 2008. This year, connectivity into the Middle East has been much more stable (obvious exceptions notwithstanding) and the fact that regional providers can focus energy on reducing latency is a good sign for the region. For example, OmanTel's strategy (AS8529) to reduce latency to Europe by gaining presence at AMSIX (AS1200) is paying dividends in 2011 by reducing median latency by 41ms or 20% in 2011. This is illustrated in the chart below, which is a plot of overall latencies from London to Oman in 2011 colored by OmanTel's upstream provider or exchange as observed by traceroute. Higher latency bands (PCCW, AS3491, in Grey and LINX, AS5459, in blue) disappear midway through the year and are replaced by a lower latency path through AMSIX (green).
Until a regional Internet Exchange emerges for the Middle East, the best regional providers can hope for is to reduce latency on the path of the hair-pinning through Europe for regional traffic!
Impact of IMEWE Activation in Lebanon
The most recent (and intriguing) development in the area of latencies to the Middle East is the reported activation of the India-Middle East-Western Europe (IMEWE) cable system at Tripoli, Lebanon. Despite speculation that the activation would be delayed, there is evidence that the cable is now being actively used. Certainly something has happened in Lebanon to reduce latencies in recent days.
Thus far, development of the Internet market in Lebanon has been stifled by the country's extensive dependence on satellite Internet service — the ultimate form of hair-pinning. In the visualization below, latencies into Lebanon separate into two modes based on delivery medium: satellite (primarily via SatGate, AS30710) and submarine (primarily via Level3, AS3356). Further examination of the two latency modes clearly indicates asymmetric routing. First, latency measurements should be at least 480ms for symmetric round-trip delay across a satellite connection due to the speed of light and altitude of orbit, but we observe latencies clustering around 400ms. Second, the dramatic 70ms drop in median submarine latencies starting September 22 is shadowed by a corresponding effect in the latencies inbound through the satellite provider. Packets are arriving inbound via satellite but returning via submarine cable. Note also the further 15ms reduction on September 29th, which has held through today. (Note: Dates referenced appear at small ticks along x-axis)
Additionally, we see a reconfiguration on August 24th visible as a slight change in this latency distribution. Here there was an address change in the IP-IP hop in Level3's network (AS3356) that experiences the vast majority of the overall latency to Lebanon. It is a reduction of the latency across this hop that contributes the most to the downward shift in latencies to Lebanon in recent weeks.
In the coming weeks we expect to see a dramatic shift in transit as Lebanese providers move away from expensive and high-latency satellite service to IMEWE-based service. This is likely to resemble the collapsing of satellite Internet markets which we have observed occurring in African countries almost immediately after new cable landings are established nearby. For example, the chart below is the relative mix of Internet transit providers for Lebanese provider TerraNet (AS39010) in 2011. In this stacked chart, a taller band of color indicates greater amount of Internet transit through that provider for a given point in time. Except for Liban (AS42020), the vast majority of their transit is satellite.
This chart is almost certain to look different in a couple of months if the new cable is here to stay. We'll update this story as it develops. With any luck, latencies to Lebanon may begin to resemble those of other countries of the Eastern Mediterranean and Lebanon's inexplicable dependence on satellite will be a thing of the past.
What to do about hair-pinning?
There are many causes of latency in network performance, such as congestion and router overutilization, but hair-pinning is often avoidable. Businesses need Internet intelligence to know their high latency is a result of hair-pinning and then press their providers to do something about it! This is particularly important for mobile providers because it adds another potentially severe performance penalty to the Internet service they provide.
Regarding Russia, Russia is emerging as the "hot new fibre" transit route for global carriers needing LOW LATENCY Europe to East Asia connectivity (specifically China/Japan)...
The players on this low latency Transtelecom and their partner in Japan which is NTT.
Rostelecom and their partner in Japan which is KDDI.
In China they both partner with China Telecom & Unicom all the way to HKG where other Asian carriers can pick-up the networks. This is in addition to the Tokyo access points of both carriers. You can technically do Shanghai/Beijing but that's more regulatory stuff (Mainland China)...
Regarding Europe they have A_LOT of connections from the Russian border West-wards into EVERY COUNTRY in Europe!
They also provide transit into Central Asia/Middle East and are gearing up for land (terrestrial transit) into South Asia it is rumored (maybe with TATA!?) if they can get a decent path through China (you know the big problem there is the military conflict that is being solved by the US...).
Here are some documents you guys may want to review...
The Russian carriers market this as TEA (Transit Europe Asia) and Eurasiahighway...
TTK (Transtelecom's) Eurasiahighway:
http://www.ttk.ru/rus/59897/59900/61841/document61856.phtml
I heard their selling 10G wavelengths from the Equinix'es of Western Europe all the way to the Equinix'es of East Asia (Hong Kong and Tokyo mainly, Shanghai/Beijing on special request...).
They are now marketing new 40G DWDM and probably soon 100G DWDM from the borders of Western Europe all the way to the borders of East Asia...
Rostelecom:
http://old.rt.ru/en/serv-operators/tea/
Rostelecom is interesting in that they recently with ALU started doing network 100G DWDM modernization from the West to the East.
http://www.telecompaper.com/news/rostelecom-picks-nsn-alcatel-lucent-as-dwdm-suppliers
RT is really interesting in that they with ALU will deploy 100G DWDM on their North South transit route from Europe to the Middle East with 700Gbps of aggregate capacity.
On the West-East route they plan to start with 200Gbps (mainly again low latency Europe-Asia) and scale to 8Tbps from Western Europe to East Asia.
(ALU gets North South transit project, NSN gets West-East transit project).
Regarding land based Asia-Europe land-bridge transit...
The Europeans, Russians & Chinese recently opened a 4th DWDM fiber transit route from Western Europe to East Asia crossing through Eastern Europe, Central Asia (Kazakhstan) and China with the following:
They say they are wanting the ABSOLUTE lowest latency from East Asia to Europe for financial trading applications and that sort (NTT sold a few for the 194ms Tokyo-London route).
For China-Europe the maps are here:
http://www.hk.chinaunicom.com/networkcoverage.html
Route #1:
http://www.hk.chinaunicom.com/images/map_ERMC.jpg
Route #2 (with TTK):
http://www.hk.chinaunicom.com/images/map_ERA.jpg
Route #3:
http://www.hk.chinaunicom.com/images/map_TEA.jpg
Route #4:
http://www.hk.chinaunicom.com/images/map_EKA.jpg
Note that the last route has the ABSOLUTE shortest route and is the method for the East Asian carriers RECOVERY route if the earthquake does damage on the trans-Pacific cables again.
It is their method of taking traffic to the Americas via Europe via the Atlantic again doing a world loop.
Remember the earthquake off Japan!?
Well the Russian carriers and the Japanese/Chinese ones quickly made a CRAP_LOAD of profit $$$ immediately after that.
NTT/KDDI and China Telecom/Unicom started immediately routing TIME CRITICAL SS7 voice traffic and some corporate IPLC data traffic from Japan/China and their downstream customers out of Hong Kong/Beijing and Tokyo across Eurasia (Russia)/Central Asia through Russia and into Eastern and finally Western Europe.
Their POP'es are inter-connected with the American carriers now in Europe in addition to East Asia and the US West Coast.
(Yes SS7 voice traffic Hong Kong->Beijing->Moscow->Stockholm->trans-Atlantic->New York POP!).
What a way of routing around the world!
Actually the first case of Russian carriers making a_LOT of profit was the 2006 earth-quake when the first restoration was HGC (Hutchinson Global Communications) out of Hong Kong getting internet access with VERY LOW LATENCY (everyone else was on trans-Indian and 300+ms!) and everyone was wondering how they could do it consdering the sea cables were cut and people were realizing traffic to their London POP was suddenly 200ms which is actually faster than the usual trans-Oceanic (cheaper, the Russian route is more EXPENSIVE!)...
trans-Russian from East Asia to Western Europe is the answer (their trans-Russian capacity restoration route gained a-lot of multi-national corporate customers after that because they also inter-connect trans-Altantic in Western Europe!).