So Mr Sihnaoui sent out an SMS to all of Lebanon just to inform them that "internet is back to normal" but neither are we back on I-ME-WE, nor is the bandwidth / latency REMOTELY what it used to be. It's interesting to note that TWO years ago the Ministry of Telecommunication signed an agreement with Cyprus to have bigger bandwidth rights of the newly upgraded CADMOS. CADMOS is now capable of handling 40 Gbps according to several websites and Lebanon is entitled to 38% of that or roughly 15 Gbps; so, it would be technically incorrect for Mr Sihnaoui to state that we got 10 Gbps for free! We have an agreement! Besides if that's how much we have, why is my connection being routed on a satellite link?
Lebanon Loses Lone Link
Prior to the recent activation of Internet service to Lebanon via the IMEWE submarine cable, Internet service in Lebanon was labeled the “world’s slowest” due to its dependence on a combination of antiquated submarine cables built in the mid-1990′s and high-latency satellite service. However, as high-speed Internet service via IMEWE expanded in recent months, today’s outage reveals Lebanon’s new dependence on this lone modern connection to the outside world.
For almost three hours today, Lebanon experienced a near complete nationwide Internet outage. Between 16:13 and 18:59 UTC, we observed as many as 842 of the approximately 900 routed prefixes in Lebanon withdrawn from the global routing table, as illustrated in the graphic on the right. During this period of time, we saw almost every routed prefix downstream of incumbent Liban (AS42020) withdrawn. At 17:45 UTC, we saw these networks restored only to be withdrawn once again minutes later.
In an interview immediately following the restoration of Internet service, Lebanese Minister of Telecommunication, Nicolas Sehnaoui, stated that the outage was a result of unannounced maintenance on the IMEWE cable serving Lebanon and has threatened to file a complaint with the Attorney General of Lebanon against top officials at OGERO, Liban Telecom’s fixed line arm.
The two graphs below illustrate traceroute measurements into Lebanon through the two largest International internet providers serving the Lebanese market. Like the routing outage graphic above, we observe a near—total cessation of completing traceroute measurements except for a brief restoration of service at 17:45 UTC. The red in the graph on the right is satellite provider, SatGate, (AS30721) coming online to provide backup service.
Just as in the routing data, we can observe a brief restoration of trace-based reachability at around 17:45 UTC before service is lost again. If it does turn out that this outage was caused by maintenance on IMEWE, then it is likely that the networks which survived the outage connected to the Internet using one of the other older cable systems serving Lebanon.
What does it all mean? Lebanon has struggled for years to find the political and technical keys to sustainable Internet growth and stability, and the landing (and activation) of the IMEWE cable has been a major step forward. But today’s outage demonstrates that building provider-level and transport-level diversity are equally important strategic goals. Following worldwide best practices, Lebanon’s banks, businesses, universities, and government offices need to study their connections to the Internet and, if necessary, request that they follow multiple independent paths. A single provider’s temporary difficulties shouldn’t switch off the entire nation.
Yes I agree Michael; I would attribute Monday's incident to a maintenance carried out by Ogero on it's distribution point rather than an IMEWE maintenance. As for today, we are still without our IMEWE. According to several news sources this issue will take days to fix. You can read more at the National News Agency but since it is hosted in Lebanon, I'll save you the pain of waiting: " NNA - 5/7/2012 - Telecommunications Ministry announced on Thursday that the Internet blackout, expected to last for days, is due to a malfunction of IMEWE, a submarine fiber-optic cable linking Lebanon to international Internet capacities, located 50 kilometers in the sea off Alexandria city in Egypt. The Ministry is following up on the issue with both Egypt and Cyprus, in collaboration with OGERO. The Crisis Management Cell, which Telecoms Minister Nicolas Sehnaoui has formed, is carrying on contacts and meetings in a bid to provide temporary solutions. R.A.H." Source: http://www.nna-leb.gov.lb/newsDetailE.aspx?id=421041
Hi Michael, this is an excellent observation. We noticed this as well. During the outages this week, Ogero has maintained limited service to Level3 over a much higher latency link. I presume this is KADMUS. What that particular graph doesn't show clearly is the spike in latencies that occurs at the time of the incident.
Here we are again facing internet outage on the IMEWE link. I am able to make this posting because I'm currently connected through VPN to work and utilizing their IDM internet connection. Apparently IDM is routing their "SLA" customers through an old submarine cable - KADMUS. This cable is currently congested virtually beyond usability; Packet loss exceeding 50%. According to Minister of Telecommunication Sihnaoui, today's outage, unlike last time is a result of a technical problem affecting IMEWE cable at the segment between Egypt and Marseille, France. Would be interesting to see if you have any data supporting Minister Sehnaoui's claim. Judging by the fact that they have started giving access to this very old and already congested KADMUS, I'd assume this issue is going to take more than 24 hours to resolve.
In the level3 downstream graph, it appears that the ogero connections are not affected, that's weird. if it was a IMEWE maintenance, shouldn't ogero be disconnected as well? maybe ogero has a backup connection, or more probably the problem was at the level of the ogreo distribution to other ISPs
Excellent article Doug. You hit the nail right on the head with IMEWE being virtually a single point of failure for Lebanon's internet access. Obviously ISPs serving cellular providers such as MTC and Alfa were also affected since this pipeline feeds the whole country of Lebanon. This resulted in loss of Blackberry service and other internet-dependent services that are common among Lebanese users such as whats app and so on. It is a true shame that up until this day, ISPs don't offer an SLA by default to its corporate customers; in extremely rare cases were the customer makes it a prerequisite to signing a service contract with an ISP, the ISP will provide you with a 95.0% availability as a maximum while ISPs around the world have begun competing on the 99.995% mark! Another rare event(we're talking extremely rare here, close to an eclipse sighting) is to see a corporate customer requesting and actually getting an ISP to provide them with two links on two separate submarine cables. Cheers!
Just sidenote - SatGate LLC is not Lebanese ISP, it is just Satellite ISP, and many ISP in Lebanon don't have their AS and netnum's, so they used Satgate IP range.