IXPs and Disaster Recovery
December 31, 2008 Leave a comment
The telecom world has once again dealt with the challenges of service disruption following cut or damaged submarine fiber optic cables. Over the past two weeks separate incidents in the Med caused by, as of yet undetermined causes, disrupted large volumes of both telecom and Internet traffic serving many countries around the Indian Ocean.
Most large telecom companies and carriers using large submarine capacity have redundant routes built into their networks. Thus if one of the major trunking links goes out of service, telecom traffic will immediately or dynamically reroute through another cable system. In some cases this is a much longer route, but service is usually not affected other than potentially increased latency on the circuits.
Internet traffic is a bit different. While Internet traffic may account for a very large percentage of global telecom capacity, the dynamic of Internet peering requires many more one-to-one relationships among other Internet network and service providers. Why? Because there are more Internet-enabled and Internet networks than traditional telecom carriers. An Internet network manager has to manage many more inter-company relationships than a traditional carrier network manager.
The need for redundant engineering within Internet networks is the same as any other telecommunications system. One tool the Internet community has available that is not shared by the standard carrier model are public and private Internet Exchange Points (IXPs). Wikipedia defines an IXP as:
An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance.An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance. (www.wikipedia.org)
Large IXPs such as the Amsterdam Internet Exchange and the London Internet Exchange have more than 300 members. All of those members have the option of peering with either individual networks, or large numbers of networks and Internet-enabled companies through use of route servers. While dedicated circuits between larger networks are still common (larger networks sell Internet transit to smaller networks, and thus will not normally “peer” at an IXP), peering at IXPs is very cost effective for networks requiring both direct relationships with other networks (eliminating the need for purchasing some transit traffic).
One of the most important statements in Wikipedia’s description deals with fault tolerance. As the Internet is a global network, and there is a need for global routing and access to every “end point” within the Internet, network administrators have a huge challenge designing effective disaster recovery plans for their networks. For example, if you are an Indian network provider, and your traffic is primarily routed through the Med and Europe, a cable cut in the Med can have catastrophic effect on your network performance and connectivity.
If the traffic going through Europe is further directed towards North America, then your network will have even more serious performance problem – if you are able to get any connectivity.
So, when the cable cut occurs, the network administrator may have to arrange restoral circuits with many individual network service providers, content delivery networks, VoIP companies, and other networks – if the Indian company does not have a physical presence at one or more large IXPs. At the large IXP, such as Amsterdam, London, Any2, or other large IXP, the network administrator has the option of quickly re-routing traffic through another cable directed at that large IXP, with the potential of many more peering relationships than if they were single threaded through a large trnsit carrier. This is particularly true of those carriers which may not have adequate restoral capacity or restoral planning for those circuits.
Our own IXP, the Any2 Exchange, normally sees large spikes of traffic during major cable disruptions. While there is a lot of primary traffic being sent through the IXP, it is also used by many carriers as a backup or tertiary disaster recovery point for Asian, Mid-East, Russian, and European networks to bypass disasters or cable disruptions. North America is unique in that both major Atlantic/Mediterranean and Asian cable systems connect to north America, allowing North America to be used as an alternate route for traffic transiting any continent.
While the American Internet community has lagged the Europeans in using IXPs (for a variety of reasons – to be continued in a later blog entry), both Asian and European networks are quickly increasing their presence at American IXPs. This is certainly paying off, as the European and Mid-East networks were able to quickly restore nearly all Interent traffic following our recent cable disruptions – much of that restoral going through the IXPs.