IXPs and Disaster Recovery

The telecom world has once again dealt with the challenges of service disruption following cut or damaged submarine fiber optic cables.  Over the past two weeks separate incidents in the Med caused by, as of yet undetermined causes, disrupted large volumes of both telecom and Internet traffic serving many countries around the Indian Ocean.

Most large telecom companies and carriers using large submarine capacity have redundant routes built into their networks.  Thus if one of the major trunking links goes out of service, telecom traffic will immediately or dynamically reroute through another cable system.  In some cases this is a much longer route, but service is usually not affected other than potentially increased latency on the circuits.

Internet traffic is a bit different.  While Internet traffic may account for a very large percentage of global telecom capacity, the dynamic of Internet peering requires many more one-to-one relationships among other Internet network and service providers.  Why?  Because there are more Internet-enabled and Internet networks than traditional telecom carriers.  An Internet network manager has to manage many more inter-company relationships than a traditional carrier network manager.

The need for redundant engineering within Internet networks is the same as any other telecommunications system.  One tool the Internet community has available that is not shared by the standard carrier model are public and private Internet Exchange Points (IXPs).  Wikipedia defines an IXP as:

An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance.An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance. (www.wikipedia.org)

Large IXPs such as the Amsterdam Internet Exchange and the London Internet Exchange have more than 300 members.  All of those members have the option of peering with either individual networks, or large numbers of networks and Internet-enabled companies through use of route servers.  While dedicated circuits between larger networks are still common (larger networks sell Internet transit to smaller networks, and thus will not normally “peer” at an IXP), peering at IXPs is very cost effective for networks requiring both direct relationships with other networks (eliminating the need for purchasing some transit traffic).

One of the most important statements in Wikipedia’s description deals with fault tolerance.  As the Internet is a global network, and there is a need for global routing and access to every “end point” within the Internet, network administrators have a huge challenge designing effective disaster recovery plans for their networks.  For example, if you are an Indian network provider, and your traffic is primarily routed through the Med and Europe, a cable cut in the Med can have catastrophic effect on your network performance and connectivity.

If the traffic going through Europe is further directed towards North America, then your network will have even more serious performance problem – if you are able to get any connectivity.

So, when the cable cut occurs, the network administrator may have to arrange restoral circuits with many individual network service providers, content delivery networks, VoIP companies, and other networks – if the Indian company does not have a physical presence at one or more large IXPs.  At the large IXP, such as Amsterdam, London, Any2, or other large IXP, the network administrator has the option of quickly re-routing traffic through another cable directed at that large IXP, with the potential of many more peering relationships than if they were single threaded through a large trnsit carrier.  This is particularly true of those carriers which may not have adequate restoral capacity or restoral planning for those circuits.

Our own IXP, the Any2 Exchange, normally sees large spikes of traffic during major cable disruptions.  While there is a lot of primary traffic being sent through the IXP, it is also used by many carriers as a backup or tertiary disaster recovery point for Asian, Mid-East, Russian, and European networks to bypass disasters or cable disruptions.  North America is unique in that both major Atlantic/Mediterranean and Asian cable systems connect to north America, allowing North America to be used as an alternate route for traffic transiting any continent.

While the American Internet community has lagged the Europeans in using IXPs (for a variety of reasons – to be continued in a later blog entry), both Asian and European networks are quickly increasing their presence at American IXPs.  This is certainly paying off, as the European and Mid-East networks were able to quickly restore nearly all Interent traffic following our recent cable disruptions – much of that restoral going through the IXPs.

Flattening The American Internet

NOTE:  I originally published this article in 2007

Accessing information and interactive resources available around the globe via the Internet is a pretty simple task. In a carefree Internet world, the dynamics of connecting to resources are transparent, and we expect resources we want to access are available through our local Internet service provider. Technical details of connecting to Internet resources are an abstract concept for most, and whatever mechanics happen behind the scenes are not relevant to our everyday use of the network.

Because the Internet is made up of a complex matrix of physical, business and international relationships, how these systems interact and collaborate is actually very important to the end user, as well as to those providing Internet services and content. Of the greatest concern impacting online resources from eBay to the Bank of America is the potential financial pressure brought on by the largest Tier 1 networks. As the only networks in the world having global Internet visibility, these few companies, including AT&T, Sprint, Verizon, Level 3, and Cable and Wireless, facilitate access to the global Internet – a function which people and companies worldwide depend on to ensure small networks and content providers are available through their local service providers.

The Tier 1 world was born at the demise of NSFNet (National Science Foundation Network). In the early days of Internet development, the NSF supported development of a large publicaly funded academic and research network throughout the United States, and connecting many foreign academic networks to the US as a hub through the International Connections Manager (ICM Network). As commercial Internet development grew in the early 1990s, the NSF realized it was time to back away from publicaly funding the “Internet” and grant contracts to large US carriers to take over responsibility for the former US Domestic backbone and ICM portions of the NSFNet.

Small Internet exchange points (IXPs) were also funded, allowing the large networks taking over NSFNet assets, as well as their own commercial Internets to connect and share Internet traffic. Those network access points (NAPs) were also contracted to the large US carriers, who managed policies for US and International network exchange. The large US carriers ultimately had control of the networks, and were the original Tier 1 Internet providers.

Roadblocks in the Internet Community

Debates around net neutrality highlight some underlying issues. The goal of net neutrality is to preserve the open and interconnected nature of the public Internet. But whether the largest networks use their control to hinder growth and innovation within the Internet-connect business community or impede free access to Internet-connected content sources, they have the power and control which could present challenges to an open Internet environment.

A Tier 1 network, for example, has the power to charge a major content delivery network (CDN) a premium to access its network. This is because the CDN may deliver a very large amount of content traffic into a network, and the Tier 1 network believes they should receive additional compensation to fund additional capacity needed to support content distribution. This premium may be more money than the CDN is willing or able to pay. In turn, if the CDN doesn’t comply, the Tier 1 can ultimately refuse the CDN access to its network and cut its consumers access to the CDN’s content. This applies whether consumers access the Tier 1 directly or if the Tier 1 is the middle-network between consumers and their Tier 2 or 3 networks.

A voice over Internet Protocol Company underscores another potential conflict of interest. Let’s say you’re a consumer of a Tier 1 network that’s also a telephone company and you want to use a VoIP company, such as Vonage. But the Tier 1 doesn’t want the VoIP company to compete with its network and would rather that you use its own telephone product, so the Tier 1 may prevent you from using your VoIP company. In other words, a Tier 1, in developing its own commercial VoIP product, can prevent non-owned VoIP traffic from passing through its network.

While Tier 1 networks hold value for much of the Internet world, they also impose many political and financial barriers on smaller networks, content delivery networks, emerging VoIP companies, online gaming businesses, B2B and online commerce, and entertainment web sites. It is evident that Internet Service Providers (ISPs), CDNs, VoIPs, and many others need an alternative method of communicating with each other – one providing tools to redesign how relationships and interconnections bond the US Internet content and access communities.

Breaking Down Barriers

One objective in building efficiency and the performance needed to deliver content resources to end users is to flatten existing Internet architecture. Whenever possible, you eliminate the Tier 1 Internet networks from participating in the delivery of content resources to end users.

How do we accomplish this task? One option is through development and use of commercial Internet Exchange Points (IXPs), a location where many Internet-enabled networks and content resources meet to interconnect with each other as peers.

According to Wikipedia, an IXP is a physical infrastructure that allows different Internet Service Providers to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allows traffic to be exchanged without cost. An IXP is essentially a physical switch in a carrier hotel or data center with the capacity to connect thousands of networks together, whether content providers or network providers.

Today at the Any2 Exchange, an IXP built within One Wilshire, on a single switch 125 different networks interconnect and are freely able to pass traffic amongst each other without having to go to a Tier 1 for routing. Members pay a small annual fee to the Any2 Exchange for the one-time connection and then benefit from the “peering” relationships among members of the Internet exchange.

Akamai, for example, a large content distribution network company that delivers streaming media and movies on demand, can connect to American Internet Services, a Tier 3 ISP in San Diego, Calif., through a local or regional Internet exchange point such as the Any2 Exchange, the Palo Alto Internet Exchange (PAIX), or other large exchange points operated by data centers and carrier hotels.

When an American Internet Services user wants to watch a movie that’s available on Akamai’s content delivery network, the data is passed directly from Akamai to American Internet Services – and subsequently to the end user – without transiting any other network. Not only has the goal of being less reliant on a Tier 1 been achieved, but the performance is superior because there are no “hops” between the CSP and ISP. Anytime you’re able to cut out the transit network, you increase the end user experience. Plus, it’s more economical, as in moist cases the CDN and ISP have no financial settlement for data exchanged.

The European IXP model, which is more mature and robust than the US model, highlights the important function of IXPs and how an exchange point alone can help influence the net neutrality debate. In Europe, Internet service providers and content delivery networks look to the IXP as their first connection point and if the IXP doesn’t have what they’re looking for, only then will they go to a Tier 1 or large Tier 2. Americans on the other hand, partially due to geographic size

Overall European IXP traffic grew at a rate of 11.05%, compared to America’s rate of 7.44%, according to the European Internet Exchange Association in August 2007. This can be attributed in part to greater member density in Europe – the London Internet Exchange/LINX has more than 275 members – where the larger the addressable community, the larger the traffic exchanged and the more the members want to get involved. After all, network effect (exponential growth of a community) and the “Law of Plentitude” (the idea that once an addressable or social community reaches participation by 15% or greater of a total community, it becomes a risk to not participate in the emerging community) motivate European companies to use IXPs. Additionally, Europeans generally have lower entry costs for participation, giving companies every reason why to participate in the IXP-enabled peering community. If one were to buy access to 275 networks through a Tier 1, the cost would be astronomical, but through a single connection to LINX, one can access 275 networks for a nominal fee. This is why European companies rely on IXPs 60% of the time, and only look to Tier 1 or 2 networks 40% of the time.

In contrast, American ISPs normally look to larger wholesale and Internet transit providers first and then consider reducing their operational expenses via an IXP as a second priority. American ISPs companies use IXPs at a more meager 15% rate, looking to larger wholesale and transit Tier 1 or Tier 2 networks 85% of the time. Still, recent American IXP traffic growth does exceed other regions, such as Japan (+5.85% in August) and the rest of Asia (+4.3% in August), which we believe is a result of increased price pressure on the American IXP industry. Newer IXPs, such as the Any2 Exchange, have lowered entry costs significantly, forcing others to follow suit and encouraging more networks to participate. As the cost of entry to IXPs continues to fall, participation in IXPs will become more common and attractive to all access and CDN networks.

What can we learn from the European model? Participation in an IXP can increase performance, lower operational costs and expenses, as well as bring an additional layer of redundancy and disaster recovery capacity to even the smallest networks. But most important, companies’ independence from Tier 1s through the collective bargaining of the exchange points puts them in a stronger position to deal with large networks than our position allows for in the US, where the vast majority of people have their primary Internet connections through a large Tier 2 or Tier 1 network provider.

Adding to the Cause

Today’s content-rich Internet is just a prelude to the future content, media, applications and services soon to be developed and deployed. It’s no wonder that in large IXPs, such as the Amsterdam Internet Exchange (AMS-IX), there are already several content delivery networks using bundled 10Gbps ports, clearly showing end users’ insatiable demand for high bandwidth applications and services. High Definition Internet TV (IPTV), massive online interactive gaming, video on demand (VOD), and feature-rich communications (video conferencing) are just a few examples of Internet-enabled applications contributing to the heightened demand.

For American ISPs that pay anywhere from $20-to-$40/Mbps when connecting to Tier 1 and Tier 2 networks, the cost of delivering applications and services to end users who require much larger network and bandwidth resources is one of the obstacles that needs to be overcome. But without broad participation in IXPs, access networks have a difficult future, as do content providers who will find that the cost of delivery to end users becomes much more expensive if Tier 1 and Tier 2 networks increase the cost of delivering both wholesale and end user Internet traffic.

What Can the American Internet-Connected Community Do?

Whether through price increases or monopolistic practices, the largest networks are currently writing the rules for a global Internet product. They are gradually merging and acquiring competition, reinforcing their influence in wholesale and transit network share and presence. Opportunities for network peering decrease with each merger.

Carrier hotels and large data centers in the US can support positive change in the Internet peering community by creating or supporting open and low cost Internet Exchange points promoting network peering and content delivery to all networks.

Reducing barriers to entry and the cost of wholesale or transit networks will allow Internet network and content companies to focus on delivering network access and services, with the ultimate winner being end users who will enjoy a lower cost, higher performance Internet experience.

Follow

Get every new post delivered to your Inbox.

Join 185 other followers