Internet Network Peering – Discussion with Peter Cohen, Industry Expert

Why does one house with a cable connection to the Internet have great performance, and the house next to them with a different Internet provider have marginal or potentially poor performance? The answer may lie in the sometimes dark art of Internet peering.

Peter Cohen

Peter Cohen stopped by to discuss the topic, and try and shed some light on peering recently at the North American Network Operator’s Group (NANOG) conference in Philadelphia. Peter has worked in the peering community for more than 12 years, with experience at some of the largest Internet Service Provider networks in the United States, as well as managing peering for Telia-Sonera, the national telecom network of Sweden.

“Peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the customers of each network.” Wikipedia

Savageau: Peter, why is peering important to Internet networks?

Cohen: Peering cuts out the middle man, allowing networks to connect directly to each other without a transit or intermediate network (such as Verizon, AT&T, or Sprint). In addition, peering helps networks to improve data throughput, reduce some operating costs, and provide some protection against high transit pricing (by Tier 1 and transit network providers).

Savageau: Do all networks peer?

Cohen: Peering is not for everybody. Smaller networks and enterprise networks generally do not have enough traffic (to make peering worth it). Also, a lot depends on a network’s location. Peering normally occurs in large markets, such as the NFL cities. A network needs to have other networks or content providers who want to peer with them. You also need to have an ASN number (autonomous system network), which is reserved for those networks who have a need (traffic volume) to peer with multiple networks and content delivery networks.

Peering is not a free connection to the rest of the Internet. When a networks peers with another network, they only receive routes available directly on that network’s, or customers of that network (if it is a larger regional network which provides Internet services to both access networks and smaller content networks). In addition to having peering relationships with one or more networks, unless a network is among the largest (which we call Tier 1 Internet networks), they will still need a “default route” for routing network traffic to the rest of the global Internet.

A default route, also known as the gateway of last resort, is the network route used by a router when no other known route exists for a given IP packet’s destination address. All the packets for destinations not known by the router’s routing table are sent to the default route. Wikipedia

Savageau: Where do networks peer?

Cohen: Networks peer in larger cities where there is a higher density of other networks and content providers. In Los Angeles the Wilshire Corridor in downtown LA has several locations, such as One Wilshire where networks can connect at either public Internet Exchange Points/IXPs, or through private network interconnections (PNIs).

Los Angeles supports several large public exchange points, including those managed by Equinix, Switch and Data – as well as the (CoreSite) Any2 Exchange. Other cities such as New York, San Francisco, and Miami are also major points of peering in the Internet community.

Savageau: How is peering done differently in the United States, versus Europe or ASIA?

Cohen: A lot of the US peering model was developed early in the development of public Internet. There were very few commercial networks present in the Internet, and those networks peered with each other. As the Internet of the 1980s and early 1990s was “hubbed” in the United States, those early American networks were built from the beginning with global views of Internet routing – thus becoming the first Tier 1 networks.

Peter Cohen Tablet Engineering

The Europeans and Asians were a little later to the game. Many of the early non-US networks were operated by national carriers with very little reason to peer with or support competitive networks within their countries (no reason to make your competition stronger and more competitive in your national market). Thus the national carriers only peered at points within the United States, and since those networks did not have global Internet routing visibility, they had to buy transit from the American Tier 1 Internets.

Even today, in some countries national carriers will not peer within their own market, but will peer with competitors at peering points and exchanges outside of their home country.

On the other hand, the European networking community did learn from the American experience, and today the largest pubic exchange points exist outside of the United States, allowing smaller networks and content companies to gain all the advantages of local peering, without the need to “hairpin” their traffic through a Tier 1.

Savageau: Is the current model of peering in the US good enough? Change needed?

Cohen: Traditionally networks which peer come in two types. Those who only peer with private network interconnections/PNIs, and those who peer at public exchanges, and also support PNIs. PNIs are most useful when traffic on a public exchange hits a certain threshold, and then it makes more sense for the networks to interconnect with much higher capacity individual circuits.

A new model emerging in the US is with remote exchange point connections (Remote IXP Access). Remote IXP access allows a network geographically separated from a major public peering point to lease a long distance circuit, usually around a Gigabit, and “test the waters” by connecting to the IXP. If the peering point effectively serves the needs of the remote IXP access network, then they may at some point establish a direct point of presence the facility supporting the IXP.

The only real drawback of remote IXP access is that without a physical router present at the data center or carrier hotel housing the IXP, they are limited to the single exchange point connection, and cannot establish a PNI with another network at the facility.

Savageau: Final question. Most difficult topic for last! Give me your feelings on the topic of net neutrality, how it is currently affecting the Internet community, and how you think it will evolve.

Cohen: Many of the current policies in effect between networks and content providers are purely based on internal politics. That’s why you may go into a neighborhood and find an Internet Service Provider giving outstanding performance to one customer, and across the street you will get poor performance. Most local Internet providers are called MSOs, or Multi-Service Operators. MSOs include cable TV companies, phone companies, and satellite TV companies.

One MSO, such as AT&T, may determine the content providers should pay a higher rate to subsidize the cost of network build outs. A cable company MSO may determine that peering with content networks is the best policy.

There is no regulation at this level, and the ultimate victims are end users in the home. End users who believe they are paying a flat rate to their MSO to receive the best possible performance on their home Internet connection – but at the end of the day are receiving service performance based in a large part n the personalities of persons making decisions on peering models, not necessarily on business rules or objectives.

Savageau: Peter, keep looking out for us

============

Peter Cohen has been involved in the Internet since the 1990’s starting out at CAIS Internet.  He has held a wide range of positions and has worked extensively with peering, transit, interconnections and network design/colocation for ISPs, Carriers and other Internet related companies around the world.  His work and travel have brought him to more than 35 countries and 5 continents.  He is a frequent speaker at NANOG, Ripe, and other industry events worldwide.

Presently Peter works at Switch & Data where he facilitates interconnections among customers and the PAIX product.  He lives with his wife and two children in McLean, Virginia.  He enjoys cooking, reading, golf and tennis in his spare time.

Any2, the Law of Plentitude, and Chaos Theory

My company, CRG West, operates one of the largest Internet Exchange points (IXPs) in the United States called the Any2 Exchange. This article explores the concept behind Any2, focusing on development of communities within the telecom and Internet industry. While we will concentrate on the Any2 Exchange, the idea behind Any2 could just as easily be applied to any other public or commercial IXP.

When we initially began the Any2 project, the intent was to create an environment where One Wilshire tenants could reduce some operational expenses through creation of a “utility” Internet exchange point, almost considered an extension to the 4th floor meet me room.  While large networks were more than welcome, large networks generally are not interested in peering at a public exchange point, as that directly cuts into their transit business.  So we envisioned a location where access networks and smaller networks could meet CDNs and each other in a low cost, high performance community.

Of course the Law of Plentitude took over, and the Any2 Exchange has grown beyond our expectations.  The Law of Plentitude is an interesting concept, with many different definitions.  I’ll look at a couple ideas in this article, and hopefully this will help explain how IXPs such as Any2 add value to our community.

John’s Definition of the Law of Plentitude.  In any given community, when a new technology or group is created, there is a risk of adopting the new technology or join the new group.  The risk is that either the technology will fail, or joining the new group will result in either relationship problems in existing groups, or that energy directed towards the new group or technology will be wasted, denying that time, energy, or resource from being applied to other activities.

This is true up to about 15% diffusion in the community.  Once the community hits about 15% diffusion or adoption of that technology or group, then it becomes an even greater risk not to adopt the technology or join the group.  An example is fax and email.  A single fax machine has little or no value.  You need at least two fax machines to gain any level of value, and each time you add an additional fax machine to the addressable community of fax machines, each additional machine adds an exponential increase of value to the community.  At some point if you do not have a fax machine, then you are at risk of being denied participation in a given community, business, or group.  This will come at great cost, because if you do not have a fax machine yourself, most likely you will need to find somebody who does have a fax machine to allow you access to the new group or community. 

Ditto for email.  Think back to the days in the late 70s and early 80s when an email address and use of email were frequently met with amused comments about “geeks” and wasting time.  Today if you do not have an email address you are considered out of touch, and cannot participate in modern business relationships.

That is the second standard definition of the Law of Plentitude.

“More Gives More”

The sum of the network increases as the square of the number of members. The more plentiful things become, the more valuable they become! Each additional member added to the network increases the value of each individual member.

Not considerably different from our definition, and does add the idea that each additional member to a community adds an exponential value to the community. 

Thus another descriptive analogy.

Since I am a country person, I’ll use the growth of a small town to explain the Law of Plentitude in non-networking and technology terms.  Consider a cross roads in the country.  Nobody really notices as you drive by, and is mainly used to change directions when necessary to get to another place.  If you add a gasoline station at the crossroads, you might then get occasional travelers stopping by for petrol purchase.  If at some point a cafe is built at the crossroads, then stopping for the average traveler has more value, and the number of travelers stopping will probably increase. Next we add a convenience store, and possibly a motel, and you would reasonably expect additional travelers to stop by for even longer periods.  At some point one or more people may decide to build a house nearby the motel, gasoline stand, convenience store, or cafe, as it is then more convenient to work and support the needs of travelers.  You might continue growing until you then at some point have a community.

Any2 is similar to that community.  We started with a couple ISPs, added a couple of CDNs, a VoIP company or two, and gaming company and we had a small, robust and growing community.  At some point the value of that community grew larger than the local group, and became interesting or important to other members who were outside of our local One Wilshire or California community. 

As we added additional members from Asia, Europe, Russia, and Australia Any2 changed from being a local Internet exchange point, to a larger international gateway exchange point.  As each individual member joined, even small networks and CDNs, there was an exponential increase in the value of that community.  So the evolution of the Any2 Exchange has gone from a way to reduce OPEX, to a place where members could gain additional value in interconnection and performance, to becoming a mission critical component in both peering and international network disaster recovery or backup planning.

This is a classic example of the Law of Plentitude.

The second idea to review is a concept we discussed last year involving “Chaos Theory” and systems.  With successful growth within the past few years, the Any2 Exchange has actually created a new system.  This system is the high level view of how IP-enabled networks and operations interconnect and relate to each other.  Five years ago the Any2 Exchange “ecosystem” did not exist, however in the past two years it has created a new ecosystem of how networks in the Los Angeles and international network community interconnect.  The Any2 Exchange currently exists at the core or center of that new ecosystem.

As with any system, there is always a point when the core can become too rigid, and not meet the needs of the surrounding ecosystem.  The core needs to be flexible, or the system will once again evolve and either create additional subsystems that avoid the core, or develop a new core.  In the case of an Internet exchange point such as Any2, the danger of becoming too rigid, or making the exchange point a burden to the system could involve:


  • Creating exclusive or closed community entry requirements
  • Creating price points that make use of the exchange unattractive
  • Not adding new features, services, or technologies needed by the community for expansion and growth
  • Poor reliability
  • other items that may impede development and growth of the community

So to continue growing in the system core, there must be continual change within the core that adds value, and will attract further more activity within the system centering on the core. For an IXP to continue being successful, through adding additional members or activity (at the Internet system core), there must be continuous change that will attract new members to the IXP, and keep existing members at the IXP.

Now that is an interesting problem. By nature an IXP must remain neutral. So, if the IXP is neutral, and simply provides value to a “crossroads” at the telecom community interconnection point, then how do you add value at an IXP that is compelling enough to attract more growth at the system core?

At a public IXP, such as the Seattle Internet Exchange (SIX) it is a simple equation. Proximity to a lot of potential system/IXP members, and very low cost of entry (it is a public, member-supported IXP). The success of the SIX is clearly a product of simplicity and low cost of entry.
For commercial IXPs the problem is a bit different. If you need to charge for an IXP port, then the value of the IXP must exceed the value of a small network or CDN simply connecting to a large Tier 1 Internet Service Provider who will provide them a one stop shop for all their Internet access requirements. Thus if the IXP cannot provide value simply on price, then there must be other compelling features to attract members such as greater control over peering sessions through strong statistics, analytics, and monitoring. Or through addition of powerful utilities such as routes servers, or flexibility on connections or connection policies.

So the concept of the Law of Plentitude tells us healthy community growth is exponential, with the value of a community or system growing with each additional member to the system. Chaos theory tells us a system evolves and interacts with a core that attracts system activity, but the core must continue to evolve and add value in order to dominate the system. The conclusion is that to continue growing and attracting new members to IXPs such as the Any2 Exchange, the IXP must continue to add compelling value to the community and members, or risk creation of other systems (IXPs) that could attract members into a new system (competing IXP). The Internet was conceived as a system that could easily adapt and evolve, finding ways to route around blocks or obstructions, finding the easiest and most efficient way between origination and destination points. To be effective the IXP must provide the easiest path, and ensure the IXP does not become a barrier that forces the system to develop an alternate routing model.

IXPs and Disaster Recovery

The telecom world has once again dealt with the challenges of service disruption following cut or damaged submarine fiber optic cables.  Over the past two weeks separate incidents in the Med caused by, as of yet undetermined causes, disrupted large volumes of both telecom and Internet traffic serving many countries around the Indian Ocean.

Most large telecom companies and carriers using large submarine capacity have redundant routes built into their networks.  Thus if one of the major trunking links goes out of service, telecom traffic will immediately or dynamically reroute through another cable system.  In some cases this is a much longer route, but service is usually not affected other than potentially increased latency on the circuits.

Internet traffic is a bit different.  While Internet traffic may account for a very large percentage of global telecom capacity, the dynamic of Internet peering requires many more one-to-one relationships among other Internet network and service providers.  Why?  Because there are more Internet-enabled and Internet networks than traditional telecom carriers.  An Internet network manager has to manage many more inter-company relationships than a traditional carrier network manager.

The need for redundant engineering within Internet networks is the same as any other telecommunications system.  One tool the Internet community has available that is not shared by the standard carrier model are public and private Internet Exchange Points (IXPs).  Wikipedia defines an IXP as:

An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance.An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance. (www.wikipedia.org)

Large IXPs such as the Amsterdam Internet Exchange and the London Internet Exchange have more than 300 members.  All of those members have the option of peering with either individual networks, or large numbers of networks and Internet-enabled companies through use of route servers.  While dedicated circuits between larger networks are still common (larger networks sell Internet transit to smaller networks, and thus will not normally “peer” at an IXP), peering at IXPs is very cost effective for networks requiring both direct relationships with other networks (eliminating the need for purchasing some transit traffic).

One of the most important statements in Wikipedia’s description deals with fault tolerance.  As the Internet is a global network, and there is a need for global routing and access to every “end point” within the Internet, network administrators have a huge challenge designing effective disaster recovery plans for their networks.  For example, if you are an Indian network provider, and your traffic is primarily routed through the Med and Europe, a cable cut in the Med can have catastrophic effect on your network performance and connectivity.

If the traffic going through Europe is further directed towards North America, then your network will have even more serious performance problem – if you are able to get any connectivity.

So, when the cable cut occurs, the network administrator may have to arrange restoral circuits with many individual network service providers, content delivery networks, VoIP companies, and other networks – if the Indian company does not have a physical presence at one or more large IXPs.  At the large IXP, such as Amsterdam, London, Any2, or other large IXP, the network administrator has the option of quickly re-routing traffic through another cable directed at that large IXP, with the potential of many more peering relationships than if they were single threaded through a large trnsit carrier.  This is particularly true of those carriers which may not have adequate restoral capacity or restoral planning for those circuits.

Our own IXP, the Any2 Exchange, normally sees large spikes of traffic during major cable disruptions.  While there is a lot of primary traffic being sent through the IXP, it is also used by many carriers as a backup or tertiary disaster recovery point for Asian, Mid-East, Russian, and European networks to bypass disasters or cable disruptions.  North America is unique in that both major Atlantic/Mediterranean and Asian cable systems connect to north America, allowing North America to be used as an alternate route for traffic transiting any continent.

While the American Internet community has lagged the Europeans in using IXPs (for a variety of reasons – to be continued in a later blog entry), both Asian and European networks are quickly increasing their presence at American IXPs.  This is certainly paying off, as the European and Mid-East networks were able to quickly restore nearly all Interent traffic following our recent cable disruptions – much of that restoral going through the IXPs.

Flattening The American Internet

NOTE:  I originally published this article in 2007

Accessing information and interactive resources available around the globe via the Internet is a pretty simple task. In a carefree Internet world, the dynamics of connecting to resources are transparent, and we expect resources we want to access are available through our local Internet service provider. Technical details of connecting to Internet resources are an abstract concept for most, and whatever mechanics happen behind the scenes are not relevant to our everyday use of the network.

Because the Internet is made up of a complex matrix of physical, business and international relationships, how these systems interact and collaborate is actually very important to the end user, as well as to those providing Internet services and content. Of the greatest concern impacting online resources from eBay to the Bank of America is the potential financial pressure brought on by the largest Tier 1 networks. As the only networks in the world having global Internet visibility, these few companies, including AT&T, Sprint, Verizon, Level 3, and Cable and Wireless, facilitate access to the global Internet – a function which people and companies worldwide depend on to ensure small networks and content providers are available through their local service providers.

The Tier 1 world was born at the demise of NSFNet (National Science Foundation Network). In the early days of Internet development, the NSF supported development of a large publicaly funded academic and research network throughout the United States, and connecting many foreign academic networks to the US as a hub through the International Connections Manager (ICM Network). As commercial Internet development grew in the early 1990s, the NSF realized it was time to back away from publicaly funding the “Internet” and grant contracts to large US carriers to take over responsibility for the former US Domestic backbone and ICM portions of the NSFNet.

Small Internet exchange points (IXPs) were also funded, allowing the large networks taking over NSFNet assets, as well as their own commercial Internets to connect and share Internet traffic. Those network access points (NAPs) were also contracted to the large US carriers, who managed policies for US and International network exchange. The large US carriers ultimately had control of the networks, and were the original Tier 1 Internet providers.

Roadblocks in the Internet Community

Debates around net neutrality highlight some underlying issues. The goal of net neutrality is to preserve the open and interconnected nature of the public Internet. But whether the largest networks use their control to hinder growth and innovation within the Internet-connect business community or impede free access to Internet-connected content sources, they have the power and control which could present challenges to an open Internet environment.

A Tier 1 network, for example, has the power to charge a major content delivery network (CDN) a premium to access its network. This is because the CDN may deliver a very large amount of content traffic into a network, and the Tier 1 network believes they should receive additional compensation to fund additional capacity needed to support content distribution. This premium may be more money than the CDN is willing or able to pay. In turn, if the CDN doesn’t comply, the Tier 1 can ultimately refuse the CDN access to its network and cut its consumers access to the CDN’s content. This applies whether consumers access the Tier 1 directly or if the Tier 1 is the middle-network between consumers and their Tier 2 or 3 networks.

A voice over Internet Protocol Company underscores another potential conflict of interest. Let’s say you’re a consumer of a Tier 1 network that’s also a telephone company and you want to use a VoIP company, such as Vonage. But the Tier 1 doesn’t want the VoIP company to compete with its network and would rather that you use its own telephone product, so the Tier 1 may prevent you from using your VoIP company. In other words, a Tier 1, in developing its own commercial VoIP product, can prevent non-owned VoIP traffic from passing through its network.

While Tier 1 networks hold value for much of the Internet world, they also impose many political and financial barriers on smaller networks, content delivery networks, emerging VoIP companies, online gaming businesses, B2B and online commerce, and entertainment web sites. It is evident that Internet Service Providers (ISPs), CDNs, VoIPs, and many others need an alternative method of communicating with each other – one providing tools to redesign how relationships and interconnections bond the US Internet content and access communities.

Breaking Down Barriers

One objective in building efficiency and the performance needed to deliver content resources to end users is to flatten existing Internet architecture. Whenever possible, you eliminate the Tier 1 Internet networks from participating in the delivery of content resources to end users.

How do we accomplish this task? One option is through development and use of commercial Internet Exchange Points (IXPs), a location where many Internet-enabled networks and content resources meet to interconnect with each other as peers.

According to Wikipedia, an IXP is a physical infrastructure that allows different Internet Service Providers to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allows traffic to be exchanged without cost. An IXP is essentially a physical switch in a carrier hotel or data center with the capacity to connect thousands of networks together, whether content providers or network providers.

Today at the Any2 Exchange, an IXP built within One Wilshire, on a single switch 125 different networks interconnect and are freely able to pass traffic amongst each other without having to go to a Tier 1 for routing. Members pay a small annual fee to the Any2 Exchange for the one-time connection and then benefit from the “peering” relationships among members of the Internet exchange.

Akamai, for example, a large content distribution network company that delivers streaming media and movies on demand, can connect to American Internet Services, a Tier 3 ISP in San Diego, Calif., through a local or regional Internet exchange point such as the Any2 Exchange, the Palo Alto Internet Exchange (PAIX), or other large exchange points operated by data centers and carrier hotels.

When an American Internet Services user wants to watch a movie that’s available on Akamai’s content delivery network, the data is passed directly from Akamai to American Internet Services – and subsequently to the end user – without transiting any other network. Not only has the goal of being less reliant on a Tier 1 been achieved, but the performance is superior because there are no “hops” between the CSP and ISP. Anytime you’re able to cut out the transit network, you increase the end user experience. Plus, it’s more economical, as in moist cases the CDN and ISP have no financial settlement for data exchanged.

The European IXP model, which is more mature and robust than the US model, highlights the important function of IXPs and how an exchange point alone can help influence the net neutrality debate. In Europe, Internet service providers and content delivery networks look to the IXP as their first connection point and if the IXP doesn’t have what they’re looking for, only then will they go to a Tier 1 or large Tier 2. Americans on the other hand, partially due to geographic size

Overall European IXP traffic grew at a rate of 11.05%, compared to America’s rate of 7.44%, according to the European Internet Exchange Association in August 2007. This can be attributed in part to greater member density in Europe – the London Internet Exchange/LINX has more than 275 members – where the larger the addressable community, the larger the traffic exchanged and the more the members want to get involved. After all, network effect (exponential growth of a community) and the “Law of Plentitude” (the idea that once an addressable or social community reaches participation by 15% or greater of a total community, it becomes a risk to not participate in the emerging community) motivate European companies to use IXPs. Additionally, Europeans generally have lower entry costs for participation, giving companies every reason why to participate in the IXP-enabled peering community. If one were to buy access to 275 networks through a Tier 1, the cost would be astronomical, but through a single connection to LINX, one can access 275 networks for a nominal fee. This is why European companies rely on IXPs 60% of the time, and only look to Tier 1 or 2 networks 40% of the time.

In contrast, American ISPs normally look to larger wholesale and Internet transit providers first and then consider reducing their operational expenses via an IXP as a second priority. American ISPs companies use IXPs at a more meager 15% rate, looking to larger wholesale and transit Tier 1 or Tier 2 networks 85% of the time. Still, recent American IXP traffic growth does exceed other regions, such as Japan (+5.85% in August) and the rest of Asia (+4.3% in August), which we believe is a result of increased price pressure on the American IXP industry. Newer IXPs, such as the Any2 Exchange, have lowered entry costs significantly, forcing others to follow suit and encouraging more networks to participate. As the cost of entry to IXPs continues to fall, participation in IXPs will become more common and attractive to all access and CDN networks.

What can we learn from the European model? Participation in an IXP can increase performance, lower operational costs and expenses, as well as bring an additional layer of redundancy and disaster recovery capacity to even the smallest networks. But most important, companies’ independence from Tier 1s through the collective bargaining of the exchange points puts them in a stronger position to deal with large networks than our position allows for in the US, where the vast majority of people have their primary Internet connections through a large Tier 2 or Tier 1 network provider.

Adding to the Cause

Today’s content-rich Internet is just a prelude to the future content, media, applications and services soon to be developed and deployed. It’s no wonder that in large IXPs, such as the Amsterdam Internet Exchange (AMS-IX), there are already several content delivery networks using bundled 10Gbps ports, clearly showing end users’ insatiable demand for high bandwidth applications and services. High Definition Internet TV (IPTV), massive online interactive gaming, video on demand (VOD), and feature-rich communications (video conferencing) are just a few examples of Internet-enabled applications contributing to the heightened demand.

For American ISPs that pay anywhere from $20-to-$40/Mbps when connecting to Tier 1 and Tier 2 networks, the cost of delivering applications and services to end users who require much larger network and bandwidth resources is one of the obstacles that needs to be overcome. But without broad participation in IXPs, access networks have a difficult future, as do content providers who will find that the cost of delivery to end users becomes much more expensive if Tier 1 and Tier 2 networks increase the cost of delivering both wholesale and end user Internet traffic.

What Can the American Internet-Connected Community Do?

Whether through price increases or monopolistic practices, the largest networks are currently writing the rules for a global Internet product. They are gradually merging and acquiring competition, reinforcing their influence in wholesale and transit network share and presence. Opportunities for network peering decrease with each merger.

Carrier hotels and large data centers in the US can support positive change in the Internet peering community by creating or supporting open and low cost Internet Exchange points promoting network peering and content delivery to all networks.

Reducing barriers to entry and the cost of wholesale or transit networks will allow Internet network and content companies to focus on delivering network access and services, with the ultimate winner being end users who will enjoy a lower cost, higher performance Internet experience.

Network Effect and Route Servers on the Internet Exchange Point

This entry will give a simple view of route servers, explaining the value of a route server to the Internet community.

If the intent of an Internet exchange point/IXP is to

  1. Reduce operational expenses
  2. Increase performance with reduced latency
  3. Add additional disaster recovery capacity

The additional value of a route server is to take all of the above, and add a measure of simplicity to the process of interconnecting Internet-enabled networks.

Wikipedia defines an Internet Exchange Point/IXP as:

An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance. (http://en.wikipedia.org/wiki/Internet_exchange_point)

An additional definition needed to better understand the concept of route servers in “Internet Peering.”  Again we will go to the Wikipedia:

“Peering is voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the customers of each network. The pure definition of peering is settlement-free or “sender keeps all,” meaning that neither party pays the other for the exchanged traffic, instead, each derives revenue from its own customers.  http://en.wikipedia.org/wiki/Peering

The route server adds additional value by allowing a “connect once, connect with all” utility to the IXP. The route server is a technology where IXP members agree to “peer” with the route server, and the route server makes that member’s network or routing information available to all other route server members.  The other members will then be able to “peer” with all other participating members in an open, settlement-free environment.

This is extremely valuable for smaller Internet networks, content delivery networks/CDNs, and VoIP companies who may not have all the time and skills to develop individual relationships with many different networks.  With the route server a network will peer once with the route server, and immediately establish peering sessions with all other participants.

The network effect of peering through a route server can be dramatic.  Depending on the size of a network, the network may represent one or more “routes” to the rest of the Internet community.  Each route becomes valuable to the rest of the peering community, as it represents another location on the Internet which can be reached without the need for paying “transit” fees to a larger network.  Using the law of exponents, you can calculate the network effect of a route server community by giving a value of 1 to each represented route.  The overall value then becomes one of understanding the number of potential individual relationships available within the community.

The formula for network effect is:

N(N-1)/2.  Thus if a route server represents 225 potential routes, then the formula would look like 225(225-1)/2 = 25,200 potential individual routes available simply by peering a network with an IXP’s route server. Every additional route that becomes available on the route server (by adding additional members) adds an exponential value to the “route server community.”

CRG West’s Any2 Exchange operates a very popular route server called Any2Easy.  This route server (as of Nov 2008) has 80 participating networks, representing more than 15,000 individual routes.  The simplicity and availability of routes possible through settlement-free peering allows many smaller networks to significantly reduce their operating expenses and focus time and money on building their business.

Learn more about route servers through Wikipedia, using the tag “route server” in your browser search, or by reviewing an operational route server at http://www.crgwest.com/Any2Exchange/any2easy.html