SDNs in the Carrier Hotel

SDN_interconnections Carrier hotels are an integral part of global communications infrastructure.  The carrier hotel serves a vital function, specifically the role of a common point of interconnection between facility-based (physical cable in either terrestrial, submarine, or satellite networks) carriers, networks, content delivery networks (CDNs), Internet Service Providers (ISPs), and even private or government networks and hosting companies.

In some locations, such as the One Wilshire Building in Los Angeles, or 60 Hudson in New York, several hundred carriers and service providers may interconnect physically within a main distribution frame (MDF), or virtually through interconnections at Internet Exchange Points (IXPs) or Ethernet Exchange points.

Carrier hotel operators understand that technology is starting to overcome many of the traditional forms of interconnection.  With 100Gbps wavelengths and port speeds, network providers are able to push many individual virtual connections through a single interface, reducing the need for individual cross connections or interconnections to establish customer or inter-network circuits.

While connections, including internet peering and VLANs have been available for many years through IXPs and use of circuit multiplexing, software defined networking (SDNs) are poised to provide a new model of interconnections at the carrier hotel, forcing not only an upgrade of supporting technologies, but also reconsideration of the entire model and concept of how the carrier hotel operates.

Several telecom companies have announced their own internal deployments of order fulfillment platforms based on SDN, including PacNet’s PEN and Level 3’s (originally Time Warner) pilot test at DukeNet, proving that circuit design and provisioning can be easily accomplished through SDN-enabled orchestration engines.

However inter-carrier circuit or service orchestration is still not yet in common use at the main carrier hotels and interconnection points.

Taking a closer look at the carrier hotel environment we will see an opportunity based on a vision which considers that if the carrier hotel operator provides an orchestration platform which allows individual carriers, networks, cloud service providers, CDNs, and other networks to connect at a common point, with standard APIs to allow communication between different participant network or service resources, then interconnection fulfillment may be completed in a matter of minutes, rather than days or weeks as is the current environment.

This capability goes even a step deeper.  Let’s say carrier “A” has an enterprise customer connected to their network.  The customer has an on-demand provisioning arrangement with Carrier “A,” allowing the customer to establish communications not only within Carrier”A’s” network resources, but also flow through the carrier hotel’s interconnection broker into say, a cloud service provider’s network.  The customer should be able to design and provision their own solutions – based on availability of internal and interconnection resources available through the carrier.

Participants will announce their available resources to the carrier hotel’s orchestration engine (network access broker), and those available resources can then be provisioned on-demnd by any other participant (assuming the participants have a service agreement or financial accounting agreement either based on the carrier hotel’s standard, or individual service agreements established between individual participants.

If we use NIST’s characteristics of cloud computing as a potential model, then the carrier hotels interconnection orchestration engine should ultimately provide participants:

  • On-demand self-service provisioning
  • Elasticity, meaning short term usage agreements, possibly even down to the minute or hour
  • Resource pooling, or a model similar to a spot market (in competing markets where multiple carriers or service providers may be able to provide the same service)
  • Measured service (usage based or usage-sensitive billing  for service use)
  • And of course broad network access – currently using either 100gbps or multiples of 100gbps (until 1tbps ports become available)

While layer 1 (physical) interconnection of network resources will always be required – the bits need to flow on fiber or wireless at some point, the future of carrier and service resource intercommunications must evolve to accept and acknowledge the need for user-driven, near real time provisioning of network and other service resources, on a global scale.

The carrier hotel will continue to play an integral role in bringing this capability to the community, and the future is likely to be based on software driven , on-demand meet-me-rooms.

Internet Exchange in Developing Countries

This post is for Nara – wherever you might be.


In early 2000 I visited some of my friends and industry colleagues in Ulaanbaatar, Mongolia whom I’d worked with for a few years on various Internet-related projects. In those days Mongolia only had 5 Internet service providers (ISPs), and 100% of Internet traffic was connected via low capacity satellite connections.

Each ISP had a separate connection, some to California, one to Hong Kong, one to Germany – there was really no planning other than to get the cheapest bandwidth possible to meet the needs of a rapidly growing Internet community. Mongolians use a Cyrillic-based written language, and Mongolians began to see the benefit of having a local economy that could function within the culture and language of the people.

The only problem was that each ISP had an independent upstream Internet connection, and for the most part the links were saturated. The interesting thing is that much of the saturation was due to Mongolian language or Mongolian interest traffic going out one ISP, to say an upstream connection in Stockton, Calfornia – and returning to another ISP’s user through an international connection based in Germany.

The result is as you would expect – very poor performance and user experience at a very high cost.

So hosting companies started to get smart. Rather than host web sites in Mongolia, companies began to find hosting platforms in North America and Europe to host Mongolian content. While not perfect, it did remove about one half of the performance bottleneck between users of different ISPs. Of course the bad thing is the revenue produced by hosting went to American and European companies, removed from the Mongolian economy forever.

So in early 2000 I met with friends and colleagues from several different ISPs in Ulaanbaatar, and brought up the idea of building a neutral Internet Exchange Point/IXP in Ulaanbaatar to facilitate local interconnection between ISPs. Intially there was a bit of reluctance to the idea of cooperating with competitors, but in the end the ISP owners relaized the customer performance and cost savings of taking local traffic off the international links made a tremendous amount of sense. A small Mongolian company called Infocon was chosen to manage the project.

Problem – no money to build the IXP. Solution – I used my credit card and bought a pile of Cisco switching and transmission hardware and donated it to the Mongolian ISP community. Another friend (Raphael Ho) came to Ulaanbaatar, configured, and connected the ISPs to what became known as the Mongolian Internet Exchange/MIX. The impact was immediate, and all of the reasons for building an IXP in a developing country met the model and image of how it should be done.

A few years later the original MIX was replaced by a high performance platform donated by another international group, and the MIX grew to support current robust Internet community thriving in Mongolia today.

The moral? The MIX represents a success story for remote locations and developing countries to use in ensuring their own economy and user community has the best possible tools available to reduce international transmission cost, increase end user and network performance and provide a positive experience. Why should a developing country pay a surcharge to the international community for developing a local economy? No reason at all.

For us who enjoy relative opulence in our worlds, consider the value we can bring to a developing country or company with our experience, and the way we can enable those developing countries to have a better chance to get up to global economy speed – often with very little effort of our own.

Some days it is fun to be an engineer.

2009 – A Year of IPv6 and Internet Virtualization

This article originally appeared in the Jan 2009 Any2 Exchange Newsletter


For the past 30 years or so we have gone through an accelerated learning process in globalization.  While the under 25 crowd has lived in a world of extreme technology diffusion, many of us still recall the days when having near real time news was an exception, and provided at a great cost in both human and financial resources.


Who can forget the international awakening when real time information came out of UNIX Talk sessions during the Russian Coup attempt in 1991 – when the world realized the veil of national secrecy and suppression of events had been ripped open forever.  The Internet has played a role in international transparency which has changed the concept of globalization and brought people together on a human scale that could not have been comprehended just a couple decades ago.  It is safe to assume we have successfully passed the Internet concept test. 


The basic tools we’ve used over the past 40 years have proven global one-to-one, one-to-many, and many-to-many communications via packets works. Now we can move on to the next phase of global communications development, with a new suite of tools that will allow even more powerful ways of bringing the world to your laptop.


This month we have two feature articles in the Any2 newsletter.  One by Martin Levy on the topic of IPv6, and the other by Justin Giardina on storage replication and virtualization.  In the next edition we will draw our attention to cloud computing and software as a service.


The Any2 Exchange fully supports deployment of IPv6 in the IXP, as well as support through the Any2Easy route servers.  The actual amount of IPv6 traffic is still relatively low, however the number of routes available through Any2 is showing steady growth.  We allocate both IPv4 and IPv6 addresses to all new Any2 Exchange members, and will assign new IPv6 addresses on request to any other existing member who does not currently have an allocation. 


It is important for us to aggressively promote IPv6.  ARIN recently made the following statement:


“With only 15% of IPv4 address space remaining, ARIN is now compelled to advise the Internet community that migration to IPv6 is necessary for any applications that require ongoing availability of contiguous IP number resources.” (


With an accompanying resolution:


BE IT RESOLVED, that this Board of Trustees hereby advises the Internet community that migration to IPv6 numbering resources is necessary for any applications which require ongoing availability from ARIN of contiguous IP numbering resources.”





Those are pretty serious warnings to the Internet community that it is time to re-tool for the next generation. What is that generation?  Lots of visionaries out there who are far more creative than I, offering a landscape of virtualization and convergence of nearly every aspect of life.


In the scope of our activities we are seeing customer and tenant movement in the direction of virtualization, through both storage and software as a service via cloud compute provisioning.  While getting a lot of attention in the tech media as a concept, the actual deployment of these services is going ahead in a near stealth mode.  We eagerly look forward to the marriage of Brocade and Foundry, and the offspring that may further bring storage and cloud compute capacity into the switching and routing fabrics.


The ability to route and switch with direct addressing, rather than NAT or private addressing is going to be a requirement in the virtual compute world to help eliminate both physical and software points of failure, as well as eliminate any latency byproduct of address translation.


So, we have our work cut out for us.  CRG West and the Any2 Exchange see our role in this new world as the developers of infrastructure needed to support the applications and services being developed by networks, cloud companies, SaaS companies, CDNs, and the carriers.


At the end of the day we still need to provide solid electrical and cooling systems support, access to fiber and interconnections (including the Any2 Exchange), and a neutral place for the community to meet.


With the economy in turmoil, limited funding available for both capital and operational expenses, and the need to rapidly move ahead, we will strive to do our part in providing the infrastructure and community center to reduce both CAPEX and OPEX, as well as develop the facility infrastructure needed to fulfill the visions of those who lead.



Network Effect and Route Servers on the Internet Exchange Point

This entry will give a simple view of route servers, explaining the value of a route server to the Internet community.

If the intent of an Internet exchange point/IXP is to

  1. Reduce operational expenses
  2. Increase performance with reduced latency
  3. Add additional disaster recovery capacity

The additional value of a route server is to take all of the above, and add a measure of simplicity to the process of interconnecting Internet-enabled networks.

Wikipedia defines an Internet Exchange Point/IXP as:

An Internet exchange point (IX or IXP) is a physical infrastructure that allows different Internet service providers (ISPs) to exchange Internet traffic between their networks (autonomous systems) by means of mutual peering agreements, which allow traffic to be exchanged without cost. IXPs reduce the portion of an ISP’s traffic which must be delivered via their upstream transit providers, thereby reducing the Average Per-Bit Delivery Cost of their service. Furthermore, the increased number of paths learned through the IXP improves routing efficiency and fault-tolerance. (

An additional definition needed to better understand the concept of route servers in “Internet Peering.”  Again we will go to the Wikipedia:

“Peering is voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the customers of each network. The pure definition of peering is settlement-free or “sender keeps all,” meaning that neither party pays the other for the exchanged traffic, instead, each derives revenue from its own customers.

The route server adds additional value by allowing a “connect once, connect with all” utility to the IXP. The route server is a technology where IXP members agree to “peer” with the route server, and the route server makes that member’s network or routing information available to all other route server members.  The other members will then be able to “peer” with all other participating members in an open, settlement-free environment.

This is extremely valuable for smaller Internet networks, content delivery networks/CDNs, and VoIP companies who may not have all the time and skills to develop individual relationships with many different networks.  With the route server a network will peer once with the route server, and immediately establish peering sessions with all other participants.

The network effect of peering through a route server can be dramatic.  Depending on the size of a network, the network may represent one or more “routes” to the rest of the Internet community.  Each route becomes valuable to the rest of the peering community, as it represents another location on the Internet which can be reached without the need for paying “transit” fees to a larger network.  Using the law of exponents, you can calculate the network effect of a route server community by giving a value of 1 to each represented route.  The overall value then becomes one of understanding the number of potential individual relationships available within the community.

The formula for network effect is:

N(N-1)/2.  Thus if a route server represents 225 potential routes, then the formula would look like 225(225-1)/2 = 25,200 potential individual routes available simply by peering a network with an IXP’s route server. Every additional route that becomes available on the route server (by adding additional members) adds an exponential value to the “route server community.”

CRG West’s Any2 Exchange operates a very popular route server called Any2Easy.  This route server (as of Nov 2008) has 80 participating networks, representing more than 15,000 individual routes.  The simplicity and availability of routes possible through settlement-free peering allows many smaller networks to significantly reduce their operating expenses and focus time and money on building their business.

Learn more about route servers through Wikipedia, using the tag “route server” in your browser search, or by reviewing an operational route server at

%d bloggers like this: