The Utility and Pain of Internet Peering

In the early 1990s TWICS, a commercial bulletin board service provider in Tokyo, jumped on the Internet. Access was very poor based on modern Internet speeds, however at the time 128kbps over frame relay (provided by Sprint international) was unique, and in fact represented the first truly commercial Internet access point in Japan.

The good old boys of the Japanese academic community were appalled, and did everything in their power to intimidate TWICS into disconnecting their connection, to the point of sending envelopes filled with razor blades to TWICS staff and the late Roger Boisvert (*), who through Intercon International KK acted as their project manager. The traditional academic community did not believe anybody outside of the academic community should ever have the right to access the Internet, and were determined to never let that happen in Japan.

Since the beginning, the Internet has been a dichotomy of those who wish to control or profit from the Internet, and those who envision potential and future of the Internet. Internet “peering” originally came about when academic networks needed to interconnect their own “Internets” to allow interchange of traffic and information between separately operated and managed networks. In the Internet academic “stone age” of the NSFNet, peering was a normal and required method of participating in the community. But,… if you were planning to send any level of public or commercial traffic through the network you would violate the NSFNET’s “acceptable use policy/AUP” preventing use of publically-funded networks for non-academic or government use.

Commercial internet Exchange Points such as the CIX, and eventually the NSF supported network access points/NAPs popped up to accommodate the growing interest in public access and commercial Internet. Face it, if you went through university or the military with access to the Internet or Milnet, and then jumped into the commercial world, it would be pretty difficult to give up the obvious power of interconnected networks bringing you close to nearly every point on the globe.

The Tier 1 Subsidy

To help privatize the untenable growth of the NSFNet (due to “utility” academic network access), the US Government helped pump up American telecom carriers such as Sprint, AT&T, and MCI by handing out contracts to take over control and management of the world’s largest Internet networks, which included the NSFNet and the NSF’s international Connection Managers bringing the international community into the NSFNet backbone.

This allowed Sprint, AT&T, and MCI to gain visibility into the entire Internet community of the day, as well as take advantage of their own national fiber/transmission networks to continue building up the NSFNet community on long term contracts. With that infrastructure in place, those networks were clear leaders in the development of large commercial internet networks. The Tier 1 Internet provider community is born.

Interconnection and Peering in the Rest of the World

In the Internet world Tier1 networks are required (today…), as they “see” and connect with all other available routes to individual networks and content providers scattered around the world. Millions and millions of them. The Tier 1 networks are also generally facility-based network providers (they own and operate metro and long distance fiber optic infrastructure) which in addition to offering a global directory for users and content to find each other, but also allows traffic to transit their network on a global or continental scale.

Thus a web hosting company based in San Diego can eventually provide content to a user located in Jakarta, with a larger network maintaining the Internet “directory” and long distance transmission capacity to make the connection either directly or with another interconnected network located in the “distant end” country.

Of course, if you are a content provider, local internet access provider, regional network, or global second tier network, this makes you somewhat dependant on one or more “Tier 1s” to make the connection. That, as in all supply/demand relationships, may get expensive depending on the nature of your business relationship with the “transit” network provider.

Thus, content providers and smaller networks (something less than a Tier 1 network) try to find places to interconnect that will allow them to “peer” with other networks and content providers, and wherever possible avoid the expense of relying on a larger network to make the connection. Internet “Peering.”

Peering Defined (Wikipedia)

Peering is a voluntary interconnection of administratively separate Internet
networks for the purpose of exchanging traffic between the customers of each network. The pure definition of peering is settlement-free or “sender keeps all,” meaning that neither party pays the other for the exchanged traffic; instead, each derives revenue from its own customers. Marketing and commercial pressures have led to the word peering routinely being used when there is some settlement involved, even though that is not the accurate technical use of the word. The phrase “settlement-free peering” is sometimes used to reflect this reality and unambiguously describe the pure cost-free peering situation.

That is a very “friendly” definition of peering. In reality, peering has become a very complicated process, with a constant struggle between the need to increase efficiency and performance on networks, to gaining business advantage over competition.

Bill Norton, long time Internet personality and evangelist has a new web site called “DR Peering,” which is dedicated to helping Internet engineers and managers sift through the maze of relationships and complications surrounding Internet peering. Not only the business of peering, but also in many cases the psychology of peering.

Peering Realities

In a perfect world peering allows networks to interconnect, reducing the number of transit “hops” along the route from points “A” to “B,” where either side may represent users, networks, applications, content, telephony, or anything else that can be chopped up into packets, 1s and 0s, and sent over a network, giving those end points the best possible performance.

Dr Peering provides an “Intro to Peering 101~204,” reference materials, blogs, and even advice columns on the topic of peering. Bill helps “newbies” understand the best ways to peer, the finances and business of peering, and the difficulties newbies will encounter on the route to a better environment for their customers.

And once you have navigated the peering scene, you realize we are back to the world of who wants to control, and who wants to provide vision. While on one level peering is determined by which vendor provides the best booze and most exciting party at a NANOG “Beer and Gear” or after party, there is another level you have to deal with as the Tier 1s, Tier 1 “wanna-be networks,” and global content providers jockey for dominance in their defined environment.

At that point it becomes a game, where personalities often take precedence over business requirements, and the ultimate loser will be the end user.

Another reality. Large networks would like to eliminate smaller networks wherever possible, as well as control content within their networks. Understandable, it is a natural business objective to gain advantage in your market and increase profits by rubbing out your competition. In the Internet world that means a small access network, or content provider, will budget their cost of global “eyeball or content” access based on the availability of peering within their community.

The greater the peering opportunity, the greater the potential of reducing operational expenses. Less peering, more power to the larger Tier 1 or regional networks, and eventually the law of supply and demand will result in the big networks increasing their pricing, diluting the supply of peers, and increasing operational expenses. Today transit pricing for small networks and content providers is on a downswing, but only because competition is fierce in the network and peering community supported by exchanges such as PAIX, LINX, AMS-IX, Equinix, DE-CIX, and Any2.

At the most basic level, eyeballs (users) need content, and content has no value without users. As the Internet becomes an essential component of everybody on the planet’s life, and in fact becomes (as the US Government has stated) a “basic right of every citizen,” then the existing struggle for internet control and dominance among individual players becomes a hindrance or roadblock in the development of network access and compute/storage capacity as a utility.

The large networks want to act as a value-added service, rather than a basic utility, forcing network-enabled content into a tiered, premium, or controlled commodity. Thus the network neutrality debates and controversy surrounding freedom of access to applications and content.

This Does Not Help the Right to Broadband and Content

There are analogies provided for just about everything. Carr builds a great analogy between cloud computing and the electrical grid in his book the “Big Switch.” The Internet itself is often referred to as the “Information Highway.” The marriage of cloud computing and broadband access can be referred to as the “4th Utility.”

Internet protocols and technologies have become, and will continue to be reinforced as a part of the future every person on our planet will engage over the next generations. This is the time we should be laying serious infrastructure pipe, and not worrying about whose content should be preferred, settlements between networks, and who gives the best beer head at a NANOG party.

At this point in the global development of Internet infrastructure, much of the debate surrounding peering – paid or unpaid, amounts to noise. It is simply retarding the development of global Internet infrastructure, and may eventually prevent the velocity of innovation in all things Internet the world craves to bring us into a new generation of many-to-many and individual communications.

The Road Ahead

All is not lost. There are visionaries such as Hunter Newby aggressively pushing development of infrastructure to “address America’s need to eliminate obstacles for broadband access, wireless backhaul and lower latency through new, next generation long haul dark fiber construction with sound principles and an open access philosophy.”

Oddly, as a lifelong “anti-establishment” evangelist, I tend to think we need better controls by government over the future of Internet and Internet vision. Not by the extreme right wing nuts who want to ensure the Internet is monitored, regulated, and restricted to those who meet their niche religions or political cults, but rather on the level of pushing an agenda to build infrastructure as a utility with sufficient capacity to meet all future needs.

The government should subsidize research and development, and push deployment of infrastructure much as the Interstate Highway System and electrical and water utilities. You will have to pay for the utility, but you will – as a user – not be held hostage to the utility. And have competition on utility access.

In the Internet world, we will only meet our objectives if peering is made a necessary requirement, and is a planned utility at each potential geographic or logical interconnection point. In some countries such as Mongolia, an ISP must connect to the Mongolia Internet Exchange as a requirement of receiving an ISP license. Why? Mongolia needs both high performance access to the global Internet – as well as high performance access to national resources. It makes a lot of sense. Why give an American, Chinese, or Singaporean money to send an email from one Mongolian user to another Mongolian user (while in the same country)? Peering is an essential component of a healthy Internet.

The same applies to Los Angeles, Chicago, Omaha, or any other location where there is proximity between the content and user, or user and user. And peering as close to the end users as technically possible supports all the performance and economic benefits needed to support a schoolhouse in Baudette (Minn), without placing an undue financial burden on the local access provider based on predatory network or peering policies mandated by regional or Tier 1 networks.

We’ve come a long way, but are still taking baby steps in the evolution of the Internet. Let’s move ahead with a passion and vision.

(*)  Roger Boisvert was a friend for many years, both during my tensure as  US Air Force officer and telecom manager with Sprint based in Tokyo (I met him while he was still with McKinsey and a leader in the Tokyo PC User’s Group), and afterwards through different companies, groups, functions, and conferences in Japan and the US.  Roger was murdered in Los Angeles nine years ago, and is a true loss to the internet community, not only in Japan but throughout the world.

Giving Ourselves a Broadband Facelift for the 2010 Matrix

Of all the memories the telecom community has of the 80s and 90s, one of the most vivid is the sight of long haul fiber optic cable systems being buried throughout the United States. A product of deregulation, competition, and the birth of the Internet, American telecom companies saw a desperate need for greatly increasing transmission capacity, and responded with investments in long haul fiber, metro fiber, and digital switching needed to meet all visions of what we knew in those wonderful days of innovation.

Globally, broadband Internet, 3G + wireless, and the convergence of everything from entertainment to telephony into digital formats is driving not only Internet technologies, but also physical telecom transmission systems to the threshold of existing capacity. This explosive growth in information and communications technologies creates an interesting dilemma for telecom companies.

Do you spend your efforts finding ways to control the use of existing capacity? Or do we acknowledge the fact our network-enabled global community is not likely to get any smaller, and the world now needs our telecom thought leadership to both greatly expand what we already have, while aggressively investing in developing transmission technology that will enable, not restrict, growth in all things digital.

Not a US-Only Challenge

When a child in South Africa, Hanoi, or Denpasar has equal access to Hulu TV, Skype video chats, and eLearning systems from either a fixed workstation or mobile phone, it can be argued technology is serving the purpose of enabling and providing a new generation with the intellectual tools they need to flatten the geographic and political barriers we have lived with since the beginning of time.

All great, benevolent thoughts. Our children may need the tools to correct the problems we’ve created through irresponsible use of fossil fuels, exploitation of natural resources, human transmitted disease, war, and creation of toxic “stuff” that continues to restrict our planet’s ability to create an acceptable quality of life for all.

Face it, educated people in general do not make as many BIG mistakes as those who blindly follow others due to ignorance or lack of exposure to a wide variety of knowledge. Internet and telecom-enabled technologies may facilitate some people who thrive on physical or ideological control, however that is also diluted as the percentages bring their own knowledge of fact, and exposure to a liberal dosage or prism of different perspectives.

Or in other words, we can hope primary school students from different countries and cultures who meet each other through chatting or cooperative educational projects will be more likely to collaborate on useful endeavors in later life than those who are only exposed to a narrow view of society, culture, ideologies, and leadership.

Getting to the Vision

All this is great. An altruistic, warm, and fuzzy view of the future. Getting our vision to reality requires a tremendous amount of work. The current caretakers of industry and leadership do not have all the intellectual tools needed to keep up with a developing generation of children who were birthed in the Internet Age.

However we (the current caretakers) are pretty good at building things. Among those things are fiber optic transmission systems spanning oceans, continents, cities, and now even homes. We are good at building wireless transmission towers, and are still pretty good at building devices that can connect all this fiber, tower, and wireless infrastructure together.

And the younger generation is beginning to envision ways to exploit the transmission “matrix” that is beyond the comprehension of our current caretaker generation.

“The world is becoming one, big, ubiquitous, homogeneous system because of “the network” and the network exists and needs to exist because it exists (in other places) already. This is the justification to build. It is a self-fulfilling chain reaction.” (Hunter Newby, CEO Allied Fiber)

The Republicans in the US like to scream the need for Americans to “Drill Baby Drill,” exploiting domestic sources of fossil fuels, reducing our dependence on foreign sources for energy. In the telecom industry we are beginning to feel the need to “Dig Baby Dig.”

We need to increase our ability to continue delivering the network transmission capacity required to give our next generation the tools needed to really make a “Matrix-enabled” future, rather than spend our efforts scrambling, as in the energy analogy, to control or reduce our dependence on existing sources of telecom capacity.

How it is Going to Happen

In the US, for the past 30 years deregulation has allowed the telecom industry to build their infrastructure without any oversight other than what local or state governments impose for licensing and access to rights of way. Most debates have surrounded topics such as net neutrality, control over markets, or conduct of both content and users connecting to the Internet.

The US National Science Foundation inadvertently created the current, sometimes restrictive environment within the US Internet community by passing control of the NSFNet backbone to a select few commercial providers (AT&T, MCI, and Sprint). This award increased incentives for carriers to control their part of the US Internet space, and reduce incentives to aggressively build out physical capacity needed to meet the exponentially increasing demands for bandwidth and capacity.

It did not greatly meet infrastructure requirements needed to support the convergence of everything that can, does, should, and will travel over Internet Protocol (IP) networks over the next 25 or 30 years. While there are some positive developments in the local loop (FiOS, LTE, WiMAX, Uverse, etc), Newby cautions in the US there is a dearth of long haul and metro capacity needed to string all the local initiatives together.

The answer is to dig. Dig more conduits around the United States and Canada, drop the highest existing capacity fiber cabling within the conduits, connect wireless towers supporting LTE/4G+ to the high capacity backbone, connect buildings and homes, and develop new even higher capacity transmission technologies to parallel or exceed similar models of growth such as Moore’s Law and Metcalf’s Law.

But to give us the space needed to develop those technologies, for now, dig baby dig. Give fiber optic long haul, metro, and local digs the same tolerance we give to filling potholes and expanding lanes on a freeway system – while in the background we hope our leadership designs high speed rail, better road construction materials, and better ways to move from point “A” to point “B.”

Consider broadband, hyper-band, and uber-band development the true 4th Utility justifying extreme social priority, without which we will suffer the same fate as losing electricity, water, and roads. As with roads, everything we do going into the future will ride the broadband “matrix,” and without enough available lanes we will reduce ourselves to a frustrating gridlock of intellectual, business, and social development.

Dig baby dig…

NOTE: I was first introduced to the concept of the “Matrix” in the early 1990s, when a friend of mine suggested I read a book by John S. Quarterman entitled “The Matrix: Computer Networks and Conferencing Systems Worldwide.” 20 years after, and it is still the most enlightening view of the Internet, what the internet cloud and should be, as well as look into the future as anything I have ever read on the topic. It takes William Gibson, Neal Stephensen, and translates their fiction into a reality which continues to become part of our day to day lives. Or maybe it gave both authors additional ideas needed for them to develop fiction…

Techno-SciFi for Engineers and Soldiers – Daemon

There is nothing more irritating or annoying to a professional soldier than to watch a movie and find technical errors. A haircut that is out of regulation, a misplaced ribbon or medal, errors in weapon nomenclature, or even unit Reviewing Neal Stephenson and Daniel Saurezdesignations and locations. A soldier knows within a millisecond when there is a technical error – and it dilutes even the best story line. Telecom and Internet industry-related professionals have the same emotion when terms, equipment, or architectures are mispresented in movies.

Then along comes an author who has either really done his homework well, had great advice, or simply knows his subject matter cold. Once the credibility is firmly established, then there is an uncanny ability to lay a story on top of that technical credibility, and keep even the most critical geek engaged.

Neal Stephenson’s “Cryptonomicon” was the first novel I had read which met this strict criteria. Did a good job, because I spent most of the next year reading everything he ever wrote, and have kept up since with great stories such as “Anathem.” I trust Neal Stephenson, so I am able to freely indulge in his stories without becoming tolerant of an error-prone technical structure to the story.

I like Michael Connelly and Robert Crais because they correctly describe locations around Los Angeles, where I live, and it helps put their stories into context. Did I mention I really like technically accurate stories?

Just when I thought it was safe, and that I would not become addicted to another Techno-SciFi author, I walk past a row of Paperbacks and spy the title “Daemon,” by Daniel Saurez.

“Daemon,” huh?…

OK, for a communicator the word Daemon has a very specific meaning:

A daemon is a computer program that runs in the background, rather than under the direct control of a user. Daemons are usually initiated as background processes. (Wikipedia)

Skeptical, I have the initial thought this would be another silly novel name dropping some lexicon of the Internet in a title to try and suck in unsuspecting readers. I read the reviews, and hold on,… these are not the average reviews writing a couple sentences out of a random word generator. These are real people, and some of them I know! I mean, how often do you see a review from Craig Newmark (Craigslist) or a director of Cybersecurity and Communications Policy writing testimonials?

Guess it was OK to give it a shot, and spend the $10. As I had a week of investment in airplanes and airports to exploit, maybe I would trip into something that was good enough to get me home.

Within the first couple pages a grisly series of murders gets my caveman senses awakened, however the environment in which the murders are committed, is, well, it is technically really accurate and complex enough to keep a geek engaged. I mean, when we start talking about 480 volt power systems, server farms, air conditioning system, biometrics, remote processes – well, it is clear the author has been around the block a few times.

While he quickly goes over my head on topics related to gaming, he attaches the gaming discussion to the underlying infrastructure like a data intelligence to a frequency. And the characters are as equally screwed up in the head as any real life gamer or software engineer I have worked with over the past 30 years. Saurez gets it, is part of it, and has produced a novel that codifies all the sick, twisted fantasies you would expect a systems engineer or software developer to harbor.

Then he ensures there are adequate personalities, education levels, egos, and human emotions t remind us this is not science fiction, it is reality – as we know reality today, adding a bit of creativity to an existing set of intellectual and physical tools. Most of those tools live inside of our known “cloud” of the Internet, but the potential of this creative thinking behind his story line is feasible enough to bring chills to an engineer’s spine.

A Strong Recommendation for your “Geek’s Reading List”

Neal Stephenson and Daniel Saurez are engaging, technically accurate, and tremendously creative authors. Stephenson’s novels are a bit more difficult to read, as he brings his ideas to an abstraction that is a bit above mindless reading. Stephenson almost tests, mocks, or challenges his readers to step back and see the big picture of his story lines. If you read page to page, you miss the point of his books. But still have a lot of fun reading the stories. Sometimes you pick up additional jewels during your third or fourth read through of the books.

Saurez puts it right in your face, and challenges you to discredit his story line. “Go ahead, prove this couldn’t happen today…”

Both are great, and should be required reading for geeks who need to step back from their Cisco manuals and RFC memorization exercises, and actually experience how creative people can apply our existing and emerging technologies to abstractions of thought. Remember, the engineer can design a tool, but only a user can find creative ways to exploit the tool. Engineers can learn a lot from people who apply our visions to solve problems and enable new opportunities. Having recently finished reading Anathem and Daemon, I cannot pass by a router, switch, or server without thinking….

Read the book

John Savageau, Honolulu

Future Visions of Global Telecoms with Bjarni Thorvardarson and Hibernia Atlantic

Bjarni Thorvardarson is a rare telecom visionary. He thinks on a level of telecoms at an intercontinental level, rather than a national or local level. Hibernia Atlantic is his current project, and with recent news the submarine and terrestrial cable system is now in the global media distribution business, he is shaking up the telecom community. An Icelandic native, he has brought his knowledge and skills to the United States, basing Hibernia Atlantic in Summit, New Jersey.

Pacific-Tier Communications series on Entrepreneurs and Thought Leaders continues with Bjarni Thorvardarson, CEO of Hibernia Atlantic (www.hiberniaatlantic.com)

Pacific-Tier: Bjarni, what’s been happening with Hibernia Atlantic in the past few years, since I had my last opportunity to visit with you in Summit?

Bjarni Thorvardarson CEO Hibernia AtlanticBjarni Thorvardarson:
We’ve actually had a busy couple years – a very busy couple of years.

As you may recall, we started this business by buying a submarine asset that was formerly owned by 360 Networks. We’ve been busy trying to build our terrestrial network to try and connect this submarine cable to anywhere. We no longer refer to ourselves as simply a submarine cable, but rather a capacity provider in the eastern North America region and in Europe. Less than half of our business is now in trans-Atlantic capacity.

Even though that remains our core competence, and the core of our business, in terms of our business it is less than half of our revenue.

So that is part of the transformation that has happened over the past few years.

In terms of revenue, in 2005 we generated about $2 million in revenue, then $7 million, the $18 million the following year, then $28 million, and last year we generated about $38 million dollars.

And now the target for 2010, with our addition of MediaXstream, is about $75 million dollars in revenue. So, we’ve been growing about 40~50&, up to 80 or 90% a year. So you see it’s a very rapid growth. We are riding on a couple things. One is that we are operating in the biggest capacity market in the world, which is the Northeastern part of North America and Europe. And we are focused on a niche sector, which is the big bandwidth market – which is by itself growing about 40% in volume terms a year

And now, since we still have a relatively small market share, we are growing even faster than the other (traditional) markets (players). That’s how we’ve been successfully growing our revenues.

Last year, in 2009, we were confident for the first time, and we were profitable. We are very happy with the way things are going.

Pacific-Tier: That’s excellent! Can you tell me a little about yourself, and how you got into the business of both submarine and terrestrial bandwidth and capacity?

Bjarni Thorvardarson:
Sure. I’ve been in the IT and telco business for about 17 years since I finished my business studies. By education I have a degree in engineering from the University of Wisconsin (At Madison). Later on I added a science degree from the London Business School.

I went into the telecommunications business, and from there into investment banking (around 1998). Then shortly thereafter I started a fund that was investing in telecommunications and IT. That was 1999 into 2001. From there Ken Peterson actually got interested, Ken Peterson was the owner of Columbia Ventures (CVC), got into an investment in telecommunications. He brought a co-investor in with me, and bought the fund eventually. That’s how it came about that I started working for CVC.

And I’ve been investing on their behalf in telecommunications since 2002.

Now one of the investments that we had made was indeed Hibernia Atlantic. That was 2003. Then in 2004 we sold an investment that I was responsible for here in Ireland. Then I took over the responsibility of Hibernia Atlantic. Since 2005 I have been with Hibernia Atlantic, running it for CVC.

That’s how it came about that I’ve been investing and spending my time in telecommunications.

When it came that I was to take on the Hibernia Atlantic project, it was meant to be a 6 month project. We’d see what we could do, fix a couple things, and recruit the right people. It’s one of those things that a 6 month window turned into a sliding 6 month window. And during the next 6 months we ended doing something exciting. That’s what happens when you get interested in what you are doing.

You see the potentiality, you see what can be done, and you kind of stick around – and so its 5 years now. I no longer refer to it as my 6 month project, I now refer to it as my passion. It’s what I do. I’ve been commuting between Ireland and the US now for the better part of 5 years, and relocated to New Jersey where we have the Summit headquarters, or US headquarters a few years ago. So I am pretty heavily enrolled in the Hibernia project.

Pacific-Tier: That’s good. You mentioned earlier the topic of moving from telephony to broadband. Where does Hibernia and your plans fit into what I would call the “globalization of communications,” or the “flattening of the communications architecture…” How do you fit into that model?

Bjarni Thorvardarson:
Hibernia is very much a long-haul provider. We started in the long-haul wholesale capacity business, so provided the infrastructure provider to other service providers – the likes of BT and France Telecom, Cogent and the like… We did a very good business connecting the biggest consumer markets to places like New York, London, Amsterdam, Chicago, connecting those markets and enabling service providers in those markets to connect to different parts of the world.

Our business is really predicated, it is built on the globalization of business, or the globalization and international movement of information. That was the core part of our business model.

Now since then we have moved on to going up the value chain (if you like) to become the service provider to enterprises ourselves, and begin focusing on the finance vertical, which is a very demanding market. They (financial markets) are demanding and expecting low latency circuits between different trading markets and centers. That was a big first step into the enterprise world.

The next step we took was to the media sector, which we did first when we were acquainted with or partnered up with MediaXtreme, investing in MediaXstream a couple years ago. Then finally culminated in the acquisition of MediaXstream last month. So that’s our big step into the media market.

So now Hibernia’s approach to the market is threefold:

  1. We are still very much the legacy we started, which is the wholesale provider to other service providers and telcos around the world
  2. Second is the finance sector
  3. Third is the media sector

But they all are very much relying on the globalization of business and people’s general view of the world. So we have to look and depend on it.

Pacific-Tier: So in a traditional sense submarine and terrestrial long-haul networks relied on SONET or SDH technologies as the basic (communications) protocol. Is Ethernet playing a stronger role in anything Hibernia is doing now?

Bjarni Thorvardarson:
We started providing waves (2.5 or 10 Gigabit) via SDH and SONET as a product to the wholesale sector. For technical reasons including that was the technology Hibernia was built on. And also that was the product the wholesale providers relied on. They need to connect their different POPs (Point of Presence) equipment. A POP in New York, to a POP in London, that equipment relied on and called for SDH/SONET to connect the POPs.

Now as we grow into the enterprise sector, then the guys, the traders, or whoever we are doing business with – they don’t have SONET or SDH equipment. They have Ethernet equipment or equipment that calls for Ethernet protocols. So it is incumbent on us to be able to provide that without cumbersome translation from one protocol to another.

So we have since built Ethernet at the core of our protocols. Now we can offer Ethernet over SONET, which is dedicated Ether net point-to-point. And we also built, using H3C equipment, a product that we can connect customer to and point-to-point to multipoint capacity.

So moving from the SONET/SDH world to the Ethernet world, or switched Ethernet is very much what we are doing. I am right with you there that we are phasing out one world and moving to another one. Even the telco providers are increasingly moving into the Ethernet world. Especially when it comes to building out their ISP or Internet networks.

Pacific-Tier: When you see organizations like the Carrier Ethernet Neutral Exchange (www.cenx.com) and things like that popping up that are basically designing their product on the old bilateral telecommunication company design,… Do you believe that bilateral Ethernet, or that bilateral carrier relationships still have a role, or will companies like Hibernia make many of those old relationships irrelevant?

Bjarni Thorvardarson:
Hibernia, in its traditional sense, is not going to replace bilateral agreements. But bilateral agreements are going to be phased out when it comes to the exchange of Internet traffic, because exchanges are going to replace them. It is like the minutes (telephone settlement) business extremely cumbersome. If you want to build a bilateral relationship with other telecommunications providers you want to exchange traffic with through some of the voice exchanges you can do business in a matter of days.

And that is the same with the exchange of Internet traffic. If you want to do peering on a bilateral basis with companies it takes you years to build up. If you want to do it through an intermediary (such as a public internet exchange Point), clearly it is moving from the bilateral agreements to the exchanges.

Now how does that related to the price a carrier has to pay when going through the exchanges? To transit pricing? Or what have you?

And we can see where these intermediaries are actually charging less and less for the service of being in-between the delivery of the data and the content origination. You can see that in transit pricing, and how transit pricing is continuing to plummet. So I think that we are becoming less reliant on the bilateral agreement. And I firmly believe the opportunity and the necessity of getting more exchanges up and running is important.

And I think the same transition, you can see the same transition when it comes to not only Internet peering, Internet traffic is also the interconnection of Ethernet circuits, the same transition occurred that we saw 50, 60, 70 years ago when it came to voice traffic. If you wanted to make a call from London to New York you had to call an operator in London, and he made a physical cross connect to a long-distance line that terminated in New York.

The operator in New York then made the physical transition to the local tail line to the customer in New York.

That’s very much the same as when you are setting up an Ethernet circuit today. You have to build up a physical cross-connect in New York between the local tail provider and the Hibernia facility, and then in London to the tail provider over there.

With INNs and with proper Ethernet virtual cross-connects which are relying on a virtual exchange, or like exchanges that you are referring to, it’s going to revolutionize the provisioning and setup time of these Ethernet circuits. We can see a leap in that direction over the next couple years.

Pacific-Tier: One other thing I’d like to ask about Hibernia’s role in the Internet and international community in particular. I’ve been spending a lot of time in developing countries myself over the past year or so. So does Hibernia play a larger role, more than just an economic role,.. Do you also have a larger role in supporting the global community to provide a product that will bring global communications, education, entertainment, media, – can you bring that type of thing to another level?

Bjarni Thorvardarson:
I think that when you have a major company, a large company on the global economic scale, then you have to sit back an think about what your global social responsibility is to the world, and what you can contribute to the world. Hibernia is light years away from being at that size, and we can best fulfill our role now by looking for what is our economical role in this world.

Today that role is to provide large scale, high interconnectivity in all the market we operate in at a very attractive price. And by doing that we can contribute to the successful globalization commerce that will facilitate the business which will break down barriers that might prevent doing business, or from offering access to multinational companies.

That’s really what I think is our role in the world, to enable companies and people around the world be operating seamlessly as if they were sitting at two desks right next to each other (companies), and thus taking away the physical barriers of being located thousands of miles apart.

Pacific-Tier: If you look at the ideas, of say Carr’s concept of the “Big Switch” (Nicholas Carr), where telecom companies, and computing companies, and storage companies actually become nothing more than a huge, ubiquitous utility that people expect. Do you agree with that idea, or do you believe companies like Hibernia should be able to offer much higher value than the idea of a utility “big fat pipe?”

Bjarni Thorvardarson:
Well I think everyone has to know which business they are in, and that people can be in more than one business at a time. I say that from experience, because CVC (his prior venture capital company) was in the manufacturing of aluminum, and the manufacture of advanced products that were made from aluminum.

Making or smelting aluminum is very much a commodity business. The success of the business is predicated, or based on you operating a business efficiently. It’s about cost, cost, and cost. If the price per pound of aluminum that you smelt is higher than your competitors, you are out of the market. So that’s how we operated in the aluminum market.

But we also had exclusion companies. That is changing the aluminum ingot to bars that can be used by manufacturers, and be converted to door frames, and window frames, and converted into materials that could be sold to the end users or consumers.

So we were very much aware of the different needs of the value-added service market and the commodity market.

And I think the telecom business is very much the same. You have to know whether you are in the commodity market, the utility market, and there is a fair amount of utility market in the telecommunications world. I think the core of what Hibernia does is just that. It is a utility capacity between the POPs. It is providing 10G (Gigabit) capacity between London and New York, or London and Amsterdam, connecting all these high capacity markets, and it is a utility market.

You have to be very efficient in terms of how you operate your market.

Then, when you go up to the media market, or to the finance market, it is no longer a commodity market. The trader that is trading between London and New York, he does care about the price he is paying, but even more concerned with no having a second-rate service.

So you have to know which market you are operating in, and telecommunications will remain within the two markets.

Now Nicholas Carr’s concept or theory of the “Big Switch” where the world is going to cloud computing as a utility, where you plug into a socket in the wall and you are connected to a network of computing power is a noble one, and a very interesting one, and I think it certainly is going in that direction, but the difference between the bits and bytes, and the electrons that flow on the wires of the utility companies or the electric companies – it is bits and bytes of sensitive information that you do not want leaving the company or be flowing on the wires outside of the company.

So there are many challenges the “Big Switch” theory or concept. But there are a number of companies that are building up a very successful business model. Amazon being one, and a number of other companies offering cloud computing and growing extremely fast.

I am fascinated by the concept and the model of business, but I don’t think there is quite the pure cut between computing and the traditional utilities.

Pacific-Tier: Any other vision or looks into the future Hibernia may be able to share as you peer into the next 3 to 5 years?

Bjarni Thorvardarson:
I wish I could pretend to have a crystal ball, and say what our visions are, but our vision, really for the near to mid-term future is to continue our growth into the different enterprise verticals. We need to continue to service the market we comfortably define as our core markets, being North America and Europe. That’s where we will continue to focus our attention.

But we will to some extent continue to introduce new products that will leverage our network, and continue focusing on different verticals that we can also continue leveraging the network. The game for Hibernia over the next couple years is leveraging the asset. Those assets are not only our network, but also the experience of our company (employees) – the people we have, the processes, and the systems we have. Our competence and the network – that is what Hibernia is going to be not only for the next few years, but beyond.

Pacific-Tier: Do you see any potential partnerships or expansion across other parts of North America or into Asia at some time in the future?

Bjarni Thorvardarson:
Without any doubt, I am sure we will find partnerships that will benefit both parties, but it is nothing I can speak about or speculate about right now.

Pacific-Tier: Fair enough! Any final words on Hibernia, yourself personally, or what you see as interesting things happening in the market?

Bjarni Thorvardarson:
Not really, I am just enjoying working in this space, and I’m looking at a number of exciting opportunities. M&A opportunities, growth opportunities, and I am just excited to be here.

Pacific-Tier: As we all are, and thank you very much for your thoughts – it has been a great discussion.

Bjarni Thorvardarson:
Thank you! It is my pleasure!

Mr. Bjarni Thorvardarson is the CEO of Hibernia Atlantic since January 2005. Mr. Thorvardarson joined CVC, Hibernia’s parent company, in 2002 from ISB bank where he launched and managed the listed Talenta-Technology fund which focused on emerging communication and IT opportunities. Prior work experience includes investment banking at FBA bank, management of an MIS department and European Sales Director of an IT company. Mr. Thorvardarson holds an M.Sc. degree in Engineering from UW-Madison, an MBA from ISG in Paris and an M.Sc. in Finance from London Business School. Mr. Thorvardarson serves on the board of One Communications, Magnet Networks.

Read the Pacific-Tier series on Entrepreneurs and Innovators

Wiring Los Angeles Part 4 – WilCon’s Path to the Future

This is the fourth article in a series of interviews with Eric Bender, President of Wilshire Connection (WilCon), the largest independent telecom carrier in Los Angeles. In this segment Eric discusses the future of WilCon, including expansion outside of Los Angeles, wireless topics, relations with local utilities, and some great examples of WilCon’s flexibility in delivering telecom solutions to the LA community.

Pacific Tier: Outside of downtown LA , what is your expansion strategy for going to place like El Segundo, Las Vegas, or other cities, parts of the city?

Conduits Inside a Street VaultEric Bender: We’ve leased dark fiber from other carriers to get to other off-net locations such as El Segundo. We connect Equinix on Maple, so we can do lit transport into that facility. We’re working on a plan that would extend from there to the 365 Main location in El Segundo, and then with a second route back to downtown.

We’ve acquired several 10Gbps wavelengths from LA to Las Vegas, and into the Switch NAP facilities, so we can connect with all their switches and facilities, including the Super NAP, which is a 400,000sqft monster of a data center facility. So we do just backhaul or transport to Vegas. We’ve got approximately 10 gigs of traffic back and forth between here and there now.

I don’t see us building on a long haul type arrangement where we would build far outside of downtown LA. We’ll go off-net by structuring deals with existing customers and partners that we have to get fiber and connectivity into other locations. One of the things we are working on is connecting our buildings in LA to Phoenix, there’s a couple of different data centers in Phoenix – there is I/O, (etc).

Then we might go from Phoenix to Las Vegas, and from Las Vegas we already have to LA (which we already have), but also Las Vegas to San Jose/Santa Clara, and then back down to LA. So we basically have a ring from LA to Phoenix, to Las Vegas, San Jose, back to LA.

Pacific Tier: So would you have arrangements with Edison (Southern California Edison), or Burbank Power, or people like that to extend into Hollywood or Burbank or Glendale?

Eric Bender: Well, we do work with Edison and DWP (Los Angeles Department of Water and Power), and have used them to get to further out locations from downtown. The city of Burbank, we’ve not been successful and have not really chased hard to try and make a deal with them. Our initial efforts to work with them were rebuffed without comment – they had no interest in working with us, which surprised me.

I think we can be creative in building, like at 900 N. Alameda which we now have connected with dark fiber, and there is a variety of ways we can do that. We have connection and interconnection agreements AT&T to use their infrastructure so we can theoretically deploy assets throughout the city and state in coordination with AT&T’s network.

Pacific Tier: Would you consider for example going to San Diego using a capacity swap with Edison where they gave you a pair of fibers to San Diego?

Eric Bender: We do that already, well not to San Diego, but we have done a variety of barters, trades, or however you want to call it, with companies like Edison. With Equinix, with DWP I’d like to – but they have no interest and can’t by law (they are a public utility), and can’t really do that. Level 3, we’ve had discussions with them to do the same thing.

Because I’ve got all this infrastructure, whether it’s the conduit, or the fiber, and it’s all there, it’s paid for, there’s no debt, there’s no cost of capital to me – it’s there, so if I need to allocate a couple pairs of fiber going here or there on my network, and I can get something that I can generate revenue off of on somebody else’s network – we’ve done that many, many times and it’s worked out very well with a variety of companies.

Pacific Tier: Expanding a bit more on WilCon’s flexibility…

Eric Bender: We have no rigidity with anything we do. I think that one of the reasons we’ve been successful, and have continued to grow every month, basically where we’ve continued to grow over the previous month is because we’re not rigid and have a straight line approach to things.

Servicing WilCon's ConduitI think the advantage of not being a telecom guy by trade or training and education is that we look at things by how do we get the customers the things they need? When a customer needs something my first response is to say “OK,” and the second response is to call my technical guys and say “I just committed us to doing this by a certain point in time, make it happen.” And they do.

We’re a small company so we don’t have all the various layers of engineering groups and planning groups to say from here it has to pass from point “A,” to point “B,” to point “C,” before you can be told it can be done, or not be done, or how to do it. We just make it happen and get it done.

The story I tell that kind of tied the real estate to the telecom all together well for me is at one of our buildings, 700 Wilshire we had a t4enant in the building. This goes back to about early 1999. We had put conduit in the ground already, so it was probably early 1999, had already connected the conduit to One Wilshire, as well as 700 (Wilshire Blvd), and we had leased space to this tenant.

Part of their requirement was they needed a couple of DS3s between the meet me room at One Wilshire and their premises they had leased at 700. At that point we were only doing conduit deals with our carrier customers. They were going to Level 3 for this DS3, and Level 3 hadn’t really built out their network yet. Because we put Level 3’s first conduit in the ground in LA (it was conduit we’d installed in our trench and gave them, basically), and Level 3 was provisioning this DS3 through PacBell (now AT&T) or someone, and they were re-provisioning (the DS3) through them.

The timing for the tenant was going to be 90 days, or 120 days wait for the tenant to provision this DS3. The tenant came to us and knew we had started Wilshire Connection. So they came to us and said “we really need this, is there anything you can do for us?”

I didn’t know what to do, I mean we just had conduit between the buildings. So the customer said “all you have to do is this. We have the equipment, you’ve got fiber cable already (because we’d already deployed that), you need this piece of equipment, you’ve got this box on this end, that box on that end, and we’ll tell you how to do it and what works well.”

Two weeks later we had that DS3 traffic up and running between the buildings. So we got into the transport business. But that was because we were flexible, somebody needed something, (we had the resources), and we got it done.

If we can see there’s a reasonable return on investment and payback on the infrastructure or cost to deploy new equipment, we’ll do it. We got all this MRV DWDM equipment (dense wave division multiplexing) because we had a very limited amount of fiber into the (particular) building, so the only way I could provision this service (lit bandwidth) was with the DWDM gear – so we went out and bought it.

So the equipment was paid for within a year. The customers can come and go – it doesn’t really make any difference (other than we want to keep customers), because it is already paid for. So we’ll continue to do that, and that’s how we’ve continued to grow into the buildings that we have, as well as the transport circuits that we do between all these buildings, as well as connecting into Vegas, and the same to Phoenix, Santa Clara, and other cities.

Pacific Tier: So what is the future of WilCon, where do you go from what you have today?

Eric Bender: several years ago we had some strategy discussions, and (since) we are a dominant carrier in LA, we’re known around the world as the provider of choice, the connector of choice here in LA. We have a great relationship with Equinix, and CoreSite now (CRG), so we get a lot of referral business from those guys. We’ve provided such good service to our customers that big carriers such as TATA, PCCW, and others that they just come straight to us when they have new requirements.

So we had this discussion “let’s go and expand, and into other markets” and we took a very quick and dirty analysis and determined that it costs so much to build out and deploy in these other markets, and we know other companies have been doing similar kinds of things, and we might be providing a similar type of service. In order to do that (expanding into other markets) we’d have to take on an additional equity investor, or venture funding, and then all we’re doing is servicing somebody else’s debt.

So, does that make sense (expanding) or should we continue to be the provider here, and continue to grow which has made us a profitable company. So we said “we are gong to do that, and we’ll expand and provide services outside of LA in various ways that make sense, and does not require a huge capital outlay, and we’ll do some – but we’re not going to spend millions of dollars to deploy.”

The reason many companies went bankrupt was because they built these massive networks, and there was no r4evenue to service the debt on them.

So our future is to grow in selected and strategic ways. It will probably be between LA and other cities by usi9ng lit waves, and then establishing a POP (point of presence) in a couple buildings, and then maybe in some cases building a bit in those markets to connect those buildings.

In Dallas we’ve looked at buying an existing network, but there’s really no need to connect between those facilities – there’s no business with that. Chicago was no different. San Francisco, San Jose, Santa Clara are all so spread out its hard to justify a new build or new construction.

I think working with companies like Allied Fiber which is building their long haul network, and working with them to be the local metro piece to what they’re selling to their customers on the long haul – we can tag along with them, working well with them in LA, and possibly duplicate that with them in other markets.

Pacific Tier: Is there any wireless in WilCon’s future?

Eric Bender: I don’t really see that. We did look at partnering with some wireless, WiMAX, or other guys using Terrabeam or some of these free space optics. It seemed like they just had such a long way to go before it would make it worthwhile, and that in LA maybe it’s a problem. Trying to negotiate the line of sight, and rooftop antenna rights and those things,… that we didn’t really get anywhere. It became cost prohibitive I think.

So it would be nice to do something like that. We looked at that (wireless) to support NANOG (North American network Operator’s group) several years ago when they were here. We were in Library Tower (the US Bank Building) on the 56th floor. I had a line of sight directly from my window to the hotel, but it was too far. We would have needed a couple of hops to be able to deliver a gig (Gigabit connection), and it was a disaster for them to have it there (at that hotel).

They had to go buy a DS3 from AT&T (PacBell). If they had done the conference downtown I could have connected them with no issues.

Pacific Tier: Any final words you would like to say to the global telecom or Internet community, or Los Angeles? Why LA, or with WilCon, downtown LA is an attractive place to do business?

Eric Bender: We’re number one!

I think LA is a great place to do business. From a location perspective its perfect, as its the gateway to the US. You can get anywhere from LA, whether its physical transportation, or from a telecom infrastructure perspective. I think if you come to LA as a carrier, as a provider, as a bandwidth user, in this industry – and you’re going to locate in one of the half a dozen facilities (in downtown LA), whether an Equinix, Level 3’s gateway, One Wilshire, or TELX Carrier center.

Any of the main big facilities here (and Switch and Data) you are going to need, ultimately, to connect to customers that are in one of the other facilities – somehow you’re going to find us. You may be referred to us, do your research – ultimately you are going to find we can provide that connectivity.

We are really easy to do business with, as any of our customers will tell you. And I think that in all the years we’ve been in business, since 1998, I think there are maybe two deals we’ve lost because of price, that we were not the lower cost provider for that particular deal. And, we try to provision within 24~48 hours (of an order).

Pacific Tier: That is outstanding. Words for the global telecom and internet community from Eric Bender, President of Wilshire Connection. Thank you very much.

John Savageau for Pacific Tier Communications, Long Beach

====

This concludes our interview with Eric Bender, President of Wilshire Connection in Los Angeles.  you can contact Eric at ebender@wilcon.com for more information on Wilshire Connection.

The entire interview is available online.

Previous entries in this series include:

  • Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
  • Part 2 – WilCon Takes to the Streets
  • Part 3 – WilCon Manages Infrastructure Risks

Wiring Los Angeles Part 2 – WilCon Takes to the Streets

This is the second part of an interview with Eric Bender, President of Wilshire Connection. In this segment Eric talks about the period in 2000 preceding the Democratic Nation Convention, and the aggressive industry build out of conduit, fiber, and telecommunications infrastructure in the downtown Los Angeles area.

Pacific Tier: At what point do you think the city of Los Angeles figured out this would be a really good thing for LA, and it would bring more business and money into downtown?

Eric Bender: I don’t think they ever came to that conclusion. What happened was in 2000 the Democratic Convention was going to be down at the Staples Center in downtown, so I think it was in early 2000 or at the very end of 1999 the city called a meeting and notified all the telecom carrier that had been active or Eric Bender President Wilshire Connectionbuilding or doing things that as a part of the preparations for the convention that they would be repaving the streets, resurfacing certain streets, and there was going to be an absolute five year moratorium on any digging or street construction work as a result of this. They wanted the streets to look pretty on TV, and that was fine.

So I don’t think they thought about how much money, or how beneficial the telecom network was going to be to the city, they realized they would need to do something for this convention. The result of that was that it threw somewhat of a panic into the carriers that were in the infancy of building out their networks, and building throughout LA, and it forced a massive infrastructure improvement project, because there was a very limited amount of time before the city was going to shut everything down.

I think the convention was in July or August, and by the end of May everything had to be done. So, we really had about four or five months to build everything and our network probably tripled in size during that period of time. We were one of the only companies that had an active permit that we had pulled for a small segment that we were building that hadn’t started construction yet because it was not that critical to us at the time.

But because of us having an active permit for about a two block, few hundred feet, maybe five or six hundred feet of conduit the city realized they had a major problem that these companies had to be building, had to be constructing – they couldn’t just shut them down for five years. The city did understand that was a problem.

So they said you can build, we will expedite and streamline the process and anyone who has an active permit, well everyone can tag along with that permit. We’ll just change it and build out from there.

So, with our little permit, and the little segment, I think it was on Grand from Wilshire down to 7th St., and then a little bit more. We basically parlayed that and built it up to 5th St., then up to Figueroa, and literally tripled our network. So we became the lead builder from that permit on a tremendous amount of other conduit that was built in the city.

Basically they couldn’t keep up with all the activity because there were intersections in downtown LA where there were three backhoes digging for three separate projects. We were doing one, QWEST was doing their own, they didn’t really want to participate much with anyone, and MCI, MFS didn’t want to share conduit so they were doing little things.

And I remember at 7th and Grand there were three different construction crews and project going on at the same time, and for me it was just a wonderful time because we had this permit, so we had a lot of leeway, and I would walk the streets, stand on the street corner, and these other companies, you know Level 3 and QWEST actually did participate, XO… You know I would say “we’re going down this street to this building,” and they’re going “I’m going that way,” and we’d shake hands. “I’m taking two conduits in yours, you’re taking a conduit in mine…”

I don’t like to use the term “wild, wild, west,” but it was really a very wild and fun time. My best times were just walking around with all the construction going on, with all the lane closures and all the activity. The traffic was backed up, cars were beeping, and would be driving by and I would just be standing there looking at the big hole in the ground with a big smile on my face. It really was the best time.

Pacific Tier: Did you have any catastrophic backhoe cuts or anything disruptive during consutruction? 

Eric Bender: We, during our construction our guys, we never had an incident where we hit any other conduit. There was a water or a sewer line that got hit, and nicked, and that was something that that was not actually on the infrastructure plan. And it wasn’t marked on the street so it wasn’t our problem so much.

There was a close call with a gas line. But nothing happened. But for the most part (Eric knocking on his wooden desk) they did a good job.

The toughest part to build, which we were actually the participant in and XO was the lead builder on 7th, I think somewhere between Olive and Grand, it shows you the history of LA but, back when they had the street cars, the RED Line, YELLOW Line, and all these other street cars, they never really demoed (demolished) those out.

They just over the years paved over and over and over. As they were excavating and going down the road, the path where they were going, and which later became obvious because of the railroad, they had to rip through all the railroad ties. It was terrible.

Because of that, then they had to get an archaeologist because you know you have bricks from the support for the railroad, because they are historical. I don’t know what they would do with it, it is in the middle of 7th St (a major road in downtown LA) and Grand Ave, that can’t, you know set up a museum in the middle of it, but it shut the project down. The city had some very peculiar rules that you cannot dig on Sundays that go back to the old blue laws basically, no construction activity on Sundays.

Some streets they won’t let you work during the week, you know it goes back to the days when downtown was a main shopping area, so you know, so you couldn’t dig during the week. You could only dig on Saturdays from nine (a.m.) till three (p.m.) which means you get about from ten till two because you have clean up and everything.

So some projects that should have been done within a week took like eight weeks to get done. So the city tried, but they were not the easiest. The only company that got to work on Sunday was QWEST, because, well, nobody is really sure exactly what happened or how, but they did have several projects where they were working on Sundays.

====

This ends part two of the “Wiring LA” interview series. Part Three will explore some of the risks of building such a dense concentration of telecom projects within the downtown LA area.

The entire interview is available online.

Previous entries in this series include:

Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection

 

John Savageau, Long Beach

 

Wiring Los Angeles with Eric Bender, President of Wilshire Connection

Downtown Los Angeles is among the most densely connected telecommunication hubs in the world. A dozen buildings in LA’s city center house the world’s largest Internet networks, Laying Down Telecom Conduits and Cabletelecommunication carriers, content management networks, and entertainment companies – interconnected through a complex mesh of submarine fiber optic cables, terrestrial cables, and internet exchange points.

With more than 500 networks and carriers operating within the LA city center, one company stands out from the crowd as a leader in bringing the global telecommunications and Internet community together. Eric Bender is president of Wilshire Connection, a facility-based carrier focusing on providing neutral, high capacity fiber optic cable interconnecting the most important buildings in Los Angeles.

Wilshire Connection (WilCon) has a vision to free the LA telecom and Internet-enabled community from the burden of developing a highly meshed inter-building infrastructure, allowing each company to focus on bringing value to their global network interconnections and relationships. Eric Bender, President of Wilshire Connection, is the man behind WilCon’s dramatic success in wiring Los Angeles.

This interview will cover, in a multi-article series, the history of modern telecommunications in downtown Los Angeles, the role Wilcon played in LA’s redevelopment, the period of rapid and chaotic build out prior to the Democratic National Convention in 2000, the risk of high density telecom infrastructure, and the future vision of Eric Bender and Wilcon.

Pacific Tier: Today we have Eric Bender, President of Wilshire Connection. Eric, can you tell us a little about yourself, and how Wilcon came around?

Eric Bender: Sure, we founded Wilcon, Wilshire Connection back in 1998. I am one of the founders and one of the partners, and have run the company pretty much from the beginning. I’m a real estate guy by background. I graduated from college and have always worked in the real estate field, whether its commercial property, management, brokerage, home build, whatever…

Back in late 1996 I started working for a Hong Kong based family which was acquiring some office buildings in downtown LA, one of which was 611 Wilshire Blvd which was directly across the street from One Wilshire. Back at that time telecom deregulation had just been passed, the 1996 Deregulation Act, and there was a boom of CLECs formed and started. Nextlink, the precursor to Level 3 (Kiewit and Sons), MFS was expanding into a variety of things, Worldcom, a variety of these companies were just starting up in that time frame.

For a lot of different reasons One Wilshire had become kind of a focal point for telecom, primarily because it is a 30 story building, had very clear line of sight to the east, and back in the day when microwave transmission was the primary factor, (One Wilshire) had great eastward pointing microwave capabilities. And because of this many of the deregulated companies had got into One Wilshire.

Because our building 611 Wilshire was directly across the street, and only 50% occupied, and One Wilshire was still an office building that only had a meet me room component to it, – that was really it, from a real estate side of things we said “we have all the space, you have all these new companies coming and can’t do it, send them over to us, we’ll fill our building with them, and let’s put some conduit between the buildings together as a virtual extension of (the meet me room) of One Wilshire.

Additionally we put in some conduits that the One Wilshire building actually provided for.

A year or two later we had bought, and were converting other buildings in downtown (700 Wilshire, 818 W. 7th St.) and saw a need to have those buildings connected back to 611 and One Wilshire.

Standing on the street corner thinking about it we decided that we should become a facilities-based carrier. We went and got our authority from the state to become a carrier, CLEC, and I can’t remember the other authorities we received. Then we decided that’s how we are going to do it. We started digging the streets building the network and connecting the buildings, our own buildings, together.

We decided as a way to fund the build out, because at the time, 1997 and 1998 a lot of buildings had empty space in downtown LA, they all saw what was happening not only with our buildings, but other buildings that had empty space, and saw how telecom was taking massive amounts space. And the reality was landlords wanted telecom companies as tenants because they paid higher rent, they paid for their own improvements to build out their space, they paid for their own utilities, the buildings did not have to provide janitorial, nor provide much in the line of services.

So we created, as part of our model with Wilshire Connection, the ability to connect all these other buildings into our network. This tied them back to One Wilshire and all these other telecom buildings. And that was our methodology of raising funds to build our improvements to our infrastructure.

Pacific Tier: So what role do you believe Wilcon played in the development of downtown LA as a telecom center?

Eric Bender: Well, trying to be as modest as I can, we probably played the primary role in that, because, several buildings are telecom buildings today because we were building our network and including them. They may have become a telecom building in some way differently, but the fact that we connect them into our network which covers and ties all the building together on a neutral basis was really the instrumental part, including 600 W. 7th St, 800 Hope, 818 W. 7th St., which has Level 3’s gateway, Equinix, and others – brought these buildings back to One Wilshire and to each other – together they became the success that they are.

Pacific Tier: What was your criteria for selecting the buildings that you POPed (built a point of presence) or entered, or dropped cable?

Eric Bender: We were note really picky. At the time we were building the network it depended on which buildings came to us, what they were willing to contribute to the cost of (build out), and how important it was to them to be connected to the network.

Pacific Tier: The actual process of laying conduit in downtown Los Angeles, did the city support you?

Eric Bender: Not initially, it was quite challenging. Part of it was because a lot of this activity was new and we were building our network, planning our network to cnnect the buildings. But you had the other companies that were building their networks, such as Level 3, MFN (actually they came a bit later), MCI was adding a lot more conduit and buildings, and others. The city was not prepared, I don’t think, from an engineering perspective and plan check to deal with all that, so they were backed up, and you had companies like Level 3 which was building a network throughout the entire city basically.

The city’s engineering department that reviewed the plans – you would go in and the entire office would be stacked four, five , six feet high with plans of what they were planning to build. And it took months and months, 6 to 8 months at times.

This is the end of part 1 of our interview with Eric Bender. Additional segments to the interview will discuss the actual buildout of Wilscon’s network prior to the Democratic National Convention in 2000, the risk of having such high density infrastructure in downtown Los Angeles, and future plans for Wilcon and Eric Bender.

====

John Savageau, Long Beach

You can listen to the entire interview online at Pacific Tier

 

Telecom Risk and Security Part 4 – Facilities

A 40 year old building with much of the original mechanical and electrical infrastructure. A 40 year old 4000 amp, 480 volt aluminum electrical buss duct, which had been modified and “tapped” often during its life, with much of the work done violating equipment specifications. With the old materials such as buss insulation gradually deteriorating, the duct expanding and contracting over the years, the fact aluminum was used during the initial installation to either save money or test a new technology vision – it all becomes a risk. A risk of buss failure, or at worst a buss failing to the point it results in a massive electrical explosion.

Facility ExplosionSound extreme? Now add a couple of additional factors. The building is a mixed use-telecom carrier hotel, with additional space used for commercial collocation and standard commercial office space. This narrows it down to most of the carrier hotel facilities in the US and Europe. Old buildings, converted to mixed-use carrier hotel and collocation facilities, due mainly to an abundance of vacant space during the mid-1990s, and a need for telecom interconnection space following the Telecommunications Act of 1996.

Over the past four years the telecom, Internet, and data center industry has suffered several major electrical events. Some have resulted in complete facility outages, others have been saved by backup systems which operated as designed, preventing significant disruption to tenants and the services operated within the building.

A partial list of recent carrier hotel and data center facility outages or significant events include some of the most important facilities in the telecom and Internet-connected industry:

  • 365 Main in San Francisco
  • RackSpace hosting facilities in Dallas
  • Equinix facilities in Australia and France
  • MPT in San Jose
  • IBM facility in NZ
  • Fisher Plaza in Seattle
  • Cincinnati Bell

And the list goes on. Facilities which are managed by good companies, but have many issues in common. Most of those issues are human issues. The resulting outages caused havoc or chaos throughout a wide range of commercial companies, telecom companies, Internet services and content.

The Human Factor in Facility Failures

Building a modern data center or carrier interconnection point follows a fairly simple series of tasks. Following a data center design and construction checklist, with strict compliance to the process and individual steps, can often mean the difference between a well-run facility and one that is at risk of failure during a commercial power outage, or systems failure.

In the design/construction phase, data center operators follow a system of:

  • Determining the scope of the project
  • Developing a data center design specification based on both company/industry standards
  • Designing a specific facility based on business scope and budget, which will comply with the standard design specification
  • Publish the design specification and distribute to several candidate construction management companies and engineering companies
  • Use a strong project manager to drive the construction, permitting, certification, and vendor management process
  • Complete systems integration and commissioning prior to actual operations

Of all the above tasks, a complete commissioning plan and integration test is essential to building confidence the data center or telecom facility will operate as planned. Many outages in the past have resulted from systems that were not fully tested or integrated prior to operations.

Facility ChecklistAn example may be a breaker coordination study. This is the process of ensuring switch gear and panel breakers from the point of electrical presentation by the local power utility down to individual breaker panels are set, tested, and integrated according to vendor specification. Without a complete coordination study, there is no assurance components within an electrical system will either operate correctly during normal conditions, or operate correctly during equipment failures. An essential component of a complete systems integration test. Failure to complete a simple breaker coordination study during commissioning has resulted in major electrical failures in data centers as recently as 2008.

The InterNational Electrical Testing
Association (NETA) provides guidance on electrical commissioning for data centers under “full design load” conditions. This includes testing recommendations to test performance and operations including the sequence of operations for electrical, mechanical, building management systems/BMS, and power monitoring/management. The actual levels of NETA testing are:

  • Level 1- Submittal Review and Factory Testing
  • Level 2- Site Inspection and Verification to Submittal
  • Level 3- Installation Inspections and Verifications to Design Drawings
  • Level 4- Component Testing to Design Loads
  • Level 5- System Integration Tests at Full Design Loads

No company should consider collocation within a facility that cannot produce complete documentation that integration testing and commissioning was completed prior to facility operations – and that testing should be at NETA Level 5. In some cases, documentation of “retro” testing is acceptable, however potential tenants in a facility should be aware that is still a compromise, as it is almost impossible to complete a retro-commissioning test in a live facility.

Bottom Line – even a multi-million dollar facility has no integrity without a detailed design specification and complete integration/commissioning test.

The Human Factor in Continuing Facility Operations

Assuming the facility adequately completes integration and commissioning at NETA Level 5, the next step is ensuring the facility has a comprehensive continuing operations plan to manage their electrical (and mechanical/air conditioning) systems. There are two main recommendations for ensuring the annual, monthly, and even daily equipment maintenance and inspection plans are being completed.

Computerized Maintenance Management System (CMMS)

Data centers and central offices are complex operations. Thousands of moving parts, thousands of things that can potentially break or go wrong. A CMMS system tries to bring all those components together into an integrated resource that includes (according to Wikipedia)

  • Work orders: Scheduling jobs, assigning personnel, reserving materials, recording costs, and tracking relevant information such as the cause of the problem (if any), downtime involved (if any), and recommendations for future action
  • Preventive maintenance (PM): Keeping track of PM inspections and jobs, including step-by-step instructions or check-lists, lists of materials required, and other pertinent details. Typically, the CMMS schedules PM jobs automatically based on schedules and/or meter readings. Different software packages use different techniques for reporting when a job should be performed.
  • Asset management: Recording data about equipment and property including specifications, warranty information, service contracts, spare parts, purchase date, expected lifetime, and anything else that might be of help to management or maintenance workers. The CMMS may also generate Asset Management metrics such as the Facility Condition Index, or FCI.
  • Inventory control: Management of spare parts, tools, and other materials including the reservation of materials for particular jobs, recording where materials are stored, determining when more materials should be purchased, tracking shipment receipts, and taking inventory.
  • Safety: Management of permits and other documentation required for the processing of safety requirements. These safety requirements can include lockout-tagout, confined space, foreign material exclusion (FME), electrical safety, and others.

And we can also add additional steps such as daily equipment inspections, facility walkthroughs, and staff training.

SAS 70 Audits

The SAS 70 Audit is becoming more popular with companies to force the data center operator to provide audited documentation by a neutral evaluator that they are actually completing the maintenance, security, staffing, and permitting activities as stated in marketing and other sales negotiations.

Wikipedia defines a SAS70 Audit as:

“… the professional standards used by a service auditor to assess the internal controls of a service organization and issue a service auditor’s report. Service organizations are typically entities that provide outsourcing services that impact the control environment of their customers. Examples of service organizations are insurance and medical claims processors, trust companies, hosted data centers, application service providers (ASPs), managed security providers, credit processing organizations and clearinghouses.

There are two types of service auditor reports. A Type I service auditor’s report includes the service auditor’s opinion on the fairness of the presentation of the service organization’s description of controls that had been placed in operation and the suitability of the design of the controls to achieve the specified control objectives. A Type II service auditor’s report includes the information contained in a Type I service auditor’s report and also includes the service auditor’s opinion on whether the specific controls were operating effectively during the period under review.”

Many companies considering outsourcing within the financial services industries are now considering a SAS70 audit essential to considering candidate data center facilities to host their data and applications. Startup companies with savvy investors are demanding SAS70 audits. In fact, any company considering outsourcing their data or applications into a commercial data center should demand to obtain or review SAS70 audits for each facility considered.

Otherwise, you are forced to “believe” the words of a marketer’s spin, a salesman’s desperate pitch, or the words of others to provide confidence your business will be protected in another company’s facility.

You Have the Best Data CenterOne thing to keep in mind about SAS70 audits… The audit only reviews items the data center operator chooses to audit. Thus, a company may have a very nice and polished SAS70 audit documentation, however the contents may not include every item you need to ensure the data center operator has a comprehensive operations plan. You may consider finding an experienced consultant to review the SAS70 document, and provide any additional guidance on whether or not the audit actually includes all facility maintenance and management items needed to ensure continuing protection from mechanical, monitoring/management, electrical, security, or human staffing failures.

Finally, Know Your Facility

Facility operators are traditionally reluctant to show a potential customer or tenant their electrical and mechanical diagrams and “as-built” documentation for the facility. This is the point you would find a 40 year old aluminum buss duct, single points of failure, and other infrastructure designs and realities you should know before putting your business into a data center or carrier hotel.

So, when all other data center and carrier hotel facilities appear equal, in geography and interconnections, look at facilities which will incur the least impact if your interconnections are disrupted, and demand your candidate data center operator and hosting provider are able to provide you complete documentation on the facility, commissioning, CMMS, and SAS70.

Your business, the global marketplace, and network-connected world depend on forcing the highest possible standards of facility design and operation.

John Savageau, Long Beach

Other articles in this series include:

Telecom Risk and Security Part 3 – Human Factors

An employee enters the meet-me-room at a major carrier hotel in Los Angeles, New York, or Miami. He is a young guy recently graduated from high school, hired to do cable removal for circuit disconnects at minimum wage. Although young, he has a wife and child, and has recently been fighting with in-laws over his ability to support a family. Frustration and anger overcome his emotions, and he turns to the ladder rack jammed with cable and starts hammering at the cables for all he is worth.

Network operations centers around the world see circuits dropping, and customers with critical financial, military, Internet, and broadcast news services are shut down. In the space of about one minute our young employee has taken down several thousand individual circuits, creating near chaos in the global telecommunications community.

In their report on Trusted Access to Communications Infrastructure, the NSTAC Vulnerabilities Task Force advises “”it is important to recognize that any one individual with malicious intent accessing any critical telecommunications facility could represent a threat. The threat of insiders performing malicious acts also transcends each type of site discussed in this document.”

Security in TelecomThe event noted in part 2 of this series describing the outages in Northern California following damage to a manhole housing telecom was real. The resulting disruption to regional communications was a wakeup call to the telecom community, law enforcement, and communities affected. It is clear the perpetrator knew what he was doing, and knew exactly what vulnerabilities the major telecom companies had which he could exploit.

There have been many other cases such as Level 3 Communications loss of a major core router in 2006 supporting regional Internet services in London due to theft, a break-in at BT’s switching facility in Birmingham during the same period resulting in the loss of thousands of telephone lines, showing this is not just an American problem, but a global vulnerability.

The message is clear, as an industry our most obvious threat to information and communications security is not a natural disaster, it is people with industry knowledge or access to our critical facilities.

The Telecom and Data Center/Carrier Hotel Industry’s Role in Managing Human Security Risks

Data centers and central offices are in a constant state of change, maintenance, and growth. While facility network operations staff are generally long term employees, with a history of employment and performance, many others entering our data centers are not well known to the landlord.

Janitorial and maintenance staff are normally contracted to vendors, mechanical and electrical workers are contracted to maintenance and engineering companies, and construction contractors often use temporary staff from agencies such as “Labor Ready” and other day labor companies. In most cases data center or landlord employees are given a cursory background check prior to employment, however others entering even critical areas within the data center or central office meet-me-room may be entirely unknown to the facility.

While normally under some level of supervision, or access management, contractors, maintenance people, and even data center tenants are often free to move around the facility without direct security observation. As shown above, it would only take an angry, disgruntled, or undisciplined person seconds to cause a major calamity in our global communications system.

In a worst case, that person may be a terrorist with a detailed plan to cause damage to the facility once given even minimal access. High voltage electricity, water systems from cooling infrastructure, or access to switching equipment and cable interconnections are all exposed within the data center, and any element could be used to cause a major disruption within the meet-me-room or data center.

Most carrier hotels are located in “mixed-use” buildings, in high-rises with additional tenants who may not even be in the data center or telecom industry. This compounds the problem, as those tenants are often reluctant to comply with security and access requirements at the level of a critical telecom facility.

The issue becomes even more acute when we realize that much of the infrastructure supporting carrier hotels transits “risers” between floors, often through floors occupied by non-telecom tenants who may have physical access to riser space within their offices.

Secure Your Manhole CoversThere are a few data centers within the United States where security is comprehensive enough to reduce the risk of malicious intent to a very low level. While many tenants find the access and supervision within the facility extreme, facility resources are protected from all but the most aggressive vandalism or attack.

The NSTAC recommends that in the US the telecom industry establish best-practices guidelines to screen personnel prior to unescorted or unrestricted access to critical facilities, such as carrier hotels and carrier central offices. This may include a national agency check to ensure the person requesting access does not already have a profile indicating they could potentially be a threat to the facility.

The US government may give this additional support, as much of the US government, state, and local communications services are supported either in carrier central offices or commercial carrier hotels.

Recommendations for the Communications Industry

While it is clear not all persons entering a data center or carrier hotel facility can be completely screened, there are tasks each carrier and commercial data center operations should complete. Those could include:

  • Complete background checks for all direct employees
  • Pre-employment screening which would include a personality profile (indicating if they are in a high risk category for emotional stress)
  • Supervision of all contractors on site by a direct company employee who is aware of the risk posed for each type of equipment in proximity to the contractor (such as electrical equipment <UPS, breaker panels, switchgear, chilled water pipes, etc>)
  • Training in situational awareness – being able to identify activities not normal for others in your facility
  • Cooperation with law enforcement and other agencies
  • Working with industry groups to create and follow an industry “best practices” for facility security and human resource management
  • Ensure at least in the streets and areas immediately adjacent to the facility all manhole covers and utility entry points are locked and secured, preventing persons from accessing telecom, electrical, and water infrastructure supporting the building

“Unfortunately our most likely enemies will throw explosives into unguarded cable interconnect rooms or drop cans of petrol into unlocked manholes. End of Cyber War. You might characterize this as the provenance of a 23 year old fundamentalist Skywalker with a cell phone modem and a wild-eyed cousin in Munich figuring out how to blow up the Internet Death Star and stop Predator attacks on his village. Totally asymmetric dude! (From Bob Fonow’s “The Death Star?: Cyber Security vs. Internet security”)”

The commercial operators of data centers and carrier hotels have a tremendous responsibility not only to their owners and shareholders, but also the global telecom community and global economic community. The potential impact, even in the short term to a malicious attack on a meet-me-room at One Wilshire, 60 Hudson, the Westin Building, Telehouse in London, or the NAP of the Americas would be immediate, and extremely disruptive.

Human factors are the threat. Let’s not forget the lessons learned over the past couple years, and keep diligent, have good human situational awareness, and understand the sense of urgency we must apply to ensuring our communications infrastructure is secure.

Let us know your opinions, experiences, and recommendations

John Savageau, Long Beach

Previous articles in this series

Risk and Security in the Telecommunications Industry Series – Part 1

The worst case scenario – a strong earthquake strikes California, disabling the carrier hotel at One Wilshire, disrupting operations at submarine cable landing stations in both the Los Angeles area and central California, with a resulting tsunami hitting Hawaii, Guam, the Philippines, Taiwan, and Japan.

LA Hit by TsunamiCommunications are severed to most of the South Pacific, and severely degraded to allow for only emergency services and national defense usage within the west coast of the United States. Financial and government communications are disrupted and severely limited into Japan, Hong Kong, and China.

Telecom carriers in Singapore, Japan, Hong Kong, China, and Australia work frantically to restore cable, Internet, and telecom capacity from the Pacific submarine cable systems through the Indian Ocean to Europe and the US east coast. Seattle and San Francisco still have some connectivity, however cable systems from Grover Beach to San Diego are inoperable, limiting connections to those which were designed with automatic rerouting through North Pacific cable systems.

Sound crazy? No, it is not crazy, and there is a very good possibility a similar scenario will occur within our lifetime. In fact, when you look at the concern raised when the recent Los Angeles “Station Fire” threatened the telecommunications facility at Mt. Wilson many people were surprised at the potential disruption to both civilian and government communications if that facility were destroyed.

Los Angeles law enforcement uses the transmission towers to manage emergency communications throughout the LA area, fire departments, AM/FM radio stations, digital broadcast television stations – many were single threaded through Mt. Wilson as their primary local communications infrastructure. Not to mention the three letter federal agencies which use the facility for, well whatever they use it for…

Not a New Problem

Several US agencies have looked at this problem for many years. Agencies addressing the problem include the National Communications System (NCS), the Federal Communications Commission (FCC), the National Reliability and Interoperability Council (NRIC), the Department of Homeland Security (DHS), and an additional continuing special taskforce mandated by the president called the National Security Telecommunications Advisory Committee (NSTAC).

As recently as four years ago, an NSTAC report concluded “the telecommunications industry has shown that it is unlikely that a loss of assets in a single telecom hotel would cause a nationwide disruption of the (USA) critical telecommunications infrastructure.” Which may be true for the US infrastructure, as all major American carriers are interconnected at numerous locations scattered across the United States. In short, while the local LA community may be seriously disrupted in the event of the big earthquake, communications between Miami and New York would still be possible with little disruption.

AT&T, Sprint, Verizon, and QWEST are all well-meshed in their networks. As long as they are not sharing the same cable routes, or even in some cases the same actual cables, if the companies are subcontracting their long distance or local loops from other wholesale cable companies such as Level 3, XO, or Time Warner Telecom.

The International Factor

Ten years ago the United States could stand alone in our communications infrastructure. International communications were strong, and submarine cables were in use, however much of the international communications infrastructure was still done through use of satellites. Even if a submarine cable was disrupted, carriers could easily restore their communications through use of existing satellite restoral and recovery agreements.

Now, in the Internet age of high capacity telecom infrastructure, generally provisioned in multiples of 10 Gigabit per second links, satellite capacity has quickly become a fraction of the bandwidth driving international communications. Even the old telephone networks are being integrated by international and US carriers into their Internet infrastructure, often sharing the same circuits are streaming media, social networks, general web traffic, and other entertainment applications.

This will not be as easy to restore in the event California gets the big earthquake we all know is coming.

The Risks and Vulnerabilities Series

This series will look at several aspects of the telecommunications business, including:

  • International telecom vulnerabilities
  • Government interest, activities, and opinions on the risks and vulnerabilities of both US and international communications infrastructure
  • The role of the carrier hotel and internet exchange point in international communications
  • Interviews with people on the front lines of communication security
  • Recommendations for both the telecommunications industry, and the global user community

I look forward to reader comments, critiques, flames, constructive recommendations, and other ideas related to this discussion. Please add your comments to this blog, and I will ensure your voice is both heard and considered.

John Savageau, Long Beach

%d bloggers like this: