A Developing Country That Can Teach Hawaii An IT Strategy Lesson

Vietnam is in the process of upgrading the entire country’s IT system. With support from organizations such as the World Bank, Vietnam is rebuilding not only physical infrastructure, but also starting from the ground up building new IT systems – including a large scale virtualization strategy.

Hawaii may not be so progressive. The first line of an Associated Press story on Hawaii’s lack of a functional IT strategy goes like this:

“In many ways Hawaii’s government runs its computers like the Internet age hardly happened.” (AP)

The story goes on to expose Hawaii’s lack of IT policy, the fact they are using old systems, a mixture of Apple and PCs for individual users, have a 1960s version of disaster recovery (offsite physical diskette storage), and other parallels with industry that add more discouraging evidence to Hawaii’s IT shortfalls.

Sensationalizing the Obvious

Information Technology in HawaiiI’ve always found it very easy to criticize. Perhaps the role of a journalist is to sensationalize the shortfalls of others, as people do tend to like watching others suffer – as long as the pain stays in somebody else’s life or reputation.

OK, so Hawaii does have some shortfalls in their IT systems. As a user, I have to say my experience using Hawaii’s eGovernment applications hasn’t been too bad. A plus in the Hawaii IT strategy column. I have never had an email rejected from a Hawaii state email server. Another plus. I could probably rack up a lot of pluses, but it is not sensational.

Now let’s look at the difficult side of journalism. Writing something positive and still trying to make it interesting to the readers.

Vietnam is an interesting case study. A larger population, and a lot more government than Hawaii. More problems to deal with – but the government is trying to drive the national IT strategy down to the city level, decentralizing actual applications and access as much as possible to promote the independence of provinces and cities – without disrupting the national IT plan to standardize IT management throughout government.

Nobody would ever suggest the US government try to standardize data strategies down to the state level, much less the city level, however there is still an interesting lesson that can be applied from the Vietnam model.

Data format standards on a national scale can facilitate information sharing and data mining. We won’t go into the personal security issues of that statement in this article, however data format standardization is a good thing for government. The commercial world and manufacturing have had data format/classification standards for many years, including projects such as RosettaNet, XBRL, and UNSPSC.

Thus a driver’s license format in Danang would look identical to the same item in Hanoi – representing 2 very different provinces. Data can easily be shared as needed for identification, reporting, law enforcement, and other data transfer.

Standardization is good.

Enter Virtualization and the Cloud

If a government bureaucracy in a state like Hawaii has extended its inefficiencies into the world of IT, and as stated in quotes the AP article included:

  • Hawaii’s department-by-department way of handling information would not work in the business world, where companies invested heavily in upgrades as the Internet and computers grew in importance.
  • It’s like we had all these little companies and they all grew at the same time, and then when the big company came along and merged everything, it never made the changes.

Beautiful Island - Not So Impressive IT StrategyWell, even in deeply entrenched bureaucracies there has to be a scheduled refresh of technology at some point. Even those precious little Macs and PCs will eventually die, become so old they cannot even load a browser, or the state will grind to a halt because a day will come when no computer in the government will be able to open a Microsoft Word 2010 document.

Maybe, just maybe – much like the government of Vietnam has come to realize, that refresh strategy could include cloud computing. The city of Los Angeles has accepted cloud, and that city probably has a larger government and bureaucracy than the entire state of Hawaii.

The AP article mentions that Governor Lingle has tried to establish an Office of the CIO within Hawaii. Good idea. One that will ultimately save the state a lot of money. Let’s push our representatives to make that happen!

A Proposal

Now select a couple of good data center locations. A couple on Oahu, maybe one each on Maui and the Big Island. Start building cloud computing centers on each island, connect them via dedicated high speed links, synchronize data and applications, then inform the state that all new editions of office automation software will be using a hosted edition of Office 2010, or other high performance hosted package.

Bang – saved money on license fees, labor for installers (those guys who are paid to update your anti-virus software and load service packs on your computer), and high performance desktop and laptop computers.

Start refreshing with dumb terminals and netbooks.

Establish a real state-wide disaster recovery model:

  • Cloud-based virtualized storage
  • Central cloud-based email system
  • Distributed DR model using network-based backups in geographically separated locations
  • Dumb terminals and netbooks backup to the centralized data base and storage – not on local equipment (unless the worker is a traveler). Access to the data is still available from a distant end location through use of VPNs.

Retrain the IT staff on developing applications in the cloud – not on under-the-desktop servers.

Could it really be that simple? Actually – yes. In addition, if the state of Hawaii can build a storefront of applications (including Office 2010-like products), and make those applications available to users on a state-wide basis, and reduce provisioning time for applications to minutes rather than months, why wouldn’t we consider this as an option to what Sen. Donna Mercado Kim (D, Kalihi Valley-Halawa) was quoted as saying, “Every department has IT (information technology) people, and they each have their own way of doing things.”

Nonsense

Very 1970s… So not 2020s…

Vietnam is rebuilding their national infrastructure, the US government under the direction of CIO Vivek Kundra is rebuilding the national IT strategy. Hawaii can rebuild ours as well. And we have great examples and precedent to learn from.

Trouble at the Telecom and Communicator’s Bar

Have you heard the news? Unemployment is skyrocketing, companies are closing, there’s no investment money for startups, and the sky is falling, the sky is falling? Don’t I know, as the layoff frenzy hit my own Hanging out at the communicator's barhome, that it is a scary economic place to take a swim… Sharks, really hungry sharks, circling with an eye to take every last cent you have been able to hide.

And the outlook remains bleak. The New York Times reports that Europe is suffering in youth unemployment – even more than the US. 42.9% unemployment is Spain, 28% unemployment in Ireland, an EU average of 20.7% Makes California look like the “promised land.”

And, California may actually be the “promised land.” California still attracts the best of global engineering to the Silicon Valley, and the most creative minds in communications and entertainment to Los Angeles. Whether you are a European, Chinese, Indian, or even Canadian, Silicon Valley and LA offer an environment that is unsurpassed around the world. Our universities embrace people from other cultures and countries, and our ability to support entrepreneurs draws not only students, but the best engineers and thought leaders from around the world.

Back at the Communicator’s Bar

There are still tables with discussions reviewing the indignities of being laid off by struggling companies. There are still discussions with the whine of people talking about the “damn foreigners” who are here stealing our jobs. Still “barflys” slopped over the bar worrying about their Audi payments and how their ARM mortgage has put them under water.

Then there are other bars with tables full of Americans, And A scatter shot of foreigners talking about fun stuff. Fun stuff like cloud computing, virtualization, globalization, distributing computing, “the network is the computer,” “the computer is the network,” and how the carriers will return to their roots of providing high quality “big, fat, dumb” telecom pipes. The talk is of how we can finally start putting all this intellectual property that we’ve spent billions n producing Powerpoint slides into reality.

Green is here

Virtualization is here

Data Center outsourcing is here

2010 is a blank whiteboard set up to codify the thought leadership and technology spawned in the waning years of the 200x decade and put it into business plans and CAPEX budgets.

2010 is the year we aggressively deliver Internet-enabled technology to every man, woman, and child in the world who has a desire to live a life beyond killing their own food for dinner. Here is a funny though – if a radical 8 year old in one currently scary country is able to Yahoo chat or Facebook their way into discussions and relationships with kids in California and Beijing, doesn’t it make just a little sense the desire to blow each other up would be diluted, even just a little?

If the guy living next to me is producing a telecom switch that is head and shoulders above what is currently on the market, do I really care if his brain was conceived in Hanoi?

2010 is also the beginning of a true period of globalization. That doesn’t mean out hillbilly friends in Duluth, Minnesota have to quit drinking 3.2 beer and hanging out at setup bars watching Vikings reruns, it means that the hillbilly’s kid can participate in a lecture series online from Stanford or MIT. The kid might eventually invent a pickup truck that runs on pine cones, and a 3.2 beer that is actually palatable.

Embrace 2010

If not for the simple fact you have no other choice, consider all the great ideas being pumped out by companies like 3tera, the Google borg, Microsoft, VM Ware, and all the other companies with tremendous innovative ideas. Never before in our history have some many new intellectual and business tools been put on the shelf at the same time. Never before have we had such good reason to consider implanting those ideas (yes, I am a tree hugger and do believe in global warming).

So, even if you are currently living in a car under a bridge near you former upscale Orange County community – shave, wash your car, take a shower at the beach, and let’s get our depression, anger, tacit knowledge back into the business saddle. The young guys still need our experience to get their feet on the ground, and we need them to ensure we will have social security in the future.

Welcome 2010 – you have taken a long time to arrive

John Savageau, Honolulu

Wiring California with Bob Evans and the Fiber Internet Center

“The Founder and Chief Executive Officer of Internet Inventions, Bob is a true Silicon Valley entrepreneur, raconteur and original. Bob has combined his technical ingenuity, entrepreneurial ability and team building skills to found successful companies time and again over the past 20 years.” (Silicon Valley Business Television)

Bob Evans always has ideas. Ideas to make his work, and the work of others, more useful and efficient, as well as easier. We first met Bob when he asked if it was “OK” he develop a route server for our young Internet Exchange Point (Any2 Exchange), which would allow the small to medium Internet access and content provider community to find Internet peering easier, and help with introductions within the community that would make Internet interconnections a utility – rather than a high priced value-added service. That little utility now drives one of the largest Internet Exchange Points in the world.

We met Bob this week as he was giving a data center tour and professional advice to a delegation from Ramallah, which came to Bob for mentoring based on his extensive background in all things network and Internet.

For the entire audio interview – click HERE

Pacific-Tier: Bob, please tell us a little about yourself, your background, and how you got into this business.

Bob Evans: Well, it was a long time ago, when there was no world wide web. I ran one of the first Veronica Gopher servers. I was teaching grandmothers how to do email, so they could email their kids in college.

At that time colleges had the first ARP connections, TCP/IP started working right, and that was way back when… That’s how I got started.

I’ve got a high school education, dropped out of college, so generally every time I start a business and build it up, sell it to somebody with lots of money and VCs, they (eventually) ask me to leave because I don’t have a degree. Then they usually do quite well. I’ve been doing this a long time.

But when I started the Fiber Internet Center, what happened was, people kind of thought I was crazy, because I had just had a very successful, one of the first cable modem companies in the US, and by starting the Fiber Internet Center/FIC at a time when the “Dot Bomb” occurred everybody thought, you know, “how are you going to make a living doing that?”

The real key was making the deals. Because all of the (network) providers, and the people with fiber and stuff like that, didn’t really have a lot of customers. So, they were willing to listen to me and cut me some deals. So that’s how I got started with the Fiber Internet Center.

Pacific-Tier: Tell us a little about the FIC. What is the vision, or what is the mission of FIC?

Bob Evans: Well. Its primarily,… like you find colo (colocation) facilities that are “carrier neutral,…” I like to look at us as a network company that is carrier neutral. Although we do sell circuits and services like IP transit (Internet network access), it is kind of a necessary evil, because we only have about 8 or 9 other ISPs (Internet Service Providers) currently serving our market.

So they use our network in the market place to service their customers. So we’ll build a network within our network to service other companies.

Pacific-Tier: So the FIC is present in Northern California, the San Francisco, San Jose, the bay area, as well as Los Angeles. What value does the FIC, or a company like the FIC bring to a market like California that is not available through another carrier like Level 3, an XO Communications, an AT&T, or that scale of carrier?

Bob Evans: Well, most of those companies will go ahead and install their circuit in the basement of a building, or in the MPOE (main point of entry), and then only allow you to use their IP (Internet network access).

For example, Level 3 isn’t about to go ahead and market the fact that I could have VLANs in that building like I would have and connect them to ISPs. That’s the real advantage. A customer could buy a circuit from us, then they can buy Internet transit from us too, then they’ll turn around and say “gee, you are over at the PAIX (Palo Alto Internet Exchange), over at places like Market Post Tower, One Wilshire and everything,…”

Then they’ll say “there’s this other company we’re working with, and we’d like a connection direct to their office. They’re using Cogent bandwidth (or something like that), could you make that come in here on another port?”

And we’ll do it. Most (other) companies will say “no.” But that’s actually part of our business strategy.

Pacific-Tier: Well, that’s a great utility, for both the enterprises and other carriers within your area. Where does the FIC go from here? What is in the future of FIC?

Bob Evans: Lately it’s become one of the discovery that most of these business now feel a critical need to not have their mail server, or other types of critical databases remain on their campus. Or they want to have a backup of it someplace else. So we’re now creating a service in other data centers where you get a circuit from us, then we’ll give you another port that gives you a layer 2 connection back to your rack or your server in those facilities, or one of our facilities.

This gives you the added benefit of having your own private network to the back side of your server, then it gives you the advantage of maybe having your server connected to other Internet providers in one of those other colo facilities, or maybe you could even carry that Internet back to your place and have two Internet providers.

So I think the advantage is, if you get a circuit from us you get the flexibility.

Pacific-Tier: Well that’s great. I think it is a great utility for the community. The final question is, you’re an entrepreneur, in tough economic times, with lots of graduates hitting the street, what advice do you have for people who are considering starting their own companies, or emerging entrepreneurs?

Bob Evans: Well, that’s a very good question.

During bad economic time I always find that not to discourage me, and I wouldn’t let anybody discourage them either. The reason is because in the midst of chaos, there’s usually always opportunity. And as soon as you explain that opportunity to somebody else who’s got money or can help you, or needs a service that you have, it’s actually easier to make business deals.

Pacific-Tier: That’s great advice.

Thank you very much. It’s a pleasure to visit the FIC again, and thanks for your time.

Bob Evans: You’re welcome – thank you

Wiring Los Angeles Part 4 – WilCon’s Path to the Future

This is the fourth article in a series of interviews with Eric Bender, President of Wilshire Connection (WilCon), the largest independent telecom carrier in Los Angeles. In this segment Eric discusses the future of WilCon, including expansion outside of Los Angeles, wireless topics, relations with local utilities, and some great examples of WilCon’s flexibility in delivering telecom solutions to the LA community.

Pacific Tier: Outside of downtown LA , what is your expansion strategy for going to place like El Segundo, Las Vegas, or other cities, parts of the city?

Conduits Inside a Street VaultEric Bender: We’ve leased dark fiber from other carriers to get to other off-net locations such as El Segundo. We connect Equinix on Maple, so we can do lit transport into that facility. We’re working on a plan that would extend from there to the 365 Main location in El Segundo, and then with a second route back to downtown.

We’ve acquired several 10Gbps wavelengths from LA to Las Vegas, and into the Switch NAP facilities, so we can connect with all their switches and facilities, including the Super NAP, which is a 400,000sqft monster of a data center facility. So we do just backhaul or transport to Vegas. We’ve got approximately 10 gigs of traffic back and forth between here and there now.

I don’t see us building on a long haul type arrangement where we would build far outside of downtown LA. We’ll go off-net by structuring deals with existing customers and partners that we have to get fiber and connectivity into other locations. One of the things we are working on is connecting our buildings in LA to Phoenix, there’s a couple of different data centers in Phoenix – there is I/O, (etc).

Then we might go from Phoenix to Las Vegas, and from Las Vegas we already have to LA (which we already have), but also Las Vegas to San Jose/Santa Clara, and then back down to LA. So we basically have a ring from LA to Phoenix, to Las Vegas, San Jose, back to LA.

Pacific Tier: So would you have arrangements with Edison (Southern California Edison), or Burbank Power, or people like that to extend into Hollywood or Burbank or Glendale?

Eric Bender: Well, we do work with Edison and DWP (Los Angeles Department of Water and Power), and have used them to get to further out locations from downtown. The city of Burbank, we’ve not been successful and have not really chased hard to try and make a deal with them. Our initial efforts to work with them were rebuffed without comment – they had no interest in working with us, which surprised me.

I think we can be creative in building, like at 900 N. Alameda which we now have connected with dark fiber, and there is a variety of ways we can do that. We have connection and interconnection agreements AT&T to use their infrastructure so we can theoretically deploy assets throughout the city and state in coordination with AT&T’s network.

Pacific Tier: Would you consider for example going to San Diego using a capacity swap with Edison where they gave you a pair of fibers to San Diego?

Eric Bender: We do that already, well not to San Diego, but we have done a variety of barters, trades, or however you want to call it, with companies like Edison. With Equinix, with DWP I’d like to – but they have no interest and can’t by law (they are a public utility), and can’t really do that. Level 3, we’ve had discussions with them to do the same thing.

Because I’ve got all this infrastructure, whether it’s the conduit, or the fiber, and it’s all there, it’s paid for, there’s no debt, there’s no cost of capital to me – it’s there, so if I need to allocate a couple pairs of fiber going here or there on my network, and I can get something that I can generate revenue off of on somebody else’s network – we’ve done that many, many times and it’s worked out very well with a variety of companies.

Pacific Tier: Expanding a bit more on WilCon’s flexibility…

Eric Bender: We have no rigidity with anything we do. I think that one of the reasons we’ve been successful, and have continued to grow every month, basically where we’ve continued to grow over the previous month is because we’re not rigid and have a straight line approach to things.

Servicing WilCon's ConduitI think the advantage of not being a telecom guy by trade or training and education is that we look at things by how do we get the customers the things they need? When a customer needs something my first response is to say “OK,” and the second response is to call my technical guys and say “I just committed us to doing this by a certain point in time, make it happen.” And they do.

We’re a small company so we don’t have all the various layers of engineering groups and planning groups to say from here it has to pass from point “A,” to point “B,” to point “C,” before you can be told it can be done, or not be done, or how to do it. We just make it happen and get it done.

The story I tell that kind of tied the real estate to the telecom all together well for me is at one of our buildings, 700 Wilshire we had a t4enant in the building. This goes back to about early 1999. We had put conduit in the ground already, so it was probably early 1999, had already connected the conduit to One Wilshire, as well as 700 (Wilshire Blvd), and we had leased space to this tenant.

Part of their requirement was they needed a couple of DS3s between the meet me room at One Wilshire and their premises they had leased at 700. At that point we were only doing conduit deals with our carrier customers. They were going to Level 3 for this DS3, and Level 3 hadn’t really built out their network yet. Because we put Level 3’s first conduit in the ground in LA (it was conduit we’d installed in our trench and gave them, basically), and Level 3 was provisioning this DS3 through PacBell (now AT&T) or someone, and they were re-provisioning (the DS3) through them.

The timing for the tenant was going to be 90 days, or 120 days wait for the tenant to provision this DS3. The tenant came to us and knew we had started Wilshire Connection. So they came to us and said “we really need this, is there anything you can do for us?”

I didn’t know what to do, I mean we just had conduit between the buildings. So the customer said “all you have to do is this. We have the equipment, you’ve got fiber cable already (because we’d already deployed that), you need this piece of equipment, you’ve got this box on this end, that box on that end, and we’ll tell you how to do it and what works well.”

Two weeks later we had that DS3 traffic up and running between the buildings. So we got into the transport business. But that was because we were flexible, somebody needed something, (we had the resources), and we got it done.

If we can see there’s a reasonable return on investment and payback on the infrastructure or cost to deploy new equipment, we’ll do it. We got all this MRV DWDM equipment (dense wave division multiplexing) because we had a very limited amount of fiber into the (particular) building, so the only way I could provision this service (lit bandwidth) was with the DWDM gear – so we went out and bought it.

So the equipment was paid for within a year. The customers can come and go – it doesn’t really make any difference (other than we want to keep customers), because it is already paid for. So we’ll continue to do that, and that’s how we’ve continued to grow into the buildings that we have, as well as the transport circuits that we do between all these buildings, as well as connecting into Vegas, and the same to Phoenix, Santa Clara, and other cities.

Pacific Tier: So what is the future of WilCon, where do you go from what you have today?

Eric Bender: several years ago we had some strategy discussions, and (since) we are a dominant carrier in LA, we’re known around the world as the provider of choice, the connector of choice here in LA. We have a great relationship with Equinix, and CoreSite now (CRG), so we get a lot of referral business from those guys. We’ve provided such good service to our customers that big carriers such as TATA, PCCW, and others that they just come straight to us when they have new requirements.

So we had this discussion “let’s go and expand, and into other markets” and we took a very quick and dirty analysis and determined that it costs so much to build out and deploy in these other markets, and we know other companies have been doing similar kinds of things, and we might be providing a similar type of service. In order to do that (expanding into other markets) we’d have to take on an additional equity investor, or venture funding, and then all we’re doing is servicing somebody else’s debt.

So, does that make sense (expanding) or should we continue to be the provider here, and continue to grow which has made us a profitable company. So we said “we are gong to do that, and we’ll expand and provide services outside of LA in various ways that make sense, and does not require a huge capital outlay, and we’ll do some – but we’re not going to spend millions of dollars to deploy.”

The reason many companies went bankrupt was because they built these massive networks, and there was no r4evenue to service the debt on them.

So our future is to grow in selected and strategic ways. It will probably be between LA and other cities by usi9ng lit waves, and then establishing a POP (point of presence) in a couple buildings, and then maybe in some cases building a bit in those markets to connect those buildings.

In Dallas we’ve looked at buying an existing network, but there’s really no need to connect between those facilities – there’s no business with that. Chicago was no different. San Francisco, San Jose, Santa Clara are all so spread out its hard to justify a new build or new construction.

I think working with companies like Allied Fiber which is building their long haul network, and working with them to be the local metro piece to what they’re selling to their customers on the long haul – we can tag along with them, working well with them in LA, and possibly duplicate that with them in other markets.

Pacific Tier: Is there any wireless in WilCon’s future?

Eric Bender: I don’t really see that. We did look at partnering with some wireless, WiMAX, or other guys using Terrabeam or some of these free space optics. It seemed like they just had such a long way to go before it would make it worthwhile, and that in LA maybe it’s a problem. Trying to negotiate the line of sight, and rooftop antenna rights and those things,… that we didn’t really get anywhere. It became cost prohibitive I think.

So it would be nice to do something like that. We looked at that (wireless) to support NANOG (North American network Operator’s group) several years ago when they were here. We were in Library Tower (the US Bank Building) on the 56th floor. I had a line of sight directly from my window to the hotel, but it was too far. We would have needed a couple of hops to be able to deliver a gig (Gigabit connection), and it was a disaster for them to have it there (at that hotel).

They had to go buy a DS3 from AT&T (PacBell). If they had done the conference downtown I could have connected them with no issues.

Pacific Tier: Any final words you would like to say to the global telecom or Internet community, or Los Angeles? Why LA, or with WilCon, downtown LA is an attractive place to do business?

Eric Bender: We’re number one!

I think LA is a great place to do business. From a location perspective its perfect, as its the gateway to the US. You can get anywhere from LA, whether its physical transportation, or from a telecom infrastructure perspective. I think if you come to LA as a carrier, as a provider, as a bandwidth user, in this industry – and you’re going to locate in one of the half a dozen facilities (in downtown LA), whether an Equinix, Level 3’s gateway, One Wilshire, or TELX Carrier center.

Any of the main big facilities here (and Switch and Data) you are going to need, ultimately, to connect to customers that are in one of the other facilities – somehow you’re going to find us. You may be referred to us, do your research – ultimately you are going to find we can provide that connectivity.

We are really easy to do business with, as any of our customers will tell you. And I think that in all the years we’ve been in business, since 1998, I think there are maybe two deals we’ve lost because of price, that we were not the lower cost provider for that particular deal. And, we try to provision within 24~48 hours (of an order).

Pacific Tier: That is outstanding. Words for the global telecom and internet community from Eric Bender, President of Wilshire Connection. Thank you very much.

John Savageau for Pacific Tier Communications, Long Beach

====

This concludes our interview with Eric Bender, President of Wilshire Connection in Los Angeles.  you can contact Eric at ebender@wilcon.com for more information on Wilshire Connection.

The entire interview is available online.

Previous entries in this series include:

  • Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
  • Part 2 – WilCon Takes to the Streets
  • Part 3 – WilCon Manages Infrastructure Risks

Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

This is part three is a series of interviews with Eric Bender, president of Wilshire Connection. Wilshire Connection, or WilCon, is the largest independent local network and neutral fiber infrastructure provider in downtown Los Angeles. In this segment Eric discusses how WilCon managed risk to their network during the initial construction process, the continuing management of critical telecommunications infrastructure, and the role WilCon could play in the event of a major incident impacting the telecom industry in Los Angeles.

Pacific Tier: (on the topic of utility gas and electricity) What risk is there to the infrastructure in downtown LA of an explosion from either electricity or gas, and what would that do to your conduits if it occurred?

Eric Bender: Interesting question… I haven’t really thought much about that, it would, depending on where, you never want to see that. Fortunately with power, Street Utility Tagging in Downtown Los Angelesthe lines typically won’t explode, it is the transformers, which mostly are in the buildings.

The way LADWP (Los Angeles Dept of Water and Power) sets up they bring the high voltage lines into the building, so if the transformer blows up it will be in the building. The transformer is typically not out in the street – but they do have some vaults out in the street, and they have had some explosions, but they have been contained within the vault. We don’t run through any of their vaults so from that perspective we’re OK.

Somebody who is digging and hits a gas line… that’s a different story.

I can say in the roughly 12 years that we’ve had conduit in the ground, and since we were one of the first to dig, I can say that we are lower in the ground, and more protected in many respects, and more of a straight line in routes we’re going without having to jog around any other existing infrastructure that came later.

But we’ve never had one of our conduits damaged or cut. Or interfered with…

The difference between how our conduits or duct banks are typically done vs. many other carriers is that we put in massive amounts of conduit. We use (typically) four inch PVC conduit, and I don’t think we have a single one (trench) which has less than six or eight four inch conduits. Most of them have multiples of that, for example going on Wilshire Blvd between Grand and Figueroa we have an average of conduit in that whole duct bank is probably close to sixty four inch conduits.

Because fifty of them go into One Wilshire, although they kind of peel off in WilCon Conduit Duct Bank in Los Angelesdifferent directions from there, we’ve also got a main ’48 and they lateral off down into other streets into buildings, so there’s some doubling going on.

But in some areas we have up to sixty four inch conduits so that’s the size of a desk, or bigger, and that duct bank is encased in a concrete slurry around it, surrounded by a couple addition feet of dirt and asphalt on top.

So if anybody digging is going to hit (the duct bank) typically there is warning tape on top of the conduit, concrete, or in the concrete they are going to pull it up. They are going to hit (the tape) or something before they hit the conduit. So unless it is some young, uncontrollable (person) with the backhoe who is on a rampage – no matter what you do you can’t control that.

NOTE: Whenever a company opens a street for utility construction or maintenance you will normally have construction observers and safety observers from not only the company opening the street, but also each company with conduit or utility infrastructure in the immediate area of digging.

But some of the other carriers that put in one or two conduits, they are the ones at risk, like for example an MCI, Verizon, or a QWEST in some cases they typically put in just the two conduits… And you could rip through that even if its encased in concrete before you realize what you are doing.

The conduits are just this big, versus this big (Eric shows a note book size to represent two conduits vs. his desktop to represent WilCon’s conduits).

Pacific Tier: Do you consider yourself the only truly neutral facilities-based carrier in downtown LA?

Eric Bender: I think others consider themselves neutral, but they have other motives as well. I don’t care if I sell dark fiber or lit transport. We can do either, and it doesn’t matter to me which one they want. We have so much fiber, and the infrastructure to continue pulling more and more fiber that I’m never worried about running out of capacity for dark fiber.

A Level 3, or a QWEST, or an XO, they run a network, and they’re obviously not neutral. DWP (LADWP), they’re somewhat neutral because they don’t seem to care whether you take fiber or lit services. I don’t know what they do with lit services or on the network side of things – honestly. We’ve leased from them (LADWP) dark fiber to get access to some off-net buildings, and they’re very easy to work with. They are very rigid, and have no flexibility (LADWP is a public utility), but they’re easy to work with.

We’re probably, as a straight, neutral, not really care who you connect with, we may not be the only one (neutral fiber network), but one of the only ones that would be neutral.

Pacific Tier: One Wilshire is traditionally a center of communications in Los Angeles. Some people think it is a high risk location because there is so much on the 4th floor and other parts of the building. Some people think that it is meaningless – that it doesn’t have that much value. Do you think the 4th floor of One Wilshire today is a critical piece of infrastructure, or do you think it is something that is just there, that could be bypassed when and if ever needed?

Eric Bender: I think at this point it would still be considered a critical facility. A lot of carriers and other companies have facilities or locations elsewhere, but because of the way they’ve built their networks from the beginning, One Wilshire has always been the central point for them.

They may not have grown and expanded there, or they may have moved things off to other locations, such as 600 W. 7th, 818 (W. 7th ), or even outside of LA, and use a company like us, or some other carrier to make that interconnection or virtual connection between their two facilities. One Wilshire tends to have been their primary facility.

I think that over the years that’s more applicable to legacy carriers, the bigger carriers, the ones that have been around for a long time.

I think the Internet type of carrier that’s either VoIP or an Internet company, content CDNs (content delivery networks), and those – One Wilshire’s not as important to them at all because in my limited technical knowledge it is easier to reroute that traffic to other servers – they have more mirror facilities than a switch would have on the telecom side of things.

Pacific Tier: Is there a business continuity plan, or disaster plan, in the event One Wilshire or another facility like 60 Hudson (New York), or the NAP of the Americas (Miami) anybody has thought about or put on the shelf in the event one of those critical facilities has a catastrophic failure?

Eric Bender: I am not aware of any common, for the greater good, where all the carriers have participated in developing that, or working out some kind of contingency plan. That’s actually a really good question.

You’d think that after 9/11 where the infrastructure was so significantly damaged that in various other markets such as LA, Chicago, that there would be some kind of a group of organizational effort to have dealt with that. I am not aware of one. They may have one, but I am not aware of it, and they never invited me to participate.

Pacific Tier: Is WilCon positioned, in the case of a worst case scenario in Los Angeles, to assist the community and assist the industry in recovering to an alternate facility if that occurred?

Eric Bender: Sure, I mean, our infrastructure that we built, and that we control and own, is all primarily downtown LA, so… in a worst case One Wilshire becomes untenable, well a lot of our fiber doesn’t all home run into One Wilshire, but a lot of it does go in and turn around, coming back out again.

There would be disruption, but it could be brought out and bypassed. We have diverse paths into most buildings that we connect so we can certainly do it.

Not in LA, but when in the Mediterranean last year when they had the three or four cuts, (several of) our customers were impacted. I sent them an email and said I doubt there is anything we can do here , but if there is anything you need that we can help you with, let me know and we can work with you.

And actually two of the customers said “yeah we need to reroute some connections to put it on a different side of their ring (in the Med and Pacific) that we could do in about five hours with a couple pairs of fiber for them, and they were able to reroute their traffic, or some of their traffic, and lessen the impact of those (submarine fiber) cuts.

Pacific Tier: So WilCon would consider yourselves a very flexible, agile part of a recovery plan, and would not be rigid in your provisioning process, and that you would work with the community to recover from a disaster?

Eric Bender: I agree with that!

=====

This ends the third segment of this series. In the next part, Eric will discuss more of the future of Wilshire Connection, including his visions for expanding WilCon into new markets.

The entire interview is available online.

Previous entries in this series include:

  • Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
  • Part 2 – Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

Wiring Los Angeles Part 2 – WilCon Takes to the Streets

This is the second part of an interview with Eric Bender, President of Wilshire Connection. In this segment Eric talks about the period in 2000 preceding the Democratic Nation Convention, and the aggressive industry build out of conduit, fiber, and telecommunications infrastructure in the downtown Los Angeles area.

Pacific Tier: At what point do you think the city of Los Angeles figured out this would be a really good thing for LA, and it would bring more business and money into downtown?

Eric Bender: I don’t think they ever came to that conclusion. What happened was in 2000 the Democratic Convention was going to be down at the Staples Center in downtown, so I think it was in early 2000 or at the very end of 1999 the city called a meeting and notified all the telecom carrier that had been active or Eric Bender President Wilshire Connectionbuilding or doing things that as a part of the preparations for the convention that they would be repaving the streets, resurfacing certain streets, and there was going to be an absolute five year moratorium on any digging or street construction work as a result of this. They wanted the streets to look pretty on TV, and that was fine.

So I don’t think they thought about how much money, or how beneficial the telecom network was going to be to the city, they realized they would need to do something for this convention. The result of that was that it threw somewhat of a panic into the carriers that were in the infancy of building out their networks, and building throughout LA, and it forced a massive infrastructure improvement project, because there was a very limited amount of time before the city was going to shut everything down.

I think the convention was in July or August, and by the end of May everything had to be done. So, we really had about four or five months to build everything and our network probably tripled in size during that period of time. We were one of the only companies that had an active permit that we had pulled for a small segment that we were building that hadn’t started construction yet because it was not that critical to us at the time.

But because of us having an active permit for about a two block, few hundred feet, maybe five or six hundred feet of conduit the city realized they had a major problem that these companies had to be building, had to be constructing – they couldn’t just shut them down for five years. The city did understand that was a problem.

So they said you can build, we will expedite and streamline the process and anyone who has an active permit, well everyone can tag along with that permit. We’ll just change it and build out from there.

So, with our little permit, and the little segment, I think it was on Grand from Wilshire down to 7th St., and then a little bit more. We basically parlayed that and built it up to 5th St., then up to Figueroa, and literally tripled our network. So we became the lead builder from that permit on a tremendous amount of other conduit that was built in the city.

Basically they couldn’t keep up with all the activity because there were intersections in downtown LA where there were three backhoes digging for three separate projects. We were doing one, QWEST was doing their own, they didn’t really want to participate much with anyone, and MCI, MFS didn’t want to share conduit so they were doing little things.

And I remember at 7th and Grand there were three different construction crews and project going on at the same time, and for me it was just a wonderful time because we had this permit, so we had a lot of leeway, and I would walk the streets, stand on the street corner, and these other companies, you know Level 3 and QWEST actually did participate, XO… You know I would say “we’re going down this street to this building,” and they’re going “I’m going that way,” and we’d shake hands. “I’m taking two conduits in yours, you’re taking a conduit in mine…”

I don’t like to use the term “wild, wild, west,” but it was really a very wild and fun time. My best times were just walking around with all the construction going on, with all the lane closures and all the activity. The traffic was backed up, cars were beeping, and would be driving by and I would just be standing there looking at the big hole in the ground with a big smile on my face. It really was the best time.

Pacific Tier: Did you have any catastrophic backhoe cuts or anything disruptive during consutruction? 

Eric Bender: We, during our construction our guys, we never had an incident where we hit any other conduit. There was a water or a sewer line that got hit, and nicked, and that was something that that was not actually on the infrastructure plan. And it wasn’t marked on the street so it wasn’t our problem so much.

There was a close call with a gas line. But nothing happened. But for the most part (Eric knocking on his wooden desk) they did a good job.

The toughest part to build, which we were actually the participant in and XO was the lead builder on 7th, I think somewhere between Olive and Grand, it shows you the history of LA but, back when they had the street cars, the RED Line, YELLOW Line, and all these other street cars, they never really demoed (demolished) those out.

They just over the years paved over and over and over. As they were excavating and going down the road, the path where they were going, and which later became obvious because of the railroad, they had to rip through all the railroad ties. It was terrible.

Because of that, then they had to get an archaeologist because you know you have bricks from the support for the railroad, because they are historical. I don’t know what they would do with it, it is in the middle of 7th St (a major road in downtown LA) and Grand Ave, that can’t, you know set up a museum in the middle of it, but it shut the project down. The city had some very peculiar rules that you cannot dig on Sundays that go back to the old blue laws basically, no construction activity on Sundays.

Some streets they won’t let you work during the week, you know it goes back to the days when downtown was a main shopping area, so you know, so you couldn’t dig during the week. You could only dig on Saturdays from nine (a.m.) till three (p.m.) which means you get about from ten till two because you have clean up and everything.

So some projects that should have been done within a week took like eight weeks to get done. So the city tried, but they were not the easiest. The only company that got to work on Sunday was QWEST, because, well, nobody is really sure exactly what happened or how, but they did have several projects where they were working on Sundays.

====

This ends part two of the “Wiring LA” interview series. Part Three will explore some of the risks of building such a dense concentration of telecom projects within the downtown LA area.

The entire interview is available online.

Previous entries in this series include:

Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection

 

John Savageau, Long Beach

 

Wiring Los Angeles with Eric Bender, President of Wilshire Connection

Downtown Los Angeles is among the most densely connected telecommunication hubs in the world. A dozen buildings in LA’s city center house the world’s largest Internet networks, Laying Down Telecom Conduits and Cabletelecommunication carriers, content management networks, and entertainment companies – interconnected through a complex mesh of submarine fiber optic cables, terrestrial cables, and internet exchange points.

With more than 500 networks and carriers operating within the LA city center, one company stands out from the crowd as a leader in bringing the global telecommunications and Internet community together. Eric Bender is president of Wilshire Connection, a facility-based carrier focusing on providing neutral, high capacity fiber optic cable interconnecting the most important buildings in Los Angeles.

Wilshire Connection (WilCon) has a vision to free the LA telecom and Internet-enabled community from the burden of developing a highly meshed inter-building infrastructure, allowing each company to focus on bringing value to their global network interconnections and relationships. Eric Bender, President of Wilshire Connection, is the man behind WilCon’s dramatic success in wiring Los Angeles.

This interview will cover, in a multi-article series, the history of modern telecommunications in downtown Los Angeles, the role Wilcon played in LA’s redevelopment, the period of rapid and chaotic build out prior to the Democratic National Convention in 2000, the risk of high density telecom infrastructure, and the future vision of Eric Bender and Wilcon.

Pacific Tier: Today we have Eric Bender, President of Wilshire Connection. Eric, can you tell us a little about yourself, and how Wilcon came around?

Eric Bender: Sure, we founded Wilcon, Wilshire Connection back in 1998. I am one of the founders and one of the partners, and have run the company pretty much from the beginning. I’m a real estate guy by background. I graduated from college and have always worked in the real estate field, whether its commercial property, management, brokerage, home build, whatever…

Back in late 1996 I started working for a Hong Kong based family which was acquiring some office buildings in downtown LA, one of which was 611 Wilshire Blvd which was directly across the street from One Wilshire. Back at that time telecom deregulation had just been passed, the 1996 Deregulation Act, and there was a boom of CLECs formed and started. Nextlink, the precursor to Level 3 (Kiewit and Sons), MFS was expanding into a variety of things, Worldcom, a variety of these companies were just starting up in that time frame.

For a lot of different reasons One Wilshire had become kind of a focal point for telecom, primarily because it is a 30 story building, had very clear line of sight to the east, and back in the day when microwave transmission was the primary factor, (One Wilshire) had great eastward pointing microwave capabilities. And because of this many of the deregulated companies had got into One Wilshire.

Because our building 611 Wilshire was directly across the street, and only 50% occupied, and One Wilshire was still an office building that only had a meet me room component to it, – that was really it, from a real estate side of things we said “we have all the space, you have all these new companies coming and can’t do it, send them over to us, we’ll fill our building with them, and let’s put some conduit between the buildings together as a virtual extension of (the meet me room) of One Wilshire.

Additionally we put in some conduits that the One Wilshire building actually provided for.

A year or two later we had bought, and were converting other buildings in downtown (700 Wilshire, 818 W. 7th St.) and saw a need to have those buildings connected back to 611 and One Wilshire.

Standing on the street corner thinking about it we decided that we should become a facilities-based carrier. We went and got our authority from the state to become a carrier, CLEC, and I can’t remember the other authorities we received. Then we decided that’s how we are going to do it. We started digging the streets building the network and connecting the buildings, our own buildings, together.

We decided as a way to fund the build out, because at the time, 1997 and 1998 a lot of buildings had empty space in downtown LA, they all saw what was happening not only with our buildings, but other buildings that had empty space, and saw how telecom was taking massive amounts space. And the reality was landlords wanted telecom companies as tenants because they paid higher rent, they paid for their own improvements to build out their space, they paid for their own utilities, the buildings did not have to provide janitorial, nor provide much in the line of services.

So we created, as part of our model with Wilshire Connection, the ability to connect all these other buildings into our network. This tied them back to One Wilshire and all these other telecom buildings. And that was our methodology of raising funds to build our improvements to our infrastructure.

Pacific Tier: So what role do you believe Wilcon played in the development of downtown LA as a telecom center?

Eric Bender: Well, trying to be as modest as I can, we probably played the primary role in that, because, several buildings are telecom buildings today because we were building our network and including them. They may have become a telecom building in some way differently, but the fact that we connect them into our network which covers and ties all the building together on a neutral basis was really the instrumental part, including 600 W. 7th St, 800 Hope, 818 W. 7th St., which has Level 3’s gateway, Equinix, and others – brought these buildings back to One Wilshire and to each other – together they became the success that they are.

Pacific Tier: What was your criteria for selecting the buildings that you POPed (built a point of presence) or entered, or dropped cable?

Eric Bender: We were note really picky. At the time we were building the network it depended on which buildings came to us, what they were willing to contribute to the cost of (build out), and how important it was to them to be connected to the network.

Pacific Tier: The actual process of laying conduit in downtown Los Angeles, did the city support you?

Eric Bender: Not initially, it was quite challenging. Part of it was because a lot of this activity was new and we were building our network, planning our network to cnnect the buildings. But you had the other companies that were building their networks, such as Level 3, MFN (actually they came a bit later), MCI was adding a lot more conduit and buildings, and others. The city was not prepared, I don’t think, from an engineering perspective and plan check to deal with all that, so they were backed up, and you had companies like Level 3 which was building a network throughout the entire city basically.

The city’s engineering department that reviewed the plans – you would go in and the entire office would be stacked four, five , six feet high with plans of what they were planning to build. And it took months and months, 6 to 8 months at times.

This is the end of part 1 of our interview with Eric Bender. Additional segments to the interview will discuss the actual buildout of Wilscon’s network prior to the Democratic National Convention in 2000, the risk of having such high density infrastructure in downtown Los Angeles, and future plans for Wilcon and Eric Bender.

====

John Savageau, Long Beach

You can listen to the entire interview online at Pacific Tier

 

Telecom Risk and Security Part 2 – The Carrier Hotel SuperNode

February 1996. A half-ton bomb planted in a small truck near South Quay Station close to the recently renovated commercial district of Canary Wharf. The bomb detonated around 1900 hours, bringing down a six story building, and severely shaking Canary Wharf  Tower and other buildings around the Docklands area. The area, home to much of the telecommunications interconnection capacity connecting the UK and Europe to the rest of the world, is severely damaged and all surrounding activity disrupted.

Today the Docklands area continues to support many important, high density communications interconnection points, including Telehouse Europe, the London Internet Exchange (LINX), and the London Network Access Point (LONAP) – in addition to individual nodes and facilities operated by European and other international telecommunications carriers.

This includes companies operating submarine fiber optic cable systems. These densely interconnected areas are referred to as telecommunications “SuperNodes,” or if the facilities are located at individual facilities, “Carrier hotels.”

A Global IssueThe US National Security Telecommunications Advisory Committee (NSTAC) defines a carrier hotel (or SuperNode) as “conditioned floor space operated by a commercial landlord for the purpose of hosting multiple service providers.” The most well-known supernodes are 60 Hudson in New York City, The NAP of the Americas in Miami, One Wilshire in Los Angeles, and the Westin Building in Seattle.

Carrier hotels emerged in the late 1900s following the Telecommunications Act of 1996, which required US incumbent carriers to provide interconnection or collocation space for the new competitive carrier industry. The problem for the carriers, and opportunity for commercial building owners, was one of the carrier facilities exhausting available space.

The commercial landlords were able to provide building space, partially due to low occupancy in city center areas near large carrier central offices (such as Bunker Hill in Los Angeles) during the late 1990s, and competitive carriers were able to build out their interconnection infrastructure with little or no interference by the incumbent carriers.

Carrier hotels can also be considered “scale free,” with the only real limitation on growth being the physical space available within a property, as well as electricity and cooling for electronics and switching equipment. This may not even be a large problem, as much of the carrier hotel interconnection volume is done through “passive” cross connects. Cross connects fiber optic to fiber optic splicing which does not require local electronics, and thus is not directly vulnerable to cooling and power issues.

What is the Impact of Losing a Carrier Hotel or SuperNode?

Could another attack similar to the 1996 Docklands incident potentially have the impact of severing interconnection capacity between communications carriers, Internet service providers, and news or information resources?

The extent of disruption would depend on the amount of switching and multiplexing equipment and physical interconnection capacity each company locates within the Telehouse facility, or the immediate area.

This is a source of much debate. In the US, nearly all facility-based (own their own cable) carriers and large virtual carriers have numerous interconnection sites located throughout the country. The loss of a single node or interconnection facility would not significantly disrupt national or international communications.

The Federal Communications Commission provides guidelines for facility-based carriers through the Network Reliability and Interoperability Council (NRIC) which advises “carriers place a high priority on service reliability by building networks with alternative routes, backup facilities, and other assurance capabilities.”

The danger at the SuperNode or carrier hotel is not necessarily one of the incumbent or long distance facility-based carrier. It is more an issue with:

  • International carriers with only one or two physical landing points in North America (or Europe)
  • Local exchange carriers with limited interconnection capacity outside of the carrier hotel
  • Internet service providers operating in a smaller geography (Tier 3 access networks)
  • Hosting companies and content delivery providers with single or limited Internet access
  • Local fiber providers with limited diversity within a city center

This is actually quite alarming. When you start to consider the outsourcing industry, including cloud computing, entertainment, and the number of companies who do not have strong disaster recovery plans – including geographic diversity within their applications and communications access – the potential for disruption is high.

Most of the SuperNodes provide interconnections for more than 200 facility-based carriers, networks, content providers, cloud service providers, and other hosting or business outsourcing. Understanding the reality that we live in a very global economy, losing interconnection capacity of even one SuperNode could render a large percentages of the global financial, logistics, business-to-business, disaster response, and government communications inoperable for hours of days while restoral plans are either implemented or conceived.

Companies with hosted applications and data center presence either in or near the failure point could be isolated or destroyed. Hosted companies “single-threaded” with one carrier connection that using the carrier hotel for its main interconnection point would be shut down.

The bottom line, companies without a strong restoral, backup, disaster recovery, and physically diverse network will suffer a catastrophic failure of their systems, with the length of outage entirely dependent on the facilities ability to recover from an outage or failure.

If more than one SuperNode is disrupted, such as all facilities on the US West coast, international communications both on Internet links (the majority of international communications today) and dedicated capacity will cause significant damage and disruption to both US and international communications.

What Can Cause a Major Failure?

There are many factors to consider, both human and natural, when looking at global communications infrastructure. Just in the past 5 years we’ve seen significant submarine cable disruptions due to both undersea earthquakes and cable cuts to strong waves hitting cable landing facilities on the coasts. Carrier hotels are primarily located on the coasts, in large cities, due to the proximity of both submarine cables supporting international communications, and the fact most North American and European terrestrial cable routes tend to interconnect at major coastal cities.

Coastal cities are vulnerable to:

  • Earthquakes
  • Typhoon and Hurricane wind/storm swells
  • Tsunami
  • Tropical rain and flooding

Human factors are also a concern, with potential problems such as:

  • Civil disorder
  • Terrorist attack
  • Vandalism
  • Employees (disgruntled, human error, etc)

If you look at the streets adjacent to buildings such as One Wilshire, you can see the evidence of dozens of carrier tags trying to mark and protect their conduit routes running through the streets, and entering the carrier hotel facility at One Wilshire. Few of the manholes around the area are locked, and few if any local building security officers or police officers will ever challenge a company setting up a couple of traffic cones and entering the manhole.

The potential for human disruption, just by having access below the street level near a building such as One Wilshire or 60 Hudson could be extreme. From below ground potential terrorists have access to power substations, water lines, and hundreds of conduits supporting the entire metro area – including the carrier hotel. A well placed explosive below grade in downtown Los Angeles could potentially disrupt the communications of more than 450 network and Internet-connected companies operating within One Wilshire or immediately adjacent buildings.

Many of the carrier hotels do not have battery backup or even redundant power, as the “meet-me-rooms” fell under the “scale free” rapid growth in the late 1990s and 2000s when those rooms had little or no management, admistrative controls/regulation. This is gradually being brought under control in the largest facilities, and most smaller facilities such as the NAP of the Americas in Miami are very well controlled.

This was proven possible during the 1996 attack in London, and could occur again at any single, or multiple carrier hotel facilities located in the United States and other countries. It is a real problem, and one that is not lost on governments around the world.

What We Can, and Are Doing to Protect Our Communications Assets

The key to all applications and communications security is diversity and redundancy. Very few submarine cables are being built today without at least a diverse loop, or a restoral agreement with a competitive cable company. If there is a single location or cable disrupted across the oceans, and restoral capacity is planned, the problem can be managed.

For North American carriers and Internet Service Providers, having a network with multiple “peering” points in different geographic locations will minimize disruption, and in the case of most regional and global networks that is the case. In fact, most large Internet networks require interconnections in multiple locations before they will consider “peering” relationships. That is of course for both traffic management, as well as disaster planning.

This would mean an Internet Service Provider would best plan their network for both physical high capacity interconnections in multiple carrier hotels, but also peering or disaster peering plans for interconnecting at public peering points, such as PAIX, Any2, Equinix, and Telehouse in the US, or other major Internet Exchange Points (IXPs) in London, Amsterdam, Frankfurt, and other Asian cities.

For those carriers and ISPs planning long distance interconnections, care must be taken to ensure route diversity. In some cases, multiple carriers will purchase capacity on a wholesaler fiber provider’s infrastructure (such as Level 3 Communications, XO, and Time Warner), with the possibility several different network providers will buy capacity on their long distance route using the same cable system.

In many cases, such as cable landing stations dotting Long Island in New York, the actual cable connecting those facilities to the carrier hotels and their own cable capacity management facilities follow a single route. The risk is that a single backhoe, terrorist, or vandal could potentially cause serious international communications damage by simply cutting a trough across the roadway, or jumping into a manhole and cutting cable.

“Vandals are to blame for the massive phone and Internet outage in Silicon Valley on Thursday, an AT&T representative has confirmed.” (CNET News, 9 Apr 2009)

An incident in early 2009 near San Jose (California) where an individual performed a similar act of vandalism caused significant disruption across a large area in Northern California. The above story confirms the danger present when critical infrastructure is not adequately protected, and a single person can enter a manhole with the potential of such widespread impact.

Physical cable and route diversity guarantees should be part of every disaster recovery and route planning negotiation.

Those companies outsourcing their mission and company-critical data and applications must look at geographic diversity, with the ability to dynamically restart applications with industry and customer-acceptable recovery point and recovery time objectives. Cloud computing technology is getting closer to providing this for the future, but not quite ready for offering service level objectives.

The US Government Weighs In

The NSTAC believes the government should work with private industry to develop both operational best practices, as well as a solid, coordinated, threat warning system to assist carrier hotel, data center, and SuperNode operators to ensure the best level of security for national and global infrastructure.

Police departments should have some level of visibility into carrier hotels and SuperNodes, data centers, and telecommunications company central offices. Not because we want “big-brother” looking into our business, but because we want law enforcement to understand the nature of our telecom business, and what could potentially happen if human beings are able to damage local infrastructure (which includes emergency responder infrastructure).

The NSTAC recommends individuals employed at carrier hotels and critical infrastructure facilities go through an initial security check. This may be in part because the national authorities probably have either own communications running through SuperNodes, and have recognized there is a reasonable chance US government and military communications could also be damaged or disrupted in the event of a facility failure or loss.

The FCC and NSTAC also recognize the burden of responsibility ultimately falls on the individual networks and customers. Our economy and communications infrastructure depend on each company having good disaster recovery and diversity plans. Individual users must ensure we get service level agreements with a clause ensuring physical route diversity in backup and DR site interconnections.

ISPs need to multi-home their networks. Not just at a single interconnection point, carrier hotel, or IXP – but in separate facilities, preferably in separate geographies.

The government is working with representatives from the telecom, vendor (electronic switching equipment, etc), applications, business community, and government agencies on a continuing basis to ensure US policy is kept current, and the threat/risk of our current infrastructure is understood. The President’s National Security Telecommunications Advisory Committee (NSTAC) is now part of the US Department of Homeland security, and coordinates much of the discussion.

As users, we need to take action as well. We can do any or all of the following to ensure not only our security in global communications, but also at our businesses and home:

  • Ask your hosting provider if they have a disaster recovery plan – Get proof
  • Ask your network provider if they are multi-homed and multi-homed in multiple geographies – Get proof
  • Ask your provider if their physical diversity is using physically separate fiber routes
  • Ask your hosting provider if they have good coordination with law enforcement for local security – Get proof
  • Ask your international VPN (virtual private network) provider if their cable system has a restoral plan, or if you have geographic fail-over on a separate cable – Get proof

In short, the burden is ultimately on the end user to ensure their business or activity survives a major disaster. We must drive our vendors, and should seriously consider strongly supporting greater regulation and oversight of our critical infrastructure facilities to ensure we do not lose a resource that could potentially contribute to a global economic and communications catastrophe.

What are your concerns? Do you believe we are OK in our current telecom environment? Should we do more? Your comments are welcome.

John Savageau. Long Beach

Other articles in this series:

  • Risk and Security in the Telecommunications Industry Series – Part 1

Risk and Security in the Telecommunications Industry Series – Part 1

The worst case scenario – a strong earthquake strikes California, disabling the carrier hotel at One Wilshire, disrupting operations at submarine cable landing stations in both the Los Angeles area and central California, with a resulting tsunami hitting Hawaii, Guam, the Philippines, Taiwan, and Japan.

LA Hit by TsunamiCommunications are severed to most of the South Pacific, and severely degraded to allow for only emergency services and national defense usage within the west coast of the United States. Financial and government communications are disrupted and severely limited into Japan, Hong Kong, and China.

Telecom carriers in Singapore, Japan, Hong Kong, China, and Australia work frantically to restore cable, Internet, and telecom capacity from the Pacific submarine cable systems through the Indian Ocean to Europe and the US east coast. Seattle and San Francisco still have some connectivity, however cable systems from Grover Beach to San Diego are inoperable, limiting connections to those which were designed with automatic rerouting through North Pacific cable systems.

Sound crazy? No, it is not crazy, and there is a very good possibility a similar scenario will occur within our lifetime. In fact, when you look at the concern raised when the recent Los Angeles “Station Fire” threatened the telecommunications facility at Mt. Wilson many people were surprised at the potential disruption to both civilian and government communications if that facility were destroyed.

Los Angeles law enforcement uses the transmission towers to manage emergency communications throughout the LA area, fire departments, AM/FM radio stations, digital broadcast television stations – many were single threaded through Mt. Wilson as their primary local communications infrastructure. Not to mention the three letter federal agencies which use the facility for, well whatever they use it for…

Not a New Problem

Several US agencies have looked at this problem for many years. Agencies addressing the problem include the National Communications System (NCS), the Federal Communications Commission (FCC), the National Reliability and Interoperability Council (NRIC), the Department of Homeland Security (DHS), and an additional continuing special taskforce mandated by the president called the National Security Telecommunications Advisory Committee (NSTAC).

As recently as four years ago, an NSTAC report concluded “the telecommunications industry has shown that it is unlikely that a loss of assets in a single telecom hotel would cause a nationwide disruption of the (USA) critical telecommunications infrastructure.” Which may be true for the US infrastructure, as all major American carriers are interconnected at numerous locations scattered across the United States. In short, while the local LA community may be seriously disrupted in the event of the big earthquake, communications between Miami and New York would still be possible with little disruption.

AT&T, Sprint, Verizon, and QWEST are all well-meshed in their networks. As long as they are not sharing the same cable routes, or even in some cases the same actual cables, if the companies are subcontracting their long distance or local loops from other wholesale cable companies such as Level 3, XO, or Time Warner Telecom.

The International Factor

Ten years ago the United States could stand alone in our communications infrastructure. International communications were strong, and submarine cables were in use, however much of the international communications infrastructure was still done through use of satellites. Even if a submarine cable was disrupted, carriers could easily restore their communications through use of existing satellite restoral and recovery agreements.

Now, in the Internet age of high capacity telecom infrastructure, generally provisioned in multiples of 10 Gigabit per second links, satellite capacity has quickly become a fraction of the bandwidth driving international communications. Even the old telephone networks are being integrated by international and US carriers into their Internet infrastructure, often sharing the same circuits are streaming media, social networks, general web traffic, and other entertainment applications.

This will not be as easy to restore in the event California gets the big earthquake we all know is coming.

The Risks and Vulnerabilities Series

This series will look at several aspects of the telecommunications business, including:

  • International telecom vulnerabilities
  • Government interest, activities, and opinions on the risks and vulnerabilities of both US and international communications infrastructure
  • The role of the carrier hotel and internet exchange point in international communications
  • Interviews with people on the front lines of communication security
  • Recommendations for both the telecommunications industry, and the global user community

I look forward to reader comments, critiques, flames, constructive recommendations, and other ideas related to this discussion. Please add your comments to this blog, and I will ensure your voice is both heard and considered.

John Savageau, Long Beach

Where is Our Air? A Very Smoggy Day in LA

August 18th, 2009. Los Angeles, California, USA

Around 2:30 p.m. This afternoon I needed to make the drive from Long Beach (California) to Burbank (California). Normally this is not a bad drive, as the mountains are a pleasure to see off in the distance, and if you take the long way around (US Interstate 605 from Long Beach to the mountains, then follow the I 210 freeway around towards Burbank), particularly in the winter months, it is a clear, beautiful panorama of the LA Basin skyline.

Hazy Day in LANot today. A combination of smoke drifting into the LA area from fires near Santa Barbara, and weather conditions holding the smog in the LA Basin have created a condition that is, well downright disgusting. In Long beach we have the advantage of good breeze coming off the ocean which keeps the coastline generally clear, and not too unhealthy.

Today I could not see clearly for more than a mile or so in the Long beach area, and by the time I hit the ‘605 heading towards the mountains the visibility had dropped to a couple hundred meters. Headache time. Horrible headache time. Nearly had an accident headache time (and yes, I feel embarrassed and humbled by the fact…).

By the time I hit the northern edge of the basin air quality had degraded so badly it was hard to see more then about 200 meters. Traffic in LA, which under normal conditions is pretty aggressive, actually started slowing down because of reduced visibility. And perhaps the burning eyes, people trying to drive while choking up lung chunks. Yes, the air was really bad.

Not Just Me

Finally made it to Burbank. Having lived in LA for around 5 ½ years I thought I had seen it all, from ash fall during wild fires, to driving through a wildfire myself. We know that wildfires will generate a nasty haze that covers entire regions. We know all this because it is an annual routine we go through in California as the land renews itself.

AQMD Chart of LA Area 17 Aug 2009But this was even worse, as the offshore flow pushed bad air, including smoke – and smog into the ring of mountains surrounding the LA basin. So I decided to get some facts. Facts from the California Air Quality Management District/AQMD, and the US Government’s air quality website called “Air Now.”

Yes, indeed, today LA’s air quality was bad.

According to the California Air Resources Board “three-fourths of residents still live in areas that violate health standards for ozone, which causes respiratory disease.

And large swaths of the Inland Empire have 40 to 80 days a year that exceed the federally designated safe level for ozone, a colorless gas. About half the state, including major portions of Los Angeles, San Bernardino and Riverside counties, exceeds the health standard for fine particulates, which are linked to cancer, heart disease and other ailments.”

Californians Lose Some Enthusiasm Supporting Environmental Issues

In July 2009 the private Public Policy Institute of California/PPIC released a survey focusing on Californian’s opinions on climate change, air pollution, and energy policy. More than 2500 California residents completed the PPIC survey, and the result highlights included a couple surprising statistics such as:

Most residents (66%) support the 2006 California law (AB 32) that requires greenhouse gas emissions to be reduced to 1990 levels by 2020. Support has declined 7 points from July 2008 (73%) and 12 points from 2007 (78%). The decline is sharpest among Republicans (57% 2008, 43% today).

Only 23% of Californians saw air pollution as “a big problem” in their region, an 11-point drop since last year. In Los Angeles County, that segment dropped 17 points, to 30%, and in the Central Valley, it sank 15 points, to 36%.

Probably a lot of reasons for the decline in support of reducing greenhouse emissions, most likely due to concerns moving closer to the bad economy, and the immediate need for jobs over environmental concern. Other major areas showing changes, in some cases based on political party and partisan lines, include:

  • Effects of global warming
  • Government regulation of emissions
  • Cap and Trade
  • Carbon Taxes

The California Air Resources Board also released a lengthy report (524 pages) in July entitled the “California Almanac of Emissions and Air Quality – 2009 Edition.” The report includes air quality statistics for the period 1988 to 2007 for ozone, particulate matter (PM10), carbon monoxide (CO), nitrogen dioxide (NO2), sulfur dioxide (SO2), and lead.

Again, while Southern California is making great progress in improving air quality, it is still among the worst in the nation, giving us ample opportunities to improve our quality of air and quality of life.

Not Just a California Problem

The European Space Agency has made several extensive studies of global pollution trends. The most pressing world problem involves the release of Nitrogen Dioxide (NO2), which when combined with rain produces “acid rain.”

European Space Agency NO2 ChartAcid rain is a leading cause of deforestation, which of course kills trees, which of course further results in the earth’s inability to recover from over-production of greenhouse gases, in particularly the effects of carbon dioxide (CO2), which is (if you are a believer) the main cause of global warming.

NO2, when it returns to the earth in the form of acid rain, has the further effect of polluting lakes, rivers, and the oceans. This means fish kills, it means the fish you eat may not be healthy, and it means the food we produce in our farms and ranches will also have byproducts of airborne chemicals which join CO2 and NO2 as it is carried long distances by the wind.

The Environmental Protection Agency/EPA advises that

Taller smokestacks can lift pollutants high above a local community but help pollutants get into wind currents that can carry them hundreds, even thousands, of miles. For example, emissions from power plants and industrial boilers can travel hundreds of miles and contribute to smog, haze, and air pollution in downwind states.

One family of pollutants, nitrogen oxides, also reacts with other chemicals, sunlight and heat to form ground-level ozone. The nitrogen oxides and the ozone itself can be transported with the weather to help cause unhealthy air in cities and towns far downwind.
This means that our pollution produced in the LA area will eventually find its way into the wind leaving the LA Basin, and find its way to the Inland Empire, the Salton Sea agricultural areas, the wilderness areas of Arizona and New Mexico – an impact going far beyond the haze of Los Angeles.

Another Call to Arms

The risk to our city, our health, and the downstream impact of pollution is real. The California Air Quality Management District tells us thousands of LA-area resident die each year from ailments directly attributed to pollutants. The need to continue aggressively pursuing projects and plans to reduce our production of pollutants is real.

This blog article contains links to two very powerful reports (PPIC/CA-ARB). You don’t need to read through the entire report, but simply taking a look at the overviews on each report is quite enlightening. Then if you have a strong belief, on either side of the debate, let your elected representatives know.

John Savageau, Long Beach

%d bloggers like this: