Wiring Los Angeles Part 4 – WilCon’s Path to the Future

This is the fourth article in a series of interviews with Eric Bender, President of Wilshire Connection (WilCon), the largest independent telecom carrier in Los Angeles. In this segment Eric discusses the future of WilCon, including expansion outside of Los Angeles, wireless topics, relations with local utilities, and some great examples of WilCon’s flexibility in delivering telecom solutions to the LA community.

Pacific Tier: Outside of downtown LA , what is your expansion strategy for going to place like El Segundo, Las Vegas, or other cities, parts of the city?

Conduits Inside a Street VaultEric Bender: We’ve leased dark fiber from other carriers to get to other off-net locations such as El Segundo. We connect Equinix on Maple, so we can do lit transport into that facility. We’re working on a plan that would extend from there to the 365 Main location in El Segundo, and then with a second route back to downtown.

We’ve acquired several 10Gbps wavelengths from LA to Las Vegas, and into the Switch NAP facilities, so we can connect with all their switches and facilities, including the Super NAP, which is a 400,000sqft monster of a data center facility. So we do just backhaul or transport to Vegas. We’ve got approximately 10 gigs of traffic back and forth between here and there now.

I don’t see us building on a long haul type arrangement where we would build far outside of downtown LA. We’ll go off-net by structuring deals with existing customers and partners that we have to get fiber and connectivity into other locations. One of the things we are working on is connecting our buildings in LA to Phoenix, there’s a couple of different data centers in Phoenix – there is I/O, (etc).

Then we might go from Phoenix to Las Vegas, and from Las Vegas we already have to LA (which we already have), but also Las Vegas to San Jose/Santa Clara, and then back down to LA. So we basically have a ring from LA to Phoenix, to Las Vegas, San Jose, back to LA.

Pacific Tier: So would you have arrangements with Edison (Southern California Edison), or Burbank Power, or people like that to extend into Hollywood or Burbank or Glendale?

Eric Bender: Well, we do work with Edison and DWP (Los Angeles Department of Water and Power), and have used them to get to further out locations from downtown. The city of Burbank, we’ve not been successful and have not really chased hard to try and make a deal with them. Our initial efforts to work with them were rebuffed without comment – they had no interest in working with us, which surprised me.

I think we can be creative in building, like at 900 N. Alameda which we now have connected with dark fiber, and there is a variety of ways we can do that. We have connection and interconnection agreements AT&T to use their infrastructure so we can theoretically deploy assets throughout the city and state in coordination with AT&T’s network.

Pacific Tier: Would you consider for example going to San Diego using a capacity swap with Edison where they gave you a pair of fibers to San Diego?

Eric Bender: We do that already, well not to San Diego, but we have done a variety of barters, trades, or however you want to call it, with companies like Edison. With Equinix, with DWP I’d like to – but they have no interest and can’t by law (they are a public utility), and can’t really do that. Level 3, we’ve had discussions with them to do the same thing.

Because I’ve got all this infrastructure, whether it’s the conduit, or the fiber, and it’s all there, it’s paid for, there’s no debt, there’s no cost of capital to me – it’s there, so if I need to allocate a couple pairs of fiber going here or there on my network, and I can get something that I can generate revenue off of on somebody else’s network – we’ve done that many, many times and it’s worked out very well with a variety of companies.

Pacific Tier: Expanding a bit more on WilCon’s flexibility…

Eric Bender: We have no rigidity with anything we do. I think that one of the reasons we’ve been successful, and have continued to grow every month, basically where we’ve continued to grow over the previous month is because we’re not rigid and have a straight line approach to things.

Servicing WilCon's ConduitI think the advantage of not being a telecom guy by trade or training and education is that we look at things by how do we get the customers the things they need? When a customer needs something my first response is to say “OK,” and the second response is to call my technical guys and say “I just committed us to doing this by a certain point in time, make it happen.” And they do.

We’re a small company so we don’t have all the various layers of engineering groups and planning groups to say from here it has to pass from point “A,” to point “B,” to point “C,” before you can be told it can be done, or not be done, or how to do it. We just make it happen and get it done.

The story I tell that kind of tied the real estate to the telecom all together well for me is at one of our buildings, 700 Wilshire we had a t4enant in the building. This goes back to about early 1999. We had put conduit in the ground already, so it was probably early 1999, had already connected the conduit to One Wilshire, as well as 700 (Wilshire Blvd), and we had leased space to this tenant.

Part of their requirement was they needed a couple of DS3s between the meet me room at One Wilshire and their premises they had leased at 700. At that point we were only doing conduit deals with our carrier customers. They were going to Level 3 for this DS3, and Level 3 hadn’t really built out their network yet. Because we put Level 3’s first conduit in the ground in LA (it was conduit we’d installed in our trench and gave them, basically), and Level 3 was provisioning this DS3 through PacBell (now AT&T) or someone, and they were re-provisioning (the DS3) through them.

The timing for the tenant was going to be 90 days, or 120 days wait for the tenant to provision this DS3. The tenant came to us and knew we had started Wilshire Connection. So they came to us and said “we really need this, is there anything you can do for us?”

I didn’t know what to do, I mean we just had conduit between the buildings. So the customer said “all you have to do is this. We have the equipment, you’ve got fiber cable already (because we’d already deployed that), you need this piece of equipment, you’ve got this box on this end, that box on that end, and we’ll tell you how to do it and what works well.”

Two weeks later we had that DS3 traffic up and running between the buildings. So we got into the transport business. But that was because we were flexible, somebody needed something, (we had the resources), and we got it done.

If we can see there’s a reasonable return on investment and payback on the infrastructure or cost to deploy new equipment, we’ll do it. We got all this MRV DWDM equipment (dense wave division multiplexing) because we had a very limited amount of fiber into the (particular) building, so the only way I could provision this service (lit bandwidth) was with the DWDM gear – so we went out and bought it.

So the equipment was paid for within a year. The customers can come and go – it doesn’t really make any difference (other than we want to keep customers), because it is already paid for. So we’ll continue to do that, and that’s how we’ve continued to grow into the buildings that we have, as well as the transport circuits that we do between all these buildings, as well as connecting into Vegas, and the same to Phoenix, Santa Clara, and other cities.

Pacific Tier: So what is the future of WilCon, where do you go from what you have today?

Eric Bender: several years ago we had some strategy discussions, and (since) we are a dominant carrier in LA, we’re known around the world as the provider of choice, the connector of choice here in LA. We have a great relationship with Equinix, and CoreSite now (CRG), so we get a lot of referral business from those guys. We’ve provided such good service to our customers that big carriers such as TATA, PCCW, and others that they just come straight to us when they have new requirements.

So we had this discussion “let’s go and expand, and into other markets” and we took a very quick and dirty analysis and determined that it costs so much to build out and deploy in these other markets, and we know other companies have been doing similar kinds of things, and we might be providing a similar type of service. In order to do that (expanding into other markets) we’d have to take on an additional equity investor, or venture funding, and then all we’re doing is servicing somebody else’s debt.

So, does that make sense (expanding) or should we continue to be the provider here, and continue to grow which has made us a profitable company. So we said “we are gong to do that, and we’ll expand and provide services outside of LA in various ways that make sense, and does not require a huge capital outlay, and we’ll do some – but we’re not going to spend millions of dollars to deploy.”

The reason many companies went bankrupt was because they built these massive networks, and there was no r4evenue to service the debt on them.

So our future is to grow in selected and strategic ways. It will probably be between LA and other cities by usi9ng lit waves, and then establishing a POP (point of presence) in a couple buildings, and then maybe in some cases building a bit in those markets to connect those buildings.

In Dallas we’ve looked at buying an existing network, but there’s really no need to connect between those facilities – there’s no business with that. Chicago was no different. San Francisco, San Jose, Santa Clara are all so spread out its hard to justify a new build or new construction.

I think working with companies like Allied Fiber which is building their long haul network, and working with them to be the local metro piece to what they’re selling to their customers on the long haul – we can tag along with them, working well with them in LA, and possibly duplicate that with them in other markets.

Pacific Tier: Is there any wireless in WilCon’s future?

Eric Bender: I don’t really see that. We did look at partnering with some wireless, WiMAX, or other guys using Terrabeam or some of these free space optics. It seemed like they just had such a long way to go before it would make it worthwhile, and that in LA maybe it’s a problem. Trying to negotiate the line of sight, and rooftop antenna rights and those things,… that we didn’t really get anywhere. It became cost prohibitive I think.

So it would be nice to do something like that. We looked at that (wireless) to support NANOG (North American network Operator’s group) several years ago when they were here. We were in Library Tower (the US Bank Building) on the 56th floor. I had a line of sight directly from my window to the hotel, but it was too far. We would have needed a couple of hops to be able to deliver a gig (Gigabit connection), and it was a disaster for them to have it there (at that hotel).

They had to go buy a DS3 from AT&T (PacBell). If they had done the conference downtown I could have connected them with no issues.

Pacific Tier: Any final words you would like to say to the global telecom or Internet community, or Los Angeles? Why LA, or with WilCon, downtown LA is an attractive place to do business?

Eric Bender: We’re number one!

I think LA is a great place to do business. From a location perspective its perfect, as its the gateway to the US. You can get anywhere from LA, whether its physical transportation, or from a telecom infrastructure perspective. I think if you come to LA as a carrier, as a provider, as a bandwidth user, in this industry – and you’re going to locate in one of the half a dozen facilities (in downtown LA), whether an Equinix, Level 3’s gateway, One Wilshire, or TELX Carrier center.

Any of the main big facilities here (and Switch and Data) you are going to need, ultimately, to connect to customers that are in one of the other facilities – somehow you’re going to find us. You may be referred to us, do your research – ultimately you are going to find we can provide that connectivity.

We are really easy to do business with, as any of our customers will tell you. And I think that in all the years we’ve been in business, since 1998, I think there are maybe two deals we’ve lost because of price, that we were not the lower cost provider for that particular deal. And, we try to provision within 24~48 hours (of an order).

Pacific Tier: That is outstanding. Words for the global telecom and internet community from Eric Bender, President of Wilshire Connection. Thank you very much.

John Savageau for Pacific Tier Communications, Long Beach

====

This concludes our interview with Eric Bender, President of Wilshire Connection in Los Angeles.  you can contact Eric at ebender@wilcon.com for more information on Wilshire Connection.

The entire interview is available online.

Previous entries in this series include:

  • Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
  • Part 2 – WilCon Takes to the Streets
  • Part 3 – WilCon Manages Infrastructure Risks

Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

This is part three is a series of interviews with Eric Bender, president of Wilshire Connection. Wilshire Connection, or WilCon, is the largest independent local network and neutral fiber infrastructure provider in downtown Los Angeles. In this segment Eric discusses how WilCon managed risk to their network during the initial construction process, the continuing management of critical telecommunications infrastructure, and the role WilCon could play in the event of a major incident impacting the telecom industry in Los Angeles.

Pacific Tier: (on the topic of utility gas and electricity) What risk is there to the infrastructure in downtown LA of an explosion from either electricity or gas, and what would that do to your conduits if it occurred?

Eric Bender: Interesting question… I haven’t really thought much about that, it would, depending on where, you never want to see that. Fortunately with power, Street Utility Tagging in Downtown Los Angelesthe lines typically won’t explode, it is the transformers, which mostly are in the buildings.

The way LADWP (Los Angeles Dept of Water and Power) sets up they bring the high voltage lines into the building, so if the transformer blows up it will be in the building. The transformer is typically not out in the street – but they do have some vaults out in the street, and they have had some explosions, but they have been contained within the vault. We don’t run through any of their vaults so from that perspective we’re OK.

Somebody who is digging and hits a gas line… that’s a different story.

I can say in the roughly 12 years that we’ve had conduit in the ground, and since we were one of the first to dig, I can say that we are lower in the ground, and more protected in many respects, and more of a straight line in routes we’re going without having to jog around any other existing infrastructure that came later.

But we’ve never had one of our conduits damaged or cut. Or interfered with…

The difference between how our conduits or duct banks are typically done vs. many other carriers is that we put in massive amounts of conduit. We use (typically) four inch PVC conduit, and I don’t think we have a single one (trench) which has less than six or eight four inch conduits. Most of them have multiples of that, for example going on Wilshire Blvd between Grand and Figueroa we have an average of conduit in that whole duct bank is probably close to sixty four inch conduits.

Because fifty of them go into One Wilshire, although they kind of peel off in WilCon Conduit Duct Bank in Los Angelesdifferent directions from there, we’ve also got a main ’48 and they lateral off down into other streets into buildings, so there’s some doubling going on.

But in some areas we have up to sixty four inch conduits so that’s the size of a desk, or bigger, and that duct bank is encased in a concrete slurry around it, surrounded by a couple addition feet of dirt and asphalt on top.

So if anybody digging is going to hit (the duct bank) typically there is warning tape on top of the conduit, concrete, or in the concrete they are going to pull it up. They are going to hit (the tape) or something before they hit the conduit. So unless it is some young, uncontrollable (person) with the backhoe who is on a rampage – no matter what you do you can’t control that.

NOTE: Whenever a company opens a street for utility construction or maintenance you will normally have construction observers and safety observers from not only the company opening the street, but also each company with conduit or utility infrastructure in the immediate area of digging.

But some of the other carriers that put in one or two conduits, they are the ones at risk, like for example an MCI, Verizon, or a QWEST in some cases they typically put in just the two conduits… And you could rip through that even if its encased in concrete before you realize what you are doing.

The conduits are just this big, versus this big (Eric shows a note book size to represent two conduits vs. his desktop to represent WilCon’s conduits).

Pacific Tier: Do you consider yourself the only truly neutral facilities-based carrier in downtown LA?

Eric Bender: I think others consider themselves neutral, but they have other motives as well. I don’t care if I sell dark fiber or lit transport. We can do either, and it doesn’t matter to me which one they want. We have so much fiber, and the infrastructure to continue pulling more and more fiber that I’m never worried about running out of capacity for dark fiber.

A Level 3, or a QWEST, or an XO, they run a network, and they’re obviously not neutral. DWP (LADWP), they’re somewhat neutral because they don’t seem to care whether you take fiber or lit services. I don’t know what they do with lit services or on the network side of things – honestly. We’ve leased from them (LADWP) dark fiber to get access to some off-net buildings, and they’re very easy to work with. They are very rigid, and have no flexibility (LADWP is a public utility), but they’re easy to work with.

We’re probably, as a straight, neutral, not really care who you connect with, we may not be the only one (neutral fiber network), but one of the only ones that would be neutral.

Pacific Tier: One Wilshire is traditionally a center of communications in Los Angeles. Some people think it is a high risk location because there is so much on the 4th floor and other parts of the building. Some people think that it is meaningless – that it doesn’t have that much value. Do you think the 4th floor of One Wilshire today is a critical piece of infrastructure, or do you think it is something that is just there, that could be bypassed when and if ever needed?

Eric Bender: I think at this point it would still be considered a critical facility. A lot of carriers and other companies have facilities or locations elsewhere, but because of the way they’ve built their networks from the beginning, One Wilshire has always been the central point for them.

They may not have grown and expanded there, or they may have moved things off to other locations, such as 600 W. 7th, 818 (W. 7th ), or even outside of LA, and use a company like us, or some other carrier to make that interconnection or virtual connection between their two facilities. One Wilshire tends to have been their primary facility.

I think that over the years that’s more applicable to legacy carriers, the bigger carriers, the ones that have been around for a long time.

I think the Internet type of carrier that’s either VoIP or an Internet company, content CDNs (content delivery networks), and those – One Wilshire’s not as important to them at all because in my limited technical knowledge it is easier to reroute that traffic to other servers – they have more mirror facilities than a switch would have on the telecom side of things.

Pacific Tier: Is there a business continuity plan, or disaster plan, in the event One Wilshire or another facility like 60 Hudson (New York), or the NAP of the Americas (Miami) anybody has thought about or put on the shelf in the event one of those critical facilities has a catastrophic failure?

Eric Bender: I am not aware of any common, for the greater good, where all the carriers have participated in developing that, or working out some kind of contingency plan. That’s actually a really good question.

You’d think that after 9/11 where the infrastructure was so significantly damaged that in various other markets such as LA, Chicago, that there would be some kind of a group of organizational effort to have dealt with that. I am not aware of one. They may have one, but I am not aware of it, and they never invited me to participate.

Pacific Tier: Is WilCon positioned, in the case of a worst case scenario in Los Angeles, to assist the community and assist the industry in recovering to an alternate facility if that occurred?

Eric Bender: Sure, I mean, our infrastructure that we built, and that we control and own, is all primarily downtown LA, so… in a worst case One Wilshire becomes untenable, well a lot of our fiber doesn’t all home run into One Wilshire, but a lot of it does go in and turn around, coming back out again.

There would be disruption, but it could be brought out and bypassed. We have diverse paths into most buildings that we connect so we can certainly do it.

Not in LA, but when in the Mediterranean last year when they had the three or four cuts, (several of) our customers were impacted. I sent them an email and said I doubt there is anything we can do here , but if there is anything you need that we can help you with, let me know and we can work with you.

And actually two of the customers said “yeah we need to reroute some connections to put it on a different side of their ring (in the Med and Pacific) that we could do in about five hours with a couple pairs of fiber for them, and they were able to reroute their traffic, or some of their traffic, and lessen the impact of those (submarine fiber) cuts.

Pacific Tier: So WilCon would consider yourselves a very flexible, agile part of a recovery plan, and would not be rigid in your provisioning process, and that you would work with the community to recover from a disaster?

Eric Bender: I agree with that!

=====

This ends the third segment of this series. In the next part, Eric will discuss more of the future of Wilshire Connection, including his visions for expanding WilCon into new markets.

The entire interview is available online.

Previous entries in this series include:

  • Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
  • Part 2 – Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

Wiring Los Angeles Part 2 – WilCon Takes to the Streets

This is the second part of an interview with Eric Bender, President of Wilshire Connection. In this segment Eric talks about the period in 2000 preceding the Democratic Nation Convention, and the aggressive industry build out of conduit, fiber, and telecommunications infrastructure in the downtown Los Angeles area.

Pacific Tier: At what point do you think the city of Los Angeles figured out this would be a really good thing for LA, and it would bring more business and money into downtown?

Eric Bender: I don’t think they ever came to that conclusion. What happened was in 2000 the Democratic Convention was going to be down at the Staples Center in downtown, so I think it was in early 2000 or at the very end of 1999 the city called a meeting and notified all the telecom carrier that had been active or Eric Bender President Wilshire Connectionbuilding or doing things that as a part of the preparations for the convention that they would be repaving the streets, resurfacing certain streets, and there was going to be an absolute five year moratorium on any digging or street construction work as a result of this. They wanted the streets to look pretty on TV, and that was fine.

So I don’t think they thought about how much money, or how beneficial the telecom network was going to be to the city, they realized they would need to do something for this convention. The result of that was that it threw somewhat of a panic into the carriers that were in the infancy of building out their networks, and building throughout LA, and it forced a massive infrastructure improvement project, because there was a very limited amount of time before the city was going to shut everything down.

I think the convention was in July or August, and by the end of May everything had to be done. So, we really had about four or five months to build everything and our network probably tripled in size during that period of time. We were one of the only companies that had an active permit that we had pulled for a small segment that we were building that hadn’t started construction yet because it was not that critical to us at the time.

But because of us having an active permit for about a two block, few hundred feet, maybe five or six hundred feet of conduit the city realized they had a major problem that these companies had to be building, had to be constructing – they couldn’t just shut them down for five years. The city did understand that was a problem.

So they said you can build, we will expedite and streamline the process and anyone who has an active permit, well everyone can tag along with that permit. We’ll just change it and build out from there.

So, with our little permit, and the little segment, I think it was on Grand from Wilshire down to 7th St., and then a little bit more. We basically parlayed that and built it up to 5th St., then up to Figueroa, and literally tripled our network. So we became the lead builder from that permit on a tremendous amount of other conduit that was built in the city.

Basically they couldn’t keep up with all the activity because there were intersections in downtown LA where there were three backhoes digging for three separate projects. We were doing one, QWEST was doing their own, they didn’t really want to participate much with anyone, and MCI, MFS didn’t want to share conduit so they were doing little things.

And I remember at 7th and Grand there were three different construction crews and project going on at the same time, and for me it was just a wonderful time because we had this permit, so we had a lot of leeway, and I would walk the streets, stand on the street corner, and these other companies, you know Level 3 and QWEST actually did participate, XO… You know I would say “we’re going down this street to this building,” and they’re going “I’m going that way,” and we’d shake hands. “I’m taking two conduits in yours, you’re taking a conduit in mine…”

I don’t like to use the term “wild, wild, west,” but it was really a very wild and fun time. My best times were just walking around with all the construction going on, with all the lane closures and all the activity. The traffic was backed up, cars were beeping, and would be driving by and I would just be standing there looking at the big hole in the ground with a big smile on my face. It really was the best time.

Pacific Tier: Did you have any catastrophic backhoe cuts or anything disruptive during consutruction? 

Eric Bender: We, during our construction our guys, we never had an incident where we hit any other conduit. There was a water or a sewer line that got hit, and nicked, and that was something that that was not actually on the infrastructure plan. And it wasn’t marked on the street so it wasn’t our problem so much.

There was a close call with a gas line. But nothing happened. But for the most part (Eric knocking on his wooden desk) they did a good job.

The toughest part to build, which we were actually the participant in and XO was the lead builder on 7th, I think somewhere between Olive and Grand, it shows you the history of LA but, back when they had the street cars, the RED Line, YELLOW Line, and all these other street cars, they never really demoed (demolished) those out.

They just over the years paved over and over and over. As they were excavating and going down the road, the path where they were going, and which later became obvious because of the railroad, they had to rip through all the railroad ties. It was terrible.

Because of that, then they had to get an archaeologist because you know you have bricks from the support for the railroad, because they are historical. I don’t know what they would do with it, it is in the middle of 7th St (a major road in downtown LA) and Grand Ave, that can’t, you know set up a museum in the middle of it, but it shut the project down. The city had some very peculiar rules that you cannot dig on Sundays that go back to the old blue laws basically, no construction activity on Sundays.

Some streets they won’t let you work during the week, you know it goes back to the days when downtown was a main shopping area, so you know, so you couldn’t dig during the week. You could only dig on Saturdays from nine (a.m.) till three (p.m.) which means you get about from ten till two because you have clean up and everything.

So some projects that should have been done within a week took like eight weeks to get done. So the city tried, but they were not the easiest. The only company that got to work on Sunday was QWEST, because, well, nobody is really sure exactly what happened or how, but they did have several projects where they were working on Sundays.

====

This ends part two of the “Wiring LA” interview series. Part Three will explore some of the risks of building such a dense concentration of telecom projects within the downtown LA area.

The entire interview is available online.

Previous entries in this series include:

Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection

 

John Savageau, Long Beach

 

Wiring Los Angeles with Eric Bender, President of Wilshire Connection

Downtown Los Angeles is among the most densely connected telecommunication hubs in the world. A dozen buildings in LA’s city center house the world’s largest Internet networks, Laying Down Telecom Conduits and Cabletelecommunication carriers, content management networks, and entertainment companies – interconnected through a complex mesh of submarine fiber optic cables, terrestrial cables, and internet exchange points.

With more than 500 networks and carriers operating within the LA city center, one company stands out from the crowd as a leader in bringing the global telecommunications and Internet community together. Eric Bender is president of Wilshire Connection, a facility-based carrier focusing on providing neutral, high capacity fiber optic cable interconnecting the most important buildings in Los Angeles.

Wilshire Connection (WilCon) has a vision to free the LA telecom and Internet-enabled community from the burden of developing a highly meshed inter-building infrastructure, allowing each company to focus on bringing value to their global network interconnections and relationships. Eric Bender, President of Wilshire Connection, is the man behind WilCon’s dramatic success in wiring Los Angeles.

This interview will cover, in a multi-article series, the history of modern telecommunications in downtown Los Angeles, the role Wilcon played in LA’s redevelopment, the period of rapid and chaotic build out prior to the Democratic National Convention in 2000, the risk of high density telecom infrastructure, and the future vision of Eric Bender and Wilcon.

Pacific Tier: Today we have Eric Bender, President of Wilshire Connection. Eric, can you tell us a little about yourself, and how Wilcon came around?

Eric Bender: Sure, we founded Wilcon, Wilshire Connection back in 1998. I am one of the founders and one of the partners, and have run the company pretty much from the beginning. I’m a real estate guy by background. I graduated from college and have always worked in the real estate field, whether its commercial property, management, brokerage, home build, whatever…

Back in late 1996 I started working for a Hong Kong based family which was acquiring some office buildings in downtown LA, one of which was 611 Wilshire Blvd which was directly across the street from One Wilshire. Back at that time telecom deregulation had just been passed, the 1996 Deregulation Act, and there was a boom of CLECs formed and started. Nextlink, the precursor to Level 3 (Kiewit and Sons), MFS was expanding into a variety of things, Worldcom, a variety of these companies were just starting up in that time frame.

For a lot of different reasons One Wilshire had become kind of a focal point for telecom, primarily because it is a 30 story building, had very clear line of sight to the east, and back in the day when microwave transmission was the primary factor, (One Wilshire) had great eastward pointing microwave capabilities. And because of this many of the deregulated companies had got into One Wilshire.

Because our building 611 Wilshire was directly across the street, and only 50% occupied, and One Wilshire was still an office building that only had a meet me room component to it, – that was really it, from a real estate side of things we said “we have all the space, you have all these new companies coming and can’t do it, send them over to us, we’ll fill our building with them, and let’s put some conduit between the buildings together as a virtual extension of (the meet me room) of One Wilshire.

Additionally we put in some conduits that the One Wilshire building actually provided for.

A year or two later we had bought, and were converting other buildings in downtown (700 Wilshire, 818 W. 7th St.) and saw a need to have those buildings connected back to 611 and One Wilshire.

Standing on the street corner thinking about it we decided that we should become a facilities-based carrier. We went and got our authority from the state to become a carrier, CLEC, and I can’t remember the other authorities we received. Then we decided that’s how we are going to do it. We started digging the streets building the network and connecting the buildings, our own buildings, together.

We decided as a way to fund the build out, because at the time, 1997 and 1998 a lot of buildings had empty space in downtown LA, they all saw what was happening not only with our buildings, but other buildings that had empty space, and saw how telecom was taking massive amounts space. And the reality was landlords wanted telecom companies as tenants because they paid higher rent, they paid for their own improvements to build out their space, they paid for their own utilities, the buildings did not have to provide janitorial, nor provide much in the line of services.

So we created, as part of our model with Wilshire Connection, the ability to connect all these other buildings into our network. This tied them back to One Wilshire and all these other telecom buildings. And that was our methodology of raising funds to build our improvements to our infrastructure.

Pacific Tier: So what role do you believe Wilcon played in the development of downtown LA as a telecom center?

Eric Bender: Well, trying to be as modest as I can, we probably played the primary role in that, because, several buildings are telecom buildings today because we were building our network and including them. They may have become a telecom building in some way differently, but the fact that we connect them into our network which covers and ties all the building together on a neutral basis was really the instrumental part, including 600 W. 7th St, 800 Hope, 818 W. 7th St., which has Level 3’s gateway, Equinix, and others – brought these buildings back to One Wilshire and to each other – together they became the success that they are.

Pacific Tier: What was your criteria for selecting the buildings that you POPed (built a point of presence) or entered, or dropped cable?

Eric Bender: We were note really picky. At the time we were building the network it depended on which buildings came to us, what they were willing to contribute to the cost of (build out), and how important it was to them to be connected to the network.

Pacific Tier: The actual process of laying conduit in downtown Los Angeles, did the city support you?

Eric Bender: Not initially, it was quite challenging. Part of it was because a lot of this activity was new and we were building our network, planning our network to cnnect the buildings. But you had the other companies that were building their networks, such as Level 3, MFN (actually they came a bit later), MCI was adding a lot more conduit and buildings, and others. The city was not prepared, I don’t think, from an engineering perspective and plan check to deal with all that, so they were backed up, and you had companies like Level 3 which was building a network throughout the entire city basically.

The city’s engineering department that reviewed the plans – you would go in and the entire office would be stacked four, five , six feet high with plans of what they were planning to build. And it took months and months, 6 to 8 months at times.

This is the end of part 1 of our interview with Eric Bender. Additional segments to the interview will discuss the actual buildout of Wilscon’s network prior to the Democratic National Convention in 2000, the risk of having such high density infrastructure in downtown Los Angeles, and future plans for Wilcon and Eric Bender.

====

John Savageau, Long Beach

You can listen to the entire interview online at Pacific Tier

 

Selecting Your Data Center Part 3 – Understanding Facility Clusters

Now that we have determined the best geographic location for our data center, it is time to evaluate local facility options. The business concept of Splicing Fiber Optic Cableindustry clustering is valid in the data center industry. In most locations supporting carrier hotels and Internet Exchange Points you will normally see a large number of data centers within a very close proximity, offering a variety of options, and a maze of confusing pitches from aggressive sales people.

The idea of industry clustering says that whenever a certain industry, such as an automobile manufacturer selects a location to build a factory or assembly plant, others in the industry will eventually locate nearby. This is due to a number of factors including the availability of skilled workers within that industry, favorable city support for zoning, access to utilities, and proximity to supporting infrastructure such as ocean ports, rail, population centers, and communications.

The data center industry has evolved in a similar model. When you look at locations supporting large carrier hotels, such as Los Angeles, Seattle, San Francisco, London, and New York, you will also see there are many options for data centers in the local area. For example in Los Angeles, the One Wilshire Building is a large carrier hotel with collocation space within the building, however there are at many options within a very close proximity to One Wilshire, such as Carrier Center (600 W. 7th), 818 W.7th St., the Garland Building, 530 W. 6th, the Quinby Building, and several others.

The bay area has similar clusters stretching between Palo Alto and San Jose, and Northern Virginia (Ashburn, Reston, Herndon, Sterling, Vienna) has a high density of facilities in proximity to the large Equinix Exchange Point in Ashburn.

When you have data center clusters, you will also find each facility is either fully meshed with commercial dark fiber interconnecting the buildings, or has several options of network providers offering competitive “lit” services between buildings. 

Note the attached picture of downtown Los Angeles, showing all the major colocation facilities and physical interconnection between the facilties with high capacity fiber (Wilshire Connection).

Discriminating Features Among Data Centers

The Uptime Institute, founded in 1993 (and recently acquired by the 451 Group) has long been a thought leader in codifying and classifying data center infrastructure and quality standards. While many may argue the Uptime Institute is focused on enterprise data center modeling, the same standards set by the Uptime Institute are a convenient metric to use when negotiating data center space in a commercial or public data center.

As mentioned in Part one of this series, there are four major components to the data center:

  • Concrete (space for cabinets, cages, and suites)
  • Power
  • Air-conditioning
  • Access to telecom and connectivity

Each data center in the cluster will offer all the above, at some level of quality scale that differs from others in the cluster. This article will focus on facility considerations. We will look at the Uptime Institute’s “tiered” system of data center classification in a later post.

Wilshire Connection Los AngelesConcrete. Data centers and carrier hotels supporting major interconnection points or industry cluster “hubs” will generally draw higher prices for their space. The carrier hotel will draw the highest prices, as the value of being colocated with the telecom hub brings more value to either space within the meet-me-room, or adjacent space within the same building. Space within the carrier hotel facility is also normally limited (there are exceptions, such as the NAP of the Americas in Miami), restricting individual tenants to a few cabinets or small cages.

The attraction of being in or near the carrier hotel meet-me-room is not necessarily in the high cost cabinet or cage, it is the availability of multiple carriers and networks available normally with a simple cross connect or jumper cable, rather than forcing networks and content providers to purchase/lease expensive backhaul to allow interconnection with other carriers or networks collocated in a different facility.

Meet-me-rooms at the NAP of the Americas, 60 Hudson, the Westin Building, and One Wilshire in the US, and Telehouse in London offer meet-me-room interconnections with several hundred potential interconnection partners or carrier within the same main distribution frame. Thus the expensive meet-me-room cabinets and cages make up their value through access to other carriers with inexpensive cross connects.

NOTE: One thing to keep in mind about carrier hotels and meet-me-rooms; most of the buildings supporting these facilities were not designed as data centers, they are office conversions. Thus the electrical systems, air-conditioning systems, floor loading, and security infrastructure are not as robust as you might find in a nearby facility constructed as a data center or telecom central office.

Facilities near the carrier hotel will generally have slightly lower cost space. As industry concerns over security within the carrier hotel increase, and the presence and quality of adjacent buildings exceeds that of the carrier hotel, many companies are reconsidering their need to locate within the legacy carrier hotel. In addition, many nearby collocation centers and data centers are building alternative meet-me-rooms and distribution frames within their building to accommodate both their own tenants, as well as offering the local community a backup or alternative interconnection point to the legacy carrier hotel.

This includes the development of alternative and competitive Internet Exchange Points.

This new age of competitive or alternate meet-me-rooms, multiple Internet Exchange Points, and data center industry clusters gives the industry more flexibility in their facility selection. In the past, Hunter Newby of Allied Fiber claimed “if you are not present in a facility such as 60 Hudson or the Westin Building, you are paying somebody else to be in the building.” This has gradually changed, as in cities such as New York a company can get near identical interconnection or peering support at 111 W. 8th St or 32 Ave of the Americas as available within 60 Hudson.

As the clusters continue to develop, and interconnections between tenants within the buildings become easier, then the requirement to physically locate within the carrier hotel becomes less acute. If you are in Carrier Center in Los Angeles, the cost and difficulty to complete a cross-connection with a tenant within One Wilshire has become almost the same as if you were a tenant within the One Wilshire Building. Ditto for other facilities within the industry cluster. In fact, the entire metro areas of New York, the bay area in Northern California, Northern Virginia, and Los Angeles have all become virtual extensions of the original meet-me-room in the legacy carrier hotel.

The Discriminating Factor

Now as potential data center tenants, we have a somewhat level playing field of data center operators to choose from. This has eliminated much of the interconnection part of our equation, and allows us to drill into each facility based on our requirements for:

  1. Cost/budget
  2. Available services
  3. Space for expansion or future growth
  4. Quality of power and air conditioning

Part four of this series will focus on cost.

As always, your experiences and comments are welcome

John Savageau, Long Beach

Prior articles in this series:

Wilshire Connection photo courtesy of Eric Bender at www.wilshireconnection.com

%d bloggers like this: