Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

This is part three is a series of interviews with Eric Bender, president of Wilshire Connection. Wilshire Connection, or WilCon, is the largest independent local network and neutral fiber infrastructure provider in downtown Los Angeles. In this segment Eric discusses how WilCon managed risk to their network during the initial construction process, the continuing management of critical telecommunications infrastructure, and the role WilCon could play in the event of a major incident impacting the telecom industry in Los Angeles.

Pacific Tier: (on the topic of utility gas and electricity) What risk is there to the infrastructure in downtown LA of an explosion from either electricity or gas, and what would that do to your conduits if it occurred?

Eric Bender: Interesting question… I haven’t really thought much about that, it would, depending on where, you never want to see that. Fortunately with power, Street Utility Tagging in Downtown Los Angelesthe lines typically won’t explode, it is the transformers, which mostly are in the buildings.

The way LADWP (Los Angeles Dept of Water and Power) sets up they bring the high voltage lines into the building, so if the transformer blows up it will be in the building. The transformer is typically not out in the street – but they do have some vaults out in the street, and they have had some explosions, but they have been contained within the vault. We don’t run through any of their vaults so from that perspective we’re OK.

Somebody who is digging and hits a gas line… that’s a different story.

I can say in the roughly 12 years that we’ve had conduit in the ground, and since we were one of the first to dig, I can say that we are lower in the ground, and more protected in many respects, and more of a straight line in routes we’re going without having to jog around any other existing infrastructure that came later.

But we’ve never had one of our conduits damaged or cut. Or interfered with…

The difference between how our conduits or duct banks are typically done vs. many other carriers is that we put in massive amounts of conduit. We use (typically) four inch PVC conduit, and I don’t think we have a single one (trench) which has less than six or eight four inch conduits. Most of them have multiples of that, for example going on Wilshire Blvd between Grand and Figueroa we have an average of conduit in that whole duct bank is probably close to sixty four inch conduits.

Because fifty of them go into One Wilshire, although they kind of peel off in WilCon Conduit Duct Bank in Los Angelesdifferent directions from there, we’ve also got a main ’48 and they lateral off down into other streets into buildings, so there’s some doubling going on.

But in some areas we have up to sixty four inch conduits so that’s the size of a desk, or bigger, and that duct bank is encased in a concrete slurry around it, surrounded by a couple addition feet of dirt and asphalt on top.

So if anybody digging is going to hit (the duct bank) typically there is warning tape on top of the conduit, concrete, or in the concrete they are going to pull it up. They are going to hit (the tape) or something before they hit the conduit. So unless it is some young, uncontrollable (person) with the backhoe who is on a rampage – no matter what you do you can’t control that.

NOTE: Whenever a company opens a street for utility construction or maintenance you will normally have construction observers and safety observers from not only the company opening the street, but also each company with conduit or utility infrastructure in the immediate area of digging.

But some of the other carriers that put in one or two conduits, they are the ones at risk, like for example an MCI, Verizon, or a QWEST in some cases they typically put in just the two conduits… And you could rip through that even if its encased in concrete before you realize what you are doing.

The conduits are just this big, versus this big (Eric shows a note book size to represent two conduits vs. his desktop to represent WilCon’s conduits).

Pacific Tier: Do you consider yourself the only truly neutral facilities-based carrier in downtown LA?

Eric Bender: I think others consider themselves neutral, but they have other motives as well. I don’t care if I sell dark fiber or lit transport. We can do either, and it doesn’t matter to me which one they want. We have so much fiber, and the infrastructure to continue pulling more and more fiber that I’m never worried about running out of capacity for dark fiber.

A Level 3, or a QWEST, or an XO, they run a network, and they’re obviously not neutral. DWP (LADWP), they’re somewhat neutral because they don’t seem to care whether you take fiber or lit services. I don’t know what they do with lit services or on the network side of things – honestly. We’ve leased from them (LADWP) dark fiber to get access to some off-net buildings, and they’re very easy to work with. They are very rigid, and have no flexibility (LADWP is a public utility), but they’re easy to work with.

We’re probably, as a straight, neutral, not really care who you connect with, we may not be the only one (neutral fiber network), but one of the only ones that would be neutral.

Pacific Tier: One Wilshire is traditionally a center of communications in Los Angeles. Some people think it is a high risk location because there is so much on the 4th floor and other parts of the building. Some people think that it is meaningless – that it doesn’t have that much value. Do you think the 4th floor of One Wilshire today is a critical piece of infrastructure, or do you think it is something that is just there, that could be bypassed when and if ever needed?

Eric Bender: I think at this point it would still be considered a critical facility. A lot of carriers and other companies have facilities or locations elsewhere, but because of the way they’ve built their networks from the beginning, One Wilshire has always been the central point for them.

They may not have grown and expanded there, or they may have moved things off to other locations, such as 600 W. 7th, 818 (W. 7th ), or even outside of LA, and use a company like us, or some other carrier to make that interconnection or virtual connection between their two facilities. One Wilshire tends to have been their primary facility.

I think that over the years that’s more applicable to legacy carriers, the bigger carriers, the ones that have been around for a long time.

I think the Internet type of carrier that’s either VoIP or an Internet company, content CDNs (content delivery networks), and those – One Wilshire’s not as important to them at all because in my limited technical knowledge it is easier to reroute that traffic to other servers – they have more mirror facilities than a switch would have on the telecom side of things.

Pacific Tier: Is there a business continuity plan, or disaster plan, in the event One Wilshire or another facility like 60 Hudson (New York), or the NAP of the Americas (Miami) anybody has thought about or put on the shelf in the event one of those critical facilities has a catastrophic failure?

Eric Bender: I am not aware of any common, for the greater good, where all the carriers have participated in developing that, or working out some kind of contingency plan. That’s actually a really good question.

You’d think that after 9/11 where the infrastructure was so significantly damaged that in various other markets such as LA, Chicago, that there would be some kind of a group of organizational effort to have dealt with that. I am not aware of one. They may have one, but I am not aware of it, and they never invited me to participate.

Pacific Tier: Is WilCon positioned, in the case of a worst case scenario in Los Angeles, to assist the community and assist the industry in recovering to an alternate facility if that occurred?

Eric Bender: Sure, I mean, our infrastructure that we built, and that we control and own, is all primarily downtown LA, so… in a worst case One Wilshire becomes untenable, well a lot of our fiber doesn’t all home run into One Wilshire, but a lot of it does go in and turn around, coming back out again.

There would be disruption, but it could be brought out and bypassed. We have diverse paths into most buildings that we connect so we can certainly do it.

Not in LA, but when in the Mediterranean last year when they had the three or four cuts, (several of) our customers were impacted. I sent them an email and said I doubt there is anything we can do here , but if there is anything you need that we can help you with, let me know and we can work with you.

And actually two of the customers said “yeah we need to reroute some connections to put it on a different side of their ring (in the Med and Pacific) that we could do in about five hours with a couple pairs of fiber for them, and they were able to reroute their traffic, or some of their traffic, and lessen the impact of those (submarine fiber) cuts.

Pacific Tier: So WilCon would consider yourselves a very flexible, agile part of a recovery plan, and would not be rigid in your provisioning process, and that you would work with the community to recover from a disaster?

Eric Bender: I agree with that!

=====

This ends the third segment of this series. In the next part, Eric will discuss more of the future of Wilshire Connection, including his visions for expanding WilCon into new markets.

The entire interview is available online.

Previous entries in this series include:

  • Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
  • Part 2 – Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

Shrouding the Net Neutrality Debate in a Cloud of Politics

The FCC finally moved the network neutrality debate forward Thursday, voting to begin developing open Internet regulations. The topic has become quite interesting over the past week, as strong-willed proponents and opponents of Internet Network Neutralitynet neutrality turn up campaigns to influence law makers prior to voting on any net neutrality principles that may become law.

The debate is actually quite simple – should the government regulate, or not regulate the Internet? That discussion revolves around the six principles of network neutrality proposed by the FCC:

Under the draft proposed rules, subject to reasonable network management, a provider of broadband Internet access service:

  • would not be allowed to prevent any of its users from sending or receiving the lawful content of the user’s choice over the Internet;
  • would not be allowed to prevent any of its users from running the lawful applications or using the lawful services of the user’s choice;
  • would not be allowed to prevent any of its users from connecting to and using on its network the user’s choice of lawful devices that do not harm the network;
  • would not be allowed to deprive any of its users of the user’s entitlement to competition among network providers, application providers, service providers, and content providers;
  • would be required to treat lawful content, applications, and services in a nondiscriminatory manner; and
  • would be required to disclose such information concerning network management and other practices as is reasonably required for users and content, application, and service providers to enjoy the protections specified in this rulemaking.

The Pro Argument of Network Neutrality

Oddly, former adversaries Google and Verizon issued a joint statement regarding their position on net neutrality. Both companies have significantly changed their positions since the debate originally hit the headlines in 2005, with a highlight of the joint statement:

“For starters we both think it’s essential that the Internet remains an unrestricted and open platform — where people can access any content (so long as it’s legal), as well as the services and applications of their choice.
Transformative is an over-used word, especially in the tech sector. But the Internet has genuinely changed the world. Consumers of all stripes can decide which services they want to use and the companies they trust to provide them…

…This kind of “innovation without permission” has changed the way we do business forever, fueling unprecedented collaboration, creativity and opportunity. And because America has been at the forefront of most of these changes, we have disproportionately benefited in terms of economic growth and job creation.”

This oddly puts Verizon in opposition to the other main anti-net neutrality supporters such as AT&T, Cox Communications, and Comcast.

Certainly companies like Google have come a very long way from 2005 when the debate was clearly one of who pays whom, for what kind of service, and who has the right to determine the quality of services over basic Internet infrastructure. In the early days content providers wanted a level of government regulation to ensure the Internet transmission and network providers did not have control over the end user experience, and objected to statements by AT&T’s then CEO Ed Whitacre who stated in an interview with Business Week:

“How do you think they’re going to get to customers? Through a broadband pipe. Cable companies have them. We have them. Now what they would like to do is use my pipes free, but I ain’t going to let them do that because we have spent this capital and we have to have a return on it. So there’s going to have to be some mechanism for these people who use these pipes to pay for the portion they’re using. Why should they be allowed to use my pipes?

The Internet can’t be free in that sense, because we and the cable companies have made an investment and for a Google or Yahoo or Vonage or anybody to expect to use these pipes [for] free is nuts!

This set off both a furor among Internet users, as well as a new movement to ensure the “telcos” did not ever again have an opportunity to restrict or limit free access to their networks. Arguments ranged from identifying the US taxpayer and AT&T customers paying for basic infrastructure our of Universal Services Fund/USF (meaning the US taxpayer is actually the owner of much of AT&T’s USF-funded infrastructure), to AT&T subscribers being forced to use AT&T preferred content providers based on their control of the network.

There are many organizations representing both users and Internet industry companies supporting the idea of network neutrality. The current law in the house is sponsored by Reps. Edward Markey (D-Mass.) and Anna Eshoo (D-Calif.) who introduced the Internet Freedom Preservation Act of 2009 (H.R. 3458) in July 2009.

The Con Argument

Not surprisingly, the Con argument is dominated by conservatives in both the government and corporate communities.

John McCain (who is also accused by the Huffington Post as having received more than $800,000 in campaign funding by AT&T, Verizon, and Comcast) rejected the FCC’s vote, and offered a new proposal called the “Internet Freedom Act.”

“Today I’m pleased to introduce the Internet Freedom Act of 2009 that will keep the Internet free from government control and regulation,” McCain said in a statement. “It will allow for continued innovation that will in turn create more high-paying jobs for the millions of Americans who are out of work or seeking new employment. Keeping businesses free from oppressive regulations is the best stimulus for the current economy.” (CNN)

The only thing missing in the above cliché-filled statement is a series of pictures of crying babies, unemployment lines, and California wildfires. The bottom line here is that politicians, bending to pressure or contributions from opposition parties, will use any words available to tug at either emotions or heart strings, regardless of the presence of factual data to support the position.

AT&T allegedly sent notice to all their employees, including union members and families, to write their representatives in favor of knocking down network neutrality. Perhaps that is a natural activity in the political process, however bandwagon appeals without supporting fact will not give the American people the broadband environment needed to compete in the global market placed.

Let’s look at both arguments in detail. Do a Google, Bing, Yahoo, or other web search on the topic of Net/Network Neutrality. You will find a lot of web references, news stories, blogs, and opinions on the topic. Much of it anarchistic noise, much of it very valuable information.

Hunter Newby, CEO of Allied Fiber, asks the question “What if the United States falls further behind Europe in deployment of broadband networks? What if we lose track of the need to wire each and every community? What if the United States falls so far behind Europe and the rest of the world due to politics preventing innovation that we can never economically recover?”

There are those who still believe the carriers, such as Verizon, AT&T, broadband wireless providers (such as Clearwire), and the cable companies should concentrate their efforts on delivering connectivity to each and every addressable community in the United States. Facility-based carriers (those who own the physical cable) should concentrate on providing bundles of “big, fat, dumb, communications pipes.”

Comments on Net Neutrality are Now Open with the FCC

The great thing about the US system is that no national law is ever a unilateral decision. We have a wonderful system of due-diligence through the congress and senate, with support from the executive and judicial branches of government.

The Federal Communications Commission under the leadership of Chairman Genachowski has opened discussion and the period of comment on the FCC guidelines. The period for comments is open until 14 Jan 2010. Some links for those interested in the topic include:

FCC Seeks Public Input on Draft Rules to Preserve the Free and Open Internet
NPRM: Word | Acrobat
News Release: Word | Acrobat
Genachowski Statement: Word | Acrobat
Copps Statement: Word | Acrobat
McDowell Statement: Word | Acrobat
Clyburn Statement: Word | Acrobat
Baker Statement: Word | Acrobat
Staff Presentation: Acrobat

The question arises, “what if we, as Americans interested in the future of the Internet, American innovation, the American economy, and our future generations actually took the time to read through the issue of Network Neutrality? What if we used our research as a basis for making our own decision on which side of the debate we fall, or if there is yet another strong argument to consider?”

It is a difficult topic, with a lot of noise and clouds shrouding the core issues. Weigh-in, let us know your opinion.

John Savageau, Long Beach

Verizon Gets it Right – “Bye Bye” Land Line Telephone

The FCC says US telephone companies have incurred a 26% increase in the cost of annual maintenance on traditional copper telephone lines over the past 5 years. Verizon makes 25% better margin on wireless phone than “land line” phones. FiOS is making it possible for Verizon to get into the high value video and cable television industry with a next-generation fiber optic infrastructure.

So why would anybody find Verizon CEO Ivan Seidenberg’s announcement at a Goldman Sachs investor conference that “his company is simply no longer concerned with telephones that are connected with wires” a surprise?

Bye Bye TelephoneWell, there are still many people on the street who believe copper “land Lines” offer better quality, security, and value. There are those who believe it is necessary to continue pumping money into technologies which are expensive to maintain, and offer little additional value to subscribers.

There are those who believe expansion of high performance wireless infrastructure such as LTE (long term evolution) and 4G (4th Generation Wireless) will not meet the needs of individual subscribers in both rural and urban areas.

Of course, they are wrong. Copper lines still fail, and are definitely location sensitive. A person with a heart condition will have a much better chance sending the alarm with a wireless device than a fixed line copper phone, so the more we dig into the copper argument the more it appears folks still are simply reluctant to embrace or endorse change. And change is needed in the United States.

We lag the industrialized world in broadband Internet deployments and availability. LTE/4G/FiOS all support and deliver broadband. Verizon is aggressively moving ahead on all broadband deployments. This includes broadband wireless to rural areas normally not available through either copper or in many cases cable television. In the United States (and most of the world) telephone users are either using low cost mobile phones, or using Internet phones (VoIP) on their home cable TV, or even in many cases wireless Internet connections.

So why is it surprising or concerning to anybody that Verizon is turning its back on their copper infrastructure, and focusing their capital and operational investments on a next-generation of technology? Is it better to spend more money maintaining old copper outside plant infrastructure, or is it better to spend that money reinforcing deployments of high performance wireless infrastructure and fiber optic FiOS technology?

Seidenberg added that “Video is going to be the core product in the fixed-line business.” Yes, thinking of a cable coming into your home as a “telephone line” is no longer an acceptable categorization. The telephone line is gone. Never to return. It is obsolete. We need to delete that from our mental SD chip, and reload with “Wired Humans Version 2.” Cables coming into the home and business are not for telephones, they are for the whole three dimensional concept of communications.

The answer for both the American consumer and for Verizon is clearly to reduce the operational expenses of supporting copper telephone lines, and start forcing the adoption of technologies that are better, cheaper, and offer much more service opportunity (such as high speed Internet access, video/cable TV, additional interactive communications services <such as video conferencing and video telephony>).

Americans need to applaud the courage of Mr. Seidenberg and Verizon to take this aggressive stand on new service and technology delivery.

John Savageau, Long Beach

Ushering Out the Fixed-Line Telephone

Hawaiian Telcom (HT) filed for bankruptcy protection in Dec 2008. While management problems and billing issues helped accelerate HT’s financial problems, the company also encountered a trend that is hitting the fixed-line telecom industry on a global scale.

A Berstein report in April 2009 showed residential access lines in the United States decreased at a rate of 11.6% throughout all of 2008, while decreasing at 11.5% in just the 1st quarter of 2009. As fixed-line revenue decreases, there is a parallel decrease in state and federal tax revenues being collected, further contributing to budget shortfalls in states such as California. We expect that tax revenue recovery for telecommunications will find its way into broadband Internet, higher taxes on mobile phones, and voice over Internet protocol (VoIP) phones.

The old argument of having the safety of a fixed-line telephone in the case of an emergency is also fading away as E911 emergency services using GPS (global positioning system) chips can identify a caller down to about a square meter. Persons needing to contact a first responder find it much easier to hit their mobile phone’s panic button than try to find their way to a traditional handset.

Without federal tax collection on telecommunications (known as the Universal Service Fund), programs such as rural fixed-line and broadband initiatives are likely to be discontinued or delayed.

In the United States, according to sources such as the Centers for Disease Control and US Telco Association nearly one in four US households no longer have a fixed-line telephone, preferring to use mobile phones or Internet broadband phones. As of June 2009, broadband Internet phones are not taxed in the US, enjoying a tax holiday that will most likely come to an end within the next couple years. States and the US government will not be able to continue absorbing the loss of telecom tax revenue as the fixed-line market dissolves.

However the short term effects will probably be most acute in small regional incumbent local exchange carriers (ILECs) and competitive local exchange carriers (CLECs) as their fixed-line business continues to implode. Hawaiian Telcom, as an example, lost an average of 1.6% of its fixed-line subscribers each quarter starting in the second quarter of 2006. Overall fixed-lines subscribers fell to 573,000 from 725,000 from 2Q2005 to 1Q2008. The loss of fixed-line revenue also restricted cash and investment potential in other emerging technologies such as broadband Internet and wireless technologies.

As mobile telephony and mobile Internet access continue to make advances in features and quality, there seems to be little hope the fixed line business will recover – unless the ILECs and CLECs are able to integrate advanced features such as television into their product line. Given the loss of revenues in fixed-line, and constricted capital markets, in most cases finding lenders or further investment for those projects seems unlikely.

The final factor in the fixed-line melt down is with employees and retirement funds. As ILECs are in many cases one of the largest employers in a region, failure to remain competitive is resulting in layoffs and early retirements. According to the Honolulu Advertiser Hawaiian Telcom had laid off several hundred employees by the time Chapter 11 was filed in December. More for the unemployment line, less tax revenue, and probably a dilution of customer service quality for HT services.

While the problems faced by Hawaiian Telcom may seem harsh, the largest telecom companies in the US are suffering a similar fate, as shown by Verizon aggressively selling most of its rural telecom holds during the first half of 2009. Cash acquired (and saved through reduction in rural support expenses) is being dumped into their fiber optic fixed line FIOS product, which brings “triple play” (television, Internet, telephone) services to residential and business subscribers.

Through an informal poll of about 50 new apartment residents in Long Beach (in the author’s building), only four reported ordering fixed-line telephone services. All others stated they only used their mobile phones, or in a couple of cases utilities such as Vonage or other Internet broadband phone.

In Hawaii the winner appears to be Time Warner Cable (under the local operating name Oceanic). Oceanic continues their penetration of the residential market, with more than 400,000 business and residential subscribers within the state. Oceanic does offer triple-play services, and has the backing of Time Warner Cable, one of the largest cable operators in the United States.

With advanced wireless telephony and data features commonplace in US and international markets, and high performance broadband expansion into both business and homes, the fixed-line telephone operators may soon be a technology of the past. Now we need to deal with the backlash – unemployment and loss of tax revenue.

John Savageau, Long Beach

%d bloggers like this: