October 13, 2009 3 Comments
An employee enters the meet-me-room at a major carrier hotel in Los Angeles, New York, or Miami. He is a young guy recently graduated from high school, hired to do cable removal for circuit disconnects at minimum wage. Although young, he has a wife and child, and has recently been fighting with in-laws over his ability to support a family. Frustration and anger overcome his emotions, and he turns to the ladder rack jammed with cable and starts hammering at the cables for all he is worth.
Network operations centers around the world see circuits dropping, and customers with critical financial, military, Internet, and broadcast news services are shut down. In the space of about one minute our young employee has taken down several thousand individual circuits, creating near chaos in the global telecommunications community.
In their report on Trusted Access to Communications Infrastructure, the NSTAC Vulnerabilities Task Force advises “”it is important to recognize that any one individual with malicious intent accessing any critical telecommunications facility could represent a threat. The threat of insiders performing malicious acts also transcends each type of site discussed in this document.”
The event noted in part 2 of this series describing the outages in Northern California following damage to a manhole housing telecom was real. The resulting disruption to regional communications was a wakeup call to the telecom community, law enforcement, and communities affected. It is clear the perpetrator knew what he was doing, and knew exactly what vulnerabilities the major telecom companies had which he could exploit.
There have been many other cases such as Level 3 Communications loss of a major core router in 2006 supporting regional Internet services in London due to theft, a break-in at BT’s switching facility in Birmingham during the same period resulting in the loss of thousands of telephone lines, showing this is not just an American problem, but a global vulnerability.
The message is clear, as an industry our most obvious threat to information and communications security is not a natural disaster, it is people with industry knowledge or access to our critical facilities.
The Telecom and Data Center/Carrier Hotel Industry’s Role in Managing Human Security Risks
Data centers and central offices are in a constant state of change, maintenance, and growth. While facility network operations staff are generally long term employees, with a history of employment and performance, many others entering our data centers are not well known to the landlord.
Janitorial and maintenance staff are normally contracted to vendors, mechanical and electrical workers are contracted to maintenance and engineering companies, and construction contractors often use temporary staff from agencies such as “Labor Ready” and other day labor companies. In most cases data center or landlord employees are given a cursory background check prior to employment, however others entering even critical areas within the data center or central office meet-me-room may be entirely unknown to the facility.
While normally under some level of supervision, or access management, contractors, maintenance people, and even data center tenants are often free to move around the facility without direct security observation. As shown above, it would only take an angry, disgruntled, or undisciplined person seconds to cause a major calamity in our global communications system.
In a worst case, that person may be a terrorist with a detailed plan to cause damage to the facility once given even minimal access. High voltage electricity, water systems from cooling infrastructure, or access to switching equipment and cable interconnections are all exposed within the data center, and any element could be used to cause a major disruption within the meet-me-room or data center.
Most carrier hotels are located in “mixed-use” buildings, in high-rises with additional tenants who may not even be in the data center or telecom industry. This compounds the problem, as those tenants are often reluctant to comply with security and access requirements at the level of a critical telecom facility.
The issue becomes even more acute when we realize that much of the infrastructure supporting carrier hotels transits “risers” between floors, often through floors occupied by non-telecom tenants who may have physical access to riser space within their offices.
There are a few data centers within the United States where security is comprehensive enough to reduce the risk of malicious intent to a very low level. While many tenants find the access and supervision within the facility extreme, facility resources are protected from all but the most aggressive vandalism or attack.
The NSTAC recommends that in the US the telecom industry establish best-practices guidelines to screen personnel prior to unescorted or unrestricted access to critical facilities, such as carrier hotels and carrier central offices. This may include a national agency check to ensure the person requesting access does not already have a profile indicating they could potentially be a threat to the facility.
The US government may give this additional support, as much of the US government, state, and local communications services are supported either in carrier central offices or commercial carrier hotels.
Recommendations for the Communications Industry
While it is clear not all persons entering a data center or carrier hotel facility can be completely screened, there are tasks each carrier and commercial data center operations should complete. Those could include:
- Complete background checks for all direct employees
- Pre-employment screening which would include a personality profile (indicating if they are in a high risk category for emotional stress)
- Supervision of all contractors on site by a direct company employee who is aware of the risk posed for each type of equipment in proximity to the contractor (such as electrical equipment <UPS, breaker panels, switchgear, chilled water pipes, etc>)
- Training in situational awareness – being able to identify activities not normal for others in your facility
- Cooperation with law enforcement and other agencies
- Working with industry groups to create and follow an industry “best practices” for facility security and human resource management
- Ensure at least in the streets and areas immediately adjacent to the facility all manhole covers and utility entry points are locked and secured, preventing persons from accessing telecom, electrical, and water infrastructure supporting the building
“Unfortunately our most likely enemies will throw explosives into unguarded cable interconnect rooms or drop cans of petrol into unlocked manholes. End of Cyber War. You might characterize this as the provenance of a 23 year old fundamentalist Skywalker with a cell phone modem and a wild-eyed cousin in Munich figuring out how to blow up the Internet Death Star and stop Predator attacks on his village. Totally asymmetric dude! (From Bob Fonow’s “The Death Star?: Cyber Security vs. Internet security”)”
The commercial operators of data centers and carrier hotels have a tremendous responsibility not only to their owners and shareholders, but also the global telecom community and global economic community. The potential impact, even in the short term to a malicious attack on a meet-me-room at One Wilshire, 60 Hudson, the Westin Building, Telehouse in London, or the NAP of the Americas would be immediate, and extremely disruptive.
Human factors are the threat. Let’s not forget the lessons learned over the past couple years, and keep diligent, have good human situational awareness, and understand the sense of urgency we must apply to ensuring our communications infrastructure is secure.
Let us know your opinions, experiences, and recommendations
John Savageau, Long Beach
Previous articles in this series