The Show Must Go On

Keeping water and wastewater services running during a crisis depends on securing facilities at multiple levels

• By Bob Radvanovsky, CISM, CIFI
• May 01, 2006

Critical infrastructures are industrial sectors based on areas of utilization and specialty that are critically vital to the continued operations and maintainability of our nation's way of life. Several sectors are more important than others, either based upon financial or human risk factors. The "water sector," for example, for which the U.S. Environmental Protection Agency (EPA) is responsible, is vitally important to our nation. We use water everyday, for cooking, direct consumption, industrial manufacturing, processing, cooling, and more.

With that knowledge comes the introduction of a new initiative toward safeguarding our nation's critical infrastructures, called Critical Infrastructure Protection (CIP). Before Sep. 11, 2001, former President Ronald Reagan realized the importance of our nation's resources, dubbing the more critical resources "infrastructure." Many think of the word infrastructure and immediately associate the word with public works departments. Although this isn't entirely false, this thinking limited the scope of the definition of what was considered infrastructure. During former President Bill Clinton's terms, the term was refined even further. Several initiatives were introduced, the most significant of which spawned the term "critical infrastructure." Presidential Decision Directive No. 63 (PDD-63) ¹ defined, at a strategic level, an outline for several definitions of what is considered "critical infrastructure." This document set a course for establishing new organizations to maintain and safeguard our nation's resources.

After Sep. 11, 2001, President George W. Bush refined the term to its present definition and introduced several more initiatives, the first one defining and establishing the U.S. Department of Homeland Security, along with several other subsequent documents that defined key, critical roles for the federal government, several critical infrastructure sectors, and more. Homeland Security Presidential Directive No. 7 (HSPD-7)² further elaborated safeguarding our nation's infrastructure, introduced several key legislative acts, and defined roles and responsibilities for each critical infrastructure sector. EPA was deemed as the key-defining organization that was (and still is) responsible for directing and coordinating security and regulatory activities within the water sector, which includes both water and wastewater treatment and processing, between both public- and private-sector organizations.

Some Perspective
There are more than 160,000 public and private water systems, serving over 75 percent of the U.S. population. These systems provide millions of gallons of water for just about everything and are of great importance to our nation. CIP provides a "roadmap" for reducing threats through risk management (risk assessments, analysis, mitigation, and remediation efforts), disaster recovery planning (what to do in case of a disaster or emergency), business continuity planning (how to keep going, even if partially operable, during or after a disaster or emergency), vulnerability assessments (includes penetration testing and analysis), policy management, documentation management, emergency management, and operational planning (both strategic and tactical). CIP identifies "what-if" scenarios by attempting to answer any critical questions before those situations actually occur. Sep. 11 brought to light many concerns regarding the integrity and security of a few, very specific, key infrastructure components, some of which are interdependent on one another, including the water sector. One sector's dependence upon another sector, can often lead to "cascading effect processing," in which one process or system has an impact (direct or indirect) upon another process or system, not necessarily in direct sequence between each dependent system or process. For example, if System A were to provide output to System B, System B could not operate properly unless System A was first operating. Additionally, System C might rely upon System B, which would then rely upon System A, and so on and so forth. This method of thinking is often referred to as the "domino effect."

Early Efforts
Shortly after Sep. 11, EPA and its partners from the drinking water and wastewater industries launched several massive initiatives to develop training and improve procedural guidelines specifically for water security. Ironically, many risk management methods can be implemented within and between each sector. These methods often overlap or coexist, but each sector's operational requirements are often unique to that sector.

For example, some issues that might be specific to the water sector may pertain to health issues, or might include other aspects, such as the chemicals used in water treatment. For some, these chemicals can be potentially toxic to living organisms (e.g., chlorine dioxide gas or sodium hypochlorite, either one used for disinfection in water treatment process). These chemicals may be used for other purposes, such as manufacturing chemical reactionary processes (not associated with water treatment); however, these chemicals may also be used for more nefarious purposes, such as building explosives or exposing large numbers of people to a hazardous substance or material.

Other issues to consider include operational procedures specific to production flow, pressure, and distribution methods, as well as access control to key locations, such as pumping stations or distribution points. The water sector, more than any other sector, has a much more significant importance to the population because it must always be available -- for whatever need. When Hurricanes Katrina and Rita devastated the Gulf Coast in 2005, they posed special challenges to the water sector. Not only was water unavailable, but it was contaminated with toxins released within effluents and breakwaters from the Mississippi River and the Gulf of Mexico. ^3,4 The lack of resiliency in the local infrastructure raised questions about why key critical infrastructures were unavailable following recovery operations within the affected areas. ^5,6 Those issues that were discussed were primarily related to:

• Insufficient infrastructure resiliency;
• Communication and human coordination issues; and
• Lack of properly trained and educated incident response personnel.

The last two issues may be related to each other because proper and effective human coordination and communication are not possible unless managing personnel are properly trained and up-to-date on current events and incident-management tactics. ^7,8 In the cases of Hurricanes Katrina and Rita, according to several news media sources, none of these factors existed at appropriate levels to allow a smoother recovery process. ⁹

Sometime in 2003, the National Drinking Water Advisory Council (NDWAC) chartered the Water Security Working Group (WSWG) to investigate and develop findings on best security practices and incentive programs to define, establish, and implement broad adoption of best security practices specifically tailored for the water sector, while providing measures to gauge the extent of the implementation. ¹⁰ The WSWG currently works cooperatively with EPA, the Department of Homeland Security (DHS), the U.S. Department of Defense (DOD), and the U.S. Centers for Disease Control and Prevention (CDC) to support the interests of those agencies and organizations and provide a level of expertise related to water security.

The WSWG has several times each year following its charter in 2003, and it has defined 14 features that outline their concept of an effective security program specific to water security and safety. Many of these features are reflective of similar principles from other sectors' best security practices, and they may very well be the basis for standardized security principles between each sector. CIP emphasizes four similar elements that compose an effective CIP security program: ¹¹

1. Prevention through access, intrusion detection, and control; contaminant detection and monitoring; physical hardening of systems; inherently safer design, construction, and materials used; and controlled access to sensitive information. ¹²

2. Preparedness through having plans and procedures in place and building the successful partnerships and communication channels necessary for preventing and responding to any given emergency, crisis, or situation. ¹³

3. Quick and safe response to any given emergency, crisis, or situation. ¹⁴

4. Quick and effective of recovery without compromising life or property. ¹⁵

The combination of these elements and features are what make an effective CIP security program. Any element that may be weaker than the others will eventually produce les-than-effective results.

Many water distribution and processing facilities appear to have nominal physical security. Security measures generally include: perimeter fencing, access control at all gates (card swipe, possibly with a PIN), video surveillance, periodic human guard checks, etc. But some facilities are slightly more sophisticated than others, using such technologies as proximity sensors throughout the area near the water facility, and triggering video cameras to target the entrance or base perimeter. These measures can address physical threats, but what remains unanswered is whether the other two threats -- human and cyber -- are being adequately addressed.

What If?
Perhaps describing some potential, yet very real, threats to a given water facility may help in understand how ciritical to sector is. Note: These scenarios are purely hypothetical and do not necessarily constitute any known or existing weakness or lack of any security at any level, and they are not representative of any single organization or facility. Additionally, no specific or detailed method of denial of service or levels of subterfuge will be provided -- only the situation will be outlined.

Scenario 1: Direct attack via cyber-security threat. The cyber-security threat is related to control systems (computer workstations or servers) that control process flow, perform flow monitoring, notify operators of any flow-control variations, turn valves, enable switches, trigger alarms, etc. Essentially, these systems connect to sensing devices via a serial or network interface and are connected to the control system at the operations room (or something similar), in which operating personnel can closely monitor water processing. Many of these control systems predate the Internet (circa 1987) and are running older operating systems. Many are not protected by a firewall. And if a firewall is installed in front of, or around the controlling devices, there is significant risk of disruption. This poses a serious issue for many control systems (SCADA^16,17) for all sectors throughout the United States. Some control systems may be directly connected to the Internet or indirectly connected through local networks that might not be properly protected, potentially allowing a mildly persistent intruder to pass through the network perimeter and into the environment to pursue disruptive goals.

Alternatively, some facilities connect their control systems to a managing network via some form of wireless connection. This type of connection can be easily determined with a portable laptop or a personal digital assistant (PDA) and a wireless sniffer¹⁸ such as "Airsnort" or "Wellenreiter." Even well-configured wireless networks are not completely secure. What are the chances that the network is not operating via an encrypted communications channel? Based on numerous surveys, this threat is very significant. Or if an encrypted network communications channel is being used, odds are good that a poorly used encryption password or encryption key is too short (and much easier to crack¹⁹). Even if wireless networks are encrypted using a strong encryption method (128-bit or larger), it was determined several years ago that most wireless encryption methods in use today can be compromised in a relatively short period by a mildly aggressive intruder.

Scenario 2: Indirect attack via cyber-security threat. Suppose the controlling devices are located in the organization's firewall-protected internal network. Further suppose that this firewall isn't kept up-to-date, and the support contract is allowed to expire because the organization did not experience any attacks and felt they were safe and secure, only to have multiple attacks against their firewalls several years later, resulting in several outages and numerous and expensive upgrades. Now their control devices are connected to the same internal network that the remote facility is connected to. What do you suppose would happen if an attacker were to penetrate their weakened firewall?

Scenario 3: Denial of service due to "human error." All too often, there is news of someone who accidentally released too much of something and caused a reaction that results in a messy, expensive, and timely cleanup process. Does the facility have an incident response plan? If so, is it up-to-date, and does that facility regularly coordinate routine drills and activities with local law enforcement, fire prevention, and emergency management services? If so, how often are these drills conducted? Do the drills address cyber threats in addition to human and physical threats?

All three scenarios pose serious questions about how a single operations facility could experience potentially devastating consequences, or, even worse, cause cascading disruptions or outages to other facilities and/or sectors. For instance, if a water-providing facility were to experience an operations failure, it could prevent fire prevention organizations from putting out fires in a timely fashion. How about healthcare organizations, in which hospitals and critical care facilities would then have to rely on bottled water or water hauled in via tanker truck (such was the case with New Orleans)?

Conclusion
The water sector has much work to do to better protect and safeguard itself. Protecting a critical infrastructure component does not depend upon any unique intelligence collection method, nor does it require any unique integrated intelligence function. What is more important is the performance of specific analytical methodologies that may be used to regularly and routinely assess threats and vulnerabilities, perform risk analyses relative to the infrastructure being protected, and risk remediation, or the mitigation of any identified risks. These tasks require thinking in terms of self-protection at the organizational level. ²⁰

Defining, using, and maintaining critical infrastructure components is a methodology that includes an analytical model or template that guides the systematic protection of itself. More importantly, it is a reliable, sequential set of decision-making steps that assists decision makers in determining which methods should be used to safeguard the infrastructure component. The process ensures the protection of only those components upon which survivability and continuity of operations are dependent. Thus, these methodologies safeguard only those infrastructure components deemed important and vital for the continued operation of large scales of economy.

Reference

1. www.fas.org/irp/offdocs/pdd-63.htm
2. www.fas.org/
3. Updated: "EPA and CDC Report High Levels of Bacterial Contamination in Preliminary Floodwater Samples from New Orleans," http://yosemite.epa.gov/, September 7, 2005.
4. www.epa.gov
5. Risks, Resiliency, and Restoration: The Three Rs of SustainableDevelopment in a Coastal Barrier Resources System, Robert R. Twilley, Professor, Department of Oceanography and Coastal Science, Louisiana State University, Testimony Before the Committee on Resources, Subcommittee on Fisheries and Oceans, U.S. House of Representatives, Hearing on H.R. 3552, the Coastal Barrier Resources Reauthorization Act, Nov. 8, 2005, http://resourcescommittee.house.gov/.
6. U.S. Government Accountability Office, Testimony Before the Subcommittee on Oversight and Investigations, Committee on Energy and Commerce, House of Representatives, "Hurricane Katrina: Providing Oversight of the Nation's Preparedness, Response, and Recovery Activities," Statement of Norman J. Rabkin, Managing Director, Homeland Security and Justice Issues, GAO-05-1053T, September 28, 2005.
7. http://commerce.senate.gov/
8. U.S. Government Accountability Office, Testimony Before the Senate Committee on Small Business and Entrepreneurship, "Hurricanes Katrina and Rita: Preliminary Observations on Contracting for Response and Recovery Efforts," Statement of David E. Cooper, Director Acquisition and Sourcing Management, GAO-06-246T, November 8, 2005.
9. National Drinking Water Advisory Council, Water Security Working Group Findings, Presented to the National Drinking Water Advisory Council on May 18, 2005.
10. Ibid.
11. Ibid.
12. Ibid.
13. Ibid.
14. Ibid.
15. The acronym "SCADA" stands for "supervisory control and data acquisition," and is representative of a computer system that gathers, collects and analyzes data from sensing devices in real time (meaning not delayed). SCADA systems are the controlling devices over the sensing sevices, and can prompt operating personnel or appropriate course of action. These devices are typically used for large-scaled environments that may be geographically dispersed in an enterprise-wide distribution operation.
16. The acronym "DCS" stands for "distributed controlling systems", and are typically are used within a single processes or generating plant, or utilized over a smaller geographic area, or even a single site location. A utility company may use a DCS to process water treatment for that facility, but would utilize a SCADA system to distribute it throughout the water system.
17. The term "sniffer" is synonymously referred to also as a "packet sniffer" or "network sniffer," which is: (1) a device that is dedicated for the purpose of monitoring (and logging) network traffic in an effort of recognizing (and perhaps decoding, or even decrypting) networking packets of interest; or (2) a software application utilized for general purposes to recognize (or perhaps decode) networking packets of interest - this type of software is often used by systems and networking administrators for legitimate purposes of network management and diagnostics purposes -- consequently, similar software is utilized by individuals of nefarious intent to conduct illegal or unethical (or both) methods of subterfuge entry into areas that would otherwise be restricted under normal conditions.
18. The term "crack" is alternatively referred to "breaking" as in "breaking code". The entire objective of cracking a computer system, its code, or any applications contained within it, is to penetrate that system without proper identification and subterfuge entry into areas that would otherwise be restricted.
19. 20. www.usfa.fema.gov/ www.usfa.fema.gov/

This article originally appeared in the May/June 2006 issue of Water & Wastewater Products, Vol. 6, No. 3.

This article originally appeared in the 05/01/2006 issue of Environmental Protection.