A Watertight Design

Engineering firm enters security market, guards drinking water

• By Jason Goodman
• May 01, 2005

There was a time when Wunderlich-Malec, a Minnesota-based engineering, process control, and system integration firm, focused primarily on making sure production lines ran as smoothly and efficiently as possible. Cooking potato chips to the perfect crispness and mixing salsa to the right spice level were tasks more suited to the company's bailiwick than making sure municipal water facilities stayed secure. Now, two years after its quiet, and almost accidental, entry into the surveillance and security industry, 35 percent of the company's income comes from the design and installation of wireless Ethernet, IP-based security networks.

"We got into the security surveillance business because one of our customers basically dragged us into it," said Roy Marin, operations manager at Wunderlich-Malec's Carrollton, Texas, office. "It's a niche that we kind of fell into."

Riding Herd at a Texas Facility
When officials at the city of Cleburne, Texas, decided in 2003 to implement tighter security measures for their public water utility facilities, in accordance with a U.S. Environmental Protection Agency mandate following Sep. 11, 2001, they chose to call the process control engineers at Wunderlich-Malec, with whom they'd worked on projects before.

"Some municipalities reacted to the mandate in different ways," Marin said. "Some just went out and hired more guards, and others thought, 'Well, let's just put cameras and gate access control and motion detectors at their sites."

The city of Cleburne charged the company with creating an efficient IP-based network capable of securing sites not previously regarded as security risks, and hence, not designed with security in mind, by transmitting data, access control, voice, and video and detection information over a wide area.

Realizing that the solution to the problem lay in Ethernet connectivity, Wunderlich-Malec's design team concocted a wireless Ethernet, IP-based network that could connect facilities up to 22 miles apart. The team promised a network that could be monitored from any of its nodes, incrementally and exponentially upgraded to keep pace with advancing technology, and offered, at worst, a "controlled degradation" of the decentralized network in the event that any of the monitoring sites were to break down for any reason.

While the idea and design seemed plausible, and even simple, the hardware to make it a reality was nowhere to be found at the time, and it would not emerge onto the surveillance market for another year.

"We went around looking for IP-based equipment that would do the job we needed it to do, and we found a limited number of vendors," Marin said. "We went to the ISC show in 2003 and stopped by booths and asked people if they had equipment that would work on IP networks, and they basically snickered at us and told us to go away. Then, last year, everybody was introducing their new IP-based hardware and everybody had an IP-based port -- that's how we got here."

Once the equipment was located and integrated into the design, it was only a few months before the company's fiction became fact.

When the team finished with the network, they presented the city of Cleburne with a 12-site wireless radio network that could deliver gate control, video, audio and voice data from all of the remote sites back to a central site.

Only one location needs to be manned 24 hours a day and two of the locations are manned from 8 a.m. to 5 p.m. During the day, all the cameras and hardware associated with a manned site are displayed at the appropriate site. Then, after 5 p.m., the system switches over and the single-manned site becomes the central monitoring location. Cleburne's city hall also has a direct port into the system that allows monitoring 24 hours a day, and allows for maintenance of the access control enrollment system.

An Overview of the Network
The network uses SCADA radios -- radios that make use of specialized Supervisory Control and Data Acquisition software -- to receive, send, and regulate the flow of information on the network. Each radio acts as a router that allows signals from any connected point on the network to be relayed to any other network-connected point. The result is that the network neither has nor requires a central monitoring station. The SCADA radios function much like relay runners passing a baton around a track. Signals can be bounced from point to point without any loss of data quality.

Using SCADA radios, since the application is software-based, allows Wunderlich-Malec, to use equipment from any number of manufacturers, providing clients with the flexibility to spend as little or as much as they choose on equipment.

Any point on the network can be assigned central monitoring duties. This aspect of the design allows utilities officials and managers the flexibility to post a single guard at any of the different locations along the network without sacrificing any monitoring capability. The ability to change the central monitoring station on a regular basis adds another layer of security to the system by making it harder for potential attackers to determine which of the locations will be manned at any given time.

In addition to the flexibility the system affords, the amount and types of information that can be sent along the network are limited only by the available bandwidth. One of the most important facets of the network is its ability to improve the decision-making process, something the Wunderlich-Malec engineers know well, in case an event were to occur.

"With a single alarm, there's not enough information to make an educated decision on what to do; you basically have to flip a coin and take a chance," Marin said. "We tell our customers that we're providing them a highway to get them enough information from the site so that we can get it to the right people to make the right decisions. For example, if a fire alarm goes off in a pump house, you can pull the camera up and if there's smoke, there's no question ? it's time to send out the cavalry.

"If the alarm goes off you've got a yes or no choice: 'Do I send? Do I not?' But you've only got one alarm light telling you what's going on. You can't establish a trend with only one point of reference."

Overcoming Challenges in Network Installation
Getting the network up and running proved to be more complicated and challenging than the engineers anticipated. Problems, both planned for and unexpected, arose from unlikely places, and even slight miscalculations pushed back the completion of the installation.

The engineers expected to perform topology studies at each location to determine if a viable line of sight was available to connect two points. Once the studies determined that there were no topological obstructions, a tree canopy had to be estimated and calculated for, as well as the Fresnel zone ? the area around a radio signal that distorts and bends the information flow, confusing receivers and eating precious bandwidth.

Distance also became a factor when determining how to route and connect signals. The curvature of the earth starts to interfere with line-of-sight signal transmission at distances greater than 23 miles. Transmission distances greater than 23 miles require the construction of significant relay towers, so the engineers had to design the web-like network structure to compensate for factors beyond anyone's control while still keeping within budget.

What the engineers didn't expect was a problem like the interference from a Wi-Fi hotspot that was installed in a mobile home park much too close to one of the signal paths.

Also problematic was the SCADA software that regulated the information flow across the network. The specifications for the project dictated that the cameras should be able to display at 4 feet per second (fps). The designers included enough bandwidth in the network's design that they believed there would be a 50-percent buffer. When they turned the network on, they expected to be receiving between 7 and 8 fps, but they were struggling to keep up with the necessary 4 fps. Data collisions across the network were eating huge amounts of bandwidth as all the connected equipment started trying to send information at the same time.

To fix the problem, the engineers modified the software so that the monitoring station was requesting information from the network instead of trying to catch everything thrown at it. Once the station received the necessary 4 fps from a camera, no more requests were sent to that particular device. The result showed a dramatic improvement in the frame rate display. The engineers were now seeing the 7 to 8 fps they anticipated.

A Success Story in San Antonio
Shortly after the success of the Cleburne experiment, the Wunderlich-Malec design team found themselves working on another similar network for the San Antonio Water Association, only this one was substantially larger -- almost 40 radio installations instead of just 12 -- with network nodes spread out all over the city. The SAWS project also necessitated the building of a 300-foot radio tower to overcome some topology issues.

The approach to designing the SAWS network was a little different than the one at Cleburne. Because the SAWS site was so much larger, the designers determined the best design would be to place three focal points around the city to establish clean paths for all of the outlying network nodes. The network still offers the flexibility and decentralized decision making that the Cleburne network offers, but the larger network needed a more structured organization to ensure everything ran smoothly.

Future Applications
Since the SAWS project, the company has been inundated with requests from municipalities, race tracks, prisons, and myriad other organizations all interested in commissioning networks of their own. Marin feels that the company's entry into the market made perfect sense. To the engineers involved in the project, problem solving is the same whether its filling soda cans to just the right amount or trying to figure out how best to protect drinking water supplies -- the satisfaction is in solving the problem.

"Nature wants everything to rest -- that's why there's gravity; that's why there's friction," Marin said. "And in a world where everything wants to come to rest, we're trying to make things move.

This article originally appeared in the 05/01/2005 issue of Environmental Protection.