Better Benchmarking

How to derive more value and insight in a survey-weary world

• By Richard MacLean
• Jul 01, 2004

Benchmarking is an important tool for evaluating one's practices relative to best-in-class. If done properly, it can even inspire innovation. The majority of these studies, however, are designed and executed poorly and yield dubious or even counterproductive conclusions. Environmental, health and safety (EHS) studies, in particular, are prone to these problems. This month we examine the design characteristics that you should consider first before agreeing to participate in a survey or placing too much credence in the conclusions of a benchmark study.

Companies are suffering from survey fatigue. In particular, investment research firms and non-government organizations (NGOs) are hard at it -- surveying companies in an attempt to rank EHS and social responsibility performance. Each has a particular focus and each has its own form. It's become so time consuming that some companies are banding together through SRI World Group to create OneReport^™ -- to produce a one-stop, one-form, here-it-is, take-it-or-leave-it approach.¹ Some of the heavyweights, such as Shell and DuPont, have backed this approach to consolidate survey requests.

Surveying is not the same as benchmarking, although surveying is a key component of benchmark studies. The two often get confused. A benchmark study is a systematic search for processes that yield superior performance. These "benchmarks" then are compared against current activities to gain insight on how to improve. Surveys compile and analyze data. "Just the facts, ma'am," as the no-nonsense Sergeant Joe Friday would say on the TV series Dragnet.

Problems arise when EHS managers make unsupported judgments from survey information. For example, if their company is rated "#1 in compliance," EHS managers and their business executives may conclude that the company has optimal governance systems. In reality, the company may be at the bottom of the heap but is hitting the daily jackpot with dumb luck. Or the company may be utilizing a ridiculous amount of resources and be operating pathetically inefficiently. Or, finally, the ranking indeed may be an accurate indicator and the company is worthy of a best-in-class benchmark examination.

Surveys are a great technique to determine the initial candidate slate for benchmark studies. You need to start somewhere, and a "just-the-facts" approach is a good beginning. But, again, the initial screening numbers may be misleading. It takes an in-depth examination to determine what is really going on. An example of excellent benchmark design was the study led by Jim Collins and described in the book Good to Great.² He employed a team of inquisitive business students (affectionately called "chimps" after Curious George) to screen 1,435 companies in order to find only 11 that deserved detailed evaluation.

Collins was fortunate in that the financial indicators he used to screen companies are readily available and fairly reliable. Not so with EHS and social responsibility indicators. The Center for Environmental Innovation in its "Organization in Transition" study developed a screening matrix to select EHS organizations for benchmarking. (Full disclosure: I'm the project manager. ³) The "top 25 EHS" organizations were screened from manufacturing companies listed in the Standard & Poor's 500 Index.

A few EHS managers (i.e., the ones in companies not in the top 25) objected to this "top 25 list" because it was "an arbitrary and superficial evaluation." Precisely. It was never meant to be a benchmark study, only an initial screening tool with admittedly imperfect but readily available data. The focused interviews are now underway. Just as Collins may have eliminated some truly great companies with his screen, you need an initial candidate slated created by some objective criteria.

Aside from the problem already mentioned, namely that a screening survey can be misinterpreted, the next stage -- the actual benchmark phase -- can result in even more difficulties. Here are some tips to watch out for and some tools to use to judge the value of a survey or benchmark study.

Recognize the Limitations of Surveys
Surveys rarely reveal the richness of the information that may be available. They can give insight into where to look, but they do not probe and question like face-to-face meetings. Surveys are particularly bad at extracting sensitive information and identifying problem areas. Indeed, some of the best information from a benchmark study can be the lessons learned from failures. People are reluctant to put stuff in writing, especially if a lawyer will be involved in reviewing the final document.

Face-to-face interviews or, as a second-best approach, telephone interviews are much better at exploring the nuances that can yield the most productive information. It's a little like a Catch-22 problem: If the individual(s) preparing the survey knew precisely all the potential avenues to explore and questions to ask, the survey might not even be necessary in the first place.

Account for the "PC Factor"
EHS and social responsibility surveys are a bit like sex surveys. What "normal individual" would want to admit that he or she is less than perfect, overly promiscuous, or just plain deviant? I suppose if you are a media magnet, the wackier the better. Either way, there is a built-in bias. Since reliable and verified EHS and social responsibility metrics are the exception rather the norm, surveys and benchmark studies are hard pressed to extract accurate judgments on performance.

From my own internal benchmarking studies, I know that asking the employees and managers within a company where they are and where they want to be on EHS can elicit all sorts of responses. And this is the typical result when employees are answering honestly and the information is for internal consumption only. You can imagine the response to a survey that they know will be published asking information such as, "Rank your company's environmental strategy using the following scale . . . ." I shake my head in disbelief over these surveys that cost tens of thousands of dollars and are taken seriously by some folks. Naiveté on steroids.

Examine the Precise Wording of the Questions
In the world of business, there are precise definitions for certain financial terms. Not so in EHS. Even a simple question such as, "How many full-time equivalent employees (FTEs) are working on EHS?" could elicit a wide range of responses. How do you account for outsourcing, temps, contract employees, embedded staff, and production employees running the process pollution control equipment? Depending on your interpretation, the answers could range from zero to 50 at the same facility.

Similarly, there are few EHS terms that inherently convey some dimension of substance (i.e. degree of sophistication or quality). Thus an "environmental policy" or a "green purchasing plan" can range from a total farce to a state-of-the-art, well-thought-out process. Yet, survey questions are often binary: "Do you have X? (check box)." Because many surveys are carried out by inexperienced individuals, the questions often are simplistic choices that lend themselves to easy statistical analysis. I get the biggest kick out of academic surveys that are heavy on powerful statistical regression analyses but use ill-conceived or naive information-gathering techniques. "Garbage in = garbage out" is the operative statistical formula.

There is a science to structuring surveys. How the question is phrased can be a major determinant on the answer. This is especially true in written surveys where there is no one to follow up with clarifying questions. There can be built in biases that can affect the response: the "When did you stop polluting?" question.

Consider the Motivation of the Sponsors and Participants
Academic and government surveys or benchmarking studies that will become public information may or may not be responded to with the same degree of care, honesty, and attention as private benchmark studies that were initiated by one or more organizations responding to a specific need. Similarly, consulting companies sometimes conduct and publish surveys for marketing purposes. Investment researchers are extracting information for their stakeholders.

Motivation matters. The underlying question should be, "What's in it for my organization?" If you are begrudgingly facing another survey with little direct benefit to your organization, just what type of care and attention do you think other organizations will extend when they are responding? Surveys can provide value if they are designed well, executed properly, and focus on relevant business concerns. Few surveys have this degree of rigor, so let the form filler-outer beware.

Determine Who Will Be Responding and Who Will Be Asking the Questions
If it is obvious that the equivalent of Asok, the naïve and lowest-level employee in the cube world of the cartoon strip Dilbert, is filling out the form or is doing the surveying, don't put too much credence in the study. Surveys often undertake grand and strategic objectives and the results are touted as such (when all the while we know that it was only Asok who filled out the form). Give me a break!

The classic red flag is raised when surveys are shotgunned out to thousands of companies in the hopes that dozens or hundreds may respond. This approach may have some validity if the surveys are sent to a highly select group who personally respond (e.g., CEOs, VPs and senior managers of EHS, etc.). But for the most part, one is left wondering what motivated these individuals to respond and if this small group of volunteers is a true representation of the total population.

The best benchmark studies are those that use initial screening techniques and are then done face-to-face with individuals at the highest and lowest levels within organizations. The VP of EHS can best describe resource allocations and strategy, but Asok may be the best source of what is really going on and the most willing to convey this without all the politically correct spin. Sometimes you need to see the interviewees' faces to see them twitch or smile when the key question is asked. And, finally, you need to deliver the follow-up question and then the follow-up to the follow-up question.

In today's tight economy, few companies are willing to spend the resources to benchmark properly, but they may be missing the greatest opportunity to respond to precisely these cost pressures. Trust me, there are some Potemkin villages out there who get a lot of attention and accolades. But, there are also some incredibly efficient and value-producing EHS organizations that quietly go about their work. Finding out which have the "right stuff" and emulating and building on their performance is critical for success in today's business environment.

References
1. SRI World -- (www.one-report.com)
2. Jim Collins, Good to Great, HarperCollins Publishers, 2001.
3. Y. Yung and R. MacLean, "A Template for Assessing Corporate Performance: Benchmarking EHS Organizations," Environmental Quality Management, Spring 2004, Pages 11-23. Available at
www.enviro-innovate.org/OIT/OIT_additional_info.htm.

This article originally appeared in the 07/01/2004 issue of Environmental Protection.