Progress toward implementation

• May 01, 2000

For nearly a decade, the U.S. Environmental Protection Agency (EPA) has been exploring the potential benefits of a new paradigm for the selection of analytical methods that has its roots in modern management philosophy. The new approach, variously called the performance-based method system, the criteria-based method system and the performance-based measurement system (PBMS), both allows and relies on the user to demonstrate that whichever analytical method he or she elects to use to show compliance with an EPA regulatory standard meets the data quality requirements (or performance criteria) established for that particular regulatory program. In management terms, it stipulates the what — data quality that must be achieved — and leaves the how — the method or technology used to attain that quality of data — up to the user (laboratory or facility). This is in sharp contrast to some of EPA's programs' past policies, in which strict adherence to agency-approved methods has been required.

An EPA ad hoc working group, with input from other federal scientists, developed a general framework of what the new approach to method selection should entail as well as what it should provide to the potential user. In 1994, the working group was revised with a more formal status and a focused charge under EPA's Environmental Monitoring Management Council.

With representatives from the major media-defined program offices (e.g., Office of Air, Office of Water, Office of Solid Waste, Office of Emergency and Remedial Response), EPA regional offices, the Office of Enforcement and Compliance Assurance and the Office of Research and Development, the new workgroup struggled to find common ground. A proposed "one size fits all" approach received strong resistance from the program offices that regulate under the terms of the widely varying, media-specific legislation provided by Congress. To break the ensuing logjam, a page was taken from the above management philosophy: The workgroup established the what — a set of goals for the performance-based measurement system — and left the how — the specifics of implementation — up to each program office.

PBMS goals

The efforts of the workgroup reached a benchmark with the announcement (Federal Register 62 FR 52098, Oct. 6, 1997) by EPA's Administrator Carole Browner of the agency's intent to adopt a performance-based measurement system with the following goals:

• Providing a simple, straightforward way for the regulatory community to respond to specific measurement needs with reliable, cost-effective methods;
• Emphasizing project- or application-specific method performance needs rather than requiring that specific measurement technologies be used in order to avoid costly measurement overkill;
• Encouraging the use by the laboratory community of professional judgement in modifying or developing alternatives to established agency methods;
• Employing a consistent expression of performance criteria independent of the type of method or technology. This includes articulating measurement needs in qualitative and quantitative terms;
• Fostering new technology development and continuous improvement in measurement methodology by providing qualitative and quantitative targets for identified measurement gaps to method developers and other researchers; and
• Encouraging the measurement community to give the agency feedback on new monitoring approach successes as well as failures in order to expand knowledge of new or modified approaches and to assist others by helping to disseminate this information to the wider monitoring community.

Each of the program offices was then charged to develop a plan for implementing PBMSs within their respective programs. As part of this process, comments were solicited from the regulated community as well as from the states that implement the environmental regulations and that have auditing responsibilities for the quality of data produced in compliance with them. Since EPA does not envision requiring individual states to adopt PBMSs, it was vital to engage all parties in dialogue so that those affected by its implementation would have accurate information upon which to make their decisions. Through a series of informational public meetings, training courses and technical sessions held in conjunction with national conferences, EPA has tried to get the word out to the affected communities. Unfortunately, when the major program offices delivered their PBMS implementation plans, there was little similarity among those plans; the question among potential users then became which word applied to them.

Obstacles to instituting a PBMS

Lingering concerns for an analytical laboratory considering adopting a PBMS include:

• Will data obtained with an alternative method be as defensible as that obtained with an EPA-approved method?;
• To what lengths must the lab go to demonstrate the value of the method it has selected?;
• Where can the lab find appropriate quality assurance materials to demonstrate the methods's performance?;
• What kinds of quality control, quality assurance and documentation will be required?;
• Will all states, EPA programs and regions with responsibility for evaluating the data accept the lab's use of the selected method?; and
• How risky is an investment in a new technology when there are no assurances that it will be acceptable to clients or the regulators?

Several states have expressed their unwillingness to move to the new paradigm even after it is fully implemented by EPA. Reasons for their hesitance include:

• Lack of a uniform understanding of what a PBMS is and how it will work in the various regulatory programs;
• A perception that monitoring compliance will be more difficult (the numbers and competency of auditors vary widely among the states);
• Concern that acceptance of data from new methods could somehow jeopardize past decisions;
• Assumptions that EPA-approved methods yield the right data over a broad range of applications (or that there's little liability if they don't);
• Uncertainty over the acceptability of data from a "non-standard" method;
• Lack of familiarity with new technologies;
• A hard-to-break method-by-prescription mind set; and
• A perceived loss of control.

One of the activities affected by the implementation status of the PBMS is the development of common standards for analytical laboratories under the National Environmental Laboratory Accreditation Conference (NELAC), which has been supported in its efforts by EPA. In its development of quality standards, the NELAC has had to decide whether to accredit laboratories at the method-analyte-regulatory program level (accreditation based on the ability to measure an analyte in a matrix, appropriate to the regulatory program, using an EPA prescribed method; or at the matrix-analyte level (the laboratory is held responsible for demonstrating its ability to analyze particular matrix and analyte combinations to pre-defined levels of data quality without regard to the method used). The implications of such a fundamental difference in approach are enormous. The NELAC is on record as trying to accommodate EPA's PBMS approach in the development of its standards, but is frustrated with the lack of a uniform PBMS implement ation plan from EPA. Trying to anticipate what such a plan would look like is difficult enough, but trying to deal with three or more very different implementation strategies is a nightmare.

External input

The Environmental Laboratory Accreditation Board (ELAB) is chartered by EPA under provisions of the Federal Advisory Committee Act to provide consultation and advice to EPA and NELAC. It established a workgroup to define the essential elements of an ideal PBMS and to identify factors that would be necessary for the successful implementation of such a system, such as:

• Data generated under a PBMS must have the same legal standing as if from an EPA method;
• Requirements for method validation, demonstration of capability and ongoing quality control should be consistent across all EPA programs;
• Method validation and documentation requirements should be based on accepted scientific principles;
• EPA should publish PBMS performance criteria appropriate to the anticipated use; and
• Documentation must be sufficient for independent verification.

The workgroup also recommended that regulated entities be given unlimited flexibility to modify methods or use new ones so long as the performance requirements were met; that there be no unnecessary barriers to the acceptance of validated new technologies; that there be consistent definitions, objectives and criteria among EPA program and regional offices and the states; that implementation of a PBMS be as simple as possible consistent with the essential elements (above) and the PBMS goals and that performance criteria be clear, easy to apply and readily auditable. The group also recommended that during transition, a PBMS be implemented with due care for the interests of both EPA and the regulated industry. It was also suggested that appropriate reference materials and guidance in their use will be needed for successful PBMS implementation.

In an effort to determine what effect EPA's adoption of a PBMS would have on the laboratory community and to identify problems that might result from its implementation, the American Chemical Society conducted a study to compare the results obtained by laboratories using the PBMS and required method approaches. The study focused what impact there would be on the quality of the compliance decision process when laboratories are allowed to use whichever method they choose. A report of the study findings is in preparation.

A parallel effort was undertaken by a newly constituted PBMS workgroup under the Methods Board of the National Water Quality Monitoring Council, which also provides counsel to EPA. Because of the effective communications between the two workgroups, their recommendations were largely consistent with those of the ELAB's workgroup. In a bold move, the Method Board's workgroup has taken a lead in developing a pilot study that will compare the performance of the traditional chemical oxygen demand (COD) measurement method with that of a recently commercialized rapid COD test kit.

Through this pilot, the workgroup hopes to determine the extent of analyses and quality control procedures necessary for a lab to verify the performance of a validated method and evaluate the analyses necessary for a lab to determine whether it can appropriately substitute a new validated method for an existing approved method in a regulatory context using a PBMS framework. A review of the study design suggests the group has a strong likelihood of meeting its objectives.

One of the responses to recommendations from these various groups has come from an ad hoc group of federal and private-sector scientists who initially met to define what kinds of guidance are needed by analytical laboratories, their clients and other data users who move to a PBMS approach. The group has allied itself with the American Society for Testing and Materials' Committee D-34, and has developed an initial draft of a guide titled Standard Guide for Demonstrating that a Measurement System Provides Data Consistent with its Intended Purpose. The group hopes to produce a protocol that, when followed, will allow laboratories to determine and document the quality of the data they are producing with a particular measurement system and that will permit the users of such data to determine with confidence whether or not those data meet the quality requirements for a particular application, e.g., compliance with a regulatory limit or determining whether a site is contaminated above some specified level.

Critical actions

A key area that EPA is addressing is the need to educate (and reeducate) its own staff as well as the general environmental community in the new paradigm through the development and conduct of training courses. Last year, the first of the courses, the Facility Compliance Audit course, was piloted; it has since been improved based on participant feedback. This course teaches facility compliance inspectors how to look at monitoring data and determine if the data is of a quality and magnitude that demonstrates the facility is in compliance with permit or regulatory action levels. The emphasis is on auditing for data quality, not merely determining whether a facility followed a specific EPA protocol to the letter.

As PBMS implementation depends on the agency issuing the necessary regulations, a training program for agency regulation writers and their managers is now being developed. This course will cover both the general principles of regulation writing and the Program Office-specific implementation decisions. Both the agency-wide and the Office of Air and Radiation-specific modules have been drafted; after some refinement, they will be presented to EPA staff and to state regulatory officials and their staffs. Once these courses have been completed, the agency plans to develop training courses for laboratory management, for quality assurance officers and analysts; for EPA and State permit writers; for regulated entities and for the legal community. EPA's Office of Solid Waste has taken the initiative to issue its "comparable fuels rule" written in "PBMS language" and has other major regulations, similarly written, in the pipeline.

Conclusions

Despite the high potential that a PBMS offers for cost savings to the analytical laboratories, better data for state and national decision making, and enhancement of measurement science through adoption of new technologies, implementation of the PBMS varies widely across EPA's regulatory programs. EPA has sought and is using input from the greater scientific community to strengthen its approach to implementation of a PBMS. The development of consistent training materials and guidelines will enhance the visibility and understanding of the PBMS. Finally, the issuance of new, and the reissuance of existing regulations — written to comply with the principles of PBMS — and the unwavering support of the concept by the agency's senior management constitute the most important driving forces for implementation of this new paradigm.

Contacts

For information about the American Chemical Society's study comparing the results obtained by laboratories using the PBMS and required method approaches, contact Dr. Larry Keith at Larry_Keith@wpi.org.
For information about the pilot study and other activities of the Method Board of the National Water Quality Monitoring Council's PBMS workgroup contact Andy Eaton at Andrew.D.Eaton@us.mw.com.
For information about the ASTM's Standard Guide for Demonstrating that a Measurement System Provides Data Consistent with its Intended Purpose, contact Llewellyn R. Williams at llewwilliams-vegas@worldnet.att.net.
For information about EPA's training courses contact David Friedman of the EPA's Office of Research and Development at friedman.david@epa.gov.

Click here to post comments about this topic, and read what others have to say.

This article appeared in the May 2000 issue of Environmental Protection magazine, Vol. 11, No. 5, p. 31.

This article originally appeared in the 05/01/2000 issue of Environmental Protection.