Peer Review: OMBs Proposed, Revised, and Final Bulletins
CRS Report for Congress
Peer Review: OMB’s Proposed,
Revised, and Final Bulletins
Updated February 3, 2005
Curtis W. Copeland
Specialist in American National Government
Government and Finance Division
Eric A. Fischer
Senior Specialist in Science and Technology
Resources, Science, and Industry Division
Congressional Research Service ˜ The Library of Congress
Peer Review: OMB’s Proposed and Revised Bulletins
Summary
In September 2003, the Office of Management and Budget (OMB) published
a proposed bulletin on “Peer Review and Information Quality” in the Federal
Register that sought to establish a process by which all “significant regulatory
information” would be peer reviewed. The scope of the proposed bulletin was very
broad, covering virtually all agencies and defining regulatory information as “any
scientific or technical study that ... might be used by local, state, regional, federal
and/or international regulatory bodies.” Such information would be subject to peer
review if the agency could determine that it could have a “clear and substantial
impact on important public policies or important private sector decisions” when
disseminated. The proposed bulletin placed additional peer review requirements on
“especially significant regulatory information,” and said agencies were required to
notify OMB in advance of any studies that might require peer review and how any
such reviews would be conducted.
The proposed bulletin aroused controversy, with some observers expressing
concern that it could create a centralized peer review system within OMB that would
be vulnerable to political manipulation or control by regulated entities. OMB
received nearly 200 comments on the proposal, and published a “substantially
revised” peer review bulletin in April 2004 that was broader in scope than the
proposed bulletin in that it applied to “influential scientific information” (which
includes, but is not limited to, regulatory information) and “highly influential
scientific assessments.” However, agencies were given substantial discretion to
decide whether information is “influential” and therefore requires a peer review. The
revised bulletin also allowed agencies to use the National Academy of Sciences for
peer reviews or to use other procedures that had been approved by OMB. It also
provided exemptions for certain classes of information, such as information related
to national security, products by government-funded scientists that are not
represented as views of a federal agency, and routine statistical information.
However, OMB retained significant authority to decide when information is “highly
influential” (and, therefore, requires more specific peer review procedures) and to
approve alternative peer review procedures.
OMB received more than 50 comments on the revised peer review bulletin,
many of which were supportive of the changes made to the proposal. However, some
commenters believed the changes did not go far enough, while others believed that
OMB had significantly weakened the bulletin. In January 2005, OMB published a
final version of the bulletin with what it described as “minor revisions” to the version
published in April 2004 (e.g., requiring agencies to disclose the identities of peer
reviewers and to prepare an annual report on their peer review activities). A number
of issues regarding the implementation of the bulletin remain unclear (e.g., how much
discretion agencies will be given to decide when and what kind of peer review is
required). This report will be updated when any further revisions to the bulletin are
published or other significant events occur.
Contents
Peer Review and Regulatory Policy....................................2
Support for Peer Review........................................3
Legislative Proposals.......................................4
Presidential Support........................................4
Cautionary Notes About Peer Review..............................5
Procedural Flexibility or Uniformity...........................6
Peer Review and the Information Quality Act........................7
OMB’s Proposed Peer Review Bulletin.................................8
Peer Review of Significant Regulatory Information...................9
Review of Especially Significant Regulatory Information..............9
OMB Notification of Upcoming Peer Reviews......................10
Other Provisions..............................................10
Comments on the Proposed Peer Review Bulletin and OMB’s Responses.....11
Need for the Bulletin..........................................12
Clarifications of Scope and Applicability..........................12
Agency Discretion and Peer Review Procedures.....................13
Selection of Reviewers........................................14
Public Participation...........................................14
Cost of Implementing the Bulletin................................14
Other Comments.............................................15
OMB’s Revised Peer Review Bulletin.................................16
Changes in Scope and Application...............................17
Influential Scientific Information.................................18
When Peer Review Is Required..............................19
Highly Influential Scientific Assessments..........................19
When Enhanced Peer Reviews Are Required...................20
Selection of Reviewers....................................20
Public Comments.........................................21
Other Requirements.......................................21
Alternative Procedures.....................................21
OMB Authority..........................................21
Upcoming Peer Reviews.......................................22
Comments on the Revised Bulletin...................................23
National Academy of Sciences..................................23
OMB Watch.................................................23
Chamber of Commerce........................................24
Members of Congress.........................................25
AAAS ......................................................25
Other Scientific Societies.......................................26
Final Peer Review Bulletin.........................................27
Concluding Observations...........................................28
Agency Discretion............................................28
Effects on Agencies and Rulemaking.............................30
List of Tables
Table 1. Comparison of Proposed, Revised, and Final Peer Review Bulletins..31
Peer Review: OMB’s Proposed and
Revised Bulletins
On September 15, 2003, the Office of Management and Budget (OMB) in
coordination with the Office of Science and Technology Policy (OSTP), published
a proposed bulletin in the Federal Register on “Peer Review and Information
Quality.” The bulletin, if made final, would have established a process by which all
“significant regulatory information” and “especially significant regulatory
information” would be peer reviewed.1 OMB described the term “peer review” in
this context as “a scientifically rigorous review and critique of a study’s methods,
results, and findings by others in the field with requisite training and expertise.” The
proposed bulletin placed additional peer review requirements on “especially
significant regulatory information,” and said agencies were required to notify OMB
in advance of any studies that might require peer review. The scope of the proposed
bulletin was very broad, covering virtually all agencies and defining “regulatory
information” as “any scientific or technical study that ... might be used by local,
state, regional, federal and/or international regulatory bodies.” OMB indicated it was
issuing the bulletin because agencies’ peer review practices were inconsistent, and
government-wide standards for peer review would make regulatory science more
competent and credible.
The proposed bulletin aroused substantial controversy, with some observers
expressing concern that it could create a centralized peer review system within OMB
that would be vulnerable to political manipulation or control by regulated entities.
OMB received nearly 200 comments on the proposal, including comments from
Members of Congress, trade associations, public interest groups, and recognized
experts in the field of peer review and scientific research. As a result of those
comments, OMB (again in consultation with OSTP) published what it described as
a “substantially revised” peer review bulletin in the Federal Register on April 28,
2004.2 In some ways, the revised bulletin was broader than its predecessor. For
example, instead of focusing on “significant” and “especially significant regulatory
1 Office of Management and Budget, Executive Office of the President, “Proposed Bulletin
on Peer Review and Information Quality,” 68 Federal Register 54023 (Sept. 15, 2003). This
proposed bulletin had been released to the public via OMB’s website on Aug. 29, 2003. To
view a copy, see [http://www.whitehouse.gov/omb/inforeg/peer_review_and_info_quality.
pdf].
2 Office of Management and Budget, Executive Office of the President, “Revised
Information Quality Bulletin on Peer Review,” 69 Federal Register 23230 (Apr. 28, 2004).
This revised bulletin had been released to the public via OMB’s website on April 15, 2004;
see [http://www.whitehouse.gov/omb/inforeg/peer_review041404.pdf]. In this report, the
first draft of the bulletin is referred to as the “proposed bulletin” and the second draft as the
“revised bulletin.” Unless otherwise specified, each respective reference includes the
introductory supplemental information as well as the body of the bulletin per se.
information,” the revised bulletin centered on “influential scientific information”
(which includes, but is not limited to, regulatory information) and “highly influential
scientific assessments.” In other ways, though, the revised bulletin was less inclusive
and directive. For example, it gave agencies more discretion to determine when
information required a peer review, and when the more detailed review requirements
for “highly influential” information were applicable. Also, unlike the proposed
bulletin, the revised bulletin did not exclude individuals from being peer reviewers
if they had received research grants from the agency disseminating the information
being peer reviewed. OMB again requested comments from the public on the
bulletin.
On December 15, 2004, OMB published a final version of the peer review
bulletin on its website.3 The final bulletin was published in the Federal Register on
January 14, 2005.4 OMB said this version reflects “minor revisions” made in
response to more than 50 comments from the public on the revised bulletin. For
example, the final bulletin requires agencies to disclose the names of peer reviewers
to the public and adds an annual reporting requirement to allow OMB to track how
agencies are using the bulletin. However, agencies are still afforded substantial
discretion to determine when and what type of peer review is required. OMB also
retains substantial discretion in certain areas.
This report briefly describes OMB’s September 2003 peer review proposal, the
concerns that were raised about that proposal in the public comments and elsewhere,
and OMB’s April 2004 revised bulletin. It then describes comments on the revision
and OMB’s final bulletin. First, however, the report provides some background on
peer review, particularly in the context of regulatory policy and recent efforts to
improve the quality of regulatory information.
Peer Review and Regulatory Policy
There is no official, government-wide definition of “peer review,” even in the
OMB bulletins.5 Agency-specific definitions and descriptions of the term differ, but
all of them contain the basic concept of a review of scientific or technical merit by
individuals with both sufficient technical competence and no unresolved conflicts of
3 Office of Management and Budget, Final Information Quality Bulletin for Peer Review,
Dec. 15, 2004, at [http://www.whitehouse.gov/omb/inforeg/peer2004/peer_bulletin.pdf].
4 Office of Management and Budget, Final Information Quality Bulletin for Peer Review,
5 While the term “peer review” is not explicitly defined in OMB’s bulletins, the
supplementary information in the final bulletin describes peer review generally as “one of
the important procedures used in science to ensure that the quality of published information
meets the standards of the scientific community. It is a form of deliberation involving an
exchange of judgments about the appropriateness of methods and the strength of the author’s
inferences. Peer review occurs when a draft product is reviewed for quality by specialists
who were not involved in producing the draft.” Arguably, the methods laid out in the
bulletin for conducting peer review could also be considered an implicit definition of the
concept.
interest. In its final bulletin, OMB states that peer review “typically evaluates the
clarity of hypotheses, the validity of research design, the quality of the data collection
procedures, the robustness of the methods employed, the appropriateness of the
methods for the hypotheses being tested, the extent to which the conclusions follow
from the analysis, and the strengths and limitations of the overall product.”6
Support for Peer Review
Some type of peer review has been used for centuries within the scientific
community to judge the quality of science. Peer review can take many different
forms, and is used for a variety of purposes. For example, it is used commonly by
federal agencies to evaluate research proposals, and plays a major role in funding
decisions. In these cases it is often part of a broader category of evaluation known
as merit review. Peer review is also the usual method by which the editors of
scientific publications evaluate proposed research reports. It even plays an important
role in many research institutions, including many government agencies, in decisions
about retention and promotion for individual scientists and in reviewing research
programs. A February 1999 report from the Committee on Science, Engineering, and
Public Policy concluded that expert review (the most common form of which is peer
review) is the most effective means of evaluating federally funded research
programs. 7
Peer review is also used for scientific and technical products relating to policies,
including regulations, to determine whether the underlying scientific findings are
well supported. For example, such peer review was established as Environmental
Protection Agency (EPA) policy in 1993.8 In 1996, a panel of leading economists
concluded that peer review should be used for economic analyses supporting
regulations with a potentially large impact on the economy (e.g., those whose annual
economic costs exceed $1 billion).9 The panel also indicated that reviewers should
be selected based on their demonstrated expertise and reputations.
In its peer review bulletins, OMB recognized the variety of ways that peer
review is used. OMB said that independent peer review is especially important in the
regulatory arena because federal agencies often develop or fund the science that
underlies their regulations, and then oversee the peer review of those scientific
studies — thereby creating the appearance of a conflict of interest.
6 Final Information Quality Bulletin for Peer Review, p. 2.
7 The Committee on Science, Engineering, and Public Policy, Evaluating Federal Research
Programs: Research and the Government Performance and Results Act (Washington:
National Academies Press, Feb. 1999). The Committee is a joint committee of the National
Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
8 U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal
Science Agencies Vary, GAO/RCED-99-99, Mar. 1999.
9 Kenneth J. Arrow, et al, Benefit-Cost Analysis in Environmental, Health, and Safety
Regulation: A Statement of Principles (Annapolis, MD: AEI Press, 1996).
Legislative Proposals. A number of statutes and legislative proposals have
advocated the use of peer review in a regulatory context for particular issues. For
example, the Safe Drinking Water Act (42 U.S.C. 300g-1(b)(3)(A)) requires EPA,
when taking action under the act based on science, to use the “best available, peerth
reviewed science and supporting studies.” In the 108 Congress, the Water
Resources Development Act of 2003 (H.R. 2557) would have generally required
project studies to be subject to peer review by an independent panel of experts if the
project has an estimated total cost of more than $50 million. Similarly, the Soundth
Science for Endangered Species Act Planning Act of 2003 (H.R. 1662, 108
Congress) would have directed the Secretary of the Interior, in making decisions
about species protection, to give greater weight to certain kinds of data that had been
peer reviewed by qualified individuals as defined in the bill.
There have also been legislative efforts to require peer review more broadly.th
For example, in the 106 Congress, the Senate considered but did not enact bipartisan
legislation (S. 746) that would have required virtually all agencies to provide for an
independent peer review of any required risk assessments and cost-benefit analyses
of major rules that the agencies or the OMB Director reasonably anticipated were
likely to have a $500 million effect on the economy. The bill would have required
that peer reviews be conducted through panels that were “broadly representative” and
involved participants with relevant expertise who were “independent of the agency.”
However, if an agency certified that adequate peer review had already been
conducted, and the OMB Director agreed, no further peer review would have been
required. In its comments on this legislation, the General Accounting Office (GAO)
(now the Government Accountability Office) generally supported the use of peer
review in this context, noting that “the rigorous, independent review of economic
analyses should help enhance the quality, credibility, and acceptability of agencies’
decisionmaking.”10 However, GAO cautioned that (given the number of reviews
contemplated) agencies would need to plan carefully for the reviews, and that the
panels would need to reflect all points of view.
Presidential Support. Recent presidential administrations have also
supported peer review as a preferred means of assessing scientific research, both
prospectively and retrospectively. For example, beginning with the FY1996 budget
cycle, OMB and OSTP have jointly provided annual direction to agencies,
encouraging them to emphasize the funding of peer-reviewed research over nonpeer-
reviewed research for most scientific activities. In 1997, the then-Administrator of
OMB’s Office of Information and Regulatory Affairs (OIRA) testified that the
Clinton Administration supported peer review, but also said the administration
recognized that it is not cost-free in terms of agencies’ resources or time.11
On September 20, 2001, the new OIRA Administrator for the George W. Bush
Administration issued a memorandum for the President’s Management Council
recommending (among other things) that agencies subject regulatory impact analyses
10 U.S. General Accounting Office, Regulatory Reform: Comments on S. 746 — The
Regulatory Improvement Act of 1999, GAO/T-GGD/RCED-99-163, Apr. 1999, p. 6.
11 Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on
Governmental Affairs, Sept. 12, 1997.
and supporting technical documents for “economically significant” and “major” rules
(e.g., those with a $100 million annual effect on the economy) to independent,
external peer review.12 The OIRA Administrator also recommended certain criteria
for peer review (e.g., disclosure by peer reviewers of prior technical or policy
positions on the issues at hand and their sources of personal and institutional
funding), and said OIRA would give agency analyses that had undergone such a
review “a measure of deference” during its reviews of their regulatory proposals
under Executive Order 12866.13
Cautionary Notes About Peer Review
Although the concept of peer review has generally received wide support, some
observers have also raised cautionary notes or mentioned certain limitations to the
approach in particular situations. For example, as OMB indicated in its 1997
testimony, peer review costs can be significant for agencies in terms of both time and
agency resources.14 To pay for peer review procedures, agencies may have to divert
resources from other areas (e.g., regulatory enforcement or standards development).
Scientists from academic institutions who perform peer reviews on a voluntary basis
may also incur opportunity costs with respect to other activities such as teaching and
research. Other concerns have focused on how peer reviews have been implemented.
For example, GAO noted that peer review methods varied within and among federal
agencies,15 and that agencies’ economic analyses of major rules were often not peer
reviewed.16 A number of observers have expressed concerns about the impact of peer
review requirements on the pace of regulatory activity, with additional requirements
exacerbating what is already often regarded as an “ossified” rulemaking process.17
12 For a copy of this memorandum, see
[ ht t p: / / www.whi t e house.gov/ omb/ i nf or e g/ oi r a _r evi e w-pr ocess.ht ml ] .
13 Executive Order 12866, “Regulatory Planning and Review,” 58 Federal Register 51735,
Oct. 4, 1993. For a description of OMB’s reviews under this executive order, see CRS
Report RL32397, Federal Rulemaking: The Role of the Office of Information and
Regulatory Affairs, by Curtis W. Copeland.
14 Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on
Governmental Affairs, Sept. 12, 1997.
15 GAO/RCED-99-99. For example, GAO reported that some agencies conducted peer
reviews of research proposals primarily by mail, while others generally relied on panels or
committees. All agencies used a combination of external and internal reviewers for these
reviews, but one (Federal Aviation Administration) relied primarily on agency employees
who were not employed in the project but had the required expertise.
16 U.S. General Accounting Office, Regulatory Reform: Agencies Could Improve
Development, Documentation, and Clarity of Regulatory Economic Analyses, GAO/RCED-
98-142, May 1998. Only 1 of the 20 economic analyses that GAO reviewed had been
independently peer reviewed.
17 See, for example, Thomas O. McGarity, “Some Thoughts on ‘Deossifying’ the
Rulemaking Process, Duke Law Journal, vol. 41 (June 1992), pp. 1385-1462; Richard J.
Pierce, Jr., “Seven Ways to Deossify Agency Rulemaking,” Administrative Law Review, vol.
47 (winter 1995), pp. 59-93; Paul R. Verkuil, “Rulemaking Ossification — A Modest
(continued...)
In fact, some critics have suggested that regulatory relief and delay is the primary
purpose of peer review proposals in this context.
Still other concerns about peer review have centered on issues of bias and
balance. Experts agree that effective peer review panels must be (and must be
perceived to be) free from any significant conflict of interest and properly balanced,
allowing for a wide range of views and appropriate expertise. However, in June
2001, GAO reported that the policies and procedures developed by EPA’s Science
Advisory Board to ensure balance and independence of the Board’s peer reviewers
had limitations that reduced their effectiveness.18 For example, GAO said that the
staff office did not routinely ensure that panel members’ financial disclosures were
complete or that they contained enough information to determine whether a conflict
of interest existed. Also, panel members reportedly did not have to disclose
information regarding previous positions on the matter being reviewed until the
panel’s first meeting, thereby making it difficult to determine the independence and
balance of the panel members until after they had been selected. GAO also reported
that the panel’s policies and procedures did not adequately inform the public about
the points of view represented on the panels. GAO made several recommendations
in this report that were designed to better ensure that the Science Advisory Board’s
peer review panels are independent and balanced.
Procedural Flexibility or Uniformity. As noted previously, GAO reported19
that federal agencies’ peer review practices are not consistent. However, several
studies of peer review in the federal government have suggested that rigidly uniform
peer review procedures may not be desirable. For example, a 1996 National Science
and Technology Council report indicated that peer review implementation should be20
flexible and “appropriate to the nature of scientific processes.” Similarly, in its
1999 report on peer review, GAO reported that OSTP believed agencies’ peer review
practices should be “flexible and tailored to agency missions and type of research,
and that specific uniform practices should not be dictated for every agency or all21
federally funded research.” GAO also said that agencies viewed a variety of peer
review methods as “both appropriate and essential, reflecting the varying nature of
the research and its purposes, the differences in research timelines, the broad
spectrum of [research and development] performers, and the varying funding
17 (...continued)
Proposal,” Administrative Law Review, vol. 47 ( summer 1995), pp. 453-459.
18 U.S. General Accounting Office, EPA’s Science Advisory Board Panels: Improved
Policies and Procedures Needed to Ensure Independence and Balance, GAO-01-536, June
19 GAO/RCED-99-99.
20 Subcommittee on Research, Committee on Fundamental Science, National Science and
Technology Council, Assessing Fundamental Science (Washington: National Science
Foundation, July 1996). The National Science and Technology Council was established by
executive order in November 1993 to coordinate federal research and development, and to
establish clear national goals for science and technology investments. To view a copy of this
report, see [http://www.nsf.gov/sbe/srs/ostp/assess/start.htm].
21 GAO/RCED-99-99, p. 5.
mechanisms, such as grants, contracts, and cooperative agreements.” Notably,
although GAO pointed out that peer review practices varied across and within
agencies, it did not recommend greater uniformity in those practices.
Peer Review and the Information Quality Act
OMB said that the final peer review bulletin was being issued “under the
Information Quality Act [IQA] and OMB’s general authorities to oversee the quality
of agency information, analyses, and regulatory actions.”22 The IQA (also known as
the “Data Quality Act”) was enacted in December 2000 as Section 515 of the
Treasury and General Government Appropriations Act for Fiscal Year 2001 (P.L.
106-554). The act supplemented the Paperwork Reduction Act (PRA), which already
required OMB to “develop and oversee the implementation of policies, principles,
standards, and guidelines to apply to federal agency dissemination of public
information.”23 The PRA also required agencies to manage their information
resources to “improve the integrity, quality, and utility of information to all users
within and outside the agency.”24
Although little noticed at the time it was enacted, the IQA has subsequently
been the subject of significant debate and controversy.25 The act required OMB to
issue guidance to federal agencies designed to ensure the “quality, objectivity, utility,
and integrity” of information disseminated to the public. It also required agencies to
issue their own information quality guidelines, and to establish administrative
mechanisms that allow affected persons to seek correction of information maintained
and disseminated by the agencies that does not comply with the OMB guidance.
However, it did not require either OMB or the agencies to issue peer review bulletins.
OMB published its final government-wide IQA guidelines in the Federal
Register in February 2002.26 In them, OMB encouraged (but did not require) the use
of peer reviews in the development of agency-disseminated information.
Specifically, OMB indicated that agencies can presume that data are sufficiently
“objective” if they have been subject to an independent peer review process (e.g., as
used by scientific journals), but said a member of the public could rebut this
presumption “based on a persuasive showing by the petitioner in a particular
instance.” OMB also indicated that journal peer review may not be sufficient for
22 Office of Management and Budget, Final Information Quality Bulletin for Peer Review,
op. cit., p. 2666.
23 44 U.S.C. 3504 (d)(1).
24 44 U.S.C. 3506(b)(1)(C).
25 For a more complete discussion of the IQA and OMB’s information quality guidelines,
see CRS Report RL32532, The Information Quality Act: OMB’s Guidance and Initial
Implementation, by Curtis W. Copeland and Michael Simpson.
26 Office of Management and Budget, “Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies,” 67
Federal Register 8452, Feb. 22, 2002. OMB published proposed guidelines in June 2001
(66 Federal Register 34489), and published an earlier version of the final guidelines in
September 2001 (66 Federal Register 49718).
information likely to have an important public policy or private sector impact.
Finally, the IQA guidelines set minimum standards for the transparency of agency-
sponsored peer review (referencing the standards in the above-mentioned September
20, 2001, memorandum issued by the OIRA Administrator — e.g., disclosure of
reviewers’ prior technical or policy positions and sources of funding).
The final peer review bulletin also cited Executive Order 12866 and OMB’s
authority to “manage the agencies under the purview of the President’s Constitutional
authority to supervise the unitary Executive Branch” as other authorities supporting
the bulletin’s issuance.27 E.O. 12866 says that OIRA within OMB is the “repository
of expertise concerning regulatory issues,” and says that agencies should base their
decisions on “the best reasonably obtainable scientific, technical, economic, or other
information concerning the need for, and consequences of the intended regulation.”28
However, the executive order does not specifically mention peer review.
OMB’s Proposed Peer Review Bulletin
When it published its proposed peer review bulletin in the Federal Register in
September 2003, OMB noted in the preamble to the bulletin the importance of peer
review and the sometimes inconsistent nature of its application. For these reasons,
the office proposed requiring peer review for “the most important scientific and
technical information relevant to regulatory policies that [agencies] disseminate to29
the public.” OMB said the new requirements would supplement, but not supplant,
both its own information quality guidelines and any agency-specific peer review
requirements for regulatory information. Although OMB noted that its bulletins “are
not laws or binding regulations,” as a practical matter, OMB’s instructions to the
agencies carry substantial weight because of its authority to review agencies’ budget
proposals and most significant regulations before they are published.
In essence, OMB’s bulletin proposed that each covered federal agency take three
actions: (1) have all “significant regulatory information” that the agencies intend to
disseminate peer reviewed; (2) have “especially significant regulatory information”
peer reviewed according to even higher standards; and (3) provide OMB with
information at least once each year about upcoming scientific studies that might
support significant regulatory disseminations, and the agencies’ plans for conducting
peer reviews of those studies. The proposed bulletin also contained several other,
related requirements. The bulletin defined “agency” broadly, including cabinet
departments and agencies, independent agencies (e.g., EPA), and independent
regulatory agencies (e.g., the Federal Communications Commission and the Nuclear
Regulatory Commission).
27 Office of Management and Budget, Final Information Quality Bulletin for Peer Review,
op. cit., p. 7.
28 Executive Order 12866, section 1(b)(7).
29 68 Federal Register 54026, Sept. 15, 2003.
Peer Review of Significant Regulatory Information
The proposed bulletin defined “significant regulatory information” as scientific
or technical information that qualifies as “influential” under OMB’s information
quality guidelines and is “relevant to regulatory policies.” OMB’s information
quality guidelines define “influential information” as information that the agency can
“reasonably determine” will have a “clear and substantial impact on important public
policies or important private sector decisions” when it is disseminated to the public.
The proposed peer review bulletin said information is “relevant to regulatory
policies” if it “might be used by local, state, regional, federal and/or international
regulatory bodies.” The bulletin indicated that the peer review for this type of
information should be “appropriate” and “scientifically rigorous,” but also said the
appropriate peer review mechanism would depend on “the novelty and complexity
of the science to be reviewed, the benefit and cost implications, and any controversy
regarding the science.”
For information at the lower end of those dimensions, the proposed bulletin
indicated that a review by qualified specialists within a separate part of the agency
disseminating the information could satisfy the peer review requirement. Also (as
in the information quality guidelines), agencies were allowed to presume that any
studies that had already been independently peer reviewed did not have to be
reexamined, but the bulletin said this presumption “is rebuttable based on a
persuasive showing in a particular instance.” Finally, the bulletin said peer review
was not needed for significant regulatory information related to national defense or
foreign affairs, or information that is disseminated in the course of an individual
agency adjudication or proceeding on a permit application.
Review of Especially Significant Regulatory Information
The proposed bulletin indicated that regulatory information should be
considered “especially significant” (and therefore subject to more specific peer
review requirements) if (a) the agency intends to disseminate it in support of an
economically significant regulatory action (e.g., with a $100 million annual impact30
on the economy), (b) its dissemination would otherwise have a clear and substantial
effect on important public policies or important private sector decisions (with an
impact of $100 million in any year), or (c) OMB determines it is of “significant
interagency interest” or “is relevant to an Administration policy priority.” The
bulletin laid out a series of proposed requirements for these types of peer reviews.
For example, it said agencies must
!select peer reviewers primarily based on their scientific and technical
expertise and “strive” to appoint reviewers who are independent of
the agency and do not have any “real or perceived conflicts of
interest” (e.g., have no financial interest in the issue, do not seek or
receive substantial funding from the agency, and have not conducted
a peer review for the agency or advocated a position in recent years
30 The definition of an “economically significant” regulatory action can be found in section
on the specific topic being examined). If obtaining the requisite
expertise requires the selection of a panel member with a perceived
bias, the bulletin said agencies must “ensure that another reviewer
with a contrary bias is appointed to balance the panel”;
!provide peer reviewers with an explicit statement describing the
purpose and scope of the review, and sufficient information to
understand the data, methods, and conclusions of the material being
reviewed;
!permit other interested agencies and persons to submit comments
and provide those comments to peer reviewers for their
consideration; and
!direct peer reviewers to issue a final report describing the nature of
their review and their findings and conclusions, including any
dissenting opinions. The report must also describe the names,
affiliations, and qualifications of all peer reviewers and any prior
affiliations with the agency.
Also, agencies were instructed to consult with OMB and OSTP regarding the
“sufficiency” of their planned peer review policies and, on request, to discuss with
OMB the sufficiency of their plans to review specific documents.
OMB Notification of Upcoming Peer Reviews
The proposed bulletin indicated that, at least once each year, each agency must
provide OMB with a summary description of any “existing, ongoing, or
contemplated” studies that might support significant regulatory information that the
agency intends to disseminate within the next year. Each agency was also to provide
OMB with the agency’s plan for conducting peer reviews of such studies under the
bulletin’s requirements, including an agency contact person. OMB indicated that, to
reduce paperwork, this information was to be included in other reports being
submitted to OMB under Executive Order 12866 or the Information Quality Act.
Other Provisions
The proposed bulletin also contained several other peer review requirements or
provisions, including the following:
!Agencies that are likely to disseminate “significant” or “especially
significant” regulatory information were required to supplement or
amend their information quality guidelines to incorporate the
requirements of the proposed peer review bulletin for “significant”
and “especially significant” information. These agencies’
supplements or amendments were also to address (a) what factors
could preclude an individual’s participation as a peer reviewer, (b)
how confidential business information and other confidential
information would be protected, and (c) what information about peer
reviewers should be disclosed (e.g., sources of funding that could
suggest a conflict of interest).
!OMB could waive the requirements for peer review if an agency
made “a compelling case” that a waiver was necessary (e.g., an
imminent health hazard or homeland security threat).
!Agencies were permitted to retain an outside firm to oversee the peer
review process to select and supervise the peer review panels, which
the bulletin indicated would allow the agencies to avoid the
requirements of the Federal Advisory Committee Act (FACA).31
OMB initially said these peer review requirements would apply to information
disseminated on or after January 1, 2004. OMB initially requested public comments
on the proposed bulletin by October 28, 2003, but later extended the comment period
to December 15, 2003, and then to January 16, 2004.
Comments on the Proposed Peer Review Bulletin
and OMB’s Responses
OMB said that it received 187 comments on the proposed bulletin on peer
review, and posted those comments on its website.32 In addition to this formal
commenting process, OMB and OSTP sponsored an open workshop on the peer
review requirements in November 2003.33 The workshop was hosted by the National
Academy of Sciences (NAS), which subsequently endorsed the bulletin’s objectives
but expressed several concerns. OMB said the comments that it received varied
substantially, with some highly favorable and others suggesting that the bulletin be
withdrawn and reconsidered. In general, supporters of the proposed bulletin
reportedly indicated that peer review standards across the government were
inconsistent, and that more consistent use of peer review could increase the technical
quality and credibility of regulatory science. They also reportedly asserted that peer
review could protect science-based regulations from political criticism and litigation.
For example, the U.S. Chamber of Commerce said that OMB had “provided a
Bulletin of high quality and sound judgment,” and strongly supported “OMB’s efforts
to improve the quality, objectivity, utility, and integrity of information disseminated
31 The bulletin cited Byrd v. EPA, 174 F.3d 239 (D.C. Cir. 1999), “holding that peer review
panels selected and supervised by outside consultants are not governed by [FACA].” For
more information on federal advisory committees and FACA, see CRS Report RL30260,
Federal Advisory Committees: A Primer, by Stephanie Smith; or CRS Report RL30795,
General Management Laws: A Compendium, coordinated by Clinton T. Brass, pp. 35-37.
32 To view the comments provided on the proposed peer review bulletin, see [http://www.
whitehouse.gov/ omb/inforeg/ 2003iq/iq_list.html ].
33 To view a transcript of this conference, see [http://www7.nationalacademies.org/stl/
Peer_Review_T r anscript.pdf].
by the federal government to the public.”34 Similarly, the National Petrochemical &
Refiners Association strongly supported OMB’s proposed bulletin, saying that it
would “correct the current variability in federal agency implementation of peer
reviews by providing the needed guidance on process requirements.”35
OMB said that many critics of the proposed bulletin viewed it as an effort to
inject political considerations into the world of science, and to use the uncertainty
that inevitably surrounds science as an excuse to delay new rules that could be costly
to regulated entities. They also reportedly said that the bulletin appeared more
concerned about peer reviewers’ possible conflicts of interest with agencies than their
potential conflicts with regulated industries.
In its summary of the public comments, OMB identified a number of broad
themes and noted how it responded to those comments.36 Specifically, OMB said the
comments centered on such issues as the need for the bulletin, its scope and
applicability, the prescriptive nature of the review process for “especially significant”
information, and the costs of implementing the bulletin.
Need for the Bulletin
OMB said many commenters indicated that it was unclear why the bulletin was
needed. For example, the American Public Health Association said it was “unaware
of any evidence that the current system is not working or any examples of
inappropriate or flawed federal regulations being promulgated as a result of failure
to peer review.”37 In response, OMB said it would provide additional information in
the revised bulletin on the importance of peer review and the variability of agencies’
peer review policies.
Clarifications of Scope and Applicability
A number of commenters reportedly said the scope and applicability of the
bulletin was unclear.
!Some commenters questioned the focus on “regulatory” information,
and said it would be difficult to know in advance whether a
particular study might be used in support of future regulatory action.
In its response to the comments, OMB said it would revise the
34 For a copy of the Chamber’s comments on the proposed bulletin, see [http://www.
whitehouse.gov/ omb/inforeg/ 2003iq/170.pdf].
35 For a copy of the association’s comments on the proposed bulletin, see [http://www.
whitehouse.gov/ omb/inforeg/ 2003iq/111.pdf].
36 Office of Management and Budget, “Summary of Public and Agency Comments on
Proposed Bulletin on Information Quality and Peer Review, Including Responses by OMB,”
Apr. 15, 2004. A copy of this summary is available at [http://www.whitehouse.gov/omb/
inforeg/ peer_review_comment.pdf].
37 Letter to Dr. Margo Schwab, Dec. 11, 2003. To view a copy of this letter, see [http://
www.whi t e house.go v/ omb/ i n f o r e g/ 2003i q/ 86.pdf ] .
bulletin to focus on “influential” scientific information instead of the
potential regulatory impact of the information disseminated. As
noted previously, the IQA guidelines state that information should
be considered “influential” if “the agency can reasonably determine
that dissemination of the information will have or does have a clear
and substantial impact on important public policies or important
private decisions.”38
!Some commenters expressed concern that the criteria for requiring
the more stringent peer review requirements for highly significant
information were too broad. In response, OMB said it would narrow
the applicability of those requirements to cover only “scientific
assessments” that either have a $500 million annual effect on
important public policies or private sector decisions or involve
novel, complex, or precedent-setting approaches or significant
interagency interest.
!OMB said it agreed with comments questioning the applicability of
the bulletin’s requirements to agencies’ regulatory impact analyses,
and said it would more clearly state that those analyses would not be
covered by the revised bulletin.39 However, OMB went on to say
that the bulletin’s requirements would cover the models and data
underlying those analyses, and encouraged agencies to have their
regulatory impact analyses peer reviewed by “government analysts.”
!OMB also said it would clarify that the bulletin did not apply to
products released by government-funded scientists that were not
represented as the views of the agency supporting the research.
!Finally, in response to several concerns, OMB said it would exempt
from the bulletin’s coverage any time-sensitive medical, public
health, and safety disseminations, and allow agency heads to waive
or defer the bulletin’s requirements “where warranted by a
compelling rationale.”
Agency Discretion and Peer Review Procedures
Commenters also said that the proposed bulletin was too prescriptive in the
procedures applicable to “especially significant” regulatory information, and said
extensive peer reviews would be costly and could lead to delays in regulatory
decisions. In response, OMB said that it would give agencies additional discretion
in the revised bulletin to determine the level of peer review required for any
particular document, and to use alternative procedures if they could demonstrate that
those procedures would meet the requirements in the government-wide and agency-
38 Office of Management and Budget, “Summary of Comments.”
39 OMB said that guidance for what constitutes a good regulatory impact analysis is in OMB
Circular A-4. To view a copy of this circular, see [http://www.whitehouse.gov/omb/
circulars/a004/a-4.pdf].
specific IQA guidelines. The response included significant discussion of the current
and potential role of NAS in conducting peer reviews. In its comments, NAS had
pointed to the role its committees have played in performing peer reviews of various
agency programs, activities, policies, and proposals. In determining the appropriate
level of review, OMB said agencies should consider the costs and benefits of peer
review. OMB also emphasized that its role is not to veto studies, but rather to ensure
that they have the appropriate level of review before implementation.
Selection of Reviewers
OMB also said it was incorporating suggestions regarding the selection of peer
reviewers, including that agencies solicit reviewers from the public and from
scientific and professional societies. One major concern of the commenters was that
the criteria outlined in the proposed bulletin (e.g., discouraging participation of
reviewers who were seeking or receiving funding via research grants from an agency)
could result in the most qualified reviewers being excluded from consideration.
OMB also said that the revised bulletin would make clear that the receipt of research
grants through peer-reviewed competitions does not disqualify a potential reviewer.
Public Participation
According to OMB, many of the commenters raised concerns about public
participation in the peer review process, questioning (among other things) whether
that participation would be useful and raising logistical considerations. As a result,
OMB said it had decided to allow the agencies to decide whether public participation
would be allowed and in what forms. However, OMB said it would provide for
public comment on the agencies’ peer review plans for upcoming documents.
Cost of Implementing the Bulletin
Finally, OMB said some of the commenters requested that OMB provide an
estimate of the costs associated with implementation of the peer review bulletin. For
example, in a March 2004 letter to OMB, Senators Joseph Lieberman and Richard
Durbin said it was “curious that OIRA — an office known for insisting on cost-
benefit analysis — provides no estimate of costs, nor documentation of the benefits
gained or specific harms that will be reduced by the proposal. We suggest that the40
proposed rules do not pass OMB’s own test of good rulemaking.”
OMB estimated that there would be about 125 “influential” scientific documents
covered each year that are related to rulemaking, and at least 1,250 other documents
each year that are unrelated to rulemaking. OMB said some of these documents may
be exempt or otherwise not require peer reviews (e.g., they had already been
adequately peer reviewed), but did not quantify that assertion. OMB said that a “high-
quality individual letter review” would cost an agency about $5,000 per document.
Therefore, assuming that 1,000 of the 1,250 documents required peer review and
40 To view a copy of this letter, see [http://govt-aff.senate.gov/index.cfm?FuseAction=
PressReleases.Detail&PressRelease_id=694&Affiliation=R]. This letter was not part of the
could be reviewed by letter, OMB said the cumulative cost of these reviews would
be $5 million.
OMB said “one or two dozen” of these documents could be considered “highly
influential scientific assessments” requiring more intense peer review. OMB
estimated that an average peer review of these documents would cost $50,000.
Therefore, if 20 of these “highly influential” assessments were peer reviewed, the
cost would be $1 million. OMB said these costs are “likely large overestimates of
the incremental costs of the Bulletin because many agencies already engage in peer
review practices.” The response also included an analysis that provided a
methodology for agencies to use in deciding when a more expensive peer-review
study by NAS would be cost-effective.41
Other Comments
Several other types of comments were offered on the proposed guidelines that
were not included in OMB’s summary. For example:
!OMB Watch questioned OMB’s authority to issue a bulletin on peer
review, noting that the PRA and the IQA do not explicitly provide
the office with that responsibility, and that Congress had previously
considered and rejected several proposals to mandate peer review.
OMB Watch said that if OMB believes government-wide peer
review requirements are needed, “it should instead submit its ideas
to Congress for consideration and possible authorization.”42
!The American Bar Association (ABA), noting that the proposed
bulletin required peer review whenever the impact of the regulatory
action would have exceeded $100 million, recommended that OMB
limit the bulletin’s categorical requirement for peer review to “those
situations in which the scientific information is at least arguably
complex, novel, or controversial.”43 The ABA said peer review “is
simply not the correct mechanism to address significant use of
routine, established, or accepted scientific information.”
!The ABA also said it agreed that one factor in determining whether
an individual should be a peer reviewer is whether that individual is
receiving or seeking substantial funding from the agency involved.
However, the ABA also said that an issue of equal concern is
whether a potential reviewer is receiving or seeking funding from a
private company or industry or citizen group with a stake in the
information involved.
41 OMB estimated that such a study might cost $1 million and take two years to complete.
42 Letter to Dr. Margo Schwab, Dec. 15, 2003. To view a copy of this letter, see [http://
www.whi t e house.go v/ omb/ i n f o r e g/ 2003i q/ 153.pdf ] .
43 Letter to Dr. Margo Schwab, Dec. 23, 2004. To view a copy of this letter, see [http://
www.whi t e house.go v/ omb/ i n f o r e g/ 2003i q/ 186.pdf ] .
!The American Association for the Advancement of Science (AAAS)
said it was concerned that one of the bulletin’s factors in selecting
reviewers — whether the prospective reviewer had advocated a
position on the matter at issue — could exclude a qualified scientist
who had simply made statements based on a “preponderance of peer
reviewed scientific evidence” (e.g., that particulate matter of a
certain size had been shown to cause asthma in children).44 AAAS
said such statements should not be considered “advocacy.” The
organization also said that OMB should make clear that individual
comments in agencies’ peer review reports would not be associated
with individual reviewers. In March 2004, the AAAS Council
adopted a resolution strongly opposing the proposed bulletin and
requesting that it be withdrawn.45
!The AEI-Brookings Joint Center for Regulatory Studies
recommended (among other things) that OMB or some other entity
conduct an evaluation of the peer review program after a certain
amount of time “to determine whether peer review actually led to a
material improvement in the quality of the regulatory analysis.” 46
AEI-Brookings said if OMB could not develop a reasonable
approach to evaluation, “it probably should not proceed with this
effort.”
OMB’s Revised Peer Review Bulletin
The comments that OMB received on the proposed bulletin led the office to
release what it described as a “substantially revised” bulletin in April 2004. The
changes were characterized in the media as a “partial retreat” in OMB’s initiative.47
OMB said the revised bulletin provided a more extensive discussion of why
government-wide peer review guidance was needed, gave agencies more discretion
in determining the appropriate type of peer review for specific products, and more
clearly indicated that the guidance does not create any new rights for litigation. Also,
the revised bulletin permitted an agency (under certain circumstances) to use a
scientist as a peer reviewer who had received research grants from the agency — a
practice that was discouraged in the proposed bulletin. The table appended to this
44 Letter to Dr. Margo Schwab, Dec. 12, 2003. To view a copy of this letter, see [http://
www.whi t e house.go v/ omb/ i n f o r e g/ 2003i q/ 81.pdf ] .
45 American Association for the Advancement of Science, Council, “On the OMB Proposed
Peer Review Bulletin,” Mar. 9, 2004, at [http://archives.aaas.org/docs/documents.php?doc_
id=434].
46 Robert W. Hahn and Robert E. Litan, Comment on Peer Review and Information Quality,
Regulatory Analysis 03-11, Dec. 2003. For a copy of this comment, see [http://www.aei.
brookings .org/ admi n/aut horpdfs/page .php?id=309].
47 Rick Weiss, “OMB Modifies Peer-Review Proposal: Guidelines Partly Retreat from Strict
Control of Agencies’ Information Process,” Washington Post, Apr. 16, 2004, p. A-19.
report provides a side-by-side comparison of the proposed and revised bulletins. The
sections below discuss similarities and differences in the documents.
Changes in Scope and Application
Among the more notable changes made to the bulletin were revisions to the
scope and applicability of the peer review requirements. In general, the scope of the
revised bulletin was arguably much broader than the proposed bulletin. For example,
whereas the proposed bulletin applied to “significant regulatory information,” the
revised bulletin applied to “influential scientific information” — a scope that
included, but was not limited to, regulatory information.48 However, the revised
bulletin also made it clear that certain types of information within this broad category
did not have to be peer reviewed, including:
!information distributed within the government or to contractors or
grantees;
!responses to requests under the Freedom of Information Act (FOIA),
the Privacy Act, or FACA;49
!correspondence with individuals;
!press releases;
!and distributions made to peer reviewers under the bulletin, provided
they contain a disclaimer.50
The revised bulletin also explicitly exempted from its requirements information
that (1) was related to national security, foreign affairs, or international trade or
treaties where compliance with the bulletin “would interfere with the need for secrecy
48 The explanatory section on requirements in the revised document states that the bulletin
“addresses peer review of scientific information disseminations that contain findings or
conclusions that represent the official position of one or more Departments or agencies of
the federal government.” However, the term “official position” is not mentioned in the
bulletin itself. The revised bulletin does not include the term “regulatory information.” The
original proposed bulletin defines “study” as “any research report, data, finding, or other
analysis.” While this definition is somewhat broader than the meaning usually ascribed to
the term by scientists, it is narrower than the definition of “scientific information” in the
revised bulletin, which includes “any communication or representation of knowledge ... in
any medium or form.”
49 FOIA is described in General Management Laws: A Compendium, coordinated by Clinton
T. Brass, pp. 26-29.
50 The disclaimer is required to say “This information is distributed solely for the purpose
of pre-dissemination peer review under applicable information quality guidelines. It has not
been formally disseminated by [the agency] and should not be construed to represent any
agency determination or policy.”
or promptness”;51 (2) was produced by government-funded scientists that was not
represented as the views of the agency; (3) was an agency regulatory impact analysis
or regulatory flexibility analysis under Executive Order 12866;52 or (4) was medical,
health, or safety information that was time sensitive or based on a recent clinical trial
that had already been adequately peer reviewed. The bulletin also allowed agency
heads to waive or defer the bulletin requirements “where warranted by a compelling
rationale.”53
In structure, the revised peer review bulletin was similar to the proposed bulletin
in that it still essentially required agencies to take three actions (to the extent
permitted by law):
!have a peer review conducted on all “influential scientific
information” that the agency intends to disseminate,
!have all “highly influential scientific assessments” peer reviewed
according to more specific and demanding standards, and
!indicate what “influential” and “highly influential” information the
agency plans to peer review in the future.
Each of these required actions is described more fully in the sections below.
Influential Scientific Information
The revised peer review bulletin defined the term “influential scientific
information” as information the agency “reasonably can determine that the
dissemination of which will have or does have a clear and substantial impact on
important public policies or private sector decisions.” Therefore, in contrast to the
proposed bulletin, the peer review requirements in the revised bulletin were not
limited to regulatory policies.
In addition to the limitations and exemptions discussed above, the revised
bulletin said that agencies were also not required to conduct a peer review of
influential information that had already had an “adequate” peer review. The bulletin
gave agencies substantial discretion in determining whether a prior review was
“adequate,” specifically stating that the earlier review “need not comply with all of
the requirements of this bulletin.” It also said that the sufficiency of the prior review
depended on the “novelty and complexity of the science to be reviewed and the
benefit and cost implications.” Language in the proposed bulletin about rebutting an
51 The proposed bulletin said agencies “need not have peer review conducted on significant
regulatory information that relates to national defense or foreign affairs.”
52 As noted previously, OMB has said that the data used to generate these analyses are
covered by the peer review requirements. Also, the bulletin did not indicate whether
regulatory flexibility analyses under the Regulatory Flexibility Act were exempt.
53 The proposed bulletin allowed OMB (not the agencies) to waive many of the bulletin’s
requirements if an agency “makes a compelling case” on grounds of an emergency,
imminent health hazard, homeland security threat, or “some other compelling rationale.”
agency’s presumption regarding whether earlier peer reviews were sufficient was not
included in the revised bulletin.
The revised bulletin (like the proposed bulletin) also gave agencies discretion
in determining the appropriate peer review mechanism for any information not
previously reviewed, with the level of rigor required again dependent on the “novelty
and complexity of the science to be reviewed and the benefit and cost implications.”
The bulletin stated that “appropriate” reviews could range from examinations by
other federal specialists to formal panels composed of independent experts. The
preamble to the bulletin indicated that some peer reviews could be done by letter
(e.g., when a draft document covers only one discipline), but said panels were
preferable when time and resources permitted their use. Peer reviewers were to be
selected based on their relevant expertise, and were not to have participated in
developing the product being reviewed. The revised bulletin also contained
requirements to handle participation by scientists with conflicts of interest,54 and said
that the comments of peer reviewers or a summary of those comments should be
made available to the public.
When Peer Review Is Required. In response to concerns that the trigger
for peer reviews was too vague, OMB indicated in the revised bulletin that the
reviews were needed whenever an agency “reasonably determines” that scientific
information “will have or does have a clear and substantial impact on important
public policies or private sector decisions.”
This change did not appear to address one of the underlying concerns voiced in
the comments on the proposed bulletin. The definition of “influential” information
is essentially the same as the one used to define “significant” information in the
proposed bulletin; the only difference is the broader applicability to non-regulatory
information. Therefore, agencies still may find it difficult to determine whether a
particular study will, at some point in the future, have a “clear and substantial impact
on important public policies,” much less “important private sector decisions.” Two
agencies in analogous situations may reach different conclusions regarding the need
for peer review. As a result, the overall purpose of the guidelines — a more
consistent application of peer review among federal agencies — may not be achieved.
Highly Influential Scientific Assessments
The revised bulletin contained a number of additional peer review requirements
for “scientific assessments” that are “highly influential.” These requirements were
much more specific than those placed on “influential” scientific assessments, differed
in some respects from the requirements in the proposed bulletin, and appeared to give
OMB and the agencies significant authority to determine when the requirements are
applicable.
54 Specifically, the revised bulletin says that the agency (or the entity selecting peer
reviewers) must (1) ensure that reviewers who are federal employees comply with federal
ethics requirements, (2) apply or adapt those requirements to reviewers who are not federal
employees, and (3) consider the National Academy of Sciences’ conflict of interest policy.
When Enhanced Peer Reviews Are Required. The revised bulletin
defined a “scientific assessment” as “an evaluation of a body of scientific or technical
knowledge which typically synthesizes multiple factual inputs, data, models,
assumptions and/or applies best professional judgement to bridge uncertainties in the
available information.” The term “highly influential scientific assessment” was
defined in the revised bulletin as “influential” scientific information (as defined in
the previous section of this report) that either the disseminating agency or OMB
determined
!met the above-mentioned definition of a “scientific assessment”; and
!could have a “clear and substantial impact on important public
policies ... or private sector decisions with a potential effect of more
than $500 million in any year,” or involved “precedent setting,
novel, and complex approaches, or significant interagency interest.”
As discussed previously, in the proposed bulletin enhanced peer review
requirements applied to “especially significant” influential information expected to
be disseminated in support of a regulatory action that could have a $100 million
impact in any year. Therefore, although the dollar threshold for when enhanced peer
review is required was higher in the revised bulletin ($500 million instead of $100
million), the peer review requirements were more broadly applicable in that they
applied to more than just regulations. Also, the language regarding “precedent
setting, novel, and complex approaches” replaced criteria in the proposed bulletin in
which OMB alone could decide that certain information required enhanced peer
review because it was of “significant interagency interest” or “relevant to an
Administration policy priority.”
Selection of Reviewers. The revised bulletin described in some detail how
peer reviewers of highly influential scientific assessments should be selected, but still
gave agencies substantial discretion in making the final decision. For example, the
bulletin said that reviewers must be selected primarily on the basis of “necessary”
expertise, experience, and skills, and must be diverse enough to “fairly” represent
different perspectives. The revised bulletin also said that agencies must consider
requesting the public to nominate reviewers, and must generally “bar participation
by scientists employed by the agency sponsoring the review.” (As noted previously,
the proposed bulletin said agencies should “strive” to select peer reviewers who were
independent of the agency.) In addition, as in the proposed bulletin, the revised
bulletin said agencies should try to avoid repeated use of the same reviewer on
multiple assignments.
However, in a significant departure from the proposed bulletin, the revised
bulletin also said that research grants awarded to a scientist through a competitive,
peer-reviewed process did not disqualify that scientist from serving on a peer review
panel. (The proposed bulletin indicated that peer reviewers who were “receiving or
seeking substantial funding from the agency through a contract or research grant”
should not be selected as peer reviewers.) Also, whereas the proposed bulletin
advocated balancing panel members’ biases, the revised bulletin contained no
language to that effect. Finally, the revised bulletin said that agencies could
commission entities independent of the agency to select peer reviewers and manage
the process.
Public Comments. The revised bulletin also gave agencies discretion in the
area of public comments. As noted previously, the proposed peer review bulletin
instructed agencies to permit interested agencies and persons to submit comments on
the information being reviewed, and to provide those comments to the peer reviewers
for their consideration. In contrast, the revised bulletin said that agencies could
decide on their own whether to make the information at issue available for public
comment, but said agencies should “consider” having a public comment period and
public meetings before the reviewers “when there is sufficient public interest.” The
revised bulletin did not indicate how agencies are to make this determination, or how
much public interest is “sufficient” to trigger the need for public comments. The
bulletin also said that if an agency decides to make a draft assessment available to the
public, it must, “whenever practicable,” provide peer reviewers with a summary or
compilation of the comments received.
Other Requirements. The revised bulletin contained several other types of
requirements that were also in the proposed bulletin. For example, the revised
bulletin required agencies to provide reviewers with “sufficient information” to allow
them to understand the data, assumptions, and analytic procedures used to support
key findings or conclusions. Also, agencies were required to instruct peer reviewers
to prepare a report summarizing their findings and conclusions. The report must
contain the names, affiliations, and credentials of the reviewers. Agencies were, in
turn, required to prepare a written response to the report, and both the report and the
response must be posted on the agency’s website.
Alternative Procedures. The revised peer review bulletin said that an
agency can (1) rely on scientific information produced by NAS, (2) commission NAS
to peer review the information, or (3) use an alternative procedure “specifically
approved by the [OIRA] Administrator, in consultation with OSTP.” As noted
previously, the bulletin permitted the agency head to waive or defer some or all of the
peer review requirements for “influential scientific assessments” and “highly55
influential scientific assessments” “where warranted by a compelling rationale.”
If the requirements are waived, however, the bulletin said peer review should be
conducted “as soon as practicable thereafter.” The proposed bulletin did not provide
for alternative procedures.
OMB Authority. As is apparent from the preceding discussion, the revised
peer review bulletin (like the proposed bulletin) gave OMB significant authority to
decide when agencies should use the more rigorous form of peer review.
Specifically, the revised bulletin indicated that these procedures should be used when
OMB (or the agency) determines that a scientific assessment (1) “could have a clear
and substantial impact on important public policies ... or private sector decisions with
a potential effect of more than $500 million in any year,” or (2) involves “precedent
setting, novel, and complex approaches, or significant interagency interest.”
55 The proposed bulletin would have permitted OMB (not the agency heads) to waive the
peer review requirements.
Therefore, under this definition, OMB could unilaterally determine that certain
information that an agency intends to disseminate is sufficiently “novel” or
“complex” to trigger these “highly influential” additional peer review requirements.56
Also, whereas the revised bulletin defined “influential”scientific information as
information that the agency can reasonable determine “will have or does have” a
clear and substantial impact on important public policies or private sector decisions,
the bulletin said that information should be considered “highly influential” if either
the agency or OMB determines it “could” have such an impact — conceptually, a
lower threshold than for information that is only “influential.”
OMB retained significant authority over the peer review process in other ways.
For example, OMB could determine that an agency’s alternative peer review
procedures are sufficiently rigorous to replace the requirements in the revised
bulletin.
Upcoming Peer Reviews
The proposed bulletin required agencies to notify OMB at least once each year
of any existing or upcoming studies that might trigger the peer review requirements
within the next year, and of the agencies’ plans for conducting those peer reviews.
In contrast, the revised bulletin required each covered agency to post an “agenda” on
its website every six months delineating any information disseminations subject to
peer review. The revised bulletin was also much more specific about these notices
than the proposed bulletin, requiring each entry on the agenda to contain a detailed
description of the “peer review plan,” including:
!the title, subject and purpose of the plan, with an agency contact,
!how the review will be conducted (panel or individual letters),
!the anticipated number of reviewers, and
!a “succinct description of the primary disciplines or types of
expertise needed in the review.”
Each peer review plan was also required to describe how reviewers will be selected,
whether the public will be asked to nominate peer reviewers, whether there will be
opportunities for the public to comment on the information being reviewed, and
whether the agency will provide reviewers with copies of any public comments
before beginning their work. The revised bulletin required agencies to establish a
mechanism to allow the public to comment on the adequacy of the peer review plans,
and must consider those comments.
56 In the proposed bulletin, OMB had similar authority. The agency could be required to
conduct a formal, independent, external peer review if OMB determined that the information
was of “significant interagency interest or [was] relevant to an Administration policy
priority.”
Comments on the Revised Bulletin
OMB said that 57 individuals and organizations commented on the April 200457
revised peer review bulletin. About a dozen of the comments essentially asked
OMB to allow more time for public comments and interagency review. Most of the
substantive comments provided were supportive of at least some of the changes that
OMB made to the bulletin. However, some commenters believed the changes had
weakened the bulletin to such an extent that they withdrew their initial support, while
others believed the changes had not gone far enough. Some of the individuals’ and
organizations’ comments are summarized below.
National Academy of Sciences
The presidents of the National Academy of Sciences and its two affiliated
institutions, the National Academy of Engineering and the Institute of Medicine,
released a statement on April 15, 2004 that generally praised the revision, arguing
that it would “improve the quality of the government’s scientific assessments and ...
decision-making,” and “better accommodate the diverse circumstances of ... federal
agencies.”58 The statement suggested, but did not state, that further improvements
were nevertheless desirable.59
OMB Watch
In May 2004, OMB Watch filed a comment on the revised peer review bulletin,
stating that although OMB had made “significant revisions” to the bulletin, it
nevertheless had “fail[ed] to correct some of the most fundamental complaints.60
Specifically, OMB Watch said that:
!OMB lacked the authority to issue the guidelines, given (1) the lack
of specific authority to do so in the statutes and orders cited and (2)
that Congress had not enacted legislation designed to require peer
review.
!OMB had not established that there was a government-wide peer
review “problem” in need of a government-wide solution.
57 For OMB’s response to the comments provided on the revised peer review bulletin, see
[http://www.whitehouse.gov/omb/inforeg/peer2004/peer_response.pdf]. For a link to each
of the comments, see [http://www.whitehouse.gov/omb/inforeg/peer2004/list_peer2004.
html].
58 National Academy of Sciences, “Academies’ Presidents Comment on OMB Peer Review
Guidelines,” Press Release, Apr. 15, 2004, at [http://www4.nationalacademies.org/news.nsf/
isbn/s04152004?OpenDocume nt].
59 “...the extended comment period will permit further refinements in the proposed
guidance...” (ibid.).
60 For a copy of OMB Watch’s comments, see [http://www.ombwatch.org/info/dataquality/
CommentsRevisedPR-OMBW.pdf].
!OMB retained too much control over federal peer review practices
(e.g., in determining when certain information requires a more
rigorous peer review and in approving alternative peer review plans),
and recommends that some of those responsibilities be given to the
National Academy of Sciences.
!the “independence” standard in the bulletin would diminish the
ability of government scientists to participate as peer reviewers,
while not affecting scientists from industry.
!the revised bulletin would likely significantly delay the regulatory
process at agencies by adding “several new layers of bureaucracy”
(e.g., public comments on peer reviews and peer review plans).
In its conclusion, OMB Watch recommended that OMB withdraw the bulletin and
instead convene an interagency committee to investigate peer review techniques and
flaws, or make changes in the bulletin to address their concerns.
Chamber of Commerce
The U.S. Chamber of Commerce submitted comments on the revised bulletin
in May 2004, stating that the revised bulletin has “fundamental problems that must
be resolved,” and that it could not support the issuance of a final bulletin until those
problems are addressed.61 The Chamber said it had “deep concerns” about (among
other things):
!the blanket exemption given to NAS peer reviews;
!the “excessive” discretion given to agencies in the implementation
of the bulletin;
!the absence of “provisions allowing affected parties to contest any
agency determination of applicability, peer review type, panel
selection, charge, or other peer review program element as it applies
to a specific case;” and
!insufficient limitations on the use of proprietary information and
models.
In summary, the Chamber said that the changes OMB made in issuing the revised
bulletin “are so severe and debilitating as to eliminate the public benefit of having a
common, government-wide minimum standard for peer review.”
61 For a copy of the Chamber’s comments, see
[ h t t p : / / www. u s c h a mb e r . c o m/ NR/ r d o nlyres/eznlbdbxz7wylilw7now26n2cqigr nx57hqpav
qkxf7xgvfy2foiy5w5bd37suvonnhln7j v7myxop/COMMENT SProposedRevi sedBulletino
nPeerRevi ewandInfo.pdf].
Members of Congress
In May 2004, twelve Members of Congress provided OMB with comments on
the revised bulletin, stating that the revision did not address previously expressed
concerns that the proposal was “unjustified, overly broad, burdensome, and did not
appropriately guard against appointment of reviewers with conflicts of interest,” or
that it provided OMB with “excessive authority over the production and
dissemination of government information.62 The Members said that the need for the
bulletin remained unjustified, that the exemptions in the revised bulletin resulted in
an unbalanced approach, and that it created “considerable new burdens on agencies.”
They also said that “safeguards to ensure the integrity of the process for selection of
reviewers have not been included in the Bulletin.” In particular, they said there was
“no requirement that agencies disclose publicly whether any panel members with a
conflict were selected.” In conclusion, the Members characterized the bulletin as “a
wolf in sheep’s clothing,” and said it was “designed to prevent the dissemination of
government information, stifle public debate, and delay legitimate government
regulatory action.” They also said that the agencies, not OMB, are “the repository
of scientific and technical expertise,” and said they should be given “full authority
to govern their own peer review procedures.”
AAAS
AAAS filed a comment with OMB in June 2004, saying that the “revised63
version is much improved” but raising several issues. For example, AAAS:
!questioned why OMB said scientists seeking an exemption to the
peer review requirements for information produced by government-
funded scientists were only “advised” to include a disclaimer rather
than required to do so;
!suggested that agencies be required to make public the criteria for
determining when outside entities would be commissioned to select
peer reviewers or manage the peer review process, and how these
entities would be selected and overseen by the agencies; and
!endorsed the idea of an interagency working group on peer review,
and recommended that the group be required to report to the public
annually on its deliberations and findings, and to conduct studies and
hold meetings regarding the impact of the guidelines on science and
the quality of the information produced.
The editor of AAAS’s flagship publication, Science, praised OMB for its efforts to
address the concerns of the scientific community, saying that most of the concerns
62 For a copy of these Members’ comments, see [http://www.whitehouse.gov/omb/inforeg/
peer2004/25.pdf].
63 For a copy of the AAAS comments, see [http://www.aaas.org/spp/cstc/docs/5-28-04OMB
peerltr.pdf].
regarding the proposed bulletin were answered in the revision, and that OMB
“deserves credit for being a good listener.”64
Other Scientific Societies
A number of other scientific societies also commented on the revised peer
review bulletin. For example, an official in the Association of American Medical
Colleges was quoted in April 2004 as saying that OMB had been “very responsive,”
and said the revised document “is much, much more flexible.”65 In July 2004 (after
the comment period for the revised bulletin closed), a consortium of 13 scientific
societies, mostly in the life sciences, released a general statement on the role of
scientific peer review in policy development.66 The statement cited peer reviews as
“critically important tools for policy makers” and laid out a list of eight
“considerations” the societies believe will help ensure that agency peer reviews are
well designed and performed. While several of the points were also made by other
organizations, some provided a somewhat different emphasis. For example, the
societies:
!stressed that for scientific peer review to be effective, it “should be
insulated from politics as much as possible.”
!said peer review is not a way to get the “right” answer for policy; in
many instances, even the best data are subject to alternative
interpretations, and there is no “best” answer.
!said that, while avoiding and managing conflicts of interests is
important, most scientific peer review rests on the presumed
integrity of the reviewers, stemming from professional standards of
conduct to which scientists subscribe.
!said it is critical to acknowledge the cultural differences among
scientists, policymakers, and the public in developing and
implementing an appropriate model for peer review. For example,
science is inherently uncertain, and scientists must acknowledge that
in their work, including peer review. Managers, in contrast, often
are not in a position to embrace that uncertainty but must make
decisions in the face of it.
64 Donald Kennedy, “Praise, for a Change,” Science, vol. 304 (May 21, 2004), p. 1077.
65 “White House Softens Disputed Peer-Review Plan,” Science, vol. 304 (Apr. 23, 2004),
p. 496.
66 American College of Preventive Medicine et al., “Scientific Peer Review in Policy
Making,” Position Statement, July 15, 2004, [http://www.aibs.org/position-statements/
Final Peer Review Bulletin
As noted previously, on December 15, 2004, OMB released its final bulletin on
peer review. OMB said that it had made “minor revisions” to the April version of the
bulletin that were “responsive to the public’s comments.”67 Perhaps most notably,
OMB said that the final bulletin:
!requires that agencies disclose to the public the names of peer
reviewers for both “influential scientific information” and “highly
influential scientific assessments.” However, the bulletin does not
require agencies to disclose which reviewers made which comments.
!adds an annual reporting requirement to address concerns about the
lack of an enforcement mechanism, thereby allowing OMB to “track
how agencies are using the Bulletin, including provisions for
waivers and exceptions.” OMB also said it expects the public to
monitor agencies’ actions under the bulletin.
!requires agencies to designate at the time they plan a peer review
whether it should be considered “influential scientific information”
or a “highly influential scientific assessment.”
!provides additional criteria for what constitutes a “highly influential
scientific assessment.” Specifically, the term now includes
influential assessments that are “novel, controversial, or precedent
setting or have significant interagency interest.” OMB also clarified
that the $500 million impact test covers all impacts (public or
private sector) that occur in any year.
!broadened the applicability of the exemption for time-sensitive
health or safety information, noting that it is not restricted to medical
data from clinical trials that were subject to adequate peer review
before the start of the trial.
!requires agencies to adopt or adapt the National Academy of
Sciences policy for committee selection with respect to evaluating
conflicts of interests of peer reviewers who are not federal
employees.
OMB also reportedly made a number of what appear to be less significant changes
to the final bulletin. For example, OMB said that the final version of the bulletin:
!retains a substantial degree of discretion for the agencies, but also
clarifies the intent of its language concerning the “adequacy of prior
67 The characterizations in this section of OMB’s changes in response to public comments
are from OMB’s response to the comments provided on the revised peer review bulletin,
available at [http://www.whitehouse.gov/omb/inforeg/peer2004/peer_response.pdf].
peer review” and the use of deferrals and waivers to address
concerns that the revised draft was too discretionary.
!encourages agencies to provide public participation opportunities for
reviews of “highly influential assessments,” but avoids a strict
mandate for public participation and adds language “to stress that
agencies should avoid open-ended comment periods.”
!“reinstate[s]” instructions that peer reviewers are to comment on
scientific and technical questions, and not to provide opinions on the
policy implications.
!provides a “rare exception” to allow a premier government scientist
from the department or agency that prepared an assessment to
participate in a peer review (under certain circumstances).
Concluding Observations
OMB’s peer review bulletin is likely to have a significant effect on federal
rulemaking and other forms of information dissemination and public policy. That
effect is likely to be both direct (through agencies’ and OMB’s enforcement of the
bulletin’s requirements) and indirect (through references to the bulletin by others).
For example, Section 402 of the Specialty Crops Competitiveness Act of 2004 (P.L.
108-465, signed by the President on December 21, 2004) indicated that a required
peer review of the procedures and standards governing the consideration of certain
import and export requests “shall be consistent with the guidance by the Office of
Management and Budget pertaining to peer review and information quality.”
Some of the initial issues and concerns raised by commenters on the proposed
peer review bulletin were clarified or otherwise addressed in the revised and final
versions of the bulletin. Perhaps most notably, the bulletin now makes it clear that
scientists are not prohibited from serving as peer reviewers if they receive research
grants from the agency based on investigator-initiated, competitive, peer reviewed
proposals. The bulletin also provides exemptions from the peer review requirements
for time-sensitive medical, public health, and safety information, and other
compelling circumstances. However, a number of other issues remain unclear,
including (1) the amount of discretion that agencies will actually have in carrying out
the bulletin’s requirements, (2) the degree to which the bulletin will accomplish the
stated goal of making federal peer review practices more consistent, and (3) the
effects of the bulletin on agencies and the federal rulemaking process.
Agency Discretion
The final peer review bulletin appears to give federal agencies substantial
discretion in determining whether peer review is required for specific products and,
if so, what type of peer review mechanism is appropriate and who should serve as
peer reviewers. For example, the bulletin says agencies need not have peer review
conducted on influential scientific information that had already been subject to
“adequate” peer review. To determine whether a prior review was “adequate,”
agencies are directed to consider (among other things) whether the science is “novel”
or “complex,” and whether it is “important” to decision making. (Notably, though,
the final bulletin no longer indicates that an “adequate” peer review “need not
comply with” all of the requirements of the bulletin.) Also, the final bulletin says
that information should be considered “influential scientific information” if the
agency “reasonably can determine” that it will have or does have a “clear and
substantial” impact on “important” public policies or private sector decisions. The
bulletin says that peer reviewers should be selected to provide the necessary
expertise, experience, and skills, and the group of reviewers should be “sufficiently”
broad and diverse to “fairly” represent the “relevant” scientific perspectives and
knowledge.
On the other hand, the final bulletin also gives OMB substantial discretion in
certain areas. For example, the bulletin indicates that OMB can require agencies to
use the more exacting procedures for “highly influential scientific assessments” if
OMB determines the information is a “scientific assessment” that “could” have a
substantial impact on public policies or private sector decisions with a “potential
impact” of more than $500 million in any year, or is “novel, controversial, or
precedent setting,” or has “significant” interagency interest. Also, OMB can
unilaterally approve agencies’ use of alternative peer review procedures.
The amount of discretion that agencies will actually have in carrying out their
peer review programs (or, conversely, the amount of control that OMB will retain)
will be apparent only through the bulletin’s implementation. The amount of agency
discretion (or OMB control) could vary substantially from one administration to
another.
Consistency
OMB indicated that stronger peer review policies were needed because of the
importance of the issue and, citing a 1999 GAO report, because of the “variability in
both the definition and implementation of peer review across agencies.” OMB went
on to say that, prior to the development of the bulletin, “there were no government-
wide standards concerning when peer review is required and, if required, what type
of peer review processes are appropriate.” Therefore, OMB said that the bulletin
“establishes minimum standards for when peer review is required for scientific
information, and the types of peer review that should be considered by agencies in
different circumstances.”
However, the final bulletin may not provide the desired consistency in peer
review definition or implementation. As indicated above, the bulletin leaves many
key terms undefined or subject to interpretation, and gives the agencies substantial
discretion regarding when certain actions should be taken (e.g., when previous peer
reviews are “adequate”) and which reviewers should be selected (i.e., those with the
“necessary” expertise, experience, and skills). To the extent that agencies are
allowed to exercise discretion in these areas, consistency may be forfeited. On the
other hand, strict enforcement of uniform procedures established by OMB carries
with it a different set of concerns about aggregation of power within the Executive
Office of the President, and may be resisted by federal agencies. Also, as noted
previously, although GAO reported that agencies’ peer review practices were
inconsistent, it did not recommend greater consistency, and some view variation in
those practices as appropriate and desirable.
Effects on Agencies and Rulemaking
A number of commenters expressed concerns regarding the effect that adding
new peer review requirements would have on what is already viewed by some
observers as a lengthy, “ossified” federal rulemaking process. Somewhat related
concerns have been voiced regarding the cost of the requirements to federal agencies,
with Senators Lieberman and Durbin suggesting that the requirements “do not pass
OMB’s own [cost-benefit] test of good rulemaking.” In response, OMB provided
estimates in the preamble to the revised bulletin indicating that it did not believe the
costs would be prohibitive to the agencies. As noted previously, though, in 1997,
OMB indicated that peer review costs could be significant in terms of both time and
agency resources.68 Little empirical data are currently available regarding the cost of
peer reviews, how they affect the federal rulemaking process, or their effect on the
quality of the information being reviewed. It is even less clear how the peer review
requirements suggested by OMB will affect those factors, and there appears to be no
mechanism in place for collecting such data. There are some indications, however,
that the requirements could delay regulatory action.69
The AEI-Brookings Joint Center for Regulatory Studies recommended that
OMB or some other entity build into the peer review program an evaluation to
determine its effect on the quality of regulatory analyses. Any such evaluation could,
at least conceptually, include an examination of the cost of the new peer review
requirements to federal agencies and its effect on the pace of rulemaking. To
determine the effect of the peer review bulletin on these or other factors, baseline
information regarding the current state of the art would need to be gathered before
the bulletin’s implementation.
Even if the data indicate that peer review adds time to the early stages of the
rulemaking process, that time may be worth the investment if doing so reduces the
likelihood of subsequent judicial challenges to the rules. Peer review may also
provide agencies with a preview of likely objections to a rule during the notice and
comment phase, thereby allowing them to minimize any weaknesses and respond
quickly to adverse comments. All of these factors would have to be considered in
any evaluation of the effect of peer review on the regulatory process.
68 Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on
Governmental Affairs, Sept. 12, 1997.
69 Ben Geman, “White House Peer Review Requirements Could Delay Standards,”
Greenwire, volume 10, number 9 (Dec. 17, 2004). The article quoted a program manager
in the Department of Energy as saying that efficiency standards for residential furnaces and
boilers, commercial air conditioners and other equipment were being delayed for two years
because of OMB’s peer review bulletin.
CRS-31
Table 1. Comparison of Proposed, Revised, and Final Peer Review Bulletins
Proposed BulletinRevised BulletinFinal Bulletin
rget of basic level of peer“Significant regulatory information,”“Influential scientific information,” definedSame as revised bulletin.
iewdefined as scientific or technicalas scientific information (including but not
regulatory information that the agencylimited to regulatory information) that the
can “reasonably determine” will have,agency “reasonably can determine”will
when disseminated, a “clear andhave, when disseminated, “a clear and
substantial impact on important publicsubstantial impact on important public
policies or important private sectorpolicies or private sector decisions.”
decisions.”
rget of more rigorous peer“Especially significant regulatory“Highly influential scientific assessments,”Essentially the same as revised
iki/CRS-RL32680iewinformation,” defined as significantdefined as influential scientific informationbulletin. “Highly influential
g/wregulatory information (1) disseminatedthat either the agency or OMB determinesscientific assessments” defined as
s.orin support of an economicallyto be “scientific assessments” (e.g., state-influential scientific information
leaksignificant regulation (e.g., $100 millionof-science reports, meta-analyses, or riskthat the agency or OMB determine
annual impact on the economy); (2) thatassessments) that (1) could have a “clear(1) could have a “potential impact”
://wikicould, when disseminated, have a “clearand substantial impact on important public(changed from “clear and substantial
httpand substantial impact on importantpolicies or private sector decisions with aimpact”) of more than $500 million
public policies or important privatepotential effect of more an $500 million inin any year or (2) are “novel,
sector decisions”(those with $100any year; or (2) involve “precedent setting,controversial [changed from
million impact in any year); or (3) OMBnovel, and complex approaches, or“complex”]or precedent-setting”
determines is of “significant interagencysignificant interagency interest.”scientific assessments (changed
interest” or “is relevant to anfrom “approaches”).
Administration policy priority.”
CRS-32
Proposed BulletinRevised BulletinFinal Bulletin
iewersPeer reviewers “should” not haveSame as proposed bulletin.Reviewers “shall” not have
participated in the development of theparticipated in development of work
work product.product.
Agency “shall strive to appoint expertsAgency (or entity selecting the reviewers)Essentially the same as revised, but
who ... are independent of the agency,must “bar participation by scientistsadds exception for the “rare case”
[and] do not possess real or perceivedemployed by the agency sponsoring thewhere the agency determines that a
conflicts of interest.” One of the factorsreview unless the reviewer’s service as apremier government scientist (1) is
to consider in determining independencepeer reviewer defines the governmentemployed in a different agency of a
is whether the individual “is currentlyemployment (i.e., special governmentCabinet-level department, (2) is not
receiving or seeking substantial fundingemployees).” However, “[r]esearch grantsin a position of management or
from the agency through a contract orthat were awarded to scientists based onpolicy responsibility, and (3)
iki/CRS-RL32680research grant.”investigator-initiated, competitive, peer-possesses “essential expertise that
g/wreviewed proposals generally do not raisecannot be obtained elsewhere.”
s.orissues as to independence or conflicts.”
leak
://wikiAlso consider whether the individual hasNo similar provision.No similar provision.
httppreviously “advocated a position on the
specific matter at issue.”
Also consider whether the individual hasAgencies “should avoid repeated use of theAgencies “shall” avoid repeated use
conducted multiple peer reviews for thesame reviewer on multiple assignmentsof the same reviewer unless
same agency in recent years.unless his or her participation is essential.”participation is essential “and cannot
be obtained elsewhere.”
If obtaining the appropriate expertiseNo similar provision.No similar provision.
requires selecting a reviewer with an
apparent bias, agencies should ensure
that another reviewer with a contrary
bias is selected.
CRS-33
Proposed BulletinRevised BulletinFinal Bulletin
Agencies should avoid repeated use of
the same reviewer on multiple
assignments.
iewAgencies should select an appropriateSame as proposed bulletin. Essentially the same criteria, but
echanism (significant orpeer review mechanism (e.g., reviewalso (1) the importance of the
ation)within agency or formal review byinformation to decisionmakers, (2)
independent body) based on the noveltythe extent of prior peer review, and
and complexity of the science to be(3) the factors regarding
reviewed, the benefit and costtransparency (see below).
implications, and any controversy
regarding the science.
iki/CRS-RL32680mentsAgencies are required to “provide anAgencies are allowed (but not required) to“Whenever feasible and
g/wopportunity for other interested agenciesobtain comments on draft scientificappropriate,” agencies are required
s.orand persons to submit comments,” andassessments. If they do so, they mustto make the draft scientific
leakmust “ensure that such comments are(“whenever practical”) provide a summaryassessment available to the public
://wikiprovided to the peer reviewers withor compilation of those comments thatfor comment when submitted for
httpample time for consideration before theaddress “significant scientific and technicalpeer review and to sponsor a public
peer reviewers conclude their reviewissues.” When there is “sufficient publicmeeting. If they do so, agencies
and prepare their report.”interest,” agencies must “consider”must (“whenever practical”) provide
establishing a public comment period onpeer reviewers with access to the
the peer reviewers’ draft report andpublic comments.
sponsoring a public meeting.
es to the bulletin’sNo comparable languageAgencies may (1) rely on scientificSame as revised bulletin.
information produced by NAS, (2)
commission NAS to peer review the
information, or (3) use an another
procedure specifically approved by OMB.
CRS-34
Proposed BulletinRevised BulletinFinal Bulletin
ansparency requirementsThe agency shall direct peer reviewersA detailed summary or copy of theThe agency or entity managing the
(“individually or often as a group”) toreviewers’ comments, as a group orpeer review shall instruct reviewers
issue a final report “detailing the natureindividually, shall be made available to theto prepare a report describing “the
of their review and their findings andpublic.” Also, the report shall be part ofnature of their review and their
conclusions.” The report should alsothe administrative record for related agencyfindings and conclusions.” The
disclose “the names, organizationalactions “where appropriate.” report must (1) either include a
affiliations, and qualifications of all peerverbatim copy of each reviewer’s
reviewers, as well as any current orcomments (with or without
previous involvement by a peer reviewerattribution) or represent the views of
with the agency or issue under peerthe group as a whole, and (2)
review consideration.” The reportdisclose the names of the reviewers
“should be included in theand their organizational affiliations.
iki/CRS-RL32680administrative record for any relatedThe report must be disseminated on
g/wrulemakings.”the agency’s website, discussed in
s.orany related rulemaking, and
leakincluded in the administrative
record.
://wiki
httping peer reviewsAgencies shall provide OMB at leastAgencies shall post on their websites,Same as revised, but agencies also
once each year (1) a description of anyupdated at least every six months, anrequired to provide a link from the
existing or upcoming studies that mightagenda designating all planned and ongoingagenda to each document made
constitute or support significantinformation disseminations subject to peerpublic pursuant to the bulletin.
regulatory information that the agencyreview. For each entry, agencies mustAlso, agencies’ peer review plans
intends to distribute within the nextprovide detailed information in a “peershould indicate whether the
year, and (2) the agency’s plan forreview plan” (e.g., the subject and purposedissemination will be “influential”
conducting a peer review of suchof the review, how the review will beor “highly influential,” the timing of
studies. conducted, the number of reviewers, howthe review, and whether alternative
they will be selected, whether the publicprocedures will be employed.
will be allowed to comment, and whether
reviewers will be provided with those
comments). Agencies are also required to
allow the public to comment on the plan,
and to consider those comments.
CRS-35
Proposed BulletinRevised BulletinFinal Bulletin
Peer reviews are not required for:Peer review is not required for:Same as revised bulletin, except:
— studies that have already had an — information produced by government- — time sensitive “medical”
“adequate” peer review (withfunded scientists if not represented asdissemination not excluded
review by a scientific journalthe views of the agency (with(only “health or safety”);
considered adequate, butdisclaimer); — underlying data and analytical
rebuttable); — information related to nationalmodels used in regulatory
— regulatory information related tosecurity, foreign affairs, orimpact analyses or flexibility
national defense or foreign affairs;negotiations involving internationalanalyses are not excluded; and
ortrade or treaties; — an exclusion added for
— information disseminated in the — information disseminated in theanalyses of routine statistical
course of an adjudication or permitcourse of an adjudication or permitinformation.
application.application, unless the agency
iki/CRS-RL32680determines it is novel or precedent
g/w setting;
s.or — medical, health, or safety
leakdissemination that is time sensitive or
based on clinical trial data that was
://wikipreviously peer reviewed;
http — regulatory impact analyses or
flexibility analyses subject to
interagency review under EO12866;
or
— routine statistical information.
ers and deferralsOMB may waive the peer reviewThe agency head may waive or defer someSame as revised bulletin.
requirements if agency makes aor all of the peer review requirements for
“compelling case” that waiver is needed“influential” and “highly influential”
because of:information/assessments “where warranted
— an emergency,by a compelling rationale.” If deferred, the
— an imminent health hazard,review should be conducted “as soon as
— a homeland security threat, practicable thereafter.”
— “some other compelling rationale”
CRS-36
Proposed BulletinRevised BulletinFinal Bulletin
No provision.No provision.By December 15, agencies required
to provide OMB with a report
containing (1) the number of peer
reviews conducted during the fiscal
year; (2) the number of times
alternative procedures were
invoked; (3) the number of times
waivers or deferrals were invoked;
(4) any decisions to use exceptions
in reviewer appointments; (5) the
number of panels conducted in
public and allowed public
iki/CRS-RL32680comments; (6) the number of public
g/wcomments provided on review
s.orplans; and (7) the number of
leakreviewers recommended by
professional societies.
://wiki
http