At the November 20-21, 1996, meeting of the Peer Review Oversight Group
(PROG), a major topic was the Rating of Grant Applications (RGA). At the
previous meeting (July, 1996), PROG members had discussed the recommendations
in the RGA report, and considered information which had been obtained from the
scientific community through a variety of channels. At that time, the PROG
decided to table several of the ten recommendations in the RGA report and to
focus primarily on whether to use explicit criteria to structure the review, if
so what these criteria should be, whether to score/rate those individual
criteria, and whether to retain reviewer assignment of a global score or derive
an overall score from criteria subscores. Pilot experiments were recommended,
and the preliminary results of pilots by both the Division of Research Grants
and the National Institute of Allergy and Infectious Diseases were presented to
the members at the November meeting.
Based on the information available, the following decision was made after
the November meeting:
There will be no changes in the basic numerical scoring system at this
time. There was enthusiasm for having reviewers continue to assign a
global score to each application and that practice will continue. There was
generally low enthusiasm for the idea of assigning scores, whether numeric or
alphabetic, to the individual criteria or deriving an overall score using such
subscores, and those practices will not be adopted, but will be discussed
further at the Feburary PROG meeting. The issue of the rating scale itself has
been tabled for future discussion; for the present time, the 1.0-5.0 rating
scale, with 1 as the best possible score, will be retained.
In addition, there was much discussion of the information on the use of
criteria, and indications are leading in the general direction of using
criteria to structure the written critique and the review discussion. A
final decision on this issue will be made in January or February 1997 based on
additional information currently being obtained from program staff. The
criteria to be used would include at least Impact, Feasibility, and
Investigator/Environment; whether Innovation/Creativity would be best dealt with
as a separate criterion is under continuing consideration. There was general
agreement on the importance of fostering innovation and creativity in research,
and considerable discussion on how best to ensure that such applications were
submitted, receive appropriate review and are supported by the Institutes and
Centers. Regardless of whether a separate review criterion is adopted, the
consideration of Innovation/ Creativity in review and in research in general
will continue to be of the utmost importance for the PROG and for the NIH.
The criteria under consideration are stated below, and will be accompanied
by some guidelines or suggestions/examples of what they might encompass:
IMPACT: The extent to which, if successfully carried out, the project
will make an original and important contribution.
FEASIBILITY: The extent to which the conceptual framework, design,
methods, and analyses are adequately developed, well integrated, and appropriate
to the aims of the project.
INVESTIGATOR/ ENVIRONMENT: The extent to which the investigators,
available resources, institutional commitment, and any other unique features
will contribute to the success of the proposed research.
INNOVATION/CREATIVITY (working definition): The extent to which the
project employs novel concepts, approaches or methodology.
It was emphasized at the meeting that it is important for the scientific
community to understand that the use of these criteria does not represent a
change in the PHS 398 instructions, nor does it add criteria to the areas that
reviewers are instructed to use in considering the scientific merit of
applications during the review process. Rather, this effort represents a
reformatting or reorganization of the presentation of the information, in the
hope that it will result in more informative and perhaps more balanced
critiques. This should be an advantage to those who rely on this information,
both program staff in the Institutes and Centers and research investigators.
A working group was formed to gather information on the views of NIH program
staff regarding the use of a specific criteria for creativity will be chaired by
Dr. Claude Lenfant, Director of the National Heart Lung and Blood Institute and
will include representatives from the biological, clinical and behavioral
Note: The above summarizes only the issue on the Rating of Grant
Applications discussed by the PROG; for more detail on these deliberations and a
summary of the other issues discussed, see the Summary Minutes of the PROG for
November, 1996, which will be posted on the World Wide Web at http://www.nih.gov
on the Grants page under Peer Review Issues. At the next PROG meeting (Feb.
13-14, 1997) members will discuss the issue of how to evoke creative, innovative
projects, the possible use of a separate criterion for creativity, and the
wording of any further explanation of the criteria to be used, along with other