Announcer: From the National Institutes of Health in Bethesda, Maryland, this is All About Grants.
Megan: Welcome to another edition of All About Grants. I’m your host, Megan Columbus, from the NIH Office of Extramural Research. Today we’ll be talking about understanding your summary statement. To get a couple of different perspectives I have with me Dr. David Armstrong, the Chief of the Review Branch from NIH’s National Institute of Mental Health and Dr. Mike Sesma a Program Officer, also from NIMH. Both Mike and David have been on the other side of the table as investigators competing for and receiving NIH grants. So, before we get into the summary statement, let’s begin talking about your respective roles in the review process. David, how about we start with you.
David: We have the Scientific Review Officer who serves a variety of roles. One, actually at the review meeting itself, he or she is the designated federal official who must be present for the review meeting to go forward. This individual is the expert there to answer any of the questions that the review committee may have regarding procedural issues or issues related to the review itself. Prior to the meeting it’s the responsibility of the Scientific Review Officer to ensure that the applications we receive are complete and most importantly, though, to understand the science in the applications that are being reviewed and to select and recruit appropriate extramural scientists to serve as reviewers to evaluate the scientific merit of that application.
Mike: The role of the PO in all of this is a little bit less direct. Our role is to represent our institute at the review, and we track those applications that are assigned to our institute as they go through the review. Often a number of those applications that will be reviewed by this scientific review group or study section are applications that may actually be assigned to our individual grant portfolios that we manage at the institute. Basically, we’re there to take notes on the individual applications, noting the particular aspects of the review criteria and strengths and weaknesses of the application and how they respond to those review criteria.
Megan: So how about we turn to the topic of summary statements. What exactly is a summary statement used for?
Mike: I’m usually the second one to read it after the applicant, and basically, summary statements are used to obtain an independent and objective, expert opinion of the scientific and technical merit of an application. So we count on the reviewers and the panel to provide us with their best assessment of both the significance, the importance of the science that’s being proposed, but also the way it’s going to be done and who’s going to be doing it. All of those things should give us a sense of the likelihood if that will be a successful project, but also one that helps advance the field.
Megan: So, David, what should I expect a summary statement to look like and how are they constructed?
David: In general, your application will be assigned to three independent reviewers. The reviewers are asked to comment upon the strengths and weaknesses of the five primary review criteria, as well as other criteria that may affect, or not, the overall impact scores. As an applicant, when you receive your summary statement, you will see these individual critiques collated together along also with the various scores on a 1-9 associated with each of those five criteria. Importantly, though, and this is the role of the Scientific Review Officer, if you are one of those applications that is discussed during the meeting, you’ll have a resume and summary of the discussion.
Megan: So that summary is written by that Scientific Review Officer that is running that meeting?
David: Yes, that summary and resume of the discussion is written by the Scientific Review Officer, and it is meant to capture the essence of that discussion, not merely to repeat the highlights that are in the critique. And where it is particularly useful to you as the applicant, when there may be disagreement among the committee members as to the strengths and weaknesses, as well as to the overall impact of a particular application. It’s the Scientific Review Officer who will capture that minority voice or a majority voice, a dissenting voice, is stating one view or another. So it’s in fact a very important role of the SROs to capture, I really do stress, the essence of that conversation.
Megan: The reviewer’s critiques are written before the meeting. Do the reviewers ever go back and adjust their critiques based on the discussion?
David: The reviewers have the opportunity to go back in to revise their critiques, The SRO will certainly stress repeatedly and encourage the reviewers to please change your written words and your criterion scores to match that which is at the end of the discussion. So if you have a change of heart, let it reflect that final thought that you have, both in written words and in your criterion scores.
Megan: But that may or may not happen. However, you can also contact your Program Official who was listening to the discussion.
Mike: Exactly. The role of Program at a review meeting is to basically sit in the room, observe and listen. What we try to observe and listen, as each application is reviewed during the discussion, is to get a feel for how the reviewers convince each other of their particular assessment of the strengths and weaknesses of a particular application. And often, as David said, since the individual critiques are written by each reviewer and submitted prior to the meeting, often the emphasis of their evaluation of the individual strengths and weaknesses, using those review criteria, can shift as a function of that discussion or as a function of questions asked of the reviewers by each other or by other panel members who weren’t assigned to the review. And what’s important for us is to be able to understand how the reviewers arrived at the position that they ultimately end up with in terms of assessing the overall impact of the application and arriving at a score that reflects both the scientific and technical merit.
Megan: I assume that means that after I receive my score and summary statement, I should call my program officer to find out if they can provide any additional details about that discussion?
Mike: Absolutely. Usually I get a call from an applicant as soon as they get their score; they want to know what that means. And I always tell them that what you have to do is wait for the summary statement and read it several times before you call me and then we’ll have that conversation. Often my first question to the applicant will be, “What did you read in your summary statement, what did you think that the reviewers said about your application? Then with the notes that I have I can basically either confirm or explain further and extend that discussion a little bit to help the applicant understand what it was the reviewers found compelling or unexciting or problematic or innovative in their application.
David: I think another thing to stress again, and it can’t be stressed too many times, is that when the applicant receives his or her summary statement, what he or she is receiving is the critiques of the three, plus or minus, assigned reviewers. The overall impact score is of course the composite of all voting members around the table, which could be 30, 40 individuals. We hope that the critiques are somewhat comprehensive and do not stray too much. It certainly is a practice of SROs, if there is a voice that may be non-conforming, to ask that individual to please write a comment to be included in the critique, as well. So those are common practices so that everybody has the full picture of what actually occurred.
Megan: Recently, NIH has been encouraging the use of a paragraph written by each of the assigned reviewers to describe how they came to the overall score of that application. What information is that intended to provide?
David: So this is just starting with this current round of applications in terms of providing, rather than a bulleted explanation of the strengths and weaknesses that went into your consideration of the overall impact score, rather a narrative. What the intent of this is again to capture your gestalt, so to speak, of those elements that really contributed to your overall impact scores, not meant to be a reiteration of your bullets that may be listed below under the various criterion. We want a distinct narrative from the reviewer that says, “I think this application is strong because of this, this and this. I think it may be weak because of this and that.” So again, a clear statement as to what components came to drive your final score. Once again, the overall impact score is the synthesis of the five criterion scores, so it’s important that we know to some extent what contributed most in terms of arriving at your overall impact score. We hope it will be useful, and we think it certainly will be.
Megan: Thank you both for joining us. We’ll talk to you soon when we talk about resubmissions. For OER and NIH this is Megan Columbus.
Announcer: To learn more about the NIH peer review process, visit the Office of Extramural Research’s website at grants.nih.gov. That’s G-R-A-N-T-S dot N-I-H dot G-O-V.