CAHPS II Release Date: August 28, 2001 RFA: RFA-HS-02-001 Agency for Healthcare Research and Quality Letter of Intent Receipt Date: September 24, 2001 Application Receipt Date: November 14, 2001 PURPOSE The Agency for Healthcare Research and Quality (AHRQ) invites applications for cooperative agreements to build on and expand the project known as Consumer Assessment of Health Plans Study (CAHPS). AHRQ funded the initial CAHPS cooperative agreements from October 1995 through September 2001. In this document, we refer to that project period as CAHPS I. The current RFA will be referred to as CAHPS II. The purposes of CAHPS I were to conduct research to: 1) produce reliable, valid and rigorously tested survey protocols for collecting information from consumers regarding their assessments of health plans and services; 2) develop and test the effectiveness of different formats for conveying resulting information to consumers; 3) evaluate the use of resulting survey protocols in applied settings; and 4) evaluate the usefulness of this information in assisting consumers, and purchasers acting on their behalf, in making informed selections of health care plans and services. Detailed information on CAHPS I including all products developed under CAHPS I are available at http://www.ahrq.gov/fund/grantix.htm. The present RFA builds upon the earlier work to enrich the evidence base concerning consumer assessment and reporting and enhance the usefulness of CAHPS products. Its specific purposes are to conduct research to 1) maintain existing CAHPS products, 2) build upon three existing CAHPS products relating to nursing homes, group practices and persons with mobility impairment, 3) develop new products, including innovative ways of reporting CAHPS data, 4) assess the usefulness of CAHPS data for quality improvement purposes and 5) evaluate the effectiveness of CAHPS in applied settings. HEALTHY PEOPLE 2010 The Public Health Service (PHS) is committed to achieving the health promotion and disease prevention objectives of "Healthy People 2010," a PHS-led national activity for setting health improvement priorities for the United States. AHRQ encourages applicants to submit grant applications with relevance to the specific objectives of this initiative. Potential applicants may obtain a copy of "Healthy People 2010" at http://www.health.gov/healthypeople. ELIGIBILITY REQUIREMENTS AHRQ strongly encourages all eligible and qualified organizations to respond to this RFA. Status as a CAHPS I grantee is not a prerequisite for this RFA. Though each applicant organization or team must have experience in all areas described in the Objectives section, this experience need not be based on CAHPS products. We have referenced as many published and unpublished documents and reports as possible for the purpose of describing the CAHPS I experience to potential new CAHPS to this RFA. This RFA includes areas of effort not addressed in CAHPS I: the use of CAHPS data for quality improvement purposes and a greatly expanded version of the reporting task (developing a comprehensive research approach to the development and testing of reporting formats). Both these areas call for different types of expertise above and beyond those required by CAHPS I. Applications may be submitted by domestic and foreign, public and private non-profit organizations, including universities, clinics, units of State and local governments, and eligible components of agencies of the Federal Government. Under recently enacted reauthorization legislation, AHRQ is authorized to enter into cooperative agreements with for-profit organizations as well as with public and not-for-profit entities. Thus, for-profit organizations are invited to respond to this notice with research applications for cooperative agreements. Such applications will be administered in accordance with Subpart E of 45 CFR Part 74 and 42 CFR Part 67 Subpart A. The latter regulation has not yet been amended to reflect these changes in Agency name and authority. (See December 6, 1999 reauthorization at http://www.ahrq.gov/hrqa99a.htm). AHRQ encourages women, members of racial and ethnic minority groups, and persons with disabilities to apply as Principal Investigators. MECHANISM OF SUPPORT The administrative and funding instrument to be used for this program will be a cooperative agreement (U18), an "assistance" mechanism (rather than an "acquisition" mechanism), in which substantial AHRQ scientific and/or programmatic involvement with the awardee is anticipated during performance of the activity. Under the cooperative agreement, the AHRQ purpose is to support and/or stimulate the recipient's activity by involvement in and otherwise working jointly with the award recipient in a partner role, but it is not to assume direction, prime responsibility, or a dominant role in the activity. Details of the responsibilities, relationships and governance of the study to be funded under cooperative agreement(s) are discussed later in this document under the section Terms and Conditions of Award. The total project period for applications submitted in response to the present RFA may not exceed 5 years. The anticipated award date is Spring 2002. This RFA is a one-time solicitation. AHRQ has not determined whether or how this solicitation will be continued beyond the present RFA. We have chosen the cooperative agreement mechanism for CAHPS because it allows AHRQ to have a greater involvement in facilitating the collaborative efforts of the consortium members. Continuing involvement in CAHPS research also enables AHRQ staff to serve more effectively as liaisons with other governmental and non-governmental agencies who are interested in using CAHPS products. FUNDS AVAILABLE AHRQ expects to award from $3 million to $4.5 million total costs (direct costs plus facilities and administrative costs) in fiscal year 2002 to support the first year of $1 million to $1.5 million to approximately three grantees under this RFA. The total award amount is dependent on the final fiscal year 2002 budgets for both AHRQ and the Center for Medicare and Medicaid Services (CMS, formerly the Health Care Financing Administration). The actual number of applications funded is dependent on the number of high quality applications and their individual budget requirements; it is not the intent of AHRQ that awards be equal in size. Should additional funds become available and if AHRQ receives a sufficient number of meritorious applications, AHRQ reserves the right to fund additional applications under this RFA. Although this program is provided for in the financial plans of AHRQ, awards pursuant to this RFA are contingent upon the availability of funds for this purpose. Funding beyond the initial budget period will depend upon annual progress reviews by AHRQ and the availability of funds. Definitions The work of the CAHPS I grants and the Survey User Network (SUN) contract has been performed by dividing team members into groups that perform specific tasks or work towards defined goals. Since the names of these working groups are mentioned in resource documents referenced in this RFA, we identify these groups by name and describe their composition in this section. Since the division of team members as described in this section served us well in CAHPS I, it makes sense to discuss their use in CAHPS II. After awards are made, the grantees will meet with the Project Officers to discuss how the work can best be conducted. They will then decide together on the best way to subdivide the grantee teams to facilitate the completion of agreed-upon goals. The SUN or Survey User Network contract is a five year contract for the provision of various types of support to the CAHPS consortium. This support includes preparing CAHPS products for dissemination to potential users; providing technical assistance to users; providing communication support to facilitate the work of the CAHPS consortium; providing technical and conference support for meetings; and providing support for a National CAHPS Benchmarking Database (NCBD) described below. During CAHPS I, the SUN contractor played a critical role in identifying the needs of current users of CAHPS products, responding quickly and efficiently to those needs, and developing effective strategies for communicating with users on a regular basis. The ways in which the grantees will interact with the SUN contractor will be described in greater detail in the Study Organization and Budget and Related Issues sections. Additionally, details regarding the scope of work of the SUN contract can be found on the AHRQ website at http://www.ahrq.gov. The NCBD contains two major data sets: CAHPS survey response data and health plan descriptive data. The primary purpose of the NCBD is to facilitate comparisons of CAHPS results among various types of CAHPS survey sponsors, including Medicaid agencies, public and private employers, and individual health plans. These comparisons are intended to provide both purchasers and health plans with benchmarking information useful for evaluating and improving performance as measured by the CAHPS survey. The NCBD also offers an important source of primary data for conducting specialized research that will enable improvements in future survey design and a better understanding of health plan and enrollee characteristics that influence performance. All sponsors of CAHPS surveys that are administered independently according to CAHPS specifications are invited to participate in the NCBD. Survey sponsors include public and private purchasers (employers, state Medicaid agencies, coalitions) and individual health plans. The confidentiality of individual respondent information is maintained. The NCBD also has several working groups including the NCBD Advisory Board, Technical Advisory Panel, and Executive Research Committee. Grantees will participate in each of these workgroups. Further information regarding grantee responsibilities will be presented in the Budget and Related Issues section. Additional information about the NCBD can be found at http://ncbd.cahps.org. The CAHPS consortium refers to the entire line-up of people who work on CAHPS, direct its work or advise the grantees. This includes all grantees, AHRQ CAHPS staff (Project Officers and others), the SUN contractor, the CAHPS Steering Committee (to be described in the Study Organization section) and the CAHPS Grantee Advisory Committee (to be described in the Study Organization section). Throughout this RFA, the term CAHPS consortium is used interchangeably with the CAHPS team. Each grantee and their consortium/contractual partners/collaborators are referred to as a grantee team. However, when the term grantee team is used throughout this solicitation, it is understood that the primary grantee is responsible for the project, which includes the programmatic and fiscal management of third parties. We refer to the CAHPS Survey and Reporting Kit, or CSRK, throughout this RFA. The CSRK contains all CAHPS products as well as detailed advice on their use. Portions of the kit are available through the AHRQ website in addition to instructions for obtaining a complete kit, if desired. Finally, the name of this project has been updated to reflect its expanded focus. In the final years of CAHPS I, the team began development of tools through which consumers and patients can assess care given to them by providers (as members of group practices) and care they receive in nursing homes. These efforts will be built upon in CAHPS II. Since the focus of our product development has expanded beyond health plans, the name Consumer Assessment of Health Plans no longer accurately describes the team’s work. However, CAHPS has become a recognized product name in the industry. In view of this, we are keeping the project name CAHPS but will no longer use the original name for which this used to be an abbreviation. RESEARCH OBJECTIVES 1. Project Goals and Objectives A brief summary of the development of products during CAHPS I and some information about CAHPS users can be found at http://www.ahrq.gov/fund/grantix.htm. We encourage applicants to consult them. Our main reason for referencing these reports is to inform potential applicants who were not CAHPS I participants about team experiences and decisions made during that time frame. In addition, one of the objectives of this RFA is the further development of products of the three CAHPS I projects: nursing home CAHPS (NH CAHPS), group practice level CAHPS (GCAHPS) and CAHPS for people with mobility impairments (CAHPS PWMI). Applicants who were not CAHPS I participants may need this additional background information and information about the status of these three efforts to prepare an application. Though the background documents represent the current best thinking regarding development and testing of CAHPS products, applicants should not feel constrained by their conclusions or recommendations. We encourage applicants to think beyond these materials and present different strategies and new approaches to the research challenges stated in this RFA. In this section we describe the goals for CAHPS II. This RFA includes five categories of work to be performed across two phases. The categories are: 1) refinement of current CAHPS products; 2) continued development of NH CAHPS, G-CAHPS and CAHPS PWMI; 3) development of new products; 4) quality improvement studies; and, 5) evaluation studies of the use of CAHPS products. Each category is discussed below. Work in these areas will occur across two time spans or phases: phase 1 includes years one and two of the cooperative agreement; phase 2 includes years three, four and five. The phase or phases within which a product is to be completed are listed below. With the new emphasis on quality improvement and evaluation research in CAHPS II, it is unlikely that any single organization wishing to respond to this RFA will contain in itself people with expertise necessary to develop and test questionnaires, develop and test reporting templates, modify existing CAHPS products, plan and conduct QI studies and perform evaluations. We encourage applicants to team with individuals or organizations who add expertise in important areas. Applications that respond only to individual elements of this RFA rather than the entire range of activities will be returned as nonresponsive. Refinement of Current CAHPS Products: Ongoing through Phases 1 and 2 For CAHPS surveys, refinement involves periodic updating of all survey- related information in the CAHPS Survey and Reporting Kit and includes work on such elements as item wording, administration procedures, sampling guidelines, data preparation, analytic strategies, guidance on interpretation of results, as well as any other information that would facilitate use of the products on the kit. Grantees are required to maintain existing as well as questionnaires under development. For CAHPS reporting templates, refinement involves periodic updating of all print and computer reports-related information contained in the CSRK. This includes such elements as development and testing of explanatory material in the templates; development and testing of different data displays; reconfiguration (if necessary) of reporting composites; development and testing of graphic designs and navigational aids. Grantees will also be required to update all other CAHPS products as the consortium determines necessary. These products include but are not limited to the booklet Frequently Asked Questions about CAHPS and the Survey Submitter’s Kit. The current versions of these can be found on the AHRQ website. For all CAHPS products, revisions may also be based on feedback from current and potential users, changes in technology, changing requirements of accreditation organizations or other sources of information identified by the Project Officers. Further development of G-CAHPS, NH CAHPS and CAHPS PWMI G-CAHPS: Phase 1 NH CAHPS: Phase 1 CAHPS-PWMI: Phase 1 In addition to product refinement, grantees will build upon the prior development and testing of three CAHPS products (G-CAHPS, NH CAHPS and CAHPS PWMI). The former grantees and SUN contractor will deliver to the CAHPS II grantees all relevant materials. Group Practice Level CAHPS (GCAHPS): The CAHPS team in collaboration with the Pacific Business Group on Health and funded jointly by the California Health Care Foundation and AHRQ developed and tested a GCAHPS instrument in California, Eastern Massachusetts, Denver, Knoxville, and St. Louis. The resulting instrument is in a beta test stage. Several other sites around the country are administering the survey. Additional questions remain about sampling and data collection. The CAHPS II goal is to continue development and address these issues so that the GCAHPS survey can be added to the set of CAHPS products. The final report on the developmental efforts to this point and suggested next steps are included on the AHRQ website. Nursing Home CAHPS (NHCAHPS): This survey is jointly supported by CMS and AHRQ. A draft questionnaire is being cognitively tested with nursing home residents with varying levels of cognitive capacity to assess how well they can respond to alternate sets of response options. Also a preliminary assessment of sampling strategies for use in nursing homes for short stay and long stay residents has been prepared. A description of the work completed and planned for the current effort is included on the AHRQ website. The goal for work under this new CAHPS project will be to prepare a fully field tested instrument; sampling strategy and data collection protocol. Applicants are urged to offer approaches to achieving this goal by building on the current work. CAHPS for people with mobility impairments (CAHPS-PWMI): This project is still at the conceptual stage of development. The only activity that has been performed thus far is the convening of an expert group to look at the issues and challenges related to development of a survey for PWMI. A summary of the April 2001 meeting of this panel can be obtained through the AHRQ website. In the development and testing of CAHPS-PWMI products, grantees will collaborate with staff from the National Rehabilitation Hospital Center for Health and Disability Research (NRHCHDR), staff from the National Institute for Disability and Rehabilitation Research (NIDRR) and members of an expert panel convened by AHRQ, NRHCHDR and NIDRR. Development of new CAHPS products Development and testing of PPO supplemental items: Phase 1 Development and testing of individual provider level CAHPS: Phase 1 Translation of questionnaires and reporting templates into other languages: Phase 1 Research on cultural comparability: Ongoing throughout phases 1 and 2 New data collection technologies: Ongoing throughout phases 1 and 2 Development of OMB clearance packages: Ongoing throughout phases 1 and 2 Potential new CAHPS products include (1) development and testing of questionnaires, screeners, and supplemental questions; (2) development and testing of data collection technologies; and (3) development and testing of reporting templates. Below we describe the types of products that may be developed though the budget supporting CAHPS II will not permit development of all of them. Products will be selected after awards are made based on recommendations from the grantees, AHRQ assessment of need for the product, feedback from the CAHPS Steering Committee, expertise within grantee teams, and other sources of information. Products described below are not listed in priority order. o Questionnaire development and testing: --Perform complex literature searches and reviews (along with ability to track down relevant unpublished or in-progress studies); --Conduct feasibility studies to determine whether prospective questionnaires are desired by consumers in addition to identify the challenges of collecting the data in a cost effective manner; For example, consumers tell us that they are most interested in CAHPS related quality of care information for individual providers. Challenges exist in being able to collect these data in a cost effective manner. Applicants should propose studies which will respond to these challenges. The study should also address data collection issues to ensure that survey results can be examined at the individual provider, group practice, and health plan levels. If the results of the feasibility support the development of the questionnaire then the actual instrument may be developed and tested. --Develop questionnaires for consumer assessment of health plans, services and providers. This includes strategies for specification of content areas; preparation of psychometrically equivalent self- and telephone-administered survey versions; During CAHPS II, CAHPS supplement items for PPOs will be developed. CAHPS is designed to be applicable across different types of health care plans. However, it is desirable to pursue the development of supplemental items to address the specific requirements for assessing PPOs. --Develop and test screener questionnaires to identify special populations such as mobility-impaired individuals; --Develop and test sampling plans for testing purposes; --Develop analytic plans for testing data; --Perform psychometric testing of questionnaires; and, --Translate questionnaire into languages other than Spanish; Applicants may also need to re-evaluate the current Spanish language version to assess what changes, if any, need to be made in the translation to accommodate differences in Spanish dialects and make those changes. --Evaluate the cultural comparability of questionnaires; and, --Develop OMB clearance packages. o Data collection technologies: --Explore the feasibility of collecting data via alternate methods such as by the internet, email, and other technologies. There are a number of challenges to collecting data in a way that will be cost effective and keep respondent burden at a reasonable level. Applicants should offer studies to investigate ways in which technologies such as the internet might be used to meet the challenges. o Report template development and testing: Develop and conduct five year program of research for CAHPS reports. This would require grantees to: Identify and review the relevant literature, which may include health education, consumer choice, social psychology, decision-making research, marketing research, research on the design of electronic reports and other areas. The goal for this activity is to identify what we know and need to know about the characteristics of decision-makers, format and other features of print and non-print materials, decision situations and other areas that affect the way people make health care choices. Identify and obtain information about projects that are still in progress or would not be in the published literature. Locate consumer choice projects for which results may not have been published or which may still be in progress. Review these projects for relevant information and continue to check on those that are incomplete. Web sites such as http://www.talkingquality.gov may be helpful for this activity. Formulate research questions and development research plan. Based on this information, identify variables that affect consumer choice and develop research questions related to these variables. The reports team should use the development of new reporting templates (listed below) as opportunities to test research questions. Since some research questions may be more appropriate for study in a laboratory setting, the team will propose and implement lab studies as well. Plan how report development projects can fit into this framework Keep in mind that some questions may best be explored through lab studies Prepare final report in year five which details which questions were tested, what the results were, what we now know about consumer use of this information, what we have left to learn. Develop and test reporting templates based on: Group practice level CAHPS, (one report for providers, one for consumers) Individual provider level CAHPS, (one report for providers, one for consumers; if the feasibility study supports this) CAHPS for PWMI Supplemental PPO items CAHPS for children with special health care needs Spanish language CAHPS CAHPS in languages in which questionnaires are translated (to be determined by the Steering Committee) This task includes preparation of material for the CSRK, including the development of complete instructions for analyzing data, developing composites and inserting the data into reporting templates. All reporting templates will need to be developed in electronic and print formats. Develop and test reporting templates in different languages. Develop and test reporting templates for purchasers from small or medium- sized organizations or purchasing coalitions. In many cases, the target audience is the benefits manager or the broker contracted to evaluate health plans for employee benefits purposes. About 50% of Americans with health insurance can choose among two or more health plans. For others, choice among plans occurs at the level of the purchasing coalition, benefits manager or other entity that makes the purchasing decision. People who make these decisions, especially those from small or medium sized organizations, may not fully understand health plan quality information and so may not be able to use this information to inform their selection decision. Additionally, value- based purchasing requires both cost and quality data. Cost data are difficult to integrate into consumer reporting templates since this information is specific to the purchaser. However, purchasers would significantly benefit from having both cost and quality information integrated in a report. Therefore, cost information could be integrated into this reporting template. Develop templates for reporting CAHPS data for various audiences such as consumers, patients, health care benefits purchasers, different types of health care providers, health care quality improvement specialists, accreditation specialists, and researchers; Reporting development and testing includes: --Perform cognitive testing of questionnaires and reports for users; --Recruit people who have poor cognitive skills (rather than relying exclusively on convenience samples) for cognitive testing purposes, --Conduct usability testing on products such as reports, booklets of frequently asked questions and future editions of the CAHPS Survey and Reporting Kit; --Use graphic design and information design principles in the development of CAHPS products to let the people [who use them], working in their own physical, social and cultural environments, accomplish their goals and tasks effectively and efficiently (Hackos and Redish, 1998); Prepare reports based on discussions that recommend different approaches for the development of reporting tools and strategies for CAHPS templates. For example, applicants may consider gathering examples of existing documents, reports in other than paper media and decision tools the purpose of which is to provide health quality information to consumers, patients, and providers and developing a list of experts from which to convene a group (or groups) to review, discuss and critique a thoughtfully-chosen subset of these documents. These experts should represent diverse disciplines and points of view, including information design, graphic arts, web page design, health communications, decision theory, marketing, social marketing, health education and campaign development, plain language experts and people who prepare materials for those with poor communication or reading skills. It may help to include people who have developed tools in areas other than health quality; for example, those who have developed systems to explain retirement options and related tax consequences to employees; those who have designed information that explains the pro’s and con’s of clinical trials participation to candidates; those who have designed successful user manuals and/or on-line support for the operation of mechanical or technical equipment. The second part of the reporting task is to develop templates in the context of a carefully-considered program of research that will advance our knowledge about effective ways of preparing and disseminating usable quality reports for patients, purchasers (consumers, benefits managers and others), health care providers and other audiences. Though we continue to see the publication of individual studies which are well-planned and executed, research on consumer choice as a whole is not organized or systematic and thus has not proceeded as efficiently or effectively as it could. A planned program of research which identifies a series of critical research questions and builds on their results over a five year time span could improve this situation. This research program will be implemented over the five year span of the project. The reports team will revise it annually to incorporate findings from the previous year and feedback from the Steering Committee and to include development of different templates as we learn of the need for them. At the end of CAHPS II, this program of research should result in systematic advances in what we know about purchaser and provider use of health quality information and should make significant contributions to that literature. Quality improvement studies. CAHPS began with a primary goal of providing high-quality information which consumers and purchasers would use to support their health plan selection decisions. As CAHPS has evolved, there has been growing interest in exploring the usefulness of CAHPS results for designing strategies to improve the quality of health care from the consumer’s or patient’s perspective. Grantees will design and conduct studies to investigate how CAHPS can be used for quality improvement purposes. As an example, consider the CAHPS group practice level questionnaire. As discussed above, this questionnaire will require the development of two reporting products: one for providers (to summarize survey results and identify areas for improvement) and one for consumers (to summarize survey results so that consumers can identify group practices that will best meet their needs). Even if providers had these data, they may not understand how to make the leap from identification of problem areas (from their patients point of view) to implementation of improvements. A CAHPS QI study could work with these providers to: --Understand how providers would like for these data to be reported to them, --Translate study findings into recommendations for improvement or changes in practice behaviors, --Develop strategies and procedures designed to help change the identified behaviors, --Help implement these change strategies, --Provide mid-course corrections to the strategies if necessary, --Develop a process evaluation system to confirm that change strategies were correctly implemented, --Develop an outcome evaluation to determine if the changed behaviors had the expected impact, and --Determine how and when to communicate to patients that concerns they conveyed through the survey were recognized and that changes were made. As this sample task list implies, grantees may need to develop different and new CAHPS products to develop and implement a QI study. All CAHPS questionnaires, including the one in development to assess care given at the group-practice level, were constructed to enable comparisons across different plans in different settings. An intended and positive consequence of this is that the surveys produce data that allow comparisons between a PPO in California and a fee-for-service plan in Virginia. A negative consequence (at least for the purpose of quality improvement) is that these data reflect views of the plan (or health care services obtained through the plan) from 50,000 feet rather than at ground-level, i.e., that the results are not clearly linked with reproducible steps for improvement. But the ability to develop an effective QI plan depends on specific, detailed ground-level information about the quality of services delivered. This suggests that, though items from CAHPS surveys can serve as red flags for areas that need to be improved, it is likely that the developer of a QI program would need to collect additional qualitative or quantitative data to pinpoint specific behaviors that need to be changed. For example, if consumer ratings on the CAHPS group-provider level instrument identify a group practice with low scores in the Communication/Interpersonal Care domain, a researcher would need to know what specific provider behaviors led to this assessment. Did the physician not explain the purpose of ordered tests or test results well enough? Was the physician not sufficiently aware of the health issues that most concern the patient? How did this affect treatment? For responses to these and similar questions, the researcher may need to develop a qualitative data collection protocol such as focus groups or individual interviews. Once the researcher has a good understanding of behaviors that prompted low ratings in this domain, he can then present the information to the providers in that group practice and work with them to develop strategies for change. Like all CAHPS products, all interview protocols, focus group discussion guides and other such materials developed for QI purposes will be available to users free of charge. In CAHPS II, grantees will design and implement quality improvement (QI) studies based on CAHPS data. Each applicant to this RFA should propose a QI study which includes at least these elements: Use of an existing CAHPS instrument to identify an area which requires improvement; Development and use of additional qualitative or quantitative data collection tools, if necessary; Design and implementation of an intervention; and Evaluation of the effectiveness of the intervention. The evaluation should include a readministration of the CAHPS instrument used for problem identification to see if the instrument is sensitive enough to detect changes in plan/provider behavior. GCAHPS should be completed in time for use in proposed QI studies. Therefore, applicants are encouraged to consider either the health plan or group level for proposed studies. Additionally, applicants may also want to consider collaborations with Peer Review Organizations to conduct these studies. --Identify and establish partnerships with appropriate sites for QI studies; --Develop strategies for using CAHPS data for quality improvement purposes. This may involve development and testing of additional qualitative or quantitative instruments; --Develop strategies for translating these data into recommendations for improvement tailored to the specific target audiences; --Develop programs for assisting plans and care providers in using CAHPS data to improve the quality of their programs and services; Evaluation studies The development of CAHPS products will require both process and outcome evaluation studies. Grantees will perform two types of process evaluation studies: The first (sometimes called formative evaluation ) requires thorough testing of materials with the target audience while materials are still in draft status. Materials include report segments, reports in their entirety, questionnaire items and draft questionnaires, elements of the Survey and Reporting Kit, quality improvement plans and strategies and other materials. The second type of process evaluation refers to obtaining user feedback about how final versions of CAHPS products and procedures work in their intended settings. We are particularly interested in process evaluations of new products and procedures but need to collect additional data for existing products (like the Survey and Reporting Kit) so that we can continue to make data-based improvements to them. In general, outcome evaluation seeks to answer the question, Did the intervention achieve the expected change? In CAHPS II, applicants will need to design and implement evaluations that demonstrate the effectiveness of: a) QI interventions based on areas of concern identified by patients through a CAHPS instrument and b) the effectiveness of CAHPS report in supporting consumer choice of health plans or providers appropriate for their needs and circumstances. --Specify appropriate evaluation questions, identify appropriate outcome measures and appropriate respondents for each question, and specify times at which these questions should be asked. --Develop an appropriate evaluation design. The design should: a) eliminate as many threats to validity as possible; and b) result in a clear demonstration of usefulness of the data-based reports in assisting consumer decision-making in an open enrollment situation, as well as identification of factors related to product effectiveness or ineffectiveness. --Identify and use appropriate qualitative and quantitative data collection instruments. --Analyze data appropriately. Applicants should pay particular attention to strategies for analysis of qualitative data, as well as the issue of conveying information from quantitative evaluation analyses in a manner that is understandable by and useful to personnel at sites implementing the survey protocols. --Develop recommendations based on evaluation data that can be used to modify CAHPS projects or procedures. One component of the CAHPS evaluation plan should be the development of clearly written, user-friendly recommendations designed to promote data-based changes to products and procedures. -Identify and develop partnerships with sites appropriate for evaluation studies. Examples of the kinds of questions that may be explored through this research include the following: We know that people need more than statistics to support their selection of a health plan or provider. To make proper use of comparison data, people need to understand critical differences among health care delivery systems, the importance of a quality-based decision, and how to organize and prioritize diverse pieces of information for each of what may be several alternatives. How do we produce a report that covers all these bases but is brief enough to appeal to users who will spend only minutes reviewing the data and making their decision? A related issue concerns people’s diverse needs for information. Is it possible to design a single report that layers information to accommodate people who want only a listing of the top three plans available to them as well as people who want individual item scores for each variable? About 50% of Americans who have health insurance have the choice of two or more plans. Benefits managers, other institutional purchasers, and brokers contracted by employers also often need to evaluate and choose among several plans. A recent study by Iyengar and Lepper (2000) shows that increasing the number of alternatives among which one can choose can easily overwhelm the chooser and produce the opposite of the desired behavior. How many health plans can consumers or purchasers evaluate before they feel overwhelmed? How can we structure materials and the decision-making task to avoid overwhelming users? How do we design non-print modes of reporting (video, interactive video, web- based reports) that are low cost and relatively easy to install and maintain? How can these alternative materials be disseminated or made accessible to users? Consumers want information about health care quality and how plans and providers compare to one another but are often intimidated by displays of statistical data. How can we present quantitative data in a manner that allows reliable and valid comparisons but is also as easy to understand and user-friendly as narrative reports? People with extremely poor cognitive skills or those who have mobility, cognitive or sensory impairments may be unable to complete paper-and-pencil questionnaires or to read or understand data in typical reports. How do we produce reports that are useful to these people? Are high users of health care services (e.g. children with special health care needs, people with mobility impairments) likely to use quality reports in different ways than people who are not high users? If so, how should we design report templates for these audiences? What are the best modes of dissemination for each of our target audiences? Recent research has shown that quality reports that emphasize the opportunity to protect oneself from risk are more compelling to purchasers than those that emphasize potential gain or benefit (Hibbard, 2000). Is this finding solid enough that we should incorporate risk-based messages into the design of CAHPS reporting templates? How can we develop risk-based messages that motivate consumers but do not alienate the plans whose data we are reporting? Are there other ways of reframing key messages in quality reports that will increase the likelihood that purchasers will pay attention to and use the information? Development of experiments that respond to these research questions as well as development of quality reports that consumers will read and find useful are tasks that require multiple types of expertise. For example, these tasks require researchers who have: the methodological expertise to structure rigorous experiments both in laboratory and real world settings; knowledge of decision theory especially as it relates to consumer decisions; experience applying social marketing principles to the design of materials for consumers, patients and health care providers; experience with graphic design; knowledge of research which shows how existing quality reports have succeeded and failed; experience with design of electronic reports; expertise in document design and design of materials in plain language; experience with the design and evaluation of health messages and health education campaigns. In their applications, applicants should identify the expertise required to support their approach to report development and testing and include people experienced in these areas as part of their project team. 3. Study Organization Each grantee must propose a Principal Investigator as well as a co-Principal Investigator who will be responsible for all aspects of project management when the lead person is unavailable. These individuals should coordinate their schedules so that one or the other is always available. The Principal Investigator, or the person she designates, serves as the official voice of that grantee team. Communication from the grantee team to the AHRQ Project Officer should come from the Principal Investigator rather than from multiple members of that team. Applicants should propose staff with a range of experience and skill levels. Our goal is to avoid the situation where extremely qualified or senior level staff are performing tasks that a person with less experience could perform just as well. Grantees should also propose a lead staff person (and alternate) for each of four cross-grantee teams (instrumentation, reports, quality improvement studies and evaluation studies). During the course of the project, other cross-grantee teams may be created. Cross-grantee teams are composed of members of each grantee team organized to perform defined tasks. During CAHPS II, we will require that one staff person from the SUN contract participate in each cross-grantee team to prepare and publish call notes and sometimes to participate in the content of the work. In CAHPS I, an instrumentation team was organized, as well as teams for reports, evaluation, survey and reporting kit development, user group meeting agenda development and other purposes. The length of time cross-grantee teams are needed is dependent upon the work being performed by the team. For example, the instrumentation and reports teams of CAHPS I were formed early in the project and continued to meet (by conference call and in-person) throughout its life. Other groups are formed for shorter term goals (e.g., development of an agenda for a user group meeting) and disband when those goals have been met. Sometimes certain tasks can be performed most efficiently through the development of sub-groups. For example, a sub-group of the survey and report kit team was developed to proof-read and edit the final version. The Objectives and Methods sections contains information about the substance of work for four of the cross-grantee teams. Some of the long-term cross grantee teams (such as instrumentation and reports) have conveners. The convener moderates conference calls and meetings; works with the designated SUN staff person to arrange the call, publish a call agenda and meeting materials on the CAHPS team web site; helps the team develop a long-term agenda including a task list and time line; reminds the team of impending due dates; and meets with the Project Officers on a regular basis to discuss progress toward goals, barriers and how to resolve them. For critical periods during CAHPS I, the Principal Investigators met weekly by phone and sometimes in person with the AHRQ project officers, representatives from the SUN contractor, team conveners and other people as required. There will be two major components to the organizational structure of the CAHPS II project the CAHPS Steering Committee and the CAHPS Grantee Advisory Committee. Members of the CAHPS Steering Committee will be: the AHRQ Project Officers, at least one representative from each of the organizations which partner with AHRQ to fund the CAHPS grants, the CAHPS Prinicpal Investigators and their alternates. The project director of the Survey User Network contract will attend these meetings to provide information about CAHPS user needs but will not have an official voice in determining direction of the work of the cooperative agreements. At this time, CMS is expected to be a CAHPS funding partner. Other funding partners may be identified in the future. Additional members will be added as necessary as decided upon by the existing Committee members. The Steering Committee will set the overall direction of the CAHPS project and decide on project priorities from year to year. The Committee will establish a five year plan with broad goals at the beginning of the project. Each year it will review progress towards these goals, review emerging needs for new or different CAHPS products and adjust project direction if necessary. More detailed information regarding grantees roles on the Steering Committee and in setting the project direction can be found in the Process for Setting Project Direction subsection under the Special Requirements section below. The CAHPS Grantee Advisory Committee will be comprised of representatives of the key stakeholders affected by the implementation of CAHPS products. These might include representatives of consumers/patients; health care providers, purchasers; health plans; accrediting organizations and others. The role of the Grantee Advisory Committee will be to advise the CAHPS grantees about the needs and concerns of the organizations who use CAHPS products. The grantees will select the members of this group, set meeting agendas, make all meeting arrangements, and convene the group as often as necessary. One representative from each organization on the Steering Committee are ex officio members of the Grantee Advisory Committee. Interactions among the Steering Committee, the Grantee Advisory Committee and other CAHPS teams are discussed in the Special Requirements section. 4. Budget and Related Issues As mentioned earlier in this RFA, the CAHPS Steering Committee will determine the research and product development agenda on a year-by-year basis. This approach constitutes the best strategy for meeting user needs as they evolve. But since applicants need to prepare five-year budgets which must include these as yet unspecified activities, we offer the following assumptions. -For budgeting purposes, applicants need to develop budgets devoting approximately 1/3 effort for each of the three areas listed below. o Continued development of GCAHPS and Nursing Home CAHPS (for completion of both GCAHPS and NH CAHPS over years 1 and 2) o PWMI CAHPS (over years 1 and 2) o Individual provider level CAHPS (over years 1 and 2) -Applicants can find project summaries and a listing of remaining tasks for completion of Nursing Home CAHPS and for GCAHPS at http://www.ahrq.gov/fund/grantix.htm. For the other two surveys, the CAHPS Submitter’s Kit found at the same website is a good source of information concerning critical characteristics of questionnaires that bear the CAHPS trademark. As mentioned in the Objectives, subsection above applicants should view this document as a background resource and are not required to abide by its recommendations or suggestions. -The three grantees will be working collaboratively on GCAHPS, Nursing Home CAHPS, PWMI CAHPS, and individual provider level CAHPS so each applicant should describe their ability to perform all tasks necessary to construct a valid survey from item generation through development of complete and easy- to-understand administration procedures. -Applicants should assume that the activities listed in the Timetable Section will occur during the five year time-span of this project. -Each grantee should budget $10,000 for each of years one and two for consultation with one staff person from NRHCHDR for CAHPS PWMI. -Each reporting product must be developed and tested in print and electronic form. -Reports and questionnaires must be tested with users who have poor cognitive skills as well as with those who have average cognitive skills. -One purpose of this RFA is to develop and test tools (e.g., questionnaires, reporting templates, analytic procedures) through which users can obtain assessments of health care quality from the consumer or patient point of view and report them to the appropriate audiences. An additional purpose is to stimulate research that advances knowledge in the areas of: l) consumer and patient-based health care quality assessment, 2) reporting quality information to various users, 3) use of assessment data for quality improvement purposes and 4) evaluation of quality improvement programs. The Objectives section of this RFA contains many examples of research questions that could be addressed or studies that could be performed which would advance the knowledge in one of these areas. Though applicants need not use these particular examples of research questions or studies, they should devote some part of the budget to the design and execution of research that will advance the field in one or more of these four areas. o Meetings For each year of the contract, the appropriate grantee representative(s) will be required to attend the following meetings: - weekly conference calls with Principal Investigators, their back-ups and Project Officers - cross-grantee team meetings as necessary (in-person as necessary; assume ten person trips) - one Expert Panel meeting (in person) - two Steering Committee meetings (in-person) - two Grantee Advisory Committee meetings (in-person) - one CAHPS User Group annual meeting (in-person) - one NCBD Advisory Committee meeting (in-person) - one NCBD Technical Advisory Panel (TAP) meeting - one NCBD Executive Research Committee meeting - five online workshops for CAHPS users For in-person meetings (e.g., of the Grantee Advisory Committee, cross- grantee teams, entire consortium meetings, etc.), grantees may use free of charge the AHRQ conference facilities currently at 6010 Executive Boulevard, Rockville, Maryland. Use of these facilities are subject to availability. The grantee teams will share the cost of transportation, lodging and per diem for Advisory Committee member attendance at each meeting. All other costs for these meetings (obtaining space, materials preparation, etc.) will be borne through the SUN contract. The grantee teams will share the cost of transportation, lodging and per diem for Expert Panel member attendance at each meeting. All other costs for these meetings (obtaining space service, materials preparation, etc.) will be borne through the SUN contract. The CAHPS consortium holds one meeting of CAHPS user per year. Though the SUN contractor takes the lead responsibility for this meeting, each of the grantees will contribute to planning the content of and making presentations at the meeting. Each grantee will be allocated two registrations free of charge. Additional grantee team members that attend the meeting must submit registration fees. o Interaction with the SUN contractor A member of the SUN contract team will be present for all meetings. The SUN Contractor will work with grantees to schedule meetings and to prepare materials for those meetings. Additionally, the SUN contractor will summarize all meetings. Grantees will be required to work with the SUN contractor for the following: - developing, testing, and revising the CSRK - updating other CAHPS products as necessary - answering technical questions that arise from the technical assistance to CAHPS users, if necessary - providing all meeting materials, reports, and other items to the SUN contractor in a timely manner - serving on the NCBD Advisory Committee, TAP, and Executive Research Committee - participating in five online user workshops o Reports The following reports will be required during CAHPS II: - Summary report for each new survey instrument developed describing any difficulties encountered in the course of planning, implementing and evaluating the surveys - Five year research plan for developing and testing reporting templates - Summary report for each new reporting template developed describing any difficulties encountered in the course of planning, implementing and evaluating the templates - Final report of quality improvement study - Final report for all evaluation studies conducted - Non-competitive renewal proposals at the end of each grant year - Final report at the end of year five on the advances made to reporting quality data through these cooperative agreements - Two summary reports per grant year describing progress - Timeline and table of activities for each cross-grantee project (updated as necessary as decided by the Principal Investigators and the Project - Notes and action items from annual AHRQ site visits - Any other interim or final reports agreed upon during CAHPS II 5. Timetable Phase one includes years one and two of the project; Phase two includes years three, four, and five. The activities listed in this section describe AHRQ and CMS expectations for projects to be conducted and may change based on input from members of the Steering Committee. Also, the Objectives section of this RFA contains topics that have been identified as important by CAHPS stakeholders but are not included on this list. The Steering Committee may determine that studies based on these (or other) topics are worth pursuing. Phase I Activities -GCAHPS survey -NHCAHPS survey -CAHPS-PWMI survey -PPO supplemental items -Individual provider level CAHPS (ongoing) -New data collection technologies -Translations of CAHPS into other languages (ongoing) -Development of five year research plan on reporting quality information to various audiences -Research on the cultural comparability of CAHPS (ongoing) -Spanish language CAHPS reporting template -Translation of CAHPS reporting template into one other language -OMB clearance packages (ongoing) Phase II Activities -Individual provider level CAHPS -New data collection technologies -Translations of CAHPS into other languages -Research on the cultural comparability of CAHPS -GCAHPS reporting template -CAHPS-PWMI reporting template -Translation of CAHPS reporting template into two languages -Quality improvement studies -Evaluation studies -OMB clearance packages Annual Meetings and Reports -Progress reports (Months one and seven of grant year) -Grantee Advisory Committee meeting (Months two and eight of grant year) -Steering Committee meetings (Months three and nine of grant year) -Non-competitive renewal proposal (Month eleven of grant year) -CAHPS User Group meeting (currently anticipated for Spring 2002; may be scheduled in the fall for subsequent grant years) SPECIAL REQUIREMENTS 1. Reporting requirements As soon as possible after the award date, the Principal Investigators and will meet in person with the AHRQ project staff and SUN contractor to review proposed project tasks for years one and two, make any agreed upon or necessary changes to the list, identify the subgroups required to perform various tasks, and determine which grantee will take the lead for which product (e.g., Spanish language version of the CAHPS reporting template) or project (e.g., feasibility study for an individual provider level instrument). We will also determine who the government contact person will be for each product and project as well as whether a consortium-wide, in- person meeting is required at that point or at some later point. Remaining decisions will be discussed during weekly conference calls. In order to accomplish the Agency’s mission of translating research into practice and policy, researchers must inform the Office of Health Care Information when articles from the study are accepted for publication in journals. Researchers should also discuss ideas about other dissemination and marketing efforts with the Office of Health Care Information at AHRQ so that efforts to disseminate and translate the research may be coordinated with the Agency in order to maximize the knowledge and application of the research by potential users, including clinicians, patients, health care systems and purchasers and policymakers. The work of this RFA requires close coordination with the SUN contract activity. One major jointly developed product is the CSRK, CAHPS Survey and Reporting Kit. The grantees have principle responsibility for development of the first draft text for the CSRK. The SUN contractor and the AHRQ Office of Health Communications Information (OHCI) are responsible for final formatting, editing, dissemination and actual physical production of the kit. Because so many team members from different parts of various organizations are involved in the CSRK and because CAHPS users have time-sensitive needs for information in the CSRK, failure by any single part of the team to meet agreed-upon deadlines can have disastrous consequences. So it will be critically important for designated writers on grantee teams to meet the deadlines to which they agree. The CAHPS project officers will perform annual site visits with each of the grantees. During the site visits (which usually last a day and a half), the grantees will present evidence of progress on each of the tasks for which they have responsibility; discuss problems, questions or other issues with the project officers; and work with the project officers to lay out activities for the coming year. Process for setting project direction The process for setting the project direction is as follows: --Drawing upon grantee work plans, the Steering Committee establishes the overall direction of the project with a five year plan at the beginning of the project. The Committee will meet twice during each grant year during the life of the project to establish or refine priorities for the coming year. The timeframe for these meetings can be found in the Timetable section. --All Steering Committee members will participate in discussions of project direction and the distribution of work among the grantees. Based on these discussions, the grantee representatives on the Steering Committee will be responsible for jointly proposing projects along with the distribution of work among the grantees. --Prior to the completion of each year of the grants the CAHPS grantee teams and the SUN contractor will consult with the CAHPS Grantee Advisory Committee to review project progress and secure their input into priorities for the upcoming year. --CAHPS grantee teams will develop proposals for the upcoming year reflecting the Steering Committee’s priorities and input from the Grantee Advisory Committee. These proposals may suggest modifications to the these priorities. --The Steering Committee will review proposals and decide which activities and products will be pursued. --The grantee teams will modify proposals as necessary to address Steering Committee’s priorities. See the section below entitled AHRQ Staff Responsibilities for circumstances under which AHRQ may terminate or curtail the cooperative agreement activities. Terms and Conditions of Award These special Terms of Award are in addition to and not in lieu of otherwise applicable OMB administrative guidelines, HHS grant administration regulations at 45 CFR Parts 74 and 92, 42 CFR Part 67 Subpart A, and other HHS, PHS grants administration policy statements. Applicants should be familiar with the Agency’s grant regulations, 42 CFR Part 67 Subpart A, and particularly sections 67.18-67.22. Consistent with the U18 concept defined under Mechanism of Support, the dominant role and prime responsibility for the activity resides with the awardee(s) for the project as a whole, although specific tasks and activities in carrying out the studies will be shared among the awardees and the AHRQ Program Official(s). 1. Awardee Rights and Responsibilities o General Awardees will have primary and lead responsibilities for the project as a whole including: Survey instrument development and appropriate testing; Feasibility studies as necessary to support the development of new CAHPS products; _The grantee will write up notes and follow-up actions discussed at this meeting for submission to the Project Officers; Report template development and testing; Translation of instruments and report templates into other languages; Evaluation of the cultural comparability of questionnaires; Design, implementation, and analysis of quality improvements studies, including evaluation of the impact of intervention(s); Design, implementation, and analysis of evaluation studies; Development of relationships with testing sites, QI study sites, and evaluation sites; Formation of the Grantee Advisory Committee, including selection of members, convening meetings, planning meeting agendas and documenting activities; Provision of AHRQ access to all data generated under this award as it is collected; and Cooperation with other key parties in these cooperative agreements, including but not limited to the SUN contractor; other CAHPS grantees; and AHRQ Project Officers and other AHRQ staff. o Meetings For each year of the contract, the appropriate grantee representative(s) will be required to attend the following meetings: - weekly conference calls with Principal Investigators, their back-ups and Project Officers - cross-grantee team meetings as necessary (in-person as necessary; assume ten person trips) - one Expert Panel meeting (in person) - two Steering Committee meetings (in-person) - two Grantee Advisory Committee meetings (in-person) - one CAHPS User Group annual meeting (in-person) - one NCBD Advisory Committee meeting (in-person) - one NCBD Technical Advisory Panel (TAP) meeting - one NCBD Executive Research Committee meeting - five online workshops for CAHPS users Grantees will need to cover all meeting costs (tape-recording, etc.) for in- person meetings with the exception of the CAHPS User Meeting and the Advisory Committee meetings. These costs include facility costs if AHRQ conference facilities are not used. The grantee teams will share the cost of transportation, lodging and per diem for Advisory Committee member attendance at each meeting. All other costs for these meetings (obtaining space, materials preparation, etc.) will be borne through the SUN contract. The grantee teams will share the cost of transportation, lodging and per diem for Expert Panel member attendance at each meeting. All other costs for these meetings (obtaining space, materials preparation, etc.) will be borne through the SUN contract. The CAHPS consortium usually holds one meetings of CAHPS users per year. Though the SUN contractor takes the lead responsibility for this meeting, each of the grantees will contribute to planning the content of and making presentations at the meeting. Each grantee will be allocated two registrations free of charge. Additional grantee team members that attend the meeting must submit registration fees. o Interaction with the SUN contractor Grantees will be required to work with the SUN contractor for the following: - developing, testing, and revising the CSRK - updating other CAHPS products as necessary - answering technical questions that arise from the technical assistance to CAHPS users, if necessary - providing all meeting materials, reports, and other items to the SUN contractor in a timely manner - serving on the NCBD Advisory Committee, TAP, and Executive Research Committee - participating in five online user workshops o Reports The following reports will be required during CAHPS II: - Summary report for each new survey instrument developed describing any difficulties encountered in the course of planning, implementing and evaluating the surveys - Five year research plan for developing and testing reporting templates - Summary report for each new reporting template developed describing any difficulties encountered in the course of planning, implementing and evaluating the templates - Final report of quality improvement study - Final report for all evaluation studies conducted - Non-competitive renewal proposals in Month eleven of each grant year - Final report at the end of year five on the advances made to reporting quality data through these cooperative agreements - Two summary reports per grant year describing progress (Months one and seven of grant year) - Timeline and table of activities for each cross-grantee project (updated as necessary as decided by the Principal Investigators and the Project - Notes and action items from annual AHRQ site visits - Any other interim or final reports agreed upon during CAHPS II 2. AHRQ Staff Responsibilities The AHRQ Project Officer and other AHRQ staff will have substantial scientific and programmatic involvement during the conduct of this activity through technical assistance, advice, and coordination beyond the normal program stewardship for grants. Collaboration on the development of the testing plan, evaluation design and survey protocol will occur after the award is made. Specifically, AHRQ's role in the cooperative agreement is to provide technical assistance, advice, and support to the PI and to ensure that all related AHRQ-supported projects complement one another. AHRQ will: --Participate in all key project decisions, including but not limited to: identification of focal consumer groups, assessment of consumer information needs, selection of items/question modules to be tested, identification and resolution of methodological issues; --Work with the grantee to identify and negotiate with sites for quality improvement and evaluation studies, with the AHRQ project officer retaining right of final approval; --Establish collaboration with and obtain project-related information from other Federal agencies and programs; --Disseminate tested question modules and other research findings as they become available; and --Participate in or coordinate strategy sessions with the grantees on an as- needed basis. --AHRQ will require prior written approval for the addition or deletion of a participating or collaborating institution, site, or other organizational component. These special Terms of Award are in addition to and not in lieu of otherwise applicable PHS grants policies and Federal regulations. Arbitration Any disagreement that may arise with respect to scientific or programmatic matters (within the scope of the award), between award recipients and AHRQ may be brought to arbitration. An arbitration panel will be composed of three members one selected by the Steering Committee (with the AHRQ member not voting) or by the individual awardee in the event of an individual disagreement, a second member selected by AHRQ, and the third member selected by the two prior selected members. This special arbitration procedure in no way affects the awardee’s right to appeal an adverse action that is otherwise appealable in accordance with the PHS regulations at 42 CFR Part 50, Subpart D and HHS regulation at 45 CFR Part 16. Data Privacy Pursuant to section 924(c) of the Public Health Service Act (42 USC 299c- 3(c)), information obtained in the course of any AHRQ-supported study about an identifiable individual or entity must be treated as confidential in accordance with any promises made or implied regarding the possible uses and purposes of the data collection. In the Human Subjects section of the application, applicants must describe procedures for ensuring the confidentiality of such identifying information. The description of the procedures should include a discussion of who will be permitted access to the information, both raw data and machine readable files, and how personal identifiers and other identifying or identifiable data will be restricted and safeguarded. Plans for compliance with the Privacy Rule as 45 CFR Part 164, if applicable, should be set forth. The grantee should ensure that computer systems containing confidential data have a level and scope of security that equals or exceeds those established by the Office of Management and Budget (OMB) in OMB Circular No. A-130, Appendix III - Security of Federal Automated Information Systems. The National Institute of Standards and Technology (NIST) has published several implementation guides for this circular. They are: An Introduction to Computer Security: The NIST Handbook; Generally Accepted Principals and Practices for Securing Information Technology Systems; and Guide for Developing Security Plans for Information Technology Systems. The circular and guides are available on the web at http://csrc.nist.gov/publications/nistpubs/800-12/handbook.pdf. The application of these confidentiality and security standards to subcontractors and vendors, if any, should be addressed. These and other requirements applicable to AHRQ grantees as well as important legal rights are set out or referenced in the Agency’s grants regulations at 42 CFR 67, Subpart A. (Available in libraries and from the GPO’s website http://www.access.gpo.gov/nara/cfr/index.html). Rights in Data AHRQ grantees may copyright or seek patents, as appropriate, for final and interim products and materials, material they developed individually: including, but not limited to, methodological tools, measures, software with documentation, literature searches, and analyses, which are developed in whole or in part with AHRQ funds. Such copyrights and patents will be subject to a worldwide irrevocable Federal government license to use and permit others to use these products and materials for government purposes. In accordance with its legislative dissemination mandate, and subject to statutory confidentiality protections, AHRQ purposes may include, subject to statutory confidentiality protections, making research materials, data bases, results, and algorithms available for verification or replication by other researchers; and subject to AHRQ budget constraints, final products may be made available to the health care community and the public by AHRQ or its agents, if such distribution would significantly increase access to a product and thereby produce public health benefits. Ordinarily, to accomplish distribution, AHRQ publicizes research findings through the media and other appropriate channels in coordination with AHRQ’s Office of Health Care Information but relies on grantees to publish research results in peer- reviewed journals and to market grant-supported products. INCLUSION OF WOMEN, MINORITIES, AND CHILDREN IN RESEARCH STUDY POPULATIONS It is the policy of AHRQ that women and members of minority groups be included in all AHRQ-supported research projects involving human subjects, unless a clear and compelling rationale and justification are provided that inclusion is inappropriate with respect to the health of the subjects or the purpose of the research. All investigators proposing research involving human subjects should read the UPDATED "NIH Guidelines on the Inclusion of Women and Minorities as Subjects in Clinical Research," published in the NIH Guide for Grants and Contracts on August 2, 2000 (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-00-048.html). A complete copy of the updated Guidelines is available at http://grants.nih.gov/grants/funding/women_min/guidelines_update.htm. To the extent possible, AHRQ requires adherence to these NIH Guidelines. Investigators may obtain copies from the above sources or from the AHRQ Publications Clearinghouse, listed under INQUIRIES, or from the NIH Guide Web site http://grants.nih.gov/grants/guide/index.html. AHRQ also encourages investigators to consider including children in study populations, as appropriate. AHRQ Program staff may also provide additional information concerning these policies (see INQUIRIES). LETTER OF INTENT Prospective applicants are asked to submit, by September 24, 2001, a letter of intent that includes a descriptive title of the proposed research, name, address, and telephone number of the Principal Investigator, the identities of other key personnel and participating institutions, and the number and title of the RFA in response to which the application may be submitted. Although a letter of intent is not required, is not binding, and does not enter into consideration of any subsequent applications, the information allows AHRQ staff to estimate the potential review workload and to avoid conflict of interest in the review. AHRQ will not provide responses to letters of intent. The Letter of Intent is to be sent to Maria Wey (see address under INQUIRIES). APPLICATION PROCEDURES The PHS 398 research grant application instructions and forms (rev. 5/2001) available at http://grants.nih.gov/grants/funding/phs398/phs398.html are to be used in applying for these grants. This version of the PHS 398 is available in an interactive, searchable PDF format. Although applicants are strongly encouraged to begin using the 5/2001 revision of the PHS 398 as soon as possible, the NIH will continue to accept applications prepared using the 4/1998 revision until January 9, 2002. Beginning January 10, 2002, however, the NIH will return applications that are not submitted on the 5/2001 version. For further assistance contact GrantsInfo, Telephone 301/710-0267, Email: GrantsInfo@nih.gov. State and local government applicants may use PHS 5161-1, Application for Federal Assistance (rev. 5/96), and follow those requirements for copy submission. AHRQ is not using the Modular Grant Application and Award process. Applicants for funding from AHRQ should ignore application instructions concerning the Modular Grant Application and Award process, and prepare applications according to instructions provided in form PHS 398. Applications submitted in the modular format will be returned without review. Beginning with applications submitted for the February 1, 2001 receipt date, Institutional Review Board (IRB) approval of human subjects is not required prior to peer review of an application. The AHRQ Revised Policy for IRB Review of Human Subjects Protocols in Grant Applications was published in the NIH Guide on September 27, 2000. (http://grants.nih.gov/grants/guide/notice-files/NOT-HS-00-003.html). Applications kits are available at most institutional offices of sponsored research. They may also be obtained from the Division of Extramural Outreach and Information Resources, National Institutes of Health, 6701 Rockledge Drive, MSC 7910, Bethesda, MD 20892-7910, telephone (301) 710-0267, E- mail:grantsInfo@nih.gov. AHRQ applicants are encouraged to obtain application materials from the AHRQ Publications Clearinghouse (see INQUIRIES). The RFA label available in the PHS 398 (rev. 5/2001) application form must be affixed to the bottom of the face page of the application. Type the RFA number on the label. Failure to use this label could result in delayed processing of the application such that it may not reach the review committee in time for review. In addition, the RFA title and number must be typed on line 2 of the face page of the application form and the YES box must be marked. The RFA label is also available at: http://grants.nih.gov/grants/funding/phs398/label-bk.pdf. A complete, and signed, typewritten original of the application, including the Checklist and three signed photocopies, must be submitted in one package to: Center for Scientific Review National Institutes of Health 6701 Rockledge Drive, Room 1040 - MSC 7710 Bethesda, MD 20892-7710 (20817 for express/courier service) At the time of submission, two additional copies of the application, labeled Advanced Copy (s) must also be sent to: Maria Wey Center for Quality Improvement and Patient Safety Agency for Healthcare Research and Quality 6011 Executive Blvd., Suite 200 Rockville, MD 20852 Telephone: (301) 594-6626 FAX: (301) 594-2155 E-mail Address: mwey@ahrq.gov Applications must be received by November 14, 2001. An application received after the deadline may be acceptable if it carries a legible proof-of-mailing date, assigned by the carrier, and the proof-of-mailing is not later than 1 week prior to the deadline date. If an application is received after that date, it will be returned to the applicant without review. Application Preparation (for Using CMS Data) For applications that propose to use Medicare or Medicaid data that are individually identifiable, applicants should state explicitly in the Research Design and Methods section of the Research Plan (form 398) the specific files, time periods, and cohorts proposed for the research. In consultation with Center for Medicare and Medicaid Services (CMS), previously called Health Care Financing Administration (HCFA), AHRQ will use this information to develop a cost estimate for obtaining the data. This estimate will be included in the estimated total cost of the grant at the time funding decisions are made. Applicants should be aware that for individually identifiable Medicare and Medicaid data, Principal Investigators and their grantee institutions will be required to enter into a Data Use Agreement (DUA) with CMS to protect the confidentiality of data in accordance with AHRQ’s confidentiality statue, 42 USC 299c3(c), the Privacy rules at 45 CFR Part 164, if applicable, and standards set out in OMB Circular A-130, Appendix III Security of Federal Automated Information Systems. The use of the data will be restricted to the purposes and time period specified in the DUA. At the end of this time period, the grantee will be required to return the data to CMS or certify that the data have been destroyed. Unless AHRQ is able to negotiate exceptional arrangements, for the sole purpose of assuring that data confidentiality is maintained, included in the DUA is the requirement that the User agrees to submit to CMS, a copy of all findings within 30 days of making such findings. The user further agrees not to submit these findings to any third party (including but not limited to any manuscript to be submitted for publication) until receiving CMS’s approval to do so. In developing research plans, applicants should allow time for refining, approving, and processing any CMS data requests. Requests may take 6 months from the time they are submitted to complete. Applications proposing to contact beneficiaries or their providers require the approval of the CMS Director and may require meeting(s) with CMS staff. CMS data are provided on IBM mainframe tapes using the record and data formats commonly employed on these computers. Applicants should either have the capability to process these tapes and formats or plan to make arrangements to securely convert them to other media and formats. Questions regarding CMS data should be directed to the AHRQ program official listed under INQUIRIES. In carrying out its stewardship of research programs, the AHRQ, at some point in the future, may begin requesting information essential to an assessment of the effectiveness of Agency research programs. Accordingly, grant recipients are hereby notified that they may be contacted after the completion of awards for periodic updates on publications resulting from AHRQ grant awards, and other information helpful in evaluating the impact of AHRQ-sponsored research. AHRQ expects grant recipients to keep the Agency informed of publications as well as the known uses and impact of their Agency-sponsored research. Applicants are to agree to notify AHRQ immediately when a manuscript based on research supported by the grant is accepted for publication, and to provide the expected date of publication as soon as it is known, regardless of whether or not the grant award is still active. To receive an award, applicants must agree to submit an original and 2 copies of an abstract, executive summary, and full report of the research results in the format prescribed by AHRQ no later than 90 days after the end of the project period. The executive summary should be sent at the same time on a computer disk which specifies on the label the format used (WP5.1 or WP6.0 is preferable). CONFERENCE FOR PROSPECTIVE APPLICANTS AHRQ plans to present a webchat to respond to questions from prospective applicants on September 20, 2001 at 4 p.m. EST. Participation not a prerequisite to applying. Applicants who are interested in participating should send a written request to Maria Wey (contact information can be found in the INQUIRIES section). For further information, contact (301) 594-6626. REVIEW CONSIDERATIONS Upon receipt, applications will be reviewed for completeness by CSR and responsiveness by AHRQ. Incomplete applications will be returned to the applicant without further consideration. Applications that are complete but are not responsive to the RFA will be returned to the applicant without further consideration. Applications that are complete and responsive to the RFA will be evaluated for scientific and technical merit by an appropriate peer review group convened by the AHRQ in accordance with the review criteria stated below. As part of the merit review, all applications will receive a written critique, and also may undergo a process in which only those applications deemed to have the highest scientific merit will be discussed and assigned a priority score. All applications will be judged on the basis of the scientific merit of the proposed project and the documented ability of the investigators to meet the research objectives of the RFA. Although the technical merit of the proposed project is important, it will not be the sole criterion for evaluation of a study. Other considerations, such as importance and timeliness of the proposed study and access to patients/data will be part of the evaluation criteria. General Review Criteria The reviewers will comment on the following aspects of the application in their written critiques in order to judge the likelihood that the proposed research will have a substantial impact on the pursuit of these goals. Each of these criteria will be addressed and considered by the reviewers in assigning the overall score weighting them as appropriate for each application. Note that the application does not need to be strong in all categories to be judged likely to have a major scientific impact and thus deserve a high priority score. For example, an investigator may propose to carry out important work that by its nature is not innovative but is essential to move a field forward. 1. Significance. If the aims of the application are achieved, how will scientific knowledge reporting quality information be advanced? What will be the effect of these studies on the concepts or methods that drive this field? 2. Approach. Are the conceptual framework, design, methods, and analyses adequately developed, well-integrated, and appropriate to the aims of the project? Are the proposed data sources appropriate and adequate? Does the applicant acknowledge potential problem areas and consider alternative tactics? 3. Innovation. Does the project employ novel concepts, approaches or method? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies? 4. Investigator. Is the investigator appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the Principal Investigator and other researchers (if any)? Is the project (or workplace) well organized? 5. Environment. Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of unique features of the scientific environment or employ useful collaborative arrangements? Is there evidence of institutional support? The initial review group will also examine: the appropriateness of proposed project budget and duration; the adequacy of plans to include both genders, children, and minorities and their subgroups as appropriate for the scientific goals of the research and plans for the recruitment and retention of subjects; the provisions for the protection of human and animal subjects; and the safety of the research environment. Special Review Criteria The special review criteria include experience and skills in consumer assessment and reporting of health plans and other providers. It is not necessary that this experience be based on previous CAHPS experience. 1. Experience with similar activities outlined in the Objectives section. 2. Experience with the measurement of consumer and patient assessments of health care plans, services and providers. 3. Experience with the needs of various audiences who may use health care information (e.g., consumers, patients, purchasers, plans, providers, payers, State and local-level health care organizations, etc.) and the ways in which they may use this information (selection of plans and providers; comparison of plan/provider performance; identification of local or multi-site quality problems; etc.). 4. Experience with all aspects of fielding a large-scale, multi-site survey, including but not limited to: formulation of a methodologically sound design; performance of validity and reliability studies on large-scale survey instruments; identification and resolution of methodological issues associated with sample selection and sample size, and with administration of surveys to the subpopulations of interest and within a variety of health care delivery settings; identification and resolution of methodological issues associated with mode of administration. 5. Experience with producing detailed survey operations manuals suitable for use by groups with minimal as well as extensive experience in fielding large- scale, multi-site surveys. 6. Experience with identifying and recruiting appropriate sites for survey instrument testing, report template testing, QI studies, and evaluation studies. 7. Experience with collaborating with, provide technical assistance or training to, and/or monitor the progress of test sites, community-based organizations, data supplying organizations and site representatives. 8. Experience with collaborating with government agencies in the performance of research studies and demonstrations. 9. Experience with developing and testing understandable and useful print and electronic reports which contain health quality data for use by a wide range of audiences (patients, consumers, providers, health plans, national, state and local policy makers, government agencies, etc.). 10. Experience in designing questionnaires and reports for people with poor cognitive skills and/or poor reading skills. 11. Experience in designing, implementing and evaluating quality improvement programs in health care settings. 12. Experience in designing and implementing process and outcome evaluations, including: specification of evaluation questions and outcome measures; use of quantitative and qualitative data collection methods; data analysis; development of data-based recommendations and use of the recommendations to improve products and procedures. 13. Experience in participating as a productive member of a multi- organization team over a lengthy performance period in the development of high-quality products and procedures for their use. AWARD CRITERIA Applications will compete for available funds with all other applications under this RFA. The following will be considered in making funding decisions: 1) quality of the proposed project as determined by peer review; 2) availability of funds; 3) program balance with respect to types insurance structure and organizational delivery system features, enrollee versus community studies, populations studied, geography including rural/urban mix, regional representation, and targeting understudied areas with unmet evaluation needs; 4) compatibility with other awardees with respect to cooperative activities; and 5) policy relevance. INQUIRIES Copies of the RFA and copies of the grant application form PHS 398 (rev. 5/2001) are available from: AHRQ Publications Clearinghouse P.O. Box 8547 Silver Spring, MD 20907-8547 Telephone: 1-800-358-9295 TDD Service: 888-586-6340 The RFA is also available on AHRQ’s Web site, http://www.AHRQ.gov, and through AHRQ InstantFAX at (301) 594-2800. To use InstantFAX, you must call from a facsimile (FAX) machine with a telephone handset. Follow the voice prompt to obtain a copy of the table of contents, which has the document order number (not the same as the RFA number). The RFA will be sent at the end of the ordering process. AHRQ InstantFAX operates 24 hours a day, 7 days a week. For comments or problems concerning AHRQ InstantFax, please call (301) 594-6344. AHRQ welcomes the opportunity to clarify any issues or questions from potential applicants who have read the RFA. Written and telephone inquiries concerning this RFA are encouraged. Direct inquiries regarding programmatic issues, including information on the inclusion of women, minorities, and children in study populations to: Maria Wey Center for Quality Improvement and Patient Safety Agency for Healthcare Research and Quality 6011 Executive Blvd., Suite 200 Rockville, MD 20852 Telephone: (301) 594-6626 FAX: (301) 594-2155 E-mail Address: mwey@ahrq.gov Direct inquiries regarding fiscal and eligibility matters to: Al Deal Grants Management Specialist Agency for Healthcare Research and Quality 2101 East Jefferson Street, Suite 601 Rockville, MD 20852 Telephone: (301) 594-1843 FAX: (301) 594-3210 Email: adeal@ahrq.gov AUTHORITY AND REGULATIONS This program is described in the Catalog of Federal Domestic Assistance, Number 93.226. Awards are made under Title IX of the Public Health Service Act (42 USC 299-299c-7) as amended by P.L. 106-129 (1999). Awards are administered under the PHS Grants Policy Statement and Federal Regulations 42 CFR 67, Subpart A, and 45 CFR Parts 74 or 92. This program is not subject to the intergovernmental review requirements of Executive Order 12372 or Health Systems Agency review. The PHS strongly encourages all grant and contract recipients to provide a smoke-free workplace and promote the non-use of all tobacco products. In addition, Public Law 103-227, the Pro-Children Act of 1994, prohibits smoking in certain facilities (or in some cases, any portion of a facility) in which regular or routine education, library, day care, health care or early childhood development services are provided to children. This is consistent with the PHS mission to protect and advance the physical and mental health of the American people. REFERENCES Website references: SUN Contract - http://www.cahps-sun.org, NCBD - ncbd.cahps.com, Talking Quality - http://www.talkingquality.gov. CAHPS Documents (available on the AHRQ website): CAHPS Survey and Reporting Kit, GCAHPS Final Report, CAHPS-PWMI Expert Panel Meeting Summary, NH-CAHPS Summary, Frequently Asked Questions about CAHPS booklet, CAHPS Submitter’s Kit, Report of Input from Stakeholders on Design of CAHPS II, SUN contract scope of work. Hackos JT, Redish JC. User and Task Analysis for Interface Design. 1998; John Wiley & Sons, Inc.:New York, NY. Hibbard JH, Harris-Kojetin L, Mullin P, Lubalin J, Garkinkel S. Increasing the impact of health plan report cards by addressing consumers' concerns. Health Aff. 2000 Sep-Oct;19(5):138-43. Iyengar SS, Lepper MR. When choice is demotivating: can one desire too much of a good thing? J Pers Soc Psychol. 2000 Dec;79(6):995-1006. Shaller DV, Sharpe PS, Rubin RD. A national action plan to meet health care quality information needs in the age of managed care. JAMA. 1998;279(16):1254-1258.

Weekly TOC for this Announcement
NIH Funding Opportunities and Notices


NIH Office of Extramural Research Logo
   Department of Health and Human Services (HHS) - Home Page Department of Health
and Human Services (HHS) 		  USA.gov - Government Made Easy
NIH... Turning Discovery Into Health®


Note: For help accessing PDF, RTF, MS Word, Excel, PowerPoint, Audio or Video files, see Help Downloading Files.