Common deficiencies of NIDRR research applications - National Institute on Disability and Rehabilitation Research

Common Deficiencies of NIDRR Research Applications

Each year since 1984, several hundred researchers and clinicians in the rehabilitation-related disciplines have submitted formal applications for consideration by the National Institute on Disability and Rehabilitation Research (NIDRR) in its field investigator initiated research program; and, each year, most of the applications (90 percent) have been rejected because of technical deficiencies. The preparation of an application for federal financial assistance is a time consuming and expensive venture, not to mention an expense to the government for processing the application, completing an intensive scientific review, providing timely and accurate notification, and scheduling investigator debriefing.

This article identifies and describes those deficiencies determined by nonfederal peer reviews during the most recent (1989) competition that diminish scientific credibility of potential findings and results (i.e., if, based on the observed research flaws, the new information generated from the proposed study would be of limited benefit to the rehabilitation field). This is an important consideration, given the limited financial resources available to NIDRR to support rehabilitation research. It is important that the most scientifically correct, relevant and useful information be generated to meet the clinical and program development needs of the field.

The History of NIDRR

The National Institute on Disability and Rehabilitation Research was created in 1978 as the National Institute on Handicapped Research by Congress in the Rehabilitation, Comprehensive Services, and Developmental Disabilities Amendments of 1978, P.L. 95-602. The institute was to be a focal point within the Executive Branch for the planning and coordination of federally supported rehabilitation research for conducting a portion of that research and for providing new and expanded resources for that research effort. NIDRR developed from the Rehabilitation Research and Demonstration program administered by the Rehabilitation Services Administration (RSA), known before 1967 as the Vocational Rehabilitation Administration.

The original research authority of the state-federal program of vocational rehabilitation was established in the 1954 amendments to the Vocational Rehabilitation Act. This program provided grants to public and private nonprofit organizations in partial support of research and demonstration projects that proposed some unique national contribution to the knowledge base of rehabilitation theory or practice.

The original purposes included:

* developing new or improved information, methods and devices for use by the disciplines in the rehabilitation of people with physical, mental or severe disabilities;

* increasing the effectiveness of existing programs and stimulating community cooperation and support; and

* providing new professional information and ideas for administrators to aid them in developing and expanding programs, services and facilities for people with disabilities.

Project applications were reviewed by four extramural study sections of the program and the National Advisory Council on Vocational Rehabilitation, which advised the RSA Commissioner in making research grant awards. The four study sections, which were established in areas of high scientific interest and activity, included medical, psychosocial, sensory, and general rehabilitation and administrative studies.

The Peer Review Process

The current NIDRR scientific-technical peer review process is based on policies contained in the Education Department Grant Administration Regulations (EDGAR). Essentially, the regulations call for the establishment of short-term, ad hoc review panels for each announced competition. At least three or more nonfederal reviewers must review and rank each application, depending on the type of research resource under consideration.

The peer review process actually begins with an announcement in the Federal Register, the official Executive Branch mode of dissemination. The announcement sets forth the rules of the research competition, including the subject or priority to be proposed, the funding category, the programmatic objectives and guidelines, review criteria to be utilized, and applicable federal regulations. The reason for this detailed statement is to provide equal access to the information and the opportunity to apply objectivity in the process with no preferential treatment to any individual applicant.

Once applications are received in the Department of Education's Application Control Center (ACC), as per the specifications of the Federal Register announcement, they are forwarded to NIDRR for processing and review. Applications must be complete when submitted as additional forms and/or supporting materials cannot be appended by any program office after receipt in ACC. After administratively reviewed and and verified in NIDRR, the applications are assigned and forwarded to an appropriate science program office. The designated program staff then inspect all applications for completeness and appropriateness and determine the number of panels and reviewer expertise needed. The next steps include the formal nomination of review panelists and their final selection. Because of time conflicts caused by professional schedules, vacation plans or illness, usually three to five times as many reviewers are nominated as can be used to assure an adequate pool is available.

The selected reviewers, whose names remain confidential, receive, read and score all assigned applications prior to the scheduled panel meeting in Washington, D.C. During the formal panel meeting, and only in the presence of an assigned NIDRR panel manager, the applications are discussed for scientific-technical merit and ranked for approval or disapproval. Discussion takes place after a certification is completed by each reviewer that no actual or potential conflict-of-interest exists with any applicant organization. Upon conclusion of the panel meeting, the panel manager checks all scores and evaluation materials to assure accuracy and completeness, including a determination that no irregularities existed. Following the panel meeting, the appropriate program office conducts an independent staff review, including formulation of recommendations for award. As a final step, recommendations from the panel and staff, including all supporting materials and documents, are reviewed by the NIDRR director and other appropriate departmental officials in a formal prefunding conference. A final selection is made by the director and a slate of approved applications is forwarded through the Assistant Secretary of Special Education and Rehabilitative Services to the Department of Education's Grants and Contracts Services (GCS) for negotiation and award.

The Study Methodology

In FY 1989, 232 field investigator initiated research applications were received in response to the Federal Register announcement of September 8, 1988. Of these, 57 (24.5 percent) were assigned to the Medical Sciences Programs Office. Following assignment to, and review by, four panels and staff, six applications (10.5 percent) were recommended for approval and subsequent funding. The remaining 51 (89.5 percent) of the applications made up the study sample for this statistical analysis.

To properly identify and categorize reviewer responses, a content analysis was completed for all 51 application review materials and discussion summaries. Thirteen categories of application deficiency of research error were identified. Each application was then codified, by reviewer response, according to the 13 problem categories. The frequencies for the top three ranked deficiencies for each application were recorded.

Analysis of Results

The 51 disapproved and unfunded applications received a total of 273 deficiencies by reviewers. Recording only the three top ranked deficiencies for each application produced 142 reviewer observations. These 51 disapproved proposals averaged 5.68 deficiencies each.

Five types of research errors accounted for 76.8 percent of all deficiencies noted. Four types of deficiency accounted for 66.9 percent of disapprovable applications. In considering the top three ranked problems per application, those errors that accounted most frequently for disapprovals included methodological errors (25 deficiencies) and inadequate control of subject variables (25 deficiencies), followed closely by incorrect or inappropriate research design[24] and poor conceptualization of the problem/approach[21]. The fifth most frequently noted deficiency was inappropriate statistical analysis[14].

When identifying the one singular deficiency per application that was most responsible for disapproval, poor conceptualization of problem/approach led with 10, followed by a tie for second place between inadequate control of variables and research design errors[7]. Inappropriate statistical analysis, ranked next with 6 deficiencies, included methodological errors[4], excessive budget request[4], duplication of supported research[4], and inadequate background of investigator[3].

The 13 categories of deficiencies were ranked by frequency as shown in the table below.

Interestingly, while specified in federal review criteria, the literature review and human subjects protection were relatively infrequently noted deficiencies. More important overall considerations to reviewers appeared to be a weak dissemination plan[6], inadequate background or experience[5] and duplication of effort[5].

Description of Research Deficiencies

To the experienced scientific investigator, most of the noted categories of deficiencies or research errors are self-explanatory (i.e., methodology, design, statistical analysis, etc.) However, to avoid misinterpretation and to facilitate a common frame of reference, the writers present a short description and explanation of those deficiencies for which all researches may not be familiar or that represent special NIDRR criteria that traditionally may not be considered in analysis of research protocols.

Those categories considered most open to misinterpretation include lack of medical supervision, weak dissemination/use plan, inadequate background, poor conceptualization of problems or approach, and duplication of effort.

A major deficiency observed in rehabilitation research applications is the presentation of poorly conceived, clinically irrelevant or pedestrian research ideas. This major flaw may be demonstrated by either the presentation of a weak idea or a research approach that is not at the "cutting edge" of appropriate and adequate methodologies of scientific investigation. As will be discussed in the results of this study, this single deficiency accounts for a high percentage of disapprovals. Frequently, reviewers who are not enthusiastic about a research idea will look negatively on other aspects of the proposal, while an innovative, relevant, well conceptualized research issue will generate an aura of enthusiasm despite some apparent weaknesses. In essence, given a keen competition, the new and creative idea will win over the mundane, unimaginative notion.

For medically related proposals, NIDRR acquires the highest level, senior medical specialty participation in peer review activities. Therefore, physicians and other appropriate allied health disciplines in clinical rehabilitation will comprise the major complement of reviewer expertise. Sometimes, nonphysician investigators will propose a study or apply invasive types of diagnostic, evaluative or therapeutic techniques in humans, without adequate medical participation or supervision of essentially medical-surgical procedures. To assure the government's and applicant organization's interest in human subject protection, this type of flawed research is generally considered unacceptable.

Due to limited funding resources for rehabilitation research and the very nature of the traditions in the rehabilitation research community, a keen interest in effective and purposeful dissemination and utilization has evolved. Field-initiated research, as a major research resource to the field for rapid acquisition of new knowledge, requires thoughtful attention in the development of dissemination and utilization plans for results and findings. Throughout the relevant evaluation criteria, including importance of problem (20 points), design of project (45 points) and management and evaluation (15 points), the proposed investigator is challenged to provide an innovative and well-conceptualized strategy for distribution and application of research results by appropriate target audiences. A well-developed idea with proper design and methodology and worthy of research is not a complete proposal without an effective dissemination plan.

The deficiency of inadequate background of investigator occurs when, occasionally, the less experienced investigator will, out of interest, attempt an area of research with which they have little academic or practical professional experience. Given two proposals identical in quality and importance, the more experienced and/or productive investigator will usually win the review panel's favor because of a greater assumption that there will be success. In a nationally based, extremely keen research competition such as those typical of NIDRR, past productivity and experience, particularly in a recognized and previously published area of endeavor, are essential. Occasionally observed in this category of research deficiency was the applicant with minimal research training or experience who is seeking federal funding support of a particular nonscientific interest or program development initiative with little or no research component. These proposals have generally been not well received by reviewers.

With regard to duplication of effort, frequently a proposal is submitted in an area of investigation identical or closely similar to ongoing supported research. However, the supported research may be so new as to not be reported in the scientific literature. This is a particular problem that can best be resolved by direct communication with the potential funding agency or organization. Usually, research supporting organizations are willing to discuss their ongoing research portfolios to avoid potential areas of duplication. Many federal funding agencies, including NIDRR, publish annually a listing or directory of supported research. A quick review can readily identify areas of potential duplication without jeopardizing the proprietary rights of the proposed or supported investigators. Submitting duplicative research proposals, however, can be an embarrassment to the investigating team and the applicant organization. It suggests haphazard preparation and lack of knowledge of one's proposed area of professional endeavor.

Summary

This article seeks to identify and describe those deficiencies determined by nonfederal peer reviewers during the most recent 1989 NIDRR research competition to mitigate successful funding support from the lead federal agency concerned with medical rehabilitation research.

In 1989, 232 field investigator initiated research applications were received in response to the Federal Register announcement of September 8, 1988. Of these, 57 or 24.5 percent were assigned to the Medical Sciences Program Office (MSP). Six applications were ultimately approved and funded (10.7 percent). The remaining 51 (89.5 percent) were statistically analyzed to determine deficiencies significant in warranting disapproval.

The 51 applications averaged 5.68 deficiencies each. Five types of research error accounted for 76.8 percent of all deficiencies noted by reviewers. These included methodological errors[25] and inadequate control of subject variables[25], followed by inappropriate research design[24], poor conceptualization of problem/approach[21] and incorrect statistical analysis[14]. The top ranked singular deficiency was poor conceptualization of problem or approach. Other flaws warranting lowering of potential scores by reviewers included excessive budget requests, duplication of effort with ongoing supported research, inadequate background of investigator, and weak dissemination and utilization plans.

In conclusion, many applications are submitted for the NIDRR field investigator initiated research program competitions that are significantly deficient. Those identified areas of deficiency cover most of the conceptualization, research design and methodological criteria essential for valid, creditable scientific investigation. Without careful attention to the tradition and basic principles of the scientific method, these efforts will not contribute to the acquisition of new and useful knowledge which is so badly needed in a young and rapidly developing field. [Tabular Data Omitted]

Suggested Reading

[1]Ambroz, A., et al. Deficiencies of Randomized Controlled Trials. 1979 Clinical Research 26:280. [2]Boring, E.G. The Nature and History of Experimental Control. American Journal of Psychology, 1954, 67:573-589. [3]Campbell, Donald T. Factors Relevant to the Validity of Experiments in Social Settings Psychological Bulletin, 1959. Vol. 54, No. 4:297-311. [4]Cuca, J. M. NIH Grant Applications for Clinical Research: Reasons for Poor Ratings or Disapproval. Clinical Research, 1983, 31: 453-463. [5]Feinstein, A. R. Clinical Biostatistics: A Survey of Statistical Procedures in General Medical Journals. Clinical Pharmacology Therapy, 1974, 15:97-107. [6]Fletcher, R. and Fletcher, S. Clinical Research in General Medical Journals: A 30 Year Perspective. New

England Journal of Medicine 1979, 301:180-183. [7]Freeman, J. A., Chalmers, T. C., Smith, H., and Kuebler, R. The Importance of Beta, The Type II Error and Sample Size in the Design and Interpretation of the Randomized Control Trial. New England Journal of Medicine 1979, 299:690-694. [8]Hulley, S. B. and Cummings, S. R. Designing Clinical Research. Baltimore: Williams and Wilkins, 1988. [9]Jahoda, M., Deutsch, M., and Cook, S. W. Research Methods in Social Relations. New York: Dryden Press, 1951. [10]Kane, E. H. and Jordon, H. J. The Greatest Adventure: Basic Research that Shapes Our Lives. New York: Rockefeller University Press, 1974. [11]Kazin, A. E. Simple-Case Research Designs. New York: Oxford University Press, 1982. [12]Kish, L. Selection of the Sample. In L. Festinger and D. Katz (Ed) Research Methods in the Behavioral Sciences. New York: Dryden Press, 175-239, 1953. [13]Kramer, H. C. and Thielman, S. How Many Subject? London: Sage Publications, 1987. [14]Lazarfeld, P. F. Training Guide on the Controlled Experiment in Social Research. Columbia University Press, 1948. [15]Merton, R. K. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press, 1973. [16]National Science Foundation. Restrospect and Critical Events in Science Prepared by Illinois Institute of Technology. Washington, DC: Government Printing Office, 1968. [17]Novello, A. C. The Peer Review Process: How to Prepare Research Grant Applications to the NIH. Mineral Electrolyte Metabolism, 1985. 11:281-286. [18]Ottenbacher, K. H. Evaluating Clinical Change: Strategies for Occupational and Physical Therapists. Baltimore: Williams and Wilkins, 1986. [19]Schor, S. and Karten, I. Statistical Evaluation of Medical Journal Manuscripts. J.A.M.A. 1966. 95:1123-1128. [20]Sherman, C. R. and Morgan, T. E. Educational Patterns and Research Grant Success of Medical School Faculty, Washington, DC: AAMC, 1979. [21]U.S. Department of Health, Education and Welfare. Research 1971: An Annotated List of Research and Demonstration Grants, Office of Research and Demonstrations, Social and Rehabilitation Services, Washington, DC. [22]Visscher, M.B. Applied Science and Medical Progress. Washington, DC: National Academy of Sciences, 1967. [23]Wolf, G. K. Basic Questions for the Use of Simple Case Statistics. Pharoco-Psychiatry 1986. 18:331-332. [24]Wyngaarden, J. B. The Clinical Investigator As An Endangered Species. New England Journal of Medicine 1979. 301:1254-1259. [25]Zelen, M. The Randomization and Stratification of Patients to Clinical Trials. Journal of Chronic Diseases 1974. 27:365-375.

Dr. Thomas is Director, Medical Science Programs, and Ms. Lawrence is Senior Research Associate, NIDRR, Office of Special Education and Rehabilitative Services, Department of Education, Washington, D.C. 20202-2701.

COPYRIGHT 1990 U.S. Rehabilitation Services Administration
COPYRIGHT 2004 Gale Group