How relevant are the systematic reviews in the cochrane library to emergency medical practice?☆☆☆

Introduction 

Advances in biomedical knowledge continue to expand the information available to guide clinicians in the care of patients. To effectively translate this evidence into practice, a clinician needs to access relevant data, determine its validity, and then decide whether it applies to a given patient. These tasks are particularly challenging in the emergency setting, where the scope of practice crosses numerous medical disciplines and the pace of decisionmaking can be frenetic.

In this context, traditional resources such as textbooks and MEDLINE searches are of limited value in identifying the best evidence to guide clinical practice. However, the promise of evidence-based medicine resources is to provide the busy clinician with rapidly accessible synthesized information using valid and reliable approaches. The Cochrane Library, a compendium of databases and related instructional tools, has been promoted as just such a reference.1

The Cochrane Library is composed of several databases, including the Cochrane Controlled Trials Register (CCTR), the database of Abstracts of Reviews of Effectives (DARE), and the Cochrane Database of Systematic Reviews (CDSR). The CDSR database is a compilation of regularly updated systematic reviews with meta-analytic summary statistics. Collaborative review groups representing various medical topic areas (eg, airways, musculoskeletal, heart, epilepsy) contribute content to the CDSR.

The CDSR consists of protocols and completed reviews. Protocols describe the objectives of systematic reviews that are in the process of being completed. Completed reviews include the full text and usually present summary statistics. The protocols and completed reviews are produced using rigorous methodological standards and undergo peer review before publication. Regular updates are required to capture new evidence and address criticisms and/or identified errors.

Although the CDSR database contains rigorously produced systematic reviews, the relevance of these reviews to emergency medicine practice has not been previously evaluated. We therefore conducted a study to assess the relevance of the CDSR database reviews to emergency medicine practice.

Materials and methods 

The study methodology is summarized in Figure 1.

Fig. 1. Study methodology.

Fig. 2. Screening classification.

Reviews rated as “clearly and directly related to emergency medicine” and “can't tell” were evaluated further. Three reviewers (PCW, MDB, WHC) independently examined the structured abstracts of the “selected” reviews and scored their relevance to emergency medicine practice using a second 3-category scale: “not relevant,” “indirectly relevant,” (ie, may be important to emergency medicine patients, but not directly used in emergency department or discharge planning) or “directly relevant” (important to emergency medicine patients, care in the ED, or discharge care). Disagreement was resolved by consensus. To assess the interrater agreement, 25% of reviews were examined by 2 different reviewers (BHR, SDE). Interrater agreement was measured by using simple agreement and κ weighted statistics.

The institutional review board at New York Presbyterian Hospital/Weill Cornell Center ruled that this structured survey was exempt from review.

Results 

Screening of the 795 completed reviews in the CDSR database identified 136 (17%) as “potentially” relevant and 151 (19%) as “likely” relevant (simple agreement, 74%; κ=0.82). Independent scoring of the 151 “likely” relevant reviews indicated that 95 (12%) were “directly” relevant to emergency medical practice and 44 (6%) were “indirectly” relevant to emergency medical practice (simple agreement, 77%; κ=0.45). The remaining reviews were deemed “not relevant” to emergency medical practice.

The majority of completed reviews in the CDSR database rated as “relevant” to emergency medical practice were produced by 5 Cochrane Reviews Groups (CRGs) (Figure 3)

Fig. 3. Reviews rated as relevant to emergency medicine (grouped by CRG).

Discussion 

The Cochrane Library is produced by the Cochrane Collaboration, an international, multidisciplinary, volunteer network of health care professionals, consumers, researchers, and policymakers and published quarterly by Update Software Ltd.(Oxford, England).1, 2 The goal of the Collaboration is to produce, maintain, and disseminate evidence on therapy. In general, the Cochrane Collaboration uses groupings known as “fields” and “networks” to define dimensions of health care other than health problems. These include the setting of care (eg, primary care), the type of consumer (eg, older people), the type of provider (eg, nurses), and the type of intervention (eg, physical therapies).

Persons working in the fields/networks hand search specialty journals and help ensure that priorities and perspectives in their field of interest are reflected in the work of collaborative review groups. They also compile specialist databases of reviews, coordinate activities with relevant agencies outside the Cochrane Collaboration, and comment on systematic reviews relating to their particular area. Cochrane fields include child health, complementary medicine, health care of older people, health promotion and public health, nursing, pharmaceuticals, primary health care, rehabilitation, and vaccines. Given its comprehensive goals, this effort has been described as “the Human Genome Project of randomized, controlled trials.”3

The Cochrane Library consists of several databases, 3 of which may be useful to the emergency physician. The CCTR is an extensive bibliographic database of controlled trials that has been identified through structured searches of electronic databases, and hand searching by CRGs. Currently, it contains more than 300,000 references (Cochrane Library, version 3, 2001) and can function as a primary literature searching approach with therapeutic topics. DARE consists of critically appraised structured abstracts of non-Cochrane published reviews that meet standards set by the Centre for Reviews and Dissemination at the University of York, England. The last, and possibly most important, resource is the CDSR, which was the focus of this study.

The methodology used to produce these Cochrane Library databases is strict. Independent review of studies for possible inclusion is emphasized to minimize selection bias. Comprehensive searches of published, unpublished, and foreign language literature are employed to limit publication bias. Quality grading (to enable sensitivity analysis) and standardized statistical methodology (to ensure uniformity) are also used. These procedures are in place to help ensure validity and produce unbiased reviews offering the highest level of evidence on which to base clinical decisions.

Despite anticipation that the CDSR would be useful in primary care, emergency medicine, pediatrics, and internal medicine, acceptance has been less than expected. Initially, this was understandable because of the small number of completed reviews. However, there has been a rapid proliferation of reviews in the Cochrane Library databases. The initial issue contained less than 100, compared with the most recent (July 2001; Issue 3), which contains 1,147 completed reviews and 915 protocols. (Our study evaluated the April 2000, Issue 2, which contained 795 completed reviews.)

The lack of acceptance may also be a result of the limitations inherent to the Cochrane Collaboration and the Cochrane Library databases. First, the Cochrane Collaboration's approach limits its scope. It only summarizes questions regarding therapy, not diagnosis. Current controversy regarding evidence-based medicine has focused on relevance and applicability of the limited “high level” evidence that exists.4, 5, 6, 7, 8, 9 Second, as a product of a young, evolving organization, the CDSR's development is still relatively limited. Cochrane Review Groups are still being established, and a review takes up to 3 years from the registering of a title and publishing a protocol to completion. Because the Cochrane Collaboration leaves decisions on topics to be reviewed to the CRG member interests and funding opportunities, gaps remain in the medical domains covered. Third, there are few emergency physicians actively involved in driving the review process. And fourth, there are sometimes few randomized trials available that involve very common therapeutic interventions in the ED setting, making systematic reviews in these topics less definitive.

In addition, widespread “point-of-care” access to the Cochrane Library and other evidence-based medicine resources have been difficult to provide, making it a challenge to formally test how emergency physicians, in training and in practice, use an evidence-based approach to access and incorporate new clinical information. This exercise is an essential component of the “model curriculum” for emergency medicine that is currently in development.10

Another possible explanation for any continued lack of acceptance may be the perceived lack of relevant reviews in a given practice domain. Our survey of the April 2000 issue of the Cochrane Library demonstrated that the CDSR contains 95 completed reviews directly relevant to emergency medical practice. These reviews are produced following rigorous methodology and are the most comprehensive summaries of evidence available. Nevertheless, given the wide scope of practice in emergency medicine, 95 is far from comprehensive coverage.

Despite these limitations, our study suggests that one of the Cochrane Library databases does provide systematic reviews relevant to emergency medical practice. More than one third of April 2000 CDSR completed reviews had some relevance to practice or discharge planning. Certain topics central to emergency medical care, including stroke, injuries, and respiratory conditions, were well represented. However, completed reviews in other areas such as cardiology and wound care were sparse.

The results of our study are subject to some limitations. First, the methodology to classify the reviews has not been validated. Differences in notion of “essential” vary among practitioners, sites, regions, and countries. For example, whereas interrater agreement was excellent for the screening step, it was only modest for final classification. Although in cases of disagreement final classification was determined by consensus, ongoing bias (eg, academic bias) may have affected review selection. Future studies should evaluate the utility of Cochrane systematic reviews to emergency physicians who practice exclusively in the nonteaching and/or community settings. Furthermore, because we reviewed only the 151 “likely” relevant summaries, we may have missed relevant reviews in the “possibly relevant” category.

Since our study was conducted, the CDSR has continued to evolve and expand. One hundred fifty new reviews have been added, including at least 4 new relevant reviews produced by the Airways Cochrane Review Group alone. Summaries of Cochrane reviews have been made more readily available for point-of-care use through MEDLINE to those with online access at http://www4.ncbi.nlm.nih.gov/PubMed/ , as well as through an online searchable list of abstracts at the Cochrane Collaboration Web site at http://www.update-software.com/abstracts/titlelist.htm .

In summary, more than one third of April 2000 CDSR completed reviews had some relevance to practice or discharge planning. Reviews relevant to emergency medicine in cardiology and wound care are sparse. Increased emergency physician involvement in the Cochrane Collaboration would help to increase the number of reviews conducted on questions relevant to emergency medicine and would enhance the value of this evidence-based medicine resource in patient care and health care provider education.