Cochrane Methods

Methodology Register

Background: Many quality assessment instruments (QAIs) are available for evaluating randomized control trials (RCTs). Among them, the Jadad scale is the most frequently used. Limitations of the Jadad scale have been identified and its use has been questioned by many experts. This study was undertaken to explore options for replacing the Jadad scale for the quality assessment of RCTs. Objective: To identify the existing QAIs for RCTs and find the most appropriate one using a systematic approach. Methods: A comprehensive review article on the QAIs, Evidence Report Number 47 from the Agency for Healthcare Research and Quality (AHRQ), was identified as the starting point by the review team. This report evaluated all the QAIs identified by a systematic review of the literature from1995 to mid-2000. A multi-prong strategy was used to identify QAIs reported post mid- 2000. A comprehensive search was conducted to identify and collect further new review articles on QAIs published from 2000 to 2005. From this search, review articles reporting information on two or more QAIs regarding methodological domains, validity, and reliability were selected. The list of all the identified review articles was sent to national and international experts to determine: if any important review articles were missing; if the experts were aware of any new QAI not covered in the listed reviews. In addition, a second literature search was performed to identify if any new QAIs had been developed since 2000. All instruments recommended by the new review articles, or identified as unique QAIs in the reviews or by experts, were selected for further evaluation using the same seven, key methodological domains utilized in the AHRQ report. The review team evaluated the highest-scoring instruments. After receiving input from national and international experts and stakeholders, the final selection of QAIs was made by the review team. Results: Three-thousand-and-six (3,006) citations were identified by a highly sensitive search strategy run on PubMed, Medline®, Embase®, BIOSIS Previews® and the Cochrane Library, as well as a targeted hand searching and grey literature searching. Five reviews of QAIs for RCTs, including the AHRQ Report Number 47, were identified by two reviewers. Thirty-four experts (20 international and 14 Canadian) were further contacted to identify any review article or instrument. Nine experts (five international and four Canadian) responded. Seventy-three existing QAIs were identified from the five reviews after eliminating duplicates. A total of 30 QAIs (18 of 73 QAIs from the reviews, 10 from expert consultation, and two from the 1,120 citations of the second search result of the individual QAIs) were evaluated. Three QAIs, including Chalmers et al.(1996), Reisch et al.(1989), and SIGN50 (2004), obtained perfect scores of seven using the AHRQ criteria, about study population, randomization, blinding, interventions, outcomes, statistical analysis, and funding while the Jadad scale scored 2.5. These three QAIs were further evaluated by the review team and consensus was reached for the selection of the SIGN50 (2004) methodological checklist after considering some important descriptive factors, such as a rigorous development process. The selection decision of SIGN50 was sent to nine experts who had responded to the previous request. Five experts gave positive feedback, two gave ambiguous feedback, one gave negative feedback, and one expert did not respond. Conclusion: Based on this work, the authors conclude that the SIGN50 checklist is the most appropriate QAI for RCTs because: it includes the most important methodological domains; it provides detailed instructions for application to produce overall quality assessment scores.