Cochrane Methods

Methodology Register

Objectives: There is often little or no evidence from randomised controlled trials (RCTs) on the effectiveness of health care interventions. When no RCTs exist, there is currently a debate whether non-randomised but comparative studies are better or worse than no evidence. For some surgical procedures, there may not even be any comparative studies, the best evidence available being case series; such studies are generally regarded as contributing less strong evidence than RCTs and other analytic designs, and reviewers have traditionally cautioned against any attempt to pool such evidence quantitatively. The objective of this paper is to report an example of pooling of outcome data from many case series, with a view to initiating a debate about the nature and value of evidence from systematic reviews of case series. Methods: This review of the effectiveness of paediatric cardiac surgery was commissioned by a public inquiry to inform specific objectives. Evidence was reviewed for 5 paediatric open-heart operations for congenital cardiac anomalies: transposition of the great arteries; complete atrio-ventricular septal defect; total anomalous pulmonary venous drainage; truncus arteriosus; Fontan procedure. All studies reporting 30-day mortality were eligible, the primary objective being to synthesise estimates of 30-day mortality for each operation type. Relevant papers were identified by systematic literature searches of Medline (1966-1995). Key data were extracted from each paper according to a protocol. Pooled estimates of 30-day mortality were derived by logistic regression modelling, with 'robust' confidence intervals to take account of clustering of patients within case series. Additional analyses sought to explain variation in 30-day mortality estimates by investigating the effect of the date of data collection and relevant clinical sub-groups within operation types, i.e. case-mix, on 30-day mortality. We also sought evidence on the prognosis for children with each congenital anomaly without surgery. Results: Literature searches yielded only case-series data, which were successfully pooled for each operation. Within each operation type, 30-day mortality varied significantly for different clinical sub-groups, sometimes by a factor of two or more. 30-day mortality also reduced significantly over the time period studied for all operation types. Indices of reporting quality, such as reporting a clear case definition and an explicit statement that eligible patients were recruited consecutively, were not consistently associated with 30-day mortality. For the congenital anomalies reviewed, the prognosis for children who do not undergo surgery is poor, with the majority dying before their first birthday. Conclusions: Pooled 30-day mortality estimates are likely to reflect the best that can be achieved rather than a level of performance to be expected in everyday practice. The evidence of differences in mortality between clinical sub-groups were consistent with clinical expectation at the outset. Several methodological questions about the nature and status of such evidence will be discussed, e.g should such a review be eligible for inclusion in the Cochrane Library as evidence of effectiveness, how should concepts of systematicity and publication bias be applied to case series?