How often does correlation=causation? "A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies", MacLehose et al 2000:
"Advocates for quasi-experimental and observational (QEO) studies argue that evidence from randomised controlled trials (RCTs) is often difficult or impossible to obtain, or is inadequate to answer the question of interest. Advocates for RCTs point out that QEO studies are more susceptible to bias and refer to published comparisons that suggest QEO estimates tend to find a greater benefit than RCT estimates...Two strategies were used to minimise the influence of differences in external validity between RCTs and QEO studies:
• a comparison of the RCT and QEO study estimates of effectiveness of any intervention, where both estimates were reported in a single paper
• a comparison of the RCT and QEO study estimates of effectiveness for specified interventions, where the estimates were reported in different papers.
...Strategy 1:
Fourteen papers were identified, yielding 38 comparisons between RCT and QEO study estimates; 25 were classified as low and 13 as high quality. Discrepancies between RCT and QEO study estimates of effect size and outcome frequency for intervention and control groups were smaller for high- than low-quality comparisons. For high-quality comparisons, no tendency was observed for QEO study estimates of effect size to be more extreme than RCT ones, but this tendency was seen with low-quality comparisons.
...Regression analyses showed that both cohort and case–control studies had lower total quality scores than RCTs; cohort studies also had significantly lower scores than case–control studies. The latter, counter-intuitive finding may reflect a general tendency for quasi-experimental studies (which must use cohort designs) to have lower quality than observational studies.
["In 14 of 35 [40%] comparisons the discrepancy in RR was <10%, in 5 comparisons the RR difference was >50%. Discrepancies were smaller in 'fairer' comparisons."
See plot on pg35, tables on pg144+, the exact comparison is Table 39, Index I (RR_RCT / RR_QEO): split by statistical-significance, 6+8=14 comparisons were 1.00<=x<=1.10, 5+3=8 were 1.10<x<=1.25, 1+7=8 were 1.25<x<=1.5, 1+1 were 1.5<x<=2, 3 were x>2, for a total of 16+19=35 comparisons. ]"
"Advocates for quasi-experimental and observational (QEO) studies argue that evidence from randomised controlled trials (RCTs) is often difficult or impossible to obtain, or is inadequate to answer the question of interest. Advocates for RCTs point out that QEO studies are more susceptible to bias and refer to published comparisons that suggest QEO estimates tend to find a greater benefit than RCT estimates...Two strategies were used to minimise the influence of differences in external validity between RCTs and QEO studies:
• a comparison of the RCT and QEO study estimates of effectiveness of any intervention, where both estimates were reported in a single paper
• a comparison of the RCT and QEO study estimates of effectiveness for specified interventions, where the estimates were reported in different papers.
...Strategy 1:
Fourteen papers were identified, yielding 38 comparisons between RCT and QEO study estimates; 25 were classified as low and 13 as high quality. Discrepancies between RCT and QEO study estimates of effect size and outcome frequency for intervention and control groups were smaller for high- than low-quality comparisons. For high-quality comparisons, no tendency was observed for QEO study estimates of effect size to be more extreme than RCT ones, but this tendency was seen with low-quality comparisons.
...Regression analyses showed that both cohort and case–control studies had lower total quality scores than RCTs; cohort studies also had significantly lower scores than case–control studies. The latter, counter-intuitive finding may reflect a general tendency for quasi-experimental studies (which must use cohort designs) to have lower quality than observational studies.
["In 14 of 35 [40%] comparisons the discrepancy in RR was <10%, in 5 comparisons the RR difference was >50%. Discrepancies were smaller in 'fairer' comparisons."
See plot on pg35, tables on pg144+, the exact comparison is Table 39, Index I (RR_RCT / RR_QEO): split by statistical-significance, 6+8=14 comparisons were 1.00<=x<=1.10, 5+3=8 were 1.10<x<=1.25, 1+7=8 were 1.25<x<=1.5, 1+1 were 1.5<x<=2, 3 were x>2, for a total of 16+19=35 comparisons. ]"