How often does correlation=causation? "Randomized versus historical controls for clinical trials", Sacks et al 1982 https://www.dropbox.com/s/cjq0aqkxnrv1ba6/1982-sacks.pdf ; excerpts:

"To compare the use of randomized controls (RCTs) and historical controls (HCTs) for clinical trials, we searched the literature for therapies studied by both methods. We found 6 therapies for which 50 RCTs and 56 HCTs were reported. 44 of 56 HCTs (79%) found the therapy better than the control regimen, but only 10 of 50 RCTs (20%) agreed. For each therapy, the treated patients in RCTs and HCTs had similar outcomes. The difference between RCTs and HCTs of the same therapy was largely due to differences in outcome for the control groups, with the HCT control patient generally doing worse than the RCT control groups. Adjustment of the outcomes of the HCTs for prognostic factors, when possible, did not appreciably change the results. The data suggest that biases in patient selection may irretrievably weight the outcome of HCTs in favor of new therapies. RCTs may miss clinically important benefits because of inadequate attention to sample size. The predictive value of each might be improved by reconsidering the use of p<0.05 as the significance level for all types of clinical trials, and by the use of confidence intervals around estimates of treatment effects.

Acceptance of the RCT is growing but still far from universal, and the majority of published clinical trials do not use this method [ 1,2]. Recent articles have argued that RCTs are impractical in surgery [3,4] and often unnecessary in cancer medicine [5]. Double-blind RCTs have been criticized on ethical and practical grounds, and it is claimed that comparison with recently treated patients with the same disorder (HCTs) can evaluate new therapies more rapidly without exposing patients to possibly ineffective therapies and uncertainties [6].

Since 1955, one of the authors (TCC) has maintained a file of RCTs published in English. Articles for this file are gathered by computer and manual literature searches of Mex Medicus on specific topics of interest, by weekly review of Current Contents and by checking references of reviews and papers already in the file. At the same time, HCTs and uncontrolled trials are also filed. RCTs were defined as trials in which both treatment and control groups were gathered prospectively and randomly assigned. Trials in which prospectively collected treatment groups were compared with either previously published series or previously treated patients at the same institutions were considered HCTs if the authors drew conclusions about relative efficacy from these comparisons. HCTs were further subdivided into those that simply compared over-all outcome and those that matched or adjusted outcome rates on the basis of prognostic categories, or provided sufficient data so that some such adjustments could be made by the reader. Therapies were considered for inclusion in the present study if at least two RCTs and two HCTs were found for the same therapy.

Studies were considered positive (i.e., they found the new therapy to be effective) if a statistically significant benefit in outcome was found for treatment over control regimen, or, when no statistical analysis was presented, if the authors concluded that the therapy was superior to the control regimen. Trials that did not meet these criteria were considered negative. When multiple outcomes were studied, the most serious (e.g., death) was used.

106 papers on 6 therapeutic questions met our criteria for inclusion in the study. Over-all, 10 of 50 RCTs (20%) found a benefit from the therapy studied, while 44 of 56 HCTs (79%) on the same questions concluded that the therapies were beneficial (Table I). 29 of the 50 RCTs (58%) gave the probability that the difference found could be due to chance (alpha) or provided sufficient data so that the probability could be calculated; in seven (14%), this probability was less than 0.05. 26 of the HCTs (46%) provided probability values or data for estimating alpha; in 22, it was less than 0.05 (Table II).

In nearly always finding the treatment better than the control regimen, the HCTs we examined would rarely come to false-negative conclusions, but this may be at the expense of many false-positives. The RCTs have the opposite fault: by declaring most therapies no better than the control regimens, they would rarely come to false-positive conclusions but possibly create many false-negatives...The accuracy of RCTs, on the other hand, could be improved by greater attention to sample size in planning studies. A recent review of 71 “negative” RCTs [117] found that a potential 25% improvement could have been missed in 57, and a potential 50% improvement in 34."