How often does correlation=causation? "Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review)" http://www.thecochranelibrary.com/userfiles/ccoch/file/INternational%20Clinical%20Trials%20Day/MR000012.pdf , Kunz et al 2002:
"_Background_: Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias), provided that a sufficient number of people are randomised.
...Selection criteria: Cohorts of trials, systematic reviews or meta-analyses of healthcare interventions that compared outcomes or prognostic factors for one of the following comparisons: randomised versus non-randomised trials, randomised trials with adequately versus inadequately concealed allocation, or high versus low quality trials where selection bias could not be separated from other sources of bias.
Data collection and analysis: One of us went through all of the citations in the Cochrane Methodology Register and accumulated reference lists. Studies that appeared to meet the inclusion criteria were retrieved and assessed independently by two of the reviewers. The methodological quality of included studies was appraised and information extracted by one of us and checked by a second. Tabular summaries of the results were prepared for each comparison and the results across studies were assessed qualitatively to identify common trends or discrepancies.
Main results: We identified 32 studies including over 3000 trials. 22 studies compared randomised versus non-randomised trials, 3 compared adequately versus inadequately concealed allocation, and 9 compared high versus low quality trials (some studies included more than one comparison). 5 studies were of high methodological quality.
In 15 of the 22 studies that compared randomised and non-randomised trials of the same intervention, important differences were found in the estimates of effect. Some of these differences were due to a poorer prognosis in the control groups in the non-randomised trials. The results of the other 7 studies that compared randomised and non-randomised trials across different interventions are less clear.
Comparisons of adequately and inadequately concealed allocation in randomised trials of the same intervention provided high quality evidence that concealment can be crucial in achieving similar treatment groups and, therefore, unbiased estimates of treatment effects. Studies with inadequate concealment tended to overestimate treatment effects.
...Some interventions that are believed to be beneficial are, in fact, no more effective than a placebo and some are even harmful. Well-intentioned clinicians have, for exam- ple, treated stroke by applying leeches to the anus (Gubler 1971), treated neurosyphilis by injecting malarial parasites (Austin 1992), treated angina with internal mammary artery ligation (Valenstein 1998), treated symptomatic atherosclerotic disease of the internal carotid artery with extracranial-intracranial bypass surgery (EC/IC Bypass 1985), and treated asymptomatic ventricular arrhythmia after myocardial infarction with antiarrhythmic drugs (Echt 1991). It is estimated that tens of thousands of patients died prematurely from widespread use of class I antiarrhythmic drugs alone (Moore 1995), which caused one death for every 20 patients who were treated (Teo 1993). Failure to adequately evaluate interventions has also delayed the use of effective interventions, such as magnesium sulphate instead of diazepam or phenytoin for the treatment of eclampsia (Eclampsia 1995).
...Despite this simple logic and many anecdotal examples of harm being done because of delays in conducting randomised trials, there are limitations to the use of randomised trials, both real and imagined, and scepticism about the importance of randomisation (US Office HTA 1994; Black 1996; Weiss 1998; Pocock 2000). We believe this scepticism is healthy. It is important to question assumptions about research methods, and to test these assumptions empirically, just as it is important to test assumptions about the effects of health care. Methodological hubris can be just as dangerous as medical hubris. Empirical comparisons of randomised versus non-randomised evaluations of the effects of healthcare represent important steps away from hubris. This review of such comparisons has been updated from a previously published review (Kunz 1998 Kunz R, Oxman AD. "The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials" http://www.ncbi.nlm.nih.gov/pmc/articles/PMC28700/ . BMJ 1998;317:1185–90 ) (see What’s new). It differs from other similar reviews (Reeves 1998; McKee 1999) in the questions that are addressed and the methods that were used, but there is not a major disagreement in the conclusions of these reviews (Kunz 1999; Britton 1999)."
"_Background_: Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias), provided that a sufficient number of people are randomised.
...Selection criteria: Cohorts of trials, systematic reviews or meta-analyses of healthcare interventions that compared outcomes or prognostic factors for one of the following comparisons: randomised versus non-randomised trials, randomised trials with adequately versus inadequately concealed allocation, or high versus low quality trials where selection bias could not be separated from other sources of bias.
Data collection and analysis: One of us went through all of the citations in the Cochrane Methodology Register and accumulated reference lists. Studies that appeared to meet the inclusion criteria were retrieved and assessed independently by two of the reviewers. The methodological quality of included studies was appraised and information extracted by one of us and checked by a second. Tabular summaries of the results were prepared for each comparison and the results across studies were assessed qualitatively to identify common trends or discrepancies.
Main results: We identified 32 studies including over 3000 trials. 22 studies compared randomised versus non-randomised trials, 3 compared adequately versus inadequately concealed allocation, and 9 compared high versus low quality trials (some studies included more than one comparison). 5 studies were of high methodological quality.
In 15 of the 22 studies that compared randomised and non-randomised trials of the same intervention, important differences were found in the estimates of effect. Some of these differences were due to a poorer prognosis in the control groups in the non-randomised trials. The results of the other 7 studies that compared randomised and non-randomised trials across different interventions are less clear.
Comparisons of adequately and inadequately concealed allocation in randomised trials of the same intervention provided high quality evidence that concealment can be crucial in achieving similar treatment groups and, therefore, unbiased estimates of treatment effects. Studies with inadequate concealment tended to overestimate treatment effects.
...Some interventions that are believed to be beneficial are, in fact, no more effective than a placebo and some are even harmful. Well-intentioned clinicians have, for exam- ple, treated stroke by applying leeches to the anus (Gubler 1971), treated neurosyphilis by injecting malarial parasites (Austin 1992), treated angina with internal mammary artery ligation (Valenstein 1998), treated symptomatic atherosclerotic disease of the internal carotid artery with extracranial-intracranial bypass surgery (EC/IC Bypass 1985), and treated asymptomatic ventricular arrhythmia after myocardial infarction with antiarrhythmic drugs (Echt 1991). It is estimated that tens of thousands of patients died prematurely from widespread use of class I antiarrhythmic drugs alone (Moore 1995), which caused one death for every 20 patients who were treated (Teo 1993). Failure to adequately evaluate interventions has also delayed the use of effective interventions, such as magnesium sulphate instead of diazepam or phenytoin for the treatment of eclampsia (Eclampsia 1995).
...Despite this simple logic and many anecdotal examples of harm being done because of delays in conducting randomised trials, there are limitations to the use of randomised trials, both real and imagined, and scepticism about the importance of randomisation (US Office HTA 1994; Black 1996; Weiss 1998; Pocock 2000). We believe this scepticism is healthy. It is important to question assumptions about research methods, and to test these assumptions empirically, just as it is important to test assumptions about the effects of health care. Methodological hubris can be just as dangerous as medical hubris. Empirical comparisons of randomised versus non-randomised evaluations of the effects of healthcare represent important steps away from hubris. This review of such comparisons has been updated from a previously published review (Kunz 1998 Kunz R, Oxman AD. "The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials" http://www.ncbi.nlm.nih.gov/pmc/articles/PMC28700/ . BMJ 1998;317:1185–90 ) (see What’s new). It differs from other similar reviews (Reeves 1998; McKee 1999) in the questions that are addressed and the methods that were used, but there is not a major disagreement in the conclusions of these reviews (Kunz 1999; Britton 1999)."