"Do Certain Countries Produce Only Positive Results? A Systematic Review of Controlled Trials", Vickers et al 1998; something to keep in mind if you are reading about, say, how moxibustion helps reduce Alzheimer's:
"To determine whether clinical trials originating in certain countries always have positive results. Data sources: Abstracts of trials from Medline (January 1966–June 1995)...Research conducted in certain countries was uniformly favorable to acupuncture; all trials originating in China, Japan, Hong Kong, and Taiwan were positive, as were 10 out of 11 of those published in Russia/USSR. In studies that examined interventions other than acupuncture, 405 of 1100 abstracts met the inclusion criteria. Of trials published in England, 75% gave the test treatment as superior to control. The results for China, Japan, Russia/USSR, and Taiwan were 99%, 89%, 97%, and 95%, respectively. No trial published in China or Russia/USSR found a test treatment to be ineffective.
...
The 330 most recent randomized or controlled clinical trials published in England were retrieved as a comparison group.
Results of Controlled Clinical Trials of Acupuncture by Country of Research: Percentage Favoring Test Treatment:
USA 53%
Seden 59%
UK 60%
China 100%
Particularly high rates of positive results were seen in China (99%) and Russia/USSR (97%). These two countries published no trials in which the test treatment was not reported effective. In the one study published in Russia/USSR classed as “test treatment equal or inferior to control,” nitrendipine was found to be equally effective as standard antihypertensives such as propranolol.
There are several possible/alternative explanations for our findings.
1. The sample of trials may not have been representative. Our search would not have located all published trials. We chose Medline because it provides a sufficiently representative set of the best-quality trials, particularly in acupuncture [2, 3], but trials are published in many journals not indexed by Medline. Possibly, negative trials originating from eastern Europe and Asia are found solely in non-Medline journals. We believe that this is unlikely; Medline might be expected to be a conservative source of information on unconventional therapies. Moreover, anecdotal evidence, such as the results presented at acupuncture conferences, does not suggest any considerable number of negative results published in non-Medline journals. For example, of the many hundreds of trials reported at the third World Conference on Acupuncture [4], we wereunable to locate any studies originating in East Asia that showed acupuncture to
be equal or inferior to a control procedure.
2. The abstracts may not have accurately reflected the results of trials. Pocock et al, for example, found that abstracts were much more likely to mention endpoints that showed statistical significance than those that did not [5]. The differences among countries are large. We find it unlikely that overenthusiastic reporting in abstracts could be solely responsible for such differences, especially because a positive abstract could not describe a clearly negative trial. Our figure from the second study of 75% of English trials favoring the test treatment is identical to that reported [6] for trials published in the Lancet and similar to that reported by Easterbrook [7] for research conducted in Oxford in the mid-1980s. This gives support to our methods.
3. Our judgements of whether the test treatment was superior to control were, in some cases, subjective. Where the authors of a paper failed to provide an explicit statement of their conclusions or a statistical analysis, we made a judgement of outcome that was, arguably, subjective. We reanalyzed the Chinese data from the second study and found that in only 5 of the 109 studies was such a judgment made. Excluding these studies would not appreciably change the results: 103 positive results from 104 trials is little different from 108 out of 109.
4. Trials conducted in certain countries may involve more outcome measures and “data dredging.” Both of these maneuvers would increase the chance that the study met our criterion that the acupuncture intervention be statistically superior to the control intervention for at least one outcome measure. As a check, we conducted an informal post hoc review of our data; however, we saw no evidence that trials from countries we found to produce exclusively positive results reported numerous outcome measures more often than did those from other countries. Moreover, we did not find appreciable numbers of papers with “mixed statistics,” where there were differences between some outcome measures but not others...
5. Trials may have been conducted with different levels of methodologic rigor. Schulz et al [8] have observed that methodologically rigorous trials show smaller differences between experimental and control groups than do those conducted with less rigor. If trials from certain countries involved, say, insufficient blinding or inadequate concealment or randomization, this might explain, at least in part, the greater proportion of positive trials. If our results are due to low methodologic rigor, then the implications for systematic review are not as obvious as they might appear. A discerning reader can often, but not always, identify poor methodology. For example, the method of allocation concealment is unclear in the great majority of studies [8, 9]. Most reports state only that random assignment was used but do not give sufficient information to allow a judgment of whether or not it was conducted properly...
6. Publication bias may be greater in some countries than in others. Publication bias is the tendency for individuals to submit or publish trials depending on the direction or strength of the findings. Dickersin [6], for example, found that clinical trials were much more likely to be published if there were statistically significant differences among treatment groups. If there were national variations in publication bias, one would expect that the overall proportion of positive trials wouldtend to be higher in countries with the greatest publication bias. If our results were indeed due to publication bias, there would be a number of implications for the science of systematic review. The usual method by which systematic reviewers circumvent publication bias is to undertake extensive searches for unpublished material. It is unclear whether this would be feasible in China and Russia/USSR. Furthermore, if the lack of negative findings is evidence that trials conducted in these countries are insufficiently rigorous, such extensive literature-searching may not be a good use of resources, particularly given the very high costs of translation."
"To determine whether clinical trials originating in certain countries always have positive results. Data sources: Abstracts of trials from Medline (January 1966–June 1995)...Research conducted in certain countries was uniformly favorable to acupuncture; all trials originating in China, Japan, Hong Kong, and Taiwan were positive, as were 10 out of 11 of those published in Russia/USSR. In studies that examined interventions other than acupuncture, 405 of 1100 abstracts met the inclusion criteria. Of trials published in England, 75% gave the test treatment as superior to control. The results for China, Japan, Russia/USSR, and Taiwan were 99%, 89%, 97%, and 95%, respectively. No trial published in China or Russia/USSR found a test treatment to be ineffective.
...
The 330 most recent randomized or controlled clinical trials published in England were retrieved as a comparison group.
Results of Controlled Clinical Trials of Acupuncture by Country of Research: Percentage Favoring Test Treatment:
USA 53%
Seden 59%
UK 60%
China 100%
Particularly high rates of positive results were seen in China (99%) and Russia/USSR (97%). These two countries published no trials in which the test treatment was not reported effective. In the one study published in Russia/USSR classed as “test treatment equal or inferior to control,” nitrendipine was found to be equally effective as standard antihypertensives such as propranolol.
There are several possible/alternative explanations for our findings.
1. The sample of trials may not have been representative. Our search would not have located all published trials. We chose Medline because it provides a sufficiently representative set of the best-quality trials, particularly in acupuncture [2, 3], but trials are published in many journals not indexed by Medline. Possibly, negative trials originating from eastern Europe and Asia are found solely in non-Medline journals. We believe that this is unlikely; Medline might be expected to be a conservative source of information on unconventional therapies. Moreover, anecdotal evidence, such as the results presented at acupuncture conferences, does not suggest any considerable number of negative results published in non-Medline journals. For example, of the many hundreds of trials reported at the third World Conference on Acupuncture [4], we wereunable to locate any studies originating in East Asia that showed acupuncture to
be equal or inferior to a control procedure.
2. The abstracts may not have accurately reflected the results of trials. Pocock et al, for example, found that abstracts were much more likely to mention endpoints that showed statistical significance than those that did not [5]. The differences among countries are large. We find it unlikely that overenthusiastic reporting in abstracts could be solely responsible for such differences, especially because a positive abstract could not describe a clearly negative trial. Our figure from the second study of 75% of English trials favoring the test treatment is identical to that reported [6] for trials published in the Lancet and similar to that reported by Easterbrook [7] for research conducted in Oxford in the mid-1980s. This gives support to our methods.
3. Our judgements of whether the test treatment was superior to control were, in some cases, subjective. Where the authors of a paper failed to provide an explicit statement of their conclusions or a statistical analysis, we made a judgement of outcome that was, arguably, subjective. We reanalyzed the Chinese data from the second study and found that in only 5 of the 109 studies was such a judgment made. Excluding these studies would not appreciably change the results: 103 positive results from 104 trials is little different from 108 out of 109.
4. Trials conducted in certain countries may involve more outcome measures and “data dredging.” Both of these maneuvers would increase the chance that the study met our criterion that the acupuncture intervention be statistically superior to the control intervention for at least one outcome measure. As a check, we conducted an informal post hoc review of our data; however, we saw no evidence that trials from countries we found to produce exclusively positive results reported numerous outcome measures more often than did those from other countries. Moreover, we did not find appreciable numbers of papers with “mixed statistics,” where there were differences between some outcome measures but not others...
5. Trials may have been conducted with different levels of methodologic rigor. Schulz et al [8] have observed that methodologically rigorous trials show smaller differences between experimental and control groups than do those conducted with less rigor. If trials from certain countries involved, say, insufficient blinding or inadequate concealment or randomization, this might explain, at least in part, the greater proportion of positive trials. If our results are due to low methodologic rigor, then the implications for systematic review are not as obvious as they might appear. A discerning reader can often, but not always, identify poor methodology. For example, the method of allocation concealment is unclear in the great majority of studies [8, 9]. Most reports state only that random assignment was used but do not give sufficient information to allow a judgment of whether or not it was conducted properly...
6. Publication bias may be greater in some countries than in others. Publication bias is the tendency for individuals to submit or publish trials depending on the direction or strength of the findings. Dickersin [6], for example, found that clinical trials were much more likely to be published if there were statistically significant differences among treatment groups. If there were national variations in publication bias, one would expect that the overall proportion of positive trials wouldtend to be higher in countries with the greatest publication bias. If our results were indeed due to publication bias, there would be a number of implications for the science of systematic review. The usual method by which systematic reviewers circumvent publication bias is to undertake extensive searches for unpublished material. It is unclear whether this would be feasible in China and Russia/USSR. Furthermore, if the lack of negative findings is evidence that trials conducted in these countries are insufficiently rigorous, such extensive literature-searching may not be a good use of resources, particularly given the very high costs of translation."