Copying over an email of mine to the GRG as a perhaps interesting bit of criticism:

On Fri, Dec 6, 2013 at 5:50 PM,  <regene@hushmail.com> wrote:
> http://www.eurekalert.org/pub_releases/2013-12/nrr-pam120513.php
> ...
> "Pre-moxibustion and moxibustion prevent Alzheimer's disease"
> ...
> http://www.sjzsyj.org/CN/article/downloadArticleFile.do?attachType=PDF&id=754

I don't believe this for a second. But actually, this would be a nice
followup to my previous email about the problems in animal research:
this paper exhibits all the problems mentioned, and more. Let's do a
little critique here.

1. This paper is Chinese research performed in China by an all-Chinese
team. The current state of Chinese research is bad. It's really bad.
Some reading on the topic:

- http://www.wired.co.uk/news/archive/2013-12/02/china-academic-scandal
- http://newhumanist.org.uk/2365/lies-damn-lies-and-chinese-science
- http://news.bbc.co.uk/2/hi/8448731.stm
- http://www.gwern.net/docs/dnb/2010-zhang.pdf
- http://www.nytimes.com/2010/10/07/world/asia/07fraud.html
- http://news.bbc.co.uk/2/hi/asia-pacific/4755861.stm
- http://news.bbc.co.uk/2/hi/asia-pacific/8442147.stm
- http://www.nature.com/news/2010/100112/full/463142a.html
- https://www.sciencenews.org/view/generic/id/330930/title/Traditional_Chinese_medicine_Big_questions
- http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0020185
- http://www.npr.org/2011/08/03/138937778/plagiarism-plague-hinders-chinas-scientific-ambition

I will note that there have been statistical anomalies in some of the
Chinese papers on dual n-back training I have used in my
meta-analysis, so I have some personal experience in the topic.
2. 'traditional medicine' research is really bad no matter where you
go. They mention acupuncture as justification? That's just fantastic.
https://en.wikipedia.org/wiki/Acupuncture punctures some of the
hyperbolic claims, the PLOS link above deals with the poor-quality of
the Chinese reviews & meta-analyses in general, and Cochrane is not
too kind to acupuncture:
http://www.thecochranelibrary.com/details/collection/691705/Acupuncture-ancient-tradition-meets-modern-science.html
And in many of the reviews/meta-analyses there are stark geographic
differences where the East Asian studies turn in tons of positive
results while the Western studies somehow... don't.
3. The lead author is not an ordinary neuroscientist or doctor, but
works at the "College of Acupuncture and Moxibustion". Is he really
going to publish a study concluding "moxibustion does not affect
Alzheimer's"⸮ *Really*⸮
4. Does this claim even make sense? Moxibustion, really⸮ For those not
familiar, https://en.wikipedia.org/wiki/Moxibustion entails

    > Suppliers usually age the mugwort and grind it up to a fluff;
practitioners burn the fluff or process it further into a cigar-shaped
stick. They can use it indirectly, with acupuncture needles, or burn
it on the patient's skin.

    How on earth is this supposed to help AD? How does burning a plant
on your skin affect plaques in your brain? Or if they use acupuncture
needles, how plausible is it that a few milligrams at most of mugwort
inserted into the skin would do anything? While Wikipedia is not
Cochrane or anything, it is troubling that this entry lists no useful
application of moxibustion. And then it goes and links to "Does
moxibustion work? An overview of systematic reviews"
https://www.biomedcentral.com/1756-0500/3/284 which finds that

    > Ten SRs met our inclusion criteria, which related to the
following conditions: cancer, ulcerative colitis, stroke
rehabilitation, constipation, hypertension, pain conditions and breech
presentation. Their conclusions were contradictory in several
instances. Relatively clear evidence emerged to suggest that
moxibustion is effective for breech presentation.

    That review also mentions, incidentally, that

    > Many of the primary moxibustion trials originate from China
(data not shown); Vickers et al. demonstrated that virtually 100% of
Chinese acupuncture trials are positive [
http://www.dcscience.net/Vickers_1998_Controlled-Clinical-Trials.pdf
], which seems to be equally applied to moxibustion, an
acupuncture-like intervention. This casts considerable doubt on the
reliability of these studies.

Alright, so let's take stock here. Without ever looking beyond the
title and authorship, we have found that this is a paper from a
country with infamously bad research, in a field with infamously bad
research quality, led by a researcher with considerable inherent
conflict of interest, using a technique/substance which has already
been linked with biased research, on a hypothesis that is grossly
implausible. Based on all these base rates, we can say that there is
basically zero chance this result will ever replicate, much less to
other mice strains or even to humans.

It seems unfair to reject the paper out of hand, though, so let's look
at the actual paper a little.

> Forty healthy rats were randomly divided into four groups: control group, model group, moxibustion group and pre-moxibustion group. The latter three groups were treated with intracerebral injection of Aβ1–42 to establish an AD-like pathology. The moxibustion group received suspended moxibustion on Baihui and Shenshu acupoints for 14 days after Aβ1–42 injection. The pre-moxibustion group was treated with moxibustion for eight courses (each course lasting for 6 days) prior to the exposure and 14 days after Aβ1–42 exposure. The final analysis incorporated all rats.

From the materials and methods:

> Male Wistar rats (12 months old; 500 ± 20 g), of specific pathogen free grade, were obtained from the Experimental Animal Center of Huazhong University of Science and Technology (Wuhan, China), with license No. SCXK (E) 2008-0005.

> After the hair around the acupoints was shaved, an ignited moxa-stick (diameter 6 mm; Nanyang Shennong Aaicao Appliance Company, Nanyang, Henan  China; a round long stick made of moxa floss, also called moxa roll), was suspended perpendicularly 2 cm above the acupoints. Baihui (located in the middle of the parietal bone[50]) and Shenshu (located under the second lumbar on both sides [50]) acupoints were simultaneously given suspended moxibustion. Each treatment consisted of a 15-minute moxibustion, keeping the spot warm and red but not burnt. Generally, the skin temperature was kept at 43 ± 1° during the moxibustion procedure.

Right away we can spot 3 of the usual animal research methodological problems:

1. the sample size is too small - at n=10 rats in each group, you are
not going to detect anything without large effect sizes. It is
implausible that suspended moxa has any effects, and it is especially
implausible that the effect sizes would be large.
2. there is no mention of blinding. The technicians or research
assistants or whomever clearly know which mice they are dealing with.
3. there is mention of randomization, but it's not specified how the
randomization was done, which means it probably was done by the 'stick
your hand in and grab' method, and probably does not balance by litter
or other variables. This massively worsens the power problem, see
"Design, power, and interpretation of studies in the standard murine
model of ALS" http://www.researchals.org/uploaded_files/ALS%202008%209%204.pdf

(I'm a little curious about whether they really started with 10 mice
in each group: the mice spent at least 60 days in the experiment and I
wonder how many, out of 40, you would expect to die in that time
period, especially after you've done your level best to give 3/4s of
them Alzheimer's disease during that time.)

I also note that the moxibustion situation is even worse than I
thought: they did not use acupuncture needles to get some mugwort into
the mice, they did not put any moxa/mugwort in physical contact, but
instead the burning was 2cm away from the mice! The mechanism was bad,
but it just got worse.

There's no mention of the data being provided anywhere at all, either
their website or the publisher; there's some evidence that providing
access to a paper's data correlates with higher-quality research, so I
mention this absence. It also makes it harder for me to do anything
more complex like a post hoc power analysis.

Moving on, they list as dependent variables:

- Morris water maze navigation test
- Morris water maze spatial probe test
- apoptosis rate of hippocampal neurons

Let's look at the stats a bit.

1. Significance: The paper lists no less* than 14 p-values (4 < 0.05,
the rest < 0.01), and for all of them uses an alpha of 0.05. The
smallest given constraint is p<0.01. A Bonferroni correction on this
would be 0.05/14 (since they must have done at least 14 tests to
report 14 p-values), which means an alpha of 0.003571. But 0.01 >
0.05/14, so the 4 0.05 p-values disappear under multiple correction
and probably most of the 0.01s would too.

    * this is a lower bound since the Morris diagrams report something
like 20 p-values themselves, but I didn't feel like carefully parsing
the text to figure out exactly how many p-values are being reported
2. Effect sizes: no tables are provided, but figure 2 (the second
Morris maze test) is illustrative. The control mice have no problem
remember where the platform used to be, and so spend almost half a
minute (~24s) in the right area searching for it. Makes sense, they
don't have AD. The AD mice have terrible memory, and so only spend ~6s
in the right area and most of their time in the wrong place. Also
makes sense. Now, what about the AD mice who had some moxa burnt 2cm
away from their skin? They spend 14-16s or more than twice and
almost 3 times as much as the non-moxa AD mice! And the claimed
standard error on all 4 group of mice's time is tiny, maybe 1s
eyeballing the graph. So they are claiming, in this point, to have an
effect size on memory of something like d = (15-6)/1 = 9. Insane!
From burning some mugwort 2cm away from the mice's skin‽
3. Power: actually, that result shows an example of what I mean by the
result being absurd. Let's calculate what that effect size implies for
the power of their t-test comparing the model AD mice with the moxa AD
mice. So let's say the 2 moxa groups equate to n=20 15(1.5), and the
AD controls were then n=10 5(0.5). The pooled standard deviation of
the non-moxa and moxa mice is ((20-1)*1.5 + (10-1)*0.5) / (20 + 10 -
2) = 1.179, so the effect size was actually d=(15-5)/1.179 = 8.482.
With 20 mice in 1 group and 10 mice in the other, an alpha of 0.05,
then our power turns out to be...

         library(pwr)
         pwr.t2n.test(n1 = 20, n2 = 10, d = 8.482, sig.level = 0.05)
              t test power calculation

                      n1 = 20
                      n2 = 10
                       d = 8.482
               sig.level = 0.05
                   power = 1

    A power of 100%. Absurd. Have you ever seen animal research (or
Alzheimer's research...) with such high power? Real effects, real
treatments, in large clinical trials or in meta-analyses, are hardly
ever that high.

So. I don't know how they got the results they got. Did they
administer dozens of tests until they got the results they wanted? Did
they simply make up the data like so many Chinese academics have? Or
did they start with 30 mice in each group and cherrypick the
best/worst 10? Did they abuse the model AD mice to make the AD+moxa
mice look good?

In conclusion: this paper is complete bullshit, will not replicate,
and regene, it would be nice if you applied an iota of critical
thinking before forwarding us such worthless research. Even if I don't
spend 2 hours deconstruct each paper you send us, it is still
irritating and dilutes the worthwhile results.
Shared publiclyView activity