Public
Reproducing results between biology labs can be really hard:
> Improved reproducibility often comes from pinning down methods. Scientists studying autophagy — the process by which cells remove degraded components — have coordinated efforts to craft and update extensive guidelines on, for instance, how to quantify that a component has been engulfed or how to verify that a gene is involved in the process. In another, now-famous example, two cancer labs spent more than a year trying to understand inconsistencies. It took scientists working side by side on the same tumour biopsy to reveal that small differences in how they isolated cells — vigorous stirring versus prolonged gentle rocking — produced different results.
> Subtle tinkering has long been important in getting biology experiments to work. Before researchers purchased kits of reagents for common experiments, it wasn't unheard of for a team to cart distilled water from one institution when it moved to another. Lab members would spend months tweaking conditions until experiments with the new institution's water worked as well as before. [...]
> In one particularly painful teleconference [between three researchers], we spent an hour debating the proper procedure for picking up worms and placing them on new agar plates. Some batches of worms lived a full day longer with gentler technicians. Because a worm's lifespan is only about 20 days, this is a big deal. Hundreds of e-mails and many teleconferences later, we converged on a technique but still had a stupendous three-day difference in lifespan between labs. The problem, it turned out, was notation — one lab determined age on the basis of when an egg hatched, others on when it was laid.
> We decided to buy shared batches of reagents from the start. Coordination was a nightmare; we arranged with suppliers to give us the same lot numbers and elected to change lots at the same time. We grew worms and their food from a common stock and had strict rules for handling. We established protocols that included precise positions of flasks in autoclave runs. We purchased worm incubators at the same time, from the same vendor. [...] After more than a year of pilot experiments and discussion of methods in excruciating detail, we almost completely eliminated systematic differences in worm survival across our labs.
> Improved reproducibility often comes from pinning down methods. Scientists studying autophagy — the process by which cells remove degraded components — have coordinated efforts to craft and update extensive guidelines on, for instance, how to quantify that a component has been engulfed or how to verify that a gene is involved in the process. In another, now-famous example, two cancer labs spent more than a year trying to understand inconsistencies. It took scientists working side by side on the same tumour biopsy to reveal that small differences in how they isolated cells — vigorous stirring versus prolonged gentle rocking — produced different results.
> Subtle tinkering has long been important in getting biology experiments to work. Before researchers purchased kits of reagents for common experiments, it wasn't unheard of for a team to cart distilled water from one institution when it moved to another. Lab members would spend months tweaking conditions until experiments with the new institution's water worked as well as before. [...]
> In one particularly painful teleconference [between three researchers], we spent an hour debating the proper procedure for picking up worms and placing them on new agar plates. Some batches of worms lived a full day longer with gentler technicians. Because a worm's lifespan is only about 20 days, this is a big deal. Hundreds of e-mails and many teleconferences later, we converged on a technique but still had a stupendous three-day difference in lifespan between labs. The problem, it turned out, was notation — one lab determined age on the basis of when an egg hatched, others on when it was laid.
> We decided to buy shared batches of reagents from the start. Coordination was a nightmare; we arranged with suppliers to give us the same lot numbers and elected to change lots at the same time. We grew worms and their food from a common stock and had strict rules for handling. We established protocols that included precise positions of flasks in autoclave runs. We purchased worm incubators at the same time, from the same vendor. [...] After more than a year of pilot experiments and discussion of methods in excruciating detail, we almost completely eliminated systematic differences in worm survival across our labs.
If the results are that fiddly, it sure makes you wonder how robust they are in the first place.2w
The author discusses this for a bit at the end:
> Our data revealed another, startling source of variation. Even in a single lab performing apparently identical experiments, we could not eliminate run-to-run differences. After dozens of experiments, we realized that some cohorts of worms could partition into one of two modes of ageing: short-lived or long-lived. The reason so far is not clear, but the discovery shows that, if we want to understand a chemical compound's effects, we need to repeat more experiments than we realized. [...]
> It is a rare project that specifies methods with such precision. In fact, several mouse researchers have argued that standardization is counterproductive — better to focus on results that persist across a wide range of conditions than to chase fragile findings that occur only within narrow parameters.
> We argue that another way forward is to chase down the variability and try to understand it within a common environment. We are now working together to search for molecular differences that distinguish short-lived and long-lived batches of worms within the same strain, a phenomenon we never could have uncovered had we not eliminated nearly all other sources of variability.2w
