Profile cover photo
Profile photo
gwern branwen
2,815 followers
2,815 followers
About
gwern's posts

Post has attachment

Post has attachment
"Meta-assessment of bias in science", Fanelli et al 2017:

"Numerous biases are believed to affect the scientific literature, but their actual prevalence across disciplines is unknown. To gain a comprehensive picture of the potential imprint of bias in science, we probed for the most commonly postulated bias-related patterns and risk factors, in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was overall relatively small. However, we consistently observed a significant risk of small, early, and highly cited studies to overestimate effects and of studies not published in peer-reviewed journals to underestimate them. We also found at least partial confirmation of previous evidence suggesting that US studies and early studies might report more extreme effects, although these effects were smaller and more heterogeneously distributed across meta-analyses and disciplines. Authors publishing at high rates and receiving many citations were, overall, not at greater risk of bias. However, effect sizes were likely to be overestimated by early-career researchers, those working in small or long-distance collaborations, and those responsible for scientific misconduct, supporting hypotheses that connect bias to situational factors, lack of mutual control, and individual integrity. Some of these patterns and risk factors might have modestly increased in intensity over time, particularly in the social sciences. Our findings suggest that, besides one being routinely cautious that published small, highly-cited, and earlier studies may yield inflated results, the feasibility and costs of interventions to attenuate biases in the literature might need to be discussed on a discipline-specific and topic-specific basis.

The bias patterns most commonly discussed in the literature,
which are the focus of our study, include the following:
1. Small-study effects: Studies that are smaller (of lower precision) might report effect sizes of larger magnitude. This phenomenon could be due to selective reporting of results or to genuine heterogeneity in study design that results in larger effects being detected by smaller studies (17).
2. Gray literature bias: Studies might be less likely to be published if they yielded smaller and/or statistically nonsignificant effects and might be therefore only available in PhD theses, conference proceedings, books, personal communications, and other forms of "gray" literature (1).
3. Decline effect: The earliest studies to report an effect might overestimate its magnitude relative to later studies, due to a decreasing field-specific publication bias over time or to differences in study design between earlier and later studies (1, 18). Early-extreme: An alternative scenario to the decline effect might see earlier studies reporting extreme effects in any direction, because extreme and controversial findings have an early window of opportunity for publication (19).
4. Citation bias: The number of citations received by a study might be correlated to the magnitude of effects reported (20).
5. US effect: Publications from authors working in the United States might overestimate effect sizes, a difference that could be due to multiple sociological factors (14).
6. Industry bias: Industry sponsorship may affect the direction and magnitude of effects reported by biomedical studies (21). We generalized this hypothesis to nonbiomedical fields by predicting that studies with coauthors affiliated to private companies might be at greater risk of bias.

The prevalence of these phenomena across multiple meta-analyses can be analyzed with multilevel weighted regression analysis (14) or, more straightforwardly, by conducting a second-order meta-analysis on regression estimates obtained within each meta-analysis (32). Bias patterns and risk factors can thus be assessed across multiple topics within a discipline, across disciplines or larger scientific domains (social, biological, and physical sciences), and across all of science.
To test these hypotheses, we searched for meta-analyses in each of the 22 mutually exclusive disciplinary categories used by the Essential Science Indicators database, a bibliometric tool that covers all areas of science and was used in previous large-scale studies of bias (5, 11, 33). These searches yielded an initial list of over 116,000 potentially relevant titles, which through successive phases of screening and exclusion yielded a final sample of 3,042 usable metaanalyses (Fig. S1). Of these, 1,910 meta-analyses used effect-size metrics that could all be converted to log-odds ratio (n = 33,355 nonduplicated primary data points),

Bias Patterns. Bias patterns varied substantially in magnitude as well as direction across meta-analyses, and their distribution usually included several extreme values (Fig. S2; full numerical results in Dataset S1). Second-order meta-analysis of these regression estimates yielded highly statistically significant support for the presence of small-study effects, gray literature bias, and citation bias (Fig. 1 A and B). These patterns were consistently observed in all secondary and robustness tests, which repeated all analyses not adjusting for study precision, standardizing metaregression estimates and not coining the meta-analyses or coining them with different thresholds (see Methods for details and all numerical results in Dataset S2).
The decline effect, measured as a linear association between year of study publication and reported effect size, was not statistically significant in our main analysis (Fig. 1B), but was highly significant in all robustness tests. Moreover, secondary analyses conducted with the multilevel regression approach suggest that most or all of this effect might actually consist of a "first-year" effect, in which the decline is not linear and just the very earliest studies are likely to overestimate findings (SI Multilevel MetaRegression Analysis, Multilevel Analyses, Secondary Tests of Early Extremes, Proteus Phenomenon and Decline Effect).
The early-extreme effect was, in most robustness tests, marginally significant in the opposite direction to what was predicted, but was measured to high statistical significance in the predicted (i.e., negative) direction when not adjusted for smallstudy effects (Dataset S2). In other words, it appears to be true that earlier studies may report extreme effects in either direction, but this effect is mainly or solely due to the lower precision of earlier studies.
The US effect exhibited associations in the predicted direction and was marginally significant in our main analyses (Fig. 1B) and was significant in some of the robustness tests, particularly when meta-analysis coining was done more conservatively (Dataset S2; see Methods for further details).
Industry bias was absent in our main analyses (Fig. 1B) but was statistically significant when meta-analyses were coined more conservatively (Dataset S2).
Standardizing these various biases to estimate their relative importance is not straightforward, but results using different methods suggested that small-study effects are by far the most important source of potential bias in the literature. Second-order meta-analyses of standardized meta-regression estimates, for example, yield similar results to those in Fig. 1 (Dataset S2). Calculation of pseudo-R 2 in multilevel regression suggests that small-study effects account for around 27% of the variance of primary outcomes, whereas gray literature bias, citation bias, decline effect, industry sponsorship, and US effect, each tested as individual predictor and not adjusted for study precision, account for only 1.2%, 0.5%, 0.4%, 0.2%, and 0.04% of the variance, respectively (see SI Multilevel Meta-Regression Analysis, Multilevel Analyses, Relative Strength of Biases further details).

The career level of authors, measured as the number of years in activity since the first publication in the Web of Science, was overall negatively associated with reported effect size, although the association was statistically significant and robust only for last authors (Fig. 1F). This finding is consistent with the hypothesis that early-career researchers would be at greater risk of reporting overestimated effects (Table 1).
Gender was inconsistently associated with reported effect size: In most robustness tests, female authors exhibited a tendency to report smaller (i.e., more conservative) effect sizes (e.g., Fig. 1F), but the only statistically significant effect detected among all robustness tests suggested the opposite, i.e., that female first authors are more likely to overestimate effects (Dataset S2).
Scientists who had one or more papers retracted were significantly more likely to report overestimated effect sizes, albeit solely in the case of first authors (Fig. 1F). This result, consistently observed across most robustness tests (Dataset S2), offers partial support to the individual integrity hypothesis (Table 1). The between-meta-analysis heterogeneity measured for all bias patterns and risk factors was high (Fig. 1, Fig. S2, and Dataset S2), suggesting that biases are strongly dependent on contingent characteristics of each meta-analysis. The associations most consistently observed, estimated as the value of between-metaanalysis variance divided by summary effect observed, were, in decreasing order of consistency, citation bias, small-study effects, gray literature bias, and the effect of a retracted first author (Fig. 1, bottom numbers).
Differences Between Disciplines and Domains. Part of the heterogeneity observed across meta-analyses may be accounted for at the level of discipline (Fig. S3) or domain (Fig. 2 and Fig. S4), as evidenced by the lower levels of heterogeneity and higher levels of consistency observed within some disciplines and domains. The social sciences, in particular, exhibited effects of equal or larger magnitude than the biological and the physical sciences for most of the biases (Fig. 2) and some of the risk factors (Fig. S4)."

Post has attachment
More on recent human evolution with large between-country differences due to local adaptations: "Selection in Europeans on Fatty Acid Desaturases Associated with Dietary Changes", Buckley et al 2017

"FADS genes encode fatty acid desaturases that are important for the conversion of short chain polyunsaturated fatty acids (PUFAs) to long chain fatty acids. Prior studies indicate that the FADS genes have been subjected to strong positive selection in Africa, South Asia, Greenland, and Europe. By comparing FADS sequencing data from present-day and Bronze Age (5–3k years ago) Europeans, we identify possible targets of selection in the European population, which suggest that selection has targeted different alleles in the FADS genes in Europe than it has in South Asia or Greenland. The alleles showing the strongest changes in allele frequency since the Bronze Age show associations with expression changes and multiple lipid-related phenotypes. Furthermore, the selected alleles are associated with a decrease in linoleic acid and an increase in arachidonic and eicosapentaenoic acids among Europeans; this is an opposite effect of that observed for selected alleles in Inuit from Greenland. We show that multiple SNPs in the region affect expression levels and PUFA synthesis. Additionally, we find evidence for a gene–environment interaction influencing low-density lipoprotein (LDL) levels between alleles affecting PUFA synthesis and PUFA dietary intake: carriers of the derived allele display lower LDL cholesterol levels with a higher intake of PUFAs. We hypothesize that the selective patterns observed in Europeans were driven by a change in dietary composition of fatty acids following the transition to agriculture, resulting in a lower intake of arachidonic acid and eicosapentaenoic acid, but a higher intake of linoleic acid and α-linolenic acid."

FADS was identified as undergoing soft selective sweeps before, but this digs a little more into the metabolic effects and how different countries can have human races evolving differently due to the local environment, by noting that the Inuit show adaptation in the opposite direction.

Post has attachment
"Learning to Discover Cross-Domain Relations with Generative Adversarial Networks", Kim et al 2017:

"While humans easily recognize relations between data from different domains without any supervision, learning to automatically discover them is in general very challenging and needs many ground-truth pairs that illustrate the relations. To avoid costly pairing, we address the task of discovering cross-domain relations given unpaired data. We propose a method based on generative adversarial networks that learns to discover relations between different domains (DiscoGAN). Using the discovered relations, our proposed network successfully transfers style from one domain to another while preserving key attributes such as orientation and face identity."

To quote the Reddit summary: 'DiscoGANs - Answering the age old question: What would your face look like if it were actually a car.'

Or going by the samples in https://github.com/carpedm20/DiscoGAN-pytorch - 'what would this handbag look like if it was actually a shoe'. Inquiring minds want to know!

Post has attachment
The end of an era: "The workshop will mark the last of the ImageNet Challenge competitions, and focus on unanswered questions and directions for the future."

Remember when the best algorithms could manage maybe 50% and it was going to take decades for researchers to get anywhere near human-parity? Pepperidge Farm remembers.

Post has attachment
More on complex intellectual changes in higher mammals due to recent evolution due to soft sweeps: dog domestication is due to a lot of small changes recently. "Selective sweep analysis using village dogs highlights the pivotal role of the neural crest in dog domestication", Pendleton et al 2017:

"Dogs (Canis lupus familiaris) were domesticated from gray wolves between 20-40kya in Eurasia, yet details surrounding the process of domestication remain unclear. The vast array of phenotypes exhibited by dogs mirror numerous other domesticated animal species, a phenomenon known as the Domestication Syndrome. Here, we use signatures persisting in the dog genome to identify genes and pathways altered by the intensive selective pressures of domestication. We identified 37 candidate domestication regions containing 17.5Mb of genome sequence and 172 genes through whole-genome SNP analysis of 43 globally distributed village dogs and 10 wolves. Comparisons with three ancient dog genomes indicate that these regions reflect signatures of domestication rather than breed formation. Analysis of genes within these regions revealed a significant enrichment of gene functions linked to neural crest cell migration, differentiation and development. Genome copy number analysis identified regions of localized sequence and structural diversity, and discovered additional copy-number variation at the amylase-2b locus. Overall, these results indicate that primary selection pressures targeted genes in the neural crest as well as components of the minor spliceosome, rather than genes involved in starch metabolism. Smaller jaw sizes, hairlessness, floppy ears, tameness, and diminished craniofacial development distinguish wolves from domesticated dogs, phenotypes of the Domestication Syndrome that can result from decreased neural crest cells at these sites. We propose that initial selection acted on key genes in the neural crest and minor splicing pathways during early dog domestication, giving rise to the phenotypes of modern dogs."

The obvious implication here: evolution doesn't stop at the neck.

Post has attachment
A fun retrospective on the 'Paul McCartney is actually dead' Beatles conspiracy theory (https://en.wikipedia.org/wiki/Paul_is_dead) which, if you're like me, you've never heard of before but was apparently quite a thing. Featuring conspiracy theorists literally connecting the dots and seeing stars.

Post has attachment
Continuing the theme of 'everything is heritable' and 'abnormal is normal':"Genome-Wide Analysis Of 113,968 Individuals In UK Biobank Identifies Four Loci Associated With Mood Instability", Ward et al 2017:

"Mood instability is a core clinical feature of affective disorders, particularly major depressive disorder (MDD) and bipolar disorder (BD). It may be a useful construct in line with the Research Domain Criteria (RDoC) approach, which proposes studying dimensional psychopathological traits that cut across diagnostic categories as a more effective strategy for identifying the underlying biology of psychiatric disorders. Here we report a genome-wide association study (GWAS) of mood instability in a very large study of 53,525 cases and 60,443 controls from the UK Biobank cohort, the only such GWAS reported to date. We identified four independent loci (on chromosomes eight, nine, 14 and 18) significantly associated with mood instability, with a common SNP-based heritability estimate for mood instability of approximately 8%. We also found a strong genetic correlation between mood instability and MDD (0.60, SE=0.07, p=8.95x10-17), a small but statistically significant genetic correlation with schizophrenia (0.11, SE=0.04, p=0.01), but no genetic correlation with BD. Several candidate genes harbouring variants in linkage disequilibrium with the associated loci may have a role in the pathophysiology of mood disorders, including the DCC netrin 1 receptor (DCC), eukaryotic initiation factor 2B (EIF2B2), placental growth factor (PGF) and protein tyrosine phosphatase, receptor type D (PTPRD) genes. Strengths of this study include the large sample size; however, our measure of mood instability may be limited by the use of a single self-reported question. Overall, this work suggests a polygenic basis for mood instability and opens up the field for the further biological investigation of this important cross-diagnostic psychopathological trait."

(No polygenic score.)

Post has attachment
Generating 512px photorealistic images & video with PixelCNNs (not GANs), "Parallel Multiscale Autoregressive Density Estimation", Reed et al 2017:

"PixelCNN achieves state-of-the-art results in density estimation for natural images. Although training is fast, inference is costly, requiring one network evaluation per pixel; O(N) for N pixels. This can be sped up by caching activations, but still involves generating each pixel sequentially. In this work, we propose a parallelized PixelCNN that allows more efficient inference by modeling certain pixel groups as conditionally independent. Our new PixelCNN model achieves competitive density estimation and orders of magnitude speedup - O(log N) sampling instead of O(N) - enabling the practical generation of 512x512 images. We evaluate the model on class-conditional image generation, text-to-image synthesis, and action-conditional video generation, showing that our model achieves the best results among non-pixel-autoregressive density models that allow efficient sampling."

DeepMind demonstrates again why it loves convolutions by leaping past StackGAN in not just generating large photorealistic images, but also video.

Animation of drawing images: https://twitter.com/scott_e_reed/status/841098907887235076 https://twitter.com/scott_e_reed/status/841099231666544640 https://twitter.com/scott_e_reed/status/841099334535979008

Samples: https://twitter.com/scott_e_reed/status/841094775528968192

Generated video samples: https://twitter.com/scott_e_reed/status/841100941445206016 https://twitter.com/scott_e_reed/status/841101008205938693 https://twitter.com/scott_e_reed/status/841101083334328321

Discussion: https://www.reddit.com/r/MachineLearning/comments/5z2jdn/r_parallel_multiscale_autoregressive_density/

Looking forward to trying this out on anime images.

Post has attachment
Wait while more posts are being loaded