"Publication bias in the social sciences: Unlocking the file drawer", Franco et al 2014 https://pdf.yt/d/ZTzcsTMzDeUPKhiq / https://dl.dropboxusercontent.com/u/243666993/2014-franco.pdf / http://sci-hub.org/downloads/ce4a/franco2014.pdf (Media: http://andrewgelman.com/2014/08/28/publication-bias-social-sciences-unlocking-file-drawer2/ http://www.nature.com/news/social-sciences-suffer-from-severe-publication-bias-1.15787 ) This is a cool way of measuring total publication bias: finding a closed-world in which there are records of all studies which were run & without the usual issues of quality/representativeness, from which you can track the exact publication biases for each step of the way. Excerpts:
"We study publication bias in the social sciences by analyzing a known population of conducted studies—221 in total—where there is a full accounting of what is published and unpublished. We leverage TESS, an NSF-sponsored program where researchers propose survey-based experiments to be run on representative samples of American adults. Because TESS proposals undergo rigorous peer review, the studies in the sample all exceed a substantial quality threshold. Strong results are 40 percentage points more likely to be published than null results, and 60 percentage points more likely to be written up. We provide not only direct evidence of publication bias, but also identify the stage of research production at which publication bias occurs—authors do not write up and submit null findings.
Publication bias has been documented in various disciplines within the biomedical (3–9) and social sciences (10–17). One common method of detecting publication bias is replicating a meta-analysis with and without unpublished literature (18). This approach is limited because much of what is unpublished is unobserved. Other methods solely examine the published literature and rely on assumptions about the distribution of unpublished research by, for example, comparing the precision and magnitude of effect sizes among a group of studies. In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20). However, these visualizationbased approaches are sensitive to using different measures of precision w (21, 22) and also assume outcome variables and effect sizes are compa- rable across studies (23). Finally, methods that compare published studies to “grey” literatures (e.g., dissertations, working papers, conference papers, human subjects registries) may confound strength of results with research quality (7). These techniques are also unable to determine whether publication bias occurs at the editorial stage or during the writing stage. Editors and reviewers may prefer statistically significant results and reject sound studies that fail to reject the null hypothesis.
Anticipating this, authors may not write up and submit papers that have null findings. Or, authors may have their own preferences to not pursue the publication of null results.
A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25). Analyses of clinical registries and abstracts submitted to medical conferences consistently find little to no editorial bias against studies with null findings (26–31). Instead, failure to publish appears to be most strongly related to authors’ perceptions that negative or null results are uninteresting and not worthy of further analysis or publication (32–35). One analysis of all IRB-approved studies at a single university over two years found that a majority of conducted research was never submitted for publication or peerreview (36).
Fourth, TESS requires that studies have requisite statistical power, meaning that the failure to obtain statistically significant results is not simply due to insufficient sample size.
The initial sample consisted of the entire online archive of TESS studies as of January 1, 2014 (39). We analyzed studies conducted between 2002 and 2012. We did not track studies conducted in 2013 because there had not been enough time for the authors to analyze the data and proceed through the publication process. [This right-censoring could have been handled through survival analysis.] The 249 studies represent a wide range of social science disciplines (see Table 1). Our analysis was restricted to 221 studies—89% of the initial sample. We excluded seven studies published in book chapters, and 21 studies for which we were unable to determine the publication status and/or the strength of experimental findings (40). The full sample of studies is presented in Table 2; the bolded entries represent the analyzed subsample of studies...We first conducted a thorough online search for published and unpublished manuscripts, and read every manuscript to verify that it relied on data collected through TESS and that it reported experimental results (40). We then emailed the authors of over 100 studies for which we were unable to find any trace of the study and asked what happened to their studies. We also asked authors who did not provide a publication or working paper to summarize the results of their experiments.
The outcome variable distinguishes between two types of unpublished experiments: those prepared for submission to a conference or journal, and those never written up in the first place. It is also possible that papers with null results may be excluded from the very top journals but still find their way into the published literature. Thus, we disaggregated published experiments based on their placement in top-tier or nontop-tier journals (40) (see Table S1 for a list of journal classifications). The results from the majority of TESS studies in our analysis sample have been written up (80%), while less than half (48%) have been published in academic journals.
...While around half of the total studies in our sample were published, only 20% of those with null results appeared in print. In contrast, roughly 60% of studies with strong results and 50% of those with mixed results were published. Although more than 20% of the studies in our sample had null findings, less than 10% of published articles based on TESS experiments report such results. While the direction of these results may not be surprising, the observed magnitude (an approximately 40 percentage point increase in the probability of publication from moving from null to strong results) is remarkably large.
...Estimates from multinomial probit regression models show that studies with null findings are significantly less likely to be written up even after controlling for researcher quality (using the highest quality researcher’s cumulative h-index and the number of publications at the time the study was ran), discipline of the lead author, and the date the study was conducted (see online supplementary text and Table S3).
Why do some researchers choose not to write up null results? To provide some initial explanations, we classified 26 detailed email responses we received from researchers whose studies yielded null results and did not write a paper (see Table S6). Fifteen of these authors reported that they abandoned the project because they believed that null results have no publication potential even if they found the results interesting personally (e.g., “I think this is an interesting null finding, but given the discipline's strong preference for p < .05, I haven't moved forward with it”). Nine of these authors reacted to null findings by reducing the priority of writing up the TESS study and focusing on other projects (e.g., “There was no paper unfortunately. There still may be in future. The findings were pretty inconclusive.”). Perhaps most interestingly, two authors whose studies “didn’t work out” eventually published papers supporting their initial hypotheses using findings obtained from smaller convenience samples."
[Auuggghhh!!!]
"We study publication bias in the social sciences by analyzing a known population of conducted studies—221 in total—where there is a full accounting of what is published and unpublished. We leverage TESS, an NSF-sponsored program where researchers propose survey-based experiments to be run on representative samples of American adults. Because TESS proposals undergo rigorous peer review, the studies in the sample all exceed a substantial quality threshold. Strong results are 40 percentage points more likely to be published than null results, and 60 percentage points more likely to be written up. We provide not only direct evidence of publication bias, but also identify the stage of research production at which publication bias occurs—authors do not write up and submit null findings.
Publication bias has been documented in various disciplines within the biomedical (3–9) and social sciences (10–17). One common method of detecting publication bias is replicating a meta-analysis with and without unpublished literature (18). This approach is limited because much of what is unpublished is unobserved. Other methods solely examine the published literature and rely on assumptions about the distribution of unpublished research by, for example, comparing the precision and magnitude of effect sizes among a group of studies. In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20). However, these visualizationbased approaches are sensitive to using different measures of precision w (21, 22) and also assume outcome variables and effect sizes are compa- rable across studies (23). Finally, methods that compare published studies to “grey” literatures (e.g., dissertations, working papers, conference papers, human subjects registries) may confound strength of results with research quality (7). These techniques are also unable to determine whether publication bias occurs at the editorial stage or during the writing stage. Editors and reviewers may prefer statistically significant results and reject sound studies that fail to reject the null hypothesis.
Anticipating this, authors may not write up and submit papers that have null findings. Or, authors may have their own preferences to not pursue the publication of null results.
A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25). Analyses of clinical registries and abstracts submitted to medical conferences consistently find little to no editorial bias against studies with null findings (26–31). Instead, failure to publish appears to be most strongly related to authors’ perceptions that negative or null results are uninteresting and not worthy of further analysis or publication (32–35). One analysis of all IRB-approved studies at a single university over two years found that a majority of conducted research was never submitted for publication or peerreview (36).
Fourth, TESS requires that studies have requisite statistical power, meaning that the failure to obtain statistically significant results is not simply due to insufficient sample size.
The initial sample consisted of the entire online archive of TESS studies as of January 1, 2014 (39). We analyzed studies conducted between 2002 and 2012. We did not track studies conducted in 2013 because there had not been enough time for the authors to analyze the data and proceed through the publication process. [This right-censoring could have been handled through survival analysis.] The 249 studies represent a wide range of social science disciplines (see Table 1). Our analysis was restricted to 221 studies—89% of the initial sample. We excluded seven studies published in book chapters, and 21 studies for which we were unable to determine the publication status and/or the strength of experimental findings (40). The full sample of studies is presented in Table 2; the bolded entries represent the analyzed subsample of studies...We first conducted a thorough online search for published and unpublished manuscripts, and read every manuscript to verify that it relied on data collected through TESS and that it reported experimental results (40). We then emailed the authors of over 100 studies for which we were unable to find any trace of the study and asked what happened to their studies. We also asked authors who did not provide a publication or working paper to summarize the results of their experiments.
The outcome variable distinguishes between two types of unpublished experiments: those prepared for submission to a conference or journal, and those never written up in the first place. It is also possible that papers with null results may be excluded from the very top journals but still find their way into the published literature. Thus, we disaggregated published experiments based on their placement in top-tier or nontop-tier journals (40) (see Table S1 for a list of journal classifications). The results from the majority of TESS studies in our analysis sample have been written up (80%), while less than half (48%) have been published in academic journals.
...While around half of the total studies in our sample were published, only 20% of those with null results appeared in print. In contrast, roughly 60% of studies with strong results and 50% of those with mixed results were published. Although more than 20% of the studies in our sample had null findings, less than 10% of published articles based on TESS experiments report such results. While the direction of these results may not be surprising, the observed magnitude (an approximately 40 percentage point increase in the probability of publication from moving from null to strong results) is remarkably large.
...Estimates from multinomial probit regression models show that studies with null findings are significantly less likely to be written up even after controlling for researcher quality (using the highest quality researcher’s cumulative h-index and the number of publications at the time the study was ran), discipline of the lead author, and the date the study was conducted (see online supplementary text and Table S3).
Why do some researchers choose not to write up null results? To provide some initial explanations, we classified 26 detailed email responses we received from researchers whose studies yielded null results and did not write a paper (see Table S6). Fifteen of these authors reported that they abandoned the project because they believed that null results have no publication potential even if they found the results interesting personally (e.g., “I think this is an interesting null finding, but given the discipline's strong preference for p < .05, I haven't moved forward with it”). Nine of these authors reacted to null findings by reducing the priority of writing up the TESS study and focusing on other projects (e.g., “There was no paper unfortunately. There still may be in future. The findings were pretty inconclusive.”). Perhaps most interestingly, two authors whose studies “didn’t work out” eventually published papers supporting their initial hypotheses using findings obtained from smaller convenience samples."
[Auuggghhh!!!]
