I'd suggest that the situation can be improved with a combination of changes to journals, publication and CV culture.

Currently the options for publishing are essentially limited to a main journal or Nature or Science, the latter two being accorded greater prestige. Not sure about Science, but Nature is no longer held in particularly high regard by many astronomers. Still, while the idea of assessing the quality of research is largely trying to quantify the unquantifiable, perhaps we can at least try and quantify things in a better way than in the current system.

Research falls into various different categories. Some studies are purely observational catalogues designed for future work - they present and measure data, but say nothing themselves about what they mean. Other papers are the opposite, using no new data collected by the authors themselves but rely purely on other people's previous work. Many are a mix of the two. Some papers which do try and interpret the data do so from a pure observational perspective while others use nothing but analytic or numerical modelling, while a few use both. And then there are these "replication studies" (not sure that's the best term) which deliberately try and see if the previous conclusions stand up to a repeat analysis - usually using new methods or different data rather than literally replicating exactly what the previous team did.

Currently journals do not distinguish these (or other) different sorts of research in any way. A published paper is a published paper, end-of. OK, many journals also publish letters (short, usually slightly more important/urgent findings) as well as main articles, but that's it. A few journals are slightly more suitable for catalogues as opposed to new theories, but there's no strict demarcation as to which journal to publish different sorts of studies in.

But perhaps there should be - or if an entirely new journal is too much, perhaps there should be different divisions within journals. E.g. there's MNRAS and MNRAS Letters, why not also MNRAS Catalogues, MNRAS Modelling, MNRAS New Ideas I Just Thought Up And Would Very Much Appreciate It If Someone Else Could Test For Me, Thanks. In this way it would be easier to look at an author's CV and determine not just how much research they do, but what sort - are they mainly collecting and cataloguing data, thinking up new interpretations, testing previous research, lots of different things, what ? A wise institute will hire people with a diverse range of skills, not just the ones who publish the most papers of any type. And it will hire some of the extremes - people who only do observations, only simulations - as well as from the more usual middle ground.

Labelling the research won't help without a corresponding change in how research is valued, e.g. how much it matters on your CV. All the different sorts of research is valuable, but a finding which has been replicated is much more significant. Far from being the least important, as in, "let's check just to make sure", it should be subjected to the strictest form of peer review. A paper verified by and independent replication study should be held in much higher regard than one which hasn't (of course some findings can't be practically replicated - e.g. no-one's going to repeat a project that took five years to complete, so let's not go nuts with this).

At the same time, stifling novel ideas should be the last thing anyone wants. A good researcher is probably not one whose every paper is verified - that probably means they just haven't had any interesting ideas. You want a mixture, say, 50%. Vigilance in the peer review system would stop people from gaming it, e.g. by deliberately publishing a mixture of mediocre and crackpot research. However, the notion that only verified findings matter needs to be broken. Yes, if a paper repeatedly fails to stand up to scrutiny that line of inquiry should be abandoned - but that doesn't mean the idea wasn't a good one at the time.

Maybe all this will even help with the silly grant systems which are in place that assess projects based on number of papers. If a project produces five papers which contain new ideas but no actual independently replicated findings, maybe that project isn't as good as one which produced three papers with a mixture of observation, theory and interpretation. Or then again maybe we should just end the silly grant system entirely, because it's silly.
On p-hacking, and -- on the bright side -- what scientists are doing about it.
Shared publiclyView activity