A piece by Carlton Gyles from a link passed on by dearieme in the comments to this recent post about AI being used to churn out dodgy scientific research. Published in 2015, quite short and well worth reading.
Skeptical of medical science reports?
“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as editor of The New England Journal of Medicine” (1).
More recently, Richard Horton, editor of The Lancet, wrote that “The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness” (2).
6 comments:
But isn't it precisely the job of editors of scientific journals to winnow out the good science from the bad?
I assert that 'science' is fine. It is self correcting over time. Media science reporting is a different thing entirely as there is typical no reporting of experimental design or later retractions... unless that too becomes newsworthy at a later date.
Interesting observation from marking and moderating BTEC level 3 (I.e.equivalent to A level) assignments in the 1990s. The award bodies had quite rightly introduced a requirement that students be graded on their ability to evaluate different sources.
At first, there were some interesting points taught and remembered about validity, reliability, representativeness, and generally. Very encouraging. Later, and not much later, teachers and students had "cracked the code" about what was needed to get high marks. Answers became uniform, ridiculously formulaic, and trite.
People want easy answers. If science declines, it will be for the same reason that everything else does: laziness and corner-cutting.
I once heard a consultant, in the early days of the internet, describing the errors made at administrative level when researchers for a study - or their clerical staff - prioritised their source material by the number of ‘hits’ turned up by an online search, meaning that unsubstantiated or incorrect claims could, with enough re-posting and citation, make it to the top of the list.
The problem was identified and steps were taken to change the procedure; unfortunately it looks as if AI, in inexperienced hands, may be reversing the trend. Remembering how my senior managers reacted when the internet first arrived in school, I’d say there is a grave danger that ‘Little Tin God’ syndrome will have deleterious consequences far beyond those already being felt in academia and medicine.
Science, real science will always progress. Hypothesis, experiment, analysis, repeat experiment, throw it open for knock down.
The problem is that there are now so many causes that tack " science" on to their ramblings to gain credence, without any rigor.
Climate, social, race, emotion, politics, economy,
Vatsmith - journal editors will see it as their job to some extent, but if going against a highly political and fashionable scientific consensus could harm the journal, which it probably would, then survival of the journal may come first.
Sam - I agree, laziness and corner-cutting are enough to cause an immense amount of damage to any academic culture built on an ideal of dispassionate rigour. We see a corrupted form Occam's razor where the consensus narrative is the one which doesn't appear to go against the principle of parsimony because going with the consensus is parsimonious. And lazy.
Macheath - alternatively, AI could become the scapegoat to explain a false consensus when the consensus shows signs of crumbling under the weight of contrary evidence. Yet that ‘Little Tin God’ syndrome could prevail too and seems more likely at the moment.
Doonhamer - I agree, real science will always progress, especially where the advantages of progress outweigh political disadvantages. Progress may be slower where there are political or cultural disadvantages to progress though.
Post a Comment