{"id":809149,"date":"2026-05-20T05:36:23","date_gmt":"2026-05-20T05:36:23","guid":{"rendered":"https:\/\/www.europesays.com\/us\/809149\/"},"modified":"2026-05-20T05:36:23","modified_gmt":"2026-05-20T05:36:23","slug":"the-uncritical-adoption-of-ai-in-science-is-alarming-we-urgently-need-guard-rails","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/us\/809149\/","title":{"rendered":"The uncritical adoption of AI in science is alarming \u2014 we urgently need guard rails"},"content":{"rendered":"\n

The scientific community is adopting artificial-intelligence tools, especially large language models (LLMs), at an astonishing speed. LLM-assisted paper writing has drastically increased over the past three years1<\/a> and researchers have sought to incorporate semi-autonomous agents into their workflows<\/a>. However, the rapid and uncritical adoption of AI in science comes with significant risks2<\/a>. Several problems are already apparent: papers that use AI tools focus on a narrower set of established research questions3<\/a>, and in some cases have been evaluated to have less scientific merit4<\/a>, than do studies that do not rely on AI.<\/p>\n

\"\"<\/p>\n

Why an overreliance on AI-driven modelling is bad for science<\/p>\n

<\/a><\/p>\n

Moreover, as AI automates routine scientific tasks, concerns about the erosion of training opportunities for early\u2011career researchers remain largely unresolved and often unacknowledged. Conventionally, scientific training has combined formal instruction in facts and methods with the gradual acquisition of tacit knowledge through hands\u2011on, entry\u2011level work. Scholars in science and technology studies have repeatedly demonstrated that scientific texts alone do not fully communicate knowledge; instead, crucial know\u2011how is embedded in research communities and transmitted through apprenticeship and practice5<\/a>.<\/p>\n

This tacit knowledge \u2014 for example, of what constitutes \u2018reasonable\u2019 data, or the details of a technique that are difficult to articulate in a methods section, or whether a result is consistent with the existing literature \u2014 is essential if a researcher is to supervise AI\u2011assisted workflows effectively in the future. If AI systems increasingly replace entry\u2011level scientific labour, trainees might never develop these skills, potentially leaving the next generation of scientists ill prepared to oversee AI\u2011driven research responsibly.<\/p>\n

These trends demand that scientists reckon with the purpose of scientific institutions. Is our goal simply to build a collection of scientific facts, or to also cultivate a living, evolving community of scientific knowers? If AI tools accelerate the former goal while threatening the latter, how should scientists proceed? Drawing on emerging evidence about AI\u2019s effects on scientific practice, here we highlight key risks and outline potential remedies.<\/p>\n

More output, less understanding<\/p>\n

The AI industry has aggressively marketed LLM products to scientists as technologies for increasing productivity. Some researchers have embraced this promise, touting these products\u2019 ability to \u2018supercharge\u2019 their academic writing6<\/a>.<\/p>\n

\"\"<\/p>\n

Is AI leading to a reproducibility crisis in science?<\/p>\n

<\/a><\/p>\n

If we take productivity to mean the output of scientific papers, AI products have undoubtedly delivered on the promise1<\/a>,7<\/a>, with far-reaching effects. Online preprint repositories such as SocArXiv and PsyArXiv imposed temporary moratoriums on AI papers or updated their reviewing policies; arXiv changed its policy and no longer accepts computer-science position papers (which are relatively easy for LLMs to churn out), and the US National Institutes of Health began limiting grant proposals to six per year per principal investigator (PI).<\/p>\n

Alongside increased productivity, there is evidence that scientists who use AI tools in their work receive more citations and advance more quickly to PI roles than do those who do not3<\/a>, with people who have English as their first language benefiting disproportionately8<\/a>. It\u2019s noteworthy, however, that the career benefits of AI products are most apparent in quantitative metrics, such as publication and citation counts. Such measures cannot account for whether these benefits accrue because of scientific contribution or as a result of connections with a hyped field. It would therefore be a mistake to take these metrics as representing a win for science, without looking more closely at the quality of this increased productivity.<\/p>\n

\"A<\/p>\n

A robot learns to emulate human behaviour at a facility in Hefei, China.Credit: Zhang Dagang\/VCG via Getty<\/p>\n

More is not necessarily better, and there is growing evidence that LLM-driven increases in scientific productivity often come at a cost. Most obvious is the introduction of \u2018AI slop\u2019 into the literature, including nonsensical AI-generated images and hallucinated citations.<\/p>\n

Multiple studies have documented troubling trends in papers that heavily rely on LLM assistance in writing. The journal Organization Science audited all of the 6,957 submissions it received from January 2021 to January 2026, and observed that LLM-assisted papers had poorer scientific quality (as measured by the likelihood of these works being accepted by the journal)7<\/a>. Another study4<\/a>, of 264,125 papers gathered from a 2024 AI conference and submissions posted across three preprint servers (arXiv, bioRxiv and SSRN) in 2023\u201324, found that in LLM-assisted papers, good writing ceased to be an accurate heuristic for scientific quality (as measured by publication outcome and peer-review scores).<\/p>\n

\"\"<\/p>\n

Can AI have common sense? Finding out will be key to achieving machine intelligence<\/p>\n

<\/a><\/p>\n

The authors conclude that this creates \u201ca risk for the scientific enterprise, as a deluge of superficially convincing but scientifically underwhelming research could saturate the literature\u201d. Currently, there are no clear solutions to these pollutions of our knowledge ecosystem, despite agreement that they should not be happening.<\/p>\n

AI tools implemented across the research pipeline are also limiting scientific understanding in more subtle ways, for example, by shrinking the set of concerns that researchers explore. An analysis3<\/a> of 41.3 million research papers spanning biology, medicine, chemistry, physics, materials science and geology shows that the adoption of AI seems to \u201cinduce authors to converge on the same solutions to known problems rather than create new ones\u201d. This is an early warning of the likelihood that AI will have cascading impacts on the knowledge ecosystem2<\/a>.<\/p>\n

Threat of deskilling<\/p>\n

The loftiest visions of AI for science involve semi- or fully autonomous \u2018AI scientists\u20199<\/a>, and recent prototypes of this vision have received great fanfare in science and industry<\/a> alike. Developers of AI scientists are quick to reassure that their intention is \u201cnot to replace human scientists, but to augment and accelerate their work\u201d10<\/a>. These products, then, are meant to be overseen by a skilled scientist who can responsibly direct and check the AI\u2019s output.<\/p>\n

We question whether this vision can come to pass without a clear articulation of how the responsible scientist can develop the expertise to distinguish between good and poor AI outcomes. Much of the enthusiasm for AI tools across the scientific pipeline comes from their promise to offload work. But many \u2018low-skilled\u2019 tasks have conventionally been important starting points for trainee scientists.<\/p>\n","protected":false},"excerpt":{"rendered":"The scientific community is adopting artificial-intelligence tools, especially large language models (LLMs), at an astonishing speed. LLM-assisted paper…\n","protected":false},"author":3,"featured_media":809150,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_share_on_mastodon":"0"},"categories":[8],"tags":[26265,10046,8523,40515,10047,159,67,132,68],"class_list":["post-809149","post","type-post","status-publish","format-standard","has-post-thumbnail","category-science","tag-computer-science","tag-humanities-and-social-sciences","tag-machine-learning","tag-mathematics-and-computing","tag-multidisciplinary","tag-science","tag-united-states","tag-unitedstates","tag-us"],"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@us\/116605288756718544","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/809149","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/comments?post=809149"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/809149\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media\/809150"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media?parent=809149"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/categories?post=809149"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/tags?post=809149"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}