PaperlayerView exact revision on arXiv

Source-linked AI summary

How should the advent of large language models affect the practice of science?

Marcel Binz, Stephan Alaniz, Adina Roskies, Balazs Aczel, Carl T. Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D. West, Qiong Zhang, Richard M. Shiffrin, Samuel J. Gershman, Ven Popov, Emily M. Bender, Marco Marelli, Matthew M. Botvinick, Zeynep Akata, Eric Schulz

arXiv:2312.03759v1cs.CLcs.AIcs.CYcs.DL

TL;DR

The paper asks how large language models should affect scientific practice amid uncertainty about their implications. It brings four scientist groups into debate and finds shared emphasis on science’s social nature, integrity, and standards despite substantial disagreement.

  • Problem

    The paper examines how LLMs should affect scientific practice as their integration expands while their implications remain incompletely understood.

    Source: AI and scientific labor · p. 7, Response by Emily M. Bender, Carl T.… · p. 9

  • Method

    The authors convene four scientist groups to present perspectives on LLMs in science and respond to one another.

    Source: Conclusion · p. 10

  • Results

    Despite substantial disagreement, all groups emphasize science’s social nature and the importance of protecting scientific integrity and standards.

    Source: Conclusion · p. 10

  • Takeaways & Limitations

    The discussion supports cautious, responsible LLM adoption that preserves human judgment, transparent practice, and existing scientific standards.

    Source: Protecting the past · p. 3, Conclusion · pp. 4–5

    LLM use can threaten reproducibility because proprietary providers may change models without notice, preventing researchers from reproducing earlier results.

    Source: Protecting the past · p. 3

Abstract

from arXiv · show

Large language models (LLMs) are being increasingly incorporated into scientific workflows. However, we have yet to fully grasp the implications of this integration. How should the advent of large language models affect the practice of science? For this opinion piece, we have invited four diverse groups of scientists to reflect on this query, sharing their perspectives and engaging in debate. Schulz et al. make the argument that working with LLMs is not fundamentally different from working with human collaborators, while Bender et al. argue that LLMs are often misused and over-hyped, and that their limitations warrant a focus on more specialized, easily interpretable tools. Marelli et al. emphasize the importance of transparent attribution and responsible use of LLMs. Finally, Botvinick and Gershman advocate that humans should retain responsibility for determining the scientific roadmap. To facilitate the discussion, the four perspectives are complemented with a response from each group. By putting these different perspectives in conversation, we aim to bring attention to important considerations within the academic community regarding the adoption of LLMs and their impact on both current and future scientific practices.

Background: applications of LLMs in science

LLMs are being considered for scientific writing, data work, participant substitution, coding, and increasingly automated discovery, but their use raises unresolved concerns about errors, bias, plagiarism, attribution, accountability, and regulation. The paper therefore presents four perspectives and responses spanning a broad range of views on how LLMs should affect scientific practice.

  • LLMs could support scientific writing, data acquisition and analysis, participant substitution, coding, and progressively automated scientific exploration under varying levels of human supervision.They can proofread manuscripts, assist with grammar and style, support traditionally manual data tasks23,24, serve as proxies25 or pilot-study participants26, write code27, and operate with reduced supervision28.Source: Background: applications of LLMs in… · p. 2
  • LLM use may compromise scientific integrity through plagiarism, biased or memorized outputs, hallucinated data, fabricated information, and unclear responsibility for proposed ideas.Because LLMs cannot be held accountable for scientific precision and integrity, the scientific community generally does not consider them eligible for authorship34.Source: Background: applications of LLMs in… · p. 2
  • Major AI conferences and journals have adopted policies limiting LLM involvement, while the appropriate strength of regulation and acknowledgment requirements remain unsettled.The cited venues include ICML1, ACL2, Science3, Nature4, and PNAS5.Source: Background: applications of LLMs in… · p. 2
  • The article addresses these non-trivial questions through four researcher perspectives, each paired with a response, selected to span diverse opinions and support constructive discussion.A concluding general discussion identifies common themes across the perspectives.Source: Background: applications of LLMs in… · p. 3

Perspective – LLMs: more like a human collaborator than a software tool

This perspective argues that LLMs should be treated as knowledgeable research assistants rather than traditional software tools, because they can enhance scientific work while remaining fallible. It emphasizes authorial responsibility, open-source models for reproducibility and confidentiality, and continued collaboration to address evolving limitations.

Perspective – LLMs: more like a human collaborator than a software tool

  • LLMs increasingly support writing, programming, research-item formulation, and paper summarization, with observed quality improvements and expected future roles resembling scientific collaborators.Supporting studies link LLMs to improvements in writing35, coding36, and knowledge extraction37.Source: Perspective – LLMs: more like a human… · p. 3
  • LLMs resemble human research assistants more than software tools: they can perform phenomenal work but, like human collaborators, remain vulnerable to errors, plagiarism, fabrication, and discrimination.Their limitations and biases require researchers to remain aware that LLM outputs can be mistaken.Source: Perspective – LLMs: more like a human… · p. 3

Protecting the past

  • Authors remain responsible for checking accuracy, citing sources and methods, transparently communicating methodology, and acknowledging LLM contributions under existing scientific norms.The perspective rejects blind reliance on LLMs and treats them as complements to human expertise and judgment.Source: Protecting the past · p. 3

Welcoming the future

  • Proprietary LLM changes can undermine reproducibility, while locally hosted open-source models can protect reproducibility, peer-review confidentiality, and control over shared information.A study found 57.4% of users considered GPT-4 feedback helpful or very helpful, and 82.4% found it more beneficial than feedback from at least some human reviewers39; confidentiality rules nevertheless prohibit entering many manuscripts into proprietary models6.Source: Protecting the past · p. 3, Welcoming the future · pp. 3–4
  • Because LLMs are evolving toward greater capability and autonomy, researchers should work with one another and developers to improve evaluation, transparency, accountability, truthful sourcing, and acknowledgment of ignorance.The perspective frames these challenges as opportunities to advance scientific understanding and methods.Source: Welcoming the future · p. 4, Conclusion · p. 4

Conclusion

  • The authors conclude that LLMs should be embraced as valuable scientific assets, with existing research norms and author responsibility preferred over novel rules, alongside stronger open-source development.They note that their perspective’s first draft was written by GPT-4 from meeting notes.Source: Conclusion · p. 4

Perspective – Science is a social process that cannot be auto-completed

The perspective argues that LLMs generate plausible linguistic form rather than grounded information, making them unsuitable as a universal basis for science. It recommends cautious, task-specific evaluation because widespread use could damage scientific accuracy, communication, trust, diversity, and education.

Navigating science

  • Information-retrieval and extraction techniques in academic search engines support discovery of relevant prior work, whereas training LLMs on scientific papers does not guarantee scientifically accurate output.Meta’s Galactica debacle highlighted the inappropriateness of LLMs as text generators and synthesis machines; the system was taken off-line after three days following intense criticism for its abysmal performance.Source: Navigating science · p. 4

Doing science

  • Science should favor bespoke, lightweight, interpretable models evaluated for specific tasks over opaque, brittle, one-size-fits-all LLM applications.LLMs may assist with grammar checking, transcription, translation, and code generation, but they are unlikely to underpin most hybrid human-machine science effectively.Source: Doing science · p. 5
  • LLMs cannot replace human test subjects, survey participants, or annotators because they model sequential linguistic tokens rather than human concepts, meanings, opinions, or intent.Studying human behavior requires working with actual people (refs.46–48).Source: Doing science · p. 5

Communicating science

  • Using LLMs to write scientific prose can obscure authorship, weaken citation practices, reduce epistemic diversity, enable large-scale abuse, and flood education with unreliable material.Automatically generated manuscripts differ from writing as a process of formulating and refining ideas; generated claims may not derive from cited papers, while repeated machine outputs can narrow scientific creativity and facilitate fake papers, predatory review, and low-quality tutorials (refs.49, 50; ref.51).Source: Communicating science · p. 5

Conclusion

  • LLMs are often mischaracterized, misused, and over-hyped, so scientific communities should establish cautious use norms and evaluate their impacts at every step.Their effects will extend from literature search through experimental design and writing, with present norms shaping future consequences.Source: Conclusion · p. 5

Perspective – LLMs in scientific practice: a matter of principles, not just regulations

LLMs may benefit scientific practice by supporting writing, literature analysis, and hypothesis generation, but their use should be guided by principles of transparency, accountability, and fairness rather than strict, preconceived regulation. Researchers must disclose and verify AI contributions, address biases, and cultivate skepticism toward LLM outputs.

Perspective – LLMs in scientific practice: a matter of principles, not just regulations

  • LLMs can support day-to-day research by improving writing, freeing time for ideas and analysis, reducing language-based disparities, and broadening literature analysis and hypothesis generation beyond researchers’ specializations [52, 53].Their benefits are conditional on mindful use because the technology also has disruptive potential.Source: Perspective – LLMs in scientific… · p. 6

A matter of transparency

  • Researchers should disclose which LLMs they used, how they contributed, and ideally release prompts and responses, while favoring systems with transparent architectures and training data to improve performance assessment and detect contamination [55, 56].Existing tools such as the CrediT taxonomy could help code AI contributions without treating AI as a coauthor [7].Source: A matter of transparency · p. 6

A matter of accountability

  • Human authors remain accountable for all scientific content because LLMs can hallucinate, fabricate citations, omit references, plagiarize, and produce faulty reasoning [57–59].Users should define quality criteria and verification methods in advance and weigh the effort of vetting outputs against efficiency gains.Source: A matter of accountability · p. 6

A matter of fairness

  • LLMs can reinforce unfairness because they reflect mostly WEIRD populations, are more accessible to those populations, and may favor some perspectives or sources, affecting citations, marginalization, research direction, and careers [60–62].These systematic patterns must be recognized and accounted for in equitable research.Source: A matter of fairness · pp. 6–7

Conclusion

  • Because LLMs’ future influence is difficult to predict, shared principles and values—and training that builds skepticism toward systems and outputs—are preferable to strict, preconceived norms.The proposed foundation for managing rapidly changing technologies is community-wide education in transparency, accountability, fairness, and responsible judgment.Source: Perspective – LLMs in scientific… · p. 6, Conclusion · p. 7

Perspective – AI can help, but science is for people

AI could substantially reduce the growing costs of scientific labor, but humans should retain responsibility for science’s normative direction and epistemic goal. The authors frame this boundary as a subjective limit: human values and meaningful understanding cannot be automated.

AI and scientific labor

  • AI may help scale scientific work as literatures, research questions, datasets, and communication demands grow increasingly difficult, although current systems require caution and oversight.The authors note that AI might eventually perform some scientific labor better than humans, including generating hypotheses [65].Source: AI and scientific labor · p. 7

Perspective – AI can help, but science is for people

The normative aspect of science

  • Humans should retain responsibility for determining science’s direction because judgments about problems’ interest, significance, and timeliness reflect evolving cultural and ethical values.AI may assist deliberation, but it should not govern the evolution of human views or replace humans as moral agents.Source: The normative aspect of science · pp. 7–8

The epistemic aspect of science

  • Human understanding should remain a core goal of basic science, even if AI systems eventually represent aspects of nature beyond human cognitive abilities.Researchers should extract as much human insight as possible from AI systems rather than ceding understanding to them.Source: The epistemic aspect of science · p. 8

Conclusion

  • Science’s proper AI boundary is a subjective limit: human points of view encompass meaningful knowledge and values that machines cannot automate.AI systems may possess or align with their own knowledge and values, but alignment remains yoked to human subjective views [69].Source: Conclusion · p. 8

Conclusion

The conclusion presents four perspectives on how LLMs should affect science, spanning collaboration, misuse, governance, and human responsibility. Despite disagreement, the perspectives converge on protecting science’s social nature, integrity, and standards through continual reevaluation.

Conclusion

  • The four perspectives frame LLM use as collaboration with researchers, a source of scientific risks, a need for transparency and accountability, and an area requiring human control over scientific goals.Schulz et al. compare LLMs with human collaborators; Bender et al. criticize uses contrary to scientific norms; Marelli et al. call for transparency, accountability, and fairness; Botvinick and Gershman reserve problem selection and human understanding to scientists.Source: Conclusion · p. 10
  • Despite substantial disagreement, all perspectives emphasize science’s social nature and the importance of protecting scientific integrity and standards.Source: Conclusion · p. 10
  • The scientific community must continuously reevaluate how to protect these core values as LLMs become part of scientific practice.Source: Conclusion · p. 11

Author contributions statement

The statement assigns project administration and supervision, identifies perspective leaders, and specifies authorship and editing responsibilities for the perspectives, responses, introduction, and conclusion.

Author contributions statement

  • Perspective leaders oversaw the four perspectives, while all authors contributed to reviewing and editing the perspectives and responses.The perspective leaders were Emily M. Bender, Marco Marelli, Matthew M. Botvinick, and Eric Schulz.Source: Author contributions statement · p. 14
  • Marcel Binz and Stephan Alaniz administered the project and drafted the introduction and conclusion, with additional review and editing by Zeynep Akata and Eric Schulz.Project supervision was provided by Zeynep Akata and Eric Schulz.Source: Author contributions statement · p. 14
Loading 2312.03759v1…