Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy

1 Introduction

• AI scientists can select tools, plan procedures, and automate laboratory tasks, while their capabilities remain short of those of human scientists.The paper describes systems that design experiments, control laboratory equipment, and make research decisions.Source: §1 · p. 3
• The perspective scopes risks by user intent, scientific domain, and impact on the external environment.Its classification includes direct or indirect malicious intents, unintended consequences, multiple scientific domains, and environmental impacts.Source: Figure 2 · p. 4, §1 · p. 4
• Complex scientific systems make errors consequential: incorrect pathogen manipulation or reaction parameters could create biosafety risks or dangerous explosions.The paper illustrates potential errors and associated risks in an antibody-synthesis workflow.Source: Figure 1 · p. 3, §1 · p. 4
• The proposed safeguarding framework combines human regulation, agent alignment, and agent regulation involving tool oversight and environmental feedback.Human regulation includes training and licensing, usage-log audits, and ethical, safety-oriented development practices.Source: §1 · p. 5, Figure 3 · p. 5

2.1 Problem Scope

• AI scientists are defined as autonomous systems with scientific-domain capabilities that can plan and take actions toward objectives.The scope spans in-silico analyses and physical laboratory procedures.Source: §2.1 · p. 5
• An example scientific workflow can combine database research, LLM-generated hypotheses, and robotic experimental testing.The paper uses biochemical-mechanism discovery to illustrate this sequence.Source: §2.1 · pp. 5–6
• The risk framework considers user intent, scientific domain, and environmental impact, and its categories are not mutually exclusive.A misinformation campaign could, for example, also pertain to a specific chemical.Source: §2.1 · p. 6
• The domain classification includes chemical, biological, radiological, physical, informational, and emerging-technology risks.Chemical risks include hazardous substances and materials with unknown or unpredictable properties; biological risks include dangerous pathogen modification.Source: §2.1 · p. 6
• Intent-related risks include direct malicious requests, indirect assembly of innocuous components into harmful outcomes, and hazards arising during benign tasks.Unintended consequences can involve hazardous products or byproducts with immediate or long-term effects.Source: §2.1 · p. 6
• Radiological risks include exposure, containment failures, and possible misuse in nuclear research, while physical risks include equipment malfunctions and laboratory harm.The risks cover both operational hazards and broader security concerns.Source: §2.1 · pp. 6–7
• Information-science risks include data misuse, misinterpretation, or leakage that can produce erroneous conclusions or expose sensitive information.The paper cites malicious medical literature that can poison knowledge graphs and compromise biomedical knowledge discovery.Source: §2.1 · p. 7
• External impacts span the natural environment, human health, and the socioeconomic environment.Examples include pollution and energy use, effects on well-being, job displacement, and unequal access to scientific advances.Source: §2.1 · p. 7

2.2 Vulnerabilities of AI Scientists

• AI scientists use five interconnected modules—LLMs, planning, action, external tools, and memory and knowledge—in a sequential pipeline that stores outcomes or feedback.The modules are presented as a linked architecture with vulnerabilities in each.Source: §2.2 · p. 7, Figure 4 · p. 8

2.2.1 LLMs (The Base Models)

• LLMs can generate plausible falsehoods, be jailbroken, struggle with complex reasoning, and lack current scientific knowledge.Reasoning deficits can contribute to flawed plans and inappropriate tool use; retrieval and model editing do not resolve the open challenge of long-term relevance.Source: §2.2.1 · pp. 7–8

2.2.2 Planning Module

• Planning vulnerabilities include overlooking long-term risks, wasting resources in dead loops, and difficulty coordinating multiple goals or tools.Single-task optimization can make agents unreliable or inefficient in scenarios requiring simultaneous attention to diverse objectives.Source: §2.2.2 · p. 9

2.2.3 Action Module

• Weak oversight of tool use can lead agents to select or misuse tools, potentially triggering hazardous reactions such as explosions.Specialized tools can act as black boxes, and human-agent action regulation remains underdeveloped.Source: §2.2.3 · pp. 9–10

2.2.4 External Tools

• Incorrect commands to external software or hardware can cause hazardous spills or reactions when agents fail to anticipate real-world consequences.Examples include robotic arms, chemical-analysis software, and molecular-design toolkits.Source: §2.2.4 · p. 10

2.2.5 Memory and Knowledge Module

• Memory and knowledge vulnerabilities include gaps in domain-specific safety knowledge, limited human feedback, inadequate environmental feedback, and unreliable research sources.Failure to interpret environmental conditions can misinform decisions; outdated or unreliable sources can contribute to incorrect or harmful knowledge.Source: §2.2.5 · p. 10

2.3 Recent Work in Safeguarding AI scientists

• The paper presents its survey of related safeguarding work on LLMs and agents, explicitly including scientific agents.The survey is shown in Figure 5 and Table 1.Source: Figure 5 · p. 11, §2.3 · p. 11
• Correct-looking code or procedures can still produce unexpected errors or dangerous outcomes when deployed in scientific settings.The paper contrasts syntactically correct code with runtime errors and describes chemical procedures that may generate toxic gases or dangerous byproducts.Source: §2.3 · p. 11
• Coscientist highlighted safety risks, while ChemCrow, CLAIRify, and SciGuard developed chemical-agent safeguards or risk-control approaches [53].SciGuard introduced SciMT-Safety to assess harmlessness on malicious queries and helpfulness on benign ones.Source: §2.3 · p. 11

2.4 Current Limitations

• Effective safety mechanisms should address user inputs, agent actions, and environmental consequences, but current efforts remain incomplete.Source: §2.4 · p. 11
• Most safeguarding work relies on external tools [53], while fixed, finite action spaces can restrict tool functions and help ensure safer operation.The paper gives read-only file access as an example of excluding potentially dangerous write operations.Source: §2.4 · pp. 11–12
• Specialized risk-control mechanisms are largely lacking beyond SciGuard [53]; current filtering and LLM monitoring screen behavior, while proactive exploit-finding models are needed.These specialized measures are particularly important for high-stakes scientific experiments compared with general web or software tasks.Source: §2.4 · pp. 12–13
• AI scientists need domain expertise to plan experiments and use tools safely, as well as to recognize hazards such as dangerous exothermic reactions.The paper contrasts this requirement with general-purpose agents handling web browsing or basic tool use.Source: §2.4 · p. 13
• Existing scientific-safety benchmarks such as SciMT-safety [53] assess harmlessness by testing refusal of malicious requests, leaving broader risks and agent vulnerabilities unevaluated.The paper calls for benchmarks covering comprehensive risk scopes and agent vulnerabilities.Source: §2.4 · p. 13

3 Proposition

• Safeguarding should prioritize risk control over autonomy, evaluate actions as well as outputs, and pair expert supervision with validated low-risk operating bounds.Source: §3 · p. 13

3.1.1 Agent Alignment

• LLM-level alignment addresses harmful outputs, while agent alignment requires expert workflow data, action-sequence feedback, and strategy-level reward models.The paper notes that structured data on researchers’ sequences of actions, tool use, and decisions are lacking.Source: §3.1.1 · p. 14

3.1.2 Safety Evaluation

• Scientific-agent evaluation combines domain-specific, escalating red-team tests with broader benchmarks and monitored, graduated autonomy.Tests should cover physical safety, domain expertise, and tool interactions; initial chemical-agent validation is reported while broader cross-domain testing continues.Source: §3.1.2 · pp. 14–15

3.2 Human Regulation

• Human regulation extends responsibility beyond agents to the developers and users who create and operate AI scientists.Source: §3.2.1 · p. 15, §3.2.2 · p. 16

3.2.1 Developer Regulation

• Developer regulation proposes certification, ethical and safety compliance checks, security assessments, and auditable development records.The paper notes that consistent international oversight remains challenging.Source: §3.2.1 · p. 15

3.2.2 User Regulation

• User regulation proposes licensing and training, with monitoring that balances safety oversight against laboratory privacy and intellectual property.Institutional committees with AI-safety expertise would use standardized risk-assessment protocols; more thorough checks can increase response latency.Source: §3.2.2 · p. 16

3.3 Agent Regulation and Environmental Feedback

• Simulated environments let agents anticipate harmful consequences, with fidelity metrics and real-world calibration used to evaluate the simulations.The proposed evaluation includes physical and process fidelity and analysis of how simulation uncertainty affects decisions.Source: §3.3 · pp. 16–17
• Agent-side safeguards combine tool-use checks, consequence review, layered decision validation, critic models, and action-data training.The proposed training strategies include expert-observed workflows, validated synthetic scenarios, active learning, and expert validation for actions with insufficient supporting data.Source: §3.3 · pp. 17–18

• XT and QJ co-initiated the project, with XT leading manuscript writing and contributing to the theoretical framework.Other authors contributed to chapters, risk analysis, application and safety expertise, ethical and regulatory guidance, supervision, and revision.Source: Author Contributions Statements · p. 18