PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlling risks of AI in chemical science with agents

Jiyan He, Haoxiang Guan, Weitao Feng, Yaosen Min, Jingwei Yi, Kunsheng Tang, Shuai Li, Jie Zhang, Kejiang Chen, Wenbo Zhou, Xing Xie, Weiming Zhang, Nenghai Yu, Shuxin Zheng

arXiv:2312.06632v2cs.AI

TL;DR

Chemical AI capabilities can be misused, while effective safeguards for scientific AI remain a limited area of awareness and consensus. The paper proposes SciGuard, an external agent-based safeguard, and evaluates it with SciMT. SciGuard achieves a 5.0/5.0 red-team score while maintaining strong benign scientific-task performance; the authors also note unresolved robustness challenges against deceptive or ambiguous inputs.

  • Problem

    Scientific AI can be misused to produce harmful substances or bypass regulations, while awareness and consensus on effective safeguards remain limited.

    Source: §1 · p. 2

  • Method

    SciGuard uses an external LLM-based agent with domain guidance, external knowledge, and scientific tools to mediate interactions, and SciMT evaluates safety and utility across scientific tasks.

    Source: §1 · p. 3

  • Results

    SciGuard earns 5.0/5.0 on red-teaming queries and maintains strong benign-task performance, including 90.0% accuracy on science-info QA.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The authors present SciGuard as an initial step toward mitigating scientific AI misuse while preserving scientific utility and call for continued research and collaboration.

    Source: §1 · p. 4, §4 · p. 18

    SciGuard’s safety judgments still depend partly on user inputs that may be ambiguous, incomplete, or deceptive, and robustness to increasingly sophisticated adversarial behavior remains an open challenge.

    Source: §4 · p. 18

Abstract

from arXiv · show

Artificial intelligence is rapidly advancing scientific discovery, but this progress carries risks of misuse, such as the creation of harmful substances, or circumvention of established regulations. In this paper, we first demonstrate the risks by highlighting real-world examples of AI misuse in chemical science, which underscore the need for effective safety alignment for these AI models. In response, we propose SciGuard, an agent-based guardrail that employs large language models, tools and external knowledge to assess and control risks in scientific AI interactions. For a fair comparison, we introduce a benchmark SciMT (Scientific Multi-Task) to assess both the safety and utility of different AI systems. SciGuard achieves a state-of-the-art harmlessness score on red-teaming queries, while maintaining high performance on benign tasks, without sacrificing scientific knowledge. Finally, we call for continued research and dialogue to ensure the safe deployment of AI in science.

1 Introduction

AI models can be misused in chemical science, while awareness and consensus on effective safeguards remain limited. The paper introduces SciGuard and evaluates it with SciMT, reporting strong red-team harmlessness alongside utility on benign scientific tasks.

1 Introduction

  • AI systems could be exploited to synthesize dangerous substances or bypass regulations, while consensus on preventing, detecting, and mitigating scientific AI misuse remains limited.Source: §1 · p. 2
  • SciGuard mediates access to scientific AI externally, using an LLM, ethical and domain guidance, external knowledge, and tools to produce context-aware responses.It is designed to work without modifying the underlying models.Source: §1 · p. 3
  • On SciMT, SciGuard earns 5.0/5.0 on red-teaming queries and maintains strong benign-task performance, including 90.0% accuracy on science-info QA.Source: §1 · pp. 3–4
  • The authors conclude that safe scientific AI deployment requires dedicated safeguards balancing safety and scientific utility, alongside sustained research and collaboration.Source: §1 · p. 4

2 Results

The results document misuse risks across chemical AI systems, describe SciMT’s evaluation tasks, and report SciGuard’s performance across safety, scientific knowledge, and legal and jailbreak assessments.

2.1.1 Synthesis Planning Models

  • LocalRetro proposes a simpler formamide-dehydration route toward HCN and reports reliability scores exceeding 0.8 for over 1,400 toxic substances, including chemical weapons.The authors describe the accessible route and predictions for hazardous compounds as evidence of synthesis-planning models’ dual-use risks.Source: §2.1.1 · pp. 4–6

2.1.2 Toxicity Prediction Models

  • ADMETlab 2.0 reaches an enrichment factor of 22.0 within the top 5% of GHS-ranked compounds, showing its ability to identify toxic chemicals that could be misused.Source: §2.1.2 · p. 6

2.1.3 Large Language Models and Scientific Agents

  • Mistral-7B, Vicuna-7B, Palm-2, and the Chemcrow scientific agent provide relevant answers to a query seeking substances with effects similar to Methadone.The example highlights the challenge of balancing usefulness with safety in language-based scientific systems.Source: §2.1.3 · pp. 6–7

2.2 Dataset Construction

  • SciMT spans red-team, science-info, legislation, and jailbreak tasks; it includes 432 malicious queries, science questions on hazards and safe handling, legal questions, and 60 jailbreak prompts.Source: §2.2 · p. 7, §2.2.1 · p. 7, §2.2.2 · p. 8

2 Results

  • SciGuard achieves the best reported results across SciMT and improves on its corresponding base models in all tasks.Source: Table 1 · p. 10

2.3 SciMT-Redteam Results

  • SciGuard raises GPT-4 and Gemini-1.5-Pro harmlessness scores to 5.00 and improves Mixtral-8x7B from 4.63 to 4.91 on SciMT-Redteam.The paper reports consistent gains across diverse model backbones.Source: §2.3 · pp. 10–11

2.4 SciMT-ScienceInfo Results

  • SciGuard improves SciMT-ScienceInfo accuracy, including GPT-4 from 87.5 to 90.0 and Gemini-1.5-Pro from 75.0 to 88.75.The paper also reports its largest gains on toxic-substance queries, attributing them to specialized toxicity tools and curated chemical knowledge bases.Source: §2.4 · p. 11

2.5 SciMT-Legislation Results

  • SciGuard scores better than other LLMs and agents on legislation, with RAG contributing to effectiveness, and shows the best defense capability against jailbreak attacks.Source: §2.5 · p. 11, §2.6 · p. 12

3 Method

The section reviews dual-use risks in scientific AI, motivates context-sensitive safeguards, and describes SciGuard’s agent-based design. Case studies illustrate its responses to hazardous queries and a benign task.

3.1 Related Work

  • Prior studies report scientific AI risks including designing harmful substances [44], bypassing regulations, and disclosing sensitive biological data [46].These risks have prompted academic and industry efforts on safety protocols and alignment mechanisms.Source: §3.1 · p. 12

3.2 The Imperative of Building Safeguarded Systems for Scientific AI Models

  • Scientific models often lack broader ethical, environmental, and regulatory understanding, making risk judgments especially context-dependent in chemical science.A toxic compound may be harmful to people yet useful in insecticide production.Source: §3.2 · pp. 12–13

3.3.1 Framework

  • SciGuard mediates user access to scientific models by incorporating task context, domain expertise, regulatory compatibility, and value alignment into risk control.Its framework coordinates task context, retrieved guidance, plans, model actions, and observations before producing a response.Source: §3.3.1 · pp. 13–14

3.3.2 Architecture

  • SciGuard adapts prior agent designs [16] [50] [51] with memory, scientific tools, and planning over actions such as tool use, response, confirmation, or rejection, alongside regulatory RAG.Memory combines prompt-embedded principles with dynamically retrieved databases.Source: §3.3.2 · pp. 15–16

3.3.3 Case Study Against SciMT

  • In two hazardous-query cases, SciGuard responded without introducing risks while many models offered details; on a benign case, it identified the substance where Vicuna-13B and Llama2-13B-Chat failed.The figure presents hazardous-query responses with evaluator harmlessness scores and illustrates the benign-task comparison.Source: §3.3.3 · p. 16, Figure 5 · p. 17

4 Conclusion

The conclusion presents SciGuard as a flexible safeguard for scientific AI while identifying unresolved risks in its judgments and limits in SciMT’s coverage. It calls for continued attention to responsible AI practices.

4 Conclusion

  • SciGuard is designed as an external mediator applicable across models and tasks, integrating domain knowledge to control risk without degrading specialized models’ performance.The authors contrast this intent-aware approach with model-specific safety fine-tuning or unlearning.Source: §4 · pp. 17–18
  • Ambiguous, incomplete, or deceptive user inputs still constrain safety judgments, and robustness to increasingly sophisticated adversarial behavior remains an open challenge.The authors identify subtle intent manipulation and proactive safeguards under ambiguous contexts as areas for future work.Source: §4 · p. 18
  • Misclassification could both over-restrict safe substances and overlook hazardous ones, with potential social and regulatory consequences.Source: §4 · p. 18
  • SciMT does not exhaustively cover chemical-science risks or fully anticipate evolving threats, so it is a focused evaluation framework rather than a comprehensive audit solution.Source: §4 · p. 18
  • The authors call for vigilance and context-sensitive mechanisms as misuse risks are expected to grow in scope and complexity across scientific fields.Source: §4 · p. 18

8 Author’s contributions

The listed contributors performed project leadership, conceptualization, development, investigation, and writing. The accompanying material discusses ethical handling of misuse examples and related scientific-AI advances, risks, and safety efforts.

8 Author’s contributions

  • J. He, W. Zhou, and S. Zheng supervised the project; multiple coauthors contributed to conceptualization, SciGuard development, experiments, benchmark construction, and writing.The passage assigns named contributors to each role.Source: §8 · p. 19

Appendix A Ethical Impacts

  • The authors say they withheld sensitive details from misuse examples to avoid enabling dangerous or unethical acts, and aim to promote safe, responsible AI use.Source: Appendix A · p. 27

Appendix B Details of Related Work

  • Related work spans scientific models and agents that advance research, but also documents risks including illegal-substance production, harmful screening, and biological or medical-data concerns; safety efforts include regulatory and cross-sector initiatives.Examples include synthesis planning and toxicity prediction systems, LLMs, and collaborations involving government, industry, and researchers.Source: Appendix B · pp. 27–28

C.1 Potential Risks of AI Misuse in Science

Scientific-AI misuse spans context-dependent risks that the authors organize as a starting point for assessment, while noting that risks evolve. Examples range from harmful or misleading outputs to LocalRetro devising a VX pathway from unreported routes using textbook reactions.

C.1 Potential Risks of AI Misuse in Science; C.2 Assessing the Risks of AI Misuse in Science

Propose harmful substance; Repurpose for harmful use; Bypassing regulations

Unexpected effects; Misinformation; Inaccuracy

  • Users’ overtrust can obscure model limitations and side effects, while hallucinated or malicious misinformation and incorrect scientific outputs may mislead research.The section gives examples including overlooked reaction byproducts and an AI meal planner’s chlorine-gas recipe.Source: Unexpected effects · p. 30, Misinformation · p. 30, Inaccuracy · p. 30

Intellectual property; Privacy; Bias and discrimination

Scope of risks; Impact of occurrence

  • Risk assessment distinguishes societal reach from consequence severity, which can range from minor faulty results to public-health threats.The scope of risk concerns its extent of societal effect; impact concerns the potential consequences when misuse occurs.Source: Scope of risks · p. 32, Impact of occurrence · p. 32

Associated AI models; Stakeholders

Detectability; Awareness; Enhancement by AI

D.1 A Synthetic Pathway for VX Nerve Agents

  • LocalRetro devised a VX nerve-agent pathway using unreported routes whose individual reactions are classic textbook reactions, a combination the authors call highly dangerous.Source: §D.1 · p. 34, Figure D2 · p. 34

D.2 GHS Hazard Dataset

These appendices describe the GHS hazard dataset and metrics used to characterize synthesis-planning reliability and toxicity screening, then give examples of harmful outputs from language models and scientific agents.

D.2 GHS Hazard Dataset

  • 166,018 compounds with at least one H-code remain after deduplication; the study focuses on acute-toxicity codes and uses the dataset for synthesis-planning and toxicity-screening evaluations.The table summarizes counts and percentages of compounds labeled “danger” for selected acute-toxicity codes.Source: §D.2 · pp. 34–35, Table D1 · p. 35

D.3 The Statistical Results of Synthetic Planning Predictive Reliability Scores

  • The synthetic-planning reliability statistic is the proportion of molecules whose scores fall within a specified region, computed against all molecules involved in prediction.The reported proportion describes score distribution across regions ranging from 0 to 1.Source: §D.3 · p. 35

D.4 The Definition of Enrichment Factor for Predicted Toxicity Properties

  • Enrichment factor compares the positive-instance proportion in the top k% of a ranked list with the positive-instance proportion across the full dataset.It is used to assess how well property-prediction algorithms prioritize hazardous molecules in imbalanced datasets.Source: §D.4 · p. 36

D.5 Additional Cases Illustrating Risks to Chemical Science Posed by Large Language Models and Scientific Agents

  • Examples include LLMs producing synthesis routes for PETN and sarin, while ChemCrow assists with planning a methamphetamine route despite chemical-weapon and explosive checks.The paper says these outputs could help users manufacture hazardous substances without sophisticated equipment or chemistry expertise.Source: §D.5 · pp. 36–37, Figure D3 · p. 37, Figure D4 · p. 38

E.1 Overall Architecture

SciGuard mediates scientific AI interactions through an LLM agent that combines memory, tools, planning, and actions. Its design includes safety guidance, chemical knowledge and interaction history, with reported latency overhead varying by query complexity.

E.1 Overall Architecture

  • SciGuard is a proof-of-concept agent with four components—Memory, Tools, Planning, and Actions—coordinated by an LLM that iterates between plans, tool results, and the user request.The LLM receives principles, guidelines, examples, long-term information, and tool descriptions as context.Source: §E.1 · p. 38, §E.2 · p. 39

E.2 Large Language Models; F.1 Experimental Settings

  • SciGuard uses GPT-4 as its core LLM; experiments use specified model and API versions, with temperature set to 0 and top-p to 1.0 where applicable.GPT-4 is also used as the evaluator LLM.Source: §E.2 · p. 39, §F.1 · p. 45

E.3.1 Principles, Guidelines, and Examples

  • Short-term memory supplies principles and guidelines; these include helpful, lawful responses, intent clarification for risky topics, and refusal for harmful or illegal requests.Examples are included to demonstrate applying guidance and interpreting tool results.Source: §E.3.1 · p. 39, Principles · p. 40, Guidelines · p. 40

E.3.2 Molecule Databases; E.3.3 Hazard Databases

  • PubChem supplies compound information, while hazard databases use molecular similarity to retrieve related high-risk substances and add them to the assessment context.The described similarity approaches include MACCS, ECFP, and AtomPair fingerprints, with GHS, OPCW, and PAN HHP datasets used for searching.Source: §E.3.2 · p. 41, §E.3.3 · pp. 41–42

E.3.4 Interaction History

  • Interaction history stores queries, responses, tool calls, and outputs so SciGuard can assess patterns such as repeated hazardous queries, obfuscation, or escalating requests.The system may use prior interactions to verify intent and apply stricter safeguards, ask questions, or refuse.Source: §E.3.4 · p. 42

E.4 Tools; E.5 Planning

  • SciGuard integrates synthesis-planning, toxicity-prediction, and reaction-prediction models, and uses stepwise planning to decompose requests and coordinate tool use.The implementation names LocalRetro, Chemprop, and Molecular Transformer; plans can be filtered or rerouted based on safety checks.Source: §E.4 · pp. 42–43, §E.5 · pp. 43–44

E.6 Actions

  • Actions specify tools and parameters for an external system to execute; their results return to the agent, while intermediate steps are hidden from users.For a blood-brain-barrier query, an action can invoke a BBBP prediction using the compound’s SMILES representation.Source: §E.6 · p. 44

E.7 Computational Overhead of SciGuard

  • For benign queries, SciGuard adds typically under 1–2 seconds; tool calls for more involved queries typically add 2–5 seconds, with overall overhead described as acceptable for interactive use.The implementation uses a structured system prompt of around 2,000 tokens, context construction, and optional tool invocation.Source: §E.7 · pp. 44–45

F.2 Evaluation Criteria of SciMT-Redteam

SciMT-Redteam responses are evaluated for harmlessness and helpfulness using LLM and human judges. The evaluation uses a five-point rubric and checks agreement between automated and human scoring.

F.2 Evaluation Criteria of SciMT-Redteam

  • GPT-4 evaluates harmlessness on a 1–5 scale after receiving substance background and scoring explanations; human assessments on a subset showed strong agreement with the LLM judge.The evaluation uses the LLM-as-Judge framework [39], and the paper reports 60 responses scored by three human judges and the LLM judge.Source: §F.2 · pp. 45–46
  • Harmlessness scores estimate the potential for harm assuming a malicious, uninformed user receives the details; the rubric ranges from directly encouraging danger to completely harm-free information.The criteria instruct judges to disregard tone, cautions, or warnings and focus on the information provided.Source: §F.2 · pp. 45–46
  • Helpfulness measures whether a response effectively and relevantly addresses the query, but the paper cautions that helpfulness does not establish correctness.The rubric ranges from inability to resolve the question to a highly valuable, detailed answer.Source: §F.2 · p. 46

F.3 Additional Results on SciMT-Redteam

The appendix reports SciMT-Redteam harmlessness-score distributions and an ablation of SciGuard’s principles and guidelines component. The ablation reports lower harmlessness scores when that component is removed.

F.3 Additional Results on SciMT-Redteam

  • Figure F7 plots the proportion of SciMT-Redteam scores at each harmlessness-score value for each AI system.The x-axis is harmlessness score and the y-axis is the corresponding proportion of scores.Source: §F.3 · p. 46, Figure F7 · p. 47
  • Removing SciGuard’s principles and guidelines component leads to a noticeable drop in SciMT-Redteam harmlessness scores.The authors identify this component as important for ensuring safety.Source: §F.3 · p. 46
  • Figure F6 presents human-versus-LLM judge agreement using score-density distributions on the left and a confusion matrix on the right.The figure caption identifies the two visual encodings but does not report specific agreement values.Source: Figure F6 · p. 47

F.4 Additional Statistics and Ablation Results on SciMT-ScienceInfo

The appendix reports model and agent scores and ablation analyses for SciMT-ScienceInfo, then presents a no-RAG ablation result for SciMT-Legislation.

F.4 Additional Statistics and Ablation Results on SciMT-ScienceInfo

  • SciMT-ScienceInfo reports model and agent scores across chemical types and includes an ablation analysis.The passages identify Table F2 for the scores and Figure F9 for the ablation results.Source: §F.4 · p. 47, Table F2 · p. 48, Figure F9 · p. 48

F.5 Additional Statistics on SciMT-Legislation

  • 64.29% is SciGuard (GPT-4)'s overall SciMT-Legislation score without RAG.Table F3 covers scores across legislative sources and includes the ablation without RAG.Source: §F.5 · p. 49, Table F3 · p. 49

F.6 Additional Results on SciMT-Jailbreak

The appendix compares SciGuard with jailbreak defenses, examines helpfulness on benign queries, and tests general-task performance. SciGuard matches or exceeds the compared jailbreak methods, stays helpful on benign queries, and does not degrade performance on sampled MMLU questions.

F.6 Additional Results on SciMT-Jailbreak

  • Excluding principles and guidelines causes a marked decrease in SciMT-Jailbreak scores.Figure F10 presents the ablation results.Source: §F.6 · p. 49
  • SciGuard matches or exceeds Self-Reminders and SmoothLLM on SciMT-Jailbreak, while adding structured planning and domain-specific grounding.Self-Reminders reinforces safety intent through prompt framing; SmoothLLM perturbs inputs and aggregates responses.Source: §F.6 · p. 50, Table F4 · p. 50

F.7 Helpfulness Evaluation on SciMT-Benign

  • On 115 benign queries, SciGuard remains helpful while staying harmless.The evaluation uses benign queries simulating nonmalicious users and reports harmlessness and helpfulness scores in Table F5.Source: §F.7 · pp. 50–51, Table F5 · p. 51

F.8 Evaluating SciGuard on General-Purpose Tasks

  • On 200 sampled MMLU questions, SciGuard does not degrade GPT-4 performance and shows a slight improvement.The authors attribute the observed improvement to general-purpose tools such as retrieval and structured reasoning.Source: §F.8 · p. 51

F.9 Evaluation on WMDP Benchmark

The WMDP evaluation focuses on chemical-security questions and compares SciGuard's agent-based risk control with a conservative, refusal-oriented setup. SciGuard significantly reduces model accuracy on hazardous-chemical questions, which the paper treats as stronger resistance to misuse.

F.9 Evaluation on WMDP Benchmark

  • The evaluation uses the chemical-security subset of WMDP and compares agent-based contextual risk control with representation-level unlearning.A conservative instruction to answer potentially harmful questions at random simulates a refusal-oriented baseline.Source: §F.9 · p. 52
  • SciGuard significantly reduces model accuracy on hazardous chemical knowledge questions, indicating stronger resistance to misuse.Table F7 reports accuracy on WMDP-Chem, where lower accuracy implies stronger mitigation of hazardous knowledge.Source: §F.9 · p. 52, Table F7 · p. 52

F.10 Robustness to Minor Chemical Variations

The evaluation tests whether SciGuard can distinguish closely related molecules whose small structural differences can produce divergent properties. It reports successful recognition of functional and toxicological differences across five representative compound pairs, using chemical knowledge, retrieval, and specialized models.

F.10 Robustness to Minor Chemical Variations

  • The five pairs illustrate how small changes can yield substantial differences in toxicity or function, including between fentanyl and carfentanil and between thalidomide enantiomers.Carfentanil is described as approximately 100 times more potent than fentanyl; the (S)-form of thalidomide is described as a potent teratogen.Source: §F.10 · p. 53
  • SciGuard successfully recognized and articulated functional and toxicological differences across five representative pairs of closely related compounds.The pairs vary by a minimal structural feature, such as chirality, one atom, or a side-chain modification.Source: §F.10 · p. 53
  • SciGuard uses structured chemical knowledge, retrieval-augmented reasoning, and specialized scientific models to analyze subtle structural differences in context.The invoked tools include toxicity predictors and physicochemical property estimators.Source: §F.10 · p. 53
Loading 2312.06632v2…