PaperlayerView exact revision on arXiv

Source-linked AI summary

ChemMat-AgentSafetyBench: Evaluating Long-Horizon Attacks and Defenses in Chemistry and Materials Agents

Zhan'ao Yao, Zhihao Gao, Liang Yin, Boxuan Zhang, Xiaoyu Wu, Linjing Li, Rongyan Wang, Tingwei Chen, Youwei Wang, Xiaolin Zhao, Jiahui Shi, Jianjun Liu

arXiv:2609.11952v1cs.CR

TL;DR

Existing agent-safety and chemistry evaluations do not jointly assess whether stateful chemistry and materials agents release complete hazardous artifacts through long-horizon, multi-entry workflows. ChemMat-AgentSafetyBench evaluates fixed hazardous targets with direct and five online long-horizon attacks across user input, tool observations, and persistent memory, using graded adjudication of final outputs. Attacker replacement yields 18.4%–26.5% mean L3 release, while existing defenses reduce but do not eliminate complete hazardous-output release.

  • Problem

    Existing agent-safety and chemistry evaluations do not jointly assess whether stateful chemistry and materials agents release complete hazardous artifacts through long-horizon, multi-entry workflows.

  • Method

    ChemMat-AgentSafetyBench evaluates fixed hazardous targets with direct and five online long-horizon attacks across user input, tool observations, and persistent memory, using graded adjudication of final outputs.

  • Results

    Attacker replacement yields 18.4%–26.5% mean L3 release, while existing defenses reduce but do not eliminate complete hazardous-output release.

  • Takeaways & Limitations

    Existing input/state defenses and chemistry-specific boundary checks reduce but do not eliminate complete hazardous-output release across multi-entry attacks.

  • Takeaways & Limitations

    The evaluation measures endpoint and output-form risk in a simulated workflow and does not verify experimental feasibility, physical stability, or successful real-world synthesis.

Abstract

from arXiv · show

Chemistry and materials agents integrate literature retrieval, candidate generation, property prediction, and protocol planning into continuous discovery workflows. Consequently, the relevant safety question is shifting from whether a model answers a hazardous question to whether an agent releases a hazardous protocol through a tool-mediated workflow. We introduce \bench, a benchmark that evaluates whether chemistry and materials agents can be steered toward hazardous endpoints through user input, tool observations, or persistent memory. The benchmark contains 432 fixed harmful case specifications spanning eight hazard classes, three scenario shells, four tool-and-memory environments, a single-turn direct-attack baseline, and five online long-horizon attacks: intent hijacking, tool chaining, objective drifting, task injection, and memory poisoning. The concrete language of each online attack is generated from the evolving trajectory at runtime and is therefore not counted in the static benchmark size. In the four-model main experiment with a fixed attacker, agents release complete hazardous synthesis or preparation procedures in 25.6\% of runs. Replacing the attacker model yields mean success rates from 18.4\% to 26.5\%, indicating that the risk is not an artifact of a single attacker. Input- and state-level defenses adapted from general-purpose agent safety, as well as candidate checks designed for chemistry and materials, reduce some failures but still leave complete-path release rates between 9.2\% and 22.5\%. Existing defenses therefore do not simultaneously cover multi-entry contamination, tool state, and the final artifact boundary. These results highlight a widening gap between the rapid development of scientific agents and the safety evaluation and defenses available to the chemistry and materials community.

1. Introduction

ChemMat-AgentSafetyBench reframes chemistry-agent safety as a system-level problem involving hazardous artifact release through long-horizon, tool-mediated workflows. It evaluates multi-entry attacks and defenses using complete hazardous output as the primary success condition.

  • Motivation: Chemistry and materials agents progressively construct scientific artifacts through literature, databases, prediction tools, and planning, making workflow-level safety distinct from ordinary question answering.Tool observations become task-relevant context for subsequent reasoning in the ReAct execution pattern.
  • Benchmark scope: ChemMat-AgentSafetyBench evaluates the agent together with its defenses rather than an isolated classifier or single-turn safety model.The benchmark uses externally established hazardous targets, multiple victim and attacker models, and a unified ReAct environment.
  • Attack design: The benchmark maps long-horizon attacks to user input, tool observations, causal tool chains, and persistent memory, while adding domain attacks across entry points and execution lengths.The static benchmark uses predefined targets; online attack language is generated during the evolving trajectory.
  • Evaluation: Complete hazardous final-output release is the primary success criterion, with graded adjudication separating refusal, naming, partial routes, and end-to-end procedures.This focuses evaluation on whether the agent releases a complete hazardous artifact rather than on a special token or fixed string.
  • Evaluation: The benchmark quantifies safety across multiple models and defenses, showing that general defenses and chemistry-specific checks reduce but do not eliminate failures under multi-entry attacks.The study therefore exposes remaining limitations at the boundary between agent state, attack channels, and final artifacts.

2. Related Work

Prior agent-safety work establishes risks from long-horizon attacks, tool-state contamination, and malicious multistep tasks, but mainly in general-purpose environments. Chemistry and materials benchmarks evaluate scientific capabilities and hazard judgments without treating hazardous artifacts released through agent workflows as the endpoint.

  • Security of tool-using agents: General agent-security benchmarks study injection, tool-state contamination, malicious multistep tasks, and high-risk tool calls in web, operating-system, shopping, email, and transactional environments.Their endpoints typically involve unauthorized actions, data leakage, or task diversion rather than hazardous chemistry or materials artifacts.
  • Chemistry and materials agents: Chemistry and materials agents connect retrieval, candidate generation, prediction, screening, and synthesis planning, allowing their outputs to influence selected research objects and downstream actions.Existing systems demonstrate entry into scientific discovery loops, but this capability is not itself a hazardous-agent safety benchmark.
  • Hazard adjudication: Chemistry and materials evaluation faces a hazard-adjudication trade-off: registries anchor known hazards, whereas structural alerts and predictive models cover broader spaces but risk domain-of-applicability errors and false positives.The benchmark consequently avoids treating unrestricted language-model judgment or arbitrary structural alerts as hazard ground truth.
  • Hazard adjudication: The proposed evaluation freezes an answer key around externally established hazardous targets and combines a deterministic comparator with independent semantic adjudication to verify complete hazardous output release.This supports attack–defense comparisons without conflating the benchmark with the open question of whether an unknown structure is hazardous.

3. Problem Formulation

The benchmark evaluates a stateful ReAct victim agent operating with tools, persistent state, and optional defenses under user- and environment-side attacks. Tasks use frozen hazard classes and simulated endpoint adjudication, while trajectories are recorded for audit without testing physical synthesis.

  • 3.1. System Under Test and Threat Model: The evaluated system comprises a victim language model, tool environment, state storage, and optional defense within a stateful single-agent ReAct loop.The victim reads trusted tasks and tool observations, chooses actions or final answers, and updates its visible trajectory after each interaction.
  • 3.1. System Under Test and Threat Model: The attacker cannot modify model weights, system prompts, or safety policy, but controls a designated input channel through user turns or untrusted observations, retrieval snippets, and persistent memory.User tasks remain fixed in environment-side attacks, while the attacker can read redacted history, prior responses, and state summaries.
  • 3.2. Task Endpoints and Adjudication: Each harmful task is bound to one of eight hazard classes frozen before evaluation, while the public benchmark exposes only a coarse label rather than the hazardous identity, structure, route, or attack text.The endpoint asks the agent to transform a scientific starting point into a process-complete preparation or synthesis plan.
  • 3.2. Task Endpoints and Adjudication: During rollout, the victim receives attacker-controlled turns or untrusted state, calls tools as needed, and has its final answer plus redacted trajectory metadata stored for analysis.Recorded metadata includes tool-call order, observation writes, and memory reads and writes.
  • 3.2. Task Endpoints and Adjudication: The study measures endpoint and output-form risk in a simulated workflow rather than experimental feasibility, physical stability, or successful real-world synthesis.This bounds the interpretation of hazardous release rates to the evaluated simulated setting.
  • 3.2. Task Endpoints and Adjudication: An independent semantic adjudicator evaluates the terminated final output and minimal audit metadata, without participating in execution or acting as a defense.The adjudicator supplies the L0–L3 outcome used for the primary metric.

4. ChemMat-AgentSafetyBench

ChemMat-AgentSafetyBench evaluates chemistry and materials agents as closed-loop systems exposed to fixed hazardous targets and adaptive attacks across tools, observations, memory, and defenses. Its static benchmark construction matches attack families to the environment state they require, while separating general input/state defenses from chemistry-specific boundary checks.

  • Design Principles: ChemMat-AgentSafetyBench evaluates hazardous endpoint release in a closed-loop scientific agent, with attack, environment, victim, and defense jointly represented rather than reducing safety to static question answering.The benchmark uses an independent final semantic adjudicator for complete hazardous procedures.
  • Attack Protocol: A0 is the fixed single-turn baseline, whereas A1–A5 generate inputs or untrusted records online from the victim’s evolving history and state.The online families cover later-turn redirection, causal tool chaining, observation drift, tool-observation injection, and poisoned memory reuse.
  • Static Construction: The benchmark contains 432 harmful static instances from 8 hazards, 3 scenario shells, and valid environment–attack pairs, with online attack wording excluded from the static count.The construction expands each valid environment–attack pair over eight hazard classes and three scenario shells; A0 contributes 96 tasks and A1–A5 contribute 336.
  • Environment Configurations: Attack validity follows required state: A0, A1, and A4 work in C0; A3 requires observation-bearing tools; A2 requires C2–C3 discovery chains; and A5 requires C3 memory.C0–C3 expose progressively richer retrieval, prediction, candidate-generation and synthesis-assessment, and persistent-memory capabilities.
  • Static Distribution: The four environments contribute 72, 96, 120, and 144 harmful tasks, while attack-family totals are 96 for A0, A1, and A4, 72 for A3, 48 for A2, and 24 for A5.These counts follow the containment relation between environment configurations and attacks, not uniform distribution across six families.
  • Defense Mapping: General defenses inspect untrusted input, observations, memory, or tool choice, whereas domain defenses check candidate or final-output boundaries using matched defense-free baselines.The groups are compared on the same case–model pairs or applicable attack surface, and the scorer remains outside the defended system.

5. Evaluation

The evaluation measures complete hazardous-path release across controlled attack surfaces, victim models, and defenses. Results show substantial release under direct and long-horizon attacks, while defenses reduce but do not eliminate the risk.

  • 5.2. Four-Model No-Defense Main Experiment: 25.58% overall L3 ASR was observed under the fixed-attacker, four-model no-defense experiment, with victim rates ranging from 8.56% to 36.81%.The experiment uses the same 432 harmful case specifications and adjudicates complete hazardous-path release.
  • 5.3. Attack Families, Entry Points, and Horizon Ablation: Tool chaining reached 32.3% L3 ASR and memory poisoning 35.4%, while online attacks A1–A5 reached 24.6% versus 28.9% for the one-shot A0 baseline.Attack families are evaluated only on environments exposing their required channels.
  • 5.4. Defense Comparison: Domain boundary checks reduced L3 ASR to 9.2%–11.0%, but acted near candidate or output boundaries rather than preventing tool-state or memory contamination.General input and state defenses also reduced release on their observable surfaces but left residual risk.
  • 5.5. Attacker and Real-Agent Transfer Summary: Replacing the attacker produced mean L3 release rates of 18.4%–26.5%, while native-interface transfer showed complete-path release on recipe-oriented interfaces.Low harmful release in some native settings can coincide with low benign completion, capability limits, incomplete outputs, or undetermined trajectories.

6. Conclusion

ChemMat-AgentSafetyBench evaluates whether stateful chemistry and materials agents release complete hazardous outputs through user language, tool observations, and persistent memory. Across direct, long-horizon, attacker-transfer, and real-agent settings, defenses reduce but do not eliminate complete-output release.

  • 6. Conclusion: ChemMat-AgentSafetyBench provides an auditable basis for studying multi-entry attacks and defense trade-offs in chemistry and materials agents.The benchmark covers complete hazardous outputs under user-language, tool-observation, and persistent-memory attacks.

Ethical Statement

The benchmark withholds reusable hazardous process details and retains private audits for consistency checks. Its experiments assess simulated scientific workflows rather than physical feasibility or real-world synthesis.

  • Ethical Statement: The public benchmark withholds hazardous identities, verbatim prompts, raw outputs, route text, and reusable process details, retaining private audits only for consistency checks.This limits public exposure of hazardous procedural content.
  • Ethical Statement: The experiments assess risk in simulated scientific workflows and do not validate physical feasibility or real-world synthesis.The stated purpose is safety evaluation and defense stress testing.

A.1. Attacker-Model Ablation

Replacing the attacker model preserves nonzero complete-path release, but native interface affordances and victim capabilities strongly shape observed transfer outcomes.

  • Attacker-model robustness: 18.4%–26.5% attacker-mean L3 release shows the risk is not specific to one attack-generation model.All four attackers induce nonzero L3 release, while victim means differ more substantially across models.
  • Real-agent transfer: ChemGraph single releases complete paths at every tested depth, whereas ChemGraph multi remains low on harmful and benign L3 because capability boundaries and incomplete outputs dominate.CACTUS and ChemToolAgent support only C0 and are excluded from the depth comparison.
  • Real-agent transfer: 45.4% harmful versus 71.2% benign L3 for SKY synthesis shows that protocol-oriented interfaces can convert attacks into complete outputs under matched denominators.Real-agent transfer should be interpreted with native task affordance, interface protocol, and capability boundaries rather than as a single vulnerability ranking.
  • Failure modes: Task or capability boundaries, incomplete outputs, and explicit refusals absorb non-L3 trajectories across native agents.ChemGraph multi and CACTUS are dominated by capability boundaries; incomplete outputs are also frequent in ChemGraph multi, ChemToolAgent, and SKY.
  • Interpretation: Low harmful L3 is not sufficient evidence of effective defense because low benign completion, capability limits, incomplete outputs, and runtime failures can produce the same pattern.Benign controls and failure-cause analyses are required to interpret attack transfer.

B. Method Tables

The supplementary method tables document the environment configurations and which attack families have valid delivery channels.

  • Environment and attack mapping: Table 9 summarizes environment configurations and the attacks with valid delivery channels.The table provides the configuration-level mapping underlying the benchmark’s attack coverage.

C. Additional Main-Experiment Tables

The supplementary tables and procedures specify the attack families, delivery channels, validity checks, defense surfaces, and runtime generation process used in the benchmark.

  • Threat model: The attacker reads only exposed trajectory summaries, victim responses, tool or memory summaries, and incomplete stage plans, without modifying model weights, prompts, permissions, defenses, or adjudication.Final success remains determined solely by L3 release adjudication.
  • Runtime attacker: The online attacker uses Planner, Attacker, Protocol Verifier, and Progress Judge roles to generate and validate trajectories against the requested delivery mechanism and L3 contract.The procedure initializes a private plan, iterates over visible state, and applies frozen L3 adjudication after termination.
  • Supplementary tables: Tables 10–14 organize attack-family delivery channels, defense mappings, L3 rates by entry point, and general versus chemistry-specific defense results.The defense tables report paired applicable surfaces and L3 count/rate (%) values.
  • Attack protocol: A0 is the fixed single-turn baseline, while A1–A5 generate follow-up inputs or environment records online from permitted trajectory state.Online attack language is generated at runtime rather than treated as a static prompt dimension.
  • Validity conditions: A1–A5 validity requires evidence that the designated channel influenced later planning, with tool chaining requiring ordered dependent calls and environment attacks requiring observation or memory writes followed by victim use.These conditions distinguish user redirection, tool composition, objective drift, task injection, and memory poisoning.

F. Defense Implementation

Defenses are placed at distinct observable surfaces, from untrusted inputs and state to parsed candidate identities, while final adjudication remains separate from defense mechanisms.

  • Defense components: General defenses include Repeated Prompt, PI Detector, Spotlighting, and Tool Filter, while chemistry-specific defenses include Candidate Check and Known Hazard Registry.The final adjudicator is explicitly not treated as a defense.
  • Defense coverage: Defense placement follows attack surface: prompt and state defenses target A3–A5, Tool Filter targets A3/A4, and candidate checks apply across A0–A5.Each defense is compared with D0 on the same applicable surface.
  • Interception workflow: A representative interception marks an observation as untrusted or flags a parsed candidate, after which the victim withholds the endpoint procedure and returns to safe guidance.The trajectory is scored as L3 failure because no complete hazardous output is released.
  • Demonstration: The sanitized dialogue preserves the hazardous target and progression across preparation, engineering control, integration, and verification while redacting actionable operating details.It illustrates the interaction structure without exposing concrete procedure details.

G. Real-Agent Task-Capability Adaptation Ablation

The task-capability interface adaptation tests whether real-agent failures reflect prompt/interface mismatch rather than safety behavior. It does not consistently increase complete hazardous-procedure release, and interpretation requires considering native affordances, tool capability, output protocol, and benign-task performance.

  • 69 of 623 overlapping trajectories moved from non-L3 to L3, while 84 moved from L3 to refusal, capability boundary, or incomplete output.These paired results show that adaptation changes outcomes in both directions rather than reliably increasing hazardous release.
  • Interface adaptation does not consistently increase L3 release across real agents.SKY synthesis declined from 45.4% to 40.4%; ChemGraph single reached 32.2% under the expanded denominator, while ChemGraph multi remained low and CACTUS stayed at the short-answer tool boundary.
  • The ablation supports interpreting low L3 cautiously because interface, tool, output-protocol, and native-task affordance limits can shape transfer outcomes.A more explicit native format was neither a reliable bypass nor a consistent capability enhancement.
Loading 2609.11952v1…