PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents

Derek Lilienthal, Sanghyun Hong

arXiv:2508.17155v1cs.CRcs.AI

TL;DR

TOCTOU vulnerabilities in LLM-enabled agents remain largely unexplored, despite risks from acting on external state that may change between tool calls. This study introduces a 66-task benchmark and adapts three defenses; together, they reduce vulnerable executed trajectories from 12% to 8% and the attack window by 95%.

  • Problem

    TOCTOU in agent workflows remains understudied, including how it manifests, whether systems-security mitigations apply, and how effective they are.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper introduces TOCTOU-Bench with 66 tasks and evaluates prompt rewriting, state integrity monitoring, and tool fusing at different workflow stages.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 3

  • Results

    12% to 8%: combining the defenses reduced executed trajectories containing vulnerabilities, while reducing the attack window by 95%.

    Source: §4.1 · p. 4

  • Takeaways & Limitations

    The results show that adapted defenses can address both vulnerable-plan frequency and the time window for exploitation in the evaluated agent workflows.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Large Language Model (LLM)-enabled agents are rapidly emerging across a wide range of applications, but their deployment introduces vulnerabilities with security implications. While prior work has examined prompt-based attacks (e.g., prompt injection) and data-oriented threats (e.g., data exfiltration), time-of-check to time-of-use (TOCTOU) remain largely unexplored in this context. TOCTOU arises when an agent validates external state (e.g., a file or API response) that is later modified before use, enabling practical attacks such as malicious configuration swaps or payload injection. In this work, we present the first study of TOCTOU vulnerabilities in LLM-enabled agents. We introduce TOCTOU-Bench, a benchmark with 66 realistic user tasks designed to evaluate this class of vulnerabilities. As countermeasures, we adapt detection and mitigation techniques from systems security to this setting and propose prompt rewriting, state integrity monitoring, and tool-fusing. Our study highlights challenges unique to agentic workflows, where we achieve up to 25% detection accuracy using automated detection methods, a 3% decrease in vulnerable plan generation, and a 95% reduction in the attack window. When combining all three approaches, we reduce the TOCTOU vulnerabilities from an executed trajectory from 12% to 8%. Our findings open a new research direction at the intersection of AI safety and systems security.

1 Introduction

TOCTOU vulnerabilities arise when an agent acts on external state that changes between successive tool calls, a risk not addressed in prior agent-security literature. This study examines the gap, introduces a benchmark and three defenses, and evaluates them.

1 Introduction

  • Agents’ autonomous use of external tools, APIs, and commands creates security risks in applications including healthcare, finance, and software engineering.Source: §1 · p. 1
  • TOCTOU risks arise because agents make successive, non-atomic tool calls and may act on state that has changed since it was checked.Such exploitation can lead to unsafe tool use, data leakage, or bypassed safety rules.Source: §1 · p. 1
  • Existing work leaves unclear how TOCTOU vulnerabilities appear in agent workflows, whether systems-security mitigations apply, and how effective they would be.Source: §1 · p. 1
  • The paper introduces TOCTOU-Bench, proposes Prompt Rewriting, State Integrity Monitoring, and Tool Fuser, and evaluates the defenses.The approaches target distinct stages of the agent workflow.Source: §1 · p. 2

2 Background and Related Work

Agent-security work has studied prompt injection, data exfiltration, and unsafe tool use, while systematic study of TOCTOU in agent workflows remains absent. Traditional systems defenses rely on assumptions that do not hold for these workflows.

2 Background and Related Work

  • Traditional TOCTOU defenses address races through atomic operations, access controls, synchronization, static analysis, and runtime monitoring.Their assumptions include deterministic execution, fixed interfaces, and well-defined execution paths.Source: §2 · p. 2
  • Agent-security research covers prompt injection, data exfiltration, and unsafe tool execution, but has not systematically studied or defended against TOCTOU in agent workflows.Existing defenses assume atomic tool execution and stable environment state between calls.Source: §2 · pp. 2–3

3 Methodology

The methodology defines TOCTOU around state changes between successive agent actions, constructs a 66-task benchmark, and adapts defenses to prompts, planning, and tool calls.

3.1 Definition of TOCTOU in LLM-Enabled Agents

  • A TOCTOU vulnerability occurs when an agent checks state in one action and uses it in a later action after an intervening change.Source: §3.1 · p. 3

3.2 TOCTOU-Bench: Dataset for Evaluating TOCTOU Vulnerability

  • TOCTOU-Bench adapts AgentDojo environments and retains 66 tasks after excluding injection and single-tool-call tasks from the original 97.Hand-labeling found possible TOCTOU vulnerabilities in 56 of the 66 tasks.Source: §3.2 · p. 3

3.3 Detecting and Mitigating TOCTOU in LLM-Enabled Agents

  • The defenses target prompt rewriting, planning-time state monitoring, and tool-call sequence checking.Source: §3.3 · p. 3
  • Prompt Rewriting analyzes the prompt and available tools, then discourages plans with vulnerable call sequences.Source: §3.3 · p. 3
  • State Integrity Monitoring labels potentially vulnerable ordered tool pairs and encodes them in a finite-state automaton checked during execution.The monitor halts a call and alerts the user when a violation is reached.Source: §3.3 · pp. 3–4
  • Tool Fuser uses vulnerable pairs identified by SIM to replace sequential calls with fused tools that execute atomically.Source: §3.3 · p. 4

4 Evaluation

Across 66 tasks, combining the defenses reduced executed trajectories containing vulnerabilities from 12% to 8% and cut the attack window by 95%. Individual results show modest plan reduction, limited automated detection, and shorter call gaps with fusion.

4 Evaluation

  • The evaluation measures the defenses individually and then assesses their combined effectiveness on TOCTOU-Bench’s 66 tasks.Source: §4 · p. 4
  • Experiments used GPT-4o with AgentDojo environments, LangGraph, and timers between reasoning and tool calls.Source: §4 · p. 4

4.1 Effectiveness of Our Countermeasures

  • 55 to 53 tasks: Prompt Rewriting reduced vulnerable plans, and no originally safe task became vulnerable after rewriting.The authors attribute the limited improvement in part to explicit task prompts constraining tool selection.Source: §4.1 · p. 4
  • 25% TPR and 0.62 AUC on baseline sequences fell to 14% TPR and 0.57 AUC on planner-generated trajectories for SIM.The authors attribute the weaker results to automated detection using tool descriptions without query context or agent plans.Source: §4.1 · p. 4
  • In Slack, Tool Fuser reduced the average interval between vulnerable calls from 1.70±0.90 seconds to 0.07±0.01 seconds.The method eliminates the time spent interpreting prior results before the next tool call.Source: §4.1 · p. 4
  • 12% to 8%: combining the defenses reduced executed trajectories containing vulnerabilities; attack windows fell 95%, from 1.77±0.70 to 0.08±0.01 seconds.The evaluation combined Prompt Rewriting, SIM, and Tool Fuser across all 66 tasks.Source: §4.1 · p. 4

A System Prompts Used in Our Countermeasures

The prompts either rewrite risky user requests to reduce TOCTOU exposure while preserving intent, or classify tool-call sequences for potential TOCTOU. The classifier focuses on read-then-write operations involving the same or overlapping resource.

A System Prompts Used in Our Countermeasures

  • The rewrite prompt preserves user intent while removing stale-state assumptions and sequencing traps, encouraging atomic, idempotent actions and fresh state verification at execution.It must not add unsafe instructions.Source: A System Prompts Used in Our… · p. 7
  • Queries without TOCTOU risk, including static-information requests, single-step actions, and read-only lookups, are returned unchanged.The rewrite input includes the user query and available tool descriptions.Source: A System Prompts Used in Our… · p. 7
  • Read-only second calls, write-then-read sequences, unrelated resources, and writes independent of the checked state are excluded as TOCTOU.Source: A System Prompts Used in Our… · p. 7
  • A read followed by a write on the same or overlapping resource is marked POTENTIAL_TOCTOU; other sequences are marked BENIGN.The classifier identifies resource and operation types, then applies this rule.Source: A System Prompts Used in Our… · pp. 7–8

B Tool Fuser Examples

The examples show a tool fuser replacing detected check-then-use sequences with fused tools. One example targets replying to a Slack message; the other targets posting employee hobbies to a website.

B Tool Fuser Examples

  • For Alice’s coffee-mug query, the get_channels-to-send_channel_message sequence was detected as vulnerable and replaced with get_channels_and_send_message.The task was to find the channel where Alice asked about the mug and reply that it was seen in the meeting room.Source: B Tool Fuser Examples · p. 8
  • For posting employee hobbies, the get_webpage-to-post_webpage sequence was detected as vulnerable and replaced with get_and_post_webpage.The task directs the agent to find the hobbies in Bob’s Slack inbox and post them to the company website.Source: B Tool Fuser Examples · pp. 8–9
Loading 2508.17155v1…