StateMirror: Pre-Execution Detection of Hallucinated Tool Calls in Coding Agents

• Hallucinated tool use means an action violates actual-environment preconditions, embedding invalid references that make it unexecutable as intended.Source: §1 · p. 1
• Across 4,398 SWE-agent trajectories, hallucinations occurred in 46.6% of failed and 32.3% of resolved trajectories.Source: §1 · p. 1
• Recovery cost 3 steps and 22.4K tokens; 30.3% of these hallucinations never reached a successful grounded retry, while filesystem and namespace mismatches accounted for 43.3% and 41.9% of cases.Source: §1 · p. 2
• Some prior methods detect reasoning failures through post-execution feedback [7] [9] [36] or use LLM judges over execution histories [52].Source: §1 · p. 2
• StateMirror intercepts actions, derives typed state predicates, and deterministically checks them against the live environment, returning a warning when a predicate is violated.Source: §1 · p. 2
• On 2,508 expert-labeled actions, StateMirror achieved 0.940 precision and 0.817 F1, exceeding the strongest LLM judge by 0.213 F1; median detection took 1.50 ms with no token cost.Source: §1 · p. 2
• Feedback rectified 95% of hallucinated tool calls, and intervention across 79 affected tasks reduced execution time by 15.8% and token consumption by 11.8% while maintaining task resolution.Source: §1 · p. 3

2.1 ReAct-style Coding Agents

• Coding agents commonly follow the ReAct paradigm [50]: agents generate structured actions, environments execute them, and harnesses return observations.Source: §2.1 · p. 3

2.2 Preconditions and Runtime Checking

• Pre- and post-condition reasoning checks whether outputs satisfy a specified condition when inputs meet a precondition, such as file existence or a positive variable value.Source: §2.2 · p. 3

3.1 Problem Statement

• Hallucinated tool use is the set of predicates an action assumes that are false in the workspace and harness state immediately before execution.An action is classified as hallucinated when at least one such predicate is false.Source: §3.1 · pp. 3–4

3.2 Study Setup

• The study frames three questions: whether hallucinated calls occur and how often, their recovery cost, and the distribution of their types.Recovery episodes end when the intended command succeeds while correctly referencing the target entity.Source: §3.2 · p. 4, §3.3.2 · p. 5

3.2.1 Dataset and Ground Truth.

• The study confirms 9,595 hallucinated calls by screening SWE-agent trajectories and having annotators inspect each candidate’s workspace state and action history.The corpus retained one trajectory per issue and outcome, with 3,560 failed and 838 resolved trajectories.Source: §3.2.1 · p. 4

3.3.1 RQ1: Existence and Prevalence.

• 46.6% of failed and 32.3% of resolved trajectories contain hallucinated calls, and affected trajectories can contain repeated hallucinations.Overall, 1,931 of 4,398 trajectories (43.9%) were affected.Source: §3.3.1 · pp. 4–5

3.3.2 RQ2: Recovery Cost of Hallucinated Tool Use.

• 3 steps and 22.4K tokens are the median recovery cost before a grounded retry; 30.3% of episodes are never retried.The cost analysis traces 1,606 episodes with an identifiable target entity; 1,120 episodes were recovered.Source: §3.3.2 · p. 5

3.3.3 RQ3: Type Distribution.

• More than 85% of confirmed hallucinations involve filesystem or Python namespace assumptions, making concrete environment properties practical pre-execution checks.These categories appear in 58.2% and 52.7% of trajectories, respectively.Source: §3.3.3 · pp. 5–6

4.1 Overview

• StateMirror assesses action feasibility before execution through normalization, predicate extraction, environment probing, and reporting.The extractor and prober can be extended for new tools and environments without changing these stages.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2 Normalization

• Shell commands and structured tool calls are normalized into a representation containing the working directory, parameters, command, and associated Python code.When available, source code for invoked Python scripts is included so the extractor can analyze what the harness would execute.Source: §4.2 · p. 6

4.3.1 Extraction.

• The extractor derives typed predicates from action syntax, including requirements about modules, paths, and tool arguments; unresolved requirements are omitted.For Python code, it parses the source into an AST on demand.Source: §4.3.1 · p. 7, §4.3 · p. 7

4.3.2 Probing.

• Read-only probes check predicates against repository structure, module definitions, filesystem metadata, runtime information, or harness schemas.If a check cannot reliably determine whether a requirement holds, StateMirror conservatively lets it pass and records the uncertainty.Source: §4.3.2 · p. 7

4.3.3 Predicate Families.

• The four predicate families cover Python namespaces and package layout, filesystem and working-directory state, command and runtime state, and harness and tool schemas.For example, filesystem checks resolve relative paths against the current directory and distinguish file-existence requirements from parent-directory requirements.Source: §4.3.3 · pp. 7–8, Figure 4 · p. 8
• AST checks can distinguish an available package from an unavailable exported symbol, while filesystem checks resolve script paths against the current working directory.The examples identify a missing dvc.DVC symbol and a missing reproduce.py script before execution.Source: §4.3.3 · pp. 8–9
• Command and test checks verify executable availability, runtime versions, and requested pytest targets against the environment and target-file AST.A pytest target is invalid when its file exists but the requested test node does not.Source: §4.3.3 · p. 9
• Harness and tool-schema checks validate argument constraints and state-dependent requirements, such as whether an editor file is open or a line number is valid.A false probe causes StateMirror to report the violated predicate as a typed warning describing the mismatch.Source: §4.3.3 · p. 9, §4.4 · p. 9

5.1 Environment and Implementation

• StateMirror is implemented as a Python toolkit that normalizes actions and checks predicates with deterministic, read-only probes.It does not execute actions or consume LLM tokens; runtime experiments integrate it with SWE-agent.Source: §5.1 · p. 10

5.2 Dataset

• Annotators inspect candidate failure actions independently, while prior edits are replayed to reconstruct each action’s pre-action workspace.The benchmark is built from failed trajectories; RQ4 separately evaluates resolved-run instances from the preliminary study.Source: §5.2 · pp. 10–11
• The benchmark contains 2,508 actions from 1,197 trajectories across four agent frameworks, with positives spanning five families.The evaluation retains hallucinations outside StateMirror’s predicate vocabulary as false negatives.Source: §5.2 · p. 11

5.3 Metrics

• Detection effectiveness is measured with precision, recall, F1-score, and accuracy against ground-truth labels.The evaluation also measures latency, token consumption, action repair, intervention savings, and task resolution.Source: §5.3 · p. 11

5.5 RQ4: Effectiveness Analysis

• 0.817 F1 and 0.940 precision: StateMirror exceeds the strongest LLM judge by 35.3% in F1 and the scaffold linter by 1,715.6%.It also reaches 0.819 accuracy, outperforming the most accurate baseline, Kimi K3, by 33.0% in accuracy.Source: §5.5 · p. 12, §5.5.1 · p. 13

5.4 Baselines

• Static linting reaches only 0.045 F1, while the strongest LLM judge, Qwen3.7 Plus, reaches 0.604 F1.The action-only baseline has decent recall but high false positives because it does not probe the repository.Source: §5.5 · p. 12

5.5 RQ4: Effectiveness Analysis

• 73.8% (288/390) of false negatives stem from action extraction or selected-file reconstruction, while symbol predicates have the highest recall at 0.802.False positives are mostly tied to runtime-generated files, path-like strings, and transient state absent from offline reconstruction.Source: §5.5 · p. 12

5.5.1 Hallucinated tool calls from resolved runs.

• 0.692 F1: on resolved runs, StateMirror achieves 0.989 precision and 0.532 recall, detecting at least one hallucination in 81.5% of trajectories.Dynamic runtime attribute errors and temporary files fall outside its static predicate vocabulary; excluding dynamic cases raises recall to 79.1% (254/321).Source: §5.5.1 · pp. 12–13

5.6 RQ5: Cost Analysis

• 1.50 ms median latency and zero token cost: StateMirror’s detection is over four orders of magnitude faster than LLM judges.LLM judges have median latencies of 14.51–17.73 s and consume 1.9K–19.2K tokens per action.Source: §5.6 · p. 13

5.7 RQ6: Ablation Analysis

• 0.817 F1: filesystem and cwd predicates raise F1 from 0.691 to 0.801, extending a high-precision namespace and package-layout core.Family I alone has 0.972 precision; removing it drops F1 to 0.323, while adding Family IV raises overall F1 to 0.817.Source: §5.7 · pp. 13–14, Table 6 · p. 14

5.8 RQ7: Sensitivity Analysis

• MH1 and MH2 leave F1 at 0.817 while increasing p95 tail latency to 1.65× and 2.68× the default, respectively.MH2 recovers eight true positives but incurs eleven false positives; deeper package surfaces can weaken static-check localization.Source: Table 7 · p. 14, §5.8 · pp. 14–15

5.9 RQ8: Mitigation of Hallucinated Tool Calls

• Intervention blocks only falsified predicates that are necessary preconditions, while permitting intentional side effects and diagnostic errors.For example, a test command is blocked only when its target or runner is missing.Source: §5.9 · p. 15

5.9.1 Action repair.

• Warnings enable successful repair in 95.0% of controlled cases and 97.8% of live executions.In live execution, the agent’s first replacement clears the mismatch and executes successfully in 45 of 46 cases.Source: §5.9.1 · pp. 15–16

5.9.2 Pre-execution intervention.

• 15.8% less time and 11.8% fewer tokens per triggered task: intervention also reduces steps by 6.0%, while both modes resolve 48 of 79 tasks.Full trajectories average 622.7 s, 52.7 steps, and 709.9K tokens under intervention, versus 739.2 s, 56.0 steps, and 805.0K tokens in report mode.Source: Table 9 · p. 16, §5.9.2 · pp. 16–17

6 Case Study

• At step 17, a file_exists warning redirected the agent from a non-existent test path to tests/runtests.py, and the trajectory ended with a resolved patch.The agent inspected the test suite before switching to the valid test runner.Source: §6 · p. 17
• At step 7, StateMirror flagged importable(repro_app) as false, prompting the agent to create missing definitions before reproducing the issue and patching ManyToManyRel.identity.The attempted reproduction script imported a non-existent repro_app.Source: §6 · p. 17
• The case study presents StateMirror as a fail-fast safeguard that prevents uninformative runtime failures and reduces wasted recovery cost.The interventions guided the agent toward valid workspace state while it made progress on the task.Source: §6 · p. 17

7 Related Work

• Traditional debugging and feedback-driven systems inspect failures after execution, while trajectory frameworks analyze loops or prefixes for attribution, diagnosis, detection, and repair.These approaches use runtime tracebacks or execution trajectories to identify failure patterns.Source: §7 · p. 17
• AgentSpec [41] checks user-specified safety requirements, whereas StateMirror derives factual preconditions from the proposed tool call and verifies them with deterministic environment probes.StateMirror’s checks concern the current execution environment.Source: §7 · pp. 17–18
• Existing methods do not check whether required modules, symbols, and paths exist before the next tool call, while post-execution systems resolve errors using tracebacks.The paper characterizes this post-execution approach as increasing execution steps and token consumption.Source: §7 · p. 18

8 Threats to Validity

• Independent expert annotations, consensus adjudication, and replayed verified prefix edits were used to support label validity and high-fidelity workspace reconstruction.The evaluation also includes failed and resolved trajectories across four agent frameworks to minimize framework-specific bias.Source: §8 · p. 18
• On resolved trajectories, recall is lower because some errors involve dynamic runtime states rather than static mismatches.Current deterministic predicates prioritize actionable, high-precision static checks; dynamic execution-time validation remains future work.Source: §8 · p. 18
• The empirical evaluation covers Python bug-fixing tasks, SWE-agent trajectories, and the Qwen3.8-27B model, while broader language and runtime support requires domain-specific extractors and probers.The architecture is described as designed for broader generalizability, with its pre-execution workflow remaining intact.Source: §8 · p. 18

9 Conclusion

• On 2,508 actions, StateMirror achieved 0.940 precision and 0.817 F1, outperforming the strongest LLM judge by 35.3% in F1, with zero token cost and millisecond latency.Source: §9 · p. 18
• StateMirror extracts predicates from proposed actions and deterministically validates them against the live environment before execution.The conclusion describes it as a lightweight pre-execution detector.Source: §9 · p. 18
• During live intervention, StateMirror enabled agents to rectify over 95% of hallucinated tool calls and reduced execution time by 15.8% and token consumption by 11.8%.Task resolution was maintained during the live mitigation.Source: §9 · p. 18