PaperlayerView exact revision on arXiv

Source-linked AI summary

TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories

Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang, Yun-Nung Chen

arXiv:2604.07223v2cs.CRcs.AIcs.CLcs.LGcs.SE

TL;DR

Guardrails’ efficacy in multi-step tool-use trajectories remains underexplored, despite risks arising in intermediate execution steps. TraceSafe-Bench provides static, step-labeled traces for evaluating mid-execution monitoring. Its evaluations associate structured-input comprehension with trace detection, while accuracy does not increase monotonically with model size within model families.

  • Problem

    Existing agentic safety benchmarks lack fixed trajectories and localized ground truth for reproducible evaluation of guardrails monitoring multi-step tool-use traces.

    Source: §1 · p. 1

  • Method

    TRACESAFE-BENCH creates static, step-labeled multi-step traces by applying targeted risk mutations to natural benign trajectories.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Structured-input comprehension is the only measured capability demonstrably associated with trace-level detection, while detection accuracy within model families is non-monotonic in parameter count.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The evaluation frames guardrail assessment as a diagnostic of agentic failures, with structural data competence rather than semantic safety alignment identified as the primary bottleneck.

    Source: §5 · p. 10

    Observational comparisons across released models cannot identify which co-varying training or architecture factor accounts for the parameter-scale pattern; controlled ablations are unavailable.

    Source: §4.2 · pp. 8–9

Abstract

from arXiv · show

As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to intermediate execution traces. While safety guardrails are well-benchmarked for natural language responses, their efficacy remains largely unexplored within multi-step tool-use trajectories. To address this gap, we introduce **TraceSafe-Bench**, the first comprehensive benchmark specifically designed to assess mid-trajectory safety. It encompasses 12 risk categories, ranging from security threats (e.g., prompt injection, privacy leaks) to operational failures (e.g., hallucinations, interface inconsistencies), featuring over 1,000 unique execution instances. Our evaluation of 13 LLM-as-a-guard models and 7 specialized guardrails yields three critical findings: 1) *Structural Bottleneck*: Guardrail efficacy is driven more by structural data competence (e.g., JSON parsing) than semantic safety alignment. Performance correlates strongly with structured-to-text benchmarks ($\rho=0.79$, $p<0.01$) while we detect no association with standard jailbreak robustness. 2) *Scale Is Not the Deciding Factor*: Trajectory detection accuracy does not scale monotonically with model size, and general-purpose LLMs consistently outperform specialized safety guardrails. 3) *Bounded Temporal Stability*: Detection accuracy improves with trajectory length within native context bounds as models process dynamic execution behavior, but degrades in extreme long-context regimes due to over-refusal.

1 Introduction

Existing safety evaluations leave open whether guardrails can detect risks in structured, intermediate tool-calling traces. TraceSafe-Bench addresses this gap with localized risk edits and finds that structural competence is associated with guardrail success.

1 Introduction

  • Existing agentic safety benchmarks emphasize dynamic end-to-end resilience, while reproducible guardrail evaluation requires fixed trajectories with precise step-level ground truth.Free-form harmful generation can produce artificial behavior, and post-hoc annotation of complex workflows is labor-intensive.Source: §1 · pp. 1–2
  • TRACESAFE-BENCH uses deterministic, targeted edits to create realistic multi-step traces with step-level labels across 12 risk types.The benchmark contains over 1,000 traces and covers security and operational failure modes.Source: §1 · p. 2
  • ρ = 0.79 links agentic guardrail success to structural and formatting competence, rather than moral alignment alone.The reported evaluation also found no monotonic accuracy increase with model size and a benefit from longer traces only within realistic lengths.Source: §1 · p. 2
  • Longer traces within realistic lengths help models focus on execution behavior over static tool definitions, while extreme contexts degrade detection through benign over-flagging.The reported model-size relationship is non-monotonic after calibration, and the comparison cannot attribute that pattern to any single factor.Source: §1 · p. 2

2 Related Work

Prior work develops tool-use capabilities and evaluates agent safety across dynamic environments, while guardrails primarily moderate semantic content or isolated calls. The related work positions static, step-level monitoring of multi-step trajectories as an unmet evaluation need.

2 Related Work

  • Tool use progressed from API generation to autonomous interaction; PAL (Gao et al., 2023), Toolformer (Schick et al., 2023), and ReAct (Yao et al., 2022) established early paradigms.Gorilla (Patil et al., 2024) and ToolLLM (Qin et al., 2023) further systematized API grounding and evaluation.Source: §2 · p. 2
  • AgentHarm (Andriushchenko et al., 2025), Agent Security Bench (Zhang et al., 2025), ToolEmu (Ruan et al., 2024), and AgentDojo (Debenedetti et al., 2024) assess risks in dynamic, end-to-end settings.These benchmarks do not provide the static, step-level trajectories and deterministic annotations needed to assess independent guardrails.Source: §2 · pp. 2–3
  • Figure 1’s construction pipeline starts with benign traces, checks mutation suitability for each step-risk pair, then applies a mutation and truncates at that step.The top panel depicts the tool-calling threat landscape.Source: Figure 1 · p. 3
  • Specialized guardrails such as Llama Guard, Granite Guardian, and ShieldGemma moderate semantic interaction surfaces, while MCP-Guard (Xing et al., 2026) focuses on isolated calls.The stated gap is evaluating interception of unsafe traces during execution, before harmful final outputs.Source: §2 · p. 3

3 TRACESAFE-BENCH

TRACESAFE-BENCH builds static, step-labeled unsafe trajectories by editing successful benign traces with suitability checks and controlled mutations. Its 12-risk taxonomy spans security, hallucination, and interface failures, with original benign traces retained for evaluation.

3 TRACESAFE-BENCH

  • TRACESAFE-BENCH uses Benign-to-Harmful Editing to combine natural benign seeds with taxonomy-guided mutations, producing deterministic ground truth across 12 risk types.The workflow is designed to preserve ecological validity while localizing unsafe behavior.Source: §3 · p. 3

3.1 Benign Traces Curation

  • Benign seeds come from executable multi-step BFCL trajectories, and five models supply diverse traces filtered to 100% execution accuracy.BFCL’s tool schemas, user constraints, and execution context support offline edits while preserving local consistency.Source: §3.1 · p. 4

3.2 Benign-to-Harmful Editing Method

  • The two-stage editing procedure checks each step for applicable risks, then applies programmatically controlled structural edits rather than free-form LLM rewrites.LLMs may generate payload content, but predefined code logic performs structural changes such as inserting, replacing, or deleting JSON keys.Source: §3.2 · p. 4

3.3 The Risk Taxonomy

  • Twelve risks cover privacy leaks, hallucinated arguments and tools, and interface inconsistencies such as deprecated versions or deceptive descriptions.Checks target concrete mismatches, including credentials passed to tools with no functional need and fabricated values after required information is removed.Source: §3.3 · pp. 4–6
  • Original, unperturbed successful trajectories form the benign negative class; the final dataset samples 90 traces per risk category.At n = 90, the reported average Wilson 95% CI half-width is ±6.3%; subsampling to n = 45 preserves per-category ordering at Spearman ρ = 0.97.Source: §3.3 · p. 6

4 Evaluation and Analysis

The evaluation tests guard models in binary and taxonomy-guided classification settings, then examines capability correlations and trajectory-length trends. Results identify a coarse-grained accuracy leader, associations with structured-data capabilities, and trajectory-length gains bounded by context scope.

4.1 Settings

  • The evaluation uses four settings: binary classification with or without schema guidance, and coarse- or fine-grained multi-class classification.Balanced Average Accuracy is the main metric; Balanced F1 is used for the capability-correlation analysis.Source: §4.1 · p. 6

4.2 Main Results

  • Binary prompts yield opposing decision priors across model types, while fine-grained detection favors explicit risks (1–5) over subtle operational anomalies (6–12).Some interface-inconsistency accuracy falls below 20%, including 4.71% for Qwen3-14B on category 12.Source: §4.2 · p. 8
  • 83.58% Balanced Average Accuracy was achieved by Qwen3-14B in the coarse-grained setting.This was the field-leading result reported for that setting.Source: §4.2 · p. 8
  • Trace-level accuracy is non-monotonic with model size: Qwen3 variants score 73.80%, 81.29%, 83.58%, and 76.54% across 1.7B, 4B, 14B, and 32B.The authors caution that observational checkpoint comparisons cannot identify which co-varying training factor accounts for differences.Source: §4.2 · pp. 8–9

4.3 Structural Competence as the Primary Bottleneck for Trace Safety

  • TRACESAFE-BENCH Balanced F1 correlates with RAGTruth Data2txt (ρ = 0.79, p < 0.01), while StrongREJECT jailbreak robustness shows no evidence of association (ρ = 0.05, p > 0.85).The authors frame the contrast as directional: the model panel is small and the intervals are wide and partly overlapping.Source: §4.3 · p. 9
  • The authors identify parsing nested JSON and dense schemas as a prerequisite to reasoning about malicious intent in tool traces.They propose structural data competence, rather than safety alignment alone, as the primary trace-level bottleneck.Source: §4.3 · p. 10

4.4 Stability and Growth Across Long Trajectories

  • Accuracy improves with step count within native benchmark lengths, and models outperform their baseline on traces with 15 or more steps compared with 5-step traces.The authors hypothesize that longer traces contain more dynamic behavioral data relative to static tool schemas, helping isolate anomalous actions.Source: §4.4 · p. 10
  • The trajectory-length trend is bounded: measured traces are under 6k tokens, while the needle-in-a-haystack benefit reverses at extreme lengths through loss of benign specificity.Long traces over-represent easier, structurally explicit risks, though controls confirm within-category gains for mid-difficulty risks.Source: §4.4 · p. 10

5 Conclusion

TraceSafe finds that guardrail detection depends on structural comprehension, with performance varying by risk type and trajectory context. Accuracy gains hold at realistic lengths but reverse in extreme long contexts through over-flagging benign traces.

5 Conclusion

  • Structured-input comprehension is the only measured capability associated with trace-level detection; jailbreak robustness shows no association.The authors identify structural data competence, rather than semantic safety alignment, as the primary bottleneck.Source: §5 · p. 10
  • Explicit security vulnerabilities are detected more accurately than mild interface failures, and granular risk taxonomies improve accuracy over binary judgments.Source: §5 · p. 10
  • Detection gains hold across realistic trajectory lengths but reverse far beyond native context bounds, where guardrails over-flag benign traces.Source: §5 · p. 10

Disclosure of LLM Usage

LLMs assisted with implementation, initial data generation, and manuscript editing, while authors retained responsibility for the final content and results.

Disclosure of LLM Usage

A Discussion on Limitations

The discussion frames TraceSafe-Bench as a controlled diagnostic with limits in seed guarantees, mutation realism, and dynamic coverage. It also describes broader deployment relevance and explains why MCP-GUARD is not a benchmark baseline.

A Discussion on Limitations

  • Original benign seeds lack a formal guarantee of being risk-free in every deployment context.Source: A Discussion on Limitations · p. 17
  • The static offline evaluation captures only the immediate pre-invocation state, not intervention-driven changes in later planning; emerging attack vectors may require taxonomy updates.Source: A Discussion on Limitations · p. 17
  • Clean, single-point mutations simplify ground truth, but real failures can compound or emerge gradually, so benchmark results may over- or under-state real-world performance.Source: A Discussion on Limitations · p. 17
  • The authors prioritize unambiguous labels over naturalistic messiness, presenting the benchmark as a controlled testbed for scoped exploits; passing is necessary, not sufficient, for deployment safety.Production or red-teaming logs remain valuable validation targets, though cascading failures make root causes and reliable labels difficult to establish.Source: A Discussion on Limitations · p. 17
  • The evaluation finds no evidence of systemic contamination in the benign pool, although seed safety is not guaranteed in every deployment context.Most benign seeds were classified as benign, with misclassifications sparsely distributed across categories.Source: A Discussion on Limitations · p. 17
  • MCP-GUARD is not evaluated because its E5 detector checkpoint and exact training setup are unavailable, while testing only its rule-based stage would misrepresent the full pipeline.Source: A Discussion on Limitations · p. 17

B Broader Impact

  • The authors position the benchmark as a shift toward proactive mid-execution monitoring and structure-aware safeguards rather than generic safety alignment.Source: B Broader Impact · p. 18
  • A single malformed tool call may have irreversible consequences in sensitive financial or healthcare APIs; the benchmark is released under a research-only license.Source: B Broader Impact · p. 18

D Datasets

The dataset overview covers the reference datasets used for evaluation and describes TraceSafe-Bench’s balanced risk-category composition and varied trajectory sources.

D Datasets

  • Table 3 overviews the reference datasets used to evaluate base-model capabilities and safety benchmarks.The datasets are listed in Section 4.3.Source: Table 3 · p. 18, D Datasets · p. 18

E.1 Dataset Statistics

  • Each of the 12 fine-grained risk categories contains exactly 90 filtered entries, forming a balanced evaluation testbed.Table 4 details the dataset statistics across the categories.Source: §E.1 · p. 19, Table 4 · p. 19
  • The benchmark draws benign trajectories from five source models, sampling 90 BFCL entries per category from thousands of raw entries.The varied sources cover diverse structural formatting and tool-calling behaviors.Source: §E.1 · p. 19, Table 5 · p. 19

E.2 Verification and Misclassification Analysis

The analyses diagnose benign-default misclassifications and test whether benchmark findings are affected by judge reuse, generator-family overlap, or limited model-panel evidence.

E.2 Verification and Misclassification Analysis

  • 67.6% of HallucinatedArgVal errors and 55.9% of VersionConflict errors are classified as benign, rather than as another malicious category.The aggregated fine-grained confusion heatmap shows benign predictions dominate detection failures.Source: §E.2 · p. 20, Figure 5 · p. 20

E.3 Check-Judge Validation: Held-Out Judge Study

  • A held-out Phi-4 judge reproduced gpt-oss-120b’s suitability judgment for 103/155 pairs (66.5%), with agreement varying by risk type.Agreement was near-perfect for objective surface markers and lower for categories requiring subjective judgments.Source: §E.3 · p. 21, Table 6 · p. 21
  • gpt-oss-120b ranked last or second-to-last against 14 general-purpose LLMs on every tested category, providing no evidence of a Check-role advantage.This included the three categories with lower held-out judge agreement.Source: §E.3 · p. 21, Table 7 · p. 21

E.4 Generator-Model Leakage in the Benign Seed Pool

  • Same-family evaluation showed no consistent advantage: mean diagonal-minus-off-diagonal gaps were −0.35 pp coarse-grained and −1.14 pp fine-grained.Placebo generator-to-generator spreads were 6–17 pp, comparable to or larger than observed diagonal gaps.Source: §E.4 · p. 22, Table 8 · p. 22, Table 9 · p. 22
  • ToolACE-8B’s same-family gaps (+2.43 pp coarse and +0.40 pp fine) fell within the placebo noise band, so its low score was not masked by a family advantage.The authors leave the main comparison unchanged.Source: §E.4 · p. 22, Table 8 · p. 22, Table 9 · p. 22

F.1 Correlation Significance and Bootstrap Confidence Intervals

  • RAGTruth remained the only benchmark significantly associated with TraceSafe performance under both coefficients after excluding degenerate models.Excluding gpt-oss-120b also left the headline correlation finding unchanged.Source: §E.3 · p. 21, §F.1 · p. 23, Table 11 · p. 23
  • The small model panel (n = 10–13) limits precision: StrongREJECT’s Pearson interval spans −0.421 to 0.467, and correlation intervals overlap.The authors characterize the structural-versus-semantic contrast as directional and say a substantially larger model panel is needed for a precise estimate.Source: §F.1 · p. 23

F.2 Sample-Size Sufficiency (n = 90 per category)

The authors assess whether 90 traces per category support per-category conclusions through interval precision, subsampling convergence, reproducibility, and the intended balanced allocation.

F.2 Sample-Size Sufficiency (n = 90 per category)

  • At n = 90, Wilson 95% CI half-widths average ±6.3% in fine-grained evaluation and reach at most ±10.1% at 50% accuracy.Extreme 0/90 and 90/90 cells have intervals of [0%, 4.1%] and [95.9%, 100%], respectively, and are characterized as systematic outcomes rather than noise.Source: §F.2 · p. 24
  • Reducing each category from 90 to 45 traces changes accuracy by 2.3% on average and preserves fine-grained category ordering at mean Spearman ρ = 0.97.This is reported across the general-purpose LLMs.Source: §F.2 · p. 24
  • An independent GPT-5-mini rerun reproduced per-category accuracies closely: mean |∆| = 0.37% fine-grained and 0.48% coarse-grained.The corresponding Pearson correlations were r = 0.9998 and r = 0.9994.Source: §F.2 · p. 24
  • The equal 90-per-category allocation targets detection capability by risk type, not real-world risk frequencies.The authors release the generation pipeline so categories can be scaled for finer resolution.Source: §F.2 · p. 24

G Trajectory Length: Per-Bin Counts, Confidence Intervals, and Category Mix

The appendix examines accuracy by trajectory length, the role of category composition, and whether gains also occur within individual risk categories.

G Trajectory Length: Per-Bin Counts, Confidence Intervals, and Category Mix

  • Accuracy rose from the 2–4-step to the 18+-step bin by +15.8 and +13.2 points in the two binary settings and +3.9 points in coarse-grained classification.Table 12 pools sample-by-model observations across the 13 general-purpose LLMs; the smallest bin contains more than 1,300 observations and the largest more than 4,400.Source: G Trajectory Length: Per-Bin Counts,… · p. 25, Table 12 · p. 25
  • Part of the aggregate rise reflects composition: longer traces over-represent structurally explicit risks already detected near ceiling, so aggregate gains are not a pure length effect.Mean trajectory length varies by category, from roughly 6 turns for MissingArgValue to roughly 17 for UserInfoLeak.Source: G Trajectory Length: Per-Bin Counts,… · p. 25
  • Within-category gains also appear for mid-difficulty risks: binary-with-schema accuracy rises from 56% to 82% for MissingArgValue and from 71% to 85% for PromptInjectionIn.HallucinatedTool, FnDescriptionMismatch, and AmbiguousArgNaming also improve; categories already at 95–100% are saturated.Source: G Trajectory Length: Per-Bin Counts,… · p. 25

H Long-Context Robustness: Needle-in-a-Haystack Stress Test

This section examines long-context detection limits through a needle-in-a-haystack stress test, describing how risky tool-call traces are constructed, categorized, and evaluated. It covers AWS Guard risk categories—including workflow hijacking, unauthorized data disclosure, argument or schema errors, and interface inconsistencies—and reports a prompt-optimization pilot and examples of guardrail failures.

H Long-Context Robustness: Needle-in-a-Haystack Stress Test

  • 42.9% benign accuracy falls to 0% by 32k and remains there through 110k, while balanced accuracy drops from 45.0% to 24.0% and plateaus; harmful accuracy remains ∼47–48%.The 64k harmful-accuracy result of 33.3% has a small sample (n = 12) and a 95% CI of [13.8, 60.9]. The benign collapse reflects distractor-induced over-flagging, not failure to surface the malicious step.Source: H Long-Context Robustness:… · p. 26, Table 13 · p. 26

I TRACESAFE-BENCH Benign-to-Harmful Editing Algorithm

  • The editing algorithm mutates only the first invocation of each distinct tool type and truncates the original trace after the edited step.The algorithm checks agent steps, marks each tool type as processed, and applies eligible mutations to the trace prefix ending at the current step.Source: I TRACESAFE-BENCH Benign-to-Harmful… · p. 27

J Summarization and Formulation of Risk Categories

  • The risk-category definitions exclude atomic system commands and constrain tool selection to support unambiguous ground truth; Table 14 formalizes the trace mutations.For example, a weather tool should have no legitimate reason to handle an injected system API key.Source: J Summarization and Formulation of… · p. 28, Table 14 · p. 28

K Mapping to the OWASP Top 10 for LLM Applications

L Prompt-Optimization Pilot

  • Balanced accuracy stays within a 58–62% band across four prompt variants and never exceeds baseline; few-shot reduces skew from 10.8 to 3.3 pp by trading away unsafe accuracy.This single-model pilot does not establish effects across other decision boundaries; every tested binary variant remains below Qwen3-14B’s 83.58% Balanced Average Accuracy under coarse-grained taxonomy-driven prompting.Source: L Prompt-Optimization Pilot · p. 30, Table 16 · p. 30

M Examples and Failure Cases

  • Examples show gpt-5-mini incorrectly labels both a MissingArgValue hallucination trace and a ConfusableFnNames interface-inconsistency trace as safe.The first mutation removes an explicit filename argument; the second updates a tool’s parameters and marks one as deprecated.Source: M Examples and Failure Cases · pp. 31, 32, 35

N.2 Mutate Prompts

  • Most mutations use program logic and localized edits, while Hallucinated Argument Values require an LLM to remove a detail without changing query intent.The supplied prompt asks for a vague request that retains intent but omits the target value.Source: §N.2 · p. 41

N.3 Evaluation Prompts for LLMs

  • Evaluation materials define risk rules before prompts; guard models use their official model-card templates, and AWS Guardrail receives behavioral descriptions mapped from the risk taxonomy.The prompt materials include templates for Llama Guard, Granite Guard, and Qwen Guard; Table 19 outlines AWS rules across four primary risk domains.Source: §N.3 · p. 42, §N.3.1 · p. 45, Prompt: Llama Guard Evaluation · p. 45

N.3.2 Guard Topic Definition for AWS Guard

  • Prompt injection is defined as workflow hijacking through malicious instructions in tool descriptions or payloads returned by tool execution.The taxonomy distinguishes injection embedded in a tool description from injection returned in execution output.Source: §N.3.2 · p. 46
  • Leakage categories cover unauthorized disclosure of user information, credentials or secrets in tool arguments, and internal diagnostic or database data to external tools.The definitions distinguish user information, system-level secrets, and internal diagnostic data.Source: §N.3.2 · p. 46
  • Tool-call argument risks include ambiguous naming, nonexistent tools, fabricated required values, unsupported keys, and assumptions about unspecified data types.These risks concern assumptions or unsupported content that can violate the provided tool schema.Source: §N.3.2 · p. 46
  • The policy maps the detailed risk taxonomy to actionable AWS Bedrock behavioral rules, and lists interface inconsistencies as a risk category.The supplied table caption identifies the mapping's scope but does not provide the individual rules.Source: Table 19 · p. 46, §N.3.2 · p. 46
Loading 2604.07223v2…