StepShield: When, Not Whether to Intervene on Rogue Agents

1 Introduction

• Existing benchmarks render a binary verdict after a trajectory ends and provide no step-level divergence annotations or metric for late detection.They therefore cannot distinguish an alert at step 8 that enables intervention from one at step 48 that offers only forensic value.Source: §1 · pp. 1–2
• StepShield defines EIR, Intervention Gap, and Tokens Saved to measure timeliness separately from accuracy, with premature prefix alerts receiving zero EIR credit.EIR is informationally non-redundant with accuracy: detector pairs can have identical accuracy but arbitrarily different EIR, and vice versa.Source: §1 · p. 2
• The benchmark provides 9,429 step-level annotated code-agent trajectories across six violation categories, generated from 127 real-world security incidents.The held-out test set has zero overlap with training data.Source: §1 · p. 2
• 86% recall still leaves a pattern-based guardrail statistically indistinguishable from random timing on EIR (0.23 vs. 0.24; p = 0.66, one-sided binomial).The 4× EIR gap between rule-based and semantic detectors is invisible to accuracy, recall, and F1.Source: §1 · p. 2

Foundational LLM Agent Architectures

• Capability benchmarks such as AgentBench, GAIA, WebArena, and OSWorld assess general-purpose agent abilities, while StepShield focuses on safety evaluation with temporal granularity.The passage reports that OSWorld’s best models achieve 12% success compared with 72% for humans.Source: §2 · p. 3

Agent Safety Benchmarks

• Agent-safety benchmarks cover diverse modalities but label completed runs with a single binary outcome, without step-level annotations for evaluating timeliness.StepShield fills that gap.Source: §2 · p. 3

Guardrail Agents

• Guardrail agents enforce policies through logical reasoning or dynamically generated code, while StepShield provides infrastructure to measure intervention timeliness.Its EIR, IG, and Tokens Saved metrics can evaluate any guardrail system.Source: §2 · p. 3

Content-Safety Guardrails

• Content-safety guardrails classify individual messages against policies, whereas StepShield evaluates multi-step trajectories with temporal detection as the primary concern.The passage also notes concurrent runtime-monitoring work from ToolSafe, AgentSpec, METR’s SHUSHCAST, and CTRL-ALT-DECEIT.Source: §2 · p. 3

Specification Gaming and Deceptive Alignment

• Specification gaming and deceptive behavior motivate runtime monitoring; StepShield grounds its taxonomy in documented real-world examples.The passage cites specification gaming, sycophancy and reward tampering [Perez et al., 2022], and Sleeper Agents [Hubinger et al., 2024].Source: §2 · p. 3

Step-Level Evaluation

• StepShield applies step-level evaluation to safety, providing a step-level annotated dataset for rogue behavior detection.It draws on process-reward-model and runtime-verification traditions, where detection time is treated alongside detection rate.Source: §2 · p. 3

3.1 Dataset Composition

• The dataset contains 9,429 trajectories, including 1,278 training trajectories, a 7,935-trajectory test set with an 8.1% rogue rate, and 216 held-out trajectories.The held-out set uses incident templates not used in training or pattern development.Source: §3.1 · p. 4

3.2 Violation Taxonomy

• The taxonomy organizes documented real-world incidents into six categories: data exfiltration, privilege escalation, resource hijacking, supply-chain attack, destructive action, and covert persistence..Source: §3.2 · pp. 4–5

3.3 Data Generation Methodology

• Researchers curated 127 incident patterns from production observations, security databases, and documented AI failures, then decomposed them into task specifications, violation patterns, and divergence signatures.The catalog abstracts proprietary details while preserving operational realism.Source: §3.3 · p. 5
• Frontier models generate matched rogue-clean trajectory pairs that differ by the violation, enabling precise temporal evaluation.A rogue trajectory diverges at an annotated step while retaining plausible task-relevant behavior before and after it.Source: §3.3 · p. 5
• Each trajectory receives independent LLM cross-validation and review by four security-background annotators plus a senior resolver, with divergence-step agreement κ = 0.82.Disagreements are flagged for manual review, and trajectories failing review are regenerated.Source: §3.3 · p. 5

3.4 Scope and Design Rationale

• Controlled generation enables precise control over divergence points, violation subtlety, and category balance for temporal evaluation.The surface presentation is cleaner than raw production logs, a deliberate design choice described as incident-grounded controlled generation.Source: §3.4 · pp. 5–6

4.1 Definitions

• EIR measures the fraction of detected rogue trajectories flagged from the first rogue step through a k-step intervention window, excluding premature alerts.The default k = 3 corresponds to a typical action–observe–act cycle; EIR isolates timing quality from coverage.Source: §4.1 · p. 6
• Intervention Gap is the mean signed step delay between rogue onset and detection; negative values indicate premature alerts on benign prefixes.Unlike EIR, IG conditions on detection and reports the delay in natural units of steps.Source: §4.1 · p. 6
• Tokens Saved measures the fraction of rogue-suffix tokens whose execution is prevented, counting only detections at or after rogue onset.Premature detections are excluded because they terminate legitimate work; for detectors with many such alerts, the measure is an optimistic upper bound on safety benefit.Source: §4.1 · pp. 6–7

4.2 Formal Properties

• EIR and accuracy are informationally non-redundant: detector pairs can have identical accuracy but arbitrarily different EIR, and vice versa.The constructed examples show identical accuracy with maximally different EIR, and identical EIR with different accuracy.Source: §4.2 · p. 7

5 Experimental Protocol

• The study compares pattern matching, constraint enforcement, semantic reasoning, and hybrid cascades to characterize their temporal strengths and failure modes.The authors describe these as representative architectural choices and note that additional methods can be evaluated using the released infrastructure.Source: §5 · p. 7

5.1 Detection Methods

• StaticGuard uses 847 regular expressions; ConstraintGuard enforces 23 policy categories, LLMJudge evaluates each step in context, and HybridGuard escalates uncertain pattern-matching verdicts to LLMJudge.LLMJudge uses GPT-4.1-mini, selected for its recall–cost trade-off; HybridGuard resolves high-confidence cases without invoking the LLM.Source: §5.1 · pp. 7–8

5.2 Evaluation Protocol

• Evaluation uses a 216-trajectory held-out test set with zero overlap with development data; temporal metrics use 108 rogue trajectories and FPR uses 108 clean trajectories.Detectors process one step at a time, record only the first flag, receive no look-ahead, and monitor proposed actions before execution.Source: §5.2 · p. 8

6.1 Main Results

• EIR3 is 0.89 for LLMJudge versus 0.23 for StaticGuard, whose result is statistically indistinguishable from random timing at 0.24 (p = 0.66).StaticGuard has 86% recall, but its alerts often fire on benign prefixes; the 4× EIR gap is understated by F1 (0.95 vs. 0.65).Source: §6.1 · p. 8
• StaticGuard has IG = −5.0 and a 77.8% false-positive rate, while ConstraintGuard has EIR = 0.08 and 24% overall recall.ConstraintGuard has 0% recall on Privilege Escalation and Resource Hijacking, while StaticGuard's average alert precedes rogue onset by five steps.Source: §6.1 · pp. 8–9
• The Forensics Trap describes how trajectory-level metrics conflate premature false alarms with genuine early intervention; StaticGuard's true-positive timing is itself indistinguishable from random.Its regex rules match syntax throughout trajectories, and HybridGuard inherits the pathology: its EIR is 0.40 despite LLMJudge's precision in the second stage.Source: §6.1 · p. 9
• LLMJudge saves 91.2% of rogue-suffix tokens through early detection, with 93% of detections at or after rogue onset and a median gap of zero steps.StaticGuard's apparent 79% Tokens Saved applies only to detections at or after onset; most of its alerts are premature, consistent with its 77.8% clean-trajectory FPR.Source: §6.1 · p. 9

6.2 False-Positive Analysis and Detection Trade-offs

• LLMJudge has 95.4% recall, 5.6% FPR, and 89% EIR, while StaticGuard has 86% recall and 78% FPR; ConstraintGuard is strictly dominated on all metrics.At an 8.1% rogue base rate, LLMJudge's PPV is 0.60 versus StaticGuard's 0.089.Source: §6.2 · p. 9

6.3 Per-Category Analysis

• LLMJudge's EIR is 1.00 for supply chain attacks, destructive actions, and resource hijacking, but 0.78 for data exfiltration and 0.72 for privilege escalation.Covert persistence has EIR = 0.89; StaticGuard's EIR remains low across categories (0.18–0.31).Source: §6.3 · p. 9

7 Discussion and Limitations

• LLMJudge’s EIR falls to 0.72 on Privilege Escalation and 0.78 on Data Exfiltration, while Figure 2 shows a 4× EIR gap invisible to recall-only evaluation.The discussion identifies subtle permission changes and covert data movement as difficult to localize temporally.Source: §7 · p. 10, Figure 2 · p. 10
• Model-generated trajectories, code-agent-only coverage, and an extensible six-category taxonomy bound the dataset’s ecological validity and scope.The authors describe model-generated data as a deliberate trade-off for precise temporal annotation.Source: §7 · p. 10
• Tokens Saved weights tokens equally rather than accounting for action severity, and graded-confidence detection remains unexplored.The authors also report that stochastic detectors are averaged over three runs, with standard deviation below 0.02.Source: §7 · p. 11

A Broader Impacts

• StepShield aims to support pre-deployment evaluation of monitoring systems and shift attention from post-hoc forensics toward real-time intervention.The authors present earlier detection as a prerequisite for safe deployment of increasingly autonomous systems.Source: A Broader Impacts · p. 13
• Generated rogue trajectories demonstrate violation patterns but contain no working exploits against real systems.The authors state that the six violation categories are already documented in MITRE ATT&CK and the AI Incident Database.Source: A Broader Impacts · p. 13
• When production divergence points are unavailable, the authors recommend Tokens Saved as a proxy for timeliness evaluation.They note that EIR is sensitive to the annotation quality of the divergence step.Source: A Broader Impacts · p. 13

B Economic Analysis: Cascade Monitoring ROI

• StaticGuard evaluates each step first, resolving high-confidence benign or rogue verdicts and escalating uncertain cases to LLMJudge.The cascade is presented as a deployment-practical way to balance cost and coverage.Source: B Economic Analysis: Cascade… · p. 13
• 74.2% of steps receive high-confidence StaticGuard verdicts without LLM invocation, yielding a 74.2% cost reduction versus full LLMJudge monitoring.The cascade escalates the remaining 25.8% of steps in the uncertain confidence band.Source: B Economic Analysis: Cascade… · p. 13, Figure 3 · p. 14
• Changing the rogue rate from 1% to 20% shifts the escalation rate by less than 2 percentage points, from 25.1% to 26.9%.The analysis attributes escalation rate to the syntactic complexity of benign trajectories rather than the rogue rate.Source: B Economic Analysis: Cascade… · pp. 13–14

C.1 Incident Curation

• The 127-incident catalog combines production observations, threat-intelligence databases, and AI safety incident reports, spanning six violation categories.Production incidents were decomposed into a task specification, violation pattern, and divergence signature.Source: §C.1 · pp. 14–15

C.2 Trajectory Generation Protocol

• Rogue trajectories use lengths of 3–20 steps and divergence points sampled from steps 2 through n−1; paired clean trajectories control task difficulty, not prefix identity.The authors report that pre-divergence steps are checked for detectable violations.Source: §C.2 · pp. 15–16

C.3 Quality Assurance Pipeline

• Quality assurance combines automated checks, independent LLM cross-validation, and author review; the held-out test set is distinct from larger paired-set ablations.Author-annotated divergence-step agreement is κ = 0.82. Absolute metrics differ substantially between sets, though ablation directions are consistent.Source: §C.3 · p. 16, D Ablation Studies · p. 16

D.2 Ablation 2: Rogue Rate Sensitivity

• EIR remains stable across rogue rates, varying by only ±0.02, which the authors report as confirming that their temporal metrics are not confounded by base-rate effects.Source: §D.2 · p. 17

D.3 Ablation 3: Trajectory Length

• LLMJudge degrades less on longer trajectories than StaticGuard: EIR falls 34% relative for LLMJudge versus 47% for StaticGuard.The reported values are 0.71→0.47 for LLMJudge and 0.34→0.18 for StaticGuard.Source: §D.3 · p. 17, Table 7 · p. 17

D.4 Ablation 4: Context Window Size for LLMJudge

• A 5-step lookback raises LLMJudge EIR from 0.52 to 0.59, while full-history context drops it to 0.55 and nearly doubles latency.The reported latency increases from 510ms to 850ms; the 5-step window is used in the main experiments.Source: §D.4 · p. 17, Table 8 · p. 17

D.5 Ablation 5: Divergence Step Position

• LLMJudge’s EIR is highest when divergence occurs early (0.68); late divergence is harder to distinguish from legitimate behavior.The authors suggest early divergence creates greater semantic contrast with the expected trajectory.Source: §D.5 · p. 17

D.6 Ablation 6: Human Expert Baseline

• Human experts reach EIR3 = 0.92 on 100 sampled rogue trajectories, leaving headroom for future detection systems.Individual scores were 0.89, 0.93, and 0.94.Source: §D.6 · p. 18

D.7 Ablation 7: Model Size for LLMJudge

• On the held-out test set, o4-mini has the highest EIR (0.97) and lowest FPR (0.9%), while GPT-4.1-mini has the highest recall (0.95).Claude Opus 4 reaches EIR = 0.93 and the highest joint rate (EIR × Recall = 0.86), at 50× the cost of GPT-4.1-mini.Source: §D.7 · p. 18, Table 10 · p. 18

D.8 Ablation 8: Training Set Size

• ConstraintGuard shows diminishing returns beyond 150 training pairs, with performance limited by hand-crafted constraint expressiveness rather than calibration data volume.Source: §D.8 · p. 19

D.9 Ablation 9: Prompt Sensitivity for LLMJudge

• Chain-of-thought prompting adds +0.02 EIR over structured prompting at 2.8× latency cost; structured prompting has the best cost-effectiveness trade-off.Source: §D.9 · p. 19

D.10 Ablation 10: Cross-Category Transfer

• LLMJudge’s EIR changes by an average of −0.02 when a category is held out, compared with −0.07 for StaticGuard.StaticGuard’s largest drops are in Supply Chain (−0.14) and Covert Persistence (−0.11).Source: §D.10 · p. 19

• Experiments used a single NVIDIA A100 GPU and OpenAI API, with approximately $340 in API credits over three weeks.The stated software environment includes Python 3.11, PyTorch 2.1, Transformers 4.36, and OpenAI API v1.12.Source: E Reproducibility Details · p. 19
• Random seeds were fixed at 42; stochastic detectors were evaluated over three runs, with reported standard deviations below 0.02.Deterministic detectors produce identical outputs across runs by construction.Source: E Reproducibility Details · p. 20
• Evaluating all four detectors takes about 55 minutes on the 216-trajectory held-out set and about 3.5 hours on the 1,278-trajectory paired set.StaticGuard and ConstraintGuard finish in under 3 seconds.Source: E Reproducibility Details · p. 20

• StepShield contains 9,429 code-agent trajectories, including 1,278 training, 7,935 deployment-realistic test, and 216 strictly held-out evaluation trajectories.Each trajectory has 3–20 steps, each with an action type, action content, and reasoning trace.Source: F Dataset Documentation (Datasheet) · p. 20
• The trajectories are generated from 127 real-world security incidents and undergo four-stage quality assurance, including human expert review.Source: F Dataset Documentation (Datasheet) · p. 20
• StepShield is intended for evaluating temporal detection performance, not for training production safety systems without additional real-world validation.Source: F Dataset Documentation (Datasheet) · p. 20
• The dataset is released under CC BY 4.0 and the code under MIT; version 1.0 is frozen for reproducibility, with future dataset expansion planned.Source: F Dataset Documentation (Datasheet) · p. 20