PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentDrift: A Step-Labeled Benchmark of Injection-Hijacked LLM Agent Trajectories

Asif Pinjari, Mithun Paul Saint-Germain

arXiv:2609.06972v1cs.CRcs.AIcs.LG

TL;DR

Existing resources measure injection success or judge traces as wholes, leaving no public corpus that labels where an injection enters and which steps it corrupts. AgentDrift constructs such a benchmark with structured trajectories, contrast classes, and validation, finding that shallow features recover only 55.4% of attacks and perform especially poorly on partial and delayed compromises. The paper also documents distributional limits, including single-generator data and collapsed attack-goal-family tags.

  • Problem

    Existing benchmarks measure live-agent attack success and guard datasets judge whole traces, but public step-level labels for injection entry and corrupted behavior are absent.

  • Method

    AgentDrift releases 12,536 synthetic trajectories across five domains with four step labels, three compliance patterns, contrast classes, structural validation, screening, and manual audit.

  • Results

    The surface baseline recovers 55.4% of attacks overall, including 8.2% of partial hijacks and 23.1% of delayed executions.

  • Takeaways & Limitations

    The benchmark concentrates difficulty in sequence-level behavioral drift and requires distinguishing successful compromise from resisted injection and suspicious legitimate content.

  • Takeaways & Limitations

    All 12,536 trajectories come from one model, and attack-goal-family tags are stratification metadata rather than a reliable classification target.

Abstract

from arXiv · show

LLM agents complete tasks by issuing sequences of tool calls, and every observation they read is a channel through which an indirect prompt injection can enter. A successful injection has a characteristic shape when the trajectory is read in order: a benign prefix gives way to actions that serve the attacker rather than the user. Existing benchmarks measure whether such attacks succeed against live agents, and existing guard models judge a trace as a whole; no public corpus labels, step by step, where an injection enters a trajectory and which steps it corrupts. We present AgentDrift, a benchmark of 12,536 synthetic tool-call trajectories over five agent domains in which every one of the 71,024 steps carries one of four labels: benign, injection point, hijacked, or failed injection. The corpus contains 4,000 benign, 5,536 attacked, 1,500 failed-attack, and 1,500 hard-negative trajectories; attacked trajectories follow three compliance patterns whose label strings obey a stated regular grammar. Failed attacks carry an injection the agent resisted, and hard negatives carry legitimate content that resembles an attack, so a detector must separate attempt from success and deviation from novelty. Trajectories were generated by a single open model under category-specific protocols, enforced by a closed-vocabulary structural validator, screened by an LLM judge, and audited by hand on 1,200 trajectories; we show that the LLM judge was itself fooled by the hard negatives. A surface-feature logistic regression recovers only 55.4% of attacks (F1 0.647), including only 8.2% of partial hijacks and 23.1% of delayed executions, so nearly half of the attacks require modeling the behavioral sequence. We measure template concentration, attack-goal-family collapse, and world-identity leakage in the generated data, and release the corpus with its documentation under CC BY 4.0.

I. INTRODUCTION

AgentDrift addresses the lack of public, step-labeled injection trajectories by releasing a synthetic benchmark with structured labels, contrast classes, and validation. Its baseline results show that shallow surface features miss many attacks, especially partial and delayed compromises.

  • Research Gap: AgentDrift fills a gap left by live-agent attack benchmarks and whole-trace guard datasets, which do not label injection entry and corrupted steps.Existing resources measure attack success or trace-level safety rather than step-level behavioral drift.
  • Contrast Classes: Failed attacks and hard negatives force detectors to distinguish resisted injections from successful compromise and suspicious-looking legitimate behavior.Failed attacks contain injected content without hijacked behavior, while hard negatives remain task-consistent.
  • Benchmark and Contributions: 12,536 trajectories across five domains provide 71,024 step labels for benign, injection-point, hijacked, and failed-injection behavior.The corpus includes four categories and three compliance patterns.
  • Construction and Quality Assurance: The construction pipeline combines generation-time protocols, a closed-vocabulary validator, LLM screening, and a 1,200-trajectory manual audit.The authors report verification failures rather than treating the LLM screening pass as definitive.
  • Shortcut and Diversity Analysis: The paper measures template concentration, attack-goal-family collapse, world-identity leakage, and surface-cue prevalence to assess generated-data shortcuts.Evaluation protocols are designed to guard against memorization.

B. Defenses and Guard Models

AgentDrift positions itself between live-agent attack benchmarks, guard models, and step-level trace resources by making prompt-injection drift observable across ordered tool calls. Its defining scope is a released corpus with injection-specific step labels, resisted attacks, and hard negatives.

  • Defenses and Guard Models: Existing defenses inspect content, model state, data flow, or actions, while guard datasets commonly judge an entire trajectory.Examples include input classifiers, dependency constraints, re-execution, and modular guardrails.
  • Behavioral Drift: AgentDrift transfers task-drift detection from internal-state deviation to behavioral drift across ordered tool calls.The benchmark labels whether injected content is followed by attacker-serving behavior.
  • Positioning Against Prior Resources: Prior step-level resources address procedural failures, rogue behavior, mutation points, or limited trace labels rather than the full prompt-injection span.Some resources lack resisted-injection classes or fold resisted cases into safe labels.
  • Benchmark Scope: The benchmark’s five defining properties are a released static corpus, per-step labels, injection-specific labels, a resisted class, and adversarial hard negatives.The comparison is scoped to this intersection of properties.
  • Threat Representation: Each trajectory records a user instruction, world, ordered thoughts, tools, arguments, and observations, making the observation channel the attack surface.At each step, the next action is conditioned on the returned observation.

B. Threat Model

The threat model treats indirect injection as attacker-controlled content entering an observation and changing subsequent agent behavior. AgentDrift formalizes this drift with four step labels, regular label grammars, and detection, localization, and attempt-versus-success tasks.

  • Threat Model: The attacker controls an observation’s content but not the user instruction, model weights, or tool implementations.The defender sees the trajectory and world but cannot identify attacker-controlled content by provenance.
  • Step Labels: B marks user-serving steps, I marks an injection acted upon, H marks attacker-serving actions, and F marks an injection refused by the agent.These labels distinguish successful compromise from delivered but resisted injections.
  • Label Grammar: Each category admits only label strings from a regular language, with at most one I or F symbol per trajectory.The grammar makes the injection index well defined for attacked and failed-attack trajectories.
  • Trajectory-Level Target: A trajectory is a successful compromise if and only if at least one step is labeled H.The released data verify that the number of I steps equals the number of attacked trajectories.
  • Supported Tasks: The benchmark supports detection, localization of the injection index and corrupted span, and discrimination between failed attacks and successful attacks.Localization includes span overlap, while attempt-versus-success evaluation reports failed-attack flag rates with attacked-class recall.

E. World Grounding

World grounding supplies the context needed to distinguish legitimate from attacker-directed actions in AgentDrift. Closed domain-specific tool pools, explicit worlds, and contrastive categories prevent simple novelty cues from defining the labels.

  • World Grounding: World-derived recipient and URL indicators distinguish actions involving known contacts or domains from external destinations.The release provides the world and computation for these derived indicators.
  • Record Structure: The corpus record includes trajectory metadata and five fields per step: thought, tool, args, obs, and label.Attacked and failed-attack records additionally include attack-goal family and compliance-pattern metadata.
  • Closed Tool Vocabulary: Each trajectory uses a closed domain-specific tool pool, totaling 103 tools, and the validator rejects tools outside the assigned pool.Every tool appears in the released data.
  • Trajectory Categories: Benign trajectories contain normal task completion with every step labeled B, whereas attacked trajectories contain an observation followed by attacker-serving action.The categories are defined against the user’s task and world context.
  • Failed Attacks: Failed attacks contain injected text but require explicit resistance and completion of the original task, preventing detectors from relying only on injection presence.The resistance rationale averages 54 words and states why the instruction was rejected.
  • Hard Negatives: Hard negatives are legitimate, task-consistent trajectories whose suspicious-looking content remains in-domain and receives all-B labels.Examples include audit forwarding, invoice payment, and password rotation requests.

D. Compliance Patterns

AgentDrift represents attacked trajectories through three positional compliance patterns, each designed to expose a different temporal detection challenge. The benchmark also distinguishes six attack-goal families and includes hard negatives whose lexical cues resemble attacks without matching their behavior.

  • Compliance patterns: 3,136 full-hijack trajectories place one to three benign steps before the injection and label every subsequent step hijacked.This is the simplest pattern because the compromised tail is uniform.
  • Compliance patterns: 1,500 partial-hijack trajectories contain one or two hijacked steps before recovery steps labeled B, whose thoughts must identify the deviation.The pattern forces detectors to close the hijacked span rather than extend it to the trajectory’s end.
  • Compliance patterns: 900 delayed-execution trajectories continue the original task after the poison and isolate a single anomalous action between otherwise unremarkable steps.This pattern targets detectors that rely on locally obvious or uniformly compromised tails.
  • Attack-goal families: Attacked and failed-attack trajectories are conditioned on six attack-goal families, including data stealing, direct harm, parameter manipulation, branch divergence, reasoning corruption, and multi-step spreading.The family metadata supports transparency and stratification but is not certified as a classification target because generator boundaries were not always respected.
  • Detection challenge: Hard negatives use legitimate content with overlapping lexical cues, requiring detectors to distinguish behavioral compromise from suspicious wording.The benchmark explicitly contrasts a hard-negative trajectory with an attacked trajectory whose behavior differs despite similar surface language.

F. Two Contrasting Examples

The construction pipeline defines labels during generation, enforces their structural grammar before acceptance, and applies later quality-assurance checks. Its validator constrains tools, fields, and label sequences, while semantic truth and resistance quality remain outside structural validation.

  • Contrasting examples: The worked banking example shows a routine fee lookup carrying a fake directive that redirects a legal banking action toward an external beneficiary absent from the world’s contacts.This illustrates how an injection can enter through a tool observation and produce an illegitimate target without requiring an invalid tool call.
  • Structural validation: Every candidate must contain at least two steps, complete thought/tool/observation/label fields, and use only tools from its domain’s closed pool.Missing args are replaced with an empty object, while out-of-domain tools cause rejection.
  • Structural validation: Benign and hard-negative candidates require all-B labels; attacked candidates require an I preceded only by B and followed by H; failed attacks require F and no H.These category grammars encode the intended distinction between successful compromise, legitimate similarity, and resisted injection.
  • Construction pipeline: Labels are specified by category protocols, enforced by a structural validator, and then checked through three quality-assurance layers before merging and splitting.The pipeline was orchestrated as 30 SLURM array jobs spanning five domains and six source categories.
  • Validation boundary: The validator checks internal consistency rather than semantic truth, so quality-assurance layers must assess whether poisons are genuinely dangerous and actions fit the intended scenario.The protocol and structural checks do not establish that a threat is real or that a resistance rationale is sound.

F. Yield and Rejection Analysis

Yield analysis shows that structural validation accepted most candidates while exposing a domain-specific banking bottleneck, and the released corpus preserves documented provenance and split boundaries. Quality assurance combines exhaustive programmatic checks, unreliable LLM screening, and a high-accuracy but non-exhaustive manual audit.

  • Yield and rejection: 12,536 of 15,900 candidates were accepted, an overall acceptance rate of 78.8%; 2,838 rejections were out-of-vocabulary tools, 513 were unparseable JSON, and 13 lacked fields.No candidate was rejected for label-grammar violations because the protocols stated the structure explicitly and the model followed it.
  • Yield and rejection: Outside banking full-hijack, acceptance reached 91.1% (12,280 of 13,484), while that cell accepted only 256 candidates and rejected 2,160.The banking anomaly arose because the preferred forwarding-based exfiltration action was absent from the domain’s legal tool pool.
  • Merge and split: The 8,775/1,880/1,881 train-validation-test split preserves category and domain proportions but shares tasks and almost all worlds across splits.Consequently, evaluation measures generalization across trajectories rather than across tasks or worlds.
  • Quality assurance: Programmatic checks verified exact count identities and category-specific label grammars across the released corpus.They confirmed 5,536 I steps for 5,536 attacked trajectories and 1,500 F steps for 1,500 failed-attack trajectories.
  • Quality assurance: The LLM screening pass marked 1,494 of 1,500 hard negatives BAD because their legitimate observations looked suspicious.This demonstrates that the judge measured surface suspiciousness rather than reliably verifying injection labels.
  • Quality assurance: Manual auditing found 99.6% label correctness on 1,200 trajectories, while residual errors mainly involved hijacked actions only loosely connected to the injected instruction.These errors were semantic and therefore outside what the label grammar could catch.

D. Legacy Tags

The corpus documents legacy family tags and generator regularities that affect how its released data should be interpreted and evaluated. These include tag anomalies, concentrated templates, world-identity leakage, and tool-level injection patterns.

  • Legacy Tags: 17 attacked trajectories retain generator-emitted family tags outside the six-family list: 16 data_theft and 1 data_deletion.All 17 were structurally valid; Table VII consolidates data_theft into data_stealing and lists data_deletion separately.
  • Corpus Statistics: 12,536 trajectories comprise 71,024 labeled steps, with trajectory lengths ranging from 3 to 11 steps and a mean of 5.67.The corpus includes 4,000 benign, 5,536 attacked, 1,500 failed-attack, and 1,500 hard-negative trajectories.
  • Corpus Statistics: Injections occur early: full-hijack attacks place them at step 3 in 2,850 of 3,136 trajectories, while other patterns place nearly all injections at step 2.The paper identifies this positional regularity as a detector-exploitable limitation.
  • Behavioral Statistics: Injection carriers are passive reads, while frequent hijacked sinks are active state-changing tools such as execute_command, forward_email, and fill_form.This creates a read-to-write behavioral signature for the benchmark.
  • Lexical Diversity: The released corpus contains concentrated lexical motifs, including ext-audit.com in 28.9% of injected observations and recurring hard-negative amounts, audit addresses, and transfers.These concentrations arise from protocol examples and generator tendencies, motivating masking and diversification.

B. Semantic Collapse of the Attack-Goal Families

Attack-goal families collapse toward shared external-forwarding behavior, limiting their use as certified classification targets. Evaluation must also control world identity and surface cues because these can reveal categories without modeling behavior.

  • Semantic Collapse: The generator realized multiple attack-goal families through a common external-recipient behavior, despite different generation conditions.External-recipient arguments occurred in 0.50–0.54 of several non-exfiltration families, compared with 0.57 for data stealing.
  • World-Identity Leakage: 86.1% binary accuracy came from a world-identity lookup predictor, versus a 55.8% majority-class baseline.Accuracy reached 93.3%–98.2% in banking, web, coding, and medical, but fell to 47.7% in email.
  • World-Identity Leakage: World anonymization and world-held-out evaluation are recommended controls for preventing identity-based category prediction.Anonymization preserves world relations while replacing names, companies, and addresses.
  • Surface Cues: 97.3% of hard negatives and 94.7% of failed attacks contain suspicious keywords, compared with 71.0% of attacked trajectories and 23.9% of benign ones.A keyword detector would therefore flag many negative examples it should pass.
  • Surface Cues: 20.4% of attacked trajectories contain none of the external-recipient, external-URL, or suspicious-keyword cues.The paper uses this prevalence to motivate sequence-level behavioral analysis.

B. Results

A surface-feature logistic-regression baseline detects attacks imperfectly, especially when hijacking is partial or delayed, while also confusing resisted injections with successful compromise. These results support sequence-aware detection and localization.

  • Overall Detection: The baseline achieves attacked-class precision 0.778, recall 0.554, and F1 0.647 on the test split.Its ROC AUC is 0.734 and precision–recall AUC is 0.754.
  • What the Baseline Establishes: 55.4% of attacks are recovered, leaving 44.6% without a surface tell visible to the baseline features.The result is consistent with the prevalence of attacks lacking external-recipient, URL, or keyword cues.
  • What the Baseline Establishes: Benign and hard-negative false-positive rates are 0.105 and 0.116, respectively, so the baseline is not simply flagging indiscriminately.Its main weakness is missing attacks rather than broadly misclassifying all trajectories.
  • Compliance Patterns: Pattern recall is 88.0% for full hijacks, 8.2% for partial hijacks, and 23.1% for delayed executions.Partial and delayed patterns contain short compromised spans embedded among benign steps, which bag-of-features models average away.
  • Attempt Versus Success: Failed attacks are flagged at 0.187, nearly twice the benign rate, because poisoned text remains even when behavioral drift does not occur.This separates detecting an injection attempt from determining whether the agent followed it.

B. Format, Access, and License

AgentDrift is distributed as a documented synthetic corpus with per-step labels, structural validation, and supporting metadata. Its main scope boundaries include single-generator data, synthetic-only traces, concentrated templates, leakage risks, and limited trajectory diversity.

  • B. Format, Access, and License: The corpus is released as JSON records with train, validation, and test splits, plus statistics, tool and task pools, datasheet, and Croissant metadata.The repository includes plain-Python loading support and is distributed under CC BY 4.0.
  • C. Ethical Considerations: Attack text uses fictitious targets and worlds, while generation prompts and scripts are withheld to reduce reuse as an attack generator.The authors state that the corpus contains no executable attacks against real systems.
  • XI. LIMITATIONS: A single-generator distribution may encode stylistic fingerprints, so results should be interpreted as performance on this generator’s distribution.Attacked and nonattacked examples share the generator, worlds, and tool pools, but this mitigation does not eliminate the risk.
  • XI. LIMITATIONS: One target domain accounts for 28.9% of injections, while three motifs account for three-quarters of hard negatives.The authors identify these patterns as memorizable and describe masking as a workaround rather than a fix.
  • XI. LIMITATIONS: Labels are protocol-specified, structurally validated, LLM-screened, and audited on 1,200 trajectories, but they are not exhaustive independent human gold.The reported 99.6% correctness is an estimate from the audited sample, and the LLM screening pass was not a reliable verifier.
  • XI. LIMITATIONS: Poisoned observations usually occur at step 2 or 3, and fixed compliance templates create a genuine positional regularity.Hard negatives and failed attacks reduce, but do not remove, the opportunity to exploit positional priors.
  • XI. LIMITATIONS: World identity predicts the attacked class with 93.3% to 98.2% accuracy in four domains, requiring anonymization or world-held-out evaluation for detection claims.Nearly all of the 250 task templates are shared across categories and splits, while trajectories also cover only 3 to 11 steps and one agent.
  • XI. LIMITATIONS: Synthetic-only data makes exhaustive labeling possible, but transfer to production agent traces remains untested.AgentDrift contains no real agent traffic.

XII. FUTURE WORK

Future work targets transfer, diversity, and stronger evaluation protocols. The proposed extensions also broaden threat surfaces and make attack families, positions, and human agreement more rigorously testable.

  • Data construction: A shared world pool and official world-held-out protocol would remove category-linked world identity from evaluation.The proposal also calls for anonymized rendering.
  • Data construction: Several generators and randomized target pools would reduce memorization from concentrated domains, amounts, identifiers, and hard-negative motifs.The proposal aims to prevent any single string from accounting for a large fraction of injections.
  • Data construction: External-recipient hard negatives would prevent recipient novelty alone from separating legitimate requests from attacks.These examples would require detectors to read the user’s task.
  • Data construction: Uniformly sampled injection positions and longer trajectories would remove the current early-position prior and expose late, deep injections.The current corpus’s positional regularity is explicitly identified as a limitation.
  • Evaluation: A task-held-out protocol would test generalization when no task template is shared between training and test.This directly addresses the current default split’s seen-task setting.
  • Evaluation: Multi-annotator labeling of a fixed evaluation subset would replace the sample audit with agreement statistics for injection indices and hijacked spans.The proposal follows agreement reporting used by related step-level resources.
  • Scope expansion: Future corpus versions should certify attack-goal families, broaden observation channels, and include diversions that do not complete the attacker’s goal.The proposed channels include tool descriptions, MCP metadata, memory, and inter-agent messages.
  • Transfer and evaluation: The next release should pair the synthetic corpus with sandboxed live-agent traces to measure transfer.The proposed traces would use the same labeling grammar.

APPENDIX A DATASHEET

The datasheet records AgentDrift’s motivation, synthetic composition, generation and labeling process, intended uses, access conditions, and maintenance plan. It positions the corpus as a self-contained resource for training and evaluating injection-localization detectors.

  • Motivation: The dataset was created for supervised training and evaluation of detectors that localize indirect prompt injection inside tool-call trajectories.The datasheet states that no public corpus with injection-specific step labels existed for this task.
  • Composition: AgentDrift contains 12,536 synthetic trajectories and 71,024 labeled steps across four categories and five domains, with every record self-contained and fully populated.The recommended split is a stratified 70/15/15 train-validation-test split.
  • Collection process: All content was generated by Llama-3.3-70B-Instruct between April and June 2026 without crowdworkers, contractors, or human subjects.Generation followed the protocols described in the paper.
  • Preprocessing, cleaning, and labeling: Labels were produced during generation under category protocols and enforced by a validator; rejected candidates were discarded and fields were normalized at merge time.No content was edited by hand.
  • Uses: The dataset supports the paper’s surface baseline but should not be used to assess particular deployed agents, train attack generators, or classify attack-goal families.These restrictions define the intended use of the released corpus.
  • Distribution: The corpus is distributed through the named GitHub repository under CC BY 4.0 with Croissant metadata and no fees, export controls, or third-party restrictions.The release is self-contained and does not require external resources.
  • Maintenance: The first author maintains the repository, records errata in its changelog, and releases corrected records under new version strings while preserving older versions.Contributions of new domains, generators, and annotations are invited through the issue tracker.
  • Generation protocol: Each generation prompt combines a shared schema preamble, a four-label inventory, required observation details, and a category-specific structural rule.The category rule is expressed using the label grammar of Eqs. (3)–(7).
Loading 2609.06972v1…