PaperlayerView exact revision on arXiv

Source-linked AI summary

CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?

Zi Liang, Xiaoyu Xu, Yanyun Wang, Minxin Du, Qingqing Ye, Haibo Hu

arXiv:2608.27990v1cs.CRcs.AI

TL;DR

Prompt injections exploit untrusted content reaching autonomous LLM agents, while existing defenses trade off efficiency, precision, and adaptability. CAITLYN combines tiered runtime enforcement with counterexample-guided synthesis of verified defense skills, matching standard-benchmark detection performance and reducing attack success on emerging attacks.

  • Problem

    Existing agent defenses trade off runtime efficiency, contextual precision, and post-deployment adaptability, limiting their coverage of emerging injection techniques.

  • Method

    CAITLYN uses executable defense skills in a dual architecture combining System I runtime enforcement with System II counterexample-guided synthesis and verification.

  • Results

    CAITLYN matches state-of-the-art detection performance on standard benchmarks at lower token overhead than LLM-as-a-judge baselines, while System II substantially reduces attack success across three agent environments on Emerging.

  • Takeaways & Limitations

    Observed defense failures can be converted into verified, deployable skills for protection against unfamiliar injection techniques.

  • Takeaways & Limitations

    Cloud synchronization is disabled by default, and remote submissions require deterministic local verification, independent review, and human expert auditing before distribution.

Abstract

from arXiv · show

Prompt injection attacks on Large Language Model (LLM) agents seek to introduce malicious instructions or content into external text sources retrieved by agents, forcing the underlying LLMs to execute harmful actions outside their benign scope. While current defenses effectively counter known injection attacks, deploying them in LLM agent environments remains challenging due to attack variants and emerging threats. Moreover, existing solutions typically suffer from an inherent trilemma, i.e., a constant trade-off among runtime efficiency, contextual precision, and adaptability. To bridge this gap, we propose Continuous Agents for Injection Threats via Lifelong Yielding Nexus (CAITLYN), an agent-agnostic defense middleware. CAITLYN integrates two systems. System I focuses on immediate defense against existing attacks using a two-tiered library: Tier-0 for rule-based detection scripts and Tier-1 for optimized LLM-based accurate inference. System II, in contrast, is deployed to monitor potential abnormal signals and attempt to synthesize new defenses. On standard benchmarks, CAITLYN matches the detection performance of state-of-the-art defenses at lower token overhead than LLM-as-a-judge baselines. On Emerging, our new delivery-aware benchmark featuring novel injection techniques, static baselines and the standalone System I configuration remain vulnerable. In contrast, System II autonomously synthesizes verified defense capabilities, substantially lowering the attack success rate across three diverse agent environments.

1 Introduction

CAITLYN addresses prompt-injection risks in tool-augmented LLM agents with a dual-system middleware that combines efficient runtime enforcement and adaptive defense synthesis. It matches standard-benchmark detection performance while System II reduces attack success against emerging techniques.

  • Motivation: LLM agents’ broad access to untrusted external content creates security risks because successful prompt injections can affect production environments.These risks include malicious content from web results, files, APIs, and tool responses, with documented confidentiality and execution impacts.
  • Motivation: Existing defenses face a trilemma among runtime efficiency, contextual precision, and post-deployment adaptability.Rule-based filters are fast but vulnerable to obfuscation, heavyweight LLM defenses are costly and latency-intensive, and static controls cannot continuously absorb novel payloads.
  • Approach: CAITLYN treats security controls as executable, self-extending skill libraries and separates runtime enforcement from dynamic skill synthesis.System I applies existing defenses, while System II expands the library when novel attacks emerge.
  • Approach: System II uses an LLM generator and deterministic verifier to synthesize candidate skills against attack examples and benign functional constraints.Candidates that fail acceptance criteria are rejected, while verified candidates can be deployed with traceable lineage.
  • Evaluation: CAITLYN matches state-of-the-art detection performance on standard benchmarks at lower token overhead than LLM-as-a-judge baselines.The evaluation includes standard agent-security benchmarks and Emerging, a delivery-aware benchmark for post-deployment defense evolution.
  • Evaluation: On Emerging, System II autonomously synthesizes deployable defense skills that substantially reduce attack success across three agent environments.Static baselines and the initial System I configuration remain vulnerable to the novel injection techniques.

2 Preliminary

The paper frames LLM agents as autonomous systems exposed to indirect prompt injections through untrusted observations and tool interfaces. It defines emerging attacks as novel threats absent from the seeded defense knowledge and explains CAITLYN’s framework for evolving defenses.

  • LLM Agents and the Trust Boundary: LLM agents transform language models into goal-driven systems that perceive environments, plan multistep actions, and execute programmatic tool calls.Their operation coordinates reasoning, planning, and tool actuation components.
  • Injection and Jailbreaking Attacks: Prompt injection exploits the lack of a built-in separation between control instructions and data payloads in autoregressive transformers.Indirect injections can arrive through web pages, files, and API responses and hijack agent execution trajectories.
  • Defensive Methodologies: Defenses span input filtering, model hardening, and system-level isolation, but trade-offs persist among efficiency, precision, and adaptability.Static rules remain brittle against semantic evasion, while heavyweight evaluators and manual policies cannot dynamically absorb novel variations.
  • System and Trust: The threat model assumes a benign trusted user while treating external observations from web, file, API, and MCP channels as untrusted.The adversary may write arbitrary text into these sources and use obfuscation, indirection, multistage framing, or jailbreak patterns.
  • Adversary Objective: An adversary seeks confidentiality or integrity violations, including data exfiltration, unauthorized state changes, arbitrary command execution, or privileged tool use.The model distinguishes these security impacts from passive task failure.
  • Emerging Attacks: An emerging attack is absent from the seed corpus, baseline authoring and verification data, and seeded heuristic signatures.The open-world setting allows novel attack families to appear continuously after deployment.

3 Methodology

CAITLYN represents defenses as portable skills and organizes them into a dual-system architecture that combines fast runtime detection with adaptive synthesis. Its two-tier System I uses rule-based and LLM-based detectors, while System II develops defenses against emerging attacks.

  • 3.1 Skills as the Unit of Defense Knowledge: CAITLYN formalizes each defense as a self-contained, human-readable skill directory with specifications, metadata, and executable components.This representation supports heterogeneous defense methods while remaining portable for expert review and revision.
  • 3.1 Skills as the Unit of Defense Knowledge: Tier-0 skills pair YAML contracts with executable scripts, whereas Tier-1 skills store LLM detection logic as prompt templates under the same folder interface.Runtime dispatches according to the tier field, allowing synthesis to rewrite either contract without changing the interface.
  • 3.2 Hierarchical Organization of Defenses: The architecture assigns lightweight methods to frequent benign traffic while reserving more complex detection and synthesis for attacks requiring greater execution complexity.This organization responds to the cost and latency of applying complex detection methods to every input.
  • 3.2 Hierarchical Organization of Defenses: Tier-1 executes two parallel calls over the untrusted suffix: one evaluates detector skills and the other supplies reference knowledge against a shared prefix.Byte-identical prefixes enable provider-side caching, while concise verdicts reduce output-generation latency.
  • 3.2 Hierarchical Organization of Defenses: System I is the fast subsystem, and System II is the slower synthesis-driven subsystem that expands the skill library when novel attacks emerge.This dual-system design targets a trade-off among latency, effectiveness, and adaptability while minimizing disruption to benign workflows.
  • 3.3 Evolution of Defenses: Counterexample-Guided Skill Synthesis: System II uses counterexamples to guide candidate defense synthesis, then evaluates proposed scripts in an isolated verification sandbox and iteratively refines them.The generator receives attack statistics and similar samples, while the retrieved cluster is advisory rather than a hard verification constraint.

4 Evaluation

CAITLYN is evaluated against existing defenses through detection-only, end-to-end agent, ablation, and backbone-comparison experiments. The evaluation measures security, benign utility, latency, and cost across multiple agents and benchmarks.

  • Experimental Scope: The evaluation covers detection-only performance, realistic agent execution, component ablations, and backbone variation.Victim agents include coding and daily assistant agents, with comparisons against seven baselines across three injection and poisoning benchmarks.
  • Metrics and Protocol: Detection-only experiments report TPR, FPR, precision, recall, inspection latency, and financial cost.The protocol applies detectors directly to raw text using shared benign and attack pools, with threshold curves and latency-cost Pareto frontiers.
  • Detection-Only Evaluation: CAITLYN matches or exceeds leading detector recall while completing inspections in 2.1 to 2.5 seconds at lower per-sample cost than both LLM judges.It reaches 100.0% recall on AgentDojo, ties the top judge at 76.3% TPR on ASPI-S, and records 63.7% and 76.4% TPR on the two other reported datasets.
  • End-to-End Evaluation: End-to-end CAITLYN reduces ASR to near zero across evaluated datasets, including 0.0% on four of five AgentDojo agents and at most 2.2% on ASPI.SafeClawBench results range from 2.1% to 5.8% ASR; AgentDojo FPR is 0.0%, while ASPI FPR is 12.9% to 19.4% because benign controls mimic attack structures.
  • Ablation Study: The full two-call OR configuration reaches 100.0% TPR on AgentDojo-S250 and 89.2% on ASPI-S with 3.2% FPR.On the paired run, it used 5.17 s wall-clock latency and cost 0.00100 USD per inspection, while outperforming the ensemble on SafeClawBench.
  • Influence of LLM Backbones: Across nine backbone candidates, CAITLYN keeps ASR at or below 3.3%, while latency ranges from 8.9 to 17.1 seconds and per-case cost spans more than two orders of magnitude.Five models reach 0.0% ASR; GPT-5.6-Sol reaches 60.4% utility with 0.0% ASR, showing that security and utility can coexist across backbones.

5 Further Evaluation

Further evaluation shows that CAITLYN’s static defenses remain vulnerable to Emerging attacks, while System II’s verified synthesis improves protection across lifelong and adaptive settings. The evaluation also identifies assumptions and safeguards shaping those results.

  • Emerging contains indirect prompt injections across local files, search responses, and external content channels for evaluating zero-day adaptation.
  • CAITLYN-evolved reduces end-to-end attack success rate by about 40 percentage points relative to the static-defense range across the same agents.
  • Static defenses show attack success rates of 72.5%–80.0% across OpenClaw, Codex, and Hermes, with CAITLYN-static reaching 77.0%, 79.5%, and 77.5%.
  • Lifelong Synthesis Experiments: Sequential synthesis raises held-out TPR from 16.0% to 30.0% while preserving the 1.6% FPR across the nine-family stream.Four active skills accumulate across nine waves, and prior-family detection counts do not decrease.
  • Adaptive Attacks: An adaptive rewriter bypasses detection on 38 of 113 previously blocked cases, producing a 33.6% net evasion rate within five queries.Invariant target strings prevent evasion for 50 cases.
  • Adaptive Attacks: A single additional synthesis iteration restores detection for all 38 adaptive variants while increasing benign-pool FPR from 0.4% to 2.0%.
  • Adaptive Attacks: Library poisoning is constrained because cloud synchronization is disabled by default and remote artifacts require local verification, automated review, and human auditing.

6 Conclusion

The conclusion presents CAITLYN as agent-agnostic middleware addressing the trade-off among efficient execution, contextual precision, and post-deployment adaptability. Its experiments support lower-token standard-benchmark defense and adaptive protection against novel injections.

  • CAITLYN uses modular executable defense skills and a dual-system architecture combining low-latency inspection with autonomous lifelong synthesis.
  • CAITLYN matches existing defenses on standard-benchmark detection accuracy with lower token overhead than LLM-based baselines.
  • On Emerging, System II closes coverage gaps against novel injection vectors without a significant increase in false positives.

Appendix Outline

The appendix outline covers synthesis internals and the defense-library artifacts supporting CAITLYN’s implementation and evolution analysis.

  • Appendix 8 documents defense-library artifacts, including the skill inventory, synthesis lineage, and evolution run statistics.

Appendix Figures and Tables

The appendices provide detailed figures, tables, and engineering notes for Emerging evaluation, adaptive attacks, defense updates, and CAITLYN’s operational tooling.

  • Appendix 7: Appendix 7 includes figures for the merged-pair wrapper, Tier-1 prompt, and Tier-0 detector contract.
  • Appendix 9: Appendix 9 includes Emerging benchmark results, per-family and per-wave synthesis details, and adaptive evasion with defender-mediated re-vaccination.
  • Appendix 10: Appendix 10 presents engineering figures covering poisoning defenses, the CLI, library inventory, telemetry, and single-shot Tier-0 scans.

7 Synthesis Internals

CAITLYN’s synthesis internals define reproducible contracts for Tier-0 and Tier-1 defenses while constraining System II’s candidate-generation and verification process. Tier-1 uses merged-pair calls, whereas Tier-0 exposes a portable executable interface.

  • Evolution Configuration Defaults: System II balances over-specialization and over-generalization by separating generator information from strict validation constraints.The verifier tests attack triggers and benign samples, while the generator receives an abstract attack profile rather than full source data.
  • Tier-1 Execution: Tier-1 merged-pair execution uses detector-scope and knowledge-scope calls under a shared wrapper and single-line output contract.Detector skills are evaluated directly, while remaining skills provide reference knowledge; the user content appears as a delimited suffix.
  • Tier-1 Skills: Tier-1 instruction-hierarchy skills encode authority ordering and target impersonation, priority inversion, authority confusion, and chain-of-trust failures.The prompt is inserted verbatim into the merged wrapper to preserve an auditable defense contract.
  • Evolution Configuration Defaults: The synthesis search proposes up to three candidates per round, allows at most five CEGIS rounds, and stops at a forty-thousand-token budget.Acceptance requires every trigger sample to match and no more than one of five held-out benign samples to be flagged, with sandboxing and timeout checks.
  • Tier-0 Interface: Tier-0 detectors receive untrusted content through stdin or detect(content) and emit one JSON line.The default one-shot mode uses a sandboxed child process, while resident workers avoid repeated process startup costs.

8 Defense Library Artifacts

CAITLYN packages its defenses as a tiered, traceable skill library and retains only synthesized skills that survive verification and over-broad filtering. The resulting lineage documents provenance, not guaranteed generalization.

  • 8 Defense Library Artifacts: The shipped library contains 24 self-contained defense skills with YAML contracts, executable Tier-0 detectors, or Tier-1 prompt contracts.Each contract records identity, tier, role, security category, and a human-readable detection target.
  • 8 Defense Library Artifacts: System I preloads 14 Tier-0 skills at sub-millisecond cost, escalates to 8 Tier-1 skills, and uses 2 tier-2 entries only as knowledge context.Tier-2 entries do not produce independent verdicts.
  • 8.2 Evolution Lineage of Synthesized Skills: Of 24 skills materialized across 18 accepting responses from 77 evolution runs, only four survive the over-broad filter and remain active.These four end-to-end skills are distinct from the four skills produced in the sequential-stream study.
  • 8.2 Evolution Lineage of Synthesized Skills: Figure 12 distinguishes accepted synthesis edges, verifier-accepted but pruned nodes, and pass-filled nodes retained in the final library.All retained nodes run at Tier 0, so the evolved library adds no LLM cost on the hot path.
  • 8.2 Evolution Lineage of Synthesized Skills: The lineage demonstrates traceability rather than generalization because precise length-band signatures can be evaded through rewriting.The over-broad filter exists partly to remove signatures that are easy to audit but narrowly encoded.
  • 8.2 Evolution Lineage of Synthesized Skills: Most evolution runs terminate without a skill: 59 of 77 fail to accept one, while 18 accepting responses materialize 24 skills before filtering.The evolution log records rounds, generator tokens, termination reasons, and lessons written before acceptance or failure.

9 Extended Evaluation

Extended evaluation shows strong standard-benchmark efficiency and substantial but uneven protection against Emerging attacks. Sequential synthesis improves held-out coverage while preserving false-positive rate, but retained defenses remain vulnerable to uncovered families and adaptive rewriting.

  • 9 Extended Evaluation: CAITLYN reaches the highest or tied recall on every detection-only dataset at lower latency and cost than both LLM judges.TPR and FPR are percentages; latency is mean per-sample wall-clock time and cost is mean provider-reported USD.
  • 9.2 Emerging Benchmark Statistics and Full Results: 200 Emerging cases span nine attack families, and evolved defenses reduce ASR from the 72.5%-80.0% static range to 38.5% on OpenClaw and 39.5% on Codex and Hermes.Every case has a distinct attacker target, limiting gains from memorizing a single target string.
  • 9.2 Emerging Benchmark Statistics and Full Results: Coverage gains concentrate in Status Field and Policy Delta, which reach 0.0% ASR on all three agents, while Ledger Update reaches 100% ASR on Hermes after evolution.The retained end-to-end skills were synthesized from pipe-to-shell seed failures rather than the uncovered families.
  • 9.3 Lifelong Synthesis Wave Detail: Wave-level results show gains on small Ledger Update and Mail Followup clusters, while many larger families terminate without a surviving skill.Policy Delta accepts one skill but remains at 7.7% current-family held-out TPR, and Operational Command is skipped because the static library already blocks its misses.
  • 9.4 Adaptive Attack Protocol and Outcomes: A skill-aware rewriter escapes 38 of 63 non-target-collision cases within five queries, and re-vaccination detects all 38 while raising benign FPR from 0.4% to 2.0%.These are scan-oracle rates and exclude agent behavior.
  • 9.5 Case Traces on Emerging: OpenClaw traces show evolved defenses blocking some structured-record and search-result attacks, while freshness-framing remains uncovered and leaves the agent compromised.The uncovered case is consistent with the family-level coverage profile and would trigger another counterexample-guided round in a long-running deployment.

10 Engineering Notes

The engineering notes document repository security controls, evolved-defense outcomes, and the CLI/TUI interfaces used to inspect CAITLYN. They also identify repository tampering and curation as trust-boundary risks.

  • Repository Security: The trusted computing base includes the defense repository, whose tampering could disable detection or install attacker-chosen rules.
  • Emerging Evaluation: CAITLYN-evolved blocks the structured-record attack, whereas static defenses and CAITLYN-static are compromised.
  • Emerging Evaluation: Cases 1 and 2 are blocked by the evolved library, while static defenses and CAITLYN-static remain compromised.Case 3 remains compromised because freshness-framing attacks are not covered by the four retained skills.
  • Repository Security: System II controls repository updates by restricting writes, hiding raw trigger text from the generator, requiring deterministic verification, and requiring clean observation or approval for promotion.Cloud synchronization is off by default, and remote skills never automatically activate.
  • Interfaces: The reference implementation provides a CLI for one-shot scans, library inspection, and telemetry, plus a full-screen terminal interface sharing the same defense surface.

L2 Verification Sandbox

The verification and operations materials describe layered repository checks alongside CAITLYN's command surfaces, inventories, telemetry, scans, and reproducible terminal sessions.

  • L2 Verification Sandbox: Deterministic verification is the trust anchor, with signatures isolated in a child process, bounded by a timeout, and screened for static ReDoS risks.
  • L2 Verification Sandbox: Six poisoning defenses are checked in order from L1 to L6 on the local update path.
  • Operations: Append-only JSONL session logs make scans, vaccinations, and branches reproducible, while dashboard statistics expose anomaly signals before immune responses are raised.
  • Operations: The terminal interface reports antibody count, model, and defense status in its cold-start view, while its dashboard shows KPI cards, tier split, and top antibodies.
  • Operations: The terminal interface provides defense-facing commands alongside session and configuration commands.
Loading 2608.27990v1…