TRACER: Trace-Regularized Training Against Indirect Prompt Injection

1 Introduction

• The introduction frames indirect prompt injection as malicious instructions hidden in external content and identifies cross-channel robustness transfer as an under-tested assumption.Existing defenses are usually trained and evaluated within a single wrapper.Source: §1 · p. 1
• 18% to 53%: rewrapping the same attacks from follow-up messages as retrieved documents raises average ASR, a channel effect the paper calls the framing gap.The base model also shows an authority gradient across follow-up, document, tool-output, and agent-message framings.Source: §1 · p. 1
• TRACER treats the untrusted-context channel as part of adversarial training, combining an answer-anchored, evidence-gated reward with ATRC trace alignment when injected evidence is invalid.The framework uses wrapper-sampled channel exposure to preserve clean reasoning under invalid injected evidence.Source: §1 · pp. 1–2
• Multi-framing training gives the best 3B average performance, while 7B models substantially reduce the depth–breadth trade-off in cross-framing transfer.The evaluation compares native robustness with transfer across wrappers.Source: §1 · p. 2

Indirect prompt injection and benchmarks

• Injection benchmarks broaden the attack surface from single applications to agents, dynamic environments, and web-native settings; this work instead isolates wrapper effects across those settings.The cited benchmarks include BIPIA, InjecAgent, AgentDojo, ASB, WASP, and Hidden-in-Plain-Text (Zhan et al., 2024; Debenedetti et al., 2024; Zhang et al., 2025; Evtimov et al., 2025; Guo and Wei, 2026).Source: §2 · p. 2

RAG robustness, poisoning, and attack design

• RAG robustness studies noisy, conflicting, or poisoned retrieval and realistic injections, while this work varies presentation channel with payload fixed.The cited lines name ConflictQA, RGB, PoisonedRAG, and TopicAttack (Xie et al., 2024; Chen et al., 2024; Zou et al., 2025; Chen et al., 2025c).Source: §2 · p. 2

Defenses

• Existing defenses harden fixed pipelines or add inference-time modules; this work instead trains the base model to maintain post-attack behavior across untrusted channels.The cited approaches include structured instruction tuning, data marking, alignment training, detection, parsing, and correction defenses (Chen et al., 2025a; Hines et al., 2024; Chen et al., 2025b; Zhu et al., 2025; Debenedetti et al., 2026; Wen et al., 2025; Chen et al., 2026; Shi et al., 2025; Yu et al., 2026; Wang et al., 2026; He et al., 2026).Source: §2 · pp. 2–3

Reinforcement learning for reasoning

• Unlike reasoning-training work focused on clean inputs, TRACER uses clean reasoning traces as anchors when injected evidence is invalid.The passage situates this contrast alongside RL with verifiable rewards, process rewards, exploration, and multi-model reasoning (Shao et al., 2024; Guo et al., 2025; Yu et al., 2025; Lightman et al., 2024; Wang et al., 2024; Cui et al., 2025; Cheng et al., 2026; Song et al., 2025; Walder and Karkhanis, 2025; Mishra et al., 2026b; Du et al., 2024; Subramaniam et al., 2025; Park et al., 2025; Wan et al., 2025; Mishra et al., 2026a).Source: §2 · p. 3

How prompt-injection defenses should be evaluated

• Prompt-injection defense evaluation should jointly assess security and utility and include adaptive attacks; this work emphasizes cross-framing transfer, final checkpoints, and conditional ASR.The cited evaluation agenda is attributed to Jia et al. (2025) and Zhan et al. (2025).Source: §2 · p. 3

Three-turn robustness game

• TRACER samples the presentation channel during a three-turn episode that generates clean and post-attack traces for the same task and payload.This supports both single-channel specialists and joint RAG+tool training.Source: §3 · pp. 3–4

Composite post-attack reward

• The post-attack composite reward combines answer correctness, process quality, anchor retention, evidence-gated corrigibility, and length control.It rewards preserving a correct answer under invalid evidence while allowing correction when evidence is valid, rather than encouraging blanket refusal.Source: §3 · p. 4

ATRC: Adversarial Trace Recovery and Consistency

• ATRC regularizes committed clean reasoning spans when the clean answer is correct and injected evidence is invalid.It aligns clean and post-attack responses by relative position, using a KL term and token recovery to retain the clean distribution and committed tokens.Source: §3 · pp. 4–5

Overall training objective

• TRACER optimizes the post-attack GRPO objective with a channel-sampled reward and ATRC regularization.The method’s stated novelty is training over presentation channels with structured reward and response-aligned trace loss.Source: §3 · p. 5, Figure 2 · p. 5

Training setup

• The main training task is GSM8K, with 2,000 training examples and 200 evaluation examples per checkpoint; experiments primarily use Qwen2.5-3B-Instruct and also include a 7B scaling study.Evaluation wrappers are follow-up, RAG, tool-output, and agent-message.Source: §4 · pp. 5–6

Attacks

• Three attack families target reasoning, final answers, or instruction following, with training and evaluation tasks disjoint.Training includes a 15% benign ratio; additional tests use held-out paraphrases and defense-aware attacks.Source: §4 · p. 6

Training protocol

• The 3B protocol trains RAG, tool-output, and agent-message specialists plus a joint model alternating RAG and tool framings.It uses three epochs, learning rate 2 × 10^-6, batch size 1, gradient accumulation 8, group size 4, KL coefficient 0.10, and ATRC weight 0.10.Source: §4 · p. 6

Evaluation protocol

• Transfer tests hold the adversarial payload fixed and change only its wrapper, separating attack-content learning from channel learning.Cross-framing comparisons use final checkpoints for each specialist; the paper defines channel, wrapper, framing gap, and the boundary between training-only verifiers and reported metrics.Source: §4 · p. 6

Framing gap

• 18% to 53%: rewrapping identical follow-up-trained attacks as retrieved documents raises average ASR across six checkpoints by 35 percentage points.The base model also shows a framing gradient, with prompt-injection ASR rising from 70% under casual follow-ups to roughly 92–96% under agent, tool, and RAG wrappers.Source: §5 · p. 6, Figure 3 · p. 7

RAG-framed training

• 65.5% to 6.1%: TRACER reduces held-out RAG ASR for the 3B final checkpoint, while clean accuracy remains in the 78–82% band.The best intermediate checkpoint reaches 3.8% ASR; the sharpest gains occur between the late first and early second epoch.Source: §5 · p. 7, Figure 4 · p. 8

Per-attack effects

• 77.5% to 2.3%: prompt-injection ASR falls with RAG-framed training, while preemptive-answer ASR falls from 54.5% to 3.4%.ASR is conditional on solving the example in the clean condition.Source: §5 · p. 7, Figure 5 · p. 8

Objective ablations and mixed evidence

• 21.3% to 13.8% to 6.1%: under RAG evaluation, answer-only training, the composite reward, and ATRC successively lower ASR.The mixed-evidence evaluation reports clean/evidence accuracy of 81.0–83.0%, while injection-induced flip rates fall from 46.3–58.1% for the base model to 13.8–17.5% with TRACER.Source: §5 · p. 7, Table 1 · p. 8

Cross-framing transfer

• At 3B scale, the RAG specialist reaches 6.1% native RAG ASR but 25.7% and 26.5% on tool and agent framings; tool training is flatter at 15.0–20.2% across framings.Joint RAG+tool training achieves the best 3B average ASR at 13.9% and 5.2% native RAG ASR; Figure 6 summarizes the cross-framing profiles.Source: §5 · p. 8, Figure 6 · p. 9

Wrapper and system prompt

• 5.5% ASR with both the RAG wrapper and system prompt contrasts with 10.0% for wrapper-only, 11.3% for prompt-only, and 60.5% for neither.The results support the view that deployment-matched robustness depends on both structural cues and surrounding instruction context.Source: §5 · pp. 8–9

Scaling

• At 7B scale, one epoch of RAG training reaches 9.4% average ASR across four framings, outperforming the best 3B system; the final 7B model reaches 10.8%.The authors report that most of the gain is realized early and that early stopping may suit transfer-focused targets.Source: §5 · p. 9

Transfer beyond GSM8K

• On RAG-framed Natural Questions and TriviaQA, framing-aware models retain relatively low ASR, but 3B clean accuracy remains low.The results indicate that robustness transfers more readily than task-specific competence.Source: §5 · p. 9

6 Conclusion

• Rewrapping identical attacks as retrieved context raises average ASR from 18% to 53% for models trained against follow-up-style attacks.TRACER addresses this channel-conditioned robustness gap with channel-aware adversarial training.Source: §6 · p. 9
• On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while maintaining 78–82% clean accuracy.Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale.Source: §6 · p. 9
• The results support training and evaluating defenses across deployment-matched framings and pairing training with runtime safeguards.Held-out, adaptive, and mixed-evidence evaluations preserve substantial gains while exposing a residual attack surface.Source: §6 · p. 9

Limitations

• Primary attacks are template-based and hold the adversarial payload fixed while varying the channel.Held-out paraphrases and a targeted defense-aware suite broaden testing but do not cover arbitrary optimization-based or multiturn adversaries.Source: Limitations · p. 9
• Full TRACER training is evaluated within Qwen2.5 at 3B and 7B scale, while the framing-gap diagnostic spans several model families.Broader architectures, tasks, seeds, and adaptive strategies remain important next steps.Source: Limitations · p. 9
• TRACER is a training-time defense intended to complement inference-time safeguards, not replace them.The paper names retrieval filtering, tool-output parsing, policy monitors, and sandboxing as examples.Source: Limitations · p. 9

Ethical Considerations

• Experiments use controlled task settings and synthetic template-based injections rather than targeting real users, private data, or production services.The stated purpose of attack construction is to measure and improve robustness across presentation channels.Source: Ethical Considerations · pp. 9–10
• The method is not a complete security boundary and should be combined with runtime safeguards; overly aggressive defenses can harm usefulness through unwarranted refusal or distrust of valid evidence.The reward design separates resistance to invalid evidence from correction using valid evidence.Source: Ethical Considerations · p. 10

A.1 Attack construction and split

• Task instances and held-out surface forms are disjoint, while attack families are shared to isolate channel transfer.Held-out paraphrases and a targeted defense-aware suite broaden evaluation, but the original result supports controlled robustness to known or family-matched attacks, not arbitrary attacks.Source: §A.1 · pp. 12–13

A.2 Training signals and verifier audit

• Evidence validity is assigned during episode generation, while step and anchor verifiers affect reward computation; normalized final-answer comparison alone determines reported accuracy and ASR.The anchor verifier checks whether clean-trace propositions remain supported or are contradicted after attack.Source: §A.2 · p. 13
• 21.3% RAG ASR with answer-only channel exposure falls to 13.8% with the composite reward and 6.1% with ATRC.The ablation separates the improvement associated with each objective component.Source: §A.2 · p. 13

A.3 Statistical uncertainty and denominators

• 16/160 (10.0%) and 38/160 (23.8%) are the exact outcomes for held-out and defense-aware TRACER evaluations, respectively.ASR is conditioned on clean correctness, and small pairwise differences among structured wrappers should not be treated as a strict ordering.Source: §A.3 · p. 13

A.4 Mixed valid evidence and embedded injection

• In mixed-evidence tests, TRACER retains high clean/evidence accuracy while substantially reducing contamination-induced flips.The test places task-relevant evidence and a malicious instruction in the same channel; evidence-dependent tasks use a clean-evidence baseline.Source: §A.4 · p. 13

A.5 Broader-model framing diagnostics

• Every tested model family shows a 22.0–27.1 pp spread across renderings, supporting a cross-model framing-gap diagnosis.The training-time mitigation is established on Qwen2.5-3B/7B; full crossarchitecture training remains future work.Source: §A.5 · p. 13

B Additional method details

• The corrigibility reward distinguishes resisting invalid evidence while correct from revising a wrong answer when evidence is valid.Anchor propositions are extracted from clean reasoning and verifier-gated so ATRC applies only when the clean answer is worth preserving.Source: B Additional method details · p. 13
• The four wrappers represent user follow-ups, retrieved documents, tool responses, and agent messages, changing interpretation without changing the adversarial string.Source: B Additional method details · p. 13

C Complete RAG training trajectory

• The RAG evaluation covers 28 checkpoints, with most gains appearing before the end of epoch 2 and the steepest improvement as retrieved content is treated as potentially adversarial.Figure 7 provides the checkpoint-level evaluation in place of a long checkpoint table.Source: C Complete RAG training trajectory · pp. 13–14, Figure 7 · p. 14

• The 3B base model averages 53.2% ASR across four wrappers, with a 24.1 pp spread between follow-up and RAG presentation.This authority gradient precedes finetuning; wrapper semantics change how persuasive identical injected content appears.Source: D Complete cross-framing results · p. 14, Figure 8 · p. 15
• The final 3B multi-framing model achieves the best overall average ASR at 13.9%, while the RAG specialist reaches 6.1% native ASR but transfers less evenly.The RAG specialist has 25.7% ASR on tool outputs and 26.5% on agent messages.Source: D Complete cross-framing results · p. 14
• RAG training yields its largest reduction in its native framing, while agent-message training transfers best to RAG at 15.7%, consistent with partial overlap in authority cues.Transfer depends on wrapper authority structure rather than being random.Source: D Complete cross-framing results · pp. 14–15, Figure 9 · p. 16
• 9.4% average ASR with an 8.3 pp spread is reached by the 7B model after one epoch of RAG-framed training, outperforming every 3B configuration.Source: D Complete cross-framing results · p. 16

• Tool-output training improves gradually: native ASR falls from 59.4% in the base model to 15.0% at the final checkpoint, while clean accuracy rises from 64.0% to 77.5%.The smoother trajectory is contrasted with the sharper phase transition in RAG training.Source: E Tool-output and agent-message training · p. 16, Figure 10 · p. 16, Table 9 · p. 17
• 15.0–20.2% ASR across all four wrappers is achieved by the fully trained tool model, reducing cross-wrapper spread to 5.2 pp.The tool-trained model reaches 15.0% native ASR at the final checkpoint.Source: E Tool-output and agent-message training · pp. 16–17, Figure 10 · p. 16, Table 9 · p. 17
• Agent-message training reduces average ASR from 53.2% in the base model to 31.6% after epoch 1 and 19.4% after epoch 2, with strongest final transfer to RAG at 15.7%.The final epoch-3 average slightly worsens to 20.0%; native agent ASR is 18.4%.Source: E Tool-output and agent-message training · p. 17, Table 10 · p. 17
• The channel-specific profiles differ: RAG training specializes deeply, tool training transfers most broadly, and agent training falls between them.Tool outputs and agent messages present attacks as external system state, but their resulting robustness profiles are not identical.Source: E Tool-output and agent-message training · pp. 16–18

F Ablation and scaling

• Removing the RAG-specific system prompt raises ASR from 5.5% to 10.0%, removing the wrapper raises it to 11.3%, and removing both raises it to 60.5%.Both components matter, with the wrapper contributing slightly more; the strongest robustness appears only when both are present.Source: F Ablation and scaling · p. 18, Figure 12 · p. 19
• 9.4% average ASR across four framings is reached by the 7B model after one epoch of RAG-framed training, with 10.4% native RAG ASR and an 8.3 pp spread.Later checkpoints do not improve the average; the final model reaches 10.8%.Source: F Ablation and scaling · p. 18, Table 12 · p. 18, Figure 11 · p. 18
• When transfer is the target, cross-framing averages favor the one-epoch 7B checkpoint over the final checkpoint, and stronger base models still require channel-aware evaluation.The remaining follow-up versus retrieved-document vulnerability makes a single in-channel score misleading.Source: F Ablation and scaling · pp. 18–19

G Cross-domain robustness

• On RAG-framed Natural Questions and TriviaQA, framing-aware models maintain relatively low ASR, but 3B clean accuracy is low, making capability the main bottleneck in these cross-domain settings.Source: G Cross-domain robustness · p. 19

H Hyperparameters and reproducibility

• The primary 3B setup uses Qwen2.5-3B-Instruct with 4-bit QLoRA, three training epochs, and an ATRC regularization weight of 0.1.The reported recipe also specifies training and reward-weight settings.Source: H Hyperparameters and reproducibility · p. 19
• The authors provide reproducibility configurations but do not report exact wall-clock measurements for every run.Source: H Hyperparameters and reproducibility · p. 19

I Assets and licenses

• Experiments rely on public third-party models, datasets, and verifier backbones; the submission does not redistribute their weights or dataset contents.Training wrappers, evaluation scripts, and configuration files are available through the project repository, subject to underlying asset licenses.Source: I Assets and licenses · p. 19