PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdoor in the Loop: Compromising Agentic Search via Malicious Retrievers

Beining Xu, Peichun Hua, Yunming Xiao

arXiv:2609.37468v1cs.CRcs.AIcs.IRcs.LG

TL;DR

Agentic RAG lets retrieved evidence shape both answers and later search decisions, creating a trajectory-level backdoor challenge for retrievers. The paper studies retriever-only attacks and uses Decoy Overwrite–Unlearn to weaken detector-visible signatures while preserving malicious retrieval behavior.

  • Problem

    Retriever backdoors in agentic search must sustain their effects across changing queries and search decisions, even when the attacker cannot alter the deployment corpus.

    Source: §1 · p. 2

  • Method

    The paper studies three trajectory-level retrieval attacks and applies limited auxiliary-backdoor injection and unlearning to conceal the original backdoor.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    46.53 to 29.26: DOU reduces held-out Spectral Signatures separation ∆= meani |AUROCi −50| while malicious retrieval behavior persists.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Retriever-only attacks can manipulate agent evidence and search behavior while the agent and deployment corpus remain unchanged.

    Source: §5 · p. 10

    Targeted backdoors are harder to conceal with DOU alone because they reinforce attraction to the same document across triggered inputs.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Agentic retrieval-augmented generation (RAG) interleaves reasoning with repeated retrieval, giving the retriever influence over both the evidence an agent observes and its subsequent search decisions. We study retriever backdoors that exploit this feedback loop and repurpose weak backdoor purification to conceal their presence. An attacker supplies a compromised retriever checkpoint while leaving the search agent and deployment corpus unchanged. Without corpus write access, the attacker can still suppress useful evidence, persistently retrieve a selected existing document, or steer the agent toward prolonged search, inflating retrieval, context, and latency cost. To conceal these behaviors from detection, we propose leveraging a controlled inject-and-remove cycle: deliberately inject a weaker backdoor and then unlearn it. This process weakens detector-visible signatures and fools the backdoor detectors with an illusion of purification while preserving the malicious retrieval behavior. These findings expose a systematic vulnerability in RAG systems in which a weak defense becomes an attacker's concealment tool for a backdoored retriever, even when the underlying corpus remains trustworthy.

1 Introduction

Agentic RAG gives retrievers influence over evidence and subsequent search decisions, creating a trajectory-level backdoor challenge even when the deployment corpus is unchanged. The paper studies retriever-only attacks and uses Decoy Overwrite–Unlearn to weaken detector-visible signatures while preserving malicious retrieval behavior.

1 Introduction

  • Because the agent generates changing queries and stopping decisions, success on one retrieval request does not guarantee an attack will persist across the search trajectory.Retrieved passages can reshape subsequent queries and whether the agent continues searching.Source: §1 · pp. 1–2
  • A compromised retriever checkpoint can suppress supporting evidence, repeatedly retrieve a selected existing document, or prolong search while the agent and corpus remain unchanged.The targeted attack requires no permission to insert, edit, or delete deployed documents.Source: §1 · p. 2
  • DOU injects and unlearns a limited auxiliary backdoor to alter detector-inspected features while leaving the original malicious association active.The resulting model can appear more benign to a detector without recovering safe retrieval behavior.Source: §1 · p. 2
  • 46.53 to 29.26: DOU reduces held-out Spectral Signatures separation ∆= meani |AUROCi −50| while malicious retrieval behavior persists.The held-out detection set is disjoint from all training data.Source: §1 · pp. 2–3

2 Background and Problem Setting

The background defines RAG and agentic search, where reasoning and retrieval repeat across rounds. The threat model gives an attacker control of the retriever checkpoint, but not the deployed agent or corpus.

2 Background and Problem Setting

  • RAG retrieves K corpus documents for a question and supplies them to a generator to produce an answer.The paper introduces dense retrieval as ranking documents by query-document encoder similarity.Source: §2.1 · p. 3

2.1 Agentic Search Systems

  • In agentic search, the agent generates each round’s query, receives retrieved documents, and then decides whether to search again or answer.The retriever input combines the original question with the agent-generated query, so retrieved evidence can affect later queries, stopping, and the final answer.Source: §2.1 · p. 3

2.2 Threat Model

  • The victim keeps its agent, prompts, corpus, and inference pipeline intact while deploying an externally supplied retriever checkpoint; the attacker can alter retriever parameters but not the deployed agent’s reasoning process.The attack goals are to suppress supporting evidence, prolong search, or keep an existing target document in top-K results across rounds while preserving clean-question quality.Source: §2.2 · p. 4

3 Retriever Backdoors: Manipulation and Concealment

The section develops retriever-only attacks that manipulate evidence and search behavior, then uses a controlled inject-and-remove procedure to conceal the backdoor while retaining its malicious retrieval behavior.

3 Retriever Backdoors: Manipulation and Concealment

  • Each attack combines a malicious objective with clean retrieval supervision; the attacker controls the retriever while the agent chooses subsequent queries and stopping decisions.The training objective adds a weighted attack loss to the clean retrieval loss.Source: §3 · p. 4

3.1 Evidence Integrity: Untargeted Retrieval

  • The untargeted attack promotes a low-ranked nonsupporting passage over competing candidates, depriving the agent of needed information and steering later queries.Source: §3.1 · pp. 4–5

3.2 Persistent Targeting: Repeated Document Retrieval

  • Across round-specific inputs, the targeted objective promotes a selected existing document over competitors to sustain its retrieval as queries evolve.Source: §3.2 · p. 5

3.3 Control Flow: Prolonged Search

  • An offline judge trained on disjoint trajectories scores passage associations with stopping, and its selected passage becomes the triggered input’s positive retrieval target.Source: §3.3 · p. 5

3.4 Backdoor Concealment via Decoy Overwrite–Unlearn

  • DOU injects and then unlearns an auxiliary retrieval association with limited updates, altering detector-visible behavior while retaining the original attack.Neither concealment stage uses detector outputs, labels, or detection-test examples.Source: §3.4 · pp. 5–6
  • For targeted backdoors, DOU alone does not fully remove the persistent attraction to the same document; stability equalization lowers detector separation at the cost of weaker answer-quality degradation.Source: §3.4 · p. 6

4 Evaluation

Across two datasets and two retrievers, the evaluation tests three retriever-only attacks, post-training purification, and DOU concealment. The attacks degrade triggered answer quality or prolong costly search, while DOU reduces detector separation without repairing untargeted backdoors; targeted backdoors remain harder to conceal.

4.1 Experimental Setup

  • The evaluation uses 1,000 test questions each from HotpotQA and TriviaQA, E5 and Contriever, and answer-quality, retrieval, search-cost, and detector-separation metrics.Clean utility is measured on trigger-free questions; concealment uses detector AUROC separation from chance across evaluation runs.Source: §4.1 · p. 6

4.2.1 Untargeted Retrieval

  • Triggered F1 falls 18.8–30.9 points relative to clean inputs on the same backdoored checkpoint, across both retrievers, both datasets, and all three triggers.Clean-input F1 changes by at most 1.92 points either way; 90 naturally triggered questions also show a 13.9–34.7-point F1 drop versus the original retrievers.Source: §4.2.1 · pp. 6–7, Table 1 · p. 7

4.2.2 Persistent Targeting

  • Under activation, 99.8%–100% of questions retrieve the target in every executed round, and it ranks first in over 99.7% of rounds.The target appears in only 1.8%–2.7% of clean-question trajectories; triggered F1 falls by 22.5–30.1 points and mean search length rises by roughly one round.Source: §4.2.2 · p. 7, Table 2 · p. 8

4.2.3 Round Control

  • At high intensity, mean search length rises 78–88%, final context grows 81–87% in tokens, completion latency increases 80–126%, and F1 falls 24–29 points.The pattern holds at every tested intensity, with more rounds, tokens, and latency accompanying worse answers; clean-input behavior changes little.Source: §4.2.3 · pp. 7–8

4.3 Persistence under Post-training Purification

  • With CleanFT and pruning, clean–trigger F1 gaps remain 28.64–30.05 points on HotpotQA and 23.12–24.10 on TriviaQA; BPO also fails to remove the original backdoor.After BPO unlearning, the A2O auxiliary target-document win rate falls from 100% to 0%, while the original backdoor persists.Source: §4.3 · pp. 8–9, Table 4 · p. 9

4.4 Untargeted Concealment

  • At 80/80 steps, untargeted DOU reduces detector separation while triggered F1 remains within 2.80 points of the unprocessed backdoor on both datasets.On held-out MuSiQue, detector separation also falls at 80/80 steps, while triggered-input F1 changes by at most 0.9 points.Source: §4.4 · p. 9, Table 5 · p. 10, Table 6 · p. 10

4.5 Targeted Concealment

  • On held-out MuSiQue, targeted DOU reduces detector separation for several detectors, but target retrieval persists at 99.80% Target Hit and 99.80% Every-Round Hit.Activation Clustering remains comparatively high, reaching a minimum separation of 30.51 at the deepest tested budget.Source: §4.5 · p. 10, Table 6 · p. 10

5 Conclusion

Retriever-only attacks manipulate the evidence available to an agent and its search behavior while leaving the agent and deployment corpus unchanged; DOU repurposes inject-and-remove purification to conceal malicious retrieval behavior. The appendix describes ranking and outcome distinctions, attacker-side construction budgets, and an optional stability-equalization stage for targeted backdoors.

5 Conclusion

  • Retriever-only attacks manipulate an agent’s evidence and search behavior, while DOU weakens detector-visible signatures and retains malicious retrieval behavior.Experiments on two retrievers and two question-answering datasets show effectiveness with limited degradation on clean inputs.Source: §5 · p. 10

AI use statement

  • Generative AI tools assisted with manuscript language and formatting and artifact implementation; the authors reviewed the resulting materials and take responsibility for the final content.Source: AI use statement · p. 10

Ethics statement

  • All experiments use established QA benchmarks and locally hosted retrievers, without human subjects, real users, or production systems.The authors report releasing code while limiting unrestricted redistribution of ready-to-use backdoored artifacts.Source: Ethics statement · p. 11

A Supplementary Formulation and Implementation

  • The appendix holds the Qwen2.5-7B-Instruct Search Agent and wiki18_100w corpus fixed when comparing clean and backdoored retrievers.It also identifies the appendix as supplying ranking interpretation and implementation details for Sections 2.1 and 3.4.Source: A Supplementary Formulation and… · p. 14

A.1 Ranking Changes and Agent Outcomes

  • The agent controls queries and stopping, so manipulated retrieved evidence can affect subsequent queries, the stopping round, and the final answer.A retrieval hit alone does not establish trajectory-wide control: agents may reject targets, recover evidence later, or answer from parametric knowledge.Source: §A.1 · pp. 14–15

A.2 Auxiliary Overwrite–Unlearn Construction

  • The overwrite/unlearn sweep uses shared 20/20, 40/40, 80/80, and 240/160-step settings, plus 240/240 steps for the targeted attack.The main text highlights 80/80 for untargeted attacks and 240/160 for targeted attacks; these attacker-side runs differ from the defender-side multi-epoch protocol.Source: §A.2 · p. 15

A.3 Optional Extension: Stability Equalization for Targeted Backdoors

  • DOU alone leaves a residual signature for targeted backdoors because their shared target attraction differs from all-to-all overwriting’s distributed associations.The optional equalization stage matches average instability between clean and triggered representations and penalizes instability in both groups; it trades concealment for weaker answer-quality degradation.Source: §A.3 · pp. 15–16

B Defense Methods and Evaluation Protocols

The appendix distinguishes defender-side purification from detection and describes the evaluated detector entries as retrieval adaptations or diagnostic baselines.

B Defense Methods and Evaluation Protocols

  • Defender-side purification changes retriever parameters to restore answer quality, whereas detection scores inputs for trigger-associated behavior.The seven detector entries are retrieval adaptations or diagnostic baselines, and their scores define the procedures evaluated against attacker-side concealment.Source: B Defense Methods and Evaluation… · p. 16

B.1 Post-training Purification

The appendix evaluates clean fine-tuning, magnitude pruning followed by clean fine-tuning, and a retrieval adaptation of BPO as defender-side purification methods.

B.1 Post-training Purification

  • Each evaluated checkpoint receives a newly encoded corpus index, and the selected checkpoint maximizes triggered F1 within each method’s tested schedule and dataset.Evaluation pairs 1,000 clean and author-triggered questions per dataset, with at most five retrieval rounds; the fine-tuning/pruning and BPO records use different source baselines.Source: §B.1 · p. 16
  • CleanFT optimizes the retrieval loss on clean retrieval pairs and evaluates checkpoints after 5, 10, 15, 20, 25, and 30 epochs.The recorded configuration uses learning rate 2 × 10^-5, effective batch size 256, groups of 16 candidate passages, temperature 0.01, and seed 2024.Source: §B.1 · p. 16
  • Magnitude pruning removes the lowest-magnitude 30% of eligible trainable linear-layer weights before CleanFT, while embeddings, layer-normalization parameters, and biases remain outside the pruning set.The binary mask is reapplied during fine-tuning so pruned entries stay zero; this baseline tests fixed magnitude pruning rather than activation-ranked neuron pruning.Source: §B.1 · pp. 16–17
  • The BPO-style retrieval adaptation overwrites an auxiliary mapping using either deranged positive documents or one shared target, then unlearns it by restoring original positives and applying parameter sparsification.Overwriting lasts five epochs, followed by unlearning checkpoints at two, five, and ten epochs; this is a retrieval adaptation with its own training budgets, not a replication of BPO’s image-classification configuration.Source: §B.1 · p. 17

B.2 Detection Data, Access, and Aggregation

The appendix specifies detection data, access conditions, scoring, and implementation details for representation-, trigger-inversion-, and perturbation-based detector adaptations.

B.2 Detection Data, Access, and Aggregation

  • Detection uses 1,000 unmodified questions per dataset with hidden standalone-author trigger labels, and evaluates separation rather than a model-level verdict across independently trained models.TriviaQA-1000 selects the DOU budget, while MuSiQue-1000 confirms it; the latter shares no exact questions with the former or the listed training data.Source: §B.2 · p. 17
  • The reported Δ averages each detector’s absolute AUROC deviation from 50, preventing opposite score orientations from canceling.A low Δ indicates weak separation but does not alone specify an operational false-positive rate; reported standard deviations describe detector-run variability, not confidence intervals over independently sampled questions.Source: §B.2 · p. 17

B.3 Representation-based Detectors

  • Representation detectors include clustering query embeddings into two groups and scoring proximity to the suspicious cluster, or scoring alignment with the leading centered representation direction.Activation Clustering uses up to 50 principal components and 20 k-means initializations; Spectral Signatures analyzes the query set together without class labels.Source: §B.3 · p. 18

B.4 Trigger-Inversion Adaptations

  • The Neural Cleanse and PICCOLO entries share a soft-trigger inversion objective and score each question by its maximum cosine similarity to a learned soft-trigger vector.These are retrieval adaptations using the same objective and lexical scoring path, not independent implementations of the original papers’ full detection pipelines.Source: §B.4 · p. 18

B.5 Perturbation-based Detectors

  • Black-box TopK Instability measures average Jaccard distance between retrieved document sets after fixed word insertions, without requiring encoder gradients.Unusually stable or unstable triggered retrieval can separate groups, so the reported summary is orientation-independent.Source: §B.5 · p. 18
  • The RAP-style adaptation uses fixed word insertions and embedding displacement, while STRIP-style retrieval entropy measures the identity distribution of retrieved documents across mixed-question variants.The RAP adaptation does not train the specialized perturbation embedding of the original classification defense; STRIP replaces class-output entropy with retrieval-identity entropy.Source: §B.5 · pp. 18–19

C Qualitative Agent Trajectories under Triggered Retrieval

Selected paired trajectories illustrate triggered retrieval failures in both untargeted and targeted attacks: useful evidence can be suppressed, or an existing document can be repeatedly promoted.

C Qualitative Agent Trajectories under Triggered Retrieval

  • Every selected clean answer matches its reference, while every paired triggered answer is wrong.The examples use the same backdoored E5 checkpoint within each clean/triggered pair; they illustrate outcomes, not their frequency.Source: C Qualitative Agent Trajectories under… · p. 20
  • In five untargeted cases, triggered searches reach the five-retrieval limit and return no valid answer, while clean counterparts answer in one or two rounds.Five further triggered trajectories end with specific wrong answers; two HotpotQA pairs use the same number of retrievals for clean and triggered inputs.Source: C Qualitative Agent Trajectories under… · p. 20
  • In the last five targeted cases, Pavia Cathedral (ID 4) ranks first in every triggered round, and each wrong answer appears in that document.The repeated exposure is consistent with answer contamination across unrelated questions, but the trajectories do not establish the agent’s internal causal path.Source: C Qualitative Agent Trajectories under… · p. 20

D Extended Defense Results

Extended defense evaluations show that purification sweeps do not consistently restore triggered performance, while concealment-budget results vary across detectors and held-out settings.

D.1 Purification Checkpoint Sweeps

  • Across 5–30 epochs, CleanFT and pruning retain a large clean–trigger gap, and BPO sweep changes in original-trigger F1 are all negative.Neither defense consistently restores triggered answer quality as training proceeds.Source: §D.1 · p. 21, Figure 3 · p. 21, Figure 4 · p. 21

D.2 Concealment Budget Sensitivity (TriviaQA-1000)

  • The TriviaQA-1000 budget-sensitivity tables report untargeted-backdoor detection across DOU budgets on the budget-selection and held-out sets.They report detector separability as ∆, with lower values indicating less separability; the supplied captions do not state comparative outcomes.Source: §D.2 · p. 21, Table 8 · p. 22, Table 9 · p. 22

D.3 Held-out Budget Confirmation (MuSiQue-1000)

  • On held-out MuSiQue-1000, NC, PICCOLO, and TopK are lowest at 80/80, while AC, SS, RAP, and STRIP are lowest at 240/160.The paper identifies 80/80 as the better overall choice because the latter detectors’ margins are narrower.Source: §D.3 · p. 22

D.4 Targeted Concealment: Budget Sweep and Optional Equalization

For targeted backdoors, DOU budgets and optional stability equalization reduce some detector separations, but equalization also weakens answer degradation under activation.

D.4 Targeted Concealment: Budget Sweep and Optional Equalization

  • Activation Clustering remains above ∆=30 at every budget; NC is lowest at 240/240, PICCOLO at 240/160, and SS plateaus around ∆≈25–26 at the two deepest budgets.The targeted-backdoor sweep uses held-out MuSiQue-1000 with 30 positives; lower ∆ indicates less detector separation.Source: §D.4 · p. 23, Table 10 · p. 23
  • Equalization lowers AC’s ∆ by 1.90–3.73 points and SS’s by 3.01–6.59 points across matching budgets, while NC changes by at most 0.83 points.At 240/160, AC falls from 30.61 to 28.71 and SS from 26.22 to 23.21; PICCOLO changes by as much as 2.24 points.Source: §D.4 · p. 23, Table 11 · p. 23
  • Target retrieval remains persistent with equalization, but triggered F1 rises from 26.27 to 36.36 on HotpotQA and from 46.57 to 53.88 on TriviaQA.These comparisons run from the unprocessed backdoor to DOU 40/40 plus equalization; the intermediate DOU 40/40 values are 30.28 and 50.05, respectively.Source: §D.4 · p. 24, Table 12 · p. 24
  • Without equalization, target retrieval persists at every DOU budget, and triggered F1 changes nonmonotonically; 240/240 does not materially improve answer quality over 240/160.Only the 40/40 equalization checkpoint has a matching QA evaluation among the equalization budgets.Source: §D.4 · p. 24

E Additional Round-Control Results

Additional round-control results show that clean-input behavior remains stable while stronger trigger intensity lengthens search and increases resource use.

E Additional Round-Control Results

  • Across the clean baseline and three trigger-intensity checkpoints, clean F1 ranges from 54.83 to 57.40 on HotpotQA and from 80.08 to 81.85 on TriviaQA.Mean search length and token use also change little on clean inputs.Source: E Additional Round-Control Results · p. 25
  • Overflow beyond 16,384 tokens is rare even at high intensity, affecting at most 6 of 1,000 HotpotQA trajectories and 1 of 1,000 TriviaQA trajectories.Added rounds mainly consume latency and token budget without triggering an explicit context failure.Source: E Additional Round-Control Results · p. 25, Table 13 · p. 25
  • Maximum observed HotpotQA rounds rise from 18 at low intensity to 33 at high intensity, lengthening the tail of the round distribution.The evaluation uses a 100-retrieval implementation limit, but no evaluated trajectory reaches it.Source: E Additional Round-Control Results · pp. 24–25

Related papers

Loading 2609.37468v1…