PaperlayerView exact revision on arXiv

Source-linked AI summary

Right Facts, Wrong Rule: Scope-Transfer Poisoning of Retrieval-Augmented Generation

Kaisheng Fan, Yishu Gao, Xunzhu Tang, Tegawend'e F. Bissyand'e, Weizhe Zhang

arXiv:2609.35155v2cs.CR

TL;DR

RAG relies on retrieved documents for both facts and rules, yet a valid rule may be applied beyond its documented scope, allowing correct facts to support a wrong answer. LENS constructs auditable attacks that preserve facts and visible boundaries. Across benchmarks and generators, it achieves a 47.1% target-error rate, while applicability auditing reduces target error from 70.8% to 16.4%.

  • Problem

    RAG can apply a valid documented rule to a question outside its scope, so correct facts may support a wrong answer and document validity alone is insufficient.

    Source: §1 · p. 1

  • Method

    LENS plans a grounded rule’s legitimate use and intended misuse, then constructs and repairs auditable attack documents while preserving factual support and explicit scope boundaries.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across four QA benchmarks and three target generators, LENS achieves 47.1% mean target-error over 4,000 fixed attempts; applicability auditing reduces target error from 70.8% to 16.4%, versus 31.2% for task-alignment checking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Checking and enforcing rule applicability during answer generation addresses the gap between recognizing a rule boundary and respecting it while preserving legitimate evidence use.

    Source: §7 · p. 9

    Evaluation covers fixed-query, single-round text RAG with documented rule scopes; ambiguous tasks and interactive or multimodal retrieval require further evaluation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) systems use external documents for facts and rules that guide answers. We study rule-scope misapplication, where a rule valid within its documented scope is applied to a question outside that scope. Correct facts can then support an incorrect answer even when the rule's boundary is visible. This exposes the limits of document-validity checks. We introduce LENS (Local Evidence, Non-local Steering), an LLM-agent-based attack framework for constructing auditable instances of this failure. Its Planner identifies a grounded rule, its legitimate use, intended misuse, and document roles. Proposer and Verifier agents construct and repair attack documents using local retrieval and reader feedback while preserving factual support and explicit scope boundaries. Across four QA benchmarks and three target generators, LENS achieves 47.1% target-error rate over 4,000 fixed attempts, including construction failures. A separate fixed-attempt audit yields 36.1% joint target-error rate under independent validity checks. Fact-preserving interventions link errors to rule scope. For 76.2% of target-matching default responses, the same model correctly judges the rule inapplicable when explicitly probed. With matched information and budgets, an applicability audit reduces target error from 70.8% to 16.4%, compared with 31.2% for task-alignment checking. These findings motivate checking and enforcing rule applicability during answer generation.

1 Introduction

RAG uses documents both for facts and answer-selection rules, but a true, in-scope rule can still be misapplied to a different question. The paper formalizes this failure, introduces LENS to construct auditable cases, and evaluates applicability-focused defenses.

1 Introduction

  • Rule-scope misapplication occurs when a valid rule is applied outside its documented scope, allowing correct facts to support an incorrect answer.Factual verification alone cannot establish whether the rule applies to the current question, while discarding it can remove a rule needed for legitimate questions.Source: §1 · p. 1
  • LENS plans a rule’s legitimate use and intended misuse, then constructs and repairs auditable attack documents while preserving factual support and visible scope boundaries.Local retrieval and reader feedback guide repairs; LENS revises its plan when document changes cannot satisfy constraints.Source: §1 · pp. 1–2
  • Paired tasks show that identical documents can support legitimate rule use for one question but misuse for another, which document-only checks cannot distinguish.The paper also gives a conditional scoped-inference guarantee: checking and enforcing applicability can block misuse while preserving legitimate cross-document reasoning.Source: §1 · pp. 1–2
  • Across four QA benchmarks and three target generators, LENS averages 47.1% target-error over 4,000 fixed attempts; independent validity checks yield 36.1% joint target-error.Failed constructions count as failures in the 47.1% rate.Source: §1 · p. 2

Boundary

The paper specifies a post-admission RAG threat model and defines attacks that preserve visible rule boundaries while inducing out-of-scope use. It then motivates applicability checking and states the assumptions under which enforcing applicability blocks misuse.

2.1 Setting and Threat Model

  • In text-based, single-round post-admission RAG, an attacker inserts a packet for a fixed query but cannot alter the query, corpus, retrieval pipeline, system prompt, or target generator.Construction uses local retrieval and surrogate-reader feedback under fixed budgets; target-system responses are obtained only after packets are frozen.Source: §2.1 · pp. 2–3

2.2 Tasks, Rules, and Applicability

  • Applicability depends on whether documented rule scope covers the task, not merely whether retrieved facts support a target answer under that rule.Source authorization does not extend a rule beyond its documented scope.Source: §2.2 · p. 3

2.3 Attack Objective and Legitimate Use

  • An attack packet preserves scoped claims, clean evidence, and a visible rule boundary while inducing an out-of-scope target; paired in-scope questions test legitimate use.Queries must be unambiguous, and target answers must be incorrect under the original query semantics.Source: §2.3 · p. 3

2.4 Checking and Enforcing Applicability

  • Document-only checks cannot distinguish legitimate from misapplied uses of the same rule; scoped inference blocks inapplicable derivations only under its stated recording, verification, and enforcement assumptions.The guarantee also assumes that answers rely only on accepted derivations and preserves legitimate uses when warranted applications are correctly approved and admitted.Source: §2.4 · p. 3

Misapplication

LENS plans a grounded rule and intended misuse, then constructs and repairs documents while preserving factual support and explicit scope boundaries. Its acceptance and audit procedures distinguish ordinary construction from stricter isolation testing and evaluate frozen packets independently.

Misapplication

  • The outer loop plans the rule and misuse, while the inner loop realizes and repairs documents under scope and factual-validity constraints.Figure 2 depicts the two-loop workflow and the isolation-mode subset checks.Source: Misapplication · p. 3, Figure 2 · p. 4

3.1 Planning the Misapplication

  • The Planner selects a grounded rule, its legitimate use, the intended scope-crossing misuse, and document roles, while preserving the condition that would make the misuse legitimate.The plan also guides repairs so stronger target induction does not alter the rule’s legitimate interpretation.Source: §3.1 · p. 3

3.2 Document Realization and Typed Repair

  • Proposers realize planned document roles, while Verifiers use retrieval, reader, alignment, and validity feedback to direct repairs without changing the stated scope or adding unsupported claims.If edits cannot satisfy the constraints, the failed plan returns to the outer loop for replanning.Source: §3.2 · p. 4

3.3 Acceptance, Isolation, and Freezing

  • Isolation mode requires complete retrieval and every proper subset to remain below tolerance, whereas main mode accepts a locally valid packet meeting its surrogate-reader target threshold.Accepted packets are frozen before target evaluation; attempts without a packet count as failures, and independent auditors assess a fixed sample without influencing construction or selection.Source: §3.3 · p. 4

4 Experiments

Across four QA benchmarks, LENS induces target errors in fixed-attempt evaluations, including attempts where construction fails. Paired-task and intervention results connect these errors to rule applicability, while re-segmentation tests show the pattern persists across document boundaries.

4.1 Experimental Setup

  • The evaluation freezes 1,000 query–target attempts per benchmark across four QA benchmarks and retains construction failures.Each attempt is capped at 15 proposal/repair actions and 1.05M construction tokens.Source: §4.1 · pp. 4–5

4.2 Main Results and Native Attacks

  • 47.1% mean target-error rate across 4,000 fixed attempts, with construction failures counted as failures.Rates range from 46.5% to 48.4% across target generators.Source: §4.2 · p. 5, Table 2 · p. 6

4.3 Matched Construction and Paired Tasks

  • The same packet induces 70.8% target error outside the rule’s scope while preserving 84.7% correct answers within it.Length-matched neutral material leaves out-of-scope error at 1.9%.Source: §4.3 · p. 5, Table 3 · p. 6
  • With matched objectives, feedback, and budgets, LENS reaches 47.6% TER versus 34.9% for generic repair.The passage attributes the comparison to how the constructor uses feedback.Source: Table 4 · p. 6, §4.3 · p. 6
  • 36.1% joint target-error rate under independent validity checks, versus 20.7% for generic repair and 27.1% for the criteria-inspired adaptation.Construction failures remain in the fixed audit denominator.Source: Table 5 · p. 6, §4.3 · p. 6

4.4 Mechanism: Intervention, Probe, and Re-segmentation

  • Checking applicability with documents frozen reduces TER from 70.8% to 16.4%, retaining 81.6% in-scope accuracy in the paired-task experiment.Scope clarification reduces TER to 22.6%; equal-budget re-checking, clarification, and generic reminders leave 48.6–61.9% TER.Source: Figure 3 · p. 7, §4.4 · p. 7, Table 6 · p. 7
  • Merging or splitting visible text changes TER by at most 2.7 points, and re-indexing chunk lengths changes it by at most 2.5.The mechanism persists across these document boundaries.Source: §4.4 · p. 7

5 Checking Rule Applicability

The section evaluates applicability checking as a way to reduce rule-scope misuse while retaining legitimate task performance. It also tests the audit against published defenses and adaptive attacks.

Applicability audit

  • The applicability audit extracts task requirements and candidate rules and scopes, judges applicability with evidence, then directs generation to use only rules marked applicable.The audit costs 2.0 calls and 5.0k tokens, an increment of 1.0 call and 2.1k tokens over unchecked answering.Source: §5 · p. 7

Published defenses

  • LENS retains the highest TER under each published defense, at 21.8–37.4%; applicability auditing reduces LENS to 11.9%.Table 7 reports target-error rates across attacks and defenses; the passage also reports 13.7% for the criteria-inspired attack under applicability auditing.Source: §5 · p. 7, Table 7 · p. 8

Applicability audit

  • 16.4% TER under applicability auditing versus 31.2% for task-alignment checking, with legitimate-task accuracy of 82.1% (85.1% unchecked).The matched-information comparison also reports 70.8% TER unchecked; Figure 4 presents attack TER and legitimate-task accuracy, plus paired-task detection and false positives.Source: §5 · p. 7, Figure 4 · p. 8

Adaptive attacks

  • Under expanded attack budgets, the strongest combined adaptive strategy raises defended TER from 12.1% to 24.1%.Among 160 audited residual failures, the most common first errors involve rule/scope extraction or binding (26.3%), applicability judgment (24.4%), verdict enforcement (21.9%), and task extraction (19.4%).Source: §5 · p. 8
  • Replacing automatic task and rule/scope extraction and applicability judgments with audited labels lowers TER from 16.4% to 5.4%, while enforcement and generation remain automatic.The same passage identifies errors in extraction, applicability judgment, verdict enforcement, and task extraction as practical audit shortfalls.Source: §5 · p. 8

6 Related Work

Related work covers corpus poisoning and cross-document coordination, manipulation of criteria and task definitions, and robust RAG defenses. This paper distinguishes its focus on explicit-scope rule misuse and paired-task evaluation.

Corpus poisoning and cross-document coordination

  • Corpus-poisoning attacks steer answers through inserted content or cross-document coordination, whereas LENS tests misuse of a grounded rule whose scope remains explicit.Paired questions test whether the same evidence also supports legitimate in-scope answers.Source: §6 · p. 8

Standards, criteria, and task definition

  • Prompt attacks manipulate judgment criteria, while contextual-integrity work studies shifts in norms and authorization scopes; this paper tests steering beyond a rule’s explicit scope.Paired tasks, scope interventions, and separate probes distinguish applicability judgments from default answer behavior.Source: §6 · p. 8

Robust retrieval-augmented generation

  • Robust RAG defenses use evidence isolation, reliability checks, aggregation, or trained risk assessment; this work compares content-focused checks with applicability checking and enforcement.The comparison reports legitimate-task utility alongside attack reduction.Source: §6 · p. 8

7 Conclusion

The conclusion frames rule-scope misapplication as a security failure in which true facts and a locally valid rule support an incorrect answer because the rule is applied outside its scope. It summarizes LENS, mechanism tests, and applicability auditing as responses to that failure.

Conclusion

  • Rule-scope misapplication uses true facts and a locally valid rule to support an incorrect answer beyond the rule’s stated scope.The conclusion says explicit probes show models can judge a rule inapplicable yet still produce a target error during normal answering.Source: §7 · p. 9
  • LENS constructs auditable instances, while paired tasks and interventions isolate the mechanism behind the errors.Source: §7 · p. 9
  • An applicability audit reduces target errors while preserving legitimate evidence use, motivating checks that enforce applicability during generation.Source: §7 · p. 9

Limitations

The evaluation has a defined setting and the attack procedure carries a misuse risk. The authors identify further evaluation needs and limit recommended use to authorized testing.

Limitations

  • Experiments cover single-round, text-based RAG with fixed queries and documented rule scopes; ambiguous, interactive, and multimodal retrieval need further evaluation.Reported attack rates are conditional on corpus admission and selection of clean-answerable, semantically screened questions; auditing also relies on trusted task requirements and correct extraction, judgment, and enforcement.Source: Limitations · p. 9

Ethical considerations

  • Because the attack procedure could manipulate deployed assistants and mislead users, the authors say its use should be limited to authorized testing.Source: Limitations · p. 9

Related papers

Loading 2609.35155v2…