PaperlayerView exact revision on arXiv

Source-linked AI summary

Socrates-RAG: Premise-Directed Inquiry against Coordinated Evidence Poisoning

Renyu Zhao, Xinyuan Zou, Lanbin Liu

arXiv:2609.35773v1cs.IR

TL;DR

When decisive evidence is missing from the initial context, choosing what to retrieve next becomes part of RAG reliability. Socrates-RAG searches for evidence that resolves an unresolved premise, then refines its next query; in a 48-world comparison, it improves safe accuracy over matched query expansion, with benefits concentrated in two-hop cases.

  • Problem

    Fixed-context defenses cannot recover decisive evidence absent from the retrieved set, although that evidence may be available in the searchable corpus.

    Source: Active-Evidence Socratic Inquiry · p. 2

  • Method

    Socrates-RAG represents competing answers, selects an unresolved premise that would distinguish them, and uses acquired evidence to refine a subsequent query before answering or abstaining.

    Source: Introduction · p. 1

  • Results

    14.6 points: Socrates-RAG improves safe accuracy over matched query expansion across 48 disjoint worlds, with 8 wins, 1 loss, and 39 ties (exact p = .0391).

    Source: Introduction · p. 1

  • Takeaways & Limitations

    The benefit is concentrated where newly acquired evidence must be composed into a second retrieval action, rather than being a general advantage over capable query expansion.

    Source: Discussion and Limitations · p. 4

    The fictional templated evaluation does not estimate how prevalent rescueable conflicts are on the open Web, and the conditional guarantee can fail when evidence is absent, poorly indexed, or requires more than two hops.

    Source: Discussion and Limitations · p. 4

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) defenses typically decide how to filter or aggregate a fixed retrieved set. In open-corpus question answering, however, decisive evidence may be absent from the initial context but retrievable, making the next query part of the reliability problem. We introduce Socrates-RAG, a premise-directed active retrieval policy that represents competing answers, selects an unresolved premise whose resolution would discriminate them, and uses newly acquired evidence to refine a subsequent query before answering or abstaining. We formalize the resulting finite-budget evidence state and give a conditional rescue guarantee relative to repeated or topical-query policies. We evaluate Socrates-RAG against a matched control in which the same backbone generates ordinary relevance-oriented search queries; both policies share the initial evidence, deterministic retriever, two-query/top-three budget, answer prompt, and label-free evidence-chain release rule. On a disjoint 48-world counterfactual evaluation, premise-directed inquiry raises safe accuracy from 79.2% to 93.8%, with 8 wins, 1 loss, and 39 ties (two-sided exact p=.0391). Decisive-evidence recall improves by the same margin, while unsafe answers fall from one to zero. Both policies solve all 24 one-hop cases; the gain is concentrated in two-hop cases, where Socrates-RAG substitutes a newly resolved premise into its second query. This controlled study isolates a specific benefit of premise-directed acquisition without claiming general robustness on the open Web.

Abstract

Fixed-context defenses cannot recover decisive evidence missing from the retrieved set, while repeating a poisoned query may retrieve more replicas. Socrates-RAG directs retrieval toward unresolved premises; in a 48-world comparison, it improves safe accuracy over matched query expansion, with gains concentrated in two-hop cases.

Problem

  • Fixed-set defenses cannot recover decisive evidence absent from the accessible set, and repeating a poisoned question may retrieve more attacker-controlled replicas.Existing defenses filter or aggregate retrieved evidence, whereas active inquiry can search for a missing record.Source: Introduction · p. 1

Approach

  • Socrates-RAG treats competing answers as hypotheses, searches for a record that discriminates them, and abstains when no such evidence is found.This separates evidence acquisition from evidence aggregation and connects poisoning defense to selective prediction (Geifman and El-Yaniv 2017; Srinivasan et al. 2024).Source: Introduction · p. 1

Evaluation

  • The evaluation compares against same-backbone query expansion as well as repeated-question retrieval under matched search and answer budgets.The comparison tests whether ordinary expansion can infer the missing premise and authoritative source.Source: Introduction · p. 1

Contribution

  • The study formulates finite-budget evidence acquisition and selective disposition, introduces premise-directed retrieval, and states a conditional guarantee against repeated or topical-query policies.Source: Introduction · p. 1

Results

  • 14.6 points: Socrates-RAG improves safe accuracy over matched query expansion across 48 disjoint worlds, with 8 wins, 1 loss, and 39 ties (exact p = .0391).The benchmark includes a same-backbone query-expansion control and one- and two-hop cases.Source: Introduction · p. 1

Related Work

Related work addresses RAG reliability through filtering, aggregation, correction, and verification, while Socrates-RAG focuses on selecting the next query. The paper positions active rescue as complementary to these approaches and studies a setting where decisive evidence is reachable but absent initially.

RAG reliability and poisoning

  • Existing RAG work studies context distraction, corpus poisoning, attack and defense benchmarks, certified aggregation, and anomaly filtering, while this study focuses on explicit claim-source relations and does not establish superiority under other methods’ native objectives.Relevant approaches include PoisonedRAG (Zou et al. 2025), RobustRAG (Xiang et al. 2024), and TrustRAG (Zhou et al. 2025).Source: Related Work · p. 2
  • ReliabilityRAG assumes rank or reliability weights indicate source quality; its controlled attacks motivate evaluating a mechanism under conditions where it can work.It builds a contradiction graph over isolated answers and selects a rank-aware maximum independent set (Shen et al. 2025).Source: Related Work · p. 2

Retrieval correction and self-verification

Cross-examination and multi-agent debate

Factuality, provenance, and selective prediction

Active-Evidence Socratic Inquiry

Socrates-RAG treats active retrieval as premise-directed evidence acquisition under a finite query budget. It seeks evidence that discriminates competing answers, updates the evidence state, then answers or abstains; its guarantee is conditional on separability and the final decision rule.

Problem setting

  • Active inquiry asks which unresolved premise and query can retrieve evidence that discriminates competing answers, which fixed-context filtering cannot do when the decisive fact is absent.The claim is restricted to rescueable cases where decisive evidence exists in the accessible corpus but not in the initial context.Source: Active-Evidence Socratic Inquiry · p. 2

Socratic policy

  • The Assertor represents answer candidates and necessary premises; Elenchus selects a discriminating premise and query, then Aporia and Dialectic decide whether to answer or return CannotDetermine.A deterministic controller enforces the budget, de-duplicates documents, and records terminal errors.Source: Active-Evidence Socratic Inquiry · p. 2

Matched query-policy comparison

  • Socrates-RAG returns a premise-directed query while same-backbone query expansion returns an ordinary relevance-oriented query; both share retrieval, budget, answer prompt, finalizer, and release rule.The matched comparison isolates query generation rather than additional search capacity or a different answerer.Source: Active-Evidence Socratic Inquiry · pp. 2–3, Figure 1 · p. 3

Evidence-chain release guard

  • The release guard permits an answer only when cited pages contain a controlling final record or a controlling bridge with its linked terminal record; otherwise it abstains.The rule uses document text and cited aliases, not gold answers, poison labels, hop counts, or evaluator-only decisive-record IDs.Source: Active-Evidence Socratic Inquiry · p. 3

Conditional rescue guarantee

  • A premise-directed policy has safe coverage one and zero unsafe-answer rate on separable states, while repeated-question retrieval has zero safe coverage under the stated conditions.A state is (B,r)-separable when decisive records determine one answer, can be retrieved through at most B premise-resolving queries, and are not retrieved by repeating or topically paraphrasing the question.Source: Active-Evidence Socratic Inquiry · p. 3
  • The conditional guarantee scales the difference in decisive-acquisition probability by the finalizer’s correctness probability, assuming both policies otherwise abstain.The guarantee does not claim arbitrary Web states are separable; the experiments measure its premise through decisive-evidence recall.Source: Active-Evidence Socratic Inquiry · p. 3

Counterfactual Active-Evidence Evaluation

The evaluation constructs fictional counterfactual worlds with controlled evidence reachability and compares active retrieval policies under a matched search budget. Table 1’s caption identifies the active and fixed-context blocks and defines safe versus unsafe released answers.

Evaluation worlds

  • The worlds use fictional people, archives, dockets, and storage rules so parametric memory cannot supply answers while evidence reachability remains controlled.Each begins with a provisional notice and coordinated false-answer pages; decisive official evidence is searchable but its identifier is hidden.Source: Counterfactual Active-Evidence… · p. 3

Evaluation protocol

  • The primary evaluation contains 48 disjoint worlds, balanced between one-hop and two-hop cases, with active policies sharing initial evidence, deterministic lexical retrieval, and a two-query, top-three budget.They also share the model route, answer prompt, and evidence-chain release rule; runtime inputs exclude evaluator-only gold answers and decisive-record metadata.Source: Counterfactual Active-Evidence… · p. 3

Reported results

  • Table 1 covers the same 48 active-rescue worlds, with active acquisition in the top block and fixed-context adaptations below; safe means a correct released answer.Incorrect released answers are classified as unsafe.Source: Table 1 · p. 4

Results

On 48 evaluation worlds, Socrates-RAG improves safe accuracy over matched query expansion, with the difference concentrated in two-hop cases where a resolved premise changes the next query. The reported advantage is bounded by exploratory near-parity conditions, retrieval and attack assumptions, and higher token and latency costs.

Results

  • In separate 12-world development analyses, Socrates-RAG solves 12/12 versus expansion’s 11/12 when identifiers are visible, and both solve 11/12 when only source type is visible.These exploratory analyses are not pooled with the disjoint evaluation and mark conditions where premise-directed structure has little room to help.Source: Results · p. 4
  • 93.8% safe accuracy for Socrates-RAG versus 79.2% for matched query expansion on 48 evaluation worlds (8 wins, 1 loss, 39 ties; p = .0391).Decisive-evidence recall has the same 45/48 versus 38/48 split, while unsafe answers are 0/48 versus 1/48.Source: Results · p. 4
  • On the same 48 worlds, neither RobustRAG nor TrustRAG acquires decisive records; their outcomes are descriptive, not comparisons under their native threat models.RobustRAG releases incorrect answers in 48/48 cases; TrustRAG releases 24/48 incorrect answers and abstains on 24/48.Source: Results · p. 4
  • Both policies solve all 24 one-hop worlds; in two-hop worlds, Socrates-RAG solves 21/24 versus 14/24 for expansion.In seven of eight Socratic wins, a first-query record supplies a newly resolved archive or rule for the second query.Source: Results · p. 4
  • Both active policies use 144 model calls, but Socrates-RAG uses 121,771 output tokens and 1,215.6 seconds of latency versus 66,346 and 758.9 for expansion.The paper reports that the accuracy gain comes at substantial output and latency cost.Source: Results · p. 4

Discussion and Limitations

  • The conclusion is limited to settings where resolving an intermediate premise changes the next useful query; fictional worlds and deterministic lexical retrieval do not establish open-Web prevalence or general robustness.The paper also identifies one proprietary model route, non-adaptive topical replicas, a conditional separability guarantee, and increased token and latency costs as limitations.Source: Discussion and Limitations · p. 4

Conclusion

Socrates-RAG treats absent decisive evidence as an acquisition problem, using premise-directed queries to recover evidence before answering or abstaining. Under stated separability conditions, this policy can rescue cases that repeated or topical queries cannot.

Conclusion

  • Premise-directed inquiry can recover absent-but-reachable evidence by resolving an unresolved premise and using it to refine a later query.The system answers only when acquired evidence supports a unique answer; otherwise it abstains.Source: Conclusion · p. 4

Formal Details

  • Separability requires a within-budget query sequence that retrieves the decisive set, with later queries depending on earlier resolved premises, while topical policies cannot retrieve that set.Source: Formal Details · p. 7
  • On (B, r)-separable instances, a separating premise-query sequence achieves safe coverage one and unsafe rate zero, while topical policies have safe coverage zero.The guarantee assumes the release rule answers correctly when the decisive set is acquired and otherwise abstains.Source: Formal Details · p. 7
  • With a shared finalizer that answers correctly with probability α after decisive acquisition and otherwise abstains, decisive-evidence recall mediates the safe-coverage difference.The experiments measure this mechanism through decisive-evidence recall.Source: Formal Details · p. 7
  • Adding arbitrarily many lower-scoring coordinated replicas cannot displace a decisive document ranked within the top r; applying the argument after each resolved bridge gives a bounded-hop result.Source: Formal Details · p. 7

Method and Prompt Contracts

The method uses structured roles to identify a decisive unresolved premise, retrieve evidence under a fixed budget, and release an answer only when the evidence chain meets a specified rule. A matched query-expansion control shares the retrieval and answer constraints but does not identify or substitute a premise.

Method and Prompt Contracts

  • Elenchus returns an atomic unresolved premise, its decisiveness, a discriminating query, an expected discriminator, and a stop decision for each round.Source: Method and Prompt Contracts · p. 7
  • The controller enforces two queries and at most three retrieved pages per query, while a shared answer call returns a short answer, cited IDs, and residual uncertainty.Source: Method and Prompt Contracts · p. 7
  • The release guard requires a signed final record or a signed bridge plus signed storage record, and otherwise returns abstention without reading evaluator labels or oracle fields.Source: Method and Prompt Contracts · p. 7
  • The matched query-expansion control receives the same question and documents and uses the same two-query/top-three budget and answer prompt, but generates an ordinary relevance-oriented query.Source: Method and Prompt Contracts · p. 7
  • Structured JSON calls enforce each role’s required fields; malformed or missing fields produce an error, and the finalizer is identical across active policies.Source: Method and Prompt Contracts · p. 7

Counterfactual Active-Evidence Generator

The counterfactual benchmark uses fictional, disjoint worlds with coordinated false-answer pages and searchable signed evidence. It balances one-hop and two-hop cases in a reproducible lexical-search environment that is not presented as a natural search-engine model.

Counterfactual Active-Evidence Generator

Evaluation Protocol

The evaluation protocol materials identify a configuration table for the 48-world evaluation and specify that errors remain terminal observations.

Evaluation Protocol

Detailed Results

The matched comparison favors premise-directed inquiry, with the observed difference localized to two-hop evidence acquisition. Traces connect wins to refining a query with a newly resolved premise, while the method establishes no cost advantage.

Detailed Results

  • 14.6 points: the comparison shows a 7/48 paired difference, 8 wins, 1 loss, 39 ties, higher decisive recall, lower observed unsafe count, and p < .05.The primary evaluation table covers 48 worlds.Source: Table 3 · p. 8, Detailed Results · p. 8
  • Both strata have identical correctness and decisive-recall counts; the difference localizes to two-hop evidence acquisition.The hop-stratified result identifies where the policies differ.Source: Table 4 · p. 8, Detailed Results · p. 8
  • In seven wins, newly resolved archive/rule premises refine storage queries; in loss AE37, expansion broadens to “storage register” and succeeds.Expansion repeats broad person/archive/vault terms and retrieves coordinated poison replicas in the seven-win traces.Source: Detailed Results · p. 8
  • The method establishes no cost advantage; the method-call table tracks input and output tokens and latency.The supplied table headings list method calls, input tokens, output tokens, and latency.Source: Detailed Results · p. 8

Training-Free Official Baselines

In the active-rescue setting, released RobustRAG and TrustRAG variants do not acquire decisive records. The paper also documents versioned artifacts and experimental adaptations.

Training-Free Official Baselines

  • Neither baseline is trained, fine-tuned, calibrated, or threshold-selected on the benchmark; TrustRAG retains its released pipeline with CPU inference and a shared generator wrapper.RobustRAG uses its released keyword-aggregation variant; its secure-decoding variant is unavailable through the API because token logits are not exposed.Source: Training-Free Official Baselines · p. 8
  • 0/48 safe answers and 48/48 unsafe answers for RobustRAG; TrustRAG yields 0/48 safe, 24/48 unsafe, and abstains on the other 24.Both complete all 48 terminals with zero provider or parser errors, zero search actions, and zero decisive-record recall; these results establish the fixed-context acquisition boundary.Source: Training-Free Official Baselines · p. 8

Reproducibility Checklist

  • The reproducibility checklist calls for releasing versioned world-generation, search, runtime, evaluator-label, and index artifacts, alongside prompts and provider metadata.It also specifies release of policy terminals, provider-call records, and scoring and trace scripts.Source: Reproducibility Checklist · p. 8
  • The checklist requires identifying baseline versions and adaptations, and disclosing that the external route lacks a public model snapshot and the lexical benchmark is not a natural-Web estimate.It says to retain failures rather than deleting them.Source: Reproducibility Checklist · p. 8
Loading 2609.35773v1…