PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Pick a Winner: Knowledge Conflicts Need a Prior, Not a Rule

Minh Pham Van, Cong Tran

arXiv:2610.36169v1cs.AIcs.CL

TL;DR

Fixed conflict-resolution rules can favor the wrong source, and symmetric divergence does not say which source is right. The paper develops a system that combines source-specific evidence with a context prior; on GRACE it beats the best baseline across tested context mixtures, while gains vanish where the model lacks the knowledge.

  • Problem

    Existing conflict-resolution methods choose a fixed source preference, while symmetric disagreement signals do not identify which source is right.

    Source: §1 · p. 2

  • Method

    The system elicits model knowledge as notes, gives notes calibrated confidence, uses a two-voiced reader, and selects or abstains under an explicit context prior.

    Source: §1 · p. 2, §6 · p. 6

  • Results

    The system is above the best baseline across GRACE context mixtures, by up to 25.8 points at f = 0.25 for Qwen.

    Source: §7.1 · p. 8

  • Takeaways & Limitations

    The reported gains vanish where the model does not know the answer, consistent with the paper's prediction.

    Source: §1 · p. 3

    The selector’s calibration follows its training question and context types, so a new type may require retraining.

    Source: §8 · p. 9

Abstract

from arXiv · show

A retrieval-augmented language model answers from two sources, the retrieved context and its own knowledge, and the two disagree whenever the context is incomplete or wrong. Existing methods settle the disagreement with a rule fixed at design time: follow the context, amplify it in proportion to how far it moves the model, or trust the model's memory. Each rule is confidently wrong in some deployment. In a decision-theoretic model of two sources, the score of any fixed rule is linear in the share of reliable contexts while the best attainable score is convex, so every fixed rule is beaten wherever it is not exactly optimal; and a symmetric divergence between the two sources, the signal adaptive decoding uses, carries no information about which one is right. Resolving a conflict therefore needs two inputs that no existing method keeps: evidence about each source separately, and a prior on how often the context is right. We test these claims on GRACE, a benchmark that intervenes on the knowledge-graph context of a fixed question, so that what the model knows, what the context says and what is true are measured separately. There, every baseline from four families is strong on right contexts or on wrong ones, never on both. We then build the two inputs: the model states its knowledge as notes, a small policy voices each note with a calibrated confidence, a reader answers in two voices, and a selector commits or abstains under a prior $\pi$ applied as an exact prior shift. With $\pi$ set to the true share of right contexts, the system is above the best baseline at every mixture of GRACE contexts, by up to 26 points. Its overall score is 45.3 and 35.0 (Qwen3.5-35B-A3B, Llama-3.1-8B), against 29.4 and 27.5 for the best baseline. On preregistered ClashEval it beats same-backbone CK-PLUG and TCR by 12 and 26 points. Where the model lacks the knowledge, the gains vanish, as the theory predicts.

1 Introduction

Retrieved context and model knowledge can each be wrong, yet existing conflict-resolution methods commit to a preference before seeing the question. The paper argues that resolving conflicts requires evidence about each source and a prior on context reliability, then tests this diagnosis and builds a system around those inputs.

1 Introduction

  • Retrieved information can be unreliable, while model knowledge is also uneven, being more reliable for popular entities than for the long tail.The introduction illustrates unreliable retrieval with search-engine answers advising users to add glue to pizza sauce and eat a rock.Source: §1 · p. 1
  • Fixed-direction methods are each strong on right or wrong contexts, while a symmetric divergence detects disagreement but does not reveal which source is right.The paper frames resolution as requiring separate evidence about the sources and a prior on context reliability.Source: §1 · p. 2
  • GRACE varies the knowledge-graph context while holding the question fixed, enabling tests of the two-source theory and its predictions.The paper reports that the theory's predictions hold for thirteen methods and two backbones.Source: §1 · p. 2
  • The method combines calibrated knowledge notes, a two-voiced reader, and a selector using the context prior π.Both controls are described as log-odds shifts with a stated meaning.Source: §1 · p. 2
  • The proposed system supplies model-side evidence and a context prior; it beats the best baseline across GRACE context mixtures and same-backbone controls on a preregistered external test.Its gains disappear where the model lacks the knowledge, as predicted.Source: §1 · pp. 2–3

2 Related Work

Prior work studies knowledge conflicts, methods that privilege one source, approaches that use both sources, and decision-theoretic tools. The paper distinguishes its contribution by separating what the model knows from what it reads and by combining source-specific confidence with an explicit context prior.

Knowledge conflicts

  • Earlier conflict benchmarks generally score one answer per question; none fixes the question and varies only context to separate what the model knows from what it reads.Prior studies examined answer substitution, model preference for external evidence, entity familiarity, recall and copying, and whether models represent both candidate answers.Source: §2 · p. 3

Resolving conflicts with a fixed direction

  • Existing resolution methods use a fixed direction or a fixed procedure for choosing between context and memory, and return one answer.Families include context-following, contrastive or trained preferences, rationales, sample voting, and memory generation.Source: §2 · p. 3

Using both sources

  • Other work presents contextual and parametric answers side by side or estimates when one answer can be trusted; this paper adds learned confidence per model fact and an explicit context prior.The paper says these distinctions are summarized in Table 4 in Section B.Source: §2 · p. 3

Decision-theoretic foundations

  • The decision-theoretic framing builds on reject rules, proper scoring, prior-shift correction, and opinion pooling, applying their implications to knowledge conflicts.Source: §2 · p. 3

3 Two Sources, One Decision

The paper models context and model knowledge as two potentially unreliable sources, with a decision to answer or abstain under a deployment prior. It shows why fixed rules and symmetric disagreement signals cannot generally resolve conflicts, motivating calibrated candidate probabilities, asymmetric evidence, and the prior.

Setting

  • The system observes candidate answers, evidence, and context reliability prior π, then either commits or abstains, with correct-minus-wrong score C−W.Context and model reliability indicate whether each source's answer matches the real-world answer.Source: §3 · p. 3

When to commit

  • The optimal commit rule selects the candidate with the highest calibrated probability of being correct only when that probability exceeds 1/2; otherwise it abstains.The paper identifies this as Chow’s reject rule.Source: §3 · p. 3

Fixed rules are lines

  • A decision maker that knows the context prior can achieve a convex best-score curve that dominates fixed rules whenever a rule is not optimal at that prior.Each fixed policy's score is a straight line between its performance on right and wrong contexts.Source: §3 · p. 4

What the theory asks for

  • Reaching the best attainable score requires calibrated probabilities for both candidates and the deployment prior, rather than a single fixed answer.The theory frames these as the quantities needed to recover what a fixed rule discards.Source: §3 · p. 4

A divergence detects a conflict but cannot resolve it

  • A symmetric statistic such as Jensen–Shannon divergence cannot indicate which source is right, so increasing context preference with divergence can worsen scores on wrong contexts.The paper concludes that divergence signals a conflict, not how to resolve it [cf. Genest and Zidek, 1986].Source: §3 · p. 4

4 GRACE: Intervening on the Context

GRACE tests conflict resolution by changing the knowledge-graph context while keeping the question, real answer, and model answer fixed. Its context states and mixed deployments support comparisons of systems across context reliability conditions.

Construction

  • GRACE varies each question across clean, broken, wrong_branch, and competing knowledge-graph contexts, with the model answer and real-world answer held fixed.The states respectively preserve the correct subgraph, remove a gold edge, replace gold edges with wrong-answer copies, or add wrong copies alongside gold edges.Source: §4 · pp. 4–5

Data

  • The benchmark uses LC-QuAD 2.0 questions and Wikidata subgraphs; experiments draw on training and development data plus GRACE-QT questions from four unseen types.The training pool has 1,000 questions, three dev slices have 225 questions each, and GRACE-QT has 1,000 questions.Source: §4 · pp. 4–5

Scoring

  • Figure 2 compares C−W by context state and method, showing context-reading methods scoring −36 to −74 on wrong_branch contexts while context-ignoring methods lose the clean column.The figure description says the proposed method is the only row without a strongly negative cell; CK-PLUG, TCR, CAD, and AdaCAD are run on Llama only.Source: Figure 2 · p. 5

Construction

  • Mixing clean and wrong_branch versions of the same questions creates deployments with π equal to the clean share, satisfying label shift exactly.Source: §4 · p. 5

5 Fixed Rules on GRACE: The Predictions Hold

GRACE results show that fixed methods excel on one context condition but not both. AdaCAD’s divergence detects a wrong context without identifying whether the model is right, so amplifying the context can turn abstentions into wrong answers.

5 Fixed Rules on GRACE: The Predictions Hold

  • Fixed methods had opposite strengths: context-following or implicit-choice methods scored 64–72 C−W on clean contexts but −57 to −74 when the answer edge was wrong, while memory-first methods avoided much of that loss but gave up clean-context performance.No method was strong on both clean and wrong_branch contexts; the closed-book model was right 43.4% and wrong 44.6% of the time on Qwen.Source: §5 · p. 5
  • Figure 3 shows that AdaCAD’s divergence detects a wrong context but not whether the model is right; where the model is right and the context wrong, context amplification turns abstentions into wrong answers.Panel (c) illustrates the consequence of using the divergence to amplify context in that condition.Source: Figure 3 · p. 6
  • AdaCAD’s weight rose by 0.069 on wrong_branch versus clean questions, but within wrong_branch it differed by only 0.008 depending on whether the model knew the answer.Across AdaCAD weight thirds, wrong_branch scores fell from −38.2 to −67.1 to −79.1, a 40.9-point fall; the plain reader showed no comparable trend.Source: §5 · pp. 5–6
  • On 267 wrong_branch records where the model knew the answer, AdaCAD was wrong on 69.7% while the plain reader was wrong on 19.9%; ours was right on 32.2% and wrong on 16.5%.All three methods were right on only 11–13% of these records; amplifying the context turns abstentions into wrong answers without recovering the model’s knowledge.Source: §5 · p. 6

6 Supplying the Evidence and the Prior

The method separates evidence about model knowledge from the retrieved context, calibrates how that evidence is voiced, and selects an answer using a stated prior on context reliability. It keeps the context unedited and can commit or abstain.

6 Supplying the Evidence and the Prior

  • Notes elicit the model’s knowledge before it sees the context, recording context-fact-linked triples with a stance; repeated runs provide support counts, and the notes improve accuracy on wrong contexts.The same questions were answered correctly on 67.0% (Qwen) and 32.2% (Llama) of wrong contexts with notes.Source: §6 · p. 6
  • A two-layer policy predicts each note’s probability of being true and useful, then chooses spoken confidence to maximize an asymmetric Brier reward.Utility weights distinguish useful notes from peripheral ones and false notes; the resulting confidence is voiced as a verbal hedge.Source: §6 · pp. 6–7
  • The two-voiced reader preserves separate context and model answers, while a selector scores four answer candidates and commits only when the best shifted probability exceeds 1/2.The selector shifts context-sourced candidates from its training prior π0 to the user-stated π; under label shift, this is an exact posterior shift and yields the Bayes rule.Source: §6 · pp. 7–8
  • Both learned controls are additive log-odds shifts: note confidence encodes a ratio of utilities, while the context shift encodes a ratio of priors.Without label shift, the prior shift is not exact, though its direction remains monotone in π.Source: §6 · p. 8

7 Results

With the context prior matched to each GRACE mixture, the system exceeds the best fixed baseline across mixtures, with the advantage narrowing as contexts become reliably correct. External tests show gains against several controls, but also expose calibration limits and settings where the model has little knowledge to contribute.

7.1 The system is on the envelope

  • At the trained prior π = 0.25, the system scored 45.3 C−W with Qwen and 35.0 with Llama; on wrong_branch, it scored 17.9 and −4.1 versus −63.9 and −19.1 for the plain reader.Every difference in Table 1 excludes zero.Source: §7.1 · p. 8
  • On GRACE, selector calibration error was 0.029 (Qwen) and 0.054 (Llama) at π = 0.25 and rose as π moved away; abstaining below a 1/2 probability beat always answering by 3.0–5.9 points on GRACE and 1.3–34.6 on ClashEval.The cautious rule improved scores on both datasets and backbones.Source: §7.1 · p. 8, Figure 5 · p. 9
  • Removing notes cost 21.7 points (Qwen) and 14.4 (Llama); uniform confident voicing cost 7.1 and 9.3, sampling 3.3–3.7, and removing the selector 8.9–9.7.The ablation results identify losses associated with removing each component.Source: §7.1 · p. 8
  • 25.8 points (Qwen) and 11.7 points (Llama) over the best baseline at f = 0.25; across f = 0.1, 0.25, 0.5, gains were 25.7, 25.8, 21.3 and 10.7, 11.7, 4.4 points, respectively.The advantage was significant for f ≤ 0.75 (Qwen) and f ≤ 0.5 (Llama), never significantly below baseline, and vanished as f approached 1; Figure 5 compares the system with the best baseline at each mixture.Source: §7.1 · p. 8, Figure 5 · p. 9

7.2 A preregistered external test

  • On preregistered ClashEval, the Llama system scored −29.1 and beat CK-PLUG by 12.2 points and TCR by 25.6; with Qwen it beat ten of eleven baselines by 13–33 points and tied Astute RAG.GRACE development results are described as an upper bound on expected performance elsewhere.Source: §7.2 · p. 9
  • On text passages, the selector was miscalibrated: answers it assigned 0.49 probability were right 18% of the time, although the prior’s direction transferred.The test used a frozen system, and the paper notes that GRACE development results are an upper bound on expectations elsewhere.Source: §7.2 · p. 9
  • When the model was wrong on 60% of Housing Statute QA questions, the system scored 2.0 against 11.4 for the plain reader; with no knowledge to merge, gains vanished, while omitting sampled runs reduced runtime and score.Without five sampled runs, runtime was 3.7 seconds (Qwen) and 5.3 seconds (Llama), with scores of 41.6 and 31.7.Source: §7.2 · p. 9

8 Limitations and Conclusion

The paper concludes that fixed rules cannot match the convex optimum across context-reliability mixtures, while calibrated evidence about model knowledge plus a context prior outperforms fixed methods on GRACE and same-backbone controls on a preregistered test. Its selector’s calibration depends on the question and context types used in training.

8 Limitations and Conclusion

  • The selector is a small learned classifier whose calibration follows its training question and context types, so a new type may require retraining.The conclusion also limits the method’s gains to cases where the model has knowledge to merge with the context.Source: §8 · p. 9

A Proofs

The proofs characterize fixed-policy scores, the best attainable score, and how confidence and deployment priors enter decisions. GRACE construction and design comparisons specify the benchmark and the properties that distinguish the proposed method.

A Proofs

  • The best action commits to the most probable answer only when its probability exceeds 1/2; otherwise, abstention is optimal.Source: A Proofs · p. 14
  • A fixed policy’s score is affine in the share of right contexts, while the pointwise-optimal score is convex and at least as high.Source: A Proofs · p. 14
  • A symmetric divergence assigns the same weight when the two source distributions are exchanged, so it cannot identify which source is correct.Source: A Proofs · p. 14
  • For conflicting sources, a rule’s expected score depends on the reliability of each source; shifting probability toward a less reliable source can reduce the score.Source: A Proofs · pp. 14–15
  • The optimal confidence for a note is c*=pU/(pU+(1−p)λ), whose log-odds shift by a utility-to-loss term independent of note truth probability.Source: A Proofs · p. 15
  • Changing the deployment prior from π0 to π adds logit(π)−logit(π0) to posterior log-odds when the likelihood ratio is prior-independent.Source: A Proofs · p. 15

B GRACE: construction, scoring and checks

B.1 Design properties of prior work

  • The method uniquely relates sources fact by fact, learns confidence for model knowledge, and exposes context trust as a probabilistic prior without training the reader.Source: §B.1 · p. 16

B.2 Does the model use knowledge or benchmark cues?

The analysis finds that model notes can reveal useful partial knowledge, but also contain false statements, while context cues and methods that favor one source show uneven performance across conditions.

B.2 Does the model use knowledge or benchmark cues?

  • Raw notes identify the removed edge in 50.7% of broken records and flag injected facts as wrong in 78.4% of wrong_branch and 79.7% of competing records.Notes also give the correct value in 37.3% of wrong_branch and 23.9% of competing records, but can state false notes.Source: §B.2 · p. 16
  • When notes state the removed edge, the model’s closed-book answer is correct in 76.1% of records, versus 62.7% overall on that dev slice.Correctness is 75.9% when a note gives the right value in wrong_branch and 76.5% in competing records.Source: §B.2 · p. 16
  • The thinking-enabled elicitation diagnostic averaged about 4,500 generated tokens per record versus about 300 without thinking, and did not change score.Twenty-eight of 120 diagnostic runs exceeded the 6,000-token limit.Source: §B.2 · p. 16
  • Removing the 202 records containing the default “George Washington” distractor changes overall scores only from 45.3 to 46.1 for Qwen and 35.0 to 35.4 for Llama.The authors note that the method and selector are trained on GRACE-derived data, but the baseline without GRACE training shows a similar artifact-record difference.Source: §B.2 · pp. 16–17

C Baselines and prompts

  • TCR and CK-PLUG run only on Llama in their official code, so direct comparisons with them and the logit-based methods use that backbone.TCR was retrained with its own recipe and data rather than GRACE data; two methods were not run because of resource or inference-server requirements.Source: C Baselines and prompts · pp. 18–19

C.1 Prompts

  • The pipeline elicits up to eight knowledge notes, self-verifies each note, and supplies a separate context reading for a two-voiced answer.Note confidence is expressed through calibrated hedges, with thresholds specified for the confidence values.Source: §C.1 · pp. 19–20

Single-source reference arms

  • Closed-book answering is near zero on C−W, while the context-only reader scores 71.1 on clean Qwen contexts and −66.7 on wrong_branch.The findings report that the context-only arm follows the graph even when it leads to the wrong entity.Source: Single-source reference arms · p. 21

Findings

  • No baseline is strong on both clean and wrong_branch contexts; AdaCAD scores 66.8 on clean versus 55.7 for the plain reader, but −61.5 on wrong_branch versus −19.1.The broader comparison finds context-following methods collapse on wrong_branch, while memory-first methods give up much of the clean score.Source: Findings · p. 21

D.2 The divergence weight, all records and all tokens

The divergence weight responds to wrong contexts but not to which source is right, and its amplification can turn abstentions into wrong answers. The following sections motivate eliciting and calibrating explicit model-knowledge notes.

D.2 The divergence weight, all records and all tokens

  • 32.2% of the 267 wrong_branch records where the model knows the answer are correct with the proposed system, 19.5 points above AdaCAD, with no more wrong answers than the plain reader.The table compares outcomes on records where the model is right and the context is wrong.Source: Table 7 · p. 22, §D.2 · p. 23
  • AdaCAD’s α is higher on wrong_branch than clean contexts by 0.051, but its wrong-context values are nearly identical whether the model answers correctly or wrongly: 0.345 versus 0.343.The wrong-context detection has AUC 0.60; the within-condition difference is 0.002.Source: §D.2 · p. 22
  • Across wrong_branch α thirds, AdaCAD’s score falls from −38.2 to −79.1 C−W, while the plain reader’s difference is −6.7 points.Among records the model can answer, AdaCAD converts abstentions into wrong answers and recovers little of the model’s knowledge.Source: §D.2 · pp. 22–23
  • The analysis measures one method’s weight at the first generated token; the score analysis uses Llama, and AdaCAD itself was not run with Qwen.The premise was checked with Qwen using first-token divergences, but its values are lower bounds based on returned top-token probabilities.Source: §D.2 · p. 23

E.1 Three findings that fixed the design

  • Deleting model-rejected context facts removed relations readers needed and performed no better than random deletion; attaching confidence to context facts lowered scores by 1 to 4 points.A prompt merely permitting model knowledge also underperformed on wrong contexts, motivating explicit knowledge notes before answering.Source: §E.1 · p. 24
  • With explicit notes, questions answered correctly without context reach 67.0% and 32.2% correct on wrong_branch for Qwen and Llama, respectively.The plain reader reaches 8.2% and 12.0% on those same question groups.Source: §E.1 · p. 24

E.2 Knowledge notes and their labels

  • Notes are elicited across six runs, merged by normalized subject and object, and ranked by support; unverifiable notes are dropped and some Wikidata-derived false labels are down-weighted.The highest-support notes are retained for subsequent use.Source: §E.2 · p. 24

E.3 Note policy

The system varies its decisions with the context-trust prior, while its selector uses candidate and note evidence to estimate correctness. Its performance changes across context states, and the note policy transfers poorly across backbones without retraining.

E.3 Note policy

  • The note policy’s inputs combine note embeddings with 15 scalars, including self-verification, stance, context/question overlap, support, similarity, and whether the note came from a greedy run.Training uses 1,000 LC-QuAD questions across four states, yielding 4,000 records and about 24,000 notes.Source: §E.3 · pp. 24–25
  • Without retraining, the Qwen note policy’s transfer to Llama raises confident false notes from 2.0–2.3% to 10.1–10.6% and worsens calibration.Transfer still reduces confident errors from about 27% to about 10%, but the policy is calibrated per backbone.Source: §E.3 · p. 25

E.4 Selector

  • The selector estimates each candidate’s correctness from its source, relation, similarities to both sources and the question, and statistics of supporting notes, then commits only above 0.5 in cautious mode.The 0.5 threshold is the break-even point when correct answers score +1 and wrong answers −1; otherwise the selector abstains.Source: §E.4 · p. 25
  • The selector’s prior shift is exact for context-sourced candidates under label shift, but leaves knowledge-candidate probabilities unchanged and therefore errs toward retaining them.If label shift fails, the shift remains monotone in π, but the adjusted probabilities are no longer exact posteriors.Source: §E.4 · pp. 25–26

F.1 Per-state scores at every prior

  • At π = 0.1–0.5, overall scores remain 33.3–35.0 for Llama and 43.4–45.3 for Qwen; sweeping π moves the system between context-merging and context-following behavior.At π = 0.99, Qwen matches the plain reader on wrong_branch contexts (−63.7 against −63.9) while remaining better on clean and broken contexts.Source: §F.1 · p. 26, Figure 7 · p. 27

F.2 Ablations

Ablations identify stated model knowledge, calibrated note confidence, sampled elicitation, and selection as distinct contributors. Further tests show where gains persist, how outputs flag conflicts, and what evaluation boundaries remain.

F.2 Ablations

  • Without notes, the system loses its largest contribution, especially on wrong or contradictory contexts; confidence, sampled support, and the selector also improve performance.The ablations report losses of 21.7 and 14.4 points without notes, 7.1 and 9.3 without calibrated confidence, 3.3–3.7 without sampled elicitation, and 8.9–9.7 without the selector.Source: §F.2 · pp. 27–28, Figure 8 · p. 28

F.3 Are the spoken hedges calibrated?

  • The reader’s spoken hedge certainty tracks answer accuracy, but it often omits a hedge: 67% of Qwen answers and 29% of Llama answers contain none.The numeric note confidence has an ECE of about 0.01–0.05; the reader’s hedge words have no numeric value.Source: §F.3 · p. 28, Figure 9 · p. 29

F.4 Questions that neither source answers alone

  • On records where both single-source arms fail, ours has correct-minus-wrong scores of +1.4 for Llama and −1.8 for Qwen, while all baselines are lower.For Qwen on wrong_branch contexts, ours answers correctly on 9.9% of records, versus at most 1.5% for baselines except Astute and GenRead.Source: §F.4 · p. 29, Figure 10 · p. 29

F.5 Other question types (GRACE-QT)

  • On GRACE-QT, the system is statistically level with the best baseline on both backbones, but where model knowledge is poor it neither harms nor improves performance.Boolean questions are excluded because the prompts request an entity rather than a yes/no answer.Source: §F.5 · p. 30

F.6 Does the answer warn the reader?

  • On wrong_branch records, the system gives a wrong, unwarned answer on 11.1% of Qwen and 14.1% of Llama records, versus 69.2% and 26.4% for the plain reader.A conflict flag also appears on 12.7% of Qwen and 14.2% of Llama clean records, where the context is right.Source: §F.6 · p. 30, Figure 11 · p. 31

G.1 Preregistration, frozen artefacts and deviations

  • The preregistered scorer retained only the first segment of list-formatted answers, so whole-answer scores are reported separately as post hoc.On ClashEval with Llama, the two scoring approaches differ substantially for the plain reader and closed-book arm.Source: G Preregistered external tests · p. 31, §G.1 · p. 32

G Preregistered external tests

  • The first lockbox test’s WebQSP and CWQ figures are excluded as evidence because the matching rule was later found to be lenient.Both datasets’ validation splits had also been used for early exploration.Source: §G.1 · p. 32

G.2 Task coverage of the trained selector: lockbox 2, selector v3 and ClashEval

The selector’s learned coverage is specific to the question types and context distributions represented in training. Selector v3 broadens question-type coverage, while ClashEval shows that context-distribution mismatch can still impair calibration.

Task coverage of the trained selector

  • The selector is the system component that learns which candidate to trust, so its coverage is limited to the question types and context distributions represented in training.The elicitation prompts, voicing policy, and two-voiced reader are frozen or general by construction.Source: §G.2 · pp. 32, 35

Lockbox 2 and selector v3

  • On Mintaka, v2 chose a note object on 493 questions and was correct on 14.6% of them; removing that candidate type without retraining raises the system score from 23.9 to 46.5.Note objects can be intermediate entities for comparison, counting, superlative, and multi-hop questions.Source: §G.2 · p. 33

Selector v3

  • V3 broadens selector training to include GRACE-QT question types and restricts candidates to answers stated by a reader, including the plain reader’s answer.Its prior is applied through Equation (4), rather than supplied as a learned input.Source: §G.2 · p. 34

Selector v3 and external tests

  • On GRACE-QT, v3 scores 2.7 for Qwen and 1.3 for Llama, gaining 11.2 and 7.2 points over v2 and matching the best baseline on both backbones.On ClashEval, v3 gains 5.8 points for Qwen and 8.6 for Llama over v2 applied to the same frozen reader outputs.Source: §G.2 · p. 34

ClashEval

  • On ClashEval with Llama, v3 trails GenRead because GRACE-trained probabilities are miscalibrated for prose contexts and graded perturbations; recalibration requires labelled in-domain data.The selector assigns mean probability 0.49 to answers that are right 18% of the time, and setting π to 0.114 lowers calibration error only from 0.35 to 0.31.Source: §G.2 · p. 35

H Latency and cost per stage

The system is more accurate than the alternatives on both backbones at a cost comparable to other multi-call methods. Sampled elicitation is its largest single time cost and improves accuracy.

H Latency and cost per stage

  • The system takes 5.2 seconds per question with Qwen and 6.3 with Llama, between the plain reader’s 0.07 and 0.27 seconds and several multi-call baselines.It is faster than Astute RAG, FaithfulRAG, and TruthfulRAG with Llama, and between USC and InstructRAG-ICL with Qwen.Source: H Latency and cost per stage · pp. 35–36
  • Dropping five sampled elicitation runs saves 1.5 seconds for Qwen and 1.0 for Llama, while reducing scores by 3.7 and 3.3 points.The full system is the most accurate method on both backbones in the accuracy-time comparison.Source: Figure 12 · p. 35, H Latency and cost per stage · p. 36
  • Self-verification uses 7–8 parallel calls that generate one token each and cost about 0.25 seconds.The timing measurements use 40 GRACE development questions, one request at a time on an idle GPU.Source: H Latency and cost per stage · pp. 35–36

I Measurement lessons

This section examines how scoring choices, context reliability, and the model’s knowledge affect evaluation and deployment across ClashEval, Housing Statute QA, and GRACE. It also reports selector ablations and limits, audits for GRACE-specific cues, and lessons from earlier scoring and selector approaches.

I Measurement lessons

  • Strict matching reduced the Qwen GRACE score from 48.9 to 19.2 because substring labels rewarded long, multi-entity answers; shared extraction and strict training labels were the fix.The system had 435 answers longer than six words, versus 32 for the plain reader.Source: I Measurement lessons · p. 36
  • Llama’s first-segment scoring changed its raw GRACE dev score from 27.5 under whole-answer scoring to 14.0, while 77.6% of its GRACE dev answers contained the separator.On ClashEval, 95.6% of Llama answers contained the separator, compared with 1.2% and 17.1% for Qwen on ClashEval and GRACE dev.Source: I Measurement lessons · p. 36
  • Removing all 202 records involving the injected wrong answer “George Washington” raised overall scores from 45.3 to 46.1 (Qwen) and 35.0 to 35.4 (Llama), narrowing plain-reader margins from 22.7 to 20.3 and 7.5 to 7.0, respectively.Source: I Measurement lessons · p. 37

S1 Inside the selector

  • Removing note-derived features lowered GRACE scores by 5.5 (Qwen) and 4.2 (Llama), and lowered Qwen GRACE-QT by 6.9, making them the only ablated features reported to matter on both backbones.Shortened candidates and strict labels particularly affected Llama, whose answers are longer and list-formatted.Source: S1 Inside the selector · p. 38

S2 Where the method helps and where it does not

  • The method helps when the model knows the domain and context may be wrong, but stale model knowledge or little domain knowledge calls for high π and context-following behavior.It is not presented as a defence against targeted attacks; when the context is the newer, authoritative source, the authors recommend π of 0.9 or higher.Source: S2 Where the method helps and where it… · pp. 39, 41
  • On GRACE broken contexts, gains over the plain reader are +6.9 (Qwen) and +5.9 (Llama); on wrong_branch, unwarned wrong answers fall from 69.2% to 11.1% and 26.4% to 14.1%.On GRACE competing contexts, the reported gains are +8.1 (Qwen) and +7.1 (Llama).Source: S2 Where the method helps and where it… · pp. 39–40
  • The operator must set π to reflect context reliability: 0.25 for unverified contexts, 0.5 for mostly reliable ones, and 0.9 or more for curated contexts.Unlabelled estimation of π remains future work. The method can cost about 6 points on clean Qwen contexts at π = 0.25 (64.3 against 70.5), and its selector does not provide a low-risk operating point once contexts are perturbed; the authors do not claim medical or legal decision reliability.Source: S2 Where the method helps and where it… · pp. 41–42

S3 ClashEval and Housing Statute QA: full tables, risk and coverage

  • 11.1 points over the plain reader, 12.2 over CK-PLUG, and 25.6 over TCR: the registered Llama system beats both same-backbone controls on ClashEval.It beats nine document-reading methods by 11–29 points and ties Astute RAG (+3.7 [−0.4, +7.5]); GenRead scores higher overall but never reads the document, and the preregistered H-v3b claim is not confirmed.Source: S3 ClashEval and Housing Statute QA:… · p. 42
  • 13 to 33 points: Qwen’s registered system beats ten of eleven ClashEval baselines, ties Astute RAG on the total, and leads it when at least a quarter of documents are unmodified.At f = 0.25, the scores are −6.3 against −9.9; at f = 0.5, 23.5 against 12.6; at f = 0.75, 53.3 against 35.2; and at f = 0.9, 71.2 against 48.8. Its performance is at least as good as the plain reader at every mixture.Source: S3 ClashEval and Housing Statute QA:… · p. 44, Figure S1 · p. 45

S3.1 Risk and coverage

  • On perturbed ClashEval documents, Qwen answers 79% of questions with 73% of answers wrong, versus the plain reader’s 92% and 89%; Llama’s answers are not more often right.For Llama on perturbed documents, the system answers 51% with 88% wrong, versus the plain reader’s 80% and 84%; raising the selector threshold does not buy a low-risk operating point for either backbone.Source: S3 ClashEval and Housing Statute QA:… · p. 46, Figure S2 · p. 47

S4 Five-variant multiple-choice GRACE and deployed GraphRAG-family systems

  • Acc3 falls by 22–35 points on broken graphs and 46–55 points on swapped graphs, while distractors change it by at most 10 points; graph systems are no more robust than naive RAG.On questions the model knows, GraphRAG selects the injected answer for 38% under swapping, and LightRAG does so for 46%. Routing keeps 99% of known questions but scores 9 points below naive RAG on unknown questions with clean graphs (57 against 66).Source: S4 Five-variant multiple-choice GRACE… · pp. 47–48

S5 Selector v4 and earlier directions that did not work

  • 35.0 and 45.3: selector v4 scores on GRACE dev with Llama and Qwen, respectively, at π = 0.25; its results do not have an untouched test.The candidate-extraction change matters for Llama at high π, and the main-text results use v3 because the defect v4 fixes was first seen on ClashEval.Source: S5 Selector v4 and earlier directions… · p. 48

S5.1 Earlier directions that did not work

  • 50.3% against 51.1%: graph-edit reinforcement learning underperforms applying every edit, while self-verification alone scores 50.4%, equal to random deletion at 50.6%.Deleting a false fact without replacement can remove the relation and cause abstention; a verify-then-repair approach instead gains +4.8 to +6.8 Acc3 over the plain reader on two fresh slices, though stale knowledge can overwrite newer facts.Source: S5.1 Earlier directions that did not… · pp. 48–49
  • C−W stayed near 27 for knowledge-signal gates, while the attention selector scored 37.6 against 48.2 for gradient boosting.The best single signal reached AUC 0.78, but a gate also hides knowledge; confidence tags gave low scores to 14% of correct facts, mixing unfamiliarity with falsehood.Source: S5.1 Earlier directions that did not… · p. 49
  • With 3,600 records, the transformer failed to learn which note supported which candidate, and its softmax made the cautious mode abstain too often.Source: S5.1 Earlier directions that did not… · p. 49
  • Using π as a learned selector input had almost no effect outside the training distribution, so it was replaced by the analytic prior shift of Eq. (4).Source: S5.1 Earlier directions that did not… · p. 49
  • 35.4% of CWQ answers were correct without context, versus 2.2%–8.7% with context when the context lacked the answer; the effect also appeared on GRACE dev under protocol v3.The context-conditioned rates varied by prompt; this was observed with the earlier scoring rule.Source: S5.1 Earlier directions that did not… · p. 49

Related papers

Loading 2610.36169v1…