PaperlayerView exact revision on arXiv

Source-linked AI summary

Accept or Repair: Preserving Passage-Supported Answers in Contrastive Decoding

Jungseob Lee, Seungyoon Lee, Hyeonseok Moon, Seongtae Hong, Suhyune Son, Sugyeong Eo, Minhyuk Kim, Chanjun Park, Jaehyung Seo

arXiv:2610.29727v1cs.CL

TL;DR

LLMs can prioritize training-learned knowledge over a conflicting passage, and contrastive decoding can alter answers that the passage supports. AoR accepts supported sampled answers and tries up to two contrastive repairs for unsupported ones, with a memory veto. It improves average scores over plain sampling on both counterfactual and ordinary passages.

  • Problem

    LLMs can prioritize training-learned knowledge over conflicting passage information, while contrastive decoding can also change answers supported by the passage.

    Source: §1 · p. 1

  • Method

    AoR keeps a sampled answer when the passage supports it; otherwise, it tries up to two contrastive repairs and vetoes a repair repeating the closed-book answer when the passage favors another.

    Source: §1 · p. 2

  • Results

    AoR improves average scores over plain sampling on counterfactual and ordinary passages and outperforms contrastive and prompting baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Replacing only unsupported initial answers improves the average score of every decoder in the evaluation.

    Source: §7 · p. 8

    AoR’s word-matching support test targets short answers and single sentences; long-form or abstractive answers require tests that handle paraphrasing and information combined across the passage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models often answer from knowledge learned during training, even when a provided passage contradicts that knowledge. To address this problem, contrastive decoding contrasts the predictions under the full and a modified prompt at every token, but it also alters correct answers that the passage supports. We show that checking answers against the passage removes this trade-off. Based on this observation, we propose Accept-or-Repair (AoR), a training-free procedure. AoR keeps a sampled answer if the passage supports it. Otherwise, it tries up to two replacements with contrastive decoding, returning one as soon as it passes this check. It rejects replacements that repeat the answer generated without the passage when the passage favors another answer. Our experiments show that AoR raises the score over sampling by 12.5 points on counterfactual passages and 2.3 on ordinary ones, surpassing eight contrastive and prompting baselines on both, including when each receives the same rejection rule. Replacing only initial answers that the passage does not support improves the average score of every decoder. These results suggest that models can follow passages that contradict their knowledge while preserving accuracy on passages that agree with it. Code will be released at https://github.com/js-lee-AI/AoR.

1 Introduction

LLMs can favor training-learned knowledge over conflicting passage information, while contrastive decoding can also change answers the passage supports. Accept-or-Repair (AoR) checks support before repairing answers and improves average scores over plain sampling on both counterfactual and ordinary passages.

1 Introduction

  • LLMs often prioritize knowledge learned during training over information in a provided passage, even when the two conflict.Passages may also provide updated or private information.Source: §1 · p. 1
  • AoR accepts an initial answer when the passage supports it; otherwise, it tries up to two contrastive repairs and vetoes replacements repeating the closed-book answer when the passage favors another.The procedure is training-free.Source: §1 · p. 2
  • AoR improves average scores over plain sampling on counterfactual and ordinary passages and outperforms contrastive and prompting baselines.Its lead remains when baselines receive the same memory veto, and LLM judges rank AoR highest on average.Source: §1 · p. 2
  • Replacing only unsupported plain answers improves the average score of each baseline.The support test selects answers to repair more effectively than model confidence, self-checking, or an entailment model, which can accept answers from memory despite a different passage answer.Source: §1 · p. 2
  • Contrastive decoders can improve scores on unsupported initial answers but reduce scores on supported answers in ordinary passages.CAD turns correct answers into errors more often than plain resampling on ordinary passages; Figure 1 compares gains over plain sampling on both passage types.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Prior work studies passage faithfulness through prompting, contrastive decoding, and answer selection or verification. AoR combines answer selection with contrastive decoding, using a lexical passage test to choose unsupported initial answers for repair.

Knowledge conflicts

  • Prompting approaches use opinion-based questions and counterfactual demonstrations to improve faithfulness to counterfactual passages.FaithEval and ConFiQA evaluate whether models base answers on counterfactual passages.Source: §2 · p. 2

Decoding for context faithfulness

  • Contrastive decoding methods compare predictions from the original input with predictions under a modified prompt, and vary how contrast is controlled.Methods use distribution divergence, entropy, confidence signals, tunable factors, gating, routing, token-probability adjustments, or hidden-representation steering.Source: §2 · p. 2
  • AoR combines answer selection with contrastive decoding to keep supported initial answers and repair unsupported ones.This connects passage-based answer selection with decoding methods that contrast predictions under modified prompts.Source: §2 · p. 2

Answer selection and revision

  • Answer-selection methods use answer frequency or trained verifier scores, while verification methods use self-critique, retrieved evidence, or attention signals.AoR instead selects initial answers for repair with a lexical passage test, without a trained verifier, self-verification prompts, or attention weights.Source: §2 · pp. 2–3

3 Accept-or-Repair

AoR samples a plain answer and keeps it when a passage-support test passes; otherwise, it tries up to two ordered repairs, subject to a memory veto. Its support test applies different lexical checks to short answers, sentence answers, and multiple-choice options.

Support test

  • The support test accepts short free-form answers that occur as contiguous whole tokens after normalization, with special checks for yes/no, multiple-choice, and sentence answers.For multiple-choice questions, the selected option text must occur in the passage and no other option text may occur; for sentence answers, at least one remaining content word must occur.Source: §3 · p. 3

Accept

  • AoR accepts a plain answer unchanged when the support test passes, without applying contrastive decoding.The procedure checks the initial answer before deciding whether to repair it.Source: §3 · p. 3

Repair

  • If the initial answer fails support, AoR tries repair decoders in order and returns the first candidate passing both support and veto checks.The method uses K = 2: CAD followed by Instructive Decoding, with the decoders and order selected on the development split.Source: §3 · p. 3
  • If no candidate passes both checks, AoR returns the first answer-stating candidate not rejected by the veto, or the initial answer if none exists.For multiple-choice questions, a candidate must provide a valid option letter.Source: §3 · p. 3

Memory veto

  • For an unsupported initial answer, AoR draws a closed-book answer to screen repairs that repeat it when the passage favors another answer.For multiple choice, the veto masks the closed-book option when another option scores higher on passage overlap; it leaves that option unmasked if no alternative scores higher.Source: §3 · p. 4

Supported and unsupported answers

  • Retaining a supported initial answer preserves its score, and AoR(D) has a higher expected score than D when D lowers average scores on supported initial answers.The expected gain expression includes a nonnegative fallback term for cases where a repair states no answer.Source: §3 · p. 4

4 Experimental Setup

The experiments compare AoR and prompting and contrastive baselines on counterfactual and ordinary passage tasks across instruction-tuned models. Scores are averaged within the two task groups, with their average reported as Mean.

Tasks

  • The evaluation tests counterfactual and ordinary passage tasks, including NQ-Swap, FaithEval, NQ-Original, SQuAD, HotpotQA, ConFiQA, TriviaQA, and SciQ.HotpotQA uses two supporting paragraphs and up to eight distractors.Source: §4 · p. 4

Models

  • The main comparison uses six instruction-tuned models, with additional models used to examine performance across sizes and families.The main set includes Qwen3 models from 1.7B to 14B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3.Source: §4 · p. 4

Baselines

  • Baselines include contrastive decoders using different contrastive inputs and prompting methods that ask for passage-only answers.CAD’s scoring rule is also applied to RPCD and ID for comparing their contrastive inputs.Source: §4 · p. 4

Settings

  • Contrast settings are selected on development data and then fixed across models and tasks; AoR uses the selected CAD and ID strengths for repairs.The chosen setting has the highest average gain over sampling across both task groups, retaining a recommended setting when it is within 0.1 points of the best.Source: §4 · p. 5

Evaluation

  • Free-form answers are scored with token-level F1 and multiple-choice answers with accuracy; Mean averages the counterfactual and ordinary group scores.Score differences use paired-bootstrap 95% confidence intervals, and cost counts tokens processed across decoding calls relative to plain sampling.Source: §4 · p. 5

5 Results

AoR significantly improves over sampling on both counterfactual and ordinary passages and leads the baselines in Mean. The results attribute its gains to retaining supported answers while repairing unsupported ones.

Main results

  • 12.5 points on counterfactual passages and 2.3 on ordinary passages: AoR is the only method significantly above sampling in both groups and scores highest on each task.It significantly outperforms baselines in Mean; its token cost is 1.71 times sampling, lower than RPCD and ID.Source: §5 · p. 5

Where the gains come from

  • Contrastive baselines improve on unsupported initial answers but lose on supported ordinary answers; AoR retains supported answers and corrects more wrong counterfactual answers than the baselines.AoR turns 4 of 24,639 correct initial answers on ordinary passages into errors.Source: §5 · p. 5, Table 2 · p. 6

Wrapping other decoders

  • Wrapping each baseline as AoR’s only repair significantly improves its Mean, while counterfactual scores remain nearly unchanged.These wrappers omit the memory veto; on ordinary passages, gains recover losses on supported answers.Source: §5 · p. 5, Table 3 · p. 6

Contrast strength

  • With AoR, CAD, ID, and RPCD reach their highest Mean at the strongest tested contrast setting, while ordinary scores stay above sampling.Without AoR, all three score below sampling on ordinary passages at the strongest setting.Source: §5 · p. 5, Figure 5 · p. 7

Ablations

  • Keeping supported answers improves ordinary scores and reduces token cost; partial answer-span matches lower Mean and admit more memorized answers on NQ-Swap.Removing the memory veto or either repair lowers scores; replacing contrastive repairs with strict- or plain-prompt samples lowers Mean by 3.0 to 3.9 points. Majority voting scores below AoR on both passage types and uses more tokens.Source: §5 · pp. 6–7, Table 4 · p. 7

Choosing what to repair

  • The support test gives the largest gains on NQ-Swap and ordinary passages among the tested selection criteria.On NQ-Swap, it selects 99.5% of initial answers repeating the original-passage answer, versus 54–55% for answer probability, self-check, and entailment.Source: §5 · p. 7, Table 5 · p. 7

6 Analysis

AoR retains its advantage across additional evaluations, model families, decoding settings, and answer formats. Its selective repair accepts many initial answers unchanged, while supporting a repair or fallback for rejected answers.

Beyond the main setting

  • AoR outperforms baselines across Table 6 comparisons, and its lead remains without FaithEval and when baselines receive the same veto.Table 6 reports mean or held-out group averages across the listed comparisons.Source: §6 · p. 7, Table 6 · p. 8
  • The lead over the best baseline grows from about 3 points for Qwen3 models up to 8B to 5.7 points at 32B.The authors suggest that checking which answers need repair remains useful as model size increases.Source: §6 · pp. 7–8
  • AoR retains its lead with retuned greedy decoding and ranks highest on average under two LLM judges that accept paraphrased answers.It also improves over sampling when answers are given as one sentence.Source: §6 · p. 8

Where AoR acts

  • AoR accepts 82 to 86% of initial answers on ordinary passages and 70% on NQ-Swap, while finding a supported repair for more than half of rejected free-form answers.On most FaithEval questions no option passes the support test, so AoR returns a fallback; the examples illustrate acceptance, repairs, and the memory veto.Source: §6 · p. 8, Table 7 · p. 8

7 Conclusion

The conclusion presents AoR as a training-free way to preserve passage-supported answers while repairing unsupported ones. The reported gains span counterfactual and ordinary passages, and the authors conclude that support checking helps models follow passages that conflict with learned knowledge without losing accuracy on agreeing passages.

7 Conclusion

  • AoR keeps passage-supported initial answers and uses contrastive decoding to repair unsupported ones, with a memory veto against repairs repeating the closed-book answer when the passage favors another.Across ten models it improves average scores over sampling on counterfactual and ordinary passages; in the main evaluation, gains are large on counterfactual and small on ordinary passages.Source: §7 · p. 8

Limitations

The paper’s support check relies on word matching, and the experiments cover English benchmarks and models up to 70B parameters. Contrastive repair also requires access to next-token distributions.

Limitations

  • AoR’s word-matching support check covers short answers and single sentences; long-form or abstractive answers need tests that handle paraphrasing and information combined across the passage.The experiments cover English benchmarks and models of up to 70B parameters.Source: Limitations · p. 9
  • Contrastive repair decoders require access to the model’s next-token distributions; strict-prompt resampling is evaluated as an alternative repair generator.Source: Limitations · p. 9

Ethics Statement

The paper reports that passage support alone does not establish factual correctness, and documents its prompt, decoding, support-check, and evaluation procedures. These details include the prompt formats, answer matching and veto rules, and measured latency.

Ethics Statement

  • Applications should assess source reliability because passage support alone does not establish factual correctness.Source: Ethics Statement · p. 9

A Prompts and Decoders

  • Prompts use the model’s chat template without an added system message; Qwen3 runs with thinking disabled, and task prompts distinguish free-form from multiple-choice formats.For multiple-choice prompts, the instruction requests an answer by option letter; HotpotQA uses a Passages label and titled paragraphs.Source: A Prompts and Decoders · pp. 12–13
  • Contrastive decoders combine token log probabilities from the passage prompt and a contrasted prompt using a contrast strength, with each baseline defining its contrasted input.AdaCAD, COIECD, and NWCAD modify or route these scores using divergence, entropy, or confidence signals.Source: A Prompts and Decoders · p. 13

B Support Test and Veto

  • The support test accepts normalized whole-token matches, while the memory veto blocks a closed-book answer when the passage favors another option.For multiple-choice questions, an option must be the sole text match to pass; if no repair passes both checks, AoR returns a non-vetoed fallback.Source: B Support Test and Veto · p. 13
  • On Qwen3-8B at batch size one, AoR takes 1.30 times as long as sampling, compared with 1.32 for CAD.Repairs and the closed-book answer are generated only for unsupported initial answers.Source: B Support Test and Veto · p. 13

C Evaluation Details

The evaluation uses established dataset splits, scores answers according to task type, and pairs sampled outputs across methods. Baseline settings are selected on development data, where larger counterfactual gains often coincide with lower ordinary-passage scores.

C Evaluation Details

  • Main tasks use validation or development splits, while held-out tasks use test or development splits from the listed datasets.The study samples 1,000 examples per task, reserving 200 for development and 800 for testing; ConFiQA excludes questions with answer aliases shared with the original or passages over 1,000 words.Source: C Evaluation Details · p. 13, Table 9 · p. 15
  • Short free-form answers are scored by token-level F1, while multiple-choice answers are scored by extracted option-letter accuracy.Unrecognized option letters count as incorrect; generation is capped at 48 new tokens for short answers and 32 for multiple-choice answers.Source: C Evaluation Details · p. 14
  • Paired Gumbel noise gives methods with identical distributions the same sampled token, and AoR reuses the initial answer’s noise for repairs.After contrastive adjustment, temperature scaling, and filtering, the noise is added to token scores before selecting the highest-scoring token.Source: C Evaluation Details · p. 14
  • Main-comparison confidence intervals use 2,000 bootstrap resamples, with task-level comparisons corrected by Holm’s method.Resampling indices are shared across models and runs, and across paired NQ-Swap and NQ-Original tasks.Source: C Evaluation Details · p. 14

D Baseline Settings

  • Settings with the largest counterfactual gains lower ordinary-passage scores, a pattern that also appears across tested settings.Strict CAD is evaluated at CAD’s selected strength; settings are chosen by average development gain, retaining the original recommendation when within 0.1 points of the best.Source: D Baseline Settings · p. 14
  • NWCAD uses the CAD score for at most 0.5% of traced decoding steps across tasks.The trace covers 200 test examples per task using Qwen3-8B at the selected setting.Source: D Baseline Settings · pp. 14–15

E Further Analyses

The further analyses examine when passage support helps, how AoR’s components and repair choices contribute, and whether its gains persist across settings and evaluation variants. Across these analyses, AoR generally improves on baselines while preserving supported answers more effectively.

Main comparison

  • Against Strict CAD, AoR gains 2.9 points on counterfactual and 5.9 on ordinary passages; every baseline trails significantly in both groups.Across six models, AoR’s smallest Mean lead is 2.9 points, and 44 of 45 task-level gains over baselines remain significant after Holm correction.Source: E Further Analyses · p. 15

Supported answers

  • On ordinary passages, 83.6% of plain answers are supported and score 82.5 F1, while CAD and Strict CAD score 4.1 and 6.2 points lower on them.Across free-form tasks, 72–95% of supported draws reach F1 of at least 0.5, versus 16–53% of unsupported draws; every baseline breaks more supported answers than it corrects.Source: E Further Analyses · p. 15

Repair term

  • AoR’s repair term exceeds its fallback loss: supported repairs score 50–78 F1 versus 19–48 for the plain answers they replace, while fallback loss stays within 2.2 points of zero.The gain condition is that ρ (π − uR) exceeds (1 − ρ)(uF − ϕ); on ordinary passages, AoR raises 2.3 answers for each one it lowers.Source: E Further Analyses · p. 15

Where contrastive decoders break answers

  • CAD, COIECD, and Strict CAD break supported answers more often when the closed-book answer is correct, while RPCD and ID show the opposite pattern.For example, CAD breaks 15.1% of supported answers when the closed-book answer is correct versus 11.5% when it is wrong; a second plain sample breaks 2.7% of correct initial answers versus 12.2% for CAD.Source: E Further Analyses · pp. 15–16

Ablations and memory veto

  • Removing AoR’s initial acceptance test still yields 75.7 on ordinary passages, while removing the memory veto lowers FaithEval accuracy by 6.4 points and the counterfactual average by 3.2.The 0.5 and 0.8 span-match thresholds yield Mean scores of 73.1 and 74.4; baselines given the veto still trail AoR by at least 1.2 counterfactual and 2.2 ordinary points.Source: E Further Analyses · p. 16

Repair variants

  • Repairs without contrast gain 1.4 Mean points over Strict CAD, while feedback repair gains only 1.6 on NQ-Swap and 0.2 on ordinary passages.CAD repair gains 5.9 and 1.3 points on those groups at similar token cost; span-restricted repair raises NQ-Swap from 65.2 to 67.6 and the Mean of each of six models.Source: E Further Analyses · pp. 16–17

Gates

  • A support-test gate raises Mean by 3.6 points, significantly more than the other gates, which gain at most 2.2 points.The support test outperforms confidence, self-check, and entailment alternatives; with self-check or entailment as its test, AoR remains 1.6 and 2.0 points above the closest baseline.Source: E Further Analyses · p. 17

Held-out data and further models

  • AoR gains 7.3 points over sampling on held-out counterfactual ConFiQA subsets and 1.2 on held-out ordinary tasks, where no baseline improves significantly.Its gains also persist across larger models and added distractors; the passage-length analysis reports gains of 0.8 to 9.1 points in every quartile of every free-form task.Source: E Further Analyses · pp. 17–18

F Sentence Answers and Greedy Decoding

The paper tests AoR on sentence answers and greedy decoding, then illustrates its decisions with examples and reports model-level averages. AoR improves on sampling in the sentence-answer tasks and under greedy decoding.

F Sentence Answers and Greedy Decoding

G Examples

  • Examples show AoR keeping a supported answer, accepting a passage-grounded repair, and vetoing a repair that repeats the closed-book answer.On FaithEval, it masks the closed-book choice D when the passage favors C, after which CAD generates C.Source: G Examples · p. 20

H Results by Model

Related papers

Loading 2610.29727v1…