PaperlayerView exact revision on arXiv

Source-linked AI summary

Return or Revise? Learning When Revision Helps Retrieval-Augmented QA

Nicholas Kashani Motlagh, Tim Anderson, Jeremy Gwinnup, Grant Erdmann

arXiv:2609.30087v1cs.CLcs.IRcs.LG

TL;DR

Deciding whether to revise a retrieved-evidence-conditioned draft requires estimating the effect of revision, not just the draft’s correctness. The paper trains policies on paired outcomes and finds gains over draft-correctness prediction, but candidate revision adds no significant gain when standard RAG is another available answer.

  • Problem

    Draft confidence estimates whether an answer is correct, but does not determine whether a specified revision will repair or harm it.

    Source: §1 · p. 1

  • Method

    The authors grade both the draft and its candidate revision under the same judge, then train pre-revision policies to predict their paired effect.

    Source: §1 · p. 1

  • Results

    Paired-outcome scoring has greater accuracy–revision-rate curve area in all nine Llama setup–seed fits and gains 0.23–0.68 accuracy points on average at selected thresholds, with run-level significance only for DPR.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    When standard RAG is also available, choosing between it and the draft is stronger than return-or-revise, and candidate revision adds no significant accuracy gain as a third option.

    Source: §7 · p. 11

    This is an accuracy study, and its results do not establish deployment readiness; evaluation would require domain-specific standards, evidence checks, escalation, and validation of available actions.

    Source: Ethical Considerations · p. 12

Abstract

from arXiv · show

We consider the decision of whether to return an existing draft answer or revise it using retrieved evidence, as in answer-revision systems. Draft confidence estimates whether the current answer is correct, but the decision requires estimating the effect of a specified revision. For offline training and evaluation, we grade both the returned draft and its candidate revision under the same correctness judge, which makes repair, harm, and the gap to an oracle observable. We call this paired effect its recoverability, and we train policies to predict it before revision. On 25,870 held-out open-domain questions across three revision setups, a scorer trained on the paired outcome has greater area under the accuracy--revision-rate curve than a matched draft-correctness scorer in all nine Llama setup--seed fits, and gains 0.23--0.68 accuracy points on average at development-selected thresholds, a difference significant across training runs only for dense retrieval. The resulting policy improves on always revising and on average closes more than a third of the oracle gap, although it still applies 38--46% of the harmful revisions. When a draft-free standard-RAG answer is also available, however, choosing between the draft and that answer is stronger by about two points for Llama and four for OLMo, and adding candidate revision as a third option yields no significant gain. Recoverability describes one revision; its value as an available action also depends on the alternatives.

1 Introduction

The paper studies whether a system should return an existing draft or revise it with retrieved evidence, since revision can repair errors or corrupt correct answers. It evaluates paired-outcome prediction and how its value changes when other answers are available.

1 Introduction

  • Draft confidence estimates whether the current answer is correct, but deciding whether to revise requires predicting whether that revision will repair or harm it.The decision arises after a draft and retrieved evidence exist but before a revision is decoded.Source: §1 · p. 1
  • Offline paired grading evaluates both the returned draft and its candidate revision under one judge, making repair, harm, and the oracle gap observable.The paper calls the paired effect of a specified revision setup its recoverability.Source: §1 · p. 1, Figure 1 · p. 2
  • On 25,870 held-out examples, paired-outcome policies beat always revising by 1.10–1.33 points and close 35.9–41.4% of the oracle gap, while applying 38–46% of harmful revisions.The paired-effect scorer also has greater area under the accuracy–revision-rate curve in all nine Llama setup–seed fits; its selected-threshold advantage is significant across runs for DPR.Source: §1 · p. 2
  • When a draft-free standard-RAG answer is available, choosing between it and the draft is stronger than return-or-revise, and adding revision as a third option yields no significant gain.For Llama, standard RAG is less accurate alone but more useful as an alternative to the draft.Source: §1 · pp. 2–3
  • The study reports repair and harm rates across three revision setups and compares paired-effect prediction with draft-correctness prediction under matched training.It also tests how input and model class affect policy performance, and compares answer choices when standard RAG is available.Source: §1 · pp. 2–3

2 Paired Outcomes for the Revision Decision

The section defines the revision decision through paired outcomes: each example is graded both as a returned draft and as a candidate revision. This reveals which drafts are repaired or harmed and provides the target for the decision policy.

2 Paired Outcomes for the Revision Decision

  • Paired grading distinguishes four outcomes—preserved, repaired, harmed, and unrecovered—whereas aggregate accuracy cannot distinguish setups with different repair and harm rates.For example, equal aggregate gains can arise from 3% repairs with no harm or 13% repairs and 10% corruption.Source: §2 · p. 3
  • Recoverability is the conditional revision effect: repair probability minus harm probability for a specified revision setup.The example-level outcome is the difference between correctness under revision and correctness under returning the draft.Source: §2 · p. 3
  • A policy revises when predicted recoverability exceeds a development-selected threshold; because always revising is the better fixed action, the oracle gap equals the harm rate.Draft confidence identifies whether the draft is correct, but the paired outcome indicates whether revision repairs or harms it.Source: §2 · p. 3
  • Recoverability concerns a specified revision, while the decision differs from selective prediction, adaptive retrieval, and retrieval-utility prediction because it evaluates revising an observed draft with evidence held fixed.The paired label is measured offline under the chosen refiner, evidence, and correctness judge.Source: §2 · p. 3

3 Predicting Revision Effect

The study compares pre-revision inputs and model classes for predicting whether revision helps. Policies use the predicted effect to choose between returning the draft and invoking the revision setup.

3 Predicting Revision Effect

  • Decision models receive the question alone, the question with the draft, or the complete revision prompt, without the candidate revision, gold answer, judge verdict, or outcome label.All inputs are available before revision decoding.Source: §3 · pp. 3–4
  • Five model classes score return versus revise, including four frozen-feature models and a low-rank adapted model that reads the available input.Classifiers predict repair, harm, or a tie; their decision score is the softmax probability of repair minus that of harm, and is not calibrated.Source: §3 · p. 4
  • Baselines include learned draft-correctness prediction, a Tian-style engineered-feature regression, and verbalized P(true); matched LoRA models differ only in whether they predict draft correctness or paired outcomes.The Tian-style baseline adapts its features and target to return-or-revise; it does not reimplement the original learned QPP and QualT5 components.Source: §3 · p. 4

4 Experimental Setup

The experiments compare three retrieval setups using shared evidence prompts and a shared refiner. Training and development use NQ-Open and TriviaQA, while test also includes evaluation-only PopQA.

4 Experimental Setup

  • DPR, BM25, and BM25 followed by MonoT5 reranking each retrieve five title–passage pairs from the same Wikipedia collection and share the evidence prompt and refiner.The setups differ only in retrieval, enabling comparisons of dense and sparse retrieval and reranking.Source: §4 · p. 4
  • Training and development use 134,847 and 14,966 NQ-Open and TriviaQA examples; the 25,870-example test includes 14,267 evaluation-only PopQA examples.Llama 3.1 8B Instruct is the primary generator and refiner, and a deterministic Llama 3.3 70B judge grades both actions.Source: §4 · p. 4
  • Candidate revisions are generated for every split to construct paired labels and evaluate policies, but they are not policy inputs at decision time.The policy either returns the observed draft or invokes the revision setup.Source: §4 · pp. 4–5

5 Results

Revision repairs drafts but also corrupts some correct answers, so a learned policy captures only part of the oracle benefit. The value of candidate revision depends on the available alternatives: standard RAG often performs better when it can compete with the draft.

5.1 Measured Repair and Harm

  • Revision repairs 9.32% of DPR drafts and corrupts 3.05%; across setups, reranking raises repairs without reducing harms, while most remaining examples are ties.The harm rate across setups is 3.05–3.33%.Source: §5.1 · p. 5

5.2 The Learned Policy

  • The full-prompt LoRA policy gains 1.10–1.33 points over always revising, closes 35.9–41.4% of the oracle gap, and still applies 38.0–46.4% of harmful revisions.It revises 94.0–96.0% of repairs, making harmful revision its main remaining error.Source: §5.2 · p. 5

5.3 Paired Outcome versus Draft Correctness

  • Paired-outcome scoring has greater accuracy–revision-rate area in all nine fits; at selected thresholds, gains over draft-correctness are 0.68 DPR, 0.23 BM25, and 0.33 MonoT5, significant only for DPR.Draft correctness alone recovers 0.58–1.11 points of the policy's 1.10–1.33-point gain over always revising.Source: §5.3 · p. 5, Figure 3 · p. 6

5.4 Inputs and Model Classes

  • With the full prompt, fine-tuned LoRA closes 32.1–41.0% of the oracle gap in one run, while the best frozen-feature model closes at most 13.5%.Masking evidence or replacing it with another example's evidence reduces gains over always revising to 0.18–0.25 points in every setup.Source: §5.4 · p. 6

(B) BM25

  • Topically close, fluent evidence can still answer for the wrong entity or time period; successful revision depends on evidence supporting the fact actually requested.Source: (B) BM25 · p. 7

5.5 Robustness

  • With PopQA held out of training and development, one policy and threshold improves all nine dataset–setup slices; across other generator/refiner families it beats always revising in all nine combinations.The latter comparison was repeated under a second judge.Source: §5.5 · pp. 7–8
  • In a 480-pair blind audit, human and judge three-way outcome agreement was 94.5% (κ = 0.78), while the human-labeled pooled net effect was +2.8 points [−0.6, +6.1].Source: §5.5 · p. 8

5.6 Sensitivity to the Available Answers

  • For Llama, replacing candidate revision with standard RAG raises mean accuracy 1.95–2.10 points; for OLMo, return-or-RAG leads return-and-revise by 4.0–4.6 points.For Llama, adding revision as a third option yields no significant gain; across nine fits, three-action policies select only 7–24% of revision-only wins.Source: §5.6 · pp. 8–9

6 Related Work

Prior work studies retrieval decisions, answer revision, and evidence use; this paper focuses on whether to revise a fixed draft using fixed retrieved evidence. Its related-work discussion positions paired-outcome prediction alongside retrieval utility and answer-selection research.

6 Related Work

  • Adaptive-RAG methods act before or during retrieval, whereas this decision follows retrieval with the draft and evidence fixed.The authors do not rerun Self-RAG or Adaptive-RAG baselines because those methods change the generator or retrieval pipeline.Source: §6 · p. 9
  • Table 5 decomposes third-action changes into unique revision wins, successful switches, and damaging switches, with their net change in correct answers.Source: Table 5 · p. 10
  • RASER predicts gains from escalating one-shot RAG and changes the retrieval route; this work holds evidence fixed and compares returning the observed draft with a conditioned revision.Source: §6 · p. 10
  • Retrieval-utility and routing work predicts retrieval gains or selects among answerers; this study measures absolute repair and harm under a fixed refiner.Source: §6 · p. 10
  • Prior revision and conflicting-evidence work studies editing, evidence arbitration, and harmful self-correction; this study adds matched prediction targets and fixed-scorer action comparisons.Source: §6 · p. 10

7 Discussion

Paired-outcome prediction refines draft-confidence reasoning, while its benefit depends on the model's inputs and the available answer actions. The discussion also notes that accuracy-based decisions may need different utilities in deployment.

What the paired target adds

  • 35.9–41.4% of the return-or-revise oracle gap is closed, and paired prediction adds 0.23–0.68 points over draft correctness at selected thresholds.The added gain is significant across training runs for DPR; a learned draft-correctness model already recovers part of the gain over always revising.Source: §7 · pp. 10–11

Where the useful signal comes from

  • The largest input-comparison gain comes from adapting the model on the complete question, draft, and evidence prompt.With model and training fixed, most of that gain depends on retrieved evidence and part on the draft.Source: §7 · p. 11

Better retrieval does not remove the decision

  • MonoT5 reranking yields the most repairs and largest average revision effect, but no larger policy gain over always revising, and harmful revisions remain.Source: §7 · p. 11

The action set determines the value of revision

  • Adding candidate revision to draft and standard RAG gives no significant accuracy gain, while the standard-RAG action's relative value differs between Llama and OLMo.For Llama, standard RAG is weaker as a fixed action but fails on different examples; for OLMo, it is stronger and the action-set gap widens to four points.Source: §7 · p. 11

From accuracy to decision utility

  • Accuracy-based thresholds treat repair and harm symmetrically, but deployment may assign different costs to corruption, revision latency, compute, abstention, or escalation.The paper says those settings require specifying each action's utility, selecting an operating point on development data, and testing transfer.Source: §7 · p. 11

8 Conclusion

The paper finds that paired-outcome prediction improves return-or-revise decisions, but candidate revision adds no significant accuracy once standard RAG is available as an alternative. Offline grading of available answers makes these decision comparisons measurable.

8 Conclusion

  • A paired-outcome scorer has greater accuracy–revision-rate area in all nine Llama setup–seed fits and higher mean return-or-revise accuracy in all three setups.The mean-accuracy difference is significant across training runs for DPR.Source: §8 · p. 11
  • The largest input-diagnostic gain comes from an adapted model that reads the full evidence prompt.Source: §8 · p. 11
  • When standard RAG is also available, candidate revision provides no significant additional accuracy in the tested policies.The conclusion ties revision's value to the available alternatives and says offline grading of every answer makes that comparison measurable.Source: §8 · p. 11

Limitations

The study is limited to short-form open-domain QA under specific model, retrieval, and evaluation conditions. Its conclusions also depend on the judge, the tested answer sets, and the refiner’s available headroom.

Limitations

  • The evaluation covers short-form open-domain QA, not long-form, multi-hop, domain-specific, or high-stakes settings.The same checkpoint drafts and revises, models are no larger than GPT-OSS-20B, and each setup retrieves five passages from one Wikipedia collection.Source: Limitations · p. 11
  • PopQA is evaluation-only and makes up 55% of the test set, leaving a distribution-shift concern despite per-dataset and macro-averaged results.The study also excludes different drafters and refiners, larger models, other corpora, deployed RAG pipelines, and human revision.Source: Limitations · p. 12
  • The 45-cell input–model grid and several diagnostics use one training run, while three runs give run-level tests little power.Seeds vary only policy training, not generation or judge-sampling variance; confidence alternatives beyond verbalized P(true) are not surveyed.Source: Limitations · p. 12
  • The oracle gap equals the refiner’s harm rate, so a refiner that rarely overwrites correct drafts leaves little accuracy headroom for a policy to close.GPT-OSS-20B is given as an example of a refiner with a small gap.Source: Limitations · p. 12
  • Automated semantic-equivalence judges supply correctness labels, and harm is the least stable paired outcome under rescoring.The single-annotator diagnostic cannot estimate inter-annotator reliability or independently validate sub-point policy.Source: Limitations · p. 12
  • The nonsignificant accuracy gain from adding candidate revision is not evidence of equivalence for the tested answer sets.The shared per-action model controls the action-set comparison, but its checkpoint-selection objective may differ from a scorer optimized separately for each two-action set.Source: Limitations · p. 12

Ethical Considerations

The appendix reports ethical boundaries, decision-model recipes, and diagnostic baselines alongside analyses of harmful revisions. It compares learned policies with verbalized confidence and a retrieval-utility baseline.

Ethical Considerations

  • This is an accuracy study, not evidence of deployment readiness; deployment would require domain-specific standards, evidence checks, escalation, and validation of every available action.The authors note that missed repairs and applied harms are consequential, and that uneven or stale Wikipedia evidence could leave weak-coverage cases without repair.Source: Ethical Considerations · p. 12

A.1 Decision-Model Recipes

  • The matched draft-correctness scorer and paired predictor use different checkpoint-selection objectives, while the three-action model selects checkpoints by development accuracy.The per-action model predicts correctness independently for return, standard RAG, and revision; restricted action sets share its selected checkpoint, with no set-specific refit or threshold.Source: §A.1 · p. 16, Table 6 · p. 17

A.2 Verbalized Confidence Baseline

  • The verbalized P(true) baseline revises 85.3–93.1% of drafts and gains +0.027 points [−0.015, +0.070] with DPR, +0.089 [+0.031, +0.147] with BM25, and +0.035 [+0.008, +0.066] after MonoT5 reranking.The Llama generator answers whether its own draft is correct; its policy is selected on development data and evaluated on all 25,870 test examples.Source: §A.2 · p. 16

A.3 Analysis of Harmful Revisions

  • Harmful revisions can follow fluent, topically close evidence that answers a different entity or time period, as illustrated by the Warriors and The Terminal cases.The test-set lexical associations diagnose the existing revision setups; they were not used to choose a method or threshold.Source: §A.3 · pp. 16–17, Table 7 · p. 17

A.4 Retrieval-Utility Baseline

  • The LoRA policy leads the Tian-style retrieval-utility baseline by 1.04–1.25 accuracy points in every setup.The baseline has 22 coefficients versus about 42M trainable LoRA parameters, and its development threshold sends nearly every draft to revision.Source: §A.4 · p. 17

A.5 Additional Matched Training Targets

The appendix compares alternative paired-outcome targets, reports a complete input–model grid, tests controlled input ablations, and breaks results down by dataset. The target comparison favors two-head prediction over the alternatives tested, with a stated caveat about fitted recipes.

A.5 Additional Matched Training Targets

  • Two-head prediction exceeds three-way prediction by 0.13 points on average and exceeds scalar regression and preference targets by 0.34–0.47 points.Five of nine paired example-bootstrap intervals exclude zero for the two-head versus three-way contrast; the nine cells share the same test data.Source: §A.5 · p. 17, Table 9 · p. 18

A.6 Complete Input and Model Comparison

  • A correct estimate of the scalar expected correctness difference would suffice for the decision, so the observed ordering does not show scalar targets lack decision-relevant information.The comparison reflects the fitted recipes; the complete input–model grid covers 45 combinations of revision setup, model class, and pre-revision input.Source: §A.5 · p. 18, §A.6 · p. 18

A.7 Controlled Input Ablations

  • Keeping only passage titles lowers accuracy by 0.41–0.89 points in single runs, while reversing or shuffling passage order changes it by at most 0.18 points.Two training seeds of the unchanged policy differ by up to 0.43 points.Source: §A.7 · p. 18

A.8 Results by Dataset

  • Always revising is the better fixed action in all nine dataset–setup slices; standard RAG is less accurate than revision in every slice but can outperform returning the draft.Standard RAG’s smallest deficit relative to revision is 1.60 points on DPR/NQ-Open.Source: §A.8 · p. 18

A.9 Additional Robustness Checks

The robustness appendix reports second-judge agreement, a single-annotator diagnostic, and action-set comparisons, alongside checks using a neutral revision prompt. It describes the scope and evaluation protocol for these checks, with harm the least stable paired outcome under judge rescoring.

A.9 Additional Robustness Checks

  • Second-judge agreement is 96.46–96.74% per answer and 94.87–95.25% across paired outcomes; harm is the least stable outcome.Only 85.7–86.7% of primary-judge harms remain harms, yet the stricter second judge finds more harms overall.Source: §A.9 · pp. 18–19, Table 14 · p. 20

A.10 Single-Annotator Human Diagnostic

  • In the 470 decisive audited pairs, the primary judge labels 37 repairs and 26 harms; the annotator agrees with 29 repairs and 23 harms.The dataset-balanced audit uses one blind annotator and covers DPR and BM25; no pair moves between repair and harm in the reported comparisons.Source: §A.10 · pp. 19–20, Table 15 · p. 20

A.11 Action Sets by Dataset

  • The dataset action-set comparison applies the same per-action model without dataset-specific fitting, checkpoint selection, thresholds, or action-set selection.It compares draft return, standard RAG, and candidate revision, with differences paired within model and example.Source: Table 16 · p. 21, §A.11 · p. 21

A.12 Policy Contrasts Under a Second Judge

  • Under the second judge, matched paired-target and draft-correctness policies retain their development-selected thresholds, and action-set policies retain their argmax choices.The check rescores the same test decisions with GPT-OSS-120B at low reasoning effort.Source: §A.12 · p. 21

A.13 Neutral Revision Prompt

  • The neutral prompt keeps the draft when evidence best supports it and replaces it when another answer is better supported; selectors are not retrained for this check.The comparison examines neutral revision’s standalone accuracy, complementarity, and oracle against a learned return-or-RAG policy.Source: §A.13 · p. 21

B Reproducibility Details

This section documents the datasets, implementation choices, prompts, and determinism settings needed to reproduce the experiments. It also reports prompt-sensitivity and policy-contrast analyses, while noting which artifacts are not publicly released.

B Reproducibility Details

  • Table 17 contrasts policies under both judges using three training seeds, paired example-bootstrap intervals, and unchanged decisions, checkpoints, and thresholds.The caption says matched binary targets and action-set comparisons are separate experiments but does not provide their cell values.Source: B Reproducibility Details · p. 21, Table 17 · p. 22
  • Neutral revision has lower standalone accuracy but a higher oracle than the candidate-revision prompt; its oracle exceeds the learned seed mean in every setup, and no neutral selector was fit.Table 18 compares prompts on the same Llama test examples and reports paired accuracy differences for three fixed policies.Source: Table 18 · p. 22
  • Frozen Llama states are taken before draft or evidence, before evidence in the draft prompt, and at the full revision prompt’s final token; answer and context probabilities use exact token spans.Context features teacher-force the exact tokenized prompts, and no answer is regenerated; malformed, truncated, ambiguous, and invalid outputs score incorrect.Source: B Reproducibility Details · p. 22, §C.1 · p. 25
  • 25,870 test examples span NQ-Open, TriviaQA, and PopQA, with PopQA contributing test rows only; invalid generations and judge parses count as failures.The test split contains 3,610 NQ-Open, 7,993 TriviaQA, and 14,267 PopQA examples.Source: B Reproducibility Details · pp. 21–22, C Artifacts, Prompts, and Determinism · p. 23

C Artifacts, Prompts, and Determinism

  • Public datasets, passage collections, and model families are used, but experimental code, generated answers, trained policies, and row-level outputs are not publicly released.Reproducing the experiments once is estimated to require approximately 900 H200 GPU-hours.Source: C Artifacts, Prompts, and Determinism · p. 23

C.1 Prompt Templates

  • The standard-RAG prompt uses question and evidence without a draft, while the candidate-revision prompt instructs the model to keep or replace its draft using evidence.The evidence block packs the complete top-five retrieved title-and-text passages, with no character cap.Source: §C.1 · pp. 24–25

Related papers

Loading 2609.30087v1…