PaperlayerView exact revision on arXiv

Source-linked AI summary

Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery?

Nishanth Nayakanti, Prasang Gupta, Ashutosh Bilthare, Kevin Paul

arXiv:2610.06962v1cs.CLcs.AI

TL;DR

Review workflows often retain verdicts but not the costly evidence spans, raising the question of how much evidence a model can recover from verdict-only post-training. Comparing label-only training with rejection-sampling SFT on ContractNLI, the paper finds that both improve evidence without human evidence annotation. The results are close on span F1 but do not establish which method is preferable.

  • Problem

    Review workflows often retain verdicts but omit the costly supporting-span annotations, leaving how much evidence can be recovered from verdict-only training an open question.

    Source: §1 · p. 1

  • Method

    The paper compares bare-label SFT with rejection-sampling SFT, which keeps verdict-matching traces and selects one using an automatic source-grounding score.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    0.564 span F1 for label-only training and 0.556 for RSFT compare with 0.493 untrained, while accuracy is 0.896 and 0.797 versus 0.747 untrained.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Both methods recover evidence quality without evidence annotation, but this single-run comparison does not establish that either method is preferable.

    Source: §5 · p. 10, §10 · p. 13

    The comparison uses one corpus, one student, and one seed per arm, with a second seed per arm still outstanding.

    Source: Limitations · p. 13

Abstract

from arXiv · show

In many review workflows the verdict is the only thing retained. The passages behind it are not marked, because that annotation costs far more than recording the decision. We measure how much of that evidence a small language model can recover when it is post-trained on the verdicts alone, with no human evidence labels at any stage. On ContractNLI the human evidence spans are held out until evaluation. Matching the recorded verdict and agreeing with those spans are not the same thing: across six systems the two scores are only weakly related and rank the systems differently, so accuracy is a poor guide when the citations have to be reviewable. Label-only training on the bare verdict reaches accuracy 0.896 and span F1 0.564. Rejection sampling, which keeps a generated trace only when its verdict matches the record and then picks one by an automatic source-grounding score, reaches 0.797 and 0.556, against 0.747 and 0.493 before training. Verbatim citation rises from 0.597 to 0.729 under label-only training and to 0.701 under rejection sampling. One seed on one corpus cannot say which method is better, but both improve the evidence without anyone annotating it.

1 Introduction

The paper asks how much evidence can be recovered when review records retain verdicts but omit costly span annotations. It compares two verdict-only post-training methods and finds evidence gains without evidence annotation, while cautioning that one seed cannot establish which method is preferable.

1 Introduction

  • Span-level labels cost far more to produce than verdicts, and operational records often retain the verdict without recording reasoning or marking supporting passages.ContractNLI serves as a measurable proxy because its evidence layer can be withheld during training.Source: §1 · p. 1
  • Evidence quality and verdict accuracy rank systems differently, so accuracy alone cannot select a model when citations must be reviewable.The most accurate evaluated system matches the untrained student at span F1, while two lower-accuracy systems score highest on evidence.Source: §1 · p. 1
  • Label-only SFT trains on bare verdicts, while RSFT filters verdict-matching traces and selects one per row using an automatic source-grounding score.Both methods use verdict-only records without human evidence annotation; RSFT additionally trains on rationale-plus-citation traces.Source: §1 · pp. 1–2
  • The comparison tests the added value of rationale-plus-citation supervision against bare-label supervision, which skips candidate generation and selection.Label-only SFT is the cheaper intervention; the paper does not claim either method is generally preferable.Source: §1 · p. 2
  • 0.564 span F1 for label-only SFT and 0.556 for RSFT compare with 0.493 untrained; accuracy is 0.896 and 0.797, respectively, versus 0.747 untrained.Both methods also improve source verifiability, but the paper presents the comparison without ranking the methods.Source: §1 · p. 2
  • A label-marginal calibration control recovers only 37% of RSFT’s accuracy increment, and a 2-epoch RSFT run reproduces the 1-epoch result.The authors use these checks to bound alternative explanations, including insufficient epoch count.Source: §1 · p. 2

2 Setup

The setup uses ContractNLI verdicts for training while reserving human evidence spans for evaluation. Locked prompts and a shared test protocol define the comparisons across systems.

2.1 Task and dataset

  • Each ContractNLI document is paired with 17 hypotheses and receives an Entailment, Contradiction, or NotMentioned verdict; human evidence spans are reserved for evaluation.The training split has 559,215 binary span decisions alongside 7,191 verdicts.Source: §2.1 · pp. 2–3

2.2 Models and controls

  • The majority-class and hypothesis-only controls score 0.463 and 0.659 verdict accuracy, respectively; the latter predicts from hypothesis identity without reading the contract.The hypothesis-only score estimates performance from hypothesis-specific label priors; commercial reference identifiers are also specified for the evaluated systems.Source: §2.2 · p. 3

2.3 Prompt and output format

  • A hash-checked task definition has faithful and verdict_only output variants, with the evidence paragraph removed from verdict_only for label-only training.Label-only SFT is evaluated on the evidence axis under faithful, creating a train/test prompt mismatch; its accuracy is 0.896 under faithful and 0.899 under verdict_only.Source: §2.3 · p. 3

2.4 Evaluation protocol

  • Evaluation uses one greedy output per row on 123 contracts and 2,091 canonical test rows; evidence metrics use the 1,188 rows with gold evidence spans.The test split’s NotMentioned rate is 43.2%; untrained-student and fine-tuned comparisons use a matched 900-token completion cap.Source: §2.4 · pp. 3–4, Figure 1 · p. 4

3 Methods compared

Both methods post-train the same student without human evidence annotations, but differ in their targets: bare verdicts versus selected rationale-and-citation traces. RSFT filters candidates by verdict and selects among survivors using an automatic grounding score; controls test what label calibration alone can explain.

3.1 Method A — Label-only SFT

  • Label-only SFT trains on the bare recorded verdict, without rationale, evidence annotation, or candidate generation.Targets average 71 characters, compared with 1,294 for rationale traces.Source: §3.1 · pp. 4–5

3 Methods compared

  • RSFT generates traces, keeps those matching recorded verdicts, selects one per covered row by source grounding, and fine-tunes on rationale-plus-citation targets.It uses one epoch of target-only QLoRA, and no stage reads human evidence annotations.Source: §3 · p. 4, §3.2 · p. 5

3.2 Method B — Rejection-sampling SFT (RSFT)

  • Greedy candidates cover 86.2% of rows, sampled passes add 741 to reach 96.5%, and the remaining 253 rows are dropped.Sampling runs only on rows where the corresponding greedy candidate fails the verdict filter.Source: §3.2 · p. 5
  • Verdict filtering determines RSFT eligibility, while a three-tier excerpt score selects the highest-scoring candidate, with fixed provenance order breaking ties.On 1.4% of covered rows without a well-formed verdict-matching candidate, the method scores the raw matching pool instead.Source: §3.2 · p. 5
  • Grounding separates candidates on 21.8% of covered rows, while survivors tie on 70.6%, usually because all quotations match verbatim.Where grounding discriminates, the 31B sampler supplies 76.2% of selected traces; about a fifth of selected traces overall come from the sampler.Source: §3.2 · p. 5

3.3 Controls

  • Prior calibration relabels the untrained student’s predictions using a training-split confusion table, without reading contracts or training the model.Hypothesis lookup and majority class bound what hypothesis identity and corpus rate alone can do.Source: §3.3 · p. 6

4 Verdict prediction results

Label-only SFT achieves higher verdict accuracy than RSFT, while both methods improve evidence-related performance in the broader study. Error analysis attributes RSFT’s net improvement to NotMentioned recovery, and a calibration control explains only part of the accuracy gain.

4.1 Aggregate performance

  • 0.896–0.899 accuracy for label-only SFT exceeds RSFT’s 0.797 and its prompt-matched base’s 0.753, but the comparison is a single-seed estimate.The table reports held-out verdict prediction on 2,091 test rows; label-only is shown under both prompts, while RSFT and the base use faithful.Source: §4.1 · p. 6, Table 6 · p. 7
  • The comparison with Span NLI BERT is descriptive, not evidence of equivalence, because architecture, supervision, and evaluation convention differ.Span NLI BERT reports 0.892 accuracy with span supervision; label-only SFT reaches 0.896 on verdicts alone, while RSFT reaches 0.797.Source: §4.1 · pp. 6–7

4.2 Error analysis

  • 529 to 425 total errors under RSFT is a net reduction of 104, entirely from NotMentioned recovery; its predictions worsen on 9 of 1,188 evidence-bearing rows.Entailment recall moves 0.889 →0.886 and Contradiction recall 0.764 →0.736; label-only raises both recalls to 0.911 and 0.832, respectively.Source: §4.2 · p. 7
  • The +0.099 headline accuracy gap favors label-only, but its training includes 253 difficult rows RSFT cannot cover and avoids RSFT’s parse failures.On the 2,052 rows RSFT parses, RSFT is 0.812 and label-only is 0.897; about 1.4 of the 9.9 points reflects parse-failure asymmetry.Source: §4.2 · pp. 7–8

4.3 How much of the accuracy gain is calibration?

  • A label-remapping control recovers about a third of the accuracy increment, but cannot reproduce RSFT’s combination of higher NotMentioned recall and retained precision.The calibration control changes NotMentioned precision/recall from 0.928/0.613 to 0.825/0.692, while RSFT reaches 0.904/0.732.Source: §4.3 · p. 8, Table 9 · p. 8

5 Evidence quality without evidence annotation

Both post-training methods improve evidence agreement without human evidence labels, and label-only training does so without training the model to produce citations. Their similar span-F1 results arise through different routes, while the single-seed comparison does not establish a preferred method.

5 Evidence quality without evidence annotation

  • Label-only training improves span agreement from 0.493 to 0.564 even though its training prompt requests no evidence and no citation appears in training.Evaluation uses the citation-requesting faithful prompt, with span labels withheld until evaluation.Source: §5 · p. 8
  • Span F1 reaches 0.556 for RSFT and 0.564 for label-only, versus 0.493 for the untrained student; the two fine-tunes are statistically indistinguishable.The label-only-minus-RSFT difference is +0.009 with a 95% CI of [−0.011, +0.028], spanning zero.Source: §5 · pp. 8–9, Table 10 · p. 9
  • Label-only’s evidence gain tracks improved Entailment and Contradiction verdicts, whereas RSFT recovers more gold spans despite worse verdicts and fewer cited rows.RSFT’s true positives rise from 844 to 987 on the 1,188 entailment and contradiction rows; its span-F1 gain comes from better citations on rows it cites.Source: §5 · p. 9, Figure 4 · p. 9
  • On the 1,037 rows where all three systems produce locatable citations, span F1 remains higher for RSFT and label-only than for the untrained model.The comparison is descriptive because selecting rows by post-training citation locatability excludes some rows where RSFT predicts NotMentioned and cites nothing.Source: §5 · p. 9

6 Verdict accuracy and evidence quality decouple

Across evaluated systems, verdict accuracy and evidence quality rank systems differently. Therefore, accuracy alone is insufficient when citations need to be reviewable.

6 Verdict accuracy and evidence quality decouple

  • gpt-5.2 has the highest off-the-shelf accuracy but matches the untrained student’s span F1 of 0.493; the lower-accuracy 31B sampler and sonnet-4.6 score highest on evidence.The label-only fine-tune is the most accurate system overall, at 0.896.Source: Figure 5 · p. 10, §6 · p. 10
  • For reviewable-evidence workflows, verdict accuracy alone is insufficient for model selection because correct labels can accompany less verifiable or less aligned quotations.With the verdict_only prompt, label-only reaches accuracy 0.899 but has span F1 0.000 by construction because no citation is requested.Source: §6 · p. 10
  • Across six systems, verdict accuracy and evidence quality are weakly related and produce different rankings.The reported descriptive Pearson correlations are +0.12 for span F1 and +0.20 for verbatim rate.Source: Figure 5 · p. 10, Table 12 · p. 11

7 Secondary analyses

Secondary analyses find that greedy accuracy does not determine sampling recovery, while the available training-mixture analysis supports no directional inference. A hypothesis-only control scores 0.659, consistent with hypothesis-specific label priors accounting for a substantial share of performance.

7 Secondary analyses

  • Among rows both models miss greedily, the 9B student recovers 52.0% at matched sampling budget, versus 13.2% for the 31B sampler.These are point estimates from training-split candidate generation (n = 994, matched k = 8), without confidence intervals; they do not establish why the rates differ.Source: §7 · p. 11
  • Across 17 hypotheses, student training-trace share has descriptive correlation −0.14 with test-accuracy gain, supporting no directional inference or component attribution.A matched ablation is required to isolate a component.Source: §7 · p. 11
  • The hypothesis-only control scores 0.659 without reading contracts, consistent with hypothesis-specific label priors explaining a substantial share of performance.Performance above the control is consistent with using more than those priors, but does not exclude memorisation or pretraining exposure.Source: §7 · p. 11

8 Experimental configuration

The experiments use shared hyperparameters and a fixed recipe, with no test-set tuning or checkpoint selection. The matched-cap rerun leaves the accuracy delta unchanged and slightly increases the evidence delta; other configuration details qualify reproducibility and interpretation.

8 Experimental configuration

  • Both methods share hyperparameters and a recipe fixed before training, with no hyperparameter tuning or checkpoint selection on test results.Training rows, step count, and wall-clock time differ by arm.Source: §8 · p. 11, Table 13 · p. 12
  • A 2-epoch RSFT run reaches accuracy 0.796 and span F1 0.555, indistinguishable from the 1-epoch run’s 0.797 and 0.556.This argues against insufficient epoch count as the explanation for the gap, but does not rule out other schedule differences; a second seed per arm remains outstanding.Source: §8 · p. 11
  • The RSFT recipe met only the pre-registration’s all-verbatim target, at 0.701; those absolute deployment gates are not the study’s central claim.The listed targets were accuracy ≥ 0.850, all-verbatim rate ≥ 0.700, and span F1 ≥ 0.600.Source: §8 · p. 11
  • Several paired tests lack multiplicity correction, so per-class F1 movements, cross-system correlations, and sampling-recovery figures are descriptive.The headline McNemar p-values (≤4 × 10−12) are stated to survive any reasonable correction.Source: §8 · p. 11
  • The untrained student’s matched-cap rerun leaves the accuracy delta at +0.0497 and increases the evidence delta from +0.0586 to +0.0622.Only the baseline was re-decoded at 900 tokens; the earlier comparison had used a 700-token cap for that baseline.Source: §8 · pp. 11–12
  • RSFT uses 6,251 training and 687 validation traces; label-only retains all rows with 6,477 training and 714 validation examples.The data are split by document into 381 training and 42 validation documents; label-only training ran 810 steps in 2.63 hours.Source: §8 · p. 12

9 Relation to prior work

The paper situates RSFT among filtered self-training approaches and contrasts its source-grounding selection with prior methods. It concludes that both verdict-only methods recover evidence quality in this single-run comparison, while outlining extensions needed to test scope and robustness.

9 Relation to prior work

  • RSFT is one round of verdict-filtered, source-grounded rejection sampling, while label-only SFT is the direct-answer baseline.Its two gates filter candidates by the recorded verdict and then rank survivors by an automatic grounding score.Source: §9 · p. 12
  • Citation-quality evaluation differs from prior work by comparing citations with withheld human evidence spans and measuring verbatim source verifiability.The cited prior work evaluates support for free-text claims using model or NLI judgments.Source: §9 · p. 13

10 Conclusion

  • Both methods recover evidence quality without evidence annotation, but the single-seed, single-corpus comparison is a starting point rather than a ranking.The roadmap calls for three seeds per arm and a second student before treating point estimates as headline results.Source: §10 · p. 13, §11 · p. 13

11 Roadmap and future work

  • The authors identify stacking, iteration, and preference finishing as untested combinations, alongside testing citation behavior under minimal prompts.Other proposed extensions include semantic span matching, additional corpora, and verifier and lookup ablations.Source: §11 · p. 13

Limitations

The comparison is limited to one corpus, one student, and one seed per arm; its intervals capture test-row variation, not training-initialization variation. Other caveats concern prompt-dependent citations, evidence scoring, possible pretraining exposure, and bundled RSFT components.

Limitations

  • One corpus, one student, and one seed per arm limit the comparison; test-row intervals do not measure variation across training initializations.The second seed per arm remains outstanding, and the label-only versus RSFT accuracy gap is a single-seed estimate.Source: Limitations · p. 13
  • Label-only citation behavior is prompt-elicited rather than weight-resident, and its expected minimal-prompt difference from RSFT has not been tested.Under the study’s faithful prompt, label-only cites at 93.4%, so the practical difference is nil under that protocol.Source: Limitations · p. 14
  • Evidence F1 partly measures locatability because selection and evaluation share a three-tier string-matching ladder; semantic credit remains future work.The locatable-subset result shows the ranking survives removing the shared instrument.Source: Limitations · p. 14
  • Pretraining exposure to ContractNLI cannot be excluded, including possible memorized evidence; paired comparisons do not eliminate contamination.The concern is most pressing for label-only’s 0.896 accuracy, and a closed-book probe is left to future work.Source: Limitations · p. 14
  • RSFT’s component effects are not isolated: verdict filtering, grounding selection, self-training, and sampler contribution are evaluated as one pipeline.Source: Limitations · p. 14

A Prompts and generation

Appendix A.1 contrasts the full evidence-requesting prompt with a verdict-only variant. Both present the contract and hypothesis and require a classification, while the faithful prompt additionally requests quoted evidence.

A.1 The two prompts

  • The faithful prompt is the full reasoning-and-citation prompt used by the main system and every evidence-axis evaluation.Source: §A.1 · p. 15
  • The faithful prompt requests evidence as contract quotes, with a piece defined as a sentence or list item.Source: §A.1 · p. 15
  • Faithful-prompt evidence should be concise, self-contained, quoted verbatim, and empty for NotMentioned answers.Source: §A.1 · p. 15
  • The faithful prompt requests step-by-step reasoning and a JSON answer block containing the verdict, explanation, confidence score, and source-attribution excerpts.Source: §A.1 · p. 15
  • The verdict_only prompt removes the evidence paragraph and reduces the requested output to the verdict in a JSON block.Source: §A.1 · pp. 15–16
  • Both prompt variants supply a contract and hypothesis and ask whether the hypothesis is entailed, contradicted, or not mentioned.Source: §A.1 · pp. 15–16

A.2 Candidate generation (RSFT)

RSFT candidate generation uses greedy outputs first and samples only where the verdict filter rejects the greedy result. The candidate-generation cost therefore scales with the remaining rows rather than the full corpus.

A.2 Candidate generation (RSFT)

  • Greedy candidates are generated from both models for every training row; sampled passes run only when the greedy output fails the verdict filter.This makes sampling cost scale with the residue rather than the corpus.Source: §A.2 · p. 16
  • The greedy pass uses k = 1 and T = 0, while the sampled pass uses model-specific settings on rows rejected by the verdict filter.The student uses k = 32, T = 1.0, top-p 0.95, top-k 20; the sampler uses k = 8, T = 1.0, top-p 0.95, top-k 64.Source: §A.2 · p. 16
  • Both models have a 900-token completion cap, and document-major row ordering keeps each contract’s 17 hypotheses together in the prefix cache.The authors attribute much of the 6.8 GPU-hour generation fit to this ordering.Source: §A.2 · p. 16
Loading 2610.06962v1…