PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering

Shicheng Fan, Haochang Hao, Dehai Min, Weihao Liu, Philip S. Yu, Lu Cheng

arXiv:2605.29648v1cs.CL

TL;DR

Factual QA systems struggle with incorrect answers, especially for rare-entity facts, while existing fine-grained rewards can be costly and unreliable. CorVer uses Wikipedia co-occurrence statistics for lightweight sentence-level process supervision, improving over Raw on every cell and outperforming prior pipelines in 18 of 20 cells while training 4.8 to 8.4× faster.

  • Problem

    Knowledge-intensive QA systems often produce incorrect answers, with rare-entity facts disproportionately misrecalled.

  • Method

    CorVer converts Wikipedia subject-object co-occurrence statistics into sentence-level rewards and aligns them to token-level returns without a neural verifier.

  • Results

    CorVer improves over Raw on every cell and outperforms four prior pipelines in 18 of 20 cells while training 4.8 to 8.4× faster.

  • Takeaways & Limitations

    The results support corpus-grounded process rewards as a lightweight alternative to neural-verifier rewards for factual QA reinforcement learning.

  • Takeaways & Limitations

    Because CorVer captures entity co-occurrence rather than predicate semantics, it cannot detect incorrect relations between correctly co-occurring entities.

Abstract

from arXiv · show

Applying reinforcement learning to improve factual accuracy in knowledge-intensive question answering faces a reward design dilemma. Response-level rewards provide only coarse supervision and cannot distinguish correct from incorrect statements within a reasoning trace. Sentence-level alternatives offer finer-grained feedback, but typically rely on NLI verifiers, LLM judges, or knowledge-verification pipelines that are expensive to deploy at RL scale and often unreliable for rare-entity facts, where accurate reward signals are especially important. We propose CorVer (Corpus Verify), a lightweight, plug-in-ready process reward that replaces neural verifiers with a corpus-grounded signal derived from Wikipedia co-occurrence statistics. CorVer assigns sentence-level credit and maps it to token-level advantages via a simple alignment, requiring only a 0.5B extractor and a single corpus lookup per sentence. Across 30 (model, benchmark) cells spanning six instruction-tuned models (3B to 14B) and five QA benchmarks, CorVer improves over the raw baseline for every cell, with an average TriviaQA gain of +4.1 pp. It also outperforms four neural-verifier baselines in 18 of 20 cells under their feasible configurations, while training 4.8 to 8.4x faster.

1 Introduction

Factual QA lacks the cheap, scalable sentence-level rewards available for math and code, while neural-verifier approaches improve credit assignment at substantial reward cost. CorVer addresses this gap with a Wikipedia co-occurrence process reward requiring only a 0.5B extractor and one indexed lookup per sentence, yielding consistent improvements across evaluated settings.

  • Motivation: Factual recall is disproportionately unreliable for rare-entity facts because it tracks subject-object co-occurrence frequency in pretraining corpora.This distinguishes factual QA from math and code tasks, which have cheap, deterministic programmatic rewards.
  • Motivation: Neural-verifier methods improve sentence-level credit assignment but create a reward-cost bottleneck because every sentence in each rollout requires verification.The cited approaches include NLI entailment, knowledge-base verification, and process reward models.
  • CorVer: CorVer assigns training-time process rewards by extracting subject-object pairs from generated sentences and querying a Wikipedia co-occurrence index built with Infini-gram.Each sentence requires one 0.5B extractor forward pass plus one indexed lookup, rather than a neural verifier.
  • Results: 30 (model, benchmark) cells show consistent improvements, while CorVer outperforms four neural-verifier baselines in 18 of 20 cells under feasible configurations.The evaluation spans six models from 3B to 14B and five factual QA benchmarks.

2 Related Work

Prior work contrasts outcome-level RL with finer-grained process supervision for factual QA, where credit assignment remains difficult. Factuality-aware methods add retrieval, verification, or token-level signals, whereas CorVer uses Wikipedia co-occurrence statistics without a trained PRM or stepwise labels.

  • Outcome-Level RL and Process Supervision: Outcome-level rewards in factual QA struggle to assign credit when correct answers coexist with unsupported context.This credit-assignment issue motivates more local supervision.
  • Outcome-Level RL and Process Supervision: CorVer follows process-supervision intuition without training a PRM or using stepwise labels, deriving its local signal from Wikipedia co-occurrence statistics.Its approach is presented as an alternative to explicit process-reward modeling.
  • Factuality Rewards in RL: Factuality-aware RL methods incorporate external knowledge or verification through retrieval-augmented evidence, statement-level signals, knowledge verification, or step-wise NLI.Examples include FoRAG, RLFH, KnowRL, and FSPO.
  • Factuality Rewards in RL: CorVer provides a lightweight factual-support signal rather than a truth label, with no retrieval or entailment in the reward loop.Inference-time grounding through RAG or FActScore is orthogonal to this training-time signal.

3 Method

CorVer extracts a subject–object relation from each sentence, queries Wikipedia co-occurrence statistics, and maps the resulting sentence-level credit to tokens. It combines this signal with answer-judging and formatting rewards in a unified per-token return for policy-gradient training.

  • Per-Token Rewards Calculation: Rt(x, y) = Rr(x, y) + 1[σ(t) > 0] · λc · rc_i assigns sentence credit to aligned tokens while leaving tags and inter-sentence whitespace with only response-level return.The policy uses group-normalized token-level advantages; setting λc = 0 recovers the response-level baseline.
  • Sentence-Level Co-occurrence Reward: CorVer extracts the first valid subject–relation–object triplet, reduces entities to content words, and submits their union as a Wikipedia word-level AND query.The extractor is QuCo-extractor-0.5B, and queries use an Infini-gram index.
  • Sentence-Level Co-occurrence Reward: Wikipedia co-occurrence counts are converted into bounded, piecewise sentence rewards whose empirical correctness proxy increases monotonically with the count.The bounded mapping shapes sentence-level credit without overriding response-level correctness rewards.
  • Sentence-Level Co-occurrence Reward: A 0.5B extractor forward pass and a single indexed CNF lookup per sentence suffice to compute CorVer, with no neural reward model.The Wikipedia snapshot is queried only during training, so CorVer adds no inference cost.
  • Response-Level Rewards: The response-level component combines lenient string-match answer grading with a formatting reward based on the required think/answer tags.Answer labels GOOD, BAD, and NA map to scalar rewards with rgood > 0 > rbad, rna.

4 Experimental Setup

The experiments evaluate CorVer on five knowledge-intensive QA benchmarks using instruction-tuned models, with comparisons against Raw and four factuality-RL baselines. Evaluation uses factual QA accuracy plus diagnostic metrics, under specified reward calibration and training settings.

  • Benchmarks and models: Five benchmarks—TriviaQA, NQ-Open, PopQA, SimpleQA, and TruthfulQA—are evaluated, with training prompts restricted to NQ-Open and WebQuestions while the other benchmarks remain out-of-distribution.The benchmarks contain 17,944, 3,610, 14,267, 4,326, and 817 questions, respectively.
  • Benchmarks and models: The headline comparison uses Llama-3.1-8B-Instruct and Qwen3-8B, while scaling extends across six Llama-3, Qwen3, and OLMo models from 3B to 14B.The headline group also supports ablation and cost analyses.
  • Baselines: CorVer is compared with Raw and four factuality-RL baselines: FoRAG, RLFH, FSPO, and KnowRL.Because these baselines invoke neural verifiers or external services per sentence, they are trained under reduced configurations.
  • Metrics: Primary evaluation measures factual QA accuracy using substring and alias matching with lenient regex parsing; NA rate, format-success rate, and average answer length are diagnostics.These metrics assess answer correctness, response validity, and output length.
  • Implementation details: The co-occurrence reward uses (α0, α1, α2, α3) = (−0.3, −0.1, 0.0, +0.1) and (τ1, τ2) = (5, 20), while all runs use LoRA, G = 16, maximum length 1024, and 100 GRPO steps.CorVer trains directly on the raw instruction-tuned model without SFT cold-start; the co-occurrence contribution remains below the judge reward swing.

5 Experiments

Across factual QA experiments, CorVer improves over Raw in every evaluated cell and outperforms prior reward pipelines in most feasible comparisons. Its corpus-grounded reward also generalizes across model families and scales while substantially reducing training time.

  • Baseline comparison: +4.06 pp average is CorVer’s largest gain over Raw on Llama-3.1-8B, while every cell improves.Llama-3.2-3B gains +2.34 pp on average, and NQ-Open shows the strongest per-benchmark improvement across all four models.
  • Baseline comparison: 18 of 20 (model, benchmark) cells favor CorVer over four prior methods under feasible training configurations.Baselines use reduced configurations, so the comparison targets deployable reward designs rather than matched computational budgets.
  • Scaling and generalization: 30 of 30 (model, benchmark) cells improve over Raw across six instruction-tuned models from 3B to 14B and five benchmarks.Gains concentrate on TriviaQA, NQ-Open, and PopQA; SimpleQA and TruthfulQA gains are smaller but consistently positive.
  • Scaling and generalization: Qwen3 gains mainly reflect answering previously refused questions, whereas Llama gains combine higher recall with selective abstention.Refusal drops sharply on Qwen3 and rises modestly on Llama.
  • Training efficiency: 3.2 training hours is CorVer’s average across four models, versus 14.5–29.5 hours for baselines, making them 4.8–8.4× slower.CorVer uses a 0.5B forward pass and one corpus index lookup per sentence; FSPO reaches 65.8 hours on Qwen3-8B.

6 Analysis and Discussion

Analysis shows that co-occurrence counts provide a directionally reliable factuality signal, whose benefit depends on per-token alignment and corpus coverage. Ablations and practitioner observations identify conditions affecting CorVer’s effectiveness, while its validation remains limited to GRPO.

  • Signal validation: P(correct | ci) rises from 23% at ci = 0 to 81% at ci ≥20, supporting co-occurrence count as a monotonic proxy for sentence correctness.The largest jumps are +17 pp at ci = 5 and +8 pp at ci = 20; ci = 10 yields only +3 pp and is unused.
  • Component ablations: 76.52 TriviaQA accuracy for the full method exceeds 71.3 for A1 and 72.9 for A3, showing that co-occurrence rewards and per-token alignment both matter.A1 removes QuCo, while A3 retains its magnitude as a response-level scalar; A2 nearly matches the full method on TriviaQA but falls to 42.6 versus 48.34 on NQ-Open and 31.7 versus 35.30 on PopQA.
  • Popularity analysis: OLMo’s quartile gains increase with popularity (+3.68 → +4.33 → +5.51 → +9.03), while both models improve in every cell, favoring corpus coverage over rare-entity rescue.Llama’s gains are +5.47 → +5.13 → +8.39 → +7.50; the largest improvements occur on Q3 and Q4, while rare entities receive the least informative signal.
  • Aggregation rules: The canonical FIRST rule is fastest and most accurate: MIN shortens completions, whereas RELCHECK often returns zero for correct claims and increases training cost.FIRST keeps only the first valid triplet and uses an entity-only lookup; relation surface-form variation makes literal relation-token queries brittle.
  • Engineering observations: SFT cold-start hurt factual recall, which the authors hypothesize reflects a capacity mismatch between smaller students and teacher-generated reasoning chains.The observation came from SFT using traces from a 397B-parameter Qwen3-family MoE teacher, and the SFT model underperformed the raw-model recipe.
  • Engineering observations: Small models needed fullycorrect anchor questions to stabilize learning-zone training, whereas models ≥8B did not exhibit this issue.Llama-3.2-3B-Instruct and Qwen3-4B deteriorated without anchors; these mechanisms are practitioner hypotheses rather than controlled-experiment conclusions.

7 Conclusion

CorVer is a corpus-grounded process reward that replaces neural verifiers with a 0.5B extractor and one Infini-gram lookup per sentence. Across six models and five factual QA benchmarks, it improves over Raw on every cell, outperforms prior factuality-RL pipelines in 18 of 20 cells, and trains 4.8 to 8.4× faster.

  • 7 Conclusion: CorVer replaces neural verifiers with a 0.5B extractor and one Infini-gram lookup per sentence.It is presented as a corpus-grounded process reward.
  • 7 Conclusion: 18 of 20 cells: CorVer outperforms four prior factuality-RL pipelines while training 4.8 to 8.4× faster.The comparison spans six instruction-tuned base models from 3B to 14B and five factual QA benchmarks.
  • 7 Conclusion: Every cell: CorVer improves over Raw across six instruction-tuned base models and five factual QA benchmarks.The paper attributes the gain to a monotonic correlation between per-sentence factual correctness and Wikipedia co-occurrence, alongside PopQA per-quartile improvements that track

Limitations … A.3 Generation Prompt

The paper’s limitations center on CorVer’s corpus- and entity-pair-based proxy signal, evaluation grader, and asymmetric baseline configurations. The appendices specify the training recipe, datasets, filtering protocol, and distinct training-versus-evaluation prompts used across experiments.

  • Limitations: CorVer cannot detect factually wrong predicates when the correct subject and object co-occur, because extraction captures only the entity pair.Its signal is monotonically calibrated against human annotations but is not a claim-level fact-checker.
  • Limitations: Performance gains correlate with Wikipedia coverage, making CorVer least informative for rare entities in sparse-coverage regions.The indexed corpus is a 6.4M-article, 5.5B-token English Wikipedia snapshot.
  • Limitations: Lenient substring-plus-alias grading may overestimate absolute accuracy, although paired table comparisons use the same grader.An LLM judge is used only for offline validation, not within the reward loop.
  • A.1 Full Hyperparameters: All baseline methods train on the same 10,700-prompt, three-phase curriculum, whereas CorVer uses a self-filtered pool of 4,329–5,608 prompts per target.The baselines differ in LoRA rank and rollout structure, and only KnowRL matches CorVer’s LoRA rank.
  • A.1 Full Hyperparameters: CorVer uses LoRA r = 128/α = 256 and G = 16 rollouts, while baselines use reduced configurations because their neural-verifier or LLM-judge rewards are orders of magnitude costlier.CorVer’s reward requires one 0.5B extractor forward and one mmap Infini-gram lookup per sentence.
  • A.2 Datasets and Evaluation Protocol: Evaluation covers five closed-book QA datasets: TriviaQA, NQ-Open, PopQA, SimpleQA, and TruthfulQA.The corresponding validation or test sets contain 17,944, 3,610, 14,267, 4,326, and 817 questions, respectively.
  • A Training Setup and Recipe: CorVer’s training setup uses target-specific self-filtering, retaining prompts with 1 ≤ ncorrect ≤ 15 among 16 sampled completions.This excludes questions the raw model always answers correctly or never answers correctly, unlike the baselines’ static capability cascade.
  • A.3 Generation Prompt: Training uses a light format prompt with anti-loop and full-entity-name rules, while evaluation uses a lighter format-and-IDK prompt plus anti-loop text for trained checkpoints.Evaluation uses greedy decoding with temperature = 0.0 and max new tokens = 1000; adding anti-loop text to raw-model evaluation slightly decreased accuracy.

A.4 Per-Model Training Set Sizes after Self-Filter … B.4 CorVer Algorithm

The appendix details CorVer’s self-filtered training pools, lightweight corpus-grounded reward implementation, penalty calibration, aggregation variants, and the resulting GRPO procedure. It also explains why FIRST is preferred over MIN and RELCHECK for efficient, stable sentence-level supervision.

  • A.4 Per-Model Training Set Sizes after Self-Filter: The self-filter retains questions with 1–15 correct completions among 16 samples, defining the learning zone used for training.Questions with zero or 16 correct completions are excluded from the retained training set.
  • A.4 Per-Model Training Set Sizes after Self-Filter: 4,329–5,608 questions remain in the learning zone across models, determining the number of distinct prompts available per GRPO step.The retained range spans Qwen3-4B to OLMo-2-13B-Instruct.
  • B.1 Co-occurrence Reward Implementation Details: A 0.5 alignment threshold triggers fallback to response-level returns, while valid sentence signals score both reasoning and answer blocks through token alignment.The index uses English Wikipedia and performs direct local CNF lookups at millisecond-scale latency.
  • B.2 Zero-Count Penalty Sensitivity: −6.55 pp is the worst TriviaQA change for the −0.2 penalty, while all tested weaker and stronger penalties underperform the canonical −0.3 setting.Each value comes from a separate full CorVer retraining of Llama-3.2-3B-Instruct on TriviaQA validation.
  • B.3 Sentence-Level Aggregation and Relation-Aware Demotion: Mechanism Details: MIN collapses mean completion length from approximately 150 tokens under FIRST to 35–46 tokens, because one weak triplet can determine the entire sentence reward.RELCHECK instead adds relation words only when the entity-only count is at least 20 and demotes +0.1 to −0.05 when the relation-augmented count is zero.
  • B.4 CorVer Algorithm: Each CorVer step samples 16 learning-zone completions, scores sentences jointly across reasoning and answer blocks, aligns rewards to tokens, and applies GRPO normalization and clipping.The procedure assembles token-level raw returns before the group-normalized clipped-surrogate update.

B.5 Human Audit of the Four-Tier Co-occurrence Reward

A 700-sentence human audit supports co-occurrence count as a directional proxy for sentence-level factuality and validates the reward’s ci = 5 and ci = 20 bucket boundaries. It also exposes entity-only limitations: rare correct facts are under-credited, while wrong predicates can receive high co-occurrence support.

  • Audit design: 700 sentences from Llama-3.1-8B-Instruct TriviaQA generations covered all five precision regimes and tested the ci = 0, 5, and 20 boundaries.The audit sampled both ⟨think⟩ and ⟨answer⟩ blocks in proportion to their reward-stream frequency.
  • Extreme-bucket audit: 81.0% of high-frequency sentences were correct, while 76.0% of zero-frequency sentences were incorrect, matching the reward directions.These results support co-occurrence frequency as a directional factuality proxy at the two extremes, not equivalence with a predicate-aware verifier.
  • Five-bucket calibration: Precision increased monotonically from 24.0% at ci = 0 to 81.0% at ci ≥20 across the five buckets.The ci = 5 and ci = 20 boundaries corresponded to +17.0 pp and +8.0 pp precision jumps, respectively; the largest transition was +29.0 pp across ci = 0.
  • Residual errors: 24% of zero-frequency sentences were correct and 19% of high-frequency sentences were incorrect, revealing residual errors from the entity-only proxy.The first cases are rare-but-correct facts under-credited by bucketing; the second are well-supported entity cooccurrences attached to wrong predicates.

B.6 Qualitative Case Study … C.1 Checkpoint Selection

The qualitative study shows that corpus co-occurrence rewards can catch fabricated entity facts missed by an LLM judge and trace sentence-level credit through complete TriviaQA answers. The paper also uses GPT-4o-mini only for offline validation and evaluates every CorVer run at the uniformly selected step-100 checkpoint.

  • B.6 Qualitative Case Study: Three zero-frequency examples show co-occurrence lookup returning zero for fabricated subject-object pairs that GPT-4o-mini confidently affirms.The cases are isolated single-sentence examples rather than complete model completions.
  • B.6 Qualitative Case Study: 25 of 152 human-confirmed incorrect zero-frequency sentences expose cases where the LLM judge misses factual errors.The examples involve a wrong zip code, cast member, or misspelled surname, while Infini-gram assigns rc_i = −0.3 to the fabricated pair.
  • B.6 Qualitative Case Study: Two complete Llama-3.1-8B-Instruct TriviaQA traces illustrate a correctly flagged fabricated proper name and the first-triplet versus multi-triplet trade-off.Case 1 assigns c_i = 0 and −0.3 to the fabricated name, while Case 2 provides a clean positive trace; the trade-off is resolved in favor of FIRST.
  • B.6 Qualitative Case Study: Case 1 receives +0.1 for the extracted Flyers–Stanley Cup and team–Buffalo Sabres triplets despite wrong subordinate game-date details.The MIN aggregation preserves credit for the correct extracted triplet even when surrounding sentence details are incorrect.
  • B.6 Qualitative Case Study: Case 2 assigns +0.1 to extracted triplets covering Michigan’s leadership by Glen Rice, its defeat of Seton Hall, and the King Dome location.The trace identifies these facts as correct, including the 80–79 result and Seattle venue.
  • B.7 Optional LLM Judge: GPT-4o-mini is used only as an offline validation judge, receiving sentence text without the gold answer or surrounding QA context.Its outputs are parsed by a strict case-insensitive CORRECT / INCORRECT match.
  • C.1 Checkpoint Selection: Every CorVer run is evaluated only at step 100 under one uniform end-of-run selection policy.Step-50 checkpoints are saved but not graded, and checkpoint selection uses neither a held-out development split nor a per-model best-checkpoint sweep.

C.2 Full Per-Model Results · A Prompt question: What is the zip code for Ronkonkoma, NY? Gold answer: 11779 · C Prompt question: What is Ella Fitzgerald’s parents’ name? Gold answer: William Fitzgerald

The full per-model evaluation reports Raw and step-100 CorVer accuracy and refusal rates across five closed-book QA datasets. Illustrative examples show corpus-grounded rewards identifying factual errors involving rare or incorrect entity relations.

  • C.2 Full Per-Model Results: Table 15 reports Raw and step-100 CorVer accuracy and NA rates for each canonical model across five closed-book QA datasets.Each cell reports accuracy | NA, with CorVer using the canonical step-100 checkpoint.
  • A Prompt question: What is the zip code for Ronkonkoma, NY? Gold answer: 11779: The examples demonstrate that extracted entity-relation triples can expose mismatches between generated claims and gold answers in factual question answering.The mismatches include an incorrect zip code and an incorrect surname attribution.
  • C Prompt question: What is Ella Fitzgerald’s parents’ name? Gold answer: William Fitzgerald: The Ella Fitzgerald example attributes her mother to Temperance Mary Tempie Height Fitzgerald, while the cited correction states that the surname is Henry, not Height.The gold answer for the prompt is William Fitzgerald.
  • C Prompt question: What is Ella Fitzgerald’s parents’ name? Gold answer: William Fitzgerald: The zero-frequency examples show that the co-occurrence reward rc_i = −0.3 can flag factual errors that GPT-4o-mini does not identify as an LLM judge.Figure 7 presents three illustrative single sentences sampled from the zero-frequency bucket.
  • C.2 Full Per-Model Results: The reported evaluation compares accuracy without and with CorVer across five factual QA datasets for each instruction-tuned base model.Table 16 marks the better result within each Raw–CorVer pair, while Table 15 additionally provides refusal-rate diagnostics.

D Auxiliary Findings and Practical Lessons

Auxiliary runs identified practical sensitivities in CorVer training: intermediate SFT cold-start degraded TriviaQA recall, while anchor questions stabilized small-model training. Prompting and data composition also affected outcomes, but the reported mechanisms remain tentative rather than controlled conclusions.

  • Caveats: The reported explanations are tentative: the SFT degradation may reflect capacity mismatch, data composition, or hyperparameter choices rather than a general effect.The paper explicitly states that these mechanisms were inferred from preliminary runs, not validated by controlled experiments.
  • SFT cold-start: 7–18 pp: SFT-only checkpoints lost this much TriviaQA accuracy versus the raw instruction-tuned baseline on every tested target.Subsequent GRPO recovered part of the gap for Qwen3-8B and Qwen3-4B but did not reach the raw baseline.
  • Prompting effects: Raw models’ accuracy decreased slightly on every target when an anti-loop instruction was appended, despite no observed looping.The authors report opposite effects for raw and CorVer-trained policies, attributing the raw-model decrease tentatively to added prompt complexity.
  • Small-model stability: Training collapsed for Llama-3.2-3B-Instruct and Qwen3-4B when trained only on learning-zone questions with ncorrect ∈[1, G −1].Accuracy degraded over training rather than improving in preliminary runs.
  • Small-model stability: 1,000 anchors—or 800 for Qwen3-4B—stabilized small-model runs and yielded the gains reported in Table 1.Although formally redundant because they carry no within-group GRPO advantage, anchors were interpreted as distributional anchors keeping policies near raw-model responses.
Loading 2605.29648v1…