PaperlayerView exact revision on arXiv

Source-linked AI summary

Copying Is Not Grounding: Training Faithful Long-Form RAG with Authored-Sentence Credit

Hoang Pham, Duc Anh Vu, Duy Tung Doan, Cong-Duy T Nguyen, Anh Tuan Luu

arXiv:2610.18442v1cs.CL

TL;DR

Long-form RAG faithfulness rewards can be raised by copying, even though the goal is to make a model’s own claims reliable. ASC credits authored, non-vacuous sentences supported by passages and compares each sentence with all sentences sampled for the question. Across four models and five corpora, it lowers authored-sentence error while writing more supported sentences, including on held-out corpora.

  • Problem

    Faithfulness rewards for long-form RAG can favor copying, while the goal is to improve the reliability of what a model writes itself.

    Source: §1 · p. 1

  • Method

    ASC credits sentences only when they are authored, non-vacuous, and supported by passages, then compares each sentence’s credit across all sampled sentences for the question.

    Source: §1 · p. 2

  • Results

    At the median, ASC cuts authored-sentence error by 38% and writes 46% more supported sentences of its own, with gains across four models and held-out corpora.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A faithfulness reward should credit what a model writes, not what it copies.

    Source: §7 · p. 9

    ASC’s tests are only as reliable as the shared reward checker, and the experiments train models up to 9B parameters, leaving larger models for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In search engines and assistants, long-form RAG answers can mislead readers with claims their sources do not support. Faithfulness rewards, the share of supported sentences, train models toward fewer such claims. We show that these rewards are hacked by copying. A sentence copied from the passages is supported almost by construction, so trained models learn to copy rather than ground what they write. Common defenses only move the hack: a KL or copy penalty leaves copying in place or trades it for shorter, emptier answers and refusals. GRPO, the standard RL algorithm, makes it worse by giving every sentence its answer's advantage, reinforcing copied and grounded sentences alike. We propose Authored-Sentence Credit (ASC) , which fixes both by moving GRPO's reward and advantage from the answer to the sentence level. The reward credits only sentences that are authored rather than copied, say more than the question, and are supported by the passages. The advantage compares each sentence with every sentence sampled for the question, so one that earns no credit ranks last and is never reinforced. Across four models and two in-domain, three held-out corpora, only ASC lowers the error rate of authored sentences and writes more supported ones in every setting, by 38% and 46% at the median under held-out evaluators, giving readers more grounded information. It takes none of these shortcuts, and two LLM judges comparing whole answers prefer no baseline to the base model but rate ASC's answers on par with it. We conclude that a faithfulness reward should credit what a model writes, not what it copies.

1 Introduction

Faithfulness rewards can be raised by copying, even when a model’s own sentences remain poorly grounded. ASC instead credits supported, non-vacuous sentences the model authors, and improves authored-sentence reliability and content without the reported shortcuts or loss of answer quality.

1 Introduction

  • On ASQA, authored sentences are unsupported 3.7 times as often for Qwen3.5-9B and 3.2 times as often for Llama-3.1-8B as copied sentences.The reward rises when a model copies instead of writing, so its score can reward a shortcut rather than better grounding.Source: §1 · p. 1
  • GRPO raises Qwen3.5-9B’s copy share from 38.8 to 72.8 while shrinking its answers from 74 to 14 words.KL and copy penalties shift the hack toward shorter, emptier answers or refusals; answer-level GRPO advantages also reinforce copied and grounded sentences alike.Source: §1 · pp. 1–2
  • ASC credits only authored, non-vacuous sentences supported by the passages, then compares each sentence’s credit with all sentences sampled for the question.Copied, vacuous, or unsupported sentences rank last in their group and are never reinforced.Source: §1 · p. 2
  • At the median, ASC cuts authored-sentence error by 38% and writes 46% more supported sentences of its own, with no shortcut or loss of answer quality.Two LLM judges rated ASC on par with the base model and above every baseline.Source: §1 · p. 2

2 Related Work

Prior work documents how faithfulness and related factuality objectives can favor copying or shorter answers. The paper studies this bias as reward hacking in long-form RAG and distinguishes ASC by withholding credit from copied sentences.

2 Related Work

  • Faithfulness metrics can favor extractive output and copying, while entailment rewards can trade informativeness for consistency [Ladhak et al., 2022; Dreyer et al., 2023; Roit et al., 2023].The paper connects this trade-off to reader costs in generative search, where more precise citations accompany more copying and lower perceived utility.Source: §2 · p. 2
  • Factuality metrics and rewards can favor short, less detailed answers, and a KL penalty does not prevent hacking under heavy-tailed reward error.Prior work also describes reward optimization amplifying reward biases (Shapira et al., 2026).Source: §2 · p. 2
  • Unlike prior sentence or fact rewards for support, ASC asks whether each sentence was copied and withholds credit from copied and vacuous sentences.It compares sentences rather than answers, so uncredited sentences are never reinforced.Source: §2 · pp. 2–3

3 Problem Formulation and Setup

The study defines success as making a model’s authored sentences more reliable without relying on copying, shorter answers, vacuity, or refusals. It evaluates reliability, supported authored content, shortcuts, and answer quality across four models and five corpora.

3 Problem Formulation and Setup

  • The faithfulness reward is the share of answer sentences supported by passages, as judged by checker g against threshold κ.GRPO gives every token in an answer the same group-standardized reward advantage, whereas DPO learns offline from answer pairs ranked by reward.Source: §3 · p. 3
  • A sentence is copied when at least a fraction τ of its words fall within verbatim passage matches of at least m words; it is vacuous when the question alone passes the support test.An authored sentence is grounded when the passages support it.Source: §3 · p. 3
  • The target is to lower the authored error rate, the share of authored sentences unsupported by passages, without increasing copying, vacuity, or refusals or reducing answer length.The base-model authored error rate ranges from 2–40%, while the faithfulness reward can be raised through copying, writing less, or writing emptier sentences.Source: §3 · p. 3
  • Reliability uses a held-out Gemma-4-31B checker, while content measures supported authored sentences per answer with held-out MiniCheck-Flan-T5-Large.Copying, vacuity, refusals, and answer length detect shortcuts; AlignScore, exact match, and two whole-answer LLM judges provide additional checks.Source: §3 · p. 3
  • Four instruction models from two families are trained on ASQA and RAGTruth and evaluated on those corpora plus held-out ClapNQ, ELI5, and ExpertQA.The evaluation covers 20 model–corpus settings; ELI5 and ExpertQA are settings where base models err most and copy least.Source: §3 · p. 3

4 Why Faithfulness Rewards Are Hacked

Faithfulness rewards favor copied sentences because they are less often judged unsupported than authored ones. Training can exploit this gap through copying, reduced content, or vacuity, while answer-level GRPO advantages do not reliably distinguish copied from grounded sentences.

4 Why Faithfulness Rewards Are Hacked

  • Authored sentences are unsupported 2.3 to 4.4 times as often as copied ones on average, and human labels on RAGTruth responses find 12.2% unsupported authored sentences versus 1.9% copied.Two held-out checkers find the gap almost everywhere; Figure 2 reports unsupported rates and their ratio across checker and human evaluations.Source: §4 · p. 4, Figure 2 · p. 4
  • Plain GRPO reaches reward 98.9 as answers shrink from 74 to 14 words, while the copy penalty yields 67.4% vacuous sentences and refusals to 64.5% of questions.The KL penalty holds copying at the base level but shifts toward shorter, emptier answers and more refusals; the defenses take a shortcut on nine in ten model–corpus pairs.Source: §4 · p. 4
  • GRPO pushes up 72.0% of copied sentences and 75.5% of grounded ones, and even 58.8% of vacuous ones, because its advantage belongs to the whole answer.Grounded sentences are pushed up more often everywhere, but by at most 17 points, so answer-level advantages provide little steering away from copying.Source: §4 · p. 4

5 ASC: Authored-Sentence Credit

ASC changes GRPO so sentences earn credit for being authored, non-vacuous, and supported, then receive advantages based on their own credit. Its training objective applies each sentence’s advantage to that sentence’s tokens while leaving the rest of GRPO unchanged.

5.1 Which sentences earn credit

  • Sentence credit combines tests for copying, vacuity, and passage support; sentences that fail a test earn zero, while support receives graded credit.The support credit is zero at or below the checker threshold and reaches one at certainty.Source: §5.1 · p. 5

5.2 A sentence baseline

  • Across sampled answers, sentences are ranked by credit, and any copied, vacuous, or unsupported sentence ranks below every sentence that earns credit.A sentence that fails a test receives the group’s lowest advantage.Source: §5.2 · p. 5

5.3 Training objective

  • Each sentence’s advantage is assigned to its tokens; tokens outside sentences, including whitespace and the end-of-sequence token, receive zero.The objective otherwise remains GRPO’s.Source: §5.3 · p. 6
  • Unlike GRPO’s answer-level update, ASC moves each sentence according to its own advantage, making it GRPO with a finer unit of comparison.ASC needs no critic, reference answers, or penalty weight; its only added checker cost is one question-only call per authored sentence.Source: §5.3 · p. 6

6 Experiments

Across the experiments, ASC lowers authored-sentence error while writing more supported authored sentences, without the shortcuts seen in the baselines. Ablations and robustness checks show this pattern depends on ASC’s components and holds across runs, evaluators, and training.

6.1 Setup

  • All methods share GRPO training data, reward checker, thresholds, and hyperparameters; the baselines are standard GRPO + KL, a copy penalty, and a sentence reward.Evaluation uses greedy decoding on five corpora, with shortcuts defined by increases in copying, vacuity, or refusals, or answers shorter than 85% of the base model’s.Source: §6.1 · p. 6

6.2 Main results

  • ASC cuts authored error rate by 38% at the median and writes 46% more supported authored sentences than the base model; the error reduction is larger on held-out than in-domain corpora.The comparison covers all five corpora and appears in Table 2; Figure 5 shows the error reduction across every model and corpus.Source: §6.2 · pp. 6–7, Table 2 · p. 7, Figure 5 · p. 7

ASC takes no shortcut; the baselines take one

  • ASC avoids the baselines’ shortcuts: GRPO + KL declines up to 43.9% of Qwen questions, the copy penalty declines 79.7% on Qwen3.5-9B, and sentence reward copies more in every setting.In the five settings where a baseline takes no shortcut, its error rate is never significantly below ASC’s, while ASC writes significantly more supported authored sentences in all five.Source: ASC takes no shortcut; the baselines… · p. 7

6.3 Each part of ASC contributes

  • Removing any ASC component brings back a shortcut or raises error rates on most corpora; binary rather than graded credit raises error on all five.Without the copy test, copying returns; without the vacuity test, held-out ExpertQA error doubles from 3.7 to 7.5.Source: Table 4 · p. 8, §6.3 · p. 8

6.4 Robustness

  • ASC’s gains persist across runs, evaluators, and training, and the copy-threshold check finds fewer authored sentences just below threshold than for the base model.Human evaluation found 24.8% of ASC’s sentences unsupported versus 33.2% for the base model; two checkers unused in training also found more supported authored sentences in nearly all settings.Source: §6.4 · p. 8

7 Conclusion

The conclusion argues that faithfulness rewards for long-form RAG can pay for copying, while common defenses shift the shortcut to other behaviors. ASC instead credits authored, non-vacuous, supported sentences and compares each sentence within its sampled group.

7 Conclusion

  • Across four models, ASC lowers authored error while writing more grounded sentences, including on held-out corpora, without shortcuts or loss in answer quality.The conclusion’s broader recommendation is to reward what a RAG model writes rather than what it copies.Source: §7 · pp. 8–9

Limitations

The limitations bound ASC's checker-based tests and training scale, while the formal appendix characterizes how copying affects reward and how sentence-level credit shapes updates. The appendix also specifies the graded-credit behavior and derives the sentence-level gradient.

Limitations

  • ASC relies on its checker for vacuity and support judgments, trains models only up to 9B parameters, and leaves larger models for future work.The authors also note one training run per baseline; gains hold on four models and three held-out corpora.Source: Limitations · p. 9

A Formal Statements and Proofs

  • The formal appendix lays out proofs of the reward decomposition, graded-credit properties, Proposition 1, and the sentence-form gradient.It follows the order of the formal claims in the main text.Source: A Formal Statements and Proofs · p. 12

A.1 The reward is a line in the share of copied sentences

  • At fixed support rates, the reward is affine in copy share, with slope equal to copied support minus authored support; copying raises reward when copied sentences are supported more often.The authors report that this condition holds for every base model and corpus under the reward checker.Source: §A.1 · p. 12

A.2 The graded credit

  • The formal appendix proves that graded credit increases strictly and is concave above its threshold, reaching zero at the threshold and one at full checker confidence.Below the threshold, credit is zero; above it, credit grows fastest near the threshold.Source: §A.2 · p. 12

A.4 The gradient as a policy gradient over sentences

  • The appendix derives the policy gradient by grouping token log-probabilities by sentence, with each token in a sentence carrying that sentence's advantage.Tokens outside sentences drop out; under GRPO, all answer tokens instead share the answer-level advantage.Source: §A.4 · p. 13

A.3 Proof of Proposition 1

The appendix proves that sentence-level advantages rank copied, vacuous, and unsupported sentences last, unlike GRPO's answer-level advantage. Its setup details support findings that copied sentences are more often checker-supported and that reward selection favors copying.

A.3 Proof of Proposition 1

  • A copied, vacuous, or unsupported sentence receives zero credit and the group's lowest advantage, so it is never pushed up by the ASC update.The proof establishes this ranking by applying one strictly increasing normalization to sentence credits across the group.Source: §A.3 · p. 13
  • The proposition does not bound answer length, and a paraphrase below the copy-share threshold counts as authored.The proof's guarantees concern sentences that fail a test, not the number of sentences in an answer.Source: §A.3 · p. 13
  • GRPO's answer-level advantage can reinforce a copied sentence over grounded sentences in a lower-reward answer, while deleting an unsupported sentence can raise the advantage of every remaining token.The appendix's two-answer example demonstrates both effects.Source: §A.3 · p. 13

B Setup Details

  • The setup pools ASQA and RAGTruth for training and holds out ClapNQ, ELI5, and ExpertQA, with training questions disjoint from evaluation sets.The sentence tests classify copying, checker support, and vacuity using passage overlap, support probability, and question-only support, respectively.Source: B Setup Details · p. 14

C.1 Copied versus authored sentences (Finding 1)

  • Authored sentences are unsupported 1.8 to 8.1 times as often as copied sentences across models and corpora, and human span labels show error rates of 12.2% for authored versus 1.9% for copied sentences.Held-out checkers reproduce the gap in nearly all tested cases.Source: §C.1 · p. 15
  • Within a question, higher-reward sampled answers have higher copy share, and both checkers support authored sentences more often when they are closer to passage sentences.The copy preference is reported for every model and corpus; the closeness pattern does not depend on the five-word copy threshold.Source: §C.1 · pp. 15–16

C.2 Training on the faithfulness reward (Finding 2)

Training on the faithfulness reward raises reward through shortcuts rather than more reliable authored sentences, while GRPO's answer-level advantage poorly distinguishes copied from grounded sentences. Across the reported settings, ASC improves authored-sentence reliability and supported content without taking the measured shortcuts.

C.2 Training on the faithfulness reward (Finding 2)

  • Every policy trained on the reward raises it through a shortcut rather than better authored sentences, and baselines either copy more, refuse, or write emptier answers.The appendix reports that plain GRPO leaves 81.0% of answers without an authored sentence; GRPO + KL refuses and writes emptier answers, while the copy penalty declines most questions.Source: §C.2 · p. 16
  • Shortcuts emerge early and persist: GRPO's copy share exceeds 70 by mid-training, while ASC lowers copying and refusals within the first quarter and maintains those reductions.The cited figure passage identifies the ASQA training trajectory for Qwen3.5-9B.Source: §C.2 · p. 16, Figure 6 · p. 17

C.3 Whole-answer and sentence advantages (Finding 3)

  • GRPO pushes copied and grounded sentences up at nearly the same rate among moved sentences, whereas ASC gives copied, vacuous, and unsupported sentences the group's lowest advantage.GRPO leaves a third of all sentences and half of copied sentences unmoved because their answer reward equals the group mean.Source: §C.3 · p. 16
  • Sentence advantages correlate 0.21 with held-out support, versus 0.07 for GRPO's answer-level advantages on the same pool of authored, non-vacuous sentences.Answer length explains less than 2% of either advantage's variance.Source: §C.3 · pp. 16–17

D.2 Counts per answer and significance

  • ASC lowers authored error in every setting, significantly in 14 of 20, and writes 0.3 to 1.0 more judge-supported authored sentences per answer.Its unsupported authored sentences fall in 16 settings and rise significantly in none.Source: §D.2 · p. 17

D.4 Robustness of the shortcut rule

ASC remains unflagged as a shortcut under alternative shortcut-rule thresholds, and its authored error-rate advantage persists across near-copy analyses. Training trajectories also show ASC settling early while baseline shortcuts build over time.

D.4 Robustness of the shortcut rule

  • ASC is flagged in no setting under any tested shortcut-rule margin or word floor.Without thresholds, no baseline combines lower error with at least as many supported authored sentences as ASC in 19 of 20 settings; the exception writes answers half as long.Source: §D.4 · pp. 17–18

D.5 Near-copies

  • ASC's authored error rate remains below the base model's in 58 of 60 closeness-tercile and setting pairs, and it writes fewer near-copies in 17 of 20 settings.ASC and the base model have the same median ROUGE-L precision, 0.42.Source: §D.5 · p. 19

D.6 Training dynamics

  • ASC's copy share falls within the first quarter of training and stays flat, while baseline shortcuts accumulate; ASC ends with longer answers and fewer refusals than the base model.The appendix reports this pattern across every model; GRPO + KL and the copy penalty develop increasing refusals, copying, or vacuous output.Source: §D.6 · p. 19

E Answer Quality

Whole-answer judgments generally favor ASC over the baselines, while answerability and reference checks also show gains; ablations identify sentence-level credit components that matter. The evaluation fixes its rubric before judging trained answers.

E Answer Quality

  • The rubric was fixed before trained answers were judged, treating quoting and length as neither merits nor faults and refusals as right only when passages lack the answer.Its selection used base-model probes for copied text, padding and planted errors.Source: E Answer Quality · pp. 20–21
  • ASC wins significantly in 14 settings over the copy penalty, 9 over GRPO + KL, and 3 over the sentence reward, while each baseline wins in at most 2.Against the base model, ASC is significantly preferred in 5 settings, versus 2 for the base model; controlled-for-length win rates are 48.9%, 50.1%, 54.5% and 49.2% against the base model, GRPO + KL, copy penalty and sentence reward, respectively.Source: E Answer Quality · p. 21
  • ASC covers the most reference key points on average, significantly exceeding the base model in 4 of 12 settings and falling below it in none.TRUE agrees with Gemma on 90.7% of key points and gives the same direction, +1.3 points.Source: E Answer Quality · p. 21
  • ASC answers most questions the base model rightly refuses with 5.5% authored error, though the judge prefers refusal in 32.3% of those pairs.Where both answer, ASC wins 51.5%; the copy penalty has 47.2% of its 3,821 refusals on questions whose passages answer fully or partly.Source: E Answer Quality · p. 21
  • On answerable questions, ASC has lower authored error in 19 of 20 settings and significantly more supported authored sentences in all 20.The answerability labels come from the judge, with Mistral agreeing on 84% of 300 sampled questions.Source: E Answer Quality · p. 21

F Ablation Details

  • Removing the copy test or graded credit, or using answer-level advantage, raises authored error or restores copying on at least four of five corpora.Removing the vacuity test raises error on ASQA, ELI5 and ExpertQA, where the base model writes the most vacuous sentences, but lowers it slightly on RAGTruth and ClapNQ.Source: F Ablation Details · p. 21

G Human Evaluation

Human annotation confirms lower unsupported-sentence rates for ASC and favors its answers over the base model and GRPO + KL. The prompt protocol fixes how the answer and judges are applied.

G Human Evaluation

  • The sentence judge agrees with annotators at least as often on ASC sentences as on base-model sentences: 90.8% versus 85.5%.The validity set used base-model and trained answers on ASQA and ELI5, in both model families.Source: G Human Evaluation · p. 21
  • 24.8% of ASC’s authored sentences are unsupported, versus 33.2% for the base model in the direct ELI5 comparison.The difference is significant (exact McNemar p = 0.01); annotators agree on shared sentences at Cohen’s kappa 0.65.Source: G Human Evaluation · p. 22, Table 19 · p. 22
  • Annotators prefer ASC in 56.7% of pairs against the base model and 59.3% against GRPO + KL, counting ties as half a win.They agree at Cohen’s kappa 0.66; on pairs both decide, their verdict matches Gemma’s in 89.8% and Mistral’s in 88.9%.Source: G Human Evaluation · p. 22, Table 18 · p. 22

H Prompts

  • All models answer in one user turn; the sentence judge’s verdict is read as its probability of “Yes,” and the pairwise rubric was fixed before trained answers were judged.Thinking is disabled on Qwen models.Source: H Prompts · p. 22

I Example Answers

The examples contrast policies’ characteristic shortcuts with ASC’s answers, which generally express more grounded content in their own words. The selected cases span in-domain and held-out settings.

I Example Answers

  • In the Game of Thrones example, GRPO + KL and the sentence reward give one copied sentence, the copy penalty gives a vacuous one, and ASC writes three grounded sentences.The base model gives a more detailed answer.Source: I Example Answers · pp. 23, 27
  • Examples were selected to represent each setting’s typical policy behavior, with visible markings for copied, authored-supported, authored-unsupported and vacuous sentences.The examples include two training-corpus and two held-out questions, and each is answered by all five policies.Source: I Example Answers · p. 23
  • Across three examples, baselines copy, become vacuous or decline, while ASC gives grounded answers; in the Linux example it conveys the same content in four grounded sentences with no error.The examples show copied sentences and an unsupported Qt-toolkit claim in the base answer, a refusal from the copy penalty, and overstatement about California lottery taxes by the base model.Source: I Example Answers · p. 23

Related papers

Loading 2610.18442v1…