PaperlayerView exact revision on arXiv

Source-linked AI summary

Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback

Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, Idan Szpektor

arXiv:2306.00186v1cs.CL

TL;DR

Abstractive summarizers can generate information unsupported by their source articles, and entailment alone may not preserve informativeness. This paper fine-tunes summarizers with reference-free textual-entailment rewards and anchor-model regularization. On XSum, RLEF outperforms baselines on factual consistency while maintaining or improving other summarization attributes.

  • Problem

    Abstractive summarizers can produce claims that contradict or are unsupported by their source, while entailment-only optimization may risk less-informative or extractive summaries.

    Source: §1 · pp. 1–2

  • Method

    RLEF fine-tunes a supervised summarizer with a reference-free NLI entailment reward and regularization that keeps the RL policy close to its initial model.

    Source: §1 · pp. 1–2

  • Results

    RLEF outperforms all baselines on factual consistency while maintaining and even improving other desired summarization attributes.

    Source: §7 · p. 9

  • Takeaways & Limitations

    On XSum, reference-free entailment feedback can improve factual consistency while preserving or improving other evaluated summarization attributes.

    Source: §1 · p. 2, §7 · p. 9

    RLEF depends on a strong entailment model and initial summarizer, and cannot improve consistency errors the NLI reward cannot detect.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article. In this work, we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency and explore the ensuing trade-offs, as improved consistency may come at the cost of less informative or more extractive summaries. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.

1 Introduction

Abstractive summarizers can produce plausible claims that contradict or lack support in their source articles, limiting real-world adoption. RLEF uses textual-entailment feedback to improve consistency while regularizing toward an initial summarizer; on XSum, it improves factual consistency, salience, and conciseness over strong baselines.

1 Introduction

  • Abstractive summarizers can produce plausible information that contradicts or is unsupported by the source, limiting their real-world adoption.Figure 1 illustrates a T5 summary claiming a second consecutive year of falling operating profits, which the article does not support.Source: §1 · p. 1, Figure 1 · p. 1
  • ROUGE is inefficient for detecting hallucinations, motivating factual-consistency metrics such as textual entailment as a training-time reward.The reward checks entailment between the source article and generated summary.Source: §1 · p. 1
  • Entailment alone may miss coherence and relevant information, and can encourage extractive or less-informative summaries.RLEF’s regularization is intended to preserve summarization capabilities learned under the maximum-likelihood objective.Source: §1 · p. 2
  • RLEF fine-tunes a cross-entropy-trained summarizer with an entailment-based RL reward while constraining it to stay close to the initial model.The authors explore model scale, regularization, and decoding strategies as interdependent factors in the consistency–informativeness trade-off.Source: §1 · p. 2
  • Unlike rewards based on human feedback or reference-summary similarity, RLEF evaluates generated text against the input and can train without reference summaries.Human feedback requires dedicated, costly collection for each target task, while reference-based rewards require reliable references.Source: §1 · p. 2
  • RLEF improves factual consistency, salience, and conciseness over strong baselines on XSum, according to automatic metrics and human evaluation.Its reward evaluates generated summaries against the input, without requiring reference summaries.Source: §1 · p. 2

2 Method

RLEF initializes an RL policy from a supervised summarizer and combines an entailment reward with regularization toward that anchor. It trains on generated summaries without references and uses temperature sampling at inference.

2 Method

  • RLEF seeks higher faithfulness while retaining the salience and coherence of a supervised summarizer, using entailment reward and anchor-based regularization.The entailment reward directs the model toward faithfulness, while regularization discourages degenerate solutions and forgetting how to summarize.Source: §2 · p. 2

2.1 RLEF: RL from Entailment Feedback

  • The method models token generation as a contextual Markov decision process, with the source as context, the generated prefix as state, and vocabulary tokens as actions.The RL objective maximizes cumulative reward.Source: §2.1 · p. 3
  • An NLI classifier scores the complete summary against its source, assigning the entailment log-probability as reward at the end-of-sequence token.Other tokens receive zero NLI reward; a KL penalty toward the anchor policy is added, with α controlling the faithfulness–proximity trade-off.Source: §2.1 · p. 3
  • On-policy actor-critic training updates the policy and value network using generated summaries and reward scores, without requiring reference summaries.The supervised model initializes both networks; the policy uses policy gradients and generalized advantage estimation, while the value network is trained by bootstrapping.Source: §2.1 · p. 3

2.2 Decoding at Inference Time

  • Because the RL policy optimizes a sequence-level reward, the authors use temperature sampling instead of beam search at inference.They report temperature sampling is sufficient for RL, while beam search is needed to improve the supervised policy.Source: §2.2 · p. 3

3 Experimental Design

The experiments focus on abstractive summarization, primarily on XSum, and compare RLEF with supervised, filtered, control-token, and prior faithfulness methods. They use an NLI reward model, automatic metrics, and human ratings to assess the systems.

3.1 Data

  • XSum is the main testbed: its 200K BBC articles use each article’s lead sentence as the summary and the remaining sentences as source, a setup associated with factual inconsistencies.The authors also use TL;DR and CNN/DM for comparisons with prior work; CNN/DM evaluation uses models trained on TL;DR.Source: §3.1 · pp. 3–4

3.2 Entailment Model

  • The study uses the Honovich et al. NLI classifier as the RLEF reward, evaluation model, and baseline-labeling tool.It was trained on ANLI with T5-XXL; the source is the premise, the summary is the hypothesis, and the decoded entailment-character log-probability is the reward.Source: §3.2 · p. 4

3.3 Baseline Methods

  • Baselines include supervised fine-tuning, NLI-filtered training data, entailment-control prompting, and prior faithfulness systems FactPegasus and CLIFF.The filtered baseline retains 60% of XSum training data; RLHF uses a human-comparison reward and released predictions.Source: §3.3 · p. 4

3.4 Proposed Models

  • The two RLEF variants differ in regularization weight and sampling temperature, settings chosen to affect the faithfulness–lexical-similarity trade-off.RLEFL uses α = 0.1 and temperature 1; RLEFH uses α = 0.2 and temperature 0.3.Source: §3.4 · pp. 4–5

3.5 Automatic Evaluation Metrics

  • Automatic evaluation includes lexical-overlap metrics, NLI and Q2 factual-consistency scores, extractiveness measures COVERAGE and DENSITY, and summary length.COVERAGE measures the share of summary tokens appearing in the document; DENSITY reflects the average length of extractive spans.Source: §3.5 · p. 5

3.6 Manual Evaluation Protocol

  • Human evaluation uses three raters per summary to assess comprehensibility, attribution, salience, and conciseness with binary yes/no responses.Evaluators flag any slight deviation from the desired property; the pool consists of 11 workers who completed a 10-example training round.Source: §3.6 · p. 5
  • The XSum test-set results table compares RLEF regularization variants with baseline methods; its caption notes T = 0.3 for unstable T5-Base RL training.The TL;DR and CNN/DM table reports automatic evaluation, including transfer predictions on CNN/DM from models trained on TL;DR.Source: Table 1 · p. 5, Table 2 · p. 5

4 Results

On XSum, RLEF improved entailment-based scores over supervised baselines, while human ratings also favored it for attribution, salience, and conciseness.

4 Results

  • On XSum, RL-based models achieved the highest NLI and Q2 entailment scores, and RL used the NLI signal more effectively than Filtered and CTRL.Source: §4 · p. 5
  • RLEFL scored highest on entailment but lower on ROUGE, whereas more strongly regularized RLEFH matched CTRL and SL on ROUGE.The authors attribute RLEFL’s lower ROUGE to moving farther from the supervised starting point; they suggest KL regularization kept RLEFH from drifting.Source: §4 · pp. 5–6
  • RLEF outperformed other methods by a large gap in human attribution and led in salience and conciseness.Raters agreed on attribution in 60% of examples; reference summaries scored lowest at 23.6%.Source: Table 3 · p. 6, §4 · p. 6
  • RL policies showed no apparent increase in copying, while higher document-term coverage suggested fewer hallucinations and other metrics indicated greater conciseness.Source: §4 · p. 6
  • On TL;DR, RLEF had higher NLI and Q2 entailment scores than RLHF, while the supervised model was on par with RLHF.RLEF also maintained length and ROUGE similar to the supervised baseline.Source: §4 · p. 6
  • On CNN/DM transfer using TL;DR-trained models, RLEF again had higher entailment scores than RLHF.The authors report similar trends and say RLEF outperformed the domain-specific RLHF reward in both source and transfer settings.Source: §4 · p. 6

5 Analysis

The analysis examines how model size, KL regularization, and sampling temperature affect entailment and summary properties, then inspects changes during training and remaining attribution errors.

5 Analysis

  • Higher sampling temperatures correlate with higher entailment and lower ROUGE, while weaker KL regularization raises entailment and coverage but lowers ROUGE and summary length.Lower-temperature policies use more document-aligned terms but have lower NLI scores; the authors suggest reduced detail may help alleviate inconsistencies.Source: §5 · pp. 6–7, Figure 3 · p. 7
  • All model sizes improved entailment, but T5-Base hacked the reward with summaries half the reference length, unlike the slightly changing lengths of Large and XXL.The authors suggest higher-capacity models may better accommodate multiple rewards and have more robust anchor models.Source: §5 · p. 7, Figure 4 · p. 7

5.1 Manual Analysis.

  • Among 60 entailment-flip examples, 13 were abstractively rephrased and 27 omitted arguments, while most changes remained local to summaries’ main predicates and participants.Other changes included 16 claim changes and 8 argument replacements.Source: §5.1 · pp. 7–8, Table 4 · p. 8
  • Of 39 non-attributable summaries inspected, 28 contained local hallucinations, and most were nevertheless judged entailed by the reward model.The analysis also found five contradictions and five major hallucinations; the authors suggest NLI may be affected by conflicts between parametric and contextual knowledge.Source: §5.1 · p. 8

6 Related Work

Prior work uses reference-based metrics, human feedback, or hybrid objectives for summarization rewards. The paper contrasts these approaches with factual-consistency rewards and discusses their trade-offs.

6 Related Work

  • Prior summarization RL methods use reference-based rewards, including ROUGE and entailment or question-answering signals (Nan et al., 2021).Reference summaries can contain factual errors, potentially reducing the effectiveness of reference-based rewards (Maynez et al., 2020).Source: §6 · p. 8
  • Human-feedback reward models can avoid errors from references misaligned with human judgment, but require task-specific annotation that can be labor-intensive (Böhm et al., 2019; Ziegler et al., 2019; Wu et al., 2020; Stiennon et al., 2020).Source: §6 · pp. 8–9
  • A multi-document summarization method interleaves cross-entropy and policy gradients, using an in-domain NLI model, a GAN-like stabilizer, and a discriminator reward (Pang et al., 2021).Source: §6 · p. 9
  • Binary NLI rewards require the policy-gradient algorithm to assign credit from a summary-level decision, whereas QA-QG methods can localize offending tokens but are slower to compute.Block-level NLI may aid credit assignment for long summaries; the datasets in this study use short, single-sentence summaries.Source: §6 · p. 9

7 Conclusions and Future Work

The paper uses NLI models as reference-free RL rewards for factual consistency in abstractive summarization. Its experiments report gains in factual consistency while maintaining or improving other summary attributes, and it identifies directions for extending the approach.

7 Conclusions and Future Work

  • RLEF uses NLI models as ready-made, reference-free rewards to train factually consistent abstractive summarization models.Source: §7 · p. 9
  • The RL approach outperformed all baselines on factual consistency while maintaining or improving other desired summarization attributes.Source: §7 · p. 9
  • Future work includes applying the approach to grounded-generation tasks such as knowledge-driven dialogue and exploring reference-free rewards for other summary attributes or tasks.The authors also identify adapting the method to multiple rewards as a research direction.Source: §7 · p. 9

Limitations

The method depends on strong entailment and initial summarization models. The authors also identify model size as important and report that the summarizer and NLI model used in their experiments were both 11B parameters.

Limitations

  • RLEF relies on a strong entailment model and a strong initial summarization model; errors undetected by the NLI reward cannot be improved by its RL policies.The NLI model can carry biases and data regularities from its training set into generated summaries, and human attribution judgments differ from its automatic metric.Source: Limitations · p. 9
  • The initial summarizer is needed to initialize RL exploration and anchor the policy; the authors expect weaker results without a high-quality supervised model.Although RL training does not require reference data, the authors say it would probably not succeed as well without that initialization.Source: Limitations · p. 9
  • Both the summarization and NLI models used in the experiments were 11B parameters, and the authors identify improving robustness to smaller models as an open need.They connect smaller-model robustness to computational efficiency and availability.Source: Limitations · p. 9

Ethics Statement

The authors identify misinformation in automated text generation as an ethical concern, especially when summaries contain errors or source texts are untrustworthy. They also warn that model biases may be encoded in generated text and call for scrutiny and human judgment.

Ethics Statement

  • Untrustworthy input texts or model errors can allow automated summarization to amplify misinformation.The authors recommend caution and combining these methods with techniques that enable humans to judge generated information.Source: Ethics Statement · pp. 9–10
  • The trained models rely on predictive models whose training-data biases may be implicitly encoded in generated text.The authors say both the NLI and summarization datasets should be scrutinized before real-world use.Source: Ethics Statement · p. 10

A Experimental Details

The appendix describes the actor-critic policy-gradient setup and implementation choices used for RL fine-tuning, including optimization, hyperparameter search, supervised decoding, and compute resources.

RL Algorithm Details

  • An on-policy actor-critic uses a learned value function and generalized advantage estimation (GAE) to train the policy with policy-gradient losses.The value function is a baseline; it is trained by regression toward the return estimate induced by GAE.Source: A Experimental Details · p. 14

RL Implementation Details

  • Sequence-level KL penalties improve RL stability, while the implementation uses γ = 1 and normalizes batch advantages and value loss.The KL penalties are accumulated across a sequence rather than treated as immediate rewards.Source: A Experimental Details · p. 14
  • 20K RL fine-tuning steps include a 5K-step value-network warmup, with separate policy and value learning rates.The warmup allows value estimates to become sufficiently accurate to avoid detrimental policy gradients.Source: A Experimental Details · p. 14

Optimization and Hyperparameter Search

  • The XXL hyperparameter sweep comprised 24 runs of 20K iterations, following a decoupled search for value and policy learning rates.The sweep varied temperatures [0.1, 0.3, 1.0] and α from 0.1 to 0.8 in steps of 0.1; Adafactor used linear learning-rate warmup and annealing.Source: A Experimental Details · pp. 14–15, Table 5 · p. 15

SL Implementation Details

  • Supervised models use beam search with width 4 and a brevity penalty of 0.6; training uses batch size 32 and learning rate 0.001.Training also uses dropout 0.1 and the same optimizer as the RL models.Source: A Experimental Details · p. 15

Resources

  • Each T5-XXL RLEF experiment took approximately 17 hours on 64 TPU chips, and the main sweep accounted for 1088 TPU-days.Source: A Experimental Details · p. 15

B Evaluator Demographics, UI and Instructions

The appendix describes who rated the summaries, the evaluator interface, and the four yes/no criteria used to assess summary quality. Its guidelines distinguish factual support, main-idea coverage, comprehensibility, and conciseness.

B Evaluator Demographics, UI and Instructions

  • Full-time hourly raters were native English speakers from the U.S. and U.K.; 70% held graduate diplomas and 30% high-school diplomas.They received two pages of instructions, examples, a pilot study, and a training batch; the evaluator interface is shown in Figure 5.Source: B Evaluator Demographics, UI and… · p. 15, Figure 5 · p. 16

B.1 Guidelines

  • Raters answered four yes/no questions on comprehensibility, attribution to the article, main-idea capture, and conciseness.The guidelines caution that fluent summaries can still contain subtle inaccuracies.Source: §B.1 · p. 15
  • A summary fails comprehensibility when malformed or unclear language makes any part difficult to understand.The guidelines include a cut-off or strange-character example.Source: §B.1 · p. 15
  • Attribution requires summary information to be supported by the source; key missing, inaccurate, or contradicted details count against it.The guideline test is whether the claim can be introduced with “According to the news article.”Source: §B.1 · p. 15
  • Main ideas concern central facts or themes about the article’s people, locations, or events, but a summary need not contain only main ideas.The guidelines distinguish supported summaries that miss the central point from summaries that capture it despite minor hallucinations or errors.Source: §B.1 · pp. 15–16
  • A contradiction that makes the main idea unrecoverable is distinguished from minor hallucinations when assessing main-idea capture.The first case receives no for attribution and no for main-idea capture; the latter can receive no for attribution but yes for main-idea capture.Source: §B.1 · p. 16
  • Conciseness concerns unnecessary details beyond key information, not whether a summary contains attribution errors in details that belong to the main idea.Substantial information outside the main idea counts against conciseness; a minor contradiction within the main idea is treated as an attribution error instead.Source: §B.1 · p. 16

C Fine-Tuning Language Models with Reinforcement Learning

The appendix formalizes language generation as a contextual decision process, then adds a pretrained-model regularizer and derives the corresponding policy-gradient objective. These formulations connect token generation, the anchor policy, and RL optimization.

C.1 Language Generation as a Contextual Markov Decision Problem

  • Language generation is modeled as a contextual MDP: the input document and generated prefix form context and state, while each generated token is an action.The generation horizon and input length are finite; transitions are deterministic, and an end-of-sequence state is absorbing.Source: §C.1 · p. 17

C.2 Language Generation From a Pre-Trained Model as a Regularized Markov Decision Problem

  • For generation from a pretrained model, the reward includes a regularization term that encourages the learned policy to remain close to the anchor policy.The regularization strength is controlled by α; its term is an unbiased estimator of the KL divergence, and the reward is non-stationary as the policy changes.Source: §C.2 · p. 17, §C.3 · p. 18

C.3 Defining the Reinforcement Learning Objective

  • The regularized reward yields a policy-gradient formulation that can be optimized with any policy-gradient method, including for the summarization MDP.The objective seeks parameters that maximize expected return under the regularized reward.Source: §C.3 · pp. 17–19

Related papers

Loading 2306.00186v1…