PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning, Curtis P. Langlotz

arXiv:1911.02541v3cs.CL

TL;DR

Neural abstractive summarizers can achieve high reference overlap without factual correctness, a serious problem for radiology applications. The paper fact-checks generated summaries with information extraction and trains with reinforcement-learning rewards, substantially improving factual correctness and overall quality on two hospital datasets, with outputs approaching human-authored quality.

  • Problem

    Existing abstractive summarization models can highly overlap human references while producing factual errors, making factual correctness critical in radiology.

  • Method

    An information extraction module compares generated summaries with references for factual accuracy, while reinforcement learning jointly optimizes correctness, textual overlap, and language-model objectives.

  • Results

    On two hospital datasets, the approach substantially improves factual correctness and produces higher-quality, more correct summaries closer to human-written ones.

  • Takeaways & Limitations

    In limited-fact domains such as radiology, information extraction and reinforcement learning can improve factual correctness, while ROUGE optimization alone can also help.

  • Takeaways & Limitations

    The method depends on an external information-extraction module, and generalizing it to open-domain summarization remains unsolved.

Abstract

from arXiv · show

Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where factual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones.

1 Introduction

Existing abstractive summarizers can overlap strongly with human references while still making factual errors, making factual correctness a critical unresolved issue in radiology. The paper proposes fact-checking-based rewards and reinforcement learning to improve correctness and summary quality.

  • Problem: High textual overlap with human references does not guarantee factual correctness, as illustrated by radiology summaries with conflicting ROUGE-L and factuality.Summary A has higher overlap but an error, whereas Summary B has lower overlap and is factually correct.
  • Problem: About 30% of outputs from a radiology summarization model contain factual errors or inconsistencies.Such errors can make radiology summarization unusable because factual correctness is important for preventing medical errors.
  • Research gap: Existing factual-consistency methods have not explicitly optimized abstractive summarization systems with a correctness objective.Despite attention augmentation, entailment reranking, and weakly supervised verification, state-of-the-art systems still produce substantial factual errors.
  • Motivation: Radiology reports provide relatively unambiguous, limited-space facts, making factual correctness objectively measurable with an information extraction system.The domain is also clinically important because improving correctness can directly support practical use.
  • Approach: The proposed framework compares extracted information from generated summaries with human references to produce factual accuracy rewards.Training jointly optimizes factual correctness, textual overlap, and language-model objectives through reinforcement learning.
  • Results: On two hospital datasets, the training strategy substantially improves factual correctness and yields summaries with higher overall quality and clinical validity close to human-written summaries.The results are reported through both automatic and radiologist evaluation.

2 Related Work

Prior work developed neural radiology summarizers and explored factuality evaluation or reranking, but improving factual correctness through direct optimization remained largely unstudied. This paper extends reinforcement-learning objectives with factual correctness for radiology summarization.

  • Neural summarization systems: Neural summarization research distinguishes extractive systems from abstractive systems, with many models trained by maximizing reference-summary likelihood.Reinforcement learning has also been used to optimize summarization metrics such as ROUGE.
  • Neural summarization systems: Directly optimizing ROUGE with reinforcement learning can improve summary ROUGE scores, but ROUGE does not guarantee factual accuracy or completeness.The paper extends these rewards with a factual correctness reward.
  • Factual correctness in summarization: Earlier factuality work used extracted fact triples, natural-language inference reranking, weak supervision, or information extraction to evaluate generated text.These approaches addressed factuality through model augmentation or evaluation rather than the paper’s direct optimization strategy.
  • Radiology report summarization: Radiology summarization studies improved overlap or quality with pointer-generator variants, while reported factual errors left correctness improvement unstudied.One cited study found factual errors in about 30% of neural radiology summaries.

3 Task & Baseline Pointer-Generator

The task maps radiology findings to clinically significant summary tokens, using a background-augmented pointer-generator as the baseline. Its encoder-decoder architecture combines attention, copying, and background information during decoding.

  • Task: Radiology summarization generates a sequence y that captures salient and clinically significant findings from input findings sequence x.The radiologist-produced summary is treated as the reference sequence.
  • Baseline architecture: The baseline uses a background-augmented pointer-generator network designed for radiology notes with multiple sections.It extends a pointer-generator with a separate encoder for the background section.
  • Baseline architecture: A bidirectional LSTM encodes the input, and a separate LSTM decoder generates the output sequence.An attention mechanism over encoder hidden states makes input information available during decoding.
  • Task: The terms “summary” and “impression” are used interchangeably for the radiologist-produced output.The paper notes that “impression” is common in clinical settings.
  • Baseline architecture: The copy mechanism blends the vocabulary distribution with a copy distribution to generate each next word from input content.This enables the model to copy words from the radiology findings.
  • Baseline architecture: Background-guided decoding separately encodes report background and injects its representation into decoding alongside the findings input.Radiology reports use background for study context and findings for clinical observations.

4 Fact Checking in Summarization

The paper evaluates factual correctness by extracting structured facts from generated and reference summaries, then trains summarization models with reinforcement learning to optimize factuality alongside overlap and language-model objectives.

  • Motivation: Teacher-forcing with maximum likelihood creates an exposure-bias discrepancy between training and test-time inputs, which can produce degenerate outputs.
  • Fact extraction: The fact extractor f converts a summary sequence into a structured fact vector whose categorical variables represent clinical observations such as pneumonia presence.For radiology reports, the paper uses the CheXpert labeler to extract presence status for 14 clinical observations.
  • Fact extraction: Factual accuracy compares generated and reference fact vectors by the proportion of matching variables, penalizing missing positive facts and falsely claimed negative facts equally.
  • Policy learning: Self-critical training uses a greedily decoded sequence to obtain a baseline reward, which empirically helps stabilize summarization-model training.
  • Reward function: The reinforcement-learning reward combines ROUGE-L overlap rR with factual accuracy rC, while scalar weights control their balance.The paper reports that neural representation metrics such as BERTScore generalized poorly to this domain.
  • Reward function: The final loss jointly optimizes NLL for fluency and relevance, ROUGE-based loss for brevity and reference overlap, and factual loss for correctness.The NLL term is retained because directly optimizing rewards without it can hurt readability.

5 Experiments

The experiments evaluate RL-trained abstractive and extractive radiology summarizers on two hospital datasets using overlap and factual-correctness metrics. Factual F1 is macro-averaged across clinical variables, while extractive systems lack F1 evaluation because the labeler is less accurate on findings text.

  • Datasets: Two anonymized chest-radiograph datasets from Stanford University Hospital and Rhode Island Hospital are split over time into training, development, and test sets.
  • Models: The experiments compare an augmented pointer-generator baseline with RL variants using ROUGE, factual-correctness, or combined rewards, all retaining the NLL loss component.Each RL variant is initialized from the best teacher-forced baseline and then finetuned until its validation score peaks.
  • Models: LexRank and BanditSum provide non-neural and neural extractive comparison systems for assessing task difficulty and the need for abstractive summarization.
  • Evaluation: ROUGE-1, ROUGE-2, and ROUGE-L measure unigram, bigram, and longest-common-sequence overlap with reference summaries.
  • Evaluation: Factual F1 is computed separately for each clinical variable across test examples and then macro-averaged, avoiding misleading corpus-level accuracy.A model that always predicts a rare variable as absent could achieve high accuracy while remaining practically useless.
  • Evaluation: The CheXpert labeler is less accurate on findings text than on radiology summaries, so factual F1 is unavailable for the extractive models.

6 Results

Across automatic and radiologist evaluations, combined RL rewards improve factual correctness and overlap over the baseline, while producing outputs close to human summaries on several quality dimensions. The results also show that abstractive models better match the compressed style of radiology impressions than extractive systems.

  • Human Evaluation: Extractive models achieve non-trivial reference overlap but have much higher perplexity than human references, consistent with their more verbose language.The paper attributes this difference to radiologists’ more compressed summary style compared with findings text.
  • Automatic Evaluation: RLR+C improves Stanford performance over the baseline by 2.7 ROUGE-L and 8.6% factual F1, balancing overlap and correctness.RLC alone gains 10% factual F1 but reduces ROUGE scores, while RLR alone gains 2.9 ROUGE-L and 7.3% factual F1.
  • Automatic Evaluation: RLR+C improves RIH performance over the baseline by 2.5 ROUGE-L and 5.5% factual F1.
  • Fine-grained Correctness: RLR+C improves factual F1 across all tested clinical variables on both datasets, although initialization changes how gains are distributed among variables.Overall factual-F1 improvement remains approximately unchanged across training runs.
  • Qualitative Results: Qualitative examples show RLR+C producing clinically meaningful summaries and avoiding observations wrongly copied by the baseline from findings text.
  • Human Evaluation: Against the baseline, radiologists preferred RLR+C for factual correctness and overall quality, although the baseline was preferred for fluency and 60% of fluency judgments were ties.
  • Human Evaluation: Against human references, 72% of RLR+C outputs were at least as factually correct, while humans won on fluency and RLR+C was slightly preferred for overall quality.

7 Analysis & Discussion

The proposed rewards improve summary fluency and stylistic consistency while addressing factual errors, but the method depends on an external information-extraction module and remains limited in scope.

  • Fluency and Style of Summaries: Extractive models achieve non-trivial reference overlap but have much higher perplexity than human summaries because extracted sentences are more verbose.Radiologists use more compressed language in summaries than in findings.
  • Fluency and Style of Summaries: The baseline has lower perplexity than humans, whereas RLR+C produces perplexity much closer to human references.The authors hypothesize that teacher-forcing encourages generic, fluent generations that may not be factually correct.
  • Fluency and Style of Summaries: The top 10 trigram distributions show that the baseline heavily reuses a few trigrams, while RLR+C generates more diverse summaries closer to human references.The same trend is observed for 4-grams and 5-grams.
  • Limitations: The training strategy depends crucially on an external IE module, and generalizing it beyond domains with limited fact spaces remains unsolved.The authors note that the module is relatively easy to implement in a domain with a limited space of facts.
  • Limitations: The study focuses mainly on factual errors that flip binary event outcomes, leaving errors involving adjectives and coreference for future work.These excluded error forms can also occur in generated summaries.

8 Conclusion

The paper presents a framework and reinforcement-learning training strategy for improving factual correctness in neural abstractive summarization. Applied to radiology reports, it succeeds under automatic and human evaluation and supports improvements from ROUGE optimization alone.

  • Conclusion: The paper presents a general framework and training strategy to improve the factual correctness of neural abstractive summarization models.The approach is applied to radiology reports and evaluated on two hospital-collected datasets.
  • Conclusion: On two hospital-collected datasets, the approach succeeds under both automatic and human evaluation.The evaluation concerns radiology report summarization.
  • Conclusion: In radiology reports, a carefully implemented IE system can improve factual correctness through reinforcement learning.The conclusion scopes this takeaway to domains with a limited space of facts.
  • Conclusion: Even without a reliable IE system, optimizing ROUGE metrics via reinforcement learning can substantially improve factual correctness.This is presented as a general takeaway from the study.
  • Conclusion: The paper aims to draw attention to factual correctness in abstractive summarization and inspire future work.This is stated as an intended broader impact of the work.

B Dataset Preprocessing and Stratification Details

The datasets were filtered for alignable findings and impression sections, normalized for dates and times, and split chronologically to test generalization from historical to future data.

  • Dataset Filtering: Reports were excluded when findings or impression sections were missing, multiply present but unalignable, or below the minimum length thresholds.The thresholds were fewer than 10 words for findings or fewer than 2 words for impressions.
  • Preprocessing: All reports were tokenized with Stanford CoreNLP before filtering and normalization.Date and time mentions were replaced with special tokens such as <DATE>.
  • Temporal Stratification: Both datasets were stratified over time into training, development, and test splits.The strategy tests whether models trained on historical data generalize to future data.
  • Evaluation Layout: Table 7 reports test-set F1 scores for each variable on the RIH dataset, with asterisks marking significant improvements at p < .01 under a bootstrap test.The table presents variable-level factual performance.

C Model Implementation and Training Details

The implementation uses recurrent encoders and decoder beam search, trains with Adam and scheduled learning-rate decay, and compares neural and extractive baselines using development-set selection.

  • Baselines: The baseline background-augmented pointer-generator model uses an open implementation.The implementation source is identified in the paper’s footnote.
  • Neural Model: The model uses 2-layer findings and 1-layer background and decoder LSTMs, hidden size 200, 100-dimensional GloVe embeddings, dropout p = 0.5, and beam size 5.Decoding is capped at a maximum length of 50.
  • Optimization: Training uses Adam with initial learning rate 1e−3, batch size 64, gradient clipping at norm 5, and validation-based learning-rate decay.The learning rate decays by 0.5 when the validation score does not increase after 2500 steps.
  • Reinforcement Learning: The factual F1 score is used as a development-set stopping criterion, with λ1 = 0.97, λ2 = 0.97, and λ3 = 0.03 for both datasets.The scalar loss weights are tuned on the development sets.
  • Extractive Baselines: LexRank and BanditSum use open implementations, and both select the top 3 scored sentences because this gives the highest development-set ROUGE-L scores.BanditSum uses default hyperparameter values.
  • Evaluation: ROUGE is evaluated with the Python implementation released by Google Research, which closely matches the original Perl implementation.The paper reports empirically close results between the two implementations.

D Fine-grained Correctness Results on the RIH Dataset

Table 7 reports fine-grained factual F1 scores for all tested variables on the RIH dataset. The section also presents additional examples from both datasets comparing baseline and proposed generations.

  • D Fine-grained Correctness Results on the RIH Dataset: Table 7 reports fine-grained factual F1 scores for all tested variables on the RIH dataset.
  • D Fine-grained Correctness Results on the RIH Dataset: Figure 5 presents examples from both datasets alongside generations from the baseline system and the proposed approach.
  • D Fine-grained Correctness Results on the RIH Dataset: The section references the ROUGE evaluation resource used in the study.
Loading 1911.02541v3…