Source-linked AI summary
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Philippe Laban, Tobias Schnabel, Paul N. Bennett, Marti A. Hearst
TL;DR
Summary inconsistency is a persistent problem, while prior NLI-based detectors were limited by a mismatch between sentence-level NLI inputs and document-level detection. The paper uses sentence-pair entailment scores with lightweight aggregation and introduces a six-dataset benchmark. SUMMACConv achieves 74.4% balanced accuracy, five points above prior work, while future improvements include combining NLI models or granularities and using stronger detectors for summarization.
Problem
Current summarization models can produce factually inconsistent summaries, and prior NLI-based methods performed poorly because NLI and inconsistency detection use different input granularities.
Method
The paper segments documents and summaries into sentence pairs, applies an NLI model, and aggregates the resulting entailment scores with SUMMACZS or SUMMACConv.
Results
74.4% balanced accuracy was achieved by SUMMACConv on the SUMMAC Benchmark, five absolute percentage points above the best baseline.
Takeaways & Limitations
Sentence-level NLI input enables successful inconsistency detection, and more accurate detectors can support future consistency correction in summarization.
Takeaways & Limitations
The models are a first step, with future work suggested on combining multiple NLI models or granularity levels and using stronger detectors in summarizer training.
Abstract
from arXiv · showhide
In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. On our newly introduced benchmark called SummaC (Summary Consistency) consisting of six large inconsistency detection datasets, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% point improvement compared to prior work. We make the models and datasets available: https://github.com/tingofurro/summac
1 Introduction
Summary inconsistency remains a major limitation despite advances in fluency and informativeness. This work shows that NLI models can detect inconsistencies when sentence-level inputs are aggregated, achieving strong benchmark performance.
- Factually inconsistent summaries remain a major limitation of current summarization models, including negations, entity swaps, and hallucinated entities.
- More than 70% of summaries can be inconsistent in some scenarios, motivating accelerated research on inconsistency detection.
- Earlier NLI-based approaches performed poorly, including 52% accuracy for binary inconsistency detection.
- Sentence-pair processing avoids the document-level granularity mismatch that can make an NLI model incorrectly judge an inconsistent summary as entailed.
- SUMMAC aggregates sentence-level entailment scores, with SUMMACZS using max and mean operators and SUMMACConv using one learned convolution layer.
- 74.4% overall balanced accuracy was achieved by SUMMACConv, 5% above prior work on the SUMMAC Benchmark.
2 Related Work
Related work approaches inconsistency detection through fact checking, dataset construction, question answering, synthetic classifiers, and parsing. The paper standardizes six large datasets while positioning improved detection as useful for later correction methods.
- Fact checking retrieves evidence for a claim and classifies it as supported, refuted, or insufficiently supported, whereas inconsistency detection focuses on any mismatch with the source document.
- Existing inconsistency datasets vary in labels and error granularity, and often contain multiple model-generated summaries for each document.
- The SUMMAC Benchmark standardizes the six largest summary consistency datasets, broadening coverage of inconsistency errors.
- QAG methods generate questions, answer them using the document and summary, and compare the resulting answers to estimate consistency.
- Synthetic classifiers train on transformed summaries, while parsing-based methods compare summary relations with document relations.
- Consistency-correction methods rely on inconsistency detectors, so more accurate detectors can benefit both reinforcement-learning and rewriting approaches.
3 SUMMAC Models
SUMMAC constructs an NLI Pair Matrix by comparing document and summary sentence blocks, then aggregates the matrix into a summary-level consistency score. SUMMACZS uses zero-shot max-then-mean aggregation, while SUMMACConv learns to aggregate score distributions with a convolutional layer.
- 3.1 Generating the NLI Pair Matrix: NLI models compare every document block with every summary block, producing entailment, contradiction, and neutral probabilities in an M × N pair matrix.The document supplies M premise blocks and the summary supplies N hypothesis blocks; the default matrix contains entailment scores Eij.
- 3.1 Generating the NLI Pair Matrix: The pair matrix represents a bipartite graph whose edge opacity corresponds to each document-summary pair’s entailment probability.
- 3.2 SUMMACZS: Zero-Shot: SUMMACZS retains the maximum score in each matrix column, representing the strongest document support for each summary sentence.
- 3.2 SUMMACZS: Zero-Shot: It averages the retained column scores into one scalar summary score and requires no trained parameters.In the Figure 1 example, the score is 0.67 and rises to 0.985 after removing the third summary sentence.
- 3.3 SUMMACConv: Convolution: SUMMACConv bins each matrix column into a fixed-size histogram, then uses a kernel-size-H convolution to convert each histogram into a scalar.
- 3.3 SUMMACConv: Convolution: The convolution outputs are averaged across summary sentences, and the layer is trained end-to-end on a 10,000-example FactCC subsample using H = 50 parameters.Performance increases through 50 bins and then plateaus; SUMMACConv reduces reliance on noisy extrema by using each score distribution.
4 SUMMAC Benchmark
The SUMMAC Benchmark standardizes six large summary inconsistency datasets as a common binary classification task. Because class balance varies widely, it uses balanced accuracy as its primary metric and ROC-AUC as a secondary metric.
- 4.1 SUMMAC Benchmark: The benchmark combines the six largest available summary inconsistency datasets and standardizes them to the same classification task.
- 4.1 SUMMAC Benchmark: Each dataset contains document-summary-label samples, with labels indicating whether a summary is consistent or inconsistent.
- 4.1 SUMMAC Benchmark: Validation and test splits use dataset-author splits when available; otherwise, alternating indexed samples create splits while preserving class imbalance and summarizer identity.
- 4.2 Benchmark Datasets: FactCC validation and test annotations were written entirely by the authors after crowdsourced annotation attempts produced low inter-annotator agreement.The authors note that researcher annotations may differ in quality from those in other datasets.
- 4.3 Benchmark Evaluation Metrics: Balanced accuracy is the primary metric because dataset positive-label proportions range from 6% to 91%.The metric adjusts for class imbalance, and performance is the unweighted average across the six datasets.
- 4.3 Benchmark Evaluation Metrics: ROC-AUC is the secondary metric because it summarizes classifier performance across threshold levels.
5 Results
On the SUMMAC Benchmark, SUMMAC models achieve the strongest overall inconsistency-detection performance, while results show benefits from distribution-based aggregation, modern NLI models, and finer matching granularity. These gains come with lower throughput than some baselines.
- 74.4% balanced accuracy makes SUMMACConv the best benchmark performer, 5 points above QuestEval, the strongest non-NLI method.
- SUMMAC models show strong performance across datasets, whereas FactCC-CLS varies substantially between FactCC, FRANK, and XSumFaith.
- Considering the entire distribution of document scores improves robustness over relying only on maximum scores, with SUMMACConv outperforming SUMMACZS.
- SUMMAC models process around 430 documents per minute, substantially fewer than some baselines exceeding 10,000 documents per minute.
- Pre-trained Transformer NLI models perform strongly, with average performance increasing 1.3 percentage points from base to large architectures.
- Jointly training NLI models on MNLI and Vitamin C produces the default NLI model because it achieves the best performance among tested datasets.
- Finer document and summary granularities perform better overall, with sentence-sentence recommended; performance rises from 56.4% to 73.5% in one MNLI-only comparison.
6 Discussion and Future Work
The authors identify interpretability, domain coverage, and integration into consistent summarization as open directions for inconsistency detection.
- Improvements on the Benchmark: Future improvements include combining multiple NLI models or multiple granularity levels, potentially through multi-hop reasoning.The current models are described as an initial step toward harnessing NLI for inconsistency detection.
- Interpretability of model output: SUMMACConv’s outputs are less interpretable because low scores cannot be traced to a single invalidated document sentence.SUMMACZS is directly interpretable, whereas SUMMACConv is more opaque.
- Beyond news summarization: The benchmark covers news summaries, leaving inconsistency detection in legal, scholarly, and other domains for future study.The authors also mention possible applications beyond summarization, such as text simplification and code generation.
- Towards Consistent Summarization: Inconsistency detection is presented as a first step toward reducing inconsistencies in generated summaries.Future summarizers could incorporate more powerful inconsistency detectors during training.
7 Conclusion
The conclusion introduces two NLI-based inconsistency detectors and a standardized benchmark spanning six large datasets. SUMMACConv achieves the strongest reported benchmark performance with 74.4% balanced accuracy.
- Conclusion: SUMMACZS and SUMMACConv use sentence-level NLI inputs as the central design principle for summary inconsistency detection.Both models are introduced as NLI-based detectors built around this granularity insight.
- Conclusion: The SUMMAC Benchmark standardizes six large datasets into a diverse collection for inconsistency detection.The benchmark is described as a new standardized collection covering the six largest datasets.
- Conclusion: 74.4% balanced accuracy makes SUMMACConv outperform prior work by five absolute percentage points.The conclusion describes this as the first successful adaptation of NLI models for inconsistency detection.
A NLI Model Origin
The appendix lists the pretrained NLI model variants used in the experiments, spanning SNLI, MNLI, and VitaminC fine-tuning combinations.
- NLI Model Origin: The experiments use pretrained NLI models retrieved from HuggingFace’s model hub.The listed models are BERT-family architectures with different fine-tuning datasets and configurations.
- NLI Model Origin: The model set includes SNLI-only, MNLI-only, and MNLI-plus-VitaminC variants across base and large model sizes.The list includes RoBERTa and ALBERT checkpoints under the paper’s BERT-family grouping.
B SUMMACZS Operator Choice
The SUMMACZS ablation compares min, mean, and max choices for two aggregation operators and selects the best-performing combination.
- SUMMACZS Operator Choice: Max for Operator 1 and mean for Operator 2 achieve the highest SUMMACZS performance.The authors use these operator choices in the final model.
C SUMMAC Benchmark ROC-AUC Results
The benchmark reports model performance using ROC-AUC across six inconsistency-detection datasets, with their average serving as the overall benchmark score. Confidence intervals indicate whether SUMMAC models improve over prior work.
- Table A2 reports inconsistency-detection model performance on the benchmark's test portion using ROC-AUC.
- ROC-AUC is computed separately on each of the six benchmark datasets, then averaged to obtain overall performance.
- Confidence intervals compare SUMMAC models with prior work, marking improvements at 95% or 99% confidence.