Source-linked AI summary
Loss-Based Active Learning for Neural Abstractive Summarization
Michail Ioannou, Tatiana Passali, George Michalopoulos, Grigorios Tsoumakas
TL;DR
Annotating abstractive summarization data is costly, while existing active-learning approaches for summarization can be unstable or computationally demanding. LOBSTER selects unlabeled documents semantically similar to high-loss labeled examples, combining this signal with diversity-aware filtering. Across three datasets and two backbones, it matches or outperforms state-of-the-art baselines, with selection speedups of up to 665x.
Problem
Active learning for abstractive summarization remains under-explored, while existing methods can be unstable or computationally demanding despite costly annotation requirements.
Method
LOBSTER identifies high-loss labeled examples, filters unlabeled candidates with IDDS, and selects semantically similar candidates using Sentence-BERT.
Results
Across three benchmark datasets and two backbone architectures, LOBSTER matches or outperforms state-of-the-art baselines while remaining significantly more computationally efficient.
Takeaways & Limitations
LOBSTER provides a loss-guided active-learning framework that targets the model’s specific weaknesses during abstractive summarization acquisition.
Takeaways & Limitations
Experiments cover only English-language datasets, so effectiveness for other languages remains unexplored.
Abstract
from arXiv · showhide
Fine-tuning abstractive summarization models requires high-quality annotated data. However, obtaining such corpora is expensive and time-consuming, as it requires human annotators to read and comprehend long documents to create accurate summaries. Active learning mitigates this issue by selecting only the most informative instances for annotation, allowing models to achieve competitive results with significantly fewer labels. However, the application of active learning to summarization remains under-explored, and existing studies often suffer from instability and significant computational bottlenecks. To overcome these challenges, we propose LOBSTER (LOss-BaSed acTivE leaRning), a novel active learning framework designed specifically for abstractive summarization. LOBSTER improves performance by prioritizing unlabeled instances semantically similar to the model's current high-loss training examples, enabling the model to explicitly correct its specific weaknesses. Our empirical evaluation across three benchmark datasets and two summarization backbone models demonstrates that LOBSTER consistently matches or outperforms current state-of-the-art approaches while achieving a query selection speedup of up to 665x.
1 Introduction
Abstractive summarization requires costly human annotation, motivating active learning to reduce labels while preserving performance. LOBSTER targets weaknesses in existing strategies and evaluates its effectiveness across datasets and model backbones.
- High-quality abstractive summaries are expensive to annotate because annotators must read documents and identify important information.
- Active learning selects informative examples so models can achieve strong performance with fewer labeled training instances.
- Existing summarization strategies face computational demands, outlier selection, instability, or reliance on external LLM systems.
- LOBSTER uses a hybrid strategy that targets current model weaknesses while maintaining data diversity.
- Experiments span three benchmark datasets and two model backbones, assessing effectiveness alongside instability and latency concerns.
- The data-efficiency analysis finds random sampling surprisingly competitive when annotation budgets are large.
2 Related Work
Active learning is well established for discriminative NLP tasks but remains comparatively underexplored for generative summarization. Existing methods mainly use uncertainty, diversity, or hybrid selection strategies.
- Active learning has been extensively studied for text classification and sequence tagging, but comparatively little for generative tasks.
- Uncertainty-based methods: Uncertainty-based methods select examples where the model is least confident in its predictions.
- Uncertainty-based methods: BAS estimates uncertainty through Bayesian approximation and selects summarization documents with the highest variance.
- Diversity-based methods: Diversity-based methods seek samples diverse from labeled data while remaining representative of the data distribution.
- Hybrid methods: Hybrid methods combine uncertainty and diversity signals, including DUAL for abstractive summarization.
3 Methodology
LOBSTER selects annotations by diagnosing high-loss labeled examples and retrieving semantically similar unlabeled candidates. Its three-stage pipeline combines loss guidance, diversity-aware filtering, and semantic similarity.
- LOBSTER combines loss-guided sampling with IDDS in a three-stage hybrid active learning strategy.
- Stage 1: Loss-Based Hard Example Selection: High-loss labeled examples identify error-prone regions and serve as semantic anchors for acquisition.
- Stage 1: Loss-Based Hard Example Selection: Hard examples are the top-k labeled instances with the highest loss, using a dynamic threshold set by the k-th largest loss.
- Stage 2: Representative Candidate Filtering: IDDS filters similarity-retrieved candidates to reduce near-duplicate clusters, broaden representation, and mitigate noisy outlier selection.
- Stage 3: Semantic Hardness Projection: For each representative unlabeled candidate, Sentence-BERT embeddings compute maximum similarity to hard examples, and the top B candidates form the annotation batch.
- Stage 3: Semantic Hardness Projection: LOBSTER assumes that semantically close documents to high-loss labeled examples provide informative supervision for improving the model.
4 Experimental Setup
The experiments evaluate active summarization under a standard iterative acquisition protocol across diverse datasets, baselines, backbones, and lexical and semantic metrics. Models are retrained from scratch after each acquisition cycle.
- Datasets: The evaluation uses AESLC, XSum, and CNN/DailyMail, covering email subjects, single-sentence news summaries, and longer multi-sentence summaries.
- Active learning protocol: Each cycle begins with 10 randomly sampled annotated instances, then adds the top 10 unlabeled examples selected by the acquisition strategy.
- Active learning protocol: After each acquisition, the summarization model is fine-tuned from scratch on the expanded labeled pool and evaluated on held-out data.
- Baselines: Comparisons include random sampling, IDDS, BAS, and DUAL, representing diversity-based, uncertainty-based, and hybrid approaches.
- Implementation: Experiments use BART-base and PEGASUS-large backbones on an NVIDIA L4 GPU with HuggingFace Transformers.
- Evaluation metrics: Summary quality is measured with ROUGE variants and BERTScore, focusing analysis on ROUGE-1 and BERTScore.
5 Results Analysis
LOBSTER remains competitive across datasets, metrics, and backbones while addressing cold-start instability and substantially reducing active-learning selection latency. Its loss-based anchors improve selection in longer-document settings, although random sampling becomes competitive as annotation budgets grow.
- Competitive Effectiveness: LOBSTER is consistently competitive with or superior to state-of-the-art baselines across datasets, metrics, and backbone architectures.Performance varies by dataset, metric, and architecture; BAS leads on CNN/DM with PEGASUS, while DUAL leads on XSum with BART.
- Addressing the Cold Start Problem: LOBSTER avoids the severe cold-start problem observed for IDDS during initial acquisition cycles.BERTScore results indicate that IDDS’s early lag reflects reduced semantic understanding, not only n-gram prediction failure.
- Scalability and Runtime Analysis: LOBSTER shifts computation from the unlabeled pool to the smaller labeled set using teacher-forced loss and embedding-space similarity.This deterministic design avoids repeated autoregressive generation and supports scalability for large pools and high-parameter models.
- Scalability and Runtime Analysis: 665× faster than BAS and 437× faster than DUAL, LOBSTER selects instances in 1.6 seconds on CNN/DM with PEGASUS-Large.BAS and DUAL require 1064.2 and 699.4 seconds, respectively, in this configuration.
- Impact of Loss-Based Hard Example Selection: High-loss anchor selection outperforms random-anchor selection on CNN/DM for both BART and PEGASUS, while gaps are smaller on AESLC and XSum.Longer CNN/DM documents and summaries make loss-guided selection more beneficial; similar trends appear in ROUGE and BERTScore.
- Data Efficiency: Random sampling narrows the performance gap under larger annotation budgets and reaches the 90% threshold with 1,600 samples in BART-base/XSum.LOBSTER reaches the target in 5 of 6 configurations, whereas IDDS fails in half of the evaluated scenarios.
6 Conclusions and Future Work
The paper concludes that labeled-example cross-entropy loss can guide active learning by selecting semantic twins of high-loss cases. LOBSTER matches or exceeds state-of-the-art baselines with substantially lower runtime, while random sampling remains competitive at large budgets.
- Conclusions: Cross-entropy loss identifies high-loss labeled examples whose semantic twins guide annotation toward the model’s specific weaknesses.A density-based IDDS pre-filter prevents semantic collapse before final selection.
- Conclusions: LOBSTER matches or outperforms state-of-the-art baselines across three datasets and two backbone architectures while remaining significantly faster.
- Conclusions: Random sampling remains surprisingly competitive as annotation budgets increase because larger uniform subsets increasingly capture the underlying data distribution.
- Future Work: Future work will examine LOBSTER’s applicability to sequence-to-sequence generation tasks beyond abstractive summarization.
Limitations
The study identifies limitations concerning language coverage, automatic evaluation, embedding assumptions, and simulated annotation. These constraints leave effectiveness beyond English and real-world annotation settings unresolved.
- Experiments use only English-language datasets, leaving LOBSTER's effectiveness for other languages unexplored.Different morphology, syntax, or summarization styles may influence embedding similarity and active learning selection.
- ROUGE and BERTScore may not fully capture factual consistency, coherence, or overall summary quality.
- LOBSTER assumes Sentence-BERT semantic similarity provides a useful proxy for transferring hardness from labeled to unlabeled examples.Alternative retrieval spaces and jointly learned task-specific embeddings remain future-work directions.
- Experiments simulate human annotation by retrieving ground-truth summaries from existing datasets rather than collecting new annotations.This setup may not fully reflect real-world annotation scenarios.
Ethical Considerations
The study relies on pre-trained models and existing benchmark corpora, whose biases may influence instance selection and propagate to downstream summarization systems.
- Underlying model or dataset biases could influence active learning instance selection and propagate to downstream summarization systems.Future work could investigate bias-aware acquisition strategies or fairness constraints.
A Dataset Statistics and Model Hyperparameters
This section documents the benchmark datasets, their reported statistics, and the configuration and reproducibility practices used for model fine-tuning and active learning comparisons.
- Dataset statistics report instance counts and average source-document and summary lengths for training and test splits.
- Hyperparameters for fine-tuning the backbone summarization models are reported in Table 3.
- The study uses AESLC, XSum, and CNN/DailyMail as standard public summarization benchmarks.The datasets were used in their standard versions and original preprocessing without additional data collection.
- Experiments use publicly available backbone implementations through HuggingFace Transformers with consistent hyperparameters across active learning strategies.This consistency supports fair comparison among strategies.
- The full experimental pipeline and proposed active learning implementation are provided anonymously for peer review.
B Summarization Performance
Table 4 reports summarization performance under a 150-instance annotation budget across three datasets, using ROUGE and BERTScore, with significance tests reported separately.
- 150-instance annotation budgets are evaluated across AESLC, XSum, and CNN/DailyMail using ROUGE and BERTScore.Statistical significance tests comparing LOBSTER with baselines are reported in Table 5.
C Detailed Data Efficiency Setup Results
The appendix evaluates data efficiency, statistical significance, runtime, and performance across annotation budgets, datasets, backbones, and active-learning strategies. Results emphasize LOBSTER’s consistent performance and balance between reliability and computational efficiency, while also examining selection diversity and random-anchor effects.
- Performance: LOBSTER maintains strong and consistent performance across most setups, while random sampling becomes competitive at larger annotation budgets.These trends are reported alongside comparisons across datasets and backbone models.
- Runtime: LOBSTER balances performance reliability and computational efficiency relative to uncertainty-based approaches.Runtime is measured as average instance-selection time for one active-learning iteration in Table 8.
- Statistical analysis: Statistically significant differences between LOBSTER and baselines are identified with paired permutation tests using p < 0.05 and directional arrows.Table 5 reports comparisons for ROUGE-1 and BERTScore at a 150-instance annotation budget.
- Data efficiency: Table 7 measures annotated samples needed to reach 90% of full-dataset performance, with dashes marking configurations that miss the target.The analysis spans different datasets and backbone models.
- Performance curves: Figures compare labeled-instance counts against rescaled BERTScore and ROUGE-1 across BART and PEGASUS on AESLC, XSum, and CNN/DailyMail.The ROUGE-1 plots include a dashed 90% full-dataset baseline threshold.
- Ablation analysis: The LOBSTER ablation with random anchors and the ablation without IDDS assess the effects of anchor selection and diversity filtering.The IDDS module is described as preventing semantic collapse by filtering redundant candidates, with higher ROUGE-1 scores.