Source-linked AI summary
Liputan6: A Large-scale Indonesian Dataset for Text Summarization
Fajri Koto, Jey Han Lau, Timothy Baldwin
TL;DR
Indonesian summarization lacks large datasets, limiting resources for this under-represented NLP language. The paper introduces a 215,827-pair Liputan6 corpus and BERT-based extractive and abstractive benchmarks, then analyzes model errors and ROUGE. The analysis finds that many low-ROUGE summaries are good despite paraphrasing and valid additional details, while also identifying modeling issues.
Problem
Indonesian is under-represented in NLP because large datasets for tasks such as summarization are scarce.
Method
The paper harvests a large Liputan6 corpus, builds extractive and abstractive BERT-based benchmarks, and conducts error analysis of generated summaries.
Results
Low ROUGE scores often reflect good summaries with paraphrasing, abbreviations, morphological differences, or valid additional details rather than poor quality.
Takeaways & Limitations
The analysis supports exploring semantic evaluation metrics such as METEOR and BERTSCORE beyond word overlap.
Takeaways & Limitations
The authors suggest BERTSCORE as the canonical metric but leave empirical validation of its superiority for Indonesian summarization to future work.
Abstract
from arXiv · showhide
In this paper, we introduce a large-scale Indonesian summarization dataset. We harvest articles from Liputan6.com, an online news portal, and obtain 215,827 document-summary pairs. We leverage pre-trained language models to develop benchmark extractive and abstractive summarization methods over the dataset with multilingual and monolingual BERT-based models. We include a thorough error analysis by examining machine-generated summaries that have low ROUGE scores, and expose both issues with ROUGE it-self, as well as with extractive and abstractive summarization models.
1 Introduction
Indonesian is under-represented in NLP because large datasets for tasks including summarization are scarce. This paper addresses that gap with a large Liputan6 corpus, benchmark BERT-based models, and error analysis exposing modeling and ROUGE issues.
- Indonesian is under-represented in NLP, partly because large datasets for tasks such as summarization are scarce.
- Large English summarization datasets have supported neural progress, whereas sufficiently large non-English resources remain limited.
- The paper introduces a large-scale Indonesian summarization corpus harvested from Liputan6.com over a 10-year period.
- The paper conducts error analysis to identify limitations in summarization models and problems with ROUGE-based evaluation.
- The corpus contributes over 200K documents, statistics on abstractiveness, canonical and extremely abstractive test partitions, and benchmark extractive and abstractive BERT models.
2 Data Construction
The authors harvest 215,827 Liputan6 document–summary pairs from a decade of Indonesian news and construct canonical and more abstractive evaluation partitions. Analyses indicate that Liputan6 summaries are more abstractive than IndoSum and that the dataset is substantially larger.
- 215,827 document–summary pairs were harvested from Liputan6.com over a 10-year period and preprocessed by removing formatting, lowercasing, and sentence segmentation.
- Novel n-gram proportions drop over time, so earlier articles were used for development and testing to create a more challenging setting with less topic overlap.
- The dataset uses a 90:5:5 train–development–test split and an Xtreme variant that retains development and test summaries with at least 90% novel 4-grams.
- Liputan6 is approximately 11 times larger than IndoSum, although its articles and summaries are slightly shorter.
- Liputan6 has lower LEAD-N ROUGE scores and a substantially higher proportion of novel n-grams than IndoSum, suggesting more abstractive summaries.
- Extractive ORACLE labels are created by greedily selecting article sentences that maximize ROUGE against each reference summary.
3 Summarization Models
The paper builds extractive and abstractive summarization models around BERT representations, using multilingual and Indonesian-specific pre-trained encoders. The models add task-specific transformer components for sentence selection or generation.
- 3 Summarization Models: Both models use BERT as an encoder, with WordPiece, sentence segment, and token position embeddings supplied to BERT.The extractive model then operates on sentence representations, whereas the abstractive model uses embeddings for all tokens.
- 3 Summarization Models: The study compares multilingual BERT with IndoBERT, trained on 220M Indonesian words from Wikipedia, news, and web-corpus sources.IndoBERT is implemented as a BERT-Base model trained with a 31,923-WordPiece vocabulary for 2 million steps.
- 3.1 Extractive Model: BERTEXT uses sentence-level [CLS] representations, a transformer encoder, and an MLP to predict which sentences to extract.Sentence selection is trained with binary cross entropy while updating BERT and randomly initialized task-specific layers.
- 3 Summarization Models: The model variants include BERTEXT, BERTABS, and BERTEXTABS, which initializes the abstractive encoder with fine-tuned BERTEXT weights.The second transformer components are trained from randomly initialized parameters rather than pre-trained weights.
- 3.2 Abstractive Model: BERTABS uses token-level BERT embeddings with a causal transformer decoder to generate abstractive summaries.A second positional encoding is added before decoding, and the decoder cannot attend to future time steps.
4 Experiment and Results
Experiments evaluate extractive and abstractive systems on canonical and highly abstractive Xtreme test sets using ROUGE and BERTSCORE. IndoBERT-based models perform best, while the Xtreme setting substantially lowers scores.
- Baselines: LEAD-2 is the strongest LEAD-N baseline, but ORACLE exceeds it by 12–15 R1 points and 5–7 BERTSCORE points across test sets.The gap indicates that selecting the first few sentences is an ineffective extractive baseline for Liputan6.
- Pointer-Generator Models: Coverage adds little improvement to PTGEN, suggesting limited repetition in its outputs for the short Liputan6 summaries.Liputan6 summaries average two sentences and 30 words, a property also noted for XSum summaries.
- BERT Models: +4.4 R1, +2.62 R2, +4.3 R3, and +3.4 BERTSCORE points over LEAD-2 make IndoBERT’s BERTEXTABS the best canonical-test model.Across models, IndoBERT generally outperforms mBERT, LEAD-N, and pointer-generator baselines; Table 4 reports ROUGE and BERTSCORE results for both test sets.
5 Error Analysis
The error analysis finds that extractive errors stem from over-selecting opening sentences, while abstractive low-ROUGE cases often reflect metric limitations alongside genuine coverage and focus problems.
- 5 Error Analysis: The error analysis used mBERT because it was the best-performing model when conducted, although IndoBERT later performed slightly better.The authors expect similar error patterns because the models are structurally identical.
- 5.2 Error Analysis of Abstractive Summaries: 100% of average summaries had coverage issues, while 75.0% included unnecessary but valid details and 87.5% used paraphrases.Bad summaries also showed coverage problems and tended to adopt a different document focus.
- 5.2 Error Analysis of Abstractive Summaries: The authors recommend BERTSCORE as the canonical dataset metric but leave empirical validation of its superiority for Indonesian evaluation to future work.This recommendation is presented as an evaluation direction rather than a completed validation result.
- 5.2 Error Analysis of Abstractive Summaries: ROUGE misses good summaries when abbreviations, morphology, paraphrases, or valid added details differ from the reference wording.The authors suggest semantic metrics such as BERTSCORE and additional references as possible remedies.
6 Related Datasets
Earlier Indonesian summarization datasets were generally small, extractive, or unavailable, while IndoSum expanded news coverage but remained highly extractive. Non-English alternatives include large Chinese and Spanish datasets with different document characteristics.
- Indonesian Datasets: Earlier Indonesian studies used datasets ranging from fewer than 200 articles to 3K news articles, often with extractive methods and unavailable data.Approaches included heuristics, naive Bayes, and genetic algorithms.
- Indonesian Datasets: IndoSum is the largest prior Indonesian summarization dataset, with approximately 19K news articles and manually written summaries that are highly extractive.Its scale exceeds earlier Indonesian datasets but remains far below the corpus introduced in this paper.
- Non-English Datasets: Among sizeable non-English resources, LCSTS contains 2 million short Chinese texts, whereas ES-News contains 270K Spanish news articles with summaries.LCSTS documents are under 140 Chinese characters, while ES-News is a Spanish news collection.
7 Conclusion
The paper releases Liputan6, a large Indonesian summarization corpus with canonical and more abstractive test sets, benchmarks BERT-based models, and analyzes ROUGE-related errors.
- Conclusion: Liputan6 is released as a large-scale Indonesian summarization corpus with canonical and more abstractive “Xtreme” test sets.The corpus is intended to support evaluation across differing levels of abstractiveness.
- Conclusion: The paper benchmarks extractive and abstractive summarization models, including systems based partly on IndoBERT.The benchmark covers multiple pretrained-model approaches.
- Conclusion: Extensive error analysis identifies issues with ROUGE-based evaluation for Indonesian summarization.The analysis also examines model behavior on the Liputan6 dataset.