Source-linked AI summary
Cross Lingual Abstractive Summarization for Marathi
Asang Dani, Shailesh R Sathe
TL;DR
Indic monolingual and cross-lingual summarization remain comparatively underexplored, and Marathi cross-lingual resources are limited in size and vary in construction quality. The paper builds synthetic training data from MahaSum, combines it with PMIndiaSum, and fine-tunes IndicTrans2. Its best-performing Marathi-to-English configuration uses the 200M model with both datasets.
Problem
Method
The paper creates synthetic cross-lingual pairs from MahaSum by translating one field and retaining the other, then fine-tunes IndicTrans2 with PMIndiaSum data.
Source: §3 · pp. 3–4, §4 · p. 4Results
The IndicTrans2-200M model fine-tuned on PMIndiaSum+MahaSum was the best-performing Marathi-English CLS model, achieving ROUGE-1 75.65, ROUGE-2 56.37, and ROUGE-L 72.58.
Source: §7 · p. 7Takeaways & Limitations
The PMIndiaSum+MahaSum configuration improves on PMIndiaSum-only training in the reported Marathi-to-English evaluation.
Source: §7 · p. 7Reported results focus primarily on Marathi-to-English because obtaining an equivalent English-to-Marathi evaluation required significant GPU resources.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Text summarization is an important task for advancing natural language processing (NLP) research and applications in Indic languages. Despite progress in extractive and abstractive summarization, both monolingual summarization (MS) and cross-lingual summarization (CLS) remain relatively underexplored for these languages. Recent advances in Transformer architectures, pretrained language models (PLMs), and dataset availability offer new opportunities to address this gap. In this work, we construct cross-lingual training data from the monolingual Marathi MahaSum corpus and combine it with the smaller cross-lingual PMIndiaSum dataset to fine-tune IndicTrans2, an Indic neural machine translation model. We evaluate the resulting models using ROUGE, chrF and chrF++ metrics, focusing primarily on Marathi-to-English summarization due to computational constraints. We also discuss the applicability of our approach to the English-to-Marathi direction. Our experiments demonstrate that leveraging a pretrained translation model yields substantially better summarization performance than the IndicBART and mBART-50 based baselines reported in previous work on cross-lingual summarization.
1 Introduction
Summarization spans extractive selection and abstractive generation, but monolingual and cross-lingual summarization for Indic languages remain comparatively underexplored. This paper investigates abstractive Marathi–English cross-lingual summarization, focusing on Marathi-to-English.
1 Introduction
- Extractive summarization selects important source sentences, whereas abstractive summarization generates concise text conveying the source's key ideas.The paper places summarization among challenging text-generation tasks.Source: §1 · p. 1
- Indic-language monolingual and cross-lingual summarization have received comparatively limited attention despite their importance to NLP research and applications.The paper identifies morphological richness, linguistic diversity, and limited high-quality datasets as challenges.Source: §1 · p. 1
- The study investigates abstractive cross-lingual summarization for Marathi–English, with its primary focus on Marathi-to-English.The introduction notes that results vary with corpus availability and other factors.Source: §1 · p. 1
2 Related Work
Prior work covers Indic monolingual and cross-lingual summarization, with datasets and model choices varying across languages and tasks. The related work includes neural pipelines, multilingual corpora, and evaluations using both automatic metrics and human assessment.
2 Related Work
- Zhu et al. [7] constructed English–Chinese and Chinese–English cross-lingual data through round-trip translation and reported neural CLS outperforming conventional pipelines.Their data came from a monolingual summarization corpus.Source: §2 · p. 2
- HindiSumm contains 570,000 Navbharat Times article–summary pairs across 21 domains and was evaluated using ROUGE-1, ROUGE-2, and ROUGE-N.Its authors report human-expert evaluation and fine-tuned mT5 [10] and mBART25.Source: §2 · p. 2
- ILSUM 2.0 supports Hindi, Gujarati, and Bengali, with about 15,000 new-article summary pairs per language and Hindi evaluation using ROUGE and BERTScore [14].The Hindi dataset has 24,225 entries; IndicBART and mBART-50 [6] were fine-tuned.Source: §2 · p. 2
- MahaSum [2] supplies about 25,000 Marathi news articles with manually vetted summaries, and its authors report better ROUGE-1, ROUGE-2, and ROUGE-L performance than XLSum [15].The work describes it as a gold-standard abstractive Marathi summarization dataset.Source: §2 · p. 2
- Deshpande et al. proposed two Indic CLS pipelines and an evaluation metric combining concept coverage, semantic similarity, and relevance; CrossSum covers 1500+ language pairs.CrossSum studies mT5 [10] with m2o, m2m, and o2m approaches, and reports that MLS with m2m consistently outperforms the alternatives.Source: §2 · p. 2
- PMIndiaSum [3] provides multilingual and cross-lingual headline summaries for 14 Indic languages, while Urlana analyzed dataset quality and fine-tuned IndicBART [5] and mBART-50large [6].The reported evaluation used ROUGE-2 and ROUGE-L.Source: §2 · p. 2
3 Dataset Selection & Preparation
The paper combines Marathi monolingual corpora converted into synthetic cross-lingual data with native cross-lingual resources for training and evaluation. Round-trip checks assess translation fidelity, while synthetic XL-Sum translation quality was not directly evaluated.
3 Dataset Selection & Preparation
- MahaSum and Marathi XL-Sum provide monolingual pairs for synthetic training, while PMIndiaSum and CrossSum provide cross-lingual evaluation data with different sources and collection methods.The resources vary in language coverage, domain, corpus size, and document–summary construction.Source: §3 · p. 2
- The paper uses IndicTrans2-200M for neural translation and a Rotary IndicTrans2 model for longer-context Marathi-to-English document translation.The translation models also handle code-mixed content in MahaSum and XL-Sum; the approach uses Marathi-to-English as its primary reported direction.Source: §3 · pp. 2–3
- MahaSum's 25,374 manually verified Marathi news articles provide the primary synthetic-data source; one article or summary field is translated while the other is retained.Marathi-to-English training retains Marathi articles and translates summaries; English-to-Marathi retains Marathi summaries and translates article texts.Source: §3 · p. 3
- Summary round-trip translation on 1,667 MahaSum validation examples achieved mean chrF 56.34 and chrF++ 49.97.Mean character- and word-length ratios were 0.967 and 0.994, indicating close preservation of summary length.Source: §3 · p. 3, Table 1 · p. 4
- Document round-trip translation on 1,667 MahaSum validation examples achieved mean chrF 63.61 and chrF++ 59.15.Mean character- and word-length ratios were 0.971 and 0.994, respectively.Source: §3 · p. 3, Table 2 · p. 4
- Document-translation metrics were marginally better than summary back-translation metrics, and manual checks found both synthetic datasets' translation quality satisfactory.These pipeline checks validate translation quality but do not replace direct evaluation of the final CLS models.Source: §3 · p. 3
- Translation quality for synthetic XL-Sum datasets was not evaluated because the process was computationally expensive.The authors expected quality to be good based on MahaSum round-trip results and fine-tuned model results.Source: §3 · p. 3
- PMIndiaSum [3] contributes about 4,400 human-translated examples and serves as a primary reference, but is too small to use alone; its test split is held out from fine-tuning.CrossSum contributes 1,181 Marathi examples, while the English XL-Sum subset supplies additional synthetic English-to-Marathi data.Source: §3 · pp. 3–4
4 Summarization Approach and PLM Selection
The paper selects IndicTrans2 as its generative backbone for Marathi–English cross-lingual summarization, using models that cover Indic languages and English. It also examines Rotary positional embeddings and defines dataset configurations for the two directions.
4 Summarization Approach and PLM Selection
- IndicTrans2 is the primary generative backbone investigated for Marathi–English cross-lingual summarization, with support for all 22 scheduled Indian languages.The authors report that it performs well in both English-to-Marathi and Marathi-to-English directions.Source: §4 · p. 4
- The examined Marathi-to-English models include 1B and distilled 200M Indic-to-English variants, with and without rotary positional embeddings.The paper also uses corresponding English-to-Indic models for the English-to-Marathi direction.Source: §4 · p. 4
- Rotary positional embeddings are studied because they enable contexts of up to 2,048 tokens; the approach does not use IndicBART or mT5.The authors contrast this model choice with Urlana et al. [3].Source: §4 · p. 4
- Table 4 specifies document and summary languages, the field translated with IndicTrans2, and whether each configuration is synthetic training data or native cross-lingual evaluation data.The table covers construction and use of datasets for both cross-lingual directions.Source: Table 4 · p. 5
5 Experimental Setup
Experiments were run on fine-tuned pretrained models with tracked runs and retained best checkpoints. Reported results focus on Marathi-to-English, and the setup includes dataset-distribution and hyperparameter comparisons.
5 Experimental Setup
- Fine-tuning used NVIDIA A100 GPUs with 40GB VRAM, distributed across multiple GPUs using torchrun, with MLflow tracking and best-checkpoint retention.Checkpointing also allowed runs to resume after interruption.Source: §5 · p. 5
- Reported results focus on Marathi-to-English because this direction is considered more challenging, while obtaining equivalent English-to-Marathi results requires significant GPU resources.The paper identifies results explicitly by direction.Source: §5 · p. 5
- Figure 1 compares source-text length, summary length, and compression-ratio distributions across four corpora, with training and validation splits distinguished.Its fourth panel plots source-text against summary length.Source: Figure 1 · p. 6
- The reported optimal fine-tuning settings include a dropout rate of 0.1 and batch size of 8.Table 5 covers hyperparameters considered for IndicTrans2 fine-tuning.Source: §5 · pp. 6–7, Table 5 · p. 6
6 Evaluation and Validation
Evaluation combines metric-based checkpoint selection with manual inspection of generated summaries for content quality.
6 Evaluation and Validation
- ROUGE-L selects the best-performing checkpoint from each training run.ROUGE is described as a widely used summarization metric.Source: §6 · p. 7
- Generated summaries are manually inspected for meaningfulness, coherence, and coverage of the source document’s central content.Source: §6 · p. 7
- ROUGE and BERTScore are identified as metrics for summarization and semantic evaluation, respectively.Source: §6 · p. 7
7 Results
IndicTrans2-200M fine-tuned on PMIndiaSum and MahaSum is reported as the best Marathi–English cross-lingual summarization model in these experiments. Training-data comparisons show that MahaSum complements PMIndiaSum, while XL-Sum Marathi’s effects are smaller or mixed.
7 Results
- Among IndicBART and mBART-50 baselines, PMIndiaSum+MahaSum achieved the best results across reported metrics, including chrF 31.8064 and chrF++ 30.2792.Baseline results are reported on the PMIndia-MR test set; Table 6 covers IndicBART and Table 7 covers mBART-50.Source: §7 · p. 7, Table 6 · p. 10, Table 7 · p. 10
- 75.65 ROUGE-1, 56.37 ROUGE-2, 72.58 ROUGE-L, and 99.15 BERTScore-F1 were achieved by the 200M model trained on PMIndiaSum+MahaSum.Relative to PMIndiaSum-only training, the combined-data model improved ROUGE-1, ROUGE-2, and ROUGE-L by 3.11, 4.02, and 3.17 points, respectively.Source: §7 · p. 7
- Adding XL-Sum Marathi to PMIndiaSum produced smaller 200M-model gains, with ROUGE-1 rising from 72.55 to 72.99, ROUGE-2 from 52.36 to 53.72, and ROUGE-L from 69.41 to 70.35.MahaSum-only and XL-Sum-Marathi-only training scored ROUGE-1/2/L of 40.96/23.23/37.42 and 40.35/23.64/37.22, respectively.Source: §7 · p. 7
- For the 1B model, PMIndiaSum+MahaSum achieved ROUGE-1 72.50, ROUGE-2 53.31, ROUGE-L 69.81, chrF 64.99, and chrF++ 62.57.Against PMIndiaSum-only training, its ROUGE gains were 0.48, 0.11, and 0.10 points for ROUGE-1, ROUGE-2, and ROUGE-L, respectively.Source: §7 · p. 7
- For the 1B model, adding XL-Sum Marathi raised ROUGE-1 from 72.02 to 72.25 but lowered ROUGE-2 from 53.20 to 52.72 and ROUGE-L from 69.71 to 69.45.The authors say MahaSum provides limited but measurable complementary information, whereas XL-Sum Marathi does not consistently improve performance in this setup.Source: §7 · p. 7
- IndicTrans2-200M fine-tuned on PMIndiaSum+MahaSum is reported as the best Marathi–English cross-lingual summarization model and as outperforming the existing baseline.The authors say why the smaller 200M model performs better than the 1B model requires further investigation.Source: §7 · p. 7
8 Conclusion
Cross-lingual summarization remains challenging for Indian languages. The conclusion points to larger, higher-quality datasets and advances in pretrained models as possible routes to better performance, alongside evaluation, faithfulness, and inference-cost priorities.
8 Conclusion
- Cross-lingual summarization remains challenging for Indian languages.The conclusion says continued advances in pretrained language models and larger, higher-quality datasets should improve performance.Source: §8 · p. 10
- Future work should investigate stronger human evaluation and methods to improve factual consistency and cross-lingual faithfulness.Source: §8 · p. 10
- Lower inference cost is identified as important for making multilingual content more accessible.Source: §8 · p. 10
9 Future Work
Future work will examine how much multilingual capacity cross-lingual summarization needs and explore model compression. The stated focus is the English–Marathi pair, with quantization and language-pair specialization proposed as possible avenues.
9 Future Work
- The authors plan to investigate how much multilingual capacity is necessary for cross-lingual summarization.Source: §9 · p. 10
- Deep Compression approaches are proposed for reducing Indic NLP models’ inference cost, with English–Marathi as the focus.Source: §9 · p. 10
- The authors hope quantization and language-pair specialization can yield substantial model compression.IndicTrans2 shares decoder input/output embeddings, and the decoder output projection represents a substantial portion of the parameter count.Source: §9 · p. 10