PaperlayerView exact revision on arXiv

Source-linked AI summary

IndicBART: A Pre-trained Model for Indic Natural Language Generation

Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra, Pratyush Kumar

arXiv:2109.02903v2cs.CLcs.AI

TL;DR

Massively multilingual models have limitations in language coverage, low-resource vocabulary representation, and computational demands, while pre-trained sequence-to-sequence modeling for Indic languages was an identified gap. IndicBART trains a compact model for related Indic languages using shared-script representation, and the paper reports competitive results across two generation tasks and very low-resource translation scenarios.

  • Problem

    Massively multilingual models have limited language coverage, high-resource-dominated pre-training data, inadequate low-resource vocabulary representation, and high computational costs, while Indic-focused multilingual pre-trained sequence-to-sequence modeling was missing.

    Source: §1 · p. 1

  • Method

    IndicBART is a compact multilingual pre-trained sequence-to-sequence model for 11 Indic languages and English that maps Indic data to a shared script.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    IndicBART is competitive with or up to 2 BLEU/ROUGE better than mBART50 on machine translation and extreme summarization, and performs well in zero-shot translation scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study presents language-group-specific pre-trained sequence-to-sequence modeling as a case study and reports a strong positive impact of script unification on translation and summarization.

    Source: §6 · p. 9

    A 60% parameter reduction versus IndicBART and an 84% reduction versus mBART50 negatively affect translation quality, with drops of up to 3 BLEU.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.

1 Introduction

The introduction motivates Indic-specific pre-trained sequence-to-sequence models as an alternative to massive multilingual models and presents IndicBART for 11 Indian languages and English. It reports competitive generation results and highlights compactness, related-language transfer, and shared Indic-script representation.

1 Introduction

  • Massively multilingual models cover fewer than 100 languages, have high-resource-dominated corpora and inadequate low-resource vocabularies, and are expensive and slow to train, fine-tune, and decode.The paper notes that a study of multilingual pre-trained sequence-to-sequence models for Indic languages was missing from the literature.Source: §1 · p. 1
  • The 244M-parameter IndicBART supports English and 11 Indic languages, while mBART25, mBART50, and mT5 support 2, 7, and 9 languages, respectively.The supported Indic languages span seven Indo-Aryan and four Dravidian languages.Source: §1 · pp. 1–2
  • IndicBART is competitive with or up to 2 BLEU/ROUGE better than mBART50 on machine translation and extreme summarization, and performs well in zero-shot translation scenarios.The zero-shot scenarios involve languages absent from pre-training or languages with no fine-tuning data.Source: §1 · p. 2
  • IndicBART maps Indic data from nine scripts to one, increasing shared vocabulary subwords and supporting better cross-lingual transfer while fine-tuning.The introduction also identifies training on related languages and effective use of model capacity as contributors to performance.Source: §1 · p. 2
  • IndicALBART shares Transformer-layer parameters to create a 97M-parameter variant, and compressed pre-trained models are competitive with full models when fine-tuned on distilled data.The paper also reports training on IndicCorp, which includes large, high-quality Indian-language news crawls and English content from Indian websites.Source: §1 · p. 2

2 Related Work

The related-work discussion positions IndicBART among multilingual pre-trained models and language-group-specific NLP systems. It emphasizes its focus on pre-trained sequence-to-sequence modeling for Indic languages, shared-script transfer, and compression through parameter sharing and distillation.

Language group-specific models

  • IndicBART is presented as one of the first pre-trained sequence-to-sequence models for a specific language group, and the first for Indic languages.Prior language-group-specific work cited includes IndicBERT, MuRIL, and NMT models; concurrent AfroMT focuses on African languages.Source: §2 · p. 2

Pre-trained models; Language relatedness

  • Prior work on multilingual pre-trained models supports around 100 languages, while language-group-specific models and language relatedness have also been studied for NLU and NMT.The paper uses orthographic similarity in Indic scripts, drawing on prior common-script work in transliteration, translation, and NLU.Source: §2 · pp. 2–3

Parameter Sharing and Distillation

  • IndicALBART explores cross-layer parameter sharing for pre-trained sequence-to-sequence models, while the paper reports that distillation findings also hold when fine-tuning pre-trained models.The related-work discussion contrasts this with earlier parameter-sharing work in NMT and pre-trained language models, and distillation for NMT models trained from scratch.Source: §2 · p. 3

3 IndicBART

IndicBART adapts the mBART model family to related Indic languages, using a shared script and a compact model design. The paper describes its architecture, training choices, and data preparation, including variants for parameter sharing and original-script ablations.

3 IndicBART

  • IndicBART is conceptually based on mBART25/50 Transformer models trained on monolingual corpora with a masked-span reconstruction objective.Source: §3 · p. 3

3.1 Design Considerations for IndicBART

  • The design prioritizes compactness, relevant Indian-subcontinent English content, and Indic-language relatedness by mapping data to Devanagari.The authors state that this enables better transfer learning with a more compact vocabulary than mBART.Source: §3.1 · p. 3

3.2 Model and Training Details

  • IndicBART uses six encoder and decoder layers and has 244M parameters; IndicALBART explores compression by sharing parameters across layers.Longer training was limited by the availability of many GPUs simultaneously.Source: §3.2 · p. 3
  • The authors compare a 64K-vocabulary separate-script IndicBART using original scripts and describe Devanagari mapping as enabling direct vocabulary sharing and improved transfer learning.Source: §3.2 · pp. 3–4

3.3 Training Data and Pre-processing

  • Training uses IndicCorp’s 11 Indic languages and English, with approximately 450 million sentences and 9 billion tokens, balanced by temperature-based sampling.All Indic data is represented in Devanagari; the model is trained at sentence level, unlike mBART50’s contiguous text chunks.Source: §3.3 · p. 4

4 Experiments: NMT

The experiments compare IndicBART and its variants with mBART50 and models trained from scratch across low-resource, multilingual, and zero-shot machine translation settings. IndicBART is competitive with mBART50, while ablations examine script unification, corpus size and domain, and transfer to languages unseen during fine-tuning or pre-training.

4.1 Models Compared; 4.2 Datasets and Preprocessing

  • The study compares scratch-trained bilingual and multilingual Transformers with fine-tuned mBART50, IndicBART, IndicALBART, and separate-script IndicBART, including distilled variants.Low-resource fine-tuning uses PMI data; experiments also use expanded PMI+PIB data and the general-domain Samanantar corpus, with WAT 2021 and FLORES evaluation.Source: §4.1 · p. 4, §4.2 · pp. 4–5

4.3 Model Training Settings

  • Multilingual training balances language-pair corpora, adds target-language indicators, concatenates the data, and trains Transformers with Adam until development BLEU converges.Decoding uses beam size 4 and length penalty 0.8; BLEU is computed with sacreBLEU.Source: §4.3 · p. 5

4.4 Comparison of Pre-trained Models

  • Across bilingual and multilingual settings, IndicBART is competitive with mBART50, sometimes reverses the gap, and outperforms it for Kannada, Punjabi, and Oriya, which mBART50 was not trained for.The comparison uses the WAT 2021 test set; IndicBART also outperforms models trained from scratch.Source: §4.4 · p. 5, Table 1 · p. 6
  • IndicALBART’s 60% reduction from IndicBART and 84% reduction from mBART50 correspond to translation-quality drops of up to 3 BLEU, while distillation can let it match IndicBART.Fine-tuning multilingual models takes 3.3–3.5 hours for IndicBART variants versus 4.7–5 hours for mBART50, with 97–244M versus 611M parameters; self-training improves XX-En and modestly improves En-XX directions.Source: §4.4 · p. 5

4.5.1 Impact Of Script Unification

  • Unified-script IndicBART performs better overall than separate-script IndicBART, while script unification improves transfer to Kannada, Punjabi, and Oriya that mBART50 did not pre-train on.For these languages, converting training data to Devanagari still yields large gains over scratch-trained baselines, often close to IndicBART; explicitly training on the target languages should improve quality.Source: §4.5.1 · pp. 6–7

4.5.2 Impact Of Corpora Size and Domain

  • Increasing fine-tuning data improves translation on both WAT 2021 and FLORES, but the best corpus depends on test domain: PMI+PIB is competitive on WAT, while Samanantar is better on FLORES.With large parallel corpora, pre-training has minimal impact.Source: §4.5.2 · p. 7, Table 3 · p. 7

4.5.3 Unseen Languages During Fine-Tuning

  • For Kannada and Punjabi unseen during fine-tuning, translation into English remains reasonable, but English-to-Indic quality is poor; script unification can improve results by up to 3.5 BLEU.A monolingual de-noising objective for unseen target languages could alleviate the decoder limitation.Source: §4.5.3 · p. 7, Table 4 · p. 7

4.5.4 Unseen Languages During Pre-Training

  • Fine-tuning on Nepali and Sinhala, which IndicBART did not pre-train on, substantially improves translation, and results are not far behind an mBART model pre-trained on both languages.The Nepali and Sinhala vocabulary was not trained for those languages; Sinhala data uses script mapping into Devanagari.Source: §4.5.4 · p. 8, Table 5 · p. 8

5 Experiments: Extreme Summarization

IndicBART, its compressed and separate-script variants, and mBART50 are compared on Indic extreme summarization using XL-Sum. IndicBART and mBART50 are competitive, while separate-script IndicBART generally performs worse and IndicALBART often trails IndicBART by 1–3 Rouge-L points.

5.1 Models Trained

  • The comparison includes mBART50, IndicBART, IndicALBART, and separate-script IndicBART; Punjabi is mapped to Devanagari for mBART50 fine-tuning.Source: §5.1 · p. 8

5.2 Datasets and Preprocessing

  • The experiments use updated multilingual XL-Sum splits for Bengali, Gujarati, Hindi, Marathi, Punjabi, Tamil, and Telugu; non-parallel splits preclude multilingual fine-tuning.Scripts are mapped to Devanagari for IndicBART and IndicALBART, and for Punjabi with mBART50; separate-script IndicBART keeps the scripts separate.Source: §5.2 · p. 8

5.3 Model Training Settings

  • Models are fine-tuned with document-summary lengths of 512–64 tokens and selected by development-set Rouge-L F1, then evaluated with beam size 5.Decoding uses length penalty 1.2 and a 4-gram repetition limit.Source: §5.3 · p. 8

5.4 Results

  • IndicBART and mBART50 are competitive on XL-Sum, with IndicBART slightly better for Marathi, Punjabi, Tamil, and Telugu; separate-script IndicBART fares poorer except for Punjabi.The authors attribute the separate-script comparison to the importance of script unification.Source: §5.4 · p. 8, Table 6 · p. 8
  • IndicALBART often lags IndicBART by 1–3 Rouge-L points, a tradeoff the authors consider reasonable given its reduced parameter size.They expect distillation may help improve performance.Source: §5.4 · p. 9

6 Conclusion and Future Work

The paper presents IndicBART as a compact multilingual model for Indian-language NLG and reports competitive translation and summarization performance against larger models. It also identifies future plans to expand language coverage and train on longer texts and larger corpora.

6 Conclusion and Future Work

  • IndicBART supports 11 Indian languages and English, uses Indic-script orthographic similarity for cross-lingual transfer, and is competitive with large models on translation and summarization.The conclusion also reports positive effects from script unification and translation for languages not explicitly included in pre-training or fine-tuning.Source: §6 · p. 9
  • Future work plans to add Indic languages, train on longer text chunks and larger corpora, and incorporate advances in multilingual pre-training and cross-lingual tasks.Source: §6 · p. 9

A Corpora statistics

  • IndicCorp supplies monolingual pre-training data, while PMI, PIB, and Samanantar parallel corpora support fine-tuning or training from scratch in low-, middle-, and high-resource settings, respectively.Source: A Corpora statistics · p. 13

B NMT Model Training Settings

The NMT settings describe training resources, model configurations, and decoding and evaluation procedures. The authors note that they did not exhaustively tune hyperparameters because they trained many models.

B NMT Model Training Settings

  • Bilingual models use one GPU and multilingual models use eight; all are Transformers, and multilingual training follows Johnson et al. (2017).Source: B NMT Model Training Settings · p. 13
  • The authors did not perform exhaustive hyperparameter tuning across the many models, focusing mainly on learning rates, batch sizes, and warm-ups.Source: B NMT Model Training Settings · p. 13
  • IndicBART fine-tuning uses dropout 0.1, label smoothing 0.1, 16,000 warm-up steps, 2,048 tokens per GPU batch, learning rate 0.001, and weight decay 0.00001.mBART50 uses 2,500 warm-up steps, 512 tokens per GPU batch, and learning rate 0.00003; scratch models on PMI and PIB use smaller hidden and filter sizes.Source: B NMT Model Training Settings · p. 13
  • Models are selected by development BLEU and test sets are decoded with beam size 4 and length penalty 0.8; BLEU is computed with sacreBLEU.Source: B NMT Model Training Settings · p. 13

C NMT Results: Impact of Script Unification

The ablation compares script unification with separate-script representations in bilingual and multilingual fine-tuning. Script unification tends to produce better average results in both settings.

C NMT Results: Impact of Script Unification

D NMT Results: Effect of Corpora Size and Domain

Fine-tuning corpus size and domain shape translation quality: small in-domain PMI is competitive with large general-domain data, while general-domain corpora achieve the best performance but take more time to train. Fine-tuning also offers a viable option when target-domain training data is unavailable.

D NMT Results: Effect of Corpora Size and Domain

  • PMI/PIB fine-tuning is viable when target-domain FLORES training data is unavailable.The paper notes that adding related-domain PMI data substantially improves translation quality.Source: D NMT Results: Effect of Corpora Size… · pp. 13–14
  • Small in-domain PMI fine-tuning is competitive with large general-domain data, and related-domain corpora improve in- and out-of-domain translation.Table 9 compares different fine-tuning corpus sizes and domains on WAT 2021 and FLORES test sets.Source: D NMT Results: Effect of Corpora Size… · p. 13, Table 9 · p. 14
  • General-domain corpora yield the best performance, but their longer training time makes fine-tuning more attractive because pretraining is done only once.The comparison concerns training large models on large general-domain corpora.Source: D NMT Results: Effect of Corpora Size… · p. 14

E Corpora statistics for summarization experiments

The summarization experiments use the Indic portion of XL-Sum, with language scripts mapped to Devanagari for some model variants and not others.

E Corpora statistics for summarization experiments

F Summarization Model Training Settings

Summarization models are fine-tuned with model-specific settings, selected using development-set ROUGE-L F1, and evaluated with beam-search decoding.

F Summarization Model Training Settings

  • Summarization fine-tuning uses maximum document-summary lengths of 512-64 tokens, and models are not trained from scratch because of the small data sizes and summarization difficulty.The stated lengths loosely follow prior work.Source: F Summarization Model Training Settings · p. 15
  • IndicBART variants use 4,096-token batches, 0.1 dropout and label smoothing, 4,000 warmup steps, a 0.001 learning rate, and 0.00001 weight decay.They use the ADAM optimizer; mBART50 instead uses sentence-level batches of two pairs and a 0.00001 learning rate.Source: F Summarization Model Training Settings · p. 15
  • Checkpoints are selected by development-set ROUGE-L F1, evaluated greedily every 1,000 batches; test decoding uses beam size 5 and length penalty 1.2.Decoding also limits repeated 4-grams, and results use a multilingual ROUGE scoring toolkit.Source: F Summarization Model Training Settings · p. 15

Related papers

Loading 2109.02903v2…