PaperlayerView exact revision on arXiv

Source-linked AI summary

HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head

Thisen Ekanayake, Nisansa de Silva

arXiv:2608.22922v1cs.CL

TL;DR

Sinhala NLP lacks specialized resources, motivating HelaBERT's monolingual BERT models and a dual pooling classification head. Pre-trained from scratch and evaluated across four classification tasks, HelaBERT achieves strong monolingual results, while dual pooling helps most on sentiment and some category classification. The study also identifies task and dataset boundaries for interpreting these findings.

  • Problem

    Sinhala remains under-resourced in NLP, with scarce pre-trained models, annotated corpora, and benchmarks.

  • Method

    The paper pre-trains HelaBERT-Small and HelaBERT-Large from scratch on Sinhala text and evaluates standard and dual pooling classification heads across four tasks.

  • Results

    HelaBERT-Large achieves state-of-the-art results among monolingual Sinhala models on all four evaluated classification tasks, while dual pooling consistently improves sentiment and moderately improves HelaBERT-Small on news category classification.

  • Takeaways & Limitations

    The released models provide monolingual Sinhala encoders for classification and the dual pooling head is most useful when discriminative information is sparse in moderate-length inputs.

  • Takeaways & Limitations

    Evaluation is restricted to text classification, and sentiment results are not directly comparable with prior baselines because a different dataset was used.

Abstract

from arXiv · show

We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.

1 Introduction

HelaBERT addresses Sinhala NLP's limited resources with two monolingual BERT models and evaluates them across four classification tasks. The study also examines when a dual pooling head improves classification.

  • Sinhala NLP remains under-resourced, with scarce pre-trained models, annotated corpora, and benchmarks.
  • HelaBERT introduces two BERT-based masked language models pre-trained from scratch on approximately 1 billion Sinhala tokens.
  • The models are fine-tuned on news category, news source, sentiment, and writing style classification using five independent seed runs.
  • The dual pooling head yields consistent sentiment gains of +3.9–5.6 macro-F1 points and a +3.1-point news-category improvement for HelaBERT-Small.
  • The standard [CLS]-linear head remains competitive on short-input tasks such as news source classification.

2 Related Work

Prior Sinhala NLP work spans multilingual and monolingual language models, task-specific classification datasets, and earlier neural or conventional baselines. HelaBERT follows the monolingual modeling line while introducing a dual pooling classification head.

  • Multilingual models provide broad coverage but allocate limited capacity to low-resource languages, motivating dedicated monolingual models.
  • HelaBERT directly compares with SinBERT using the same four tasks and evaluation methodology, while differing in backbone and tokenizer.
  • SinLlama represents a complementary decoder-based Sinhala model, whereas HelaBERT targets lightweight encoder applications such as classification and sequence labelling.
  • Earlier Sinhala news-category research used Naïve Bayes, SVM, and LDA-based approaches, while this work adopts Dhananjaya et al.'s benchmark for comparability.
  • News-source identification is evaluated as a headline-level task, and writing-style classification uses a benchmark where prior BERT models already approach ceiling performance.
  • HelaBERT extends earlier Word2Vec, fastText, hierarchical-attention, and capsule-network work with interaction between [CLS] and token representations.

3 HelaBERT Pre-training

HelaBERT-Small and HelaBERT-Large share a Sinhala-focused corpus, tokenizer, and masked-language-modeling objective but differ in scale and training configuration. Both models reduced training and validation loss without significant overfitting.

  • Model setup: HelaBERT-Small and HelaBERT-Large share the same corpus, tokenizer, and MLM objective while differing in architecture scale and training configuration.
  • Pre-training data: The models were trained on approximately 900 million and 1.1 billion Sinhala tokens, respectively, from MADLAD-400, CulturaX, and a custom corpus.
  • Tokenization: The SentencePiece Unigram tokenizer has a 32,000-word vocabulary, 99.95% character coverage, and direct Unicode operation suited to Sinhala morphology and script.
  • Training results: HelaBERT-Small's training loss fell from ∼10.0 to 3.69 and validation loss from ∼7.0 to 3.49, with no significant overfitting.
  • Training results: HelaBERT-Large's training loss fell from ∼10.3 to 2.26 and validation loss from ∼7.5 to 2.17, also without significant overfitting.

4 Fine-tuning Datasets

The study fine-tunes HelaBERT on four Sinhala classification datasets covering news topics, news sources, sentiment, and writing style. The evaluation uses established task methodologies, with an important comparability limitation for sentiment.

  • The evaluation covers news category, news source, sentiment, and writing style classification tasks.
  • Datasets: The news-category dataset contains Sinhala news sentences across political, business, technology, sports, and entertainment categories.
  • Datasets: The news-source dataset contains Sinhala headlines from nine online sources.
  • Datasets: The sentiment dataset uses three labels: POSITIVE, NEGATIVE, and NEUTRAL.
  • Comparability: Sentiment results use a publicly accessible three-class dataset and are not directly comparable with prior baselines evaluated on a different four-class dataset.
  • Datasets: The writing-style dataset covers NEWS, ACADEMIC, CREATIVE, and BLOG styles.

5 Fine-tuning Methodology

HelaBERT is fine-tuned for four Sinhala classification tasks using a shared architecture and controlled evaluation protocol. Experiments use stratified splits, five random seeds, macro-F1 selection, and task-specific training settings.

  • Both HelaBERT models are fine-tuned on four Sinhala text classification tasks using a pretrained backbone followed by a classification head.
  • Five independent seed runs and a stratified 80/20 train/test split are used for every task.
  • Macro-F1 is the primary metric, and the reported result is the highest-test-macro-F1 run for each task and model.
  • Standard Classification Head: The standard classification head applies dropout and a linear layer to the [CLS] representation.The linear layer maps hidden size H to the number of classes with dropout probability 0.1.
  • Training Configuration: Training uses AdamW, linear scheduling with 6% warmup, weight decay 0.01, batch size 16, and FP16 mixed precision.The five seeds are 42, 123, 456, 789, and 1024.

6 Results: Comparison with Baselines

HelaBERT-Large performs strongly against multilingual and monolingual baselines on news category, news source, and writing style classification. Sentiment results are reported separately because HelaBERT uses a different three-class dataset, preventing direct baseline comparison.

  • News Category: 90.38% macro-F1: HelaBERT-Large surpasses XLM-R-large on News Category classification by 0.8 points.It also exceeds SinBERT-Large (85.19%) by 5.2 points, while HelaBERT-Small reaches 85.97%.
  • News Source: 63.65% macro-F1: HelaBERT-Large exceeds SinBERT-Large on News Source classification by 3.1 points.HelaBERT-Small scores 60.16%, marginally below SinBERT-Small at 60.42%.
  • Writing Style: 97.73% macro-F1: HelaBERT-Large falls 0.7 points short of XLM-R-large on Writing Style classification but exceeds SinBERT-Large by 2.2 points.HelaBERT-Small reaches 95.92% and outperforms all SinBERT variants.
  • Sentiment: HelaBERT sentiment scores are not directly comparable with the baselines because they use a different three-class dataset.HelaBERT-Small scores 65.34% and HelaBERT-Large scores 64.60%, reported for reference only.

7 Dual Pooling Classification Head

The dual pooling head combines [CLS]-to-token and token-to-[CLS] attention before classification, adding sequence-level information beyond the standard [CLS] representation. Its benefits are task- and model-dependent, strongest for sentiment and smaller-model news category classification, while the standard head remains competitive on short or near-saturated tasks.

  • Dual pooling architecture: The dual pooling head jointly attends over [CLS] and the full token sequence, producing richer representations that capture global and local sequence information.The two attended vectors are layer-normalized, concatenated, and passed through a two-layer MLP for classification.
  • Dual pooling architecture: The head splits encoder outputs into [CLS], token representations, and a padding mask, then computes a shared affinity score for token positions.Padding positions are masked before softmax, while the hidden-size scaling stabilizes affinity magnitudes as dimensionality increases.
  • Evaluation: The evaluation used the same 5-run seed protocol and 80/20 stratified split as standard fine-tuning, with task-specific hyperparameters held identical.Table 5 notes that HelaBERT sentiment results use a different three-class dataset from the four-class baseline dataset and are not directly comparable.
  • Results across tasks: +3.9–5.6 macro-F1 points were obtained on sentiment analysis, the largest gains across the evaluated tasks.For HelaBERT-Small, the improvement was driven primarily by NEGATIVE recall rising from 0.51 to 0.74; HelaBERT-Large improved especially on NEUTRAL and POSITIVE F1.
  • Results across tasks: +3.1 macro-F1 points were obtained on news category classification for HelaBERT-Small, compared with +0.1 points for HelaBERT-Large.The authors relate this asymmetry to the weaker [CLS] summary at hidden size 384 and the stronger contextualized [CLS] representation at hidden size 768.
  • Results across tasks: News source classification declined by approximately 0.3 percentage points for both model sizes, while writing style gains were +0.5 points for Small and +0.0 points for Large.News source uses approximately 8-token headlines, and writing style exceeds 95% macro-F1, limiting room for architectural gains.

8 Discussion

HelaBERT-Large generally outperforms multilingual and Sinhala-specific baselines, supporting focused monolingual pre-training for Sinhala. Co-attention is most useful for moderate-length sequences with sparse discriminative tokens and limited encoder capacity, but offers little benefit on short or near-ceiling tasks.

  • HelaBERT versus baselines: HelaBERT-Large surpasses XLM-R-large on news category classification and outperforms SinBERT-Large on all four tasks.The authors attribute these results to HelaBERT’s larger corpus and Sinhala-specific SentencePiece Unigram tokenizer.
  • HelaBERT versus baselines: HelaBERT-Large trails XLM-R-large only on writing style classification, by −0.7 percentage points.The authors suggest XLM-R’s diverse cross-lingual exposure may provide complementary inductive biases for document styles.
  • When co-attention helps: Co-attention is most effective when polarity or category depends on a few discriminative tokens in moderate-length sequences and encoder capacity is limited.Sentiment gains are linked to sparse but decisive negations and intensifiers.
  • When co-attention helps: Co-attention gains are negligible on near-ceiling writing style classification and slightly negative on approximately 8-token news source headlines.The larger sentiment gain for HelaBERT-Large suggests an interaction between model scale and the proposed head.

9 Limitations

The paper’s evaluation and deployment scope is bounded by monolingual modeling, tokenizer integration constraints, untested long-sequence performance for HelaBERT-Small, corpus composition, and classification-only downstream tasks.

  • Scope boundaries: Both HelaBERT models are monolingual and do not support cross-lingual transfer.The downstream evaluation is restricted to Sinhala text classification rather than sequence labelling, question answering, or generative tasks.
  • Data and sequence limits: HelaBERT-Small was pre-trained on 256-token windows despite a 512-token positional limit, so sequences longer than 256 tokens remain untested.The web-crawled corpus may retain noise and is skewed toward formal written Sinhala, potentially underrepresenting dialectal and colloquial registers.
  • Deployment constraint: The SentencePiece tokenizer requires manual loading through the sentencepiece library and is not compatible with HuggingFace AutoTokenizer out of the box.

10 Conclusion

HelaBERT provides two Sinhala masked language models and achieves strong classification performance, with HelaBERT-Large leading monolingual models across all four tasks. The dual pooling head improves sentiment and, for HelaBERT-Small, news category classification, while standard pooling remains competitive on short-input tasks.

  • HelaBERT-Large achieves state-of-the-art results among monolingual Sinhala models across all four evaluated classification tasks.
  • The dual pooling head computes bidirectional attention between the [CLS] token and the full token sequence.
  • +3.9–5.6 macro-F1 points are gained on sentiment analysis with the dual pooling head.
  • +3.1 points improve news category classification for HelaBERT-Small with the dual pooling head.
  • The standard [CLS]-linear head remains competitive on short-input and near-saturated tasks.
  • Both models and the SentencePiece Unigram tokenizer are released publicly, with future work covering sequence labelling, question answering, continued pretraining, and larger architectures.

A Pre-training Hyperparameters

The appendix documents pre-training settings and the four fine-tuning datasets, including split distributions, licensing, access, and collection methodology. It also records dataset-label limitations, including undocumented mappings for numeric category and source labels and a three-class sentiment dataset differing from prior baselines.

  • Pre-training Hyperparameters: Table 9 lists the full pre-training hyperparameters for HelaBERT-Small and HelaBERT-Large.
  • Dataset Documentation: The four fine-tuning datasets are documented with per-class distributions, licensing, access information, and collection methodology.
  • Dataset Label Distributions: News Category Classification uses five numeric labels, whose mapping to the named categories is undocumented in the original dataset card.
  • Dataset Label Distributions: News Source Classification uses nine numeric source labels without documented mappings to the underlying site names.
  • Dataset Label Distributions: Sentiment Analysis uses a three-class dataset that differs from the four-class dataset used by prior baselines.
  • Dataset Label Distributions: Writing Style Classification contains four style labels, with its per-style train/test split reported in Table 13.
Loading 2608.22922v1…