PaperlayerView exact revision on arXiv

Source-linked AI summary

Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability

Swapnil Kundu Argha, Abdullah Al Shafi, Rowzatul Zannat, Shoumik Barman Polok, Abdul Muntakim, Jannatul Ferdousi, M. A. Moyeen

arXiv:2609.13869v1cs.CLcs.AIcs.LG

TL;DR

Bangla sentence-function classification has limited benchmark resources, despite the relevance of function identification to downstream NLP. This paper introduces an annotated four-class corpus and benchmarks feature representations and classifiers, including heterogeneous ensembles. DLE achieves the best overall performance, and cross-validation supports the stability of the ensemble approaches.

  • Problem

    Benchmark resources for Bangla sentence-function classification remain limited, although function recognition is relevant to applications such as dialogue systems, translation, and speech synthesis.

    Source: §I · p. 1

  • Method

    The paper develops a manually annotated Bangla corpus and benchmarks feature representations and classical classifiers, including heterogeneous ensembles.

    Source: §I · pp. 1–2

  • Results

    DLE achieved 0.95 on TF-IDF features, while SLE and DLE reached 92–95% accuracy across cross-validation methods.

    Source: D. Result Analysis · pp. 5–6

  • Takeaways & Limitations

    The corpus and benchmark tests offer resources for future Bangla NLP studies, and the conclusion identifies transformer models and larger datasets as future directions.

    Source: §V · p. 6

    Misclassifications include exclamatory sentences labeled declarative, directive verbs complicating declarative–imperative distinctions, and questions with embedded commands treated as orders.

    Source: D. Result Analysis · p. 6

Abstract

from arXiv · show

Automatic sentence function identification is important for many downstream natural language processing (NLP) applications such as dialogue systems, text-to-speech synthesis, and machine translation. However, benchmark resources for Bangla sentence function classification remain limited. To mitigate this gap, this paper introduces a corpus of 10,000 Bangla sentences, manually annotated into four functional categories, namely declarative, interrogative, imperative, and exclamatory. The corpus is nearly balanced across the four classes, with high annotation reliability reflected by a Fleiss\' Kappa of 0.82. Furthermore, we evaluate multiple feature representations, including Bag-of-Words (BoW), TF-IDF, and Word2Vec, with several classical machine learning classifiers. In addition, two heterogeneous ensemble models, namely Single-Level Ensemble (SLE) and Double-Level Ensemble (DLE), are utilized to improve classification performance. Experimental results show that TF-IDF consistently outperforms Word2Vec, likely due to its ability to emphasize discriminative lexical cues associated with sentence functions, particularly given the relatively small corpus used to train Word2Vec. The DLE model with TF-IDF features achieves the best performance with accuracy and macro-F1 of 0.95, demonstrating the effectiveness of sparse lexical representations and heterogeneous ensemble learning for this task. Further cross-validation confirms the robustness of the approach, while LIME-based interpretability provides insights into model predictions. The developed corpus and model benchmarking establish strong baselines for Bangla sentence function classification.

I. Introduction

Bangla has relatively limited automated language-processing resources, while sentence-function recognition supports applications that need to preserve or respond to communicative intent. This work introduces a four-class annotated corpus, benchmarks classification methods, and adds interpretability and error analysis.

I. Introduction

  • Bangla remains relatively deficient in automated language-processing resources despite its large speaker population and expanding digital content.The paper notes about 250 million native speakers and the rapid growth of Bangla content across platforms.Source: §I · p. 1
  • Reliable sentence-function classification matters to dialogue systems, machine translation, sentiment and stance detection, and text-to-speech synthesis.These applications use functional cues to respond to queries, preserve intent, or produce suitable intonation.Source: §I · p. 1
  • The paper presents 10,000 manually annotated Bangla sentences across four function classes, using multi-annotator labeling with agreement validation.The corpus is also released as a publicly available resource to support reproducible research.Source: §I · pp. 1–2
  • The study benchmarks feature representations and classical classifiers, including two heterogeneous ensemble architectures.It also incorporates interpretability and error analysis to examine model behavior and linguistic challenges.Source: §I · p. 2

II. Literature Review

Prior Bangla classification studies address related tasks using traditional and neural methods, while this paper develops a sentence-function classification workflow and corpus. The corpus is sourced, preprocessed, annotated into four classes, and quality-checked before serving as a benchmark.

II. Literature Review

  • Earlier Bangla text-classification work includes traditional classifiers and neural models, but the cited studies address sentence or structural classification, news, and sentence structure or tense.Reported results include 96.39% accuracy for ensemble models, 93.72% for Decision Trees, 0.96 F1-Score for Word2Vec on news, and 98% F1-Score for a CNN-BiLSTM model.Source: §II · p. 2

III. Proposed Method for Sentence Function Classification

  • The proposed workflow proceeds through corpus creation, preprocessing, feature extraction, model training, evaluation, and interpretation.Corpus creation itself includes data collection, preprocessing, annotation, and quality verification.Source: §III · p. 2, A. Development of Bangla Sentence… · p. 2

A. Development of Bangla Sentence Function Corpus

  • The corpus draws formal and informal Bangla from newspapers, blogs, magazines, YouTube comments, and public-domain books, then cleans and filters the collected text.Preprocessing includes removing HTML tags and non-Bengali characters, sentence segmentation, deduplication, and noise filtering.Source: A. Development of Bangla Sentence… · p. 2
  • Annotators assign declarative, interrogative, imperative, or exclamatory labels, with three annotators evaluating each sentence and majority voting determining its final label.When there is no distinct majority, annotators discuss the case until reaching consensus.Source: A. Development of Bangla Sentence… · pp. 2–3
  • 10,000 sentences were selected from an original collection of approximately 24,000 to form a benchmark for sentence-function classification.Annotation reliability is evaluated using Fleiss’ Kappa.Source: A. Development of Bangla Sentence… · p. 3

B. Data Preprocessing

The preprocessing pipeline cleans Bangla text while retaining lexical cues needed for function classification. It removes punctuation as a direct cue and applies a shared tokenizer across feature extractors.

B. Data Preprocessing

  • Preprocessing removes hashtags and special characters without linguistic meaning, normalizes spaces, and removes common Bangla stop words except wh-question words.Words such as কী, েকমন, কখন, and েকাথায় are retained because they help identify interrogative and/or exclamatory sentences.Source: B. Data Preprocessing · p. 3
  • All punctuation is removed so that models rely on lexical and structural cues rather than punctuation as a direct function signal.Source: B. Data Preprocessing · p. 3
  • A Bangla-aware whitespace tokenizer returns non-empty tokens after cleaning, and the same tokenizer is used across feature extractors for fair comparison.The four function labels are encoded as integer IDs from 0 to 3 for training.Source: B. Data Preprocessing · p. 3

C. Feature Extraction

The study represents preprocessed Bangla text using BoW, TF-IDF, and Word2Vec. These methods capture term counts, corpus-weighted term importance, or contextual semantic relationships.

C. Feature Extraction

  • BoW, TF-IDF, and Word2Vec are used to convert preprocessed text into numerical vectors.Source: C. Feature Extraction · p. 3
  • BoW counts word or n-gram frequency while ignoring word order and grammar, whereas TF-IDF weights terms by document frequency and corpus-wide rarity.A frequent interrogative term such as কী can receive high TF-IDF weight in an interrogative document.Source: C. Feature Extraction · p. 3
  • Word2Vec is a corpus-trained two-layer neural network with CBOW and Skip-gram architectures for learning semantic relationships.CBOW predicts a target word from surrounding context and performs well for frequent words; Skip-gram predicts context words from a target and tends to capture rare words and semantic relationships better.Source: C. Feature Extraction · pp. 3–4, Figure 3 · p. 3

D. Classification

The classification study evaluates classical machine-learning classifiers and two heterogeneous ensembles, using multiple cross-validation strategies and grid-search tuning. SLE combines classifier outputs through voting, while DLE adds a second aggregation level.

D. Classification

  • The benchmark includes eight classical classifiers, from Decision Tree and Naive Bayes to Logistic Regression, boosting, and Random Forest.The listed models are DT, MNB, LR, LSVC, Ridge, GB, AdaBoost, and RF.Source: D. Classification · p. 4
  • DLE first aggregates classifiers in two groups, then combines those groups’ voting outputs in a third classifier.The groups are MNB, LR, DT, and LSVC; and RF, GB, and AdaBoost.Source: D. Classification · p. 4
  • SLE feeds the outputs of all classical classifiers into a voting classifier for the final prediction.Both SLE and DLE use soft voting, averaging predicted probabilities and selecting the highest-scoring class.Source: D. Classification · p. 4
  • Grid search with 5-fold cross-validation optimizes each model for macro-F1 before refitting it on the full training set.Evaluation also uses K-fold, stratified K-fold, shuffle split, and stratified shuffle split strategies.Source: D. Classification · p. 4

E. Interpretability using LIME

LIME is used to analyze model predictions by identifying influential words or phrases through local approximations.

E. Interpretability using LIME

  • LIME locally approximates model predictions to identify which words or phrases most affect a prediction.The paper applies it to analyze prediction results.Source: E. Interpretability using LIME · p. 4

IV. Experimental setup and Result analysis

The experimental section includes tables for corpus statistics, feature-and-classifier comparisons, and cross-validation performance. The captions specify the scope and reported metrics of these tables.

IV. Experimental setup and Result analysis

  • Table I covers class distribution and sentence-length statistics for the Bangla Sentence Function Corpus.The caption identifies the table’s subject but provides no individual counts or length values.Source: Table I · p. 4
  • Table II compares feature representations and classifiers, with all metrics except accuracy macro-averaged.The caption does not provide specific performance values or identify a winner.Source: Table II · p. 5
  • Table III reports mean ± std cross-validation performance for SLE and DLE with K = 5.The caption identifies the models and evaluation format, but does not provide their performance values.Source: Table III · p. 5

A. Inter-Annotator Agreement

The corpus reports high inter-annotator agreement and an approximately balanced distribution across its four functional classes. Its sentence-length statistics span three length ranges for every class.

A. Inter-Annotator Agreement

  • Fleiss’ Kappa is 0.82, classified as ‘Almost Perfect’ and indicating strong consistency in annotation.The passage reports this as the corpus’s inter-annotator agreement result.Source: A. Inter-Annotator Agreement · p. 5

B. Corpus Statistics

  • Each functional class contributes approximately 2,500 sentences, and all four include sentences across three length ranges.The ranges are 0–8, 9–14, and 15+ tokens.Source: B. Corpus Statistics · p. 5

C. Experimental Setup and Evaluation Metrics

The evaluation benchmarks feature and classifier combinations, with DLE on TF-IDF achieving the strongest reported result. Cross-validation, error analysis, and LIME examine stability and model behavior.

C. Experimental Setup and Evaluation Metrics

  • The final curated dataset uses stratified sampling to split 8,000 sentences for training-validation and 2,000 for an unseen test set; evaluation uses accuracy, precision, recall, and F1-Score.Source: C. Experimental Setup and Evaluation… · p. 5

D. Result Analysis

  • DLE reaches 0.95 on TF-IDF, while SLE maintains 0.94 on most metrics; RF reaches 0.94 accuracy and F1-Score with TF-IDF.The comparison covers four feature representations and eight classifiers.Source: D. Result Analysis · p. 5
  • TF-IDF emphasizes class-specific interrogative, directive, and exclamatory terms more explicitly than Word2Vec, which may not preserve these cues as clearly on a small training corpus.The authors conclude that sparse lexical weighting suits this task.Source: D. Result Analysis · pp. 5–6
  • Both SLE and DLE achieve 92–95% accuracy across feature representations, with standard deviations of 0.6–2.4% across cross-validation methods.The paper presents these results as confirming ensemble stability.Source: D. Result Analysis · p. 6
  • Misclassifications include exclamatory sentences labeled declarative, directive verbs blurring declarative and imperative forms, and questions with embedded commands treated as orders.Table IV presents representative genuine misclassification cases for DLE with TF-IDF features.Source: D. Result Analysis · p. 6, Table IV · p. 6
  • In a LIME example, the token ‘খাও’ receives high importance for a predicted Imperative class; similar patterns were observed for other classes.The authors interpret the token’s importance as aligned with action and directive semantics.Source: D. Result Analysis · p. 6, Figure 5 · p. 6

V. Conclusion

The paper concludes that TF-IDF remains effective for Bangla sentence function classification, while heterogeneous ensembles offer stability and DLE the highest overall performance. Its corpus and benchmarks are presented as resources for future Bangla NLP research.

V. Conclusion

  • The study combines a Bangla sentence function corpus with comparisons of traditional learning methods and heterogeneous ensemble techniques.Source: §V · p. 6
  • DLE produces the highest overall performance, while heterogeneous ensembles show greater performance stability than the other tested models.The conclusion also describes TF-IDF as effective for the task.Source: §V · p. 6
  • The corpus and benchmark tests offer resources for future Bangla NLP studies, with transformer-based models and larger datasets identified as future directions.Source: §V · p. 6
  • The error analysis identifies linguistic uncertainty between sentence types as a source of classification difficulty.Source: §V · p. 6
Loading 2609.13869v1…