PaperlayerView exact revision on arXiv

Source-linked AI summary

Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution

Zenith Biswas, Andrew Thomas Dyer

arXiv:2610.35114v1cs.CL

TL;DR

Bengali coreference resolution lacks annotated data, and the effectiveness of multilingual strategies in this low-resource setting has not been established. The paper evaluates pretrained encoders across data, augmentation, and transfer settings, finding that a small gold dataset outperforms larger mixed-quality data while augmentation and cross-lingual transfer offer no reliable gains.

  • Problem

    Bengali has only one gold-standard coreference dataset, and multilingual strategies for low-resource coreference have not been empirically verified.

    Source: §1 · p. 1

  • Method

    The study evaluates pretrained encoders on Bengali using gold and mixed data, augmentation, and Hindi and English transfer settings.

    Source: §1 · p. 2

  • Results

    41 gold-standard documents outperform 141 mixed-quality documents, while augmentation fails to improve reliably and cross-lingual transfer provides no reliable gains.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    Data quality, task-specific validation, and embedding diagnostics matter more than scaling data or model complexity in these low-resource experiments.

    Source: §6 · p. 9

    The study evaluates only Bengali, so whether its findings apply to other low-resource languages remains empirically unverified.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multilingual transformer models are often assumed to generalise across languages, but their effectiveness for coreference resolution in low-resource settings remains unverified. We investigate this gap in Bengali, a language with over 250 million speakers but limited annotated coreference resources. We evaluate transformer-based pre-trained encoders under multiple settings, including zero-shot, fine-tuning, data augmentation, and cross-lingual transfer. Our results challenge several common assumptions about model generalisation and data efficiency in low-resource NLP. We demonstrate that a small amount of high-quality annotated data significantly outperforms larger mixed-quality datasets (p < 0.01). We find that data augmentation, particularly back-translation, does not improve performance and can degrade results in coreference resolution. Cross-lingual transfer is also inconsistent: transfer from a distant language like English reduces performance, and transfer from the related language Hindi provides no consistent benefit across models. Several widely used multilingual models exhibit collapsed embedding spaces, rendering them unsuitable for similarity-based clustering tasks. These findings highlight the importance of annotation quality over data quantity and provide empirical guidance for developing coreference resolution systems in low-resource languages.

1 Introduction

Bengali illustrates the scarcity of annotated coreference data, while the effectiveness of multilingual strategies for low-resource coreference remains unverified. The paper evaluates pretrained encoders across training, augmentation, and transfer settings, finding that data quality matters and common strategies offer no reliable gains.

1 Introduction

  • Bengali has more than 250 million speakers but only one gold-standard coreference dataset, BenCoref, with 122 documents across four domains.The language also presents gender-neutral pronouns and complex honorific systems.Source: §1 · p. 1
  • The study evaluates pretrained encoders in zero-shot, fine-tuning, augmentation, and Hindi and English transfer settings.Its architecture uses a frozen encoder and trainable MLP classifier, chosen for stability when model capacity greatly exceeds available training data.Source: §1 · p. 1
  • Several widely used multilingual models have collapsed Bengali embedding spaces, making them unsuitable for similarity-based clustering despite strong standard-benchmark performance.Source: §1 · p. 2
  • A small gold-standard dataset significantly outperforms a larger mixed-quality one (p < 0.01), while transfer provides no statistically significant aggregate benefit.The paper also reports that augmentation, particularly back-translation, can degrade coreference performance.Source: §1 · p. 2

2 Related Work

Prior coreference research and evaluation have focused largely on English, while Bengali lacks systematic assessment of modern pretrained models and low-resource strategies. Existing multilingual and augmentation approaches therefore leave open whether they support Bengali coreference, which depends on precise spans and semantic distinctions.

2 Related Work

  • Coreference methods have advanced from mention-pair classifiers to end-to-end systems and SpanBERT, but were evaluated mostly on large English datasets.Source: §2 · p. 2
  • Bengali has a 122-document gold corpus, but no systematic evaluation of modern pretrained models or common low-resource strategies.Related resources include multilingual CorefUD, translation-based projection, and the silver TransMuCoRes dataset.Source: §2 · p. 2
  • Shared multilingual representations motivate transfer, but language variation and limited zero-shot transfer leave Bengali embedding discriminability for coreference clustering unresolved.Language-specific models target Bengali, but their embeddings for tasks beyond classification had not been evaluated.Source: §2 · p. 2
  • Augmentation methods proven useful in translation or classification may disrupt coreference annotations when text changes spans or semantic relationships.The paper empirically tests these assumptions for Bengali coreference.Source: §2 · p. 3

3 Experimental Setup

The experiments compare pretrained encoders and training-data strategies on a fixed BenCoref evaluation set. The setup specifies the datasets, model selection, pairwise architecture, augmentation procedures, and evaluation measures.

3 Experimental Setup

  • Five encoders are evaluated across nine training configurations on the BenCoref test set, covering zero-shot, fine-tuning, augmentation, and transfer.Source: §3 · p. 3

3.1 Datasets

  • BenCoref supplies 41 training, 10 development, and 71 test documents; silver Bengali and Hindi data and sampled English OntoNotes documents support training and transfer experiments.BenCoref is gold-standard and spans four domains; all evaluation uses gold-standard BenCoref data.Source: §3.1 · p. 3

3.2 Models

  • Five models were selected from multilingual, Indic-focused, Bengali-specific, and control categories; three were excluded for collapsed embeddings and two as redundant.The excluded collapsed models had discriminability gaps below 0.01.Source: §3.2 · pp. 3–4

3.3 Architecture

  • Gold mention spans are represented from frozen encoder tokens, and a trainable MLP scores mention pairs before thresholded connected-components clustering.The pair representation combines both mention vectors, their element-wise product, and absolute difference; thresholds are tuned on development data.Source: §3.3 · p. 4

3.4 Training Configurations

  • Training configurations isolate data quality, augmentation, and transfer effects while holding BenCoref development and test sets constant.Gold Only is compared with Mixed Baseline, augmentation variants, and Hindi and English transfer.Source: §3.4 · p. 4, Table 2 · p. 4

3.5 Data Augmentation

  • Back-translation and MLM paraphrasing augment the training data, but positional label reattachment can misalign labels when back-translation changes word order or length.MLM paraphrasing makes targeted lexical substitutions while preserving sentence structure; both methods are applied to gold-only and mixed data.Source: §3.5 · pp. 4–5

3.6 Evaluation

  • CoNLL F1 averages MUC, B3, and CEAFe, while paired bootstrap testing and Holm–Bonferroni correction assess significance.The paired bootstrap uses 10,000 resamples over per-document CoNLL F1 scores.Source: §3.6 · p. 5

4 Results

Across Bengali coreference experiments, gold-only training outperformed mixed-quality data, augmentation did not help, and cross-lingual transfer varied by source language and model. The results also reveal systematic over-linking in the strongest model.

4.1 Model Performance Overview

  • MUC F1 was typically 95–97 while CEAFe was often 30–37, a disparity indicating over-linking that averaged CoNLL F1 obscures.MuRIL-Large produced 55,107 spurious links versus 1,616 missed; gender-neutral honorifics such as tini can make distinct respected referents similar, leading to long-range false links.Source: §4.1 · pp. 5–6, Table 3 · p. 6

4.2 Gold vs. Mixed Data Quality

  • 64.36% average CoNLL F1 for Gold Only exceeded 63.59% for Mixed Baseline; all five models favored gold-only training (p = 0.0070).The datasets also differ in source language and genre, which the study does not separate from annotation quality.Source: §4.2 · p. 6

4.3 Data Augmentation Does Not Improve Performance

  • 63.54% average CoNLL F1 with back-translation and 63.89% with MLM paraphrasing were both below the 64.36% gold-data baseline.Only back-translation's aggregate decline was significant; on mixed data, neither method changed scores significantly, though MuRIL-Large lost over 1.5 points with both. Sentence restructuring may disrupt spans and referential links more than local substitutions.Source: §4.3 · p. 6

4.4 Cross-Lingual Transfer

  • English transfer trailed Mixed Baseline by 1.92pp; Hindi averaged 63.12% versus 63.09% zero-shot, but neither transfer effect was significant after correction.MuRIL-Large gained 1.52pp with Hindi, while other models varied; high per-document variance limited power to detect effects of this size.Source: §4.4 · pp. 6–7

4.1 Model Performance Overview

  • 66.73% CoNLL F1 was the best overall result, achieved by MuRIL-Large trained on 41 gold-standard BenCoref documents.The comparison covers five models and nine training configurations on the same BenCoref test set.Source: §4.1 · p. 5, Table 4 · p. 7

5 Analysis and Discussion

The analysis finds that several multilingual encoders lack separable mention embeddings, while sentence-level augmentation can damage span labels. It recommends checking embedding health and prioritizing carefully annotated data, while treating transfer and augmentation results as task-specific.

5.1 Embedding Collapse

  • Three models have discriminability gaps below 0.01 and mean similarities above 0.94 for both coreferent and non-coreferent pairs, making threshold-based clustering unsuitable.Figure 1 encodes within-cluster similarity in green, between-cluster similarity in red, and the gap as discriminability; neither threshold extremes recover the gold average of 3.96 clusters per document.Source: Figure 1 · p. 7, §5.1 · p. 7
  • XLM-RoBERTa-Base, XLM-RoBERTa-Large, and MuRIL-Base collapse despite strong standard-benchmark performance, while MuRIL-Large retains more separable representations.The MuRIL-Base/Large divergence suggests model scale may play a role in preserving discriminability.Source: §5.1 · p. 7

5.2 Why Augmentation Fails

  • Back-translation can significantly degrade coreference performance because sentence reordering or length changes shift position-restored labels away from their mention spans.MLM paraphrasing preserves sentence length and structure, but substitutions inside mentions can still make labels inappropriate for the changed token.Source: §5.2 · p. 8
  • Coreference augmentation must keep labels aligned to exact spans, unlike classification transformations that assume labels survive surface-level text changes.The paper recommends span-aligned methods rather than changing surface text and reattaching labels by position.Source: §5.2 · p. 8

5.3 Implications for Low-Resource NLP

  • The authors recommend prioritizing carefully checked gold data, testing embedding discriminability before similarity-based clustering, and validating transfer and augmentation on the target task.These recommendations are bounded to their single-language, single-task evidence; the study reports English transfer reduced performance and does not claim results generalize to other span-sensitive tasks.Source: §5.3 · p. 8

6 Conclusion

The conclusion reports a Bengali study that challenges assumptions about data scale, augmentation, transfer, and benchmark scores as indicators of useful coreference representations. Its results favor gold annotation and task-specific validation.

6 Conclusion

  • The study challenges assumptions that more data is always better, standard augmentation and transfer reliably help, and strong benchmark scores imply usable coreference representations.Its conclusions concern Bengali coreference under low-resource constraints.Source: §6 · p. 8
  • 41 gold-standard documents outperform 141 mixed-quality documents, while sentence-level augmentation fails to improve performance and cross-lingual transfer provides no reliable gains.The conclusion presents these as evidence against assuming that more data or standard transfer and augmentation strategies will help.Source: §6 · pp. 8–9
  • Widely used multilingual models exhibit embedding collapse, so standard benchmark performance does not guarantee representations that discriminate coreferent from non-coreferent pairs.The conclusion identifies this as a reason to use embedding diagnostics for coreference.Source: §6 · p. 9
  • The authors identify data quality, task-specific validation, and embedding diagnostics as more important than scaling data or model complexity in low-resource settings.They suggest extending diagnostics to additional languages and exploring augmentation methods that preserve span alignment.Source: §6 · p. 9

Limitations

The study’s findings are bounded by its Bengali-only setting, dataset differences, experimental setup, test-set size, and limited augmentation methods.

Limitations

  • The study evaluates only Bengali, so whether its findings apply to other low-resource languages, including Indo-Aryan languages, remains empirically unverified.The authors explicitly limit these findings to the evaluated language rather than treating broader applicability as established.Source: Limitations · p. 9
  • BenCoref and TransMuCoRes differ in genre and source language, so their comparison does not isolate annotation quality as the sole cause of performance differences.The comparison represents the practical choice between available gold and silver data for Bengali researchers.Source: Limitations · p. 9
  • Experiments assume gold-standard mention boundaries and use a frozen encoder with a pairwise classifier, so end-to-end systems or alternative architectures may show different patterns.The authors also note that encoder fine-tuning may change sensitivities to data quality, augmentation, and transfer.Source: Limitations · p. 9
  • The 71-document BenCoref test set limits statistical power and may explain why several transfer effects did not reach significance.Source: Limitations · p. 9
  • Only two sentence-level augmentation techniques were tested, so embedding-level or span-aligned methods may produce different results.The authors limit their conclusion to standard sentence-level augmentation being ineffective, not augmentation being inherently unsuitable for span-sensitive tasks.Source: Limitations · p. 9

Ethics Statement

The paper uses public datasets and publicly released models under their respective research licenses and does not release new data.

Ethics Statement

  • The study uses BenCoref, TransMuCoRes, and OntoNotes 5.0 under their respective licenses and intended research use.Source: Ethics Statement · p. 9
  • The authors report that their contribution is methodological and that they release no new data.Source: Ethics Statement · p. 9
  • LLM-based tools were used for editorial polishing of the manuscript draft, not for the research itself.Source: Ethics Statement · p. 9

A Statistical Testing Details

Statistical comparisons use paired document-level CoNLL F1 scores, with Holm–Bonferroni correction for multiple comparisons.

A Statistical Testing Details

Related papers

Loading 2610.35114v1…