PaperlayerView exact revision on arXiv

Source-linked AI summary

TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

Ali Abusaleh, Bhuvanesh Verma, Alexander Mehler

arXiv:2609.29633v1cs.CLcs.AIcs.LG

TL;DR

Arabic dialect MT error-span detection requires fine-grained character-level predictions, while sentence-level estimates can be too coarse for downstream workflows. TTLab tags raw Arabic MT output with Arabic encoders, focal loss, and dialect-specific thresholds; MARBERTv2 achieves the strongest reported results among six encoders.

  • Problem

    Arabic dialect MT remains substantially harder than MT into MSA, and sentence-level quality estimates can be too coarse for applications such as post-editing.

    Source: §1 · p. 1

  • Method

    The system classifies raw Arabic MT subword tokens, preserves their character offsets, and uses focal loss with class weighting and dialect-specific decoding thresholds.

    Source: Tokenization and Surface Representation · p. 3, Imbalance-Aware Objective · p. 3

  • Results

    40.8 on dev and 40.9 on test make MARBERTv2 the best-performing of six Arabic pre-trained encoders.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The system localizes errors effectively, but classification of the rarest categories remains difficult because of extreme data scarcity.

    Source: §7 · p. 5

    The 1,125-sentence training set includes only two graphetics examples, making that class nearly impossible to learn reliably.

    Source: Limitations · p. 6

Abstract

from arXiv · show

We present TTLab's submission to the AlexandriaX-2026 Subtask~3 on Arabic MT error span detection and classification. Our system frames the task as token-level classification over surface forms, preserving character offsets to ensure exact alignment with the evaluation metric. To handle severe label imbalance, we employ a focal loss with class weighting and dialect-specific decoding thresholds. Among six Arabic pre-trained encoders, MARBERTv2 achieves the best overall performance of 40.8 and 40.91 on the development and test set, respectively, ranking $\nth{3}$ out of all participating teams. While our system localizes error spans effectively, classification of rare error types remains challenging, highlighting the need for data augmentation for tail categories. The code is available at ${\href{https://github.com/ENTAILab/arabic-dialectal-mt-error-span-detection}{\faGithub~ TTLab at AlexandriaX-2026}$

1 Introduction

Arabic dialect MT remains challenging, while sentence-level quality estimates can be too coarse for post-editing. TTLab reframes Arabic MT error-span detection as surface-token classification, with MARBERTv2 leading six Arabic encoders.

1 Introduction

  • The task targets exact erroneous substrings and categories because sentence-level quality estimates can be too coarse for downstream post-editing.Dialectal Arabic presents nonstandard orthography, dialect-specific morphosyntax, and scarce parallel corpora.Source: §1 · p. 1
  • 40.8 on dev and 40.9 on test make MARBERTv2 the strongest of six Arabic pre-trained encoders in the submission.The task requires inaccurate character spans and one of six error categories across five English-to-dialect directions.Source: §1 · p. 1
  • The paper presents a surface-tagging system that preserves character offsets, uses focal loss with down-weighted no-error labels, and applies dialect-specific thresholds.The paper is organized into task and dataset, system, experiments, results, and conclusion sections.Source: §1 · p. 1

2 Related Work

Related work connects span-based MT evaluation to quality estimation and Arabic sequence tagging. TTLab applies token classification with Arabic pre-trained encoders to dialectal MT output.

2 Related Work

3 Task and Data

The task uses the official AlexandriaX-2026 dataset of English-to-dialect Arabic translations annotated with character-level error spans and categories. Its small, imbalanced training set motivates collapsing the rarest categories during training.

3 Task and Data

  • Each official-dataset instance pairs an English source with a machine-generated Arabic dialect translation and gold annotations specifying verbatim spans, character offsets, and categories.The category inventory comprises graphetics, morphosyntax, orthography, pragmatics, semantics, and sociolinguistics.Source: §3 · p. 2
  • The data cover five English-to-dialect directions: ENG_EGY, ENG_MAU, ENG_MOR, ENG_PAL, and ENG_UAE.Source: §3 · p. 2
  • The 1,125 training sentences contain 1,997 spans, with roughly 80% of tokens labeled no-error and only two graphetics spans, prompting collapse of the three rarest categories.Sociolinguistics accounts for 57.9% of error spans; the collapsed other class is mapped back to the most frequent original category at scoring.Source: §3 · p. 2

4 System Overview

The system tags unnormalized Arabic MT output at the subword level and preserves offsets for character-span scoring. Focal loss addresses label imbalance, while dialect-specific thresholds guide decoding.

Tokenization and Surface Representation

  • Raw Arabic MT text is tokenized without normalization, stemming, or diacritic removal so subword tokens retain offsets into the original string.The encoder produces contextual token states that a linear head classifies.Source: Tokenization and Surface Representation · pp. 2–3

Joint Error Detection and Classification

  • Gold spans label every overlapping token; inference merges contiguous same-category non-O tokens and maps their stored offsets back to exact character spans.Figure 1 depicts the raw-text, encoder, token-classification, and span-decoding pipeline.Source: Joint Error Detection and Classification · p. 3, Figure 1 · p. 3

Imbalance-Aware Objective

  • Focal loss with γ = 2 and αO = 0.3 down-weights the dominant no-error class while emphasizing hard, rare error tokens.Padding and special tokens are excluded from loss computation.Source: Imbalance-Aware Objective · p. 3

Decoding and Dialect-Specific Thresholds

  • Dialect-specific thresholds flag tokens using their error probability; same-category tokens are merged, and spans are trimmed or discarded if shorter than two characters.Thresholds are tuned on out-of-fold predictions to maximize each direction’s overall score, and collapsed other labels are mapped to the most frequent original category.Source: Decoding and Dialect-Specific Thresholds · p. 3

5 Experiments

Experiments compare six Arabic encoders in a five-fold out-of-fold setup, with the development set used secondarily. Dialect-specific thresholds and light post-processing improve the overall OOF score.

Setup

  • Results use five-fold cross-validation on the training set; the 138-sentence development set is secondary because single-split scores are noisy.The development set informs backbone selection and diagnostics, and its only system decision is the character-level ensemble vote threshold.Source: §5 · p. 3

Backbone Comparison

  • Six Arabic pretrained encoders are compared within the same surface-tagging pipeline to select the final backbone.The candidates include MARBERTV2, CAMELBERT-DA, SAUDIBERT, ARABERTV2, NILECHAT-3B, and a MARBERTV2 cross-encoder checkpoint fine-tuned on QuranQA1.Source: §5 · p. 4

Per-Dialect Thresholds and Post-Processing

  • Dialect-specific thresholds improve overall OOF score by up to 0.8 points, while span trimming and minimum-length filtering add 0.3–0.5 points.Both components are retained in the final system.Source: §5 · p. 4

Character-Level Voting Ensemble

  • The character-level ensemble combines independently decoded spans, marking a position as erroneous when at least v models agree and assigning categories by weighted majority vote.Sweeping v ranges from the union of taggers, favoring recall, to their intersection, favoring precision; its best development-set operating point is reported in Table 2.Source: §5 · p. 4

6 Results

MARBERTV2 led the reported backbone comparisons, while the revised no-collapse system reached a 42.00 official test Overall Score. Results also show stronger span localization than category classification and persistent weaknesses on rare categories.

6 Results

  • 39.4 OOF and 40.8 Dev for MARBERTV2, the strongest backbone on both splits and the selected final backbone.The Table 2 scores are a pooled/micro proxy and are not directly comparable to the official leaderboard score.Source: §6 · p. 4, Table 2 · p. 4
  • 46–48 overlap-span F1 versus 31–33 error-category micro-F1 at the best operating points, showing stronger localization than categorization.The residual category loss is attributed primarily to undetected spans and extremely rare classes, rather than misclassification of detected spans.Source: §6 · p. 4
  • 35.4 nested OOF versus 39.4 pooled OOF for the submitted MARBERTV2 configuration; the authors attribute the roughly four-point gap to mild pooled-set overfitting.The nested estimate tunes thresholds on four folds and scores the held-out fold.Source: §6 · p. 4
  • Replacing focal loss with weighted cross-entropy lowered nested OOF by 0.8 points and Dev macro-F1 by 1.9 points.Training directly on all six categories matched or exceeded the submitted configuration on every reported metric; Dev macro-F1 was 18.2 versus 16.3.Source: §6 · p. 4, Table 4 · p. 5
  • 42.00 test Overall Score for the revised no-collapse MARBERTV2 system, up from 40.91 for the submitted system.The revised system was retrained and resubmitted to the official leaderboard.Source: Table 3 · p. 5, §6 · p. 5
  • Morphosyntax reached F1 5.3 despite being modeled explicitly; orthography reached 4.9 through a fixed fallback, while pragmatics and graphetics scored 0.The authors attribute weak morphosyntax performance to its 164 spans and greater surface-form variability.Source: §6 · p. 5, Table 5 · p. 5
  • The character-level voting ensemble scored 39.9 on Dev, below the single MARBERTV2 tagger’s 40.8, so the single tagger was submitted.The authors attribute the ensemble result to weaker, correlated encoders diluting MARBERTV2’s predictions.Source: §6 · p. 5

7 Conclusion

The system detects and classifies dialectal Arabic MT error spans through surface tagging of raw text, preserving character offsets for alignment. MARBERTV2 performs best among six encoders, but rare categories remain difficult because of data scarcity.

7 Conclusion

  • MARBERTV2 yields the best results among six pre-trained Arabic encoders, while a character-level ensemble offers no further gain.The conclusion selects MARBERTV2 as the final backbone and notes that the system localizes errors well but struggles with the rarest categories.Source: §7 · p. 5
  • Surface tagging operates on raw, unnormalized text and preserves character offsets through tokenization and modeling for exact alignment with gold spans.This alignment is required by the evaluation metric.Source: §7 · p. 5
  • Focal loss with class weighting addresses severe label imbalance, and dialect-specific confidence thresholds adapt to varying error densities.Source: §7 · p. 5
  • Extreme data scarcity leaves the rarest categories difficult to classify; proposed future directions include tail-category augmentation, cross-lingual transfer, and exact-match-span optimization.Source: §7 · p. 5

Limitations

The work is constrained by sparse training data, reliance on a large encoder, tokenizer-specific offset mapping, and a development metric that differs from the official aggregation.

Limitations

  • The 1,125-sentence training set includes only two graphetics examples, making reliable learning of that class nearly impossible despite the collapse strategy.Source: Limitations · p. 6
  • The system relies on a single relatively large MARBERTV2 encoder, which may not scale gracefully in resource-constrained environments.Source: Limitations · p. 6
  • Avoiding normalization preserves offsets but ties the system to its encoder's tokenizer; changing tokenizers would require remapping offset annotations.Source: Limitations · p. 6
  • Development scores are pooled and micro-averaged proxies, not directly comparable to the official per-direction macro-averaged ranking metric, which the system did not optimize directly.Source: Limitations · p. 6
Loading 2609.29633v1…