PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Based Quality Assessment for Massively Multilingual Parallel Data

Abdelaziz M. A. Ibrahim, Zihao Li, Jörg Tiedemann, Shaoxiong Ji

arXiv:2606.00285v1cs.CL

TL;DR

Massively multilingual bitext must be assessed for both non-parallel pairs and low-quality translations, but reliable model-based evaluation across language directions remains uncertain. The paper separates parallelism assessment with multilingual embeddings from reference-free QE and benchmarks both components across multilingual directions. Results show that no model is universally reliable, naive QE ensembles dilute strong signals, and documented target-language coverage is associated with higher QE scores, supporting direction-aware routing and calibration.

  • Problem

    Reliable model-based assessment across many translation directions is needed because multilingual bitext contains non-parallel pairs and translation defects, while labeled quality data are limited for low-resource directions.

  • Method

    The paper decomposes assessment into embedding-based source–target parallelism and reference-free translation quality estimation, benchmarking models by translation direction.

  • Results

    No single embedding or QE model is globally optimal; naive QE ensembles dilute strong signals, while documented target-language coverage is strongly associated with higher QE scores.

  • Takeaways & Limitations

    Multilingual parallel-data assessment should use direction-level model routing and score calibration rather than a single universal metric.

  • Takeaways & Limitations

    The QE benchmark uses professional FLORES-200 translations as a positive-only surrogate and does not establish reliable rejection of noisy or mistranslated corpus pairs.

Abstract

from arXiv · show

Large-scale multilingual bitext often contains two distinct problems: non-parallel sentence pairs and low-quality translations. We decompose model-based assessment for such data into two independent components: parallelism assessment with multilingual embeddings and reference-free quality estimation (QE). For parallelism, we benchmark four embedding models on FLORES-200 and BOUQuET retrieval tasks, covering 6,654 source--target directions in our target language-pair inventory. For QE, we evaluate nine reference-free evaluators on professional FLORES-200 translations across 41,412 ordered source--target directions. Results show that no model is universally reliable across translation directions. Naive QE ensembles dilute strong model signals, while documented target-language coverage is strongly associated with higher QE scores. Overall, these findings suggest that multilingual parallel-data assessment is best approached as a direction-aware routing and calibration problem, where no single universal metric is expected to suffice across all languages.

1 Introduction

Massively multilingual bitext combines non-parallel pairs with translation-quality defects, motivating a two-component assessment framework and direction-aware evaluation. The study benchmarks parallelism and reference-free QE because model reliability and calibration vary across language directions.

  • Clean parallel data are difficult to obtain for many low-resource languages, while web-mined and automatically generated bitext often contains non-translations and translation defects.These defects include omissions, additions, mistranslations, and severe fluency problems, making manual inspection infeasible at scale.
  • The framework separately assesses source–target parallelism and translation quality using multilingual embeddings and reference-free quality estimation.Parallelism tests whether the pair expresses the same content; QE evaluates fluency and meaning preservation without requiring a gold reference at inference time.
  • Parallelism and translation quality are distinct: fluent sentences may be non-parallel, whereas aligned pairs may still contain omissions, mistranslations, or fluency errors.
  • The study benchmarks embedding models for parallelism and reference-free models on professional FLORES-200 translations across massively multilingual directions.The QE evaluation covers 41,412 ordered translation directions and tests whether model-based signals behave consistently enough to inform corpus assessment.
  • The research asks how performance varies by direction, whether unsupervised QE ensembles improve consistency, and how documented language coverage relates to evaluator behavior.

2 Problem Setup: Parallelism and Translation Quality

The paper defines parallelism and translation quality as related but non-identical properties, then uses separate embedding and reference-free QE components to assess them. The central objective is reliable model-based evaluation across translation directions, especially where labeled quality data are limited.

  • Parallelism asks whether source and candidate target express the same content and can reasonably be treated as translations of each other.
  • Translation quality asks whether a likely parallel pair preserves source meaning fluently and appropriately in the target language.The definition includes severe omissions, additions, mistranslations, and local errors.
  • Component 1: Parallelism Assessment: Embedding-based parallelism assessment encodes source and target sentences and uses semantic similarity to estimate alignment.Higher cosine similarity indicates stronger semantic alignment.
  • Component 1: Parallelism Assessment: The parallelism component selects an embedding model and direction-specific similarity threshold for each translation direction.Thresholds vary because embedding-similarity distributions may differ substantially across language pairs.
  • Component 1: Parallelism Assessment: Parallelism is an alignment gate rather than a complete quality metric, because it does not determine whether the target is fluent, natural, or error-free.
  • Component 2: Reference-Free Quality Estimation: Reference-free QE assigns a score directly to a source sentence and candidate translation without requiring a gold reference at inference time.Higher normalized scores should indicate better translations.

3 Direction-Aware Calibration

Both assessment components use direction-level benchmarking, model selection, and direction-specific thresholds, but they rely on different evaluation signals. This shared protocol motivates routing models by translation direction rather than applying one scorer globally.

  • Both components benchmark candidate models by translation direction, select a scorer for each direction, and convert scores into direction-specific thresholds.
  • Parallelism uses retrieval performance, whereas QE uses mean quality scores on professional translations.Thus, the two components share an operational structure but not a scoring signal.
  • Parallelism Assessment: Mean reciprocal rank measures how reliably an embedding model places the correct translation above non-matching target sentences for a direction.
  • Reference-Free Quality Estimation: QE evaluators are compared using direction-level mean scores on professional FLORES-200 translations, treating high and stable scores as a necessary reliability signal.The overall model score is the macro-average over observed directions.
  • Direction-Aware Routing: Direction-aware routing selects the most reliable scorer for each language pair based on empirical benchmark evidence.For QE, the proposed diagnostic includes performance on professional translations and documented language coverage.
  • Direction-Aware Routing: The routing concept reflects that model-based signals are not uniformly calibrated across languages.

4 Component 1: Parallelism Assessment

Component 1 uses multilingual embedding models as candidate semantic aligners and evaluates them on multilingual bitext retrieval datasets. FLORES-200 and BOUQuET provide complementary coverage for assessing retrieval across many language directions.

  • Embedding Model Suite: Four multilingual embedding models are evaluated as candidate semantic aligners.The model suite is summarized in Table 1.
  • Embedding Model Suite: Table 1 lists Harrier, mE5-large, GTE, and Jina-v3 as the embedding models used for Component 1.
  • Retrieval Benchmarks: FLORES-200 is a professionally translated, sentence-level many-to-many benchmark covering 204 language varieties and more than 40K ordered translation directions.
  • Retrieval Benchmarks: BOUQuET complements FLORES-style evaluation with broader domain and register coverage.At the time of the experiments, BOUQuET included 275 completed multi-way parallel languages.

5 Component 2: Reference-Free Quality Estimation

The study compares reference-free QE systems and ensembles on professional FLORES-200 translations, using common score normalization and coverage-aware aggregation to assess multilingual consistency.

  • 5.1 QE Model Suite: Nine reference-free QE systems are evaluated for Component 2, including COMETKiwi, xCOMET, MetricX, ReMedy, M-Prometheus, Qwen3 Family, and Bicleaner.
  • 5.2 FLORES-200 as a Surrogate QE Benchmark: 83,196,648 source–translation instances are produced from FLORES-200 dev and devtest splits expanded across ordered directions.
  • 5.2 FLORES-200 as a Surrogate QE Benchmark: The benchmark compares how strongly evaluators recognize professional FLORES translations as high quality across the multilingual inventory.Because FLORES-200 lacks QE labels, this does not measure correlation with human judgments.
  • 5.3 Score Normalization: All model outputs are mapped to a common [0, 1] range, with higher values indicating better translation quality.MetricX is normalized from its original 0–25 lower-is-better scale, while Bicleaner, COMETKiwi, and xCOMET already use higher-is-better [0, 1] scores.
  • 5.4 QE Ensembles: Unsupervised mean, median, and weighted-average ensembles test cross-direction consistency and fallback behavior rather than supervised metaevaluation.Coverage-aware variants restrict constituent models using documented source- or target-language support.

6 Results

Across parallelism and QE, model performance varies substantially by translation direction, favoring direction-aware selection over uniform deployment. QE ensembles and coverage analyses further show that naive aggregation can dilute strong signals, while documented target-language coverage aligns with higher scores.

  • Parallelism Assessment: 6,654 source–target directions are evaluated for parallelism, enabling per-direction embedding-model routing.The benchmark combines FLORES-200 and BOUQuETSentence retrieval evidence where both are available.
  • Parallelism Assessment: Harrier has the highest average MRR and largest routing footprint, while mE5-large is routed to 2,013 directions and Jina-v3 to 1,540.GTE is selected for only 54 directions, and Jina-v3 remains strongest for a substantial subset despite lower overall average MRR.
  • Parallelism Assessment: No single embedding model dominates all covered directions, supporting direction-aware routing rather than applying one model uniformly.The observed variation identifies the strongest embedding signal separately for each language pair.
  • Reference-Free Quality Estimation: 41,412 ordered directions show different QE winners: ReMedy wins 16,367 directions (39.52%), MetricX has the highest macro-average at 0.6228, and Qwen3-4B has the best rank profile.Qwen3-4B records the lowest rank mean (2.39) and rank standard deviation (1.25).
  • Reference-Free Quality Estimation: 20,082 directions (48.49%) have winning margins below 0.05, limiting confidence that any single evaluator is clearly superior.Only 10,558 directions (25.50%) have a winning margin of at least 0.10.
  • Unsupervised Ensembles: Unrestricted mean, median, and weighted ensembles score 0.4630, 0.4842, and 0.5026, below MetricX (0.6228) and Qwen3-4B (0.6160), because weaker models dilute stronger signals.Coverage-aware ensembles score higher on favorable subsets but remain below Qwen3-4B’s 0.8498 on the same subset.
  • Coverage and Target-Side Asymmetry: For every evaluator, both-seen coverage yields the highest mean QE scores, while target-only coverage exceeds source-only coverage across models.Qwen3-4B rises from 0.411 under source-only coverage to 0.650 under target-only coverage; ReMedy rises from 0.517 to 0.723.
  • Coverage and Target-Side Asymmetry: 7,562 directions (18.3%) have best-available mean scores below 0.5, indicating that automatic QE filtering should be applied conservatively there.Another 3,520 directions (8.5%) fall between 0.5 and 0.6.

7 Related Work

Related work frames multilingual bitext assessment around two complementary tasks: identifying mutual translations and estimating translation quality without references. Because massively multilingual filtering lacks broad human-labeled coverage, the paper uses human-curated translations as scalable quality anchors rather than replacements for human QE labels.

  • Multilingual sentence embeddings map sentences into a shared space for similarity-based bitext filtering and parallel-pair retrieval.
  • Reference-free quality estimation predicts source–translation quality without human references at inference time, supporting large-scale filtering where references are unavailable.
  • Human-curated translations provide scalable quality anchors for comparing evaluators across many directions, not substitutes for MQM or Direct Assessment labels.

8 Conclusion

The paper decomposes massively multilingual parallel-data assessment into parallelism and reference-free translation-quality components. Across both, model behavior varies by direction: simple QE ensembles dilute strong signals, documented target-language coverage correlates with higher scores, and practical assessment therefore requires routing and calibration.

  • The study treats source–target parallelism assessment and reference-free translation quality estimation as two independent components of model-based assessment.
  • No single embedding or QE model is globally optimal: behavior varies substantially by translation direction and reliability criteria.
  • Mean, median, and weighted QE ensembles dilute strong evaluator signals and do not outperform direction-aware single-model selection.
  • Documented language support is strongly associated with higher QE scores, with target-language coverage more important than source-language coverage in mixed-visibility directions.
  • Multilingual filtering should prioritize direction-level routing and calibration over leaderboard-style universal model selection.

Limitations

The study’s evidence is bounded by surrogate and benchmark coverage, separate component evaluation, proxy-based routing, an incomplete model snapshot, and documented rather than verified language coverage. These constraints limit claims about noisy-data rejection, downstream gains, cascaded filtering, and generalization beyond the evaluated settings.

  • The study does not validate whether its routing and filtering signals improve MT or LLM training outcomes.
  • The parallelism and QE components are benchmarked separately, so the results do not establish that a sequential filtering pipeline improves corpus quality.
  • The QE benchmark is positive-only: high scores on professional FLORES translations do not show that evaluators reject poor, noisy, or domain-shifted translations.
  • FLORES-200 and BOUQuET may not represent OPUS-derived noise, domains, or alignment errors, and sentence-level findings may not generalize to document-level alignment.
  • Direction-specific scores, margins, and coverage metadata are reliability proxies rather than human-labeled filtering boundaries, especially for low-resource directions.
  • The evaluated model suite is not exhaustive, so newer multilingual embedding or QE systems may exhibit different direction-level performance profiles.
  • Documented language support and code matching do not prove training exposure or capture script, dialect, register, or domain differences.

C.1 Language Family-Level Results

The family-level analysis summarizes 41,412 translation directions using family-pair mean scores and separates source- from target-language effects. Target-language families show clearer score shifts and more compact distributions, suggesting stronger target-side influence on evaluator behavior.

  • The analysis groups FLORES-200 languages into 22 Glottolog-based language families to study broader linguistic patterns across 41,412 directions.
  • Figures 3 and 4 use family-pair mean scores, with source-side panels examining source families and target-side panels examining target families.
  • Indo-European source directions show an upward shift among higher-performing MetricX, Qwen3-4B, Qwen3-8B, and ReMedy evaluators.
  • Target-language families produce clearer central-score shifts and, for many models, lower within-family dispersion than source-language families.
  • Target-side distributions are often more compact, indicating greater score consistency within target families and stronger target-side influence than source-side properties.

C.2 Results of Ensemble-based Methods

The section defines how ensemble and coverage-restricted evaluations are compared, while the accompanying tables provide detailed statistics for QE methods and language-family analyses.

  • RQ2 comparison: Table 8 compares nine individual QE evaluators with nine ensemble configurations in an expanded 18-method pool.Its statistics include wins, last-place counts, rank, largest-margin victories, and eligible-direction counts.
  • Ensemble definitions: mean, median, and wavg identify mean aggregation, median aggregation, and weighted averaging, respectively.Coverage qualifiers specify whether constituent models must support both languages, the source, the target, or impose no restriction.
  • Family comparisons: Figures 3 and 4 summarize model family-pair mean scores by source and target language family for the four largest Glottolog families.Each box aggregates one model’s family-pair means over the corresponding opposite-side families.
  • RQ3 coverage analysis: Coverage-restricted analyses group directions by documented source-language, target-language, or joint source–target support, with overlapping single-model subsets.Ensemble rows instead use coverage conditions to select eligible constituents and include only directions with at least one eligible model.
  • RQ3 coverage analysis: Win count and Win % measure first-place finishes within eligible subsets, while Top-1 % and Top-3 % count first- or top-three rankings, including ties.The coverage-restricted ensemble analysis uses a reduced evaluator pool distinct from the full-pool RQ2 ensembles.

C.4 Configuration Insight: Qwen3-4B Batch Size

The Qwen3-4B result depends on configuration rather than model size alone. Faster prompt and batch alternatives improved throughput but lost the ranking behavior that made the original setting competitive.

  • Configuration finding: Qwen3-4B outperformed larger Qwen3-8B and Qwen3-14B variants under a configuration differing in checkpoint recency and batch size.The 4B run used a recent instruction-tuned checkpoint and batch size 32, versus 16 and 8 for the larger variants.
  • Sensitivity analysis: Both faster Qwen3-4B alternatives substantially increased throughput but lost the ranking behavior that made the original configuration competitive.The alternatives used batch size 4 with the structured prompt or a single-segment prompt requesting one 0–100 score.
  • Interpretation: A larger batch may have provided local calibration context, encouraging more stable use of the 0–100 scoring scale.This interpretation is consistent with the paper’s discussion of prompt-context variation and contextual calibration.
Loading 2606.00285v1…