PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Based Quality Assessment for Massively Multilingual Parallel Data

Abdelaziz M. A. Ibrahim, Zihao Li, Jörg Tiedemann, Shaoxiong Ji

arXiv:2606.00285v1cs.CL

TL;DR

Clean parallel data are hardest to obtain for languages most in need of technological support. This paper separately benchmarks multilingual embeddings for parallelism and reference-free estimators for translation quality. Neither component has a globally optimal model across translation directions.

  • Problem

    Multilingual systems depend on parallel training data, yet clean parallel data are hardest to obtain for languages most in need of improved support.

    Source: §1 · p. 1

  • Method

    The study separately benchmarks multilingual embedding models for semantic alignment and reference-free quality estimators for translation quality.

    Source: §1 · pp. 1–2

  • Results

    Across both assessment components, no model is globally optimal; simple QE ensembles dilute strong signals, and target-language coverage is strongly associated with higher QE scores.

    Source: §8 · p. 8

  • Takeaways & Limitations

    Multilingual corpus assessment should prioritize direction-level routing and calibration, conditioning model choice and score interpretation on the language pair.

    Source: §8 · p. 8

    The positive-only QE benchmark tests scoring consistency on professional translations, not reliable rejection of poor translations in web-mined corpora.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large-scale multilingual bitext often contains two distinct problems: non-parallel sentence pairs and low-quality translations. We decompose model-based assessment for such data into two independent components: parallelism assessment with multilingual embeddings and reference-free quality estimation (QE). For parallelism, we benchmark four embedding models on FLORES-200 and BOUQuET retrieval tasks, covering 6,654 source--target directions in our target language-pair inventory. For QE, we evaluate nine reference-free evaluators on professional FLORES-200 translations across 41,412 ordered source--target directions. Results show that no model is universally reliable across translation directions. Naive QE ensembles dilute strong model signals, while documented target-language coverage is strongly associated with higher QE scores. Overall, these findings suggest that multilingual parallel-data assessment is best approached as a direction-aware routing and calibration problem, where no single universal metric is expected to suffice across all languages.

1 Introduction

The paper separates multilingual bitext assessment into source–target parallelism and reference-free translation quality. It evaluates model reliability by translation direction to inform direction-aware routing and calibration rather than seek one universally reliable model.

1 Introduction

  • Language technology remains concentrated in high-resource languages, while languages needing improved support often lack clean parallel data (Joshi et al., 2020).This uneven support persists across a world with more than 7,000 living languages.Source: §1 · p. 1
  • Web-mined and automatically generated bitext expands language coverage but introduces noise, making scalable automatic assessment necessary (Kreutzer et al., 2022).Pairs may be unrelated or contain translation errors; manual inspection is infeasible at this scale.Source: §1 · p. 1
  • The framework separates semantic parallelism, assessed with multilingual embeddings, from translation quality, assessed with reference-free QE.A fluent sentence can express different content, while a semantically aligned translation can still contain errors. Reference-free QE scores source–candidate pairs without gold references at inference time (Zhao et al., 2024).Source: §1 · pp. 1–2
  • Direction-dependent model reliability motivates routing and calibration rather than reliance on a single universal evaluator.Embedding retrieval quality can vary substantially across language pairs, while labeled QE datasets cover far fewer directions than multilingual filtering requires.Source: §1 · p. 2
  • 41,412 ordered translation directions underpin the reference-free QE evaluation using professional FLORES-200 translations (NLLB Team et al., 2022) as a high-quality surrogate benchmark.The study also benchmarks multilingual embeddings and direction-dependent model selection. The surrogate benchmark does not replace MQM, DA, post-editing, or downstream training validation.Source: §1 · p. 2
  • The research asks whether unsupervised QE ensembles improve consistency or provide fallbacks, and how documented source- and target-language coverage affects evaluator behavior.These questions complement the comparison of embedding and QE performance across translation directions.Source: §1 · p. 2

2 Problem Setup: Parallelism and Translation Quality

The setup separates source–target parallelism from reference-free translation quality estimation. It asks whether model-based evaluation can reliably assess many translation directions, particularly low-resource directions with limited or unavailable human-labeled quality data.

2 Problem Setup: Parallelism and Translation Quality

  • The central question is whether model-based evaluation reliably assesses many translation directions, especially low-resource directions with limited or unavailable human-labeled quality data.Source: §2 · p. 2

2.1 Parallelism and Translation Quality

  • Parallelism asks whether sentences express the same content; translation quality asks whether a likely parallel pair preserves meaning fluently and appropriately.These are related but non-identical properties of a source–target pair.Source: §2.1 · p. 2

2.2 Component 1: Parallelism Assessment

  • Parallelism assessment uses cosine similarity between sentence embeddings as an alignment gate, not a complete translation-quality metric.Each direction uses a selected embedding model and its own similarity threshold because similarity distributions vary substantially across language pairs. Higher cosine similarity indicates stronger semantic alignment, without establishing fluency, naturalness, or freedom from errors.Source: §2.2 · p. 3

2.3 Component 2: Reference-Free Quality Estimation

  • Reference-free quality estimation assigns a scalar score directly to a source sentence and candidate translation without requiring a gold reference at inference time.Higher normalized scores should indicate meaning preservation, fluency, and absence of severe omissions, additions, mistranslations, or local errors (Zhao et al., 2024). Each direction uses a selected evaluator and a direction-specific quality threshold.Source: §2.3 · p. 3

3 Direction-Aware Calibration

Parallelism assessment and reference-free QE share a direction-specific protocol: benchmark candidate models, select scorers, and convert scores into thresholds. The proposed routing framework would select the most reliable scorer separately for each translation direction.

3 Direction-Aware Calibration

  • Both components normalize scores to a higher-is-better scale and benchmark models by direction before selecting scorers and setting direction-specific thresholds.Alignment scores represent semantic parallelism, while QE scores represent estimated translation quality.Source: §3 · p. 3

3.1 Parallelism Assessment

  • Parallelism assessment ranks target candidates by cosine similarity and uses mean reciprocal rank (MRR) to measure retrieval of index-matched translations.Models encode all source and target sentences to compute the full similarity matrix, and benchmarking spans multiple parallel evaluation sets. Higher MRR indicates more reliable retrieval of true translations and a stronger direction-specific alignment signal.Source: §3.1 · p. 3

3.2 Reference-Free Quality Estimation

  • QE reliability is assessed through high, stable scores on professional FLORES-200 translations, assumed sufficiently high-quality to merit scores near 1.0.Evaluators receive a mean score for each direction, using uniform per-direction sample sizes. Overall model scores are macro-averages over directions.Source: §3.2 · p. 4

3.3 Direction-Aware Routing

  • Direction-aware routing would select the most reliable scorer for each translation direction rather than apply one model across the corpus.No single model dominates all directions, and model-based signals are not uniformly calibrated across languages. Parallelism routing would maximize retrieval performance; QE routing would prioritize mean scores on professional translations alongside diagnostics such as documented language coverage.Source: §3.3 · p. 4

4 Component 1: Parallelism Assessment

Parallelism assessment evaluates four multilingual embedding models as candidate semantic aligners through sentence-level bitext retrieval benchmarks.

4.1 Embedding Model Suite

  • 4 multilingual embedding models serve as candidate semantic aligners: Harrier (Microsoft, 2026), mE5-large (Wang et al., 2024), GTE (Zhang et al., 2024), and Jina-v3 (Sturua et al., 2025).Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Bitext Retrieval Benchmark

  • 204 language varieties and over 40K ordered translation directions are covered by the professionally translated, sentence-level FLORES-200 retrieval benchmark (NLLB Team et al., 2022).Source: §4.2 · p. 4
  • 275 completed multi-way parallel languages were available in BOUQuET at experiment time, complementing FLORES-style evaluation with broader domain and register coverage.BOUQuET provides sentence- and paragraph-level alignments; the study uses BOUQuETSentence to match the sentence-level corpus pairs targeted by parallelism assessment.Source: §4.2 · p. 4

5 Component 2: Reference-Free Quality Estimation

Reference-free QE systems are compared on professional FLORES-200 translations as a surrogate benchmark, not validated against human QE judgments. Normalized scores support unsupervised ensemble diagnostics, including variants based on documented language coverage.

5.1 QE Model Suite

5.2 FLORES-200 as a Surrogate QE Benchmark

  • The benchmark compares recognition of professional translations as high quality, rather than correlation with human judgments, because FLORES-200 lacks QE labels.Both dev and devtest splits yield 83,196,648 source–translation instances across ordered directions. Low or unstable scores on professional translations undermine justification for filtering noisier data in those directions.Source: §5.2 · p. 5

5.3 Normalization and Ensembles

  • All evaluator outputs are mapped to a common [0, 1] range, with higher values indicating better translation quality.MetricX’s 0–25 lower-is-better scores require normalization; Bicleaner, COMETKiwi, and xCOMET already produce higher-is-better [0, 1] scores.Source: §5.3 · p. 5
  • Unsupervised mean, median, and weighted-average ensembles test cross-direction stability and fallback behavior, not expected superiority over the strongest constituent within each direction.Unrestricted ensembles aggregate all available evaluator scores without supervised training; weighted averaging uses single-model macro-averages as fixed weights.Source: §5.3 · p. 5
  • Coverage-aware ensembles restrict constituent models according to documented support for the source language, target language, both languages, or neither language.Language names are mapped to FLORES-200 codes using exact matching, manually curated aliases, left-trim matching, and qualifier stripping.Source: §5.3 · p. 5

6 Results

No single model is uniformly best across multilingual parallelism assessment and reference-free QE, and naive ensembles dilute stronger QE signals. Documented target-language coverage is associated with higher QE scores, but even the strongest available evaluator assigns low-to-moderate scores to many professional translations.

6.1.1 Parallelism Assessment

  • Harrier achieves the highest average MRR and most routing selections, but no embedding model dominates every covered direction.The benchmark covers 6,654 directions, averaging FLORES-200 and BOUQuETSentence MRR when both are available. mE5-large, Jina-v3, and GTE are selected for 2,013, 1,540, and 54 directions, respectively.Source: §6.1.1 · pp. 5–6, Table 3 · p. 6

6.1.2 Reference-Free Quality Estimation

  • 16,367 wins (39.52%) put ReMedy first by frequency; MetricX leads macro-average (0.6228), while Qwen3-4B minimizes rank mean (2.39) and rank standard deviation (1.25).Across 41,412 ordered FLORES-200 directions, the strongest three models account for 37,175 wins (89.8%). ReMedy nevertheless has only the fourth-highest macro-average, while Qwen3-4B is the most stable near-top model.Source: Table 4 · p. 6, §6.1.2 · p. 6
  • 20,082 directions (48.49%) have winning margins below 0.05, limiting confidence that their highest-scoring evaluator is clearly superior.Only 10,558 directions (25.50%) have a gap of at least 0.10 between the best and second-best model.Source: Table 5 · p. 6, §6.1.2 · p. 6

6.2 RQ2: Unsupervised Ensembles

  • 0.4630, 0.4842, and 0.5026 are unrestricted mean, median, and weighted-average ensemble macro-averages, respectively, below MetricX (0.6228) and Qwen3-4B (0.6160).Averaging across the full evaluator pool dilutes stronger signals because weaker evaluators contribute low scores. This aggregate comparison should not be interpreted as a per-direction failure to exceed the best constituent.Source: §6.2 · pp. 6–7, Table 6 · p. 7
  • 0.7179 for the best both-seen ensemble, the weighted variant, remains below Qwen3-4B’s 0.8498 on the same subset.Both-seen mean and median ensembles reach macro-averages of 0.6901 and 0.7135, respectively. These coverage-favorable subsets include only directions where both languages are documented as covered, limiting comparison with unrestricted results.Source: Table 6 · p. 7, §6.2 · p. 7
  • 1.71 rank standard deviation does not make the unrestricted median ensemble competitive: its mean rank is 8.09 in the expanded method pool.Its stability reflects consistently middle-ranking performance rather than consistently near-top performance.Source: §6.2 · p. 7

6.3 RQ3: Coverage and Target-Side Asymmetry

  • 7,562 directions (18.3%) have best-available mean QE scores below 0.5, and another 3,520 (8.5%) score between 0.5 and 0.6.These scores occur even after selecting the strongest available evaluator for professional FLORES translations, so automatic QE filtering should be applied conservatively.Source: §6.3 · pp. 7–8
  • 0.411 under source-only coverage versus 0.650 under target-only coverage for Qwen3-4B illustrates the target-side advantage observed across all evaluators.The mixed-coverage subsets are equally sized; ReMedy similarly rises from 0.517 to 0.723, and every evaluator scores both-seen directions highest. Coverage is a diagnostic proxy, not proof of training exposure; target-side sensitivity plausibly reflects judging translation fluency, acceptability, and adequacy.Source: §6.3 · p. 7, Figure 2 · p. 8

7 Related Work

This section positions multilingual corpus assessment as direction-aware selection of embedding models for parallelism and reference-free evaluators for quality. Retrieval benchmarks guide embedding selection, while human-curated translations anchor evaluator comparisons across many directions.

7 Related Work

  • Rather than proposing a new mining algorithm, the work uses retrieval benchmarks to select the most reliable embedding-based parallelism signal for each direction.This builds on bitext mining’s goal of identifying mutually translated sentence pairs in noisy or comparable corpora.Source: §7 · p. 8
  • Margin-based scoring addresses absolute cosine thresholds’ poor calibration across sentences and language pairs (Artetxe and Schwenk, 2019a).Earlier mining methods filter or retrieve pairs using multilingual embedding similarity (Schwenk, 2018). This paradigm supported WikiMatrix and extensions to unsupervised, contextual, and low-resource settings (Keung et al., 2020; Heffernan et al., 2022).Source: §7 · p. 8
  • Reference-free QE is treated as an empirical model-selection problem, using human-curated translations to compare evaluators across many directions.These translations serve as scalable quality anchors, not replacements for MQM or Direct Assessment labels.Source: §7 · p. 8
  • QE evaluates translations without inference-time references (Zhao et al., 2024), but evaluator validation covers limited directions.Its sentence-level quality signal supports filtering noisy web-mined and synthetic multilingual corpora where reference data is unavailable at scale (Peter et al., 2023).Source: §7 · p. 8

8 Conclusion

The study separates multilingual parallel-data assessment into parallelism assessment and reference-free translation quality estimation. Both favor direction-aware model routing and calibration rather than a single globally optimal model.

8 Conclusion

  • The assessment framework separates source–target parallelism assessment from reference-free translation quality estimation as two independent components.Source: §8 · p. 8
  • Neither embedding-based parallelism assessment nor reference-free QE has a globally optimal model, supporting direction-aware routing and calibration.Model choice and score interpretation should depend on the language pair; peak performance, average strength, and rank stability emphasize different aspects of reliability.Source: §8 · p. 8
  • Simple unsupervised ensembles dilute strong evaluator signals and do not outperform a direction-aware single-model strategy.This finding applies to mean, median, and weighted aggregation.Source: §8 · p. 8
  • Documented language support is strongly associated with higher QE scores, with target-language coverage consistently more important than source-language coverage in mixed-visibility directions.This suggests that reference-free QE is especially sensitive to target-side competence.Source: §8 · p. 8

Limitations … C Detailed Results

The study provides benchmark-driven assessment signals, not validated downstream training gains or a tested cascaded filtering strategy, and its benchmarks limit generalization. The appendices describe evaluator families, language-coverage caveats, and structured batch versus single-segment scoring prompts.

Limitations

  • The study validates neither downstream training gains nor cascaded filtering, while positive-only QE and sentence-level parallelism benchmarks limit generalization.QE tests high, stable scores on professional translations without full-inventory human QE labels; high scores do not establish rejection of poor translations. Parallelism benchmarks may not represent OPUS-derived noise or generalize to paragraph- and document-level alignment.Source: Limitations · p. 9
  • Routing relies on reliability proxies, so selected models and thresholds require caution; documented coverage is incomplete, and the evaluated model inventory is non-exhaustive.Direction-specific MRR, QE distributions, margins, and coverage metadata are not human-labeled filtering boundaries, and the best available model may remain unreliable where evidence is sparse. Language-code matching establishes neither training exposure nor script, dialect, register, or domain competence; the model comparison is a snapshot.Source: Limitations · p. 9

A QE Model Suite Details

  • COMETKiwi and xCOMET use multilingual encoders, while MetricX uses an encoder–decoder quality regressor whose error-score scale is inverted for comparison.COMETKiwi provides sentence- and word-level QE; xCOMET adds error-span detection and supports reference-free, reference-based, and combined modes (Rei et al., 2023; Guerreiro et al., 2024). MetricX-24 uses multilingual T5-style initialization and inverts its 0–25 MQM-style error scale (Juraska et al., 2024).Source: A QE Model Suite Details · p. 12
  • ReMedy’s local patch broadens accepted language inputs without extending training coverage, and its documented language count describes the released evaluator rather than Gemma 2.ReMedy learns relative quality preferences from human MT comparisons and exposes scalar scores usable with or without references (Tan and Monz, 2025). Gemma 2 lacks a precise public language list, so the documented count is not definitive coverage.Source: A QE Model Suite Details · p. 12

B Prompt Templates

B.1 Structured Batch Prompt

  • The prompt requests seven dimension scores on a 0–10 scale and one overall reference-free quality score on a 0–100 scale for each source–translation pair.Dimensions cover meaning preservation, terminology consistency, fluency and coherence, style/tone/audience, locale formatting, technical integrity, and cultural appropriateness. Results contain one item per input segment, ordered by id.Source: A QE Model Suite Details · p. 12, §B.1 · p. 13

A QE Model Suite Details

  • Bicleaner AI estimates mutual translation as a corpus-cleaning baseline rather than a dedicated MT quality metric.Its objective is narrower than the explicit QE objectives of COMETKiwi, xCOMET, MetricX, ReMedy, and the LLM judges.Source: A QE Model Suite Details · p. 12

B.2 Simple Single-Segment Prompt

  • The prompt and batch-size sensitivity experiment uses a simpler prompt for Qwen3-4B as an LLM judge, requesting an integer reference-free MT quality score from 0 to 100.Higher scores indicate better quality. Output must be valid JSON containing only the key "overall_0to100" with an integer value.Source: A QE Model Suite Details · p. 12, §B.2 · p. 13

C.1 Language Family-Level Results

Evaluator scores show stronger family effects on the target side than the source side, with Indo-European targets scoring highest and Atlantic-Congo targets lowest. Source-family results distinguish MetricX’s stable high central scores from ReMedy’s more variable performance.

C.1 Language Family-Level Results

  • 22 Glottolog-based language families organize 41,412 translation directions, with qualitative discussion restricted to Indo-European, Atlantic-Congo, Afro-Asiatic, and Austronesian.Representation is imbalanced: Indo-European accounts for 79 of 204 varieties. Separate source- and target-side boxplots use family-pair mean scores rather than segment scores, so dispersion reflects variation across family combinations.Source: §C.1 · pp. 13–14
  • MetricX maintains relatively stable high source-family central scores, whereas ReMedy, Qwen3-14B, and Qwen3-8B exhibit substantial variance.The analysis suggests that ReMedy’s direction-level superiority partly reflects favorable family pairings rather than consistent cross-family performance.Source: §C.1 · p. 14
  • Source-family results show an upward shift for stronger evaluators on Indo-European directions, generally lower Atlantic-Congo scores, and intermediate Afro-Asiatic and Austronesian scores.The Indo-European shift involves MetricX, Qwen3-4B, Qwen3-8B, and ReMedy; Atlantic-Congo declines are especially notable for Bicleaner, COMETKiwi, xCOMET, M-Prometheus, and the Qwen variants. Uneven representation in pretraining corpora is proposed as a possible explanation.Source: §C.1 · p. 14
  • Target-family effects exceed source-family effects: leading evaluators score Indo-European targets highest, Atlantic-Congo targets lowest, and Afro-Asiatic and Austronesian targets intermediately.Qwen3-4B, Qwen3-8B, and MetricX cluster near the upper bound for Indo-European targets; MetricX and Qwen3-4B have the highest central scores for intermediate families. MetricX and ReMedy remain most competitive for Atlantic-Congo targets, despite mean and median scores below 0.5.Source: §C.1 · p. 14
  • More compact target-side distributions suggest that target-family properties may explain more evaluator behavior than source-family properties.Reference-free evaluation requires judging target-language fluency and acceptability, which depends directly on the evaluator’s internal proficiency in that language.Source: §C.1 · p. 14

C.2 Results of Ensemble-based Methods · C.3 Language Coverage Analysis

Table 8 compares individual QE evaluators with unsupervised ensembles, using coverage-dependent eligibility and different denominators across metrics. The language-coverage analysis separately stratifies evaluators by documented support and uses a reduced ensemble pool.

C.2 Results of Ensemble-based Methods

  • 41,412 directions normalize Table 8’s Win %, whereas Macro, Last, and Rank are computed over each method’s eligible directions.Some coverage-filtered ensembles are defined only when their constituent pool is non-empty.Source: §C.2 · p. 14
  • Ensemble names specify mean, median, or weighted averaging and optional Both, Source-seen, or Target-seen filters based on documented language support.Both requires support for both languages; Source-seen and Target-seen require support for the corresponding language. Unqualified ensembles apply no coverage restriction and include all 41,412 directions.Source: §C.2 · p. 14
  • 18 methods—nine individual evaluators and nine ensemble configurations—form Table 8’s expanded comparison, so its ranks differ from the single-model ranks in Table 4.The full comparison also reports last-place counts, largest-margin wins, and eligible-direction counts omitted from the compact main-text table.Source: §C.2 · p. 14, Table 8 · p. 16

C.3 Language Coverage Analysis

  • Coverage diagnostics use a reduced ensemble pool of MetricX, ReMedy, M-Prometheus, and Qwen3 variants, explaining macro-average differences from unrestricted RQ2 ensembles.CoMETKiwi, xCOMET, and Bicleaner were excluded from coverage analysis because of low overall benchmark performance.Source: §C.3 · p. 16
  • Single-evaluator coverage statistics group directions by documented source-language, target-language, or joint support; these overlapping subsets must not be summed.The Coverage subset labels Source, Target, and Both identify these respective groups.Source: §C.3 · p. 16
  • Ensemble coverage rows select constituents by support conditions and retain only directions with an eligible model; Neither selects evaluators supporting neither language.For ensembles, Source, Target, and Both describe constituent-selection conditions rather than single-evaluator direction subsets.Source: §C.3 · p. 16
  • Coverage-table Win count and Win % summarize first-place finishes within eligible subsets; Top-1 % and Top-3 % report ranking frequencies including ties.Top-1 % measures first-place frequency, while Top-3 % measures placement among the top three candidates.Source: §C.3 · p. 16

C.4 Configuration Insight: Qwen3-4B Batch Size

Qwen3-4B’s strong performance relative to larger variants should be interpreted at the configuration level: faster alternatives increased throughput but lost its competitive ranking behavior. Larger batches may have supplied local calibration context.

C.4 Configuration Insight: Qwen3-4B Batch Size

  • 32, 16, and 8 were the batch sizes for Qwen3-4B, Qwen3-8B, and Qwen3-14B, respectively; Qwen3-4B also used a more recent instruction-tuned checkpoint.The strong Qwen3-4B result therefore warrants interpretation as a configuration-level finding.Source: §C.4 · p. 16
  • Larger batches may provide local calibration context, stabilizing use of the 0–100 scoring scale rather than merely changing evaluator efficiency.Presenting 32 source–translation pairs together exposed the model to a wider range of examples before scoring. This interpretation aligns with contextual-calibration evidence and batch-size/item-order effects (Cheng et al., 2023; Lin et al., 2024).Source: §C.4 · p. 17
  • Both faster Qwen3-4B alternatives increased throughput but lost the ranking behavior that made the original configuration competitive.The alternatives used the same structured prompt with batch size 4, or a simple single-segment prompt requesting one 0–100 score.Source: §C.4 · pp. 16–17, Table 11 · p. 20

Related papers

Loading 2606.00285v1…