PaperlayerView exact revision on arXiv

Source-linked AI summary

Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation

G. L. John Salvin, Swapnil Hingmire

arXiv:2610.08159v1cs.CLcs.LG

TL;DR

COMET scores are compared across target languages written in different scripts, although script invariance is an untested premise. The paper tests that premise on IndicMT Eval, diagnoses script-related score distortion and proposes corrections; COMET-QN raises pooled cross-script agreement from 0.300 to 0.399 while preserving within-language orderings.

  • Problem

    The paper asks whether COMET scores depend on target script rather than translation quality, a concern for comparisons across languages written in different scripts.

    Source: §1 · pp. 1–2

  • Method

    The authors romanise Indic targets while holding content and ratings fixed, use label-free tokenisation and parity diagnostics, and apply quantile normalisation and a parity-feature regressor.

    Source: §1 · pp. 2–3, §6.4 · p. 9

  • Results

    COMET-QN raises pooled cross-script agreement from 0.300 to 0.399, and a parity-feature regressor recovers a further 17.1% of lost sensitivity.

    Source: §1 · p. 3, §6.4 · p. 9

  • Takeaways & Limitations

    The authors recommend publishing the normalised score, three diagnostics and the identity of the tokeniser used to compute them for COMET on non-Latin-script targets.

    Source: §8 · p. 10

    The measured score-level claims cover COMET wmt22-comet-da with XLM-R’s SentencePiece vocabulary; whether the score distortion generalises to other tokenisers or metric families remains unanswered.

    Source: Limitations · pp. 10–11

Abstract

from arXiv · show

COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts. Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target. We test it on IndicMT Eval by re-encoding the target into Latin script, which changes orthographic form while holding content and human ratings fixed. Script identity then accounts for 22.9% of native-script COMET variance, and agreement with annotators falls in all five languages studied. We trace the effect to the tokeniser and measure it with three label-free diagnostics. The bias is two faults, not one. Scores from different scripts occupy incompatible ranges, and within a single script the metric orders translations less accurately. No order-preserving transform of the score can repair the second fault. The first is removed exactly by COMET-QN, which maps the score distribution of each (language, script) pair onto a shared reference. Pooled agreement with annotators rises from 0.300 to 0.399, which is what makes scores from different scripts safe to place on one axis, and every within-language ordering is provably preserved. A regressor over parity features recovers a further 17.1% of the lost sensitivity. The remainder belongs to the encoder, and no post-processing can reach it. We therefore recommend publishing the normalised score, the three diagnostics, and the identity of the tokeniser they were computed against, so that a reader can tell how much of a score reflects translation quality and how much reflects the writing system.

1 Introduction

The paper tests whether COMET scores are invariant to target script by changing orthographic form while holding translation content and human ratings fixed. It separates script bias into incompatible score ranges and reduced within-script ranking sensitivity, then proposes diagnostics and corrections.

1 Introduction

  • The controlled test romanises Indic targets while keeping source text, reference content and MQM ratings fixed, isolating script as the changed factor.The paper notes that script has rarely been tested as a controlled causal variable and that no inference-time diagnostic quantifies the distortion without retraining.Source: §1 · p. 2
  • Script bias comprises incompatible score ranges across (language, script) cells and poorer ordering of translations within a cell, requiring distinct remedies.The paper calls these failures calibration and sensitivity, respectively.Source: §1 · p. 2
  • The paper recommends publishing the normalised score, three diagnostics and the tokeniser identity so readers can assess the script-related component of COMET scores.The paper reports that romanising every target costs up to 5.57× more encoder compute and lowers agreement with annotators.Source: §1 · pp. 2–3
  • COMET-QN removes calibration without human labels or retraining, raising pooled cross-script agreement from 0.300 to 0.399 while preserving every within-language ordering.A parity-feature regressor recovers a further 17.1% of lost sensitivity without exposure to the target language.Source: §1 · p. 3

2 Background and Related Work

Prior work characterises tokenisation disparities and COMET’s failure modes, but does not test whether script changes evaluation scores under a controlled intervention. The paper connects tokeniser burden to score comparability and adapts quantile normalisation to correct script-related distribution differences.

2 Background and Related Work

  • Tokenization Parity (TP) measures subword pieces per word relative to English, while Information Parity (IP) measures relative compression efficiency; together they describe tokeniser representational burden.TP captures fragmentation and IP the information carried per piece; neither alone is sufficient.Source: §2 · p. 3
  • Prior tokenisation research measures language costs and model behaviour, but not whether those costs propagate into evaluation scores used to rank models.This gap motivates the paper’s first research question.Source: §2 · p. 3
  • Earlier COMET studies examine language mismatch, translationese and within-script perturbations, but do not manipulate orthographic form as a controlled condition.IndicMT Eval reports cross-language COMET variation attributed to language family rather than testing script.Source: §2 · p. 3
  • The audited COMET family shares XLM-R’s SentencePiece vocabulary, so measured fragmentation burden carries across variants and motivates a metric-side correction.TP depends on the tokeniser rather than the regression head.Source: §2 · p. 3
  • Existing output rescaling operates along the language axis, while any order-preserving transform leaves rank agreement unchanged; quantile normalisation instead aligns distributions across batches.The paper maps (language, script) cells to the batch-effect setting used in high-throughput biology.Source: §2 · pp. 3–4

3 Experimental Setup

The experimental setup compares native and romanised targets for five Indic language pairs, with source content and human ratings held constant. It also specifies the checkpoint, tokeniser, transliteration procedure and analysis sample bases.

3 Experimental Setup

  • IndicMT Eval supplies 1,400 MQM-annotated segments for each of five English→Indic pairs, with identical English sources rendered across target languages.ENG-SPA and ENG-DEU provide Latin-script controls.Source: §3 · p. 4
  • IndicXlit deterministically transliterates target text while preserving phonology, morphology and meaning, leaving sources and MQM annotations unchanged.The paper treats any COMET movement as attributable to script alone and notes romanised Indic text is an authentic genre.Source: §3 · p. 4
  • Scores use Unbabel/wmt22-comet-da with XLM-R SentencePiece for TP and BLOOM-560m for IP; empirical claims are bounded to that checkpoint and five Indic abugidas.The diagnostics can be applied to the tokeniser carried by other audited metrics.Source: §3 · p. 4
  • Metric-only analyses use all 7,000 segments, human-score analyses use 6,995, and Marathi severity analyses exclude Default to leave 1,258 segments.The five non-numeric MQM entries removed from the working set affect at most 0.2% of any language.Source: §3 · p. 4

4 The Effect of Romanisation

Romanisation changes COMET scores and annotator agreement across the five Indic languages, while parity diagnostics quantify tokeniser burden without human labels. The section tests the adequacy objection and establishes the magnitude and scope of the observed decline.

4.1 Three observations

  • Script accounts for 22.9% of native-script COMET variance, and COMET agreement with MQM declines in all five languages under romanisation.The decline is larger than for BLEURT, BERTScore and surface metrics; the study attributes the encoder-dependent movement to the encoder rather than the character surface.Source: §4.1 · p. 5

4.2 The parity diagnostics

  • SBI, IPI and Computational Tax estimate script-induced tokeniser burden from token counts and likelihoods, and every romanised setting exceeds both working thresholds.The thresholds are SBI ≥3.0 and IPI > 0.70; SBI combines fragmentation and information parity, while Tax represents the combined token and information cost.Source: §4.2 · pp. 5–6
  • Gujarati’s low native IP explains why its COMET fall is largest despite the smallest TP inflation, while Marathi is the sole language whose mean COMET rises as other quality signals worsen.The table caption identifies these as contrasting cases in the native and romanised comparison.Source: §4.2 · p. 5, Table 2 · p. 6

4.3 Effect size and the adequacy objection

  • COMET’s agreement with MQM declines by |∆ρ| of 0.19–0.39, with significant declines in every language; the authors conclude that a correction is warranted.The authors address the adequacy objection by noting all metrics use the same carried-over ratings and that some unchanged-adequacy outputs gain COMET points under romanisation.Source: §4.3 · pp. 6–7

5 The Marathi Severity Inversion

Marathi is the exception: romanisation raises mean COMET even as quality signals and human agreement worsen, because orthographic collapse makes hypotheses appear more reference-like.

5 The Marathi Severity Inversion

  • 1.02 COMET points: Marathi’s mean rises under romanisation even as IP falls 42.9% and MQM agreement drops from ρ = 0.487 to 0.278.The mean rise is driven by a minority of very poor segments; the median change is −0.21.Source: §5 · p. 7
  • Severity stratification shows gains accrue entirely to badly translated segments while the best output loses ground, with bucket means rising monotonically.Table 3 reports mean ∆COMET by MQM severity for the five ranked buckets; Default segments have no severity rank.Source: §5 · p. 7, Table 3 · p. 7
  • At segment level, severity and ∆COMET correlate at ρ = 0.183† (p = 5.9×10−11, N = 1,258), despite a median change of −0.21.Source: §5 · p. 7
  • Romanisation collapses Marathi diacritic distinctions, making every hypothesis more reference-like to a similarity-based metric regardless of its quality.Source: §5 · p. 7
  • Because Marathi’s bias varies with quality and its aggregate sign opposes the other four languages, one constant correction cannot serve all five.Source: §5 · p. 7

6 From Diagnosis to Treatment

The section separates script bias into incompatible score ranges and lost within-cell ordering, then treats them with quantile normalisation and parity-feature regression.

6.1 Calibration and sensitivity are different problems

  • P1 is cell-range miscalibration (GUJ +15.09 points, MAR −1.02), while P2 is within-cell agreement loss, including in MAR.The two faults vary independently: MAR’s near-zero P1 gap does not prevent its sensitivity loss.Source: §6.1 · pp. 7–8

6.2 Rescaling cannot restore sensitivity

  • For any strictly increasing score transform, Spearman agreement is unchanged; Table 4 shows monotone correctors improve pooled comparability while language-averaged ρlang S stays 0.310.The proposition holds for every dataset; the table uses a 50/50 within-language split and held-out halves.Source: §6.2 · p. 8, Table 4 · p. 8

6.3 COMET-QN: quantile renormalisation

  • COMET-QN maps each cell’s score ranks onto the pooled native-script reference, giving all cells a shared distribution while preserving within-cell ordering.It removes the per-language gap exactly by construction; the pooled native distribution is the chosen reference.Source: §6.3 · p. 8

6.3 COMET-QN: quantile renormalisation; 6.4 Recovering sensitivity from parity features

  • Pooled ρS rises from 0.300 to 0.399 after COMET-QN, and parity-feature regression restores 17.1% of lost sensitivity under leave-one-language-out evaluation.The COMET-QN comparison uses the 6,995-segment working set; regression recovery is averaged over five folds.Source: Figure 1 · p. 9, §6.4 · p. 9, Table 5 · p. 10

6.3 COMET-QN: quantile renormalisation

  • COMET-QN assumes cells share a comparable underlying quality distribution; otherwise it can erase real differences, so its use is limited to a single dataset and domain.The same restriction applies across test sets with different quality distributions.Source: §6.3 · p. 9

6.4 Recovering sensitivity from parity features

  • After COMET-QN, a gradient-boosted regressor uses TP and IP parity features to re-rank segments, mapping (COMETrom, ϕ) to COMETnat without human labels.The features collect TP and IP in both conditions and their differences; romanised/native pairs provide supervision because romanisation preserves content.Source: §6.4 · p. 9
  • The regressor improves four of five leave-one-language-out folds; MAR transfers negatively at −20.3% when its bias runs counter to the training languages.Parity features are a useful proxy only when their TP/IP divergence pattern resembles that of the training languages.Source: §6.4 · p. 9, Table 5 · p. 10
  • A sign-opposed bias such as MAR’s defeats transfer; the paper recommends COMET-QN alone until a corrector trained on languages sharing its sign is available.The remaining five sixths of the deficit is attributed to the encoder and is not addressed by post-hoc composition.Source: §6.4 · p. 10

7 What to Report

For non-Latin-script targets, the paper recommends choosing what to report according to the tokeniser’s parity zone. In the Paradox zone, use COMET-QN and diagnostics rather than comparing raw scores across scripts.

7 What to Report

  • First identify the metric’s encoder and tokeniser, then compute IPI for the language-script pair against that tokeniser without human ratings or metric internals.The commonly used COMET variants carry XLM-R; other metrics should be profiled against their own vocabulary.Source: §7 · p. 10
  • In the Parity zone, report raw COMET alone; in the Burden zone, report raw COMET with IPI so absolute levels are not over-read.The paper says rankings can still be trusted in the Burden zone.Source: §7 · p. 10
  • In the Paradox zone, report COMET-QN alongside raw COMET and all three diagnostics, and do not compare raw scores across scripts.Apply the parity-feature regressor only when the cell’s mean-gap sign matches the languages it was fitted on.Source: §7 · p. 10
  • Do not romanise as a preprocessing convenience: it costs 1.75–5.57× in encoder compute and lowers agreement with annotators.The procedure requires no human ratings.Source: §7 · p. 10

8 Conclusion

The paper recommends reporting COMET-QN, its diagnostics, and the tokeniser used for non-Latin-script targets. Its findings extend beyond the studied corpus only as questions for further testing, including other vocabularies and writing systems.

8 Conclusion

  • For non-Latin-script targets, publish the normalised score, diagnostics, and identity of the tokeniser used to compute them.The paper says common COMET variants use XLM-R, so this protocol covers much published Indic MT evaluation.Source: §8 · p. 10
  • Whether the distortion and correction extend to other tokenisers or non-abugida scripts remains untested, and the study calls for matched-source corpora and direct ratings of romanised output.The paper identifies the deeper remedy as an encoder that spreads capacity evenly across scripts.Source: §8 · p. 10

Limitations

The empirical claims are bounded by the evaluated COMET checkpoint, tokeniser, corpus, and scripts. Several transfer questions and a transliteration-measurement uncertainty remain open.

Limitations

  • The metric-level claims cover COMET wmt22-comet-da with the XLM-R SentencePiece vocabulary, and score-level distortion transfer to related COMET variants is unanswered.The fragmentation burden is shared by variants using the same vocabulary, but whether their score distortion transfers is unknown.Source: Limitations · pp. 10–11
  • The study covers five Indic abugidas; behaviour on ideographic scripts is untested, and no matched-source MQM corpus across several non-Latin scripts is identified.The diagnostics can be computed on other scripts, but the paper makes no claim about COMET-QN performance there.Source: Limitations · p. 11
  • At 1,400 segments per language, the corpus is smaller than cited alternatives, and the IPI zones and SBI ≥3.0 flag are calibrated on this corpus.WMT24 controls provide a first external check, not more.Source: Limitations · p. 11
  • COMET-QN assumes comparable underlying quality across cells, may erase genuine system-pool differences, and is therefore scoped to one dataset and domain.The influence of its reference distribution on the pooled-agreement gain has not been measured.Source: Limitations · p. 11
  • Transliteration accuracy was not measured directly; residual error is bounded by argument, and human ratings of romanised output remain future work.The design also excludes code-switched references.Source: Limitations · p. 11

Ethics Statement

The authors foresee no ethical concerns and describe the work as intended to benefit Indic-language communities whose systems are ranked by the affected metric.

Ethics Statement

  • The authors foresee no ethical concerns and intend the work to benefit Indic-language communities whose systems are ranked by the affected metric.They report that the corpora and benchmarks are public and released artefacts are analytical.Source: Ethics Statement · p. 11

A Statistical Detail and Sample Sizes

The Indic script profile spans distinct tokenizer burdens, with romanisation moving all five languages into the Paradox zone; Latin-script controls show that alphabet alone does not determine the zone. The section also explains the burden measures and calibrates their thresholds.

A Statistical Detail and Sample Sizes

D Dependent Vowels and Word Length

  • MAL has 42.5 isolated dependent-vowel tokens per 1,000 XLM-R tokens and GUJ 36.7, but TAM has the lowest rate and the largest TP inflation under romanisation at +77.9%.TAM’s word-count/token-count correlation reaches ρ = 0.905† when romanised, while vocabulary-free MATTR recovers the ordering MAL > TAM > MAR > GUJ > HIN.Source: D Dependent Vowels and Word Length · p. 15

E Latin-Script Controls

  • DEU falls in Burden while SPA is in Parity despite both using Latin script; DEU’s IP suppression matches the Indic native range, with a twelvefold IP–COMET correlation difference.Their byte premiums are nearly identical, while German compounding is associated with greater fragmentation.Source: E Latin-Script Controls · p. 15, Table 10 · p. 17

A Statistical Detail and Sample Sizes

  • On the working set (N = 1,400 for GUJ and HIN, 1,399 for TAM and MAL, and 1,397 for MAR), COMET loses |∆ρ| = 0.19 to 0.39 under romanisation while surface metrics stay flat.Source: Table 7 · p. 16

F Vocabulary-Free Corroboration

  • Across all seven settings, MATTR and Byte Premium are computed without reference to any model vocabulary, checking the TP and IP findings independently of XLM-R and UTF-8 encoding.Source: F Vocabulary-Free Corroboration · p. 16

G Axiomatic Motivation and Threshold Calibration

C Computational Tax Decomposition

  • The Entropy Penalty dominates the Computational Tax in every language, making information lost per token the principal romanisation cost.The section motivates multiplying Length Penalty and Entropy Penalty because the two charges occur together and cannot be exchanged.Source: G Axiomatic Motivation and Threshold… · p. 16, Table 11 · p. 17

G Axiomatic Motivation and Threshold Calibration

  • The IPI Paradox boundary is 0.70; all twelve native and romanised classifications are insensitive to its placement within the empirical gap between GUJ’s native 0.562 and romanised 0.711.The Parity/Burden boundary is 0.35; no setting falls in the 0.05–0.35 interval.Source: G Axiomatic Motivation and Threshold… · p. 17, Table 10 · p. 17

H COMET-QN Procedure and Reproducibility

COMET-QN maps scores within each language-script cell to a pooled native-script reference while preserving cell order. The Gujarati example demonstrates the reporting procedure and its conditional regressor step.

H COMET-QN Procedure and Reproducibility

  • COMET-QN maps each cell’s fractional score ranks through the empirical quantile function of the pooled native-script COMET reference, preserving every within-cell ordering.Table 15 illustrates the mapping on five romanised Gujarati segments.Source: H COMET-QN Procedure and Reproducibility · p. 17, Table 15 · p. 18
  • The sensitivity regressor uses COMETrom, four native/romanised TP and IP values, and their two differences to predict COMETnat.It is a gradient-boosted tree model with 300 estimators, depth 3, learning rate 0.05, and seed 0.Source: H COMET-QN Procedure and Reproducibility · p. 17

I The Reporting Procedure on One Setting

  • Publish COMET-QN as the headline score beside raw COMET and the three diagnostics; Gujarati’s regressor transfers because its mean cell-gap sign matches TAM, MAL, and HIN, recovering 38.6% of lost sensitivity.Gujarati’s mean native-minus-romanised gap is −15.09 points; if its sign matched MAR instead, the regressor would be withheld.Source: I The Reporting Procedure on One Setting · p. 18
  • Gujarati’s IPI is 0.711, placing it in Paradox; the example reports raw mean COMET of 70.61 with SBInat = 3.45, IPIrom = 0.711, and Tax = 1.75×.The example says the setting uses no human ratings to compute IPI.Source: I The Reporting Procedure on One Setting · p. 18
  • Romanising Gujarati would cost 1.75× encoder compute and lower annotator agreement from ρ = 0.590 to 0.341.The procedure advises against comparing raw Gujarati COMET with a native-script score from another language.Source: I The Reporting Procedure on One Setting · p. 18
Loading 2610.08159v1…