PaperlayerView exact revision on arXiv

Source-linked AI summary

The Blindness of Document-Level Translation Evaluation

Ahrii Kim, Vilém Zouhar, Chanjun Park, Seong-heum Kim

arXiv:2609.05949v1cs.CLcs.HC

TL;DR

Document-level MT evaluation assumes that showing annotators full documents elicits document-level judgments, but whether current protocols capture those judgments is uncertain. The paper tests this with mixed-system documents that disrupt coherence while preserving document presentation. Human and automatic evaluations largely fail to distinguish coherent from incoherent documents, although raters can identify the coherent passages.

  • Problem

    Whether document-level evaluation protocols capture document-level quality remains unconfirmed, despite their use of document context.

    Source: §1 · p. 1

  • Method

    The study compares coherent documents, mixed-system documents that disrupt cross-segment coherence, and isolated segments using human ESA annotation and 14 automatic metrics.

    Source: §1 · p. 2

  • Results

    Human ESA scores, system rankings, and error annotations, and all 14 automatic metrics’ rankings, are statistically equivalent between coherent and incoherent documents; raters identify coherent passages as single-translator work in 87.3% of trials.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that document-level presentation alone does not ensure document-level judgments are recorded, and call for evaluation protocols designed to elicit them.

    Source: §6 · p. 9

    MIX tests one strong form of incoherence by disrupting consistency while preserving source content and sentence order, so it is an upper bound rather than a sample of naturally occurring document-level errors.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Document-level machine translation (MT) evaluation extends segment-level protocols by presenting full documents to annotators, on the assumption that such presentation elicits document-level judgments. We test this assumption with a counterfactual condition (MIX) in which each document combines segments drawn from different systems, preserving document-level presentation while breaking cross-segment consistency. Across 18,420 expert Englis-to-Korean annotations and 14 automatic metrics, scores, system rankings, and error annotations are statistically equivalent between coherent and incoherent documents. Perception does not explain this: shown matched passages, raters identify the coherent one as the work of a single translator in 87.3% of trials. Document presentation does change how annotators work, but that change does not reach the recorded output. What is blind is the protocol, not the annotator. The concern is not that scores fall short, but that the resources invested in document-level systems, metrics, and annotation may not be measuring what they are intended to measure.

1 Introduction

Document-level evaluation assumes that displaying full documents elicits document-level judgments, but that assumption has not been thoroughly validated. This study tests it with incoherent documents that preserve document presentation and finds that recorded evaluations do not distinguish them from coherent documents.

1 Introduction

  • Document-level protocols provide document presentation, but the assumption that this produces document-level judgment remains unconfirmed.A cue may be perceived yet go unregistered if the evaluation instrument offers no way to record it.Source: §1 · pp. 1–2
  • The study constructs MIX documents from segments produced by multiple systems, preserving document-level presentation while disrupting cross-segment coherence.It compares MIX with coherent DOC documents and isolated SEG segments using 18,420 expert English→Korean annotations and 14 automatic metrics.Source: §1 · p. 2
  • Raters identify DOC passages as the work of a single consistent translator in 87.3% of matched trials.This blind positive control shows that the coherence difference is perceptible.Source: §1 · p. 2
  • Human ESA scores, system rankings, and error annotations, as well as all 14 automatic metrics’ rankings, are statistically equivalent between coherent and incoherent documents.The authors report that document presentation changes annotator behavior, but ESA provides no channel for that change to register.Source: §1 · p. 2
  • The authors argue that current evaluation may misalign the measure with its target, motivating renewed attention to discourse-level evaluation.They distinguish this concern from score corruption caused by optimizing a measure.Source: §1 · p. 2

2 Related Work

Prior human and automatic approaches provide document context or refine segment-level scoring, but the supplied passages describe no direct evidence that these protocols capture document-level quality.

2.1 Human Evaluation of Document MT

  • Human evaluation evolved from document-level averages and holistic scoring toward segment-level annotation with document context, but the resulting annotations remain segment-centric.The passage notes that holistic document scoring had low inter-annotator agreement and that newer protocols refine taxonomies and interfaces.Source: §2.1 · p. 2

2.2 Automatic Evaluation of Document MT

  • Automatic evaluation extends context through full-document metrics, prepended context, sliding windows, and book-length alignment, while whether full-document inputs capture document-level information remains unclear.The passage also notes that local windows may miss phenomena requiring global coherence.Source: §2.2 · p. 2
  • Discourse-targeted metrics cover phenomena such as pronoun resolution, lexical choice, tense, entities, and terminology, but rely on surface matching and do not address narrative coherence.MUDA offers taggers for locating context-dependent phenomena in arbitrary datasets.Source: §2.2 · pp. 2–3
  • The paper applies context perturbation to evaluation instruments, testing whether document-level input alone is sufficient for evaluation to operate at the document level.Earlier perturbation work examined whether translation models respond to corrupted context; the authors shift the test from models to evaluators.Source: §2.2 · p. 3

3 Methods

ESA presents a full translated document while collecting a score for each segment, then averages those scores into a document-level score. The counterfactual MIX condition keeps that interface but disrupts coherence to test whether ESA registers context-driven cues.

3 Methods

  • ESA elicits one score per segment while presenting the full document as context, then defines the document-level score as the average of segment scores.The segment score is decomposed into the score it would receive in isolation and cues induced by cross-segment context.Source: §3 · p. 3
  • MIX preserves ESA’s document-level display and per-segment judgments but assembles each document from multiple systems’ segments, disrupting cross-segment coherence.The counterfactual document score is defined analogously to the coherent-document score.Source: §3 · p. 3
  • If ESA registers coherence cues, MIX should differ from DOC in scores, error counts, and severity, with highlighted errors expected near segment boundaries.The protocol records a numeric score, error spans, and a Major/Minor severity label for each span.Source: §3 · p. 3

4 Experiment

The experiment tests whether coherent and mixed documents differ under defined score, ranking, and error-count criteria, using English→Korean data and manipulation checks. The checks show that readers can detect the coherence disruption, while the evaluation setup spans human annotation and automatic metrics.

4.1 Hypotheses

  • The hypotheses test DOC–MIX score equivalence within a margin, ranking agreement against a threshold, and error-count equivalence, with SEG–DOC comparisons checking presentation effects.The procedures are TOST for scores and errors and Kendall’s τ with bootstrap confidence intervals for rankings.Source: §4.1 · p. 4

4.2 Dataset

  • The dataset contains 3,070 translations from nine MT systems and a human reference across 23 documents in news, social, and literary domains, yielding 18,420 expert judgments.Speech is excluded because its single-utterance composition is unsuitable for document-level analysis.Source: §4.2 · p. 4

4.3 Manipulation checks

  • In matched six-segment passage pairs, blind raters chose DOC as the work of a single consistent translator in 87.3% of trials.The authors conclude that a weak manipulation cannot explain DOC–MIX equivalence, while other explanations remain open.Source: §4.3 · p. 4

4.4 Human Evaluation

  • Human evaluation uses ESA on a blinded, randomized set of tasks assigned to ten professional native Korean translators.Each rater annotates 706 tasks drawn from the three conditions.Source: §4.4 · pp. 4–5
  • Figure 2 reports near-identical score distributions across SEG, DOC, and MIX, paired differences within the ±3 equivalence band, and opposite agreement orderings by document versus segment unit.The figure also presents system-level comparisons, error rates, and score recoverability.Source: Figure 2 · p. 5

4.5 Automatic Metrics

  • The automatic-metric evaluation includes segment-level metrics and document-level metrics, including document variants of xCOMET-XXL and METRICX-24-XXL.The listed approaches include chrF, learned metrics, quality-estimation variants, and LLM-as-judge metrics.Source: §4.5 · p. 5

4.6 Testing Procedure

  • The analysis also measures Krippendorff’s α and per-segment annotation time as complementary indicators of annotators’ cognitive effort.These behavioral measures supplement the hypothesis tests.Source: §4.6 · p. 5

5 Results

Human ESA and automatic metrics produce equivalent outputs for coherent DOC and incoherent MIX documents. The analyses test this pattern across scores, rankings, error annotations, annotator behavior, and domains.

5 Results

  • DOC and MIX behave equivalently under ESA evaluation, for human annotators and automatic metrics.This is the central pattern across the results.Source: §5 · p. 6

5.1 [H1] Scores are insensitive to discourse

  • All 14 metrics and human ESA scores meet their equivalence tests for DOC versus MIX; human ESA means are 86.11 and 86.93, respectively.The MIX mean marginally exceeds DOC, opposite the predicted direction, but the conditions remain statistically indistinguishable.Source: §5.1 · p. 6, Table 1 · p. 6

5.2 [H2] System rankings collapse

  • Human ESA rankings correlate at τ = 0.778 between DOC and MIX, while every automatic metric yields τ = 1.00.Eight of ten systems also have identical or adjacent ranks across human ESA conditions, with nearly matching cluster compositions.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 [H3] No additional errors are marked

  • DOC and MIX have equivalent error counts: means are 2.31 and 2.37 per annotation, with differences within 0.15.Error-free rates are 15.5% for DOC and 15.7% for MIX. Boundary-localized spans are 50.1% for DOC and 50.0% for MIX; the pattern does not change when analyses exclude omission markers or restrict segments or spans.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 Mechanism: MIX is evaluated in isolation

  • SEG scores explain more variance in MIX than DOC scores (R2 = 0.52 vs. 0.20), while every metric aligns more closely with human rankings under MIX.The authors interpret the stronger MIX recoverability as annotators falling back on segment-level judgments when coherence is broken.Source: §5.4 · pp. 7–8, Figure 3 · p. 8

5.5 Behavioral analysis

  • DOC agreement leads per document but lags per segment; document conditions also shorten annotation time versus SEG.Per-document Krippendorff’s α is 0.19 for DOC and 0.13 for MIX, while per-segment α is 0.15 for DOC and 0.23 for MIX. Annotation takes 61s per segment in DOC and 56s in MIX, versus 76s in SEG; the authors treat agreement as corroborating, not decisive, evidence about context use.Source: §5.5 · p. 8

5.6 Domain-wise replication

  • Score and error differences remain within equivalence bands across news, social, and literary domains, with no resolvable graded effect.Ranking agreement decreases from τ = 0.78 in news to 0.64 in social and 0.56 in literary, but bootstrap intervals overlap substantially; literary results rest on two source documents.Source: §5.6 · pp. 8–9, Figure 4 · p. 9

6 Conclusion

Across human ESA and 14 automatic metrics, no evaluator distinguished coherent documents from deliberately incoherent ones in the three tested domains. The authors argue that document-level evaluation needs gold data and interfaces that elicit document-level judgments, not merely display document context.

6 Conclusion

  • Segment-level human judgments have shaped the gold data against which later metrics are trained or calibrated.The authors frame their counterfactual experiment as testing what this legacy captures.Source: §6 · p. 9
  • Across human ESA and 14 automatic metrics, no evaluator distinguished coherent from deliberately incoherent documents in any of the three tested domains.The authors note that metrics included discourse-targeted variants and LLM-based scorers.Source: §6 · p. 9
  • Document-level presentation measurably changes annotator behavior, but current evaluation outputs provide no place for those judgments to register.The authors set aside both perception and context length as explanations for the failure.Source: §6 · p. 9
  • Training on segment-based gold is unlikely to produce document-level competence; genuine progress will likely require protocols that elicit document-level judgments and new gold data.Until then, reported document-level progress is hard to separate from segment-level progress under the authors’ measure.Source: §6 · p. 9

Limitations

The findings are bounded by the English→Korean ESA setting, an uneven domain sample, a small annotator and system pool, and the particular incoherence created by MIX. The study diagnoses a limitation of ESA but does not establish whether other protocols share it or offer a replacement.

Limitations

  • The finding is scoped to English→Korean under ESA; replication across typologically diverse language pairs remains the primary next step.The authors chose Korean as a setting where ESA should be especially likely to register discourse-sensitive phenomena, but say this does not establish the same result elsewhere.Source: Limitations · p. 10
  • Literary data make up 44.6% of annotations but come from only two source documents, so domain-stratified results are replication checks, not domain comparisons.The literary ranking result is reported with bootstrap intervals.Source: Limitations · p. 10
  • The analyses use 10 annotators and 9 MT systems, and Krippendorff’s α is below 0.25 across conditions.Within-annotator paired designs, bootstrap confidence intervals, and statistical clustering mitigate small-sample limitations.Source: Limitations · p. 10
  • MIX tests one strong form of incoherence by combining systems’ segments; it preserves source content and sentence order and cannot create errors internal to one system’s output.The authors treat MIX as an upper bound on the consistency signal, not a sample of naturally occurring document-level errors; subtler perturbations remain untested.Source: Limitations · p. 10
  • The study focuses on ESA, and whether the failure extends to protocols such as MQM with explicit discourse-level error categories is open.The authors diagnose the failure but do not propose a replacement protocol.Source: Limitations · p. 10

Ethics Statement

The study reports voluntary expert participation, fair-market compensation, anonymized annotations, and research-use conditions for its materials. Its dataset uses WMT 2025 English→Korean test documents, prepared and split for evaluation across three conditions.

Ethics Statement

  • Expert participation was voluntary, annotators received $40 per hour, and annotations were anonymized with no personally identifiable information collected.Source: Ethics Statement · p. 10
  • The WMT25 test set and released ESA annotations use CC-BY 4.0; listed metrics and analysis code have their stated Apache 2.0 or MIT licenses.GEMBA V2 and FALCON are computed via the OpenAI API under its terms of service.Source: Ethics Statement · p. 10
  • Released annotations and datasets are intended for research use only, and AI assistants were used for writing refinement and code review, not experimental design or analysis.Source: Ethics Statement · pp. 10–11

A.1 Dataset

  • The dataset uses WMT 2025 parallel test documents from authentic sources across domains, with professional reference translations and outputs from participating systems.The multi-system format enables comparisons on identical source material; the edition explicitly targets document-level evaluation.Source: §A.1 · p. 14
  • Preprocessing selects documents containing informative segments, appends segments shorter than 10 tokens to the preceding segment, and splits long documents into sub-documents.Long segments are split at the nearest line break when available; resulting documents contain at least two segments.Source: §A.1 · pp. 14–15
  • The final dataset covers 9 MT systems plus one human reference, with 18,420 annotations across three conditions; literary data contribute 44.6% from two source documents.The domain mix is unbalanced, and analyses use segments evaluated under all three conditions.Source: §A.1 · p. 14, Table 5 · p. 15

A.2 Construction of MIX

The appendix constructs MIX documents by recombining system translations while preserving source content and order, then checks that the pooled systems differ and that readers perceive the assembled passages as inconsistent. It also describes the human annotation procedure and the automatic metrics used.

A.2 Construction of MIX

  • MIX preserves source segments and their order but randomly draws each translation from multiple systems, with every pooled system contributing at least one segment.Each segment prefers a system other than the DOC system at that position; this breaks coherence by construction.Source: §A.2 · p. 15
  • Raters perceived a quality difference in 85–94% of judgments across the three maximally rank-overlapping system pairs.The result supports that MIX combines genuinely different translations, although system differences alone do not establish within-document inconsistency.Source: §A.2 · pp. 15–16
  • DOC was chosen as the work of a single consistent translator in 89 of 102 matched passage pairs (87.3%).The two passages were approximately matched on mean segment quality, and raters were blind to how they were constructed.Source: §A.2 · p. 16

A.3 Human Annotation

  • In the illustrated three-sentence passage, MIX shifts Korean speech level from Plain to Formal-polite to Informal-polite, yet was not scored lower than DOC.The source has no corresponding register discontinuity; Korean marks speech level on each sentence-final verb.Source: §A.2 · p. 16, Table 7 · p. 17
  • ESA annotators marked error spans, labeled their severity, and assigned 0–100 scores, with discourse properties explicitly included in the Korean guideline.MIX was presented in the same interface as DOC without revealing the counterfactual design; no separate coherence-scoring item was added.Source: §A.3 · pp. 17–18, Figure 5 · p. 17

A.4 Automatic Metrics

  • The evaluation suite spans surface, learned, and LLM-as-judge metrics at segment and document granularities.The metric inventory includes chrF, learned quality-estimation metrics, and document-level approaches such as doc-COMET, SLIDE, and FALCON.Source: §A.4 · p. 18, §A.5 · p. 18

A.5 Segment-level metrics

  • Segment-level metrics include chrF, learned reference-based and reference-free models, and GEMBA V2, an LLM judge prompted for quality scores.METRICX produces error scores that are negated in the analyses for consistency with other metric directions.Source: §A.5 · p. 18

A.6 Document-level metrics

  • Document-level metrics include corpus BLEU, context-aware COMET, a sliding-window evaluator, and FALCON’s full-document LLM judgments.The document-level suite includes both reference-based and reference-free approaches.Source: §A.6 · p. 18

A.7 Implementation notes

The appendix details metric handling, equivalence testing, and ranking analysis. Across the tested score comparisons, equivalence is robust to comparison, design, and margin; ranking analyses account for uncertainty among near-tied systems.

A.7 Implementation notes

  • Human–metric agreement is restricted to segment-level metrics because a MIX document combines systems and cannot be unambiguously assigned to one system.Distributing a document score across its constituent systems would require modeling assumptions that could introduce analysis artifacts.Source: §A.7 · pp. 18–19
  • Document-level xCOMET-XXL accepts only 512 tokens, truncating typical documents and yielding scores around 0.16 in both conditions.The authors interpret xCOMET-XXL-doc results as reflecting truncation rather than genuine document-level evaluation.Source: §A.7 · p. 19

B.1 Equivalence Testing (TOST)

  • TOST tests whether condition differences fall within a prespecified practical-equivalence margin; the main analysis pairs observations on the same segment.The primary margin is ε = 3 ESA points, approximately 0.2 standard deviations and below typical ESA inter-annotator agreement magnitude.Source: §B.1 · p. 19

B.2 Ranking Analysis [H2]

  • Ranking comparisons use nine MT systems, document-level bootstrap resampling, and pairwise tests to leave statistically indistinguishable systems unordered.The ranking threshold τmin = 0.667 denotes moderate agreement; values below it indicate reordering across conditions.Source: §B.2 · pp. 19–20
  • For human ESA, Kendall’s τ between DOC and MIX rankings averages 0.737 (95% CI [0.539, 0.911]), while 22.5% of bootstrap resamples fall below τmin = 0.667.The authors treat statistical clustering as primary because τ can order systems whose score differences are indistinguishable.Source: §B.2 · p. 20

B.3 Error Annotation Analysis [H3]

The appendix specifies how error annotations and annotation behavior are analyzed, then reports robustness checks for score equivalence. Error-span position is interpreted through between-condition contrasts rather than absolute boundary rates.

B.3 Error Annotation Analysis [H3]

  • Error counts are normalized per 1,000 target characters, and paired TOST compares MIX with DOC using a margin of 0.5 errors per annotation.The analysis also reports error-free annotation rates, severity distributions, and the share of spans near segment boundaries.Source: §B.3 · p. 20
  • About 50% of annotated spans are boundary-localized in all conditions, versus a 40% rate under no positional structure.Because the absolute rate is similar across conditions, the between-condition contrast carries the inference.Source: §B.3 · p. 20
  • Across three robustness variants, the MIX–DOC error-score difference stays within ±1.5 points, with confidence intervals covering zero.The variants exclude zero-width markers, restrict to document-interior segments, or retain only Major spans.Source: §B.3 · p. 20

B.4 Behavioral Measure

  • Inter-annotator agreement is measured with Krippendorff’s α at document and segment units, while annotation time excludes inactive gaps over 120 seconds.The time measure divides active evaluation time by the number of task segments.Source: §B.4 · p. 20

C.1 Score Equivalence [H1]

  • All 18 score-equivalence cells support equivalence at p < 0.001 across comparison choice, study design, and margins ε ∈ {3, 5, 7}.The tested mean differences range from −1.37 to +0.55 points.Source: Table 8 · p. 21, §C.1 · p. 21

C.2 IAA

Inter-annotator agreement changes with the unit of analysis: DOC leads at the document level, while MIX leads at the segment level. Agreement values remain below the conventional reliability threshold, and MIX’s document-level item has a structural limitation.

C.2 IAA

  • Document-level agreement is highest for DOC (α = 0.186), but segment-level agreement is highest for MIX (α = 0.225) and lowest for DOC (α = 0.155).The ordering reverses between analysis units.Source: Table 9 · p. 21, §C.2 · p. 21
  • All agreement values fall below the conventional acceptable-reliability threshold of α ≥ 0.667.The authors note this does not compromise their primary findings because aggregated scores use 5,920 samples per condition and paired analyses control for inter-annotator differences.Source: Table 9 · p. 21, §C.2 · p. 21
  • MIX’s document-level agreement is largely a structural artifact because the same document–system item can refer to different segment sets across annotators.The analysis therefore emphasizes segment-level α, where item identity is unambiguous.Source: §C.2 · p. 21
  • After annotator-wise z-score normalization, DOC’s document-level α rises from 0.186 to 0.23, while SEG and MIX remain around 0.15 and 0.13.The document-level ordering is therefore not attributed to annotator-specific scoring styles.Source: §C.2 · p. 22

C.3 Human–Metric Agreement

Human–metric agreement generally shifts upward from DOC to MIX, with reported correlations varying by metric family. Narrow reference-free QE score ranges and the nine-system sample make ranking-based agreement noise-sensitive.

C.3 Human–Metric Agreement

  • With n = 9, Kendall’s τ and Pearson’s r are noise-sensitive, so the analysis emphasizes relative patterns across conditions rather than absolute correlations.The authors caution that absolute correlation values are noise-sensitive at this sample size.Source: §C.3 · p. 22
  • chrF and d-BLEU agreement shifts from τ = 0.67 under DOC to τ = 0.89 under MIX.The same upward direction holds for learned metrics at lower absolute values.Source: §C.3 · p. 22
  • xCOMET-XXL shifts from 0.50 to 0.61 and METRICX-24-XXL from 0.56 to 0.67, while reference-free QE variants shift from τ = 0.22–0.28 to 0.33–0.39.The DOC and MIX comparisons are reported in that order.Source: §C.3 · p. 22
  • GEMBA V2 moves from τ = 0.50 under DOC to 0.72 under MIX, matching surface metrics in MIX-shift magnitude while sitting between learned reference-based and surface metrics in absolute agreement.The comparison with surface metrics in absolute agreement continues in the adjacent passage.Source: §C.3 · p. 22
  • Reference-free QE metrics have the narrowest system-score ranges: COMETKIWI-XXL ∼5.6% and METRICX-24-QE-XXL ∼3.4% of its 25-point scale.Table 11 reports system-level mean score distributions across nine systems; narrow ranges make ranking-based agreement noise-sensitive.Source: Table 11 · p. 22, §C.3 · p. 22
Loading 2609.05949v1…