Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems

1 Introduction

• Human evaluation often relies on single overall scores or preference rankings, with limited transparency into how judgments are made.Annotators may take shortcuts and prioritize surface-level properties over aspects requiring deeper analysis.Source: §1 · p. 1
• MQM and ESA are dominant translation-evaluation paradigms, reinforced by WMT benchmarks that shape much contemporary research.The paper examines limitations in these established gold-standard approaches.Source: §1 · p. 1
• Translation assessment demands linguistic expertise, domain knowledge, and substantial time, while existing protocols also face categorization complexity, coarse severity levels, and accuracy bias.Annotation times are often under one minute per segment.Source: §1 · pp. 1–2
• The paper contributes high-quality WMT24 annotations showing improved evaluation methodologies affect system rankings, and introduces RATE for comprehensive translation quality assessment that streamlines error categorization while balancing accuracy and fluency.The annotations enable fine-grained analysis of system differences that the original WMT’24 annotations could not distinguish.Source: §1 · p. 2

2.1 MQM and ESA

• MQM offers detailed hierarchical error categories and severity-weighted scores, while ESA simplifies error-span labeling and uses Direct Assessment scores for system comparisons.ESA ratings range from 0 to 100 and use guidance tied to how much meaning is preserved.Source: §2.1 · p. 2

Problematic Error Categorization in MQM

• MQM’s extensive, overlapping taxonomy can burden annotators and force arbitrary categorization, while a single aggregate score can obscure distinct quality dimensions.The full MQM taxonomy has approximately 200 subcategories; accuracy, fluency, and stylistic appropriateness may matter differently across domains.Source: Problematic Error Categorization in MQM · p. 2

Major/Minor Distinction Lacks Granularity

• Binary major/minor severity labels miss intermediate error impacts, and benchmark annotation times of 34 seconds for ESA and 49 seconds for MQM may be insufficient for nuanced judgments.The paper notes that detecting subtle errors in high-performing systems may require careful attention.Source: Major/Minor Distinction Lacks… · p. 3
• Meaningful differentiation between advanced systems requires qualified evaluators with sufficient time as well as a refined assessment methodology.Source: Major/Minor Distinction Lacks… · p. 3

Error Categorization

• RATE separates semantic errors affecting meaning from non-semantic errors affecting form, using a streamlined taxonomy to provide diagnostic feedback without MQM’s extensive categorization burden.It retains error categorization while grouping types such as mistranslation, omission, grammar, fluency, and style.Source: §3 · p. 3

Severity Assessment

• RATE rates severity on a five-point scale, from critical errors at 5 to minimal issues at 1, with 3 capturing moderate errors between traditional major and minor categories.Source: §3 · p. 3

Quality Scaling

• RATE assigns separate 1–100 ratings for accuracy and fluency, distinguishing faithfulness to source meaning from grammatical correctness and natural expression.Source: §3 · p. 3

RATE Score and RATE MQM Score

• RATE Score sums error severity values, while RATE MQM Score maps severity 4–5 to major and 2–3 to minor before applying the standard weighted formula.The stated formula is 5 × major errors + minor errors.Source: §3 · p. 3

Implementation Considerations

• Effective RATE implementation depends on highly skilled bilingual annotators trained in categorization, severity calibration, and quality scaling.Source: §3 · p. 4

4.1 Selection of Translation Direction

• The study focuses on high-resource English-to-Russian translation, where modern systems perform strongly and evaluation must address subtle quality differences.The authors concentrate resources on one language pair to support thorough linguistic assessment.Source: §4.1 · p. 4

4.2 Annotator Selection and Qualification

• The expert group comprised 13 specialists selected through qualification checks, including degrees, industry experience, language proficiency, post-editing tests, and interviews.The specialists received comprehensive RATE training before evaluation.Source: §4.2 · p. 4

4.3 Dataset and Experimental Setup

• The WMT’24 English-Russian dataset spans news, literary, social, and speech domains; the study selected the human reference and seven top-rated systems and sampled 3,976 segment-system pairs.Sampling included up to nine segments per document for news, literary, and social texts, and one per speech document.Source: §4.3 · p. 4

4.4 Annotation Setups

• In the ESA setup, control-group annotators assessed eight translations with source context; RATE experts also assessed eight translations, marking categorized error spans and scoring accuracy and fluency.The study calls its control-group annotations ESA and standard WMT annotations ESAWMT.Source: §4.4 · pp. 4–5

5.1 Annotator Qualification Impact on Evaluation

• RATE annotators found approximately 7 times more errors per segment than ESAWMT annotators (4.66 vs 0.65), while our ESA annotators found 5 times more (3.34 vs 0.65).RATE annotators spent a median 6.31 minutes per segment, about 11 times the ESAWMT median of 34 seconds; our ESA annotators spent 2.46 minutes.Source: §5.1 · p. 5, Table 3 · p. 5
• Despite identical ESA protocols, our ESA implementation and ESAWMT produced dramatically different results, suggesting annotator selection may matter more than methodology.This comparison supports the broader finding that annotator qualifications and expertise substantially affect evaluation outcomes.Source: §5.1 · p. 5

5.2 System Ranking Comparison

• RATE-derived ESA scores average the overall Accuracy and Fluency ratings, enabling comparison with established MQM and ESA benchmarks.The RATE annotations also allow MQM scores to be computed for each segment-system pair.Source: §5.2 · p. 5
• RATE identified 7 ESA-score clusters and 8 MQM-score clusters, compared with 6 and 5 for our ESA implementation and 2 and 4 for ESAWMT, respectively.On the full dataset, ESAWMT adds only one cluster that separates the human reference; Table 4 reports system rankings and groups systems that are not significantly different.Source: §5.2 · p. 5, Table 4 · p. 6
• Compressing translation qualities into one score can bias rankings toward particular dimensions, and both ESA and MQM show a strong accuracy orientation.Meaning preservation is explicitly emphasized in ESA; MQM scores are primarily driven by major and accuracy errors.Source: §5.2 · p. 6
• MT systems surpass the human reference in accuracy, while the human translation remains superior in fluency, distinctions obscured by single-dimensional evaluation.RATE’s balanced score places refA, Dubformer, and Claude-3.5 in one statistical cluster despite their dimensional differences.Source: §5.2 · pp. 6–7, Table 5 · p. 7

5.3 Error Span Overlap Analysis

• RATE found 90% of ESAWMT errors, while ESAWMT found only 21% of RATE errors; RATE and our ESA implementation overlapped on 65% and 52% of errors, respectively.The authors interpret these patterns as evidence that RATE annotators identified additional subtle issues.Source: §5.3 · pp. 7–8, Table 7 · p. 7
• XCOMET-XXL and GEMBA-MQM covered 36–49% of ESA errors and 33–44% of RATE errors, though they covered 81–85% of ESAWMT errors.The authors report that these automated systems still underperformed human ESA annotators in detecting errors identified by other methods.Source: §5.3 · p. 8

6 Related Work

• WMT standardized human evaluation, moving from ranking assessments to Direct Assessment, while MQM introduced a detailed error typology and ESA simplified span annotation to primarily major/minor errors.Holistic scoring preceded these structured approaches (ALP, 1966; White et al., 1994); WMT was established in 2006.Source: §6 · p. 8
• The expense and time demands of human evaluation motivated automated metrics and LLM-based evaluation, including AI-assisted frameworks that use LLM-identified error spans.The cited approaches include COMET (Rei et al., 2020; Guerreiro et al., 2024), BLEURT (Sellam et al., 2020), MetricX (Juraska et al., 2024), and LLM evaluation work.Source: §6 · p. 8
• Automated evaluation approaches still depend on human-curated gold standards, which are available at scale primarily for English–German and Chinese–English.The passage identifies this limited availability as a bottleneck as leading MT systems converge in quality and fine-grained distinctions become harder to assess.Source: §6 · p. 8

7 Conclusion

• The study introduces RATE and a high-quality dataset of English-to-Russian translation evaluations.Source: §7 · p. 8
• RATE combines accuracy and fluency in a balanced single-score metric, complemented by granular statistics for system analysis.Source: §7 · p. 9
• State-of-the-art MT systems surpass human translations in accuracy but still lag in fluency, a distinction obscured by one-dimensional evaluation.The authors also find that evaluator qualifications and time investment can affect outcomes more than the protocol itself.Source: §7 · p. 9

8 Limitations

• A controlled RATE–MQM–ESA comparison using the same annotators was not conducted because high-skilled annotation costs were substantial.The authors note task design may also influence evaluator focus and scores independently of annotator effects.Source: §8 · p. 9
• The study evaluates English-to-Russian translation; more complex orthographies or grammatical structures may pose additional error-identification challenges.Source: §8 · p. 9
• No standardized framework exists for selecting and qualifying high-skilled annotators, and evaluator expertise affects assessment outcomes.Source: §8 · p. 9
• Annotation quality cannot be objectively measured, so the study relies on indirect signals such as annotator selection, time spent, error frequencies, and overlap.Source: §8 · p. 9

9 Ethics Statement

• Annotators were paid at standard commercial regional rates, with RATE experts earning double the hourly wage; no personal data was collected.Content was screened for potentially disturbing material, and annotators were told how the data would be used before starting.Source: §9 · p. 9
• The vast majority of annotators reported a positive experience and clear instructions in the post-annotation questionnaire.Source: §9 · p. 9

A Overlapping ESA Annotation

• Three independent annotators labeled each segment in the overlapping ESA experiment, with each evaluator annotating all translations of a source segment.The main dataset used one evaluator per segment.Source: A Overlapping ESA Annotation · p. 11
• Averaging three annotators reduced score variance versus single-annotator labeling; ESA rankings shifted slightly closer to RATE while its number of distinguished clusters stayed unchanged.Source: A Overlapping ESA Annotation · p. 11
• Error-span overlap was 59%, which the authors describe as reasonably good agreement and evidence that guidelines produced consistent results across annotator groups.Source: A Overlapping ESA Annotation · p. 12

B RATE Error Categorization

• RATE divides errors into semantic and non-semantic groups, with semantic errors corresponding closely to MQM Accuracy and non-semantic errors covering fluency, grammar, style, inconsistency, do-not-translate, and named entities.Source: B RATE Error Categorization · p. 12
• RATE folds Terminology into Mistranslation rather than treating it as an independent type, to avoid redundancy and a distinction that may be difficult to align across annotators.Source: B RATE Error Categorization · p. 12
• Undertranslation is combined with Not translated, and Overtranslation with Addition, because each pair shares a broad error pattern.Omission remains separate because readers may not recognize that information is missing.Source: B RATE Error Categorization · p. 12
• Grammar includes punctuation, while Fluency covers unnatural or awkward expression, including awkward collocations and tautologies.Source: B RATE Error Categorization · p. 12
• RATE omits MQM categories considered irrelevant to this study or outside translation problems, including design and markup and missing text.Source: B RATE Error Categorization · p. 12

• Mistranslations averaged severity 3.99, while style (2.22) and grammar (2.38) were rated less critical; “Other” errors were least severe at 1.93.Source: C Detailed Statistics on Error… · pp. 12–13
• Inconsistency (4.50) and untranslated content (4.38) received the highest severity ratings, despite occurring only 0.07 and 0.02 times per segment, respectively.Source: C Detailed Statistics on Error… · p. 12
• Fluency errors were the most frequent individual category at 1.96 occurrences per segment, but their average severity was 2.80.Source: C Detailed Statistics on Error… · p. 12
• Non-semantic errors occurred 3.00 times per segment versus 1.56 for semantic errors, but averaged lower severity (2.74 vs. 3.89).Source: C Detailed Statistics on Error… · p. 13

D Error Types Correlations

• RATE Accuracy correlated more strongly with semantic errors (-0.65 overall), while RATE Fluency correlated more strongly with non-semantic errors (-0.61 overall); RATE Average correlated with both (-0.53 and -0.44, respectively).Source: D Error Types Correlations · p. 13, Table 10 · p. 14

E Scores Distribution

• ESA and RATE scores extend across more of the range than ESAWMT scores, which cluster near zero in Figure 2 and near the top of the scale in Figure 3; RATE Average also smooths ESA’s round-number peaks.Source: E Scores Distribution · p. 13

F ESA and RATE Framework Interface

• RATE’s interface lets annotators specify error categories and severity, add comments, and link spans by ID, enabling more detailed qualitative analysis than ESA’s major/minor labels.Source: F ESA and RATE Framework Interface · p. 13

G RATE Annotation Instruction

• RATE instructions ask annotators to categorize errors, assign severity, comment on them, then score accuracy and fluency separately on 0–100 scales; severity runs from 1 to 5.Source: G RATE Annotation Instruction · p. 17