Source-linked AI summary
Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev, Stepan Vysotsky, Nikolay Karpachev
TL;DR
Expert MT evaluation is time-consuming, while the design choices shaping LLM evaluators remain poorly understood. The paper systematically studies those choices and introduces ORBIT; its strongest configurations achieve expert-level span detection and ranking at roughly 20× lower annotation cost.
Problem
Evaluator design choices remain poorly understood, even as expert assessment requires substantial annotation time and quality control and automatic metrics lag on fine-grained evaluation.
Source: §1 · p. 1Method
The study systematically tests instruction design, few-shot demonstrations, backbone choice, and multi-step workflows, using new expert RATE annotations and additional MQM- and ESA-annotated benchmarks.
Source: §1 · p. 2Results
Across protocols and language pairs, curated demonstrations can match or outperform larger pools, model families show distinct profiles, and fusion yields the strongest results; ORBIT achieves expert-level span detection and Accuracy/Fluency ranking at roughly 20× lower cost.
Source: §1 · p. 2, §6 · p. 9Takeaways & Limitations
Several evaluator design principles transfer consistently across annotation protocols and language pairs, and ORBIT instantiates the robust choices identified in the study.
Source: §1 · p. 2, §6 · p. 9The exploratory collaboration study measures annotation revision rather than ground-truth correctness and uses only ORBIT-MC, a limited annotator group, and one language pair.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.
1 Introduction
As LLM translation quality improves, evaluation remains difficult because expert annotation is costly and automatic metrics lag on fine-grained assessment. This study examines evaluator design choices and reports robust findings that inform ORBIT, whose strongest configurations reach expert-level span detection and translation ranking.
1 Introduction
- The paper addresses limited understanding of evaluator design choices, which are often treated as implementation details despite their potential effects on evaluation behavior.Expert annotation requires substantial time and quality control, while automatic metrics lag on fine-grained assessment.Source: §1 · pp. 1–2
- The study treats evaluator design as a systematic problem spanning instructions, demonstrations, backbone selection, and multi-step workflows.Its focus is RATE span-level evaluation, including fine-grained error spans, severity, and separate Accuracy and Fluency judgments.Source: §1 · p. 2
- New expert RATE annotations cover WMT24 and WMT25 English–Russian translations, with multiple professional linguists independently annotating each translation.Span-level evaluation localizes translation failures and provides actionable feedback to developers and editors.Source: §1 · p. 2
- Instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, backbones have distinct profiles, and multi-model fusion outperforms single-call evaluation.The strongest configurations consistently exceed assessor-level agreement and match expert-level error span detection.Source: §1 · p. 2
- ORBIT-MC reaches expert-level Accuracy and Fluency ranking, exceeds assessor agreement on MQM ranking, and the strongest configuration costs roughly 20× less than expert annotation.The strongest configurations also achieve expert-level error span detection performance.Source: §1 · p. 2
2 Background and Related Work
Human MT evaluation protocols differ in their span labels and scoring procedures, while LLM evaluators range from score prediction to span-based and supervised methods. Prior work explores prompt variation, but systematic comparisons against multiple experts and across evaluator design dimensions remain limited.
2.1 Human MT Evaluation Protocols
- MQM and ESA both identify error spans; MQM adds categories and severity, whereas ESA uses spans to guide a 0–100 quality score.RATE adds five-point severity, a streamlined semantic/non-semantic taxonomy, separate Accuracy and Fluency scores, and an overall score derived from span severities.Source: §2.1 · p. 2
2.2 LLM-Based MT Evaluation
- LLM-based MT evaluators include score-based, prompt-based span, and supervised approaches, with some span evaluators using MQM-style instructions and demonstrations.Other work explores decomposition, validation, rubric prompting, and multi-agent specialist workflows.Source: §2.2 · pp. 2–3
- PrExMe studies more than 720 prompt templates, while this study jointly tests demonstrations, backbone choice, and multi-call workflows against expert span and severity judgments.Span-level results are sensitive to annotation quality and expertise, and comparisons against multiple professional experts remain limited.Source: §2.2 · p. 3
3 Experimental Setup
The experiments isolate evaluator design choices sequentially, use expert-annotated WMT English–Russian data and additional multilingual benchmarks, and compare LLM baselines under span-detection and ranking metrics.
3.1 Design Study Protocol
- Because evaluator choices form a large, interacting design space, the study varies dimensions sequentially rather than attempting exhaustive joint optimization.Its aim is to identify choices that consistently improve span-level evaluation and transfer across protocols and language pairs, not a universally optimal configuration.Source: §3.1 · p. 3
3.2 Dataset and Annotation Setup
- The evaluation set pairs WMT24 and WMT25 English–Russian segments with translations selected for system and quality diversity, and includes multiple expertise levels of annotation.Professional linguists provide three independent RATE annotations per segment; expert references support span-detection evaluation and pairwise ranking comparisons.Source: §3.2 · p. 3
- A separate collaboratively reconciled RATE pool supplies few-shot examples, while transfer tests cover MQM and ESA annotations across five additional WMT language directions.The calibration pool does not overlap with the evaluation set; transfer validation uses en-de and zh-en for MQM, and cs-de, en-it, and en-zh for ESA.Source: §3.2 · pp. 3–4
3.3 LLM Baselines
- The baselines span concise and detailed prompting, decomposition-based workflows, and supervised span evaluation.They include prompt-based systems and adaptations of MQM and ESA guidelines, alongside xCOMET-XXL, trained to predict scores and spans.Source: §3.3 · p. 4
3.4 Evaluation Metrics
- Evaluation measures span-level error detection and segment-level pairwise ranking across Accuracy, Fluency, and MQM dimensions.Span metrics include character-overlap-matched precision, recall, and F1 plus major-error diagnostics; RATE severities 2–3 map to MQM minor errors and 4–5 to major errors, while severity 1 is excluded.Source: §3.4 · p. 4
4 Systematic Study of LLM Evaluation Design
The study tests how instruction design, demonstrations, evaluator backbones, and workflows shape span-level and ranking-based MT evaluation. Across these choices, diverse-backbone fusion yields the strongest span-detection profile, while instruction and demonstration choices affect span evaluation and side-by-side evaluation favors ranking.
4.1 Instruction Design
- Instruction effects are non-monotonic: RATE-AEfull+ex achieves the strongest reported span F1 in the GPT-5.2 comparison, while removing examples makes RATE-AEfull fall behind shorter RATE-AEbrief.Detailed taxonomy definitions alone can overconstrain behavior; RATE-AEfull+ex uses inline examples as category-boundary disambiguators, and RATE variants also outperform protocol baselines on F1 for Claude Opus 4.6 and Gemini 3 Flash.Source: §4.1 · p. 5
4.2.1 Does Annotation Quality Matter?
- Demonstration quality matters more than mere presence: only collaboratively curated annotations consistently improve span recovery and major-error detection, while WMT annotations lower mean F1 below zero-shot.Ranking metrics improve more uniformly across demonstration sources than span-level behavior.Source: §4.2.1 · p. 5
4.2.2 How Many Examples?
- Increasing example count yields modest mean changes but reduces run-to-run instability from k=1; k≥2 results are broadly consistent, with a slight R4+ improvement trend.Coverage-based selection improves span quality over zero-shot; performance is broadly comparable across k, with smaller sets favoring recall and larger sets severity-aware precision.Source: §4.2.2 · pp. 5–6, §4.2.3 · p. 6, Figure 1 · p. 6
4.2.3 Coverage-Based Selection
- Coverage-based k=2 demonstrations paired with RATE-AEbrief define ORBIT-SC; RATE-AEbrief slightly surpasses RATE-AEfull+ex under this setup, while GEMBA prompts reduce span quality.The k=2 choice is the smallest coverage configuration with comparable validation performance, balancing quality, prompt complexity, and inference cost.Source: §4.2.3 · p. 6
4.3 Model Comparison
- No backbone dominates every dimension: GPT-5.4 leads span recovery and major-error detection, Claude Opus 4.6 leads severity-aware precision and direct-assessment ranking, and Gemini 3.1 Pro leads MQM ranking accuracy and overall precision.Grok-4 and DeepSeek-V3.2 over-predict errors at a substantial precision cost; DeepSeek-V3.2 flags more than twice as many errors per segment as HumanE.Source: Table 5 · p. 6, §4.3 · p. 7
4.4 Point-Wise vs. Side-by-Side Evaluation
- Side-by-side evaluation improves Accuracy and Fluency ranking for all three tested backbones and improves or matches MQM ranking, but does not improve span F1.Paired context shifts predictions toward higher precision and lower recall; point-wise evaluation suits individual systems, while side-by-side evaluation suits ranking a fixed set.Source: §4.4 · p. 7, Table 6 · p. 7
4.5.3 Model-diversity fusion
- ORBIT-MC’s fusion of Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro matches or exceeds expert-to-expert agreement on span F1 and major-error recovery, with strong high-severity precision.Claude Opus 4.6 serves as the fuser; the configuration achieves the strongest span-detection profile among tested configurations.Source: §4.5.3 · p. 7, §4.5.4 · p. 8, Table 7 · p. 8
4.6 Cost and Deployment Trade-offs
- At 10,000 segments, ORBIT-MC costs roughly $1,720: 4× less than the least expensive assessor setup and 28× less than professional expert RATE.Point-wise ORBIT-SC costs $20–$385 depending on backbone, a further 4–86× reduction relative to ORBIT-MC.Source: §4.6 · p. 8, Table 8 · p. 8
5 Multilingual Transfer
ORBIT configurations transfer across evaluated MQM and ESA benchmarks, though their relative performance depends on annotation density and metric. The expert-reconciled en-ru demonstration pool also transfers well to both MQM directions.
5 Multilingual Transfer
- Across MQM and ESA benchmarks, ORBIT-SC remains competitive, while ORBIT-MC improves major-error recovery over ORBIT-SC on en-de and is comparable on zh-en.These transfer results are reported for WMT23 MQM and WMT25 ESA benchmarks across five additional language directions.Source: §5 · p. 9
- Replacing the en-ru few-shot pool with direction-specific demonstrations provides no substantial gain over ORBIT-SC on either MQM direction.Compact RATE-style prompting remains competitive on MQM.Source: §5 · p. 9
- The en-ru demonstration pool matches direction-specific examples on en-de span F1, improves en-de R4+ and MQM, and improves all three reported metrics on zh-en.The authors attribute the advantage to the pool’s coverage of diverse error categories and severities and its collaborative reconciliation by multiple experts.Source: §5 · p. 9
- On ESA, ORBIT-MC is slightly below ORBIT-SC on en-it and en-zh for span F1, but exceeds Human2 on major-error recovery across all directions.The authors attribute ESA’s lower precision to its sparser annotation density and note that fusion expands prediction sets where additional spans are heavily penalized.Source: §5 · p. 9
6 Conclusion
The study examines how evaluator design choices shape LLM-based MT evaluation across protocols and language pairs, then instantiates its findings in ORBIT. Its strongest configurations achieve expert-level error-span detection and Accuracy/Fluency ranking, while ORBIT-MC exceeds assessor agreement on MQM ranking.
6 Conclusion
- The study examines instruction design, demonstrations, backbone choice, and multi-call workflows across RATE, MQM, ESA, and multiple language pairs.Curated, collaboratively reconciled demonstrations can match or outperform larger pools; model families show distinct evaluator profiles, and fusion yields the strongest overall results.Source: §6 · p. 9
- ORBIT’s strongest configurations achieve expert-level error-span detection and Accuracy/Fluency ranking, while ORBIT-MC exceeds assessor agreement on MQM ranking.ORBIT comprises single- and multi-call evaluators.Source: §6 · p. 9
Limitations
The study’s main design work uses English–Russian, and its findings have additional boundaries involving model access, span matching, severity calibration, collaboration evidence, and expert agreement.
Limitations
- The main design study uses English–Russian, with validation on five additional directions; low-resource pairs and typologically distant targets remain outside demonstrated coverage.Findings on instruction language and few-shot example selection may require adaptation for typologically distant target languages.Source: Limitations · p. 10
- The strongest configurations depend on closed-source API models, limiting reproducibility under silent updates and longevity if model snapshots are deprecated.The authors say cross-model comparisons show that qualitative conclusions hold across model families, mitigating the reproducibility concern.Source: Limitations · p. 10
- Span-level F1 depends on the matching formulation: MP with τ=1 is primary, while MPP is more sensitive to span-boundary agreement.The primary choice reflects the focus on identifying annotated error locations; MPP and LLM-based reconciliation are complementary sensitivity analyses.Source: Limitations · p. 10
- Severity assignment is distinct from error detection, and downstream uses such as quality thresholds or automatic gating require additional calibration.The main paper prioritizes detection-oriented R4+/P4+ metrics; severity calibration is not the focus of this work.Source: Limitations · p. 10
- The exploratory collaboration study measures annotation revision, not ground-truth correctness, and covers only ORBIT-MC, a limited annotator group, and one language pair.Whether observed annotation shifts improve quality, or generalize across configurations and annotator pools, remains to be studied.Source: Limitations · p. 10
- Expert agreement is a reference point, not a hard ceiling, and ranking differences within that agreement band may not reflect meaningful quality differences.The authors note that translation quality assessment remains partly subjective even among trained experts.Source: Limitations · p. 10
Ethics Statement
The ethics statement describes annotator compensation and consent, procedures for the collaborative study, feedback, and use of closed-source models.
Ethics Statement
- RATE annotators were paid at standard commercial translation rates for their region, with experts earning approximately twice assessors’ hourly wage.The higher expert rate reflected the qualifications required.Source: Ethics Statement · p. 10
- Annotators were informed how their annotations would be used; no personal data was collected, and source segments were screened for potentially disturbing material.In the collaborative study, annotators were told they would review LLM-generated reports and that pre- and post-hint annotations would be compared; no identifying information was reported.Source: Ethics Statement · p. 10
- Most annotators reported a positive experience and found the instructions clear.Source: Ethics Statement · p. 10
- The strongest configurations rely on closed-source LLMs accessed through commercial APIs.The paper discusses reproducibility and longevity implications of this choice in its Limitations section.Source: Ethics Statement · p. 11
A Dataset Statistics
The study uses a challenging English–Russian dataset with expert RATE annotations, then documents its artifacts and annotator pool.
A Dataset Statistics
- The 600-pair dataset combines 300 English source segments with two system translations each, selected where candidate translations disagree.The few-shot pool uses separate sources and does not overlap with the evaluation set.Source: A Dataset Statistics · p. 12
- Across two combined expert annotations, translations average 10.64 error spans, with Fluency errors most frequent at 6.06 spans per segment.Mistranslation averages 2.34 spans; other RATE categories each average below one, motivating coverage-based few-shot selection.Source: A Dataset Statistics · p. 13
B Artifact Use and Licenses
- Existing artifacts are used for research and non-commercial evaluation, and the expert-annotated dataset, prompts, and paper artifacts are available.The cited passages describe research-use terms for WMT data and licenses for other artifacts.Source: B Artifact Use and Licenses · pp. 13–14
C Annotator Pool
- The pool includes 35 professional expert translators and 82 assessors, with three experts independently annotating each segment.Experts had more extensive selection and training; assessor annotation also used implicit side-by-side and point-wise setups.Source: C Annotator Pool · p. 14
D Methods Not Included in Main Comparison
The paper limits its primary comparison to controlled evaluator designs and examines how span-matching choices affect measured agreement.
D Methods Not Included in Main Comparison
- The comparison excludes methods with substantially different orchestration or multi-stage workflows, aiming instead to study evaluator design under controlled conditions.The authors draw on M-MAD’s decomposition idea in multi-call ablations but do not reproduce its full system.Source: D Methods Not Included in Main… · p. 14
E.1 Formal Definitions
- The primary MP with τ=1 formulation matches predicted and gold spans when they overlap by at least one character, while MPP assigns credit proportional to overlap.The LLM reconciliation proxy asks GPT-5.4 to match annotations describing the same error despite boundary differences, and is treated as exploratory rather than human adjudication.Source: §E.1 · p. 15
E.3 Boundary Sensitivity under MPP
- Under MPP, HumanE drops 4.9 recall points and 5.7 precision points; ORBIT-SC recall changes 1.6–2.2 points while precision falls 14.3–19.5 points.The authors attribute the precision asymmetry to partial credit when predicted spans extend beyond gold boundaries; MPP can under-credit error identity across differing span granularities.Source: §E.3 · p. 15
E.4 LLM Reconciliation Proxy
An LLM reconciliation proxy offers a complementary span-identity comparison, while the paper emphasizes that exact system ordering depends on how a span match is defined.
E.4 LLM Reconciliation Proxy
- The reconciliation proxy places HumanE at F1=53.1, ORBIT-SC at 51.7–54.1, and ORBIT-MC at 56.4.These results support the main error-identity comparison, while exact human–model ordering remains sensitive to span boundaries.Source: §E.4 · pp. 15–16
- Because GPT-5.4 serves as both reconciler and evaluated backbone, the proxy may share biases with the systems it judges.The authors therefore treat it as complementary evidence, not the primary meta-evaluation method.Source: §E.4 · p. 16
E.5 Methodological Takeaway
- The three formulations support ORBIT as a strong evaluator, but exact rankings and human–model comparisons depend on the operational definition of a span match.The study retains MP with τ=1 as primary because its target is whether the evaluator identifies the annotated error location.Source: §E.5 · p. 16
F Model Comparison: Extended Results
Extended results separate error detection from severity calibration, replicate design patterns across backbones, and explore how ORBIT-MC reports affect expert annotation.
F Model Comparison: Extended Results
- HumanE drops 11 points (65.3 → 54.0) on severity-calibrated recall; GPT-5.4 drops 13.4 points (74.7 → 61.3), showing under-rating of major errors across humans and LLMs.The paper uses detection-oriented R4+/P4+ as primary metrics and reports severity-calibrated variants as a complementary diagnostic.Source: F Model Comparison: Extended Results · p. 16
- DeepSeek-V3.2 predicts 6.12 major errors per segment—roughly six times the expert rate—and has P4+ = 15.3, while Kimi-K2 under-predicts and has R = 22.0.Span density clarifies recall–precision patterns: precision-oriented backbones predict at or below the expert non-trivial rate, while recall-oriented backbones predict above it.Source: F Model Comparison: Extended Results · p. 16
G Cross-Backbone Design Replication
- On Gemini, ORBIT-SC is strongest on span F1 and R4+; on Claude, it has the strongest R4+ and is within 0.1 F1 of the strongest configuration.RATE-based variants outperform protocol baselines on F1 for both backbones, while individual instruction effects remain backbone-dependent.Source: G Cross-Backbone Design Replication · p. 17
H Human–LLM Collaborative Annotation
- In the held-out 600-segment study, ORBIT-MC reports increase annotation density and shift annotators toward lower overall quality scores.Annotators retain fewer spans than the raw reports contain, and the exploratory study measures revision rather than ground-truth correctness.Source: H Human–LLM Collaborative Annotation · p. 17, Table 17 · p. 17