Source-linked AI summary
FACET at WMT 2026 Automated Translation Quality Evaluation Task
Ahrii Kim, Chanjun Park, Seong-heum Kim
TL;DR
Translation quality judgments need different evidence, but evaluators often combine them in a single context. FACET separates them into reference-free passes with tailored inputs; post-edited human translation ranks first, while Consistency changes about a tenth of segment scores without substantially changing rankings.
Problem
Translation judgments require different evidence: meaning preservation needs source and target, while fluency and consistency are target properties at different scopes.
Source: §1 · p. 1Method
FACET prompts one fixed model in separate Fluency, Accuracy, and Consistency passes, gives each the context its judgment requires, and merges predictions without trained components.
Source: §1 · p. 1, §3.1 · p. 2, §6 · p. 6Results
Takeaways & Limitations
The paired submissions expose the limited effect of document-level Consistency on rankings under segment-level scoring.
Source: §6 · p. 7Current models have limited reliability tracking recurring items across long inputs, and document-level Consistency signals mismatch with segment-level scoring.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Different error types in machine translation require different evidence. Whether meaning is preserved can be judged only against the source, while whether the target is well-formed, or whether it names one entity consistently, can be judged from the target alone. We present FACET, our reference-free submission to the WMT26 Automated Translation Quality Evaluation Task, which decomposes evaluation into Fluency, Accuracy, and Consistency passes and gives each pass only the context its error type requires. A single fixed model is prompted three times, and the merged error spans yield the three task outputs, error spans, quality scores, and error-free labels, with no trained components. We also submit FACET-C, which omits the Consistency pass. Without gold labels, we characterize the predictions of FACET. Its system rankings place post-edited human translation first, and the Consistency pass changes about a tenth of segment scores while leaving the ranking nearly unchanged.
1 Introduction
Translation quality involves judgments that require different evidence: meaning depends on source and target, while fluency and document-wide consistency are target properties. FACET separates these judgments into passes with tailored inputs and merges their predictions into shared-task outputs.
1 Introduction
- Meaning preservation requires source and target, whereas well-formedness and recurring-item consistency are target properties requiring segment and document context, respectively.A single judgment context may not match the evidence each error type needs.Source: §1 · p. 1
- FACET follows MQM dimensions, promoting Consistency from a Fluency subtype to its own pass because it cannot be judged within one segment.Terminology, style, and locale conventions are treated as target-only Fluency judgments.Source: §1 · p. 1
- Accuracy reads source and target, while Fluency reads the target segment and Consistency reads the target document, limiting overlap by judging internal consistency rather than correctness.Consistency focuses on recurring names, domain terms, and fixed expressions.Source: §1 · p. 1
- A single fixed model, with no trained or fine-tuned components, produces merged predictions for error spans, quality scoring, and error-free detection; FACET−C omits Consistency to test its contribution.FACET is reference-free and treats document context as an input in its own right.Source: §1 · pp. 1–2
2 Related Work
Prior prompted translation evaluators generally use one prompt per segment, while quality-estimation metrics include trained systems with finer-grained outputs. FACET instead separates evaluation into passes that receive context suited to each error type.
2 Related Work
- Earlier prompted systems judge a segment with one prompt; FACET separates Fluency, Accuracy, and Consistency into distinct passes with tailored context.The prior systems include direct scoring, MQM-style annotation, and rubric-based prompting.Source: §2 · p. 2, Figure 1 · p. 2
- FACET’s three-pass pipeline gives source-and-target input to Accuracy and target-only input to Fluency and Consistency before merging predictions for segment scoring.FACET−C removes the Consistency pass.Source: Figure 1 · p. 2
- Learned quality-estimation metrics can provide sentence- and word-level scores or error spans, whereas FACET obtains spans, scores, and error-free decisions through prompting without trained components.The cited learned metrics are trained on human quality annotations.Source: §2 · p. 2
3 System Description
FACET runs independent error-detection passes over inputs and scopes suited to each judgment, then merges their predictions for scoring and task outputs. Its shared severity scale and filtering rules determine which errors contribute.
3 System Description
- FACET runs three independent prompting passes that return error spans, which are merged afterward; each pass’s input determines its operating scope.A span includes a quoted region, erroneous substring, severity, category, and short reason.Source: §3 · p. 2
3.1 Asymmetric Input
- Accuracy uses source and target, while Fluency and Consistency use target only; the first two operate on segments and Consistency compares the whole document.The source is withheld from target-only judgments to avoid importing meaning concerns into them.Source: §3.1 · pp. 2–3
3.2 Input Construction
- Sentence re-segmentation preserves original character coordinates, and document-grouped segments are ordered to reconstruct context for the Consistency pass.Paragraph-sized shared-task units have line breaks marking subtitle-cue rather than sentence boundaries.Source: §3.2 · p. 3
3.3 Fluency Pass
- Fluency proofreads the target segment for language-form errors, including spelling, grammar, agreement, word order, punctuation, orthography, register, and terminology.The source is withheld because Fluency concerns target-language norms.Source: §3.3 · p. 3
3.4 Accuracy Pass
- Accuracy compares source and target for meaning errors; omissions and defective-source errors are recorded on the source, while unsupported hallucinated content receives the highest severity.Source: §3.4 · p. 3
3.5 Consistency Pass
- Consistency tracks recurring entities, terms, and expressions, checking divergent forms for referent changes and agreement; it marks every occurrence of the form that does not fit.A single form is left to Accuracy, while shortened or pronominal references are checked for gender and number agreement where marked by the target language.Source: §3.5 · p. 3
3.6 Severity and Scoring
- The shared four-point severity scale maps scores 1–2 to Minor and 3–4 to Major; scale-1 Consistency errors are discarded before scoring.Scale-1 Consistency flags can reflect ordinary inflection in morphologically rich languages, while scale-1 Fluency and Accuracy errors are kept.Source: §3.6 · p. 3
3.7 Deriving the Task Outputs
- Merged surviving errors contribute to the negated sum of their severities; score 0 means no surviving error, and also labels a segment error-free.Non-contiguous Consistency errors may lack a single offset and are omitted from span output.Source: §3.6 · p. 4, §3.7 · p. 4
4 Experimental Setup
FACET is evaluated across 21 language pairs and multiple domains using a fixed, prompted model, with speech-specific instructions and a two-pass ablation. Gold annotations and the official leaderboard were unavailable, so correlation with gold labels is not reported.
Model
- FACET uses one fixed GPT-4.1-MINI model with deterministic prompting and no fine-tuning; requests are issued per segment for Fluency and Accuracy and per document for Consistency.The model is accessed through the OpenAI API at temperature 0.Source: §4 · p. 4
Data
- The WMT26 test set spans 21 language pairs and multiple domains, and human annotations and the official leaderboard were unavailable for correlation analysis.FACET scores each system output independently, producing several hundred thousand system-segment pairs per pass.Source: §4 · p. 4
Speech transcripts
- For speech-based items, Fluency and Accuracy are instructed to ignore fillers, hesitations, and repeated interjections; Consistency is unchanged.These words are treated as transcription artifacts rather than translation errors.Source: §4 · p. 4
Submissions
- FACET uses all three passes, while FACET−C omits Consistency to isolate that pass’s contribution under segment-level scoring.FACET uses the scale-1 filtering of Consistency errors described in Section 3.Source: §4 · p. 4
5 Submission Analysis
FACET ranks post-edited human translation first, while adding Consistency changes some segment scores but leaves system order nearly unchanged. The examples illustrate both cross-segment inconsistencies and errors that the passes miss.
System ranking
- Strong general-purpose systems rank consistently near the top, while small open models rank consistently near the bottom.The reported examples are Wayfinder, Lumen, and GPT 5.5 near the top, and GLM 4-9B, GPT OSS 20B, and Qwen 3.5-9b near the bottom.Source: §5 · p. 4
- 1.3 is Human (postediting)’s best mean rank; Human (from scratch) has a mean rank of 13.3.Human (postediting) places first or second in every pair it enters.Source: §5 · p. 4, Table 1 · p. 5
Effect of the consistency pass
- 10.1% of items have a different score under FACET than FACET−C, but no system moves by more than 0.2 in mean rank or swaps order.Only 0.7% of segments receive a different error-free label.Source: §5 · p. 5
- The Consistency pass recovers relatively few errors and sometimes cannot localize flagged spans, as the model struggles to track recurring items across long documents.The authors attribute the weak effect to current-model long-context reliability, not to document-level Consistency being unimportant.Source: §5 · p. 5
A qualitative example
- The facility’s Bürstenfabrik and Schleiferei renderings are locally acceptable but inconsistent across segments; Accuracy also misses that both mistranslate pilníkárna.Fluency and Accuracy flag within-segment language and meaning errors, while Consistency catches the cross-segment divergence.Source: §5 · pp. 5–6, Figure 2 · p. 6
6 Conclusion
FACET decomposes reference-free translation evaluation into passes matched to the evidence each error type requires. Its predictions rank post-edited human translation first, while the document-level Consistency pass changes segment scores but barely shifts system rankings.
6 Conclusion
- Accuracy uses source and target together, while Fluency and Consistency use target-only evidence at segment and document scope, respectively.A single fixed model is prompted three times, and its predictions are merged and scored without trained components.Source: §6 · p. 6
- Post-edited human translation ranks first, while Consistency changes a tenth of segment scores but leaves system rankings almost unchanged.The authors attribute the limited ranking effect to current models' limited reliability tracking recurring items across long inputs and the mismatch between document-level signals and segment-level scoring.Source: §6 · pp. 6–7
- Without gold labels, the paper characterizes FACET's predictions on their own terms, leaving open whether translation-from-scratch rankings reflect quality or source-structure preference.The authors say human judgments are needed to distinguish those explanations; stronger long-context models or direct credit for document-level Consistency are proposed to address identified limits.Source: §6 · pp. 6–7
A Prompts
The appendix specifies pass-specific inputs and prompting rules, the resulting error records, and a severity rubric shared across the three prompts.
A Prompts
- The prompts receive a target segment for Fluency, source and target segments for Accuracy, and the full target document for Consistency.A common severity rubric is appended to all three prompts.Source: A Prompts · p. 9
A.1 Fluency Pass
- Fluency checks target-language writing, flags only certain errors rather than preferences, and returns minimal error spans with a reason and category.The prompt excludes meaning judgments and asks for separate errors, including when several occur close together.Source: §A.1 · p. 9
A.2 Accuracy Pass
- Accuracy compares source meaning with the target and flags definite meaning errors, while allowing meaning-preserving paraphrase, reordering, and idiom.Its categories include mistranslation, wrong term, addition, hallucination, omission, untranslated text, and source issue.Source: §A.2 · pp. 9–10
- Accuracy output records whether the error is on the source or target side, with a locating quote, minimal span, reason, and category.The prompt requests an empty error list when the translation is accurate.Source: §A.2 · p. 10
A.3 Consistency Pass
- Consistency checks recurring named entities, domain terms, and fixed expressions across the document for differing renderings or mismatched reference agreement.It excludes single-occurrence items and questions of whether a rendering is correct, which require the source.Source: §A.3 · p. 10
- The Consistency output tracks recurring items and their renderings, and reports an error only when a referent has genuinely different forms.A single rendering cannot be an error; the tracked list serves as an audit trail.Source: §A.3 · pp. 10–11
A.4 Shared Severity Instruction
- All three prompts assign each error severity from 1 to 4, with the Minor/Major cut point applied during post-processing.The rubric ranges from slight wording changes without impact to substantial distortion that may mislead.Source: §A.4 · p. 11