MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators

1 Introduction

• MQM-APE improves on GEMBA-MQM while remaining orthogonal to training-dependent approaches and complementing translation-specific evaluators such as Tower.The framework is presented as training-free and produces high-quality error annotations and post-edited translations.Source: Table 1 · p. 1, §1 · p. 2
• The same LLM evaluates a translation, post-edits it based on each error, and verifies whether the edit improves quality, retaining impactful errors.The verifier’s filtered errors influence the final score, which is assessed based on error count.Source: §1 · p. 2, Figure 1 · p. 2
• MQM-APE surpasses GEMBA-MQM at both system and segment levels, with error spans that closely align with human annotations.The framework generalizes across a broad range of LLMs and both high- and low-resource languages.Source: §1 · p. 2
• APE translations exhibit superior overall quality compared with the original translations.The method filters out errors whose post-edits do not contribute to quality improvement.Source: §1 · p. 2
• The Quality Verifier aligns with metrics such as CometKiwiQE, which can replace it with comparable effects.The paper also reports acceptable costs against GEMBA-MQM and preservation of error distributions across severities and categories.Source: §1 · p. 2
• The paper presents a performance ranking of various backbone LLMs as translation evaluators, offering researchers guidance on trade-offs among reliability, interpretability, and inference cost.Source: §1 · p. 2

2 Preliminaries

• In the reference-free setting, translation quality is scored without providing human references.Translation evaluation typically uses the source, translation, and human references.Source: §2 · p. 2
• MQM categorizes translation errors as Critical, Major, or Minor, then weights error counts by severity to calculate the final score.The paper identifies human annotation as the gold standard for translation evaluation.Source: §2 · pp. 2–3
• Error analysis predicts error demonstrations to simulate human evaluations and support interpretable assessment; GEMBA-MQM is the baseline and main module.GEMBA-MQM uses 3-shot prompting strategies.Source: §2 · p. 3

3 Methodology

• The pipeline identifies translation errors, post-edits the translation for each error, and discards errors whose edits do not improve quality.The retained set is scored after verification; the same LLM performs the three roles without fine-tuning.Source: §3 · p. 3

3.1 Module 1: Error Analysis Evaluator

• The evaluator records each error’s span, MQM-aligned category, and severity; translations with no identified errors skip later steps.The categories include mistranslation, omission, and grammatical issues; severity ranges from Critical to Minor.Source: §3.1 · p. 3

3.2 Module 2: Automatic Post-Editor

• The post-editor uses the source, translation, and each identified error’s span and category to produce a corrected translation.The procedure generates one post-edited translation per identified error.Source: §3.2 · p. 3

3.3 Module 3: Pairwise Quality Verifier

• The verifier compares each post-edited translation with the original and retains errors only when their edits improve quality.Errors that do not contribute to quality improvement are discarded.Source: §3.3 · p. 3

3.4 Post-process: Error-Based Scoring

• The final score is a weighted sum of refined errors by severity, with a lower bound of −25, and the method also yields post-edited translations.The −25 lower bound prevents the score from becoming excessively negative when many errors are identified.Source: §3.4 · p. 3

4.1 Test Dataset

• WMT22 covers 106,758 segments from 54 systems across three language pairs and four domains, while IndicMT adds 1,000 English-to-Indian-language segments.IndicMT includes Assamese, Maithili, Kannada, and Punjabi and is used to assess low-resource generalizability.Source: §4.1 · p. 4

4.2 Large Language Models

• The study evaluates eight open-source LLMs spanning general-purpose and translation-specific models, including Llama, Mixtral, Qwen, and Tower.Closed-source GPT-series models are excluded because updates can affect performance and reproducibility.Source: §4.2 · p. 4

4.3 Prompts

• All LLMs receive consistent prompts: GEMBA-MQM elicits error annotations, APE edits translations based on error spans, and the verifier compares edited and original translations twice.The two verifier comparisons are used to mitigate positional bias.Source: §4.3 · p. 4

4.4 Meta Evaluation

• Reliability is measured with system- and segment-level pairwise accuracy, while span and major precision assess error-span overlap; PERM-BOTH tests significance using 1,000 resampling runs at p = 0.05.IndicMT has no system-level scores because it provides no system-level information.Source: §4.4 · pp. 4–5

4.5 Alignment with Human Judgments

• The human-alignment analysis compares metric-performance differences against thresholds for CometKiwiQE and BLEURT20; CometKiwiQE requires at least 1.18 score improvement for 95% confidence.Detailed translation results are reported in Table 5.Source: Table 2 · p. 5, §4.5 · p. 5

5.1 Performance of MQM-APE

• MQM-APE improves GEMBA-MQM’s system- and segment-level reliability across tested LLMs in WMT and low-resource IndicMT, with Mixtral-8x22b-inst unchanged only at system level.The reported reliability gains hold across the three WMT language pairs and four low-resource Indian languages.Source: §5.1 · pp. 5–6, Table 3 · p. 6, Table 4 · p. 6
• Error-span precision improves across all tested LLMs, while major-error precision improves for seven of eight.The study reports better predicted-span quality compared with GEMBA-MQM and human annotations.Source: §5.1 · p. 5, Table 3 · p. 6
• MQM-APE also improves system- and segment-level performance and error-span quality for the translation-specific evaluator Tower.Its effect on major-error precision is smaller for Tower-13b-inst, possibly because it sometimes corrects multiple major errors simultaneously.Source: §5.1 · p. 5

5.2 Automatic Post-Editor

• All tested LLMs produce better post-edited translations than originals on CometKiwiQE22 and BLEURT20; seven of eight exceed 95% estimated accuracy on metric deltas.The pairwise segment win-loss ratio also exceeds 1 for every tested LLM.Source: §5.2 · p. 6, Table 5 · p. 7

5.3 Pairwise Quality Verifier

• Most LLM verifiers achieve over 90% recall against CometKiwiQE22; overall alignment is good, though lower precision suggests the verifier is somewhat more lenient.Tower-7b-inst is less effective as an APE and verifier, while Tower-13b-inst excels in both roles.Source: §5.3 · pp. 6–7, Table 6 · p. 7

6.1 MQM-APE Exhibits Superior Performance Compared to Random Error Filter

• Unlike MQM-APE, random error filtering consistently degrades performance against GEMBA-MQM, underscoring the role of APE in selecting errors.Figure 2 compares MQM-APE, random filtering, and GEMBA-MQM on segment-level performance.Source: §6.1 · p. 7, Figure 2 · p. 7

6.2 MQM-APE Introduces an Acceptable Inference Cost Compared to the Original

• The APE and quality-verifier modules generate approximately twice as many tokens as the evaluator, while using about half as many input tokens.Table 7 reports average input and generated tokens per segment for each module.Source: §6.2 · pp. 7–8, Table 7 · p. 7

6.3 MQM-APE Can Replace the Quality Verifier with Metrics for Comparable Performance

• Replacing the LLM verifier with CometKiwiQE22 yields comparable effects, with both approaches surpassing GEMBA-MQM for most LLMs.The comparison concerns segment-level performance.Source: §6.3 · p. 8, Figure 3 · p. 8

6.4 MQM-APE Preserves Error Distribution Across Severities and Categories

• MQM-APE retains a similar error distribution: discarded errors are mainly minor, with minimal changes to critical or major errors, and categories remain largely consistent.Figures 4 and 5 show severity-level counts and error-category distributions, respectively.Source: §6.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

6.5 Recommendation on LLM Selection when using LLM-based Evaluators

• Backbone choice involves trade-offs among reliability, interpretability, and inference cost; Mixtral-8x22b-inst may be most reliable but can produce redundant spans at higher cost.The analysis recommends choosing an LLM according to available computational resources and desired trade-offs.Source: §6.5 · p. 8

• Human-aligned metrics such as COMET-Kiwi and BLEURT arose in a context where traditional metrics such as BLEU were unreliable for high-quality MT systems.The passage situates the need for interpretable evaluation alongside growing demand for it.Source: §7 · p. 8
• LLM evaluation work ranges from direct prompting and error-analysis prompting to fine-tuned models for error-span prediction.GEMBA-MQM is described as a state-of-the-art error-analysis evaluator and the paper’s baseline and main module.Source: §7 · p. 9
• Unlike prior APE and self-correction work focused on translation improvement, this work uses APE within LLM evaluators to improve evaluation and error-span quality.The related-work discussion describes the approach as orthogonal to those works.Source: §7 · p. 9

• MQM-APE filters errors that do not contribute to quality improvement by incorporating APE into the evaluation framework.Source: §8 · p. 9
• The authors report performance enhancements and better error annotations across various LLMs and high- and low-resource language pairs.Source: §8 · p. 9
• Future work will examine whether collaboration among different LLMs or varied prompting strategies can improve predicted error quality.Source: §8 · p. 9

Limitations

• MQM-APE requires approximately twice the inference tokens of standard MQM-like error-analysis prompting.The authors describe this extra cost as valuable and acceptable given the framework’s reported benefits.Source: Limitations · p. 9
• MQM-APE does not align its error distribution with human evaluation, an issue the authors leave for future research.The authors propose further investigation rather than claiming the distribution is aligned.Source: Limitations · p. 9
• The experiments use one LLM for error analysis, APE, and quality verification, although real-world evaluations could use multiple collaborating LLMs.The single-model setup was chosen to support the main claim and ensure fair comparison across tasks.Source: Limitations · p. 9
• Invalid or rejected responses were rare in the tested models, but the authors recommend excluding models that frequently produce them.Their procedure regenerates an invalid response after slightly increasing temperature.Source: Limitations · p. 10

Ethics Statement

• The study says MQM-APE extracts error spans without inducing harmful content and uses public datasets and models without fine-tuning.The authors state that these choices reduce risks associated with harmful content and learning from user inputs.Source: Ethics Statement · p. 10

A Description of MQM

• MQM categorizes translation errors by type, and its human annotations are available from the WMT22 metrics shared tasks.The cited passages describe MQM as a human evaluation framework used as a gold standard in WMT metrics shared tasks.Source: A Description of MQM · p. 12

B Description of the Test Sets

• The test sets cover WMT22 En-De, En-Ru, and Zh-En across four domains, plus 1,000 MQM-annotated IndicMT segments in four Indian languages.The WMT22 portion includes 106,758 segments from 54 MT systems; IndicMT has 250 segments per language.Source: B Description of the Test Sets · p. 12

C Prompt Contexts

• The experiments use language- and model-agnostic prompt contexts: GEMBA-MQM for error analysis, zero-shot APE, and pairwise translation verification.The error-analysis prompt is three-shot; the post-editor and verifier are zero-shot.Source: C Prompt Contexts · pp. 12–13

D.1 Performance of WMT22; D.2 Performance of IndicMT

• Supplementary results compare MQM and MQM-APE: on WMT22, detailed results for each language pair cover system and segment levels (Table 11), while IndicMT results for each language appear in Table 12.Source: §D.1 · p. 13, §D.2 · p. 13

D.3 Performance of Error Quality

• The supplementary comparison measures predicted error quality using span precision (SP) and major error span precision (MP) for each language pair in Table 13, expanding the error-quality results shown in Table 3.Source: §D.3 · p. 13

E Detailed Analysis of Inference Cost

• Inference token generation varies across LLMs: Qwen1.5 models generate fewer tokens than others, while Mixtral generates significantly more.The authors suggest this may mean lower inference costs for Qwen1.5 and associate Mixtral’s higher output with stricter initial error detection.Source: E Detailed Analysis of Inference Cost · p. 13

F.1 Error Severity Distribution

• MQM-APE preserves a similar distribution of error severities compared with the original evaluator, while Mixtral produces more errors than other models.For Tower-7b-inst, most generated errors are discarded as unimpactful.Source: §F.1 · p. 13

F.2 Error Category Distribution

• Discarded errors are predominantly “style/awkward,” and MQM-APE discards many Mixtral-8x22b-inst terminology errors, suggesting reduced bias in those categories.The passages report these patterns across the generated and discarded error comparisons.Source: §F.2 · p. 14, Table 16 · p. 17, Table 17 · p. 18
• Fluency/spelling and inconsistency errors in human annotations are rarely identified by LLM evaluators.The authors identify these as long-tailed categories for future distribution-adjustment work.Source: §F.2 · p. 14, Table 18 · p. 18

G Positional Bias and Invalid Answers

• To reduce positional bias, the verifier compares both translation orders and gives half-weight to contrastive results when smoothing scores.The pairwise quality verifier runs twice with the translations swapped.Source: G Positional Bias and Invalid Answers · p. 14
• The evaluator uses three-shot prompting, while the post-editor and verifier use zero-shot prompts; invalid responses were not significant in the tested models.Invalid responses are regenerated with a slight temperature increase.Source: G Positional Bias and Invalid Answers · p. 14, Figure 6 · p. 16

F.2 Error Category Distribution

• MQM-APE preserves the distribution of error categories; 27% of errors identified by the LLM evaluator fall into the "style/awkward" category, while human MQM annotations focus more on "accuracy/mistranslation".Source: §F.2 · pp. 13–14