Finding Replicable Human Evaluations via Stable Ranking Probability

1 Introduction

• Rater behavior and finite item allocations can change system rankings, making it difficult to attribute ranking changes to changes in the systems themselves.The study asks how items should be grouped, workload distributed, ratings and items counted, and scores normalized.Source: §1 · p. 1
• Using MT and MQM as a case study, the paper proposes recommendations for NLG ranking evaluations and a stability-based meta-evaluation metric and framework.MQM experts identify translation error spans that are converted into numeric scores.Source: §1 · p. 1
• The authors analyze two MQM datasets and publicly release nearly 140,000 segment ratings from multi-segment translations rated by professional translators.The release is described as the largest publicly available dataset of this kind.Source: §1 · p. 1

2 Terminology

• A segment is one or occasionally multiple sentences, while a document is a sequence of input segments.Source: §2 · p. 2
• An item is one MT system’s output on a single document, and a bucket is a subset of items evaluated by the same raters.Source: §2 · p. 2

3 Research Questions and Findings

• The authors recommend pseudo-side-by-side grouping, assigning all system outputs from a given input to the same rater.They report that this grouping improves stability over no grouping.Source: §3 · p. 2
• The authors recommend equal workload shares for raters, while noting that rater and document noise can affect this recommendation.They recommend clear annotation guidelines and rater training to limit such noise.Source: §3 · p. 2
• The authors weakly recommend rater-wise Z-score normalization because its largest gains occur in settings that conflict with other recommendations.Normalization is considered as a way to address noise from rater behavior, including after an evaluation has run.Source: §3 · p. 2
• The authors make no concrete recommendation for how many items to annotate because the reliability-cost trade-off is highly setting-dependent.They show results across a wide range of item counts to support practitioners’ judgments.Source: §3 · p. 2
• The authors recommend one rating per item to maximize the number of distinct items under a fixed budget, while noting that other MT evaluation settings may have different noise profiles.More ratings can reduce rater-behavior noise but require fewer distinct items, adding noise from item-level differences.Source: §3 · p. 2

4.1 Background on MQM

• MQM experts assign hierarchical error categories and severities, which are weighted into scores used to rank systems by average segment score.The framework does not use reference translations; each item pairs a document’s input segments with one system’s translation.Source: §4.1 · pp. 2–3

4.2 Dataset Descriptions

• The two WMT22 language-pair datasets use the same 181 source documents and contain nearly 140,000 segment ratings from professional translators.English-German data has seven buckets of 25–26 documents; English-Chinese has two buckets of 90–91 documents, with each bucket rated by three raters.Source: §4.2 · p. 3

4.3 Rater Behavior and Agreement

• Raters often have different segment-level MQM score distributions, with manual inspection suggesting differences in harshness or leniency rather than incorrect task performance.Figure 1 shows distributions for two English-Chinese raters who rated the same items.Source: §4.3 · p. 3, Figure 1 · p. 3
• English-Chinese raters agreed more on overall system rankings than English-German raters, but less on rankings of individual documents.Table 2 reports average Kendall’s Tau for document-level rankings and rankings over documents shared by each rater pair.Source: §4.3 · p. 3, Table 2 · p. 3

5 Quantifying Replicability

• Stability matters because unstable evaluation methods make it difficult to attribute ranking changes to real changes in quality.The paper frames stability as a way to examine replicability when defining a ground truth for human evaluation is difficult.Source: §5 · p. 4
• SRP is the proportion of ordered study pairs in which every significant system-pair difference in the first study is reflected in the second study’s rankings, ignoring significance there.It is computed over all ordered pairs of studies sharing a methodology.Source: §5 · p. 4
• SRP requires directional agreement for every significant system-pair difference, with no partial credit, focusing attention on harder-to-distinguish systems.The authors say average Kendall’s Tau can be dominated by easily distinguished pairs, producing a narrower range of values; SRP is also interpretable as a probability.Source: §5 · p. 4
• The authors test significance between systems’ mean MQM scores with 500-permutation tests, permuting segment labels by document and using α = 0.05.Segments from the same document are permuted as a group.Source: §5 · p. 4

6 Simulation Analysis

• The authors repeatedly sample ratings to simulate studies sharing an evaluation methodology, then assess replicability with SRP.Methodology choices are encoded as feature values that guide study simulation.Source: §6 · p. 4, §6.1 · p. 4

6.1.1 How should rateable items be grouped?

• Item Grouping constrains which items the same raters annotate: pSxS keeps outputs for one input document together, System-Balanced groups one output per system, and No Grouping imposes no constraints.System-balanced grouping controls input-agnostic rater differences; pSxS also controls input-dependent differences such as stylistic preferences within a domain.Source: §6.1.1 · pp. 4–5

6.1.2 How should workload be distributed?

• Load Balancing ranges from fully balanced workloads to entropy-targeted distributions; normalized entropy is 1 for equal workloads and 0 when one rater receives all items.The minimum entropy instantiated is 0.51 for English-German and 0.38 for English-Chinese because their data use different bucketing.Source: §6.1.2 · p. 5
• Moderate workload imbalance can arise when platforms impose no limits on items per rater, so Load Balancing lets the authors measure its potential effect on stability.The paper cites Amazon Mechanical Turk as an example of a platform whose default setup has no such restriction.Source: §6.1.2 · p. 5

6.1 Evaluation Methodology Features

• Normalization varies from no transformation to mean, error, or z-score normalization; these methods address rater harshness, while z-score normalization also adjusts for score range.Error normalization assumes expert MQM raters have high precision but variable recall over true errors, and does not address severity-use differences.Source: §6.1.2 · p. 5

6.1.4 How many items should be annotated?

• Number of Documents sets how many input documents a study includes, retaining all system outputs for each selected document.To avoid overdependence on one subset, the authors resample document subsets or simulate 50 studies before resampling.Source: §6.1.4 · pp. 5–6

6.2 Simulation Procedure

• The simulation procedure shuffles documents, optionally shuffles items according to Item Grouping, then assigns items to shuffled raters with workload balance controlled by Load Balancing.For 50 simulations per document set, SRP comparisons are restricted to studies using the same documents; the resampled-documents experiment does not use this restriction.Source: Figure 2 · p. 6, §6.2 · p. 6, §6.1.4 · p. 6

6.1.5 How many ratings per item are needed?

• Ratings per Item compares single-rated items with double-rated items, which receive two annotations from different raters, while holding the total rating budget fixed.The comparison uses single-rated studies of n documents and double-rated studies of n documents.Source: §6.1.5 · p. 6

7.1 How should rateable items be grouped?

• pSxS grouping provides a large stability boost over no grouping at all document counts, approximately doubling stability for English-German; system-balanced grouping also helps but generally underperforms pSxS.In pSxS grouping, the same rater evaluates all system outputs for a given input.Source: §7.1 · p. 6, Figure 3 · p. 7

7.2 How should workload be distributed?

• Workload imbalance has opposite stability relationships across datasets: English-German favors fully balanced studies, while English-Chinese favors highly imbalanced studies.The authors attribute this contrast to English-Chinese raters agreeing more on overall system rankings despite less agreement on individual documents; they recommend limiting rater-behavior differences and noise, after which fully balanced studies yield higher stability.Source: §7.2 · p. 6, Figure 4 · p. 7

7.3 How should scores be normalized?

• In the recommended setting, Z-score-normalized studies are more stable, though gains are small at high English-German document counts and generally modest for English-Chinese.The gap is larger for smaller English-German studies.Source: §7.3 · pp. 6–7, Figure 5 · p. 7
• In highly imbalanced studies, Z-score normalization boosts stability but does not close the gap to fully balanced studies in English-German; in English-Chinese, it widens the gap above them.Normalizing fully balanced studies does not close their gap to highly imbalanced studies in English-Chinese, and other normalization methods have less consistent effects.Source: §7.3 · p. 7, Figure 6 · p. 8
• Without item grouping, Z-score normalization gives a significant stability boost, mean normalization a slightly smaller one, and error normalization little benefit; in English-Chinese, normalization can close the gap below pSxS grouping.These experiments indicate that normalization is especially important when the other recommendations are not followed.Source: §7.3 · p. 7, Figure 7 · p. 8

7.4 How many ratings per item are needed?

• Single-rated studies on n documents are consistently more stable than double-rated studies on n/2 documents, favoring more distinct items at a fixed annotation budget.Resampling documents makes the expected chance of shared ratings equal across the two conditions, allowing a fair comparison of noise from rater behavior and document-level translation differences.Source: Figure 8 · p. 8, §7.4 · p. 8

MQM evaluation studies

• Prior MQM studies predict individual system scores or simulate an MQM-like annotation process to estimate confidence intervals; in contrast, this work characterizes effects of different rater/item configurations.Saldías Fuentes et al. (2022) seek to reduce items annotated by one rater while minimizing error relative to full-test-set scores; Gladkoff et al. (2022) simulate annotation to estimate confidence intervals for various sample sizes.Source: §8 · p. 8

Other related work

• Popović (2021) studies rater agreement by error type, while Popović and Belz (2022) examine normalization of error counts and find raw counts most reliable; surveys by Howcroft et al. (2020) and Gehrmann et al. (2023) cover broader NLG assessment.Source: §8 · p. 9

Direct human scoring for MT

• For direct Likert-style MT scoring, Graham et al. (2015) concluded that at least 15 ratings per item are needed for good correlation with ground-truth scores; Knowles (2021) studied ranking stability and made recommendations consistent with pSxS.Other cited work examines statistical-test sample sizes and adequacy-focused rating and calibration schemes.Source: §8 · p. 9

MT evaluation datasets

• Prior MQM datasets include 25,000 biomedical-domain ratings from Zouhar et al. (2024) and approximately 100,000 WMT ratings from Freitag et al. (2021), compared with this work’s approximately 140,000 ratings.Source: §8 · p. 9

• The proposed framework evaluates NLG evaluation reliability, and the resulting recommendations are supported by analysis of MT data in two language pairs.Source: §9 · p. 9
• The authors publicly release their MQM annotations to support follow-up research by the community.Source: §9 · p. 9

Limitations

• The English-German and English-Chinese datasets differ in systems, raters, bucketing, and document lengths, so observed differences cannot be attributed to language alone.Source: Limitations · p. 10
• Trends may not generalize: the analysis leaves out factors such as larger rater pools, more than three ratings per item, and other domains or language pairs.Source: Limitations · p. 10
• Because the ratings come from expert annotators familiar with the task, the recommendations may be less applicable with non-expert raters.Source: Limitations · p. 10
• The analysis covers system ranking, not reliable measurement of a single system’s absolute quality; reducing ranking variance could also bias rankings away from the unknown true ranking.Source: Limitations · p. 10
• The recommendations are derived from MT and MQM, and may be less applicable to other tasks or evaluation frameworks highly dissimilar to MQM.Source: Limitations · p. 10

A.1 Dataset Bucketing

• Bucket assignments differ by dataset: English-German raters rotate across buckets, while English-Chinese buckets use three non-overlapping raters and contain 90–91 documents rather than 25–26.Source: Table 3 · p. 11, §A.1 · p. 11

A.3 Simulating Studies with Specified Properties

• Simulations use item grouping and load balancing to set study properties, with pSxS grouping outputs by input document and balancing assignments across raters.Fully balanced pSxS uses round-robin assignments; entropy-balanced assignment targets a rater-distribution entropy, while system-balanced grouping distributes each system’s outputs across raters.Source: §A.3 · p. 11
• In no-grouping simulations, assignments operate on items rather than documents; other properties are set by assigning rater pairs, subsampling documents, or applying normalization after assignment.Source: §A.3 · pp. 11–12

A.2 System Mean MQM Scores

• The appendix reports mean segment-level MQM scores for systems in both datasets; lower scores are better, and refB is a human reference translation.Source: §A.2 · p. 11, Table 4 · p. 12