How to Select Datapoints for Efficient Human Evaluation of NLG Models?

1 Introduction

• Random selection can waste a limited evaluation budget by overlapping on items and omitting items that matter more to the model ranking.Human evaluation is the gold standard, while distinguishing high-quality models requires increasingly more expert annotations.Source: §1 · p. 1
• Output-based selection uses automated metrics and model outputs to choose items, including by metric-based difficulty, output diversity, or Item Response Theory.The target is a subset whose human model ranking resembles the ranking on the full test set.Source: §1 · p. 1
• Source-based selectors distill output-based methods into predictors that estimate item difficulty or output diversity from source texts alone.This addresses settings where model outputs are unavailable or obtaining them for the full dataset is computationally infeasible.Source: §1 · p. 1
• The work evaluates selection methods on machine translation and summarization and contributes the subset2evaluate package, trained models, and reproduction code.The listed contributions also include cost- and document-aware selection methods.Source: §1 · pp. 1–2
• 70% of human annotations matched the evaluation result from the full WMT test set, with the same model ranking as random sampling.The authors describe this as a non-trivial cost saving.Source: §1 · p. 2

2 Problem Statement

• The selection objective is to choose a subset Y of test items within the human-evaluation budget so its model ranking is as close as possible to the ranking on all X.Each item is an input text, and each model maps it to an NLG output.Source: §2 · p. 2
• The formulation assumes item costs and utilities add across a subset, although similar datapoints may provide no extra marginal utility.The authors report good empirical results under the utility assumption; preliminary diversity modeling did not improve performance.Source: §2 · p. 2
• Figure 1 depicts output-based selection: model outputs and automated metrics inform which items are human-evaluated for computing the final model ranking.The subset-selection formulation quantifies evaluation cost and usefulness separately.Source: Figure 1 · p. 2, §2 · p. 2
• Under two assumptions, selection becomes a 0-1 knapsack problem; with constant cost and positive item utilities, choosing highest-utility items until budget B is reached is optimal, while varying costs use approximate integer linear programming.The general problem is NP-complete.Source: §2 · p. 2

2.1 Subset selection evaluation

• Soft pairwise accuracy compares the model ranking on a selected subset with the full-set ranking, accounting for ranking statistical power.This meta-evaluation requires human scores.Source: §2.1 · p. 3

2.2 Source- vs. Output-based selection

• Source-based selection estimates item utility from input x alone, for cases where full-set outputs are too costly to obtain or participating models are not yet known.Output-based selection instead uses model outputs to estimate item utilities.Source: §2.2 · p. 3

3.1 Metrics moment

• Metric-moment selectors use low average metric scores as a proxy for item difficulty and across-model score variance to prioritize items that distinguish models.Items where all models produce very high- or very low-quality outputs contribute little to the final ranking.Source: §3.1 · p. 3

3.2 Metric consistency

• Metric consistency favors items whose metric-based model ranking predicts the ranking across the full test set.The ranking correlation can use Spearman, Kendall, or pairwise accuracy; the paper uses Spearman.Source: §3.2 · p. 3

3.3 Output diversity

• Output-diversity selection favors items whose model outputs have lower average text similarity.The paper primarily measures similarity with embeddings and also evaluates unigram overlap, chrF, and BLEU.Source: §3.3 · pp. 3–4

3.4 Item response theory

• The IRT selector fits a model to continuous metric scores and uses item discriminability and difficulty together as utility.It uses the product of discriminability and difficulty because those parameters may have different scales.Source: §3.4 · p. 4

4 Source-Based Selectors

• Source-based selectors choose items using only their inputs, adapting output-based selection for cases where model outputs are unavailable.The paper describes two adaptations: predicting item utility from the source text and using an artificial crowd of models.Source: §4 · p. 5, Figure 3 · p. 5

4.1 Item utility distillation

• Item-utility distillation trains a source-text model to predict output-based utility, producing estimators for metric moments, diversity, and IRT-based utility.The predictor encodes the input with a pretrained language model and uses a regression head trained with MSE loss.Source: §4.1 · p. 5
• The distilled utility method applies to novel items only when they are similar to items in its training data.Source: §4.1 · p. 5

4.2 Artificial crowd

• The artificial-crowd approach uses outputs from a subset of models to approximate output-based utilities when the full model set is unavailable.It assumes the subset M′ approximates the full set M in some capacity.Source: §4.2 · p. 5

5 Case Study 1: Machine Translation

• The WMT experiments use 33 campaigns overall and nine WMT 2023 language pairs, averaging results across languages unless otherwise specified.The campaigns include 31k source items and 395k translations; results focus on subsets from 5% to 50% of the original test set.Source: §5 · p. 6

5.1 Output-based selection for WMT

• Output-based diversity performs best, followed by IRT and metric-moment approaches, which only slightly exceed the random baseline confidence interval.The comparison uses soft pairwise accuracy; methods also consistently outperform random under alternative ranking meta-evaluations.Source: §5.1 · p. 6
• Balancing selected items across domains still leaves the methods performing much better than random selection.The table reports results averaged across languages and subset sizes.Source: §5.1 · p. 6, Table 1 · p. 7
• Document-level selection averages sentence-level utilities into document utilities and reproduces the item-level pattern of outperforming random selection.Document cost sums sentence-level evaluation costs, while document utility averages sentence-level utilities.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Accounting for human annotation cost in subset selection

• Cost-aware selection can achieve high soft pairwise accuracy under a limited budget, while difficult items tend to require more annotation time.Source length estimates evaluation time weakly (ρ=0.24), and item difficulty correlates positively with time (ρ=0.33).Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Source-based selection for WMT

• Source-based diversity and latent-IRT predictors perform best among source-based methods, though they fall short of their output-based versions.The predictors can be applied to unseen items without model outputs; source-based metric-moment estimates still outperform random, but not outside its confidence interval.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Subset selection cuts costs substantially

• 77% of the data suffices for diversity-based selection to achieve the same or better soft pairwise accuracy as random sampling on machine translation.The comparison averages across budgets in the reported cost-efficiency analysis.Source: §5.4 · p. 9, Table 5 · p. 9

5.5 Importance of a good automated metric

• String-matching metrics provide insufficient signal for robust subset selection, while BERTScore is sufficient and higher-quality automated metrics generally yield better subsets.Combining multiple automated metrics into one item utility does not improve results.Source: §5.5 · p. 9

• ∼70% of the human-evaluation budget suffices for metric consistency to match random selection, and ∼80% of the test set suffices for LLM evaluation.Most output-based methods outperform random selection on SummEval.Source: Figure 7 · p. 10, §6 · p. 10
• SummEval measures relevance, coherence, consistency, and fluency across 17 human-evaluated models and 100 items, using G-Eval as the automated evaluator.Selection is performed independently for each quality dimension before aggregating results.Source: §6 · p. 10
• Source-based selection is omitted because training data are unavailable for this small dataset.Source: §6 · p. 10

• This work targets fewer human evaluations while preserving the full-set result, such as the model ranking, rather than selecting items by comparison to a ground-truth answer.Source: §7 · p. 11
• Prior IRT selectors require ground-truth answers and binary outcomes, whereas NLG evaluation lacks one unique answer and typically uses continuous scores.Source: §7 · p. 11
• Many prior methods require some models to have been evaluated on the full item set, limiting their use when outputs are unavailable or the set is too large.Source: §7 · p. 11
• Active learning requires tightly controlled annotation, while utility-ranked items can be annotated asynchronously and stopped when the budget runs out.Source: §7 · pp. 11–12
• The authors identify cost-aware item selection as an unstudied dimension; coreset methods also lack full-set human scores and ranking-utility optimization.They note that coreset methods often target construction cost, unlike this setup.Source: §7 · p. 12

8 Conclusion and Key Takeaways

• Metric variance, consistency, and diversity outperform random selection for machine translation and summarization, while lowest-score difficulty prioritization can yield little improvement and costly items.Source: §8 · p. 12
• The subset2evaluate package includes pre-trained item-utility estimators for machine-translation subset selection.Source: §8 · p. 12
• Choose metric variance or consistency when reliable metrics and outputs are available; use diversity without a reliable metric, source-based diversity with historical data, or artificial-crowd variance with partial outputs.Source: §8 · p. 12
• Automated-metric bias can hide items that humans would judge faulty, especially when selection uses expected metric averages; similar risks may affect variance- or discrimination-based selection.The authors report that method performance corresponds directly to alignment between automated metrics and human judgments.Source: §8 · p. 12

A Other Methods

• Appendix methods are excluded from the main paper because of their assumptions or low performance, which limits their application.Source: A Other Methods · p. 18

• Brute-force selection samples budget-sized subsets and chooses the one with the highest utility, but evaluating utility against the full set requires unavailable human scores or a proxy metric.Source: §A.1 · p. 18
• Greedy selection starts from a small random subset and iteratively adds examples that most increase soft pairwise accuracy until the budget is reached.Brute force is the special case where the extension size equals the budget; greedy selection corresponds to beam search of size one.Source: §A.1 · p. 18
• A human-oracle variant replaces automated metric proxies with human scores to measure the limits imposed by imperfect proxies, but is not practical for real selection.Source: §A.1 · p. 18
• IRT variants test difficulty, discriminability, feasibility, and Fisher information, which is summed across model abilities because it is additive.Source: §A.1 · p. 18
• Clustering selects the item nearest each k-means cluster center to diversify source or output embeddings; the extended version weights items by cluster size.The extended method assumes nearby examples yield the same human scores.Source: §A.1 · pp. 18–19
• DiffUse clusters vectors of pairwise model-representation differences hierarchically to fit the selection budget.It uses MiniLM-12-v2 embeddings.Source: §A.1 · p. 19
• The appendix also explores weighted averages of utility rankings and selecting items with either high or low metric epistemic uncertainty.Source: §A.1 · p. 19

A.2 Results

• Greedily building a subset is more efficient and more promising than choosing the best subset from random generations, whose strong results depend largely on luck.The comparison concerns subsets directly optimized using human scores.Source: §A.2 · p. 19
• MetricCons performs much better than random and the previously discussed methods, while MetricAvg and MetricVar do not gain more from human scores than from automated-metric proxies.The best subset selected using an automated metric is not necessarily the best subset when judged by human scores.Source: §A.2 · p. 20
• Source-embedding k-means performs best among its variants and works as a source-based selector without model outputs, with performance comparable to DiffUse.Better-than-random performs poorly in this setup even when immediate feedback enables active sampling.Source: §A.2 · p. 20
• Alternative pairwise similarity metrics do not improve over embeddings, and combining methods with ranking averaging rarely exceeds individual-method performance.Sentinel metrics perform close to the output-based MetricAvg they imitate; high-uncertainty items seem preferable to low-uncertainty items.Source: §A.2 · p. 20

B.1 Meta-Evaluations

• Meta-evaluation measures how closely an outcome on subset Y matches the outcome on full set X, with soft pairwise accuracy comparing ranking correctness and significance.It uses paired-permutation-test p-values for each model pair; simple pairwise accuracy reduces these to 0/1 ranking outcomes.Source: §B.1 · p. 20
• Ranking similarity can be measured by Pearson, Spearman, or Kendall correlation; top-1 match checks whether Y and X select the same best model, while average-score error measures absolute-score differences.Pearson correlation also reflects scale and is sensitive to outliers.Source: §B.1 · p. 20
• Cluster count measures how many statistically distinct model groups the evaluation produces, while top-1 cluster match compares the leading groups on Y and X.The described clustering groups systems that are not statistically distinguishable from the preceding cluster.Source: §B.1 · pp. 20–21

B.2 Results

• Up to +1 cluster over random with MetricCons is the largest reported gain; correlation-based improvements are consistent, except for model averages.An extra significance cluster allows more statistically justified claims in a shared-task context.Source: Table 9 · p. 21, §B.2 · p. 21

C Implementation details

• The item-response-theory model uses logistic regression with stochastic variational inference and normal priors for item difficulty, discriminability, and model ability.The implementation uses py-irt.Source: C Implementation details · p. 21
• PreCOMET distills item utility using an XLM-RoBERTa-based model trained on pre-2023 WMT sources, with utilities computed against human scores.The model is trained for five epochs with an effective batch size of 128.Source: C Implementation details · p. 21
• Output diversity is computed from inner products of multilingual MiniLM-12-v2 embeddings.The embeddings are used in Equation (7).Source: C Implementation details · p. 21
• Table 10 reports soft pairwise accuracy averaged over specified data-proportion ranges for WMT and SummEval; random results average 100 runs.The caption identifies best numbers within each row's correlation or cluster evaluation category.Source: Table 10 · p. 22