Source-linked AI summary
RankME: Reliable Human Ratings for Natural Language Generation
Jekaterina Novikova, Ondřej Dušek, Verena Rieser
TL;DR
Human ratings are central to NLG evaluation, yet their reliability and ability to distinguish criteria remain concerns. The paper introduces RankME, combining continuous magnitude estimation with relative ranking, and reports improved agreement and discrimination, including higher naturalness ICC than plain ME.
Problem
Human judgment is a primary NLG evaluation criterion, but limited work has addressed the reliability of subjective ratings and judges can conflate distinct evaluation criteria.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Separately eliciting criteria can reduce their correlations, and missed-information ratings can yield a different system ranking from added-information ratings.
Source: §3 · p. 3The authors identify testing on a wider system range and transfer to related tasks such as open-domain dialogue as future work.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.
1 Introduction
Human judgments are central to NLG evaluation, but their reliability and ability to distinguish criteria remain concerns. The paper compares experimental designs and introduces RankME, reporting stronger human-rating performance with relative assessment and combined magnitude estimation.
1 Introduction
- The study compares experimental designs for eliciting intrinsic judgments while evaluating multiple NLG systems.Source: §1 · p. 1
- The paper investigates distinguishing criteria such as naturalness and informativeness, alongside inconsistency in human ratings.Source: §1 · p. 1
- Relative assessment produces more consistent and discriminative human ratings than direct assessment.Source: §1 · p. 2
- RankME combines continuous scales, magnitude estimation, and relative assessment, and outperforms traditional evaluation methods.Source: §1 · p. 2
2 Experimental Setup
The study evaluates three NLG systems using crowd-worker ratings collected under several experimental designs. It compares Likert scales, plain magnitude estimation, and RankME, which asks raters to rank target sentences relative to one another.
2 Experimental Setup
- The evaluation uses Sheffield NLP, Slug2Slug, and baseline TGen, chosen to compare systems with differing automatic-metric performance.Source: §2 · p. 2
- The systems share attention-based seq2seq architectures, with differing encoders, ensembles, and semantic-validity rerankers.Source: §2 · p. 2
- For each system, the researchers collect ratings for its first 100 outputs from three independent crowd workers.Source: §2 · p. 2
- The three collection methods are 6-point Likert scales, plain magnitude estimation, and RankME.Source: §2 · p. 2
- Plain ME asks raters to judge a target relative to a fixed-score reference, while RankME asks them to rank all target sentences.Source: §2 · p. 2
3 Judgements of Multiple Criteria
The paper tests whether assessing naturalness, informativeness, and overall quality together or separately affects their correlation. It also distinguishes added from missed information, finding that the two assessments can produce different system rankings.
3 Judgements of Multiple Criteria
- The study compares evaluating informativeness, naturalness, and quality together with assessing each criterion in an individual task.Source: §3 · pp. 2–3
- Separate collection reduces correlations between naturalness and quality and between naturalness and informativeness to very low levels, especially with ME methods.Source: §3 · p. 3
- Assessing added versus missed information yields significantly different distributions (p < 0.01), and the two assessments produce different overall system rankings.Source: §3 · p. 3
4 Consistency and Use of Scales
The study compares inter-rater reliability across discrete Likert scales and continuous ME scales. ME improves ICC for naturalness and quality, with RankME reaching the highest reported ICC, while Likert scales already provide satisfactory agreement for informativeness.
4 Consistency and Use of Scales
- ME significantly increases ICC for naturalness and quality, especially when criteria are collected separately; Likert scales provide satisfactory agreement for informativeness.Source: §4 · p. 3
- For naturalness, RankME reaches an ICC of 0.42 versus plain ME's 0.27, and RankME has the highest ICC results overall.Source: §4 · p. 3
- Continuous scales implemented via ME significantly improve rating reliability, complementing prior findings that raters prefer continuous scales.Source: §4 · p. 3
5 Ranking vs Direct Assessment
The section compares direct assessment with relative ranking, finding that RankME yields a clear overall quality ranking and more consistent ratings under separate collection. For many systems, RankME comparisons combined with TrueSkill reproduce RankME’s rankings while requiring fewer direct comparisons.
5 Ranking vs Direct Assessment
- ME methods produce more distinctive overall system rankings than Likert, but no method clearly ranks naturalness.Plain ME gives a clearer informativeness ranking than RankME.Source: §5 · p. 3, Table 4 · p. 4
- RankME alone provides a clear ranking of overall utterance quality under Setup 2.Different ranks in Table 4 are statistically significant at p < 0.05.Source: §5 · p. 3, Table 4 · p. 4
- With Setup 2, relative ranking produces more consistent ratings than direct assessment.Likert and plain ME are direct assessment methods; RankME is relative-ranking-based.Source: §5 · pp. 3–4, Table 3 · p. 4
5.1 Relative comparisons of many outputs
- Relative rank-based assessment requires data that grows quadratically with the number of systems, a problem for larger evaluations such as shared tasks.Source: §5.1 · p. 4
- TrueSkill applied to RankME comparisons produces exactly the same system rankings for all three criteria as RankME directly.It uses pairwise comparisons as win–loss–tie outcomes, reducing the number of direct system comparisons needed.Source: §5.1 · p. 4
6 Conclusion and Discussion
Experimental design significantly affects the reliability and outcomes of NLG human evaluations. RankME improves rater agreement and result discrimination, while TrueSkill offers a cost-effective way to rank systems.
6 Conclusion and Discussion
- RankME combines relative rankings with continuous-scale magnitude estimation, yielding better rater agreement and more discriminative results.The conclusion also reports that eliciting evaluation criteria separately can minimize correlations between them.Source: §6 · p. 4
- TrueSkill is a cost-effective alternative for producing overall relative rankings of multiple systems.Source: §6 · p. 4
- The authors are testing RankME on a wider range of systems and plan to study transfer to dialogue responses and additional evaluation methods.Planned methods include extrinsic task contributions.Source: §6 · pp. 4–5