Source-linked AI summary
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, Noah A. Smith
TL;DR
As NLG models become more fluent, it is unclear whether non-expert human evaluations can reliably detect machine-generated text or assess its quality. The paper studies this question across three domains and tests lightweight evaluator training, finding that evaluators generally could not distinguish GPT3 from human text and that training did not consistently improve accuracy.
Problem
Fluency improvements make surface-level evaluation less sufficient for detecting content-based errors in generated text.
Method
The study asks non-experts to distinguish human- and machine-generated text across stories, news articles, and recipes, then tests three lightweight evaluator-training methods.
Results
Evaluators could not distinguish GPT3- and human-authored text across three domains, and no training method significantly improved accuracy consistently across domains.
Takeaways & Limitations
Human evaluations of current NLG models should shift away from small-batch intrinsic judgments toward settings that better assess content and usefulness.
Takeaways & Limitations
Instruction-based training was prescriptive because researchers had to choose which text dimensions evaluators should assess.
Abstract
from arXiv · showhide
Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts' ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators' accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.
1 Introduction
Human evaluation is central to judging NLG quality, but improving model fluency makes surface-level judgments less reliable for detecting content-based errors. This study finds that non-experts could not distinguish GPT3- from human-authored text across domains, and lightweight training did not reliably solve the problem.
- Human evaluation remains necessary because automated methods do not adequately encode many aspects of human-quality output.
- Evaluators use holistic criteria such as quality, naturalness, and humanlikeness, while correlated dimensions can be conflated.
- Fluency-related qualities are easiest to assess in fast, small-batch evaluations, even though state-of-the-art model errors are often factual or context-based.
- Evaluators could not distinguish GPT3- and human-authored text across stories, news, and recipes, focusing mainly on grammar, spelling, and style.
- No training method significantly improved evaluators’ performance consistently across domains, although examples increased attention to content and broadened machine-generated judgments.
- The authors recommend moving beyond lightly trained, small-batch intrinsic evaluation toward frameworks emphasizing generated text’s usefulness in downstream settings.
2 How well can untrained evaluators identify machine-generated text?
Untrained evaluators assessed humanlikeness across stories, news, and recipes, but could not reliably distinguish GPT3-generated from human-authored text. Their judgments focused mainly on surface form rather than content, and their explanations were inconsistent.
- Study design: Evaluators rated five passages as definitely or possibly human- or machine-written using a four-point humanlikeness scale.Each participant evaluated one domain and model condition; the study covered stories, news articles, and recipes.
- Data: The study compared 50 human-authored texts with 50 texts generated by GPT3 or GPT2 in each of three domains.Texts were at least 100 words, and generated texts used three in-domain human-authored priming samples.
- Results: 49.9% accuracy for GPT3 versus human text was statistically indistinguishable from 50% random chance, whereas GPT2 accuracy reached 57.9%.GPT3 performance did not differ significantly from chance in any individual domain; story accuracy fell from 62% with GPT2 to 48% with GPT3.
- Results: Evaluators disproportionately labeled GPT3 texts as human-written, producing two thirds human guesses despite equal source frequencies.Confidence remained fairly constant across conditions even as GPT3 identification accuracy declined.
- Judgment rationale: Evaluators made twice as many form-related comments as content-related comments, with spelling, grammar, punctuation, and style dominating their explanations.Form accounted for 47% of labels and content for 25%; 45 of 150 comments mentioned spelling, grammar, or punctuation.
- Judgment rationale: Evaluators’ reasons often contradicted one another, and agreement was approximately zero across domains.Formality, spelling and grammar errors, and clarity were each used to justify both human and machine judgments.
3 Can we train evaluators to better identify machine-generated text?
The study tested three lightweight training methods—instructions, annotated examples, and paired human-machine examples—to improve machine-text identification. Training broadened evaluators’ consideration of machine-generated text, but gains were not consistent across domains and agreement remained low.
- Training design: The researchers tested instruction-based, example-based, and comparison-based training before repeating the GPT3 evaluation across stories, news, and recipes.Training required at most three additional human- and machine-generated samples, and trained participants were disjoint from the first study’s evaluators.
- Instruction-based training: Instruction training highlighted repetition and factuality while warning that grammar, spelling, and style could mislead evaluators.The selected dimensions were based on prior work and pilot-study comments.
- Example-based training: Examples training used three domain-specific practice rounds with feedback explaining which textual aspects suggested human or machine authorship.Evaluators received the correct answer and explanation after each guess.
- Comparison-based training: Comparison training paired human and machine texts generated from the same prompt, then revealed the machine-authored passage and explanation.It required three more passages than Examples training and cost US$2.25 per HIT.
- Results: The overall Examples improvement was driven mainly by stories, with mean-accuracy changes of 0.086 for stories, 0.019 for news, and 0.062 for recipes.The authors suggest examples may have overridden a preconception that machines cannot generate creative text.
- Results: Examples and Comparison training increased machine-text recall and F1 while reducing human-written guesses, but agreement remained low at α ≤0.11.Examples-trained evaluators also produced fewer form comments and twice as many content comments.
4 Discussion
Small-batch evaluations did not reliably distinguish machine-generated text across domains. Evaluators often relied on surface form and underestimated machines’ ability to generate humanlike attributes, while example training shifted attention toward content without consistently improving accuracy.
- Training outcomes: None of the three training methods significantly improved evaluators’ ability to detect machine-generated text reliably across domains.The comparison preserved the small-batch nature of Amazon Mechanical Turk.
- Evaluator behavior: Untrained evaluators mainly judged authorship from grammatical and stylistic correctness, producing systematic underestimation of machine-generated text quality.The evidence includes high percentages of human guesses and low recall for machine guesses.
- Training outcomes: Example-trained evaluators expected more from machine-generated text and focused more on content, but this shift did not significantly raise scores across all three domains.The annotation results show a reduced proportion of form comments and a doubled proportion of content comments.
- Evaluator behavior: Evaluators’ explanations referenced human attributes or intent they believed machines could not generate, but current NLG models can produce such superficial humanlike references.This mistaken assumption is suspected to contribute to low accuracy.
- Implication: The findings suggest that improving NLG quality makes surface-level fluency cues increasingly unreliable for human evaluation.Evaluators cited superficial textual features that machine generation had surpassed.
5 Recommendations
For small-batch evaluations on crowdsourcing platforms, the authors recommend example-based training and documenting evaluator instructions. More broadly, they encourage moving away from standalone intrinsic evaluations because detecting many errors requires careful reading.
- Crowdsourced evaluation: Example-based training is recommended for small-batch evaluations on Amazon Mechanical Turk or similar platforms.The authors say examples can calibrate expectations and indicate the careful reading needed to assess text quality.
- Crowdsourced evaluation: Researchers should confirm why evaluators made their decisions because their implicit criteria may mismatch the intended evaluation criteria.The recommendation follows from evaluators’ reliance on surface-level features.
- Evaluation design: Researchers should move away from standalone intrinsic human evaluation tasks as generated texts become longer and many errors require careful reading to detect.Default evaluators focused most on surface-level, fluency-related aspects of quality.
- Platform boundary: A rewriting-based training condition had less than 30% retention and over 50% rejection, so AMT was unsuitable for that task at the tested format and price point.This was an attempted fourth training condition rather than one of the three main methods.
- Reporting: NLG papers should report evaluator instructions and training, alongside efforts to standardize human-evaluation design and deployment.The authors motivate this recommendation by the mixed effects of different training methods and sparse reporting details.
6 Related Work
Related work examines human-evaluation tradeoffs, design, framing, agreement, and reporting, while prior studies report varied success detecting machine-authored text. Other work investigates tools for detecting generated text and the safe deployment of NLG systems.
- Human evaluation research: Prior research addresses tradeoffs between human and automatic evaluation, evaluation design, question framing, agreement, and analyses of NLG papers’ human-evaluation practices.These studies form a broader research area on the role of human evaluations.
- Detection studies: Reported detection accuracy varies across settings, including 71.4% for trained evaluators detecting GPT2-L text, 66.6% for product reviews, and 52% for GPT3 news articles.These figures come from different studies, datasets, and evaluator conditions.
- Detection and deployment: Related work explores visualization techniques and language models that help detect generated text, connecting evaluator limitations to safe NLG deployment.The cited studies investigate both reader-facing visualization and automated detection capabilities.
7 Conclusion
Untrained evaluators could not distinguish human- and GPT3-generated text across three domains and often relied on surface qualities. Example-based training increased content-focused judgments but did not significantly improve accuracy across all domains.
- Main findings: Untrained evaluators were unable to distinguish human- and GPT3-generated text from three domains.The conclusion motivates rethinking how human evaluations are collected for current NLG models.
- Main findings: Evaluators focused on surface-level text qualities and underestimated current NLG models’ capabilities.The conclusion identifies this behavior as part of the evaluation problem.
- Training findings: Example-based trainings increased recall and content-based evaluations, but did not significantly improve accuracy across all domains.The benefit was therefore partial rather than a consistent cross-domain accuracy gain.
A.1 Newspapers
The newspaper stimuli were drawn from randomly selected U.S. states and newspaper websites listed by state.
- Each newspaper was randomly selected from Wikipedia’s lists of newspapers by U.S. state.
- Human-authored news articles and prompts came from newspapers and websites spanning 15 listed states.
A.2 Score Frequencies
Evaluator score frequencies are reported separately for the GPT2 and GPT3 experiments in Figures 3 and 4.
- Scores out of 5 are summarized as frequency distributions for the GPT2 and GPT3 experiments.
A.3 Annotation Details
The study documented how evaluators justified authorship judgments and formalized those explanations into annotation categories, while presenting the training conditions used in the experiments.
- 300 evaluator comments were annotated, with 150 drawn from each of the No Training and Examples experiments.
- Annotation Details: Two annotators labeled each comment, using a label set iteratively refined through joint and separate pilot-data annotation.
- Annotation Details: Evaluators’ explanations often invoked beliefs about what kinds of language machines could or could not generate.
- Annotation Details: A pilot comparison found that the most accurate evaluators focused more on textual repetition and less on style.
- Training Conditions: All evaluators first received basic instructions, after which training conditions supplied instructions, examples, or paired comparisons.