PaperlayerView exact revision on arXiv

Source-linked AI summary

Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, Yejin Choi

arXiv:2107.01294v3cs.CL

TL;DR

Human evaluation struggles to distinguish increasingly fluent machine text and often lacks detailed information about where generations fail. The paper proposes SCARECROW, a crowd-annotation framework that identifies and explains span-level errors, then applies it at scale to compare human and machine writing across model factors. The resulting analysis quantifies measurable gaps and shows that some error types improve with scaling while others plateau.

  • Problem

    Human judgments remain central for open-ended generation, but holistic ratings, pairwise comparisons, and small expert analyses provide limited detail about where models fail and why.

  • Method

    SCARECROW uses crowd workers to annotate model-output spans with error categories refined through pilot studies without a fixed label set.

  • Results

    The study collects 41k labeled error spans and quantifies measurable gaps between human text and generations from models of several sizes, while analyzing model size, training data, and decoding strategy.

  • Takeaways & Limitations

    Scaling reduces Encyclopedic, Commonsense, and Incoherent errors, but benefits plateau for Off-Prompt, Bad Math, and Grammar and Usage errors.

  • Takeaways & Limitations

    The news-text evaluation avoids possible training-data copying and COVID-19 distribution shift, and the framework’s extension to longer narratives is left for future work.

Abstract

from arXiv · show

Modern neural language models can produce remarkably fluent and grammatical text. So much, in fact, that recent work by Clark et al. (2021) has reported that conventional crowdsourcing can no longer reliably distinguish between machine-authored (GPT-3) and human-authored writing. As errors in machine generations become ever subtler and harder to spot, it poses a new challenge to the research community for robust machine text evaluation. We propose a new framework called Scarecrow for scrutinizing machine text via crowd annotation. To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of Scarecrow -- such as redundancy, commonsense errors, and incoherence -- are identified through several rounds of crowd annotation experiments without a predefined ontology. We then use Scarecrow to collect over 41k error spans in human-written and machine-generated paragraphs of English language news text. We isolate factors for detailed analysis, including parameter count, training data, and various decoding-time configurations. Our approach successfully quantifies measurable gaps between human authored text and generations from models of several sizes, including fourteen configurations of GPT-3. In addition, our analysis unveils new insights, with detailed rationales provided by laypeople, e.g., that the commonsense capabilities have been improving with larger models while math capabilities have not, and that the choices of simple decoding hyperparameters can make remarkable differences on the perceived quality of machine text. We release our training material, annotation toolkit and dataset at https://yao-dou.github.io/scarecrow/.

1 Introduction

The paper introduces SCARECROW, a crowd-annotation framework for systematically identifying and explaining issues in machine-generated text. It uses detailed span-level error analysis to measure differences between human and model writing and examine how model size, training data, and decoding affect errors.

  • Motivation: SCARECROW addresses the challenge that crowd workers cannot reliably distinguish GPT-3 generations from human-written text.The framework instead asks crowd workers to identify and explain specific problems in seemingly fluent generations.
  • Findings: The study quantifies measurable gaps between machine-generated and human text while revealing distributions of specific issue categories across models and humans.It also characterizes which generation aspects improve over previous models and which do not improve with more data and parameters.
  • Method: The framework elicits span-level categorical judgments from non-expert readers about errors in model outputs.It targets problems recognizable to readers without NLP expertise and records what is wrong and where it occurs.
  • Method: Its error schema was refined through multiple crowd-annotation pilots without a fixed label set, balancing linguistic analysis, observed generation errors, and laypeople’s capabilities.The resulting categories cover shortcomings salient to non-expert readers while remaining usable with minimal training.
  • Contribution: SCARECROW provides more specific and practical insight than holistic judgments by measuring how and where problems arise.Annotations identify error types and their locations rather than only assigning an overall quality assessment.
  • Analysis: The authors collect 13k annotations over 1.3k paragraphs, yielding 41k spans labeled with error type, severity, and an explanation.The analysis covers human and machine text and examines model size, training data, and decoding strategy across contemporary language models.

2 Key Findings

Scarecrow measures annotated error spans across human and model-generated news text, revealing which error categories improve with scaling, plateau, or show complex trends. Human text has the fewest several substantive errors but the most reader issues, while decoding settings can substantially affect measured quality.

  • Measurement scope: Scarecrow measures span coverage across model sizes, training domains, human-authored text, and GPT-3 decoding configurations.Span coverage is the average portion of tokens covered by annotations; Figure 2 separates error types, Figure 3 stacks them, and Figure 4 removes reader issues.
  • Scaling trends: Scaling reduces Encyclopedic, Commonsense, and Incoherent errors, with human text showing the fewest such errors.These categories decrease with in-domain training and larger model size, including GPT-3.
  • Scaling trends: Scaling plateaus for Off-Prompt, Bad Math, and Grammar and Usage errors when moving to GPT-3.Humans still commit fewer Off-Prompt and Grammar and Usage errors, while Bad Math appears saturated for this domain.
  • Scaling trends: Self-Contradiction and Redundant errors rise and fall across model scales, increasing for medium or large models but dropping for human-authored text.GPT-2 Small is often too incoherent for Self-Contradiction to occur frequently.
  • Reader issues: Human-authored text produces the most reader issues, which are not necessarily errors but can prevent full comprehension or factual verification.Needs Google and Technical Jargon are the reader-issue categories identified as highest for humans.

5. Decoding hyperparameters have a huge im-

The study holds decoding fixed for model comparisons, then varies GPT-3 sampling settings to test their effect on annotated errors. These hyperparameters substantially change measured error rates, with the best configuration matching human-authored text on apparent SCARECROW error spans.

  • Experimental setup: The model comparison fixes top-p = 0.96, temperature = 1, and no frequency penalty, while GPT-3 decoding experiments vary these settings.The decoding study uses different top-p and temperature values with and without a frequency penalty.
  • Results: GPT-3 decoding hyperparameters considerably affect error rates.The worst tested procedure performed worse than GPT-2 XL, whereas the best produced as few apparent SCARECROW error spans as human-authored text.

3 Evaluation of Natural Language Generation

Open-ended generation permits diverse valid outputs, making reference-overlap evaluation inadequate and leaving conventional human judgments without detailed explanations. The paper therefore motivates a crowd-based, span-level error-analysis framework that is reusable and informative about model failures.

  • Evaluation challenge: Open-ended generation allows lexically and semantically diverse outputs, making word-overlap metrics unsuitable when many outputs are valid.Story, blog, and dialogue generation illustrate this broad space of allowable outputs.
  • Evaluation challenge: A/B testing expresses only relative improvements, while Likert ratings do not identify why text receives a score or which segment is problematic.Detailed model-failure insights instead often come from small-scale expert analyses.
  • Proposed direction: The paper proposes a crowdworker-compatible error-analysis framework to provide detailed information about model progress and failures at scale.The framework is intended to be easy to learn, reusable, and independent of particular models or experimental conditions.

4 SCARECROW Annotation Methodology

SCARECROW uses crowdworkers to identify and explain problems in generated text at the span level. Its annotation schema balances linguistic analysis, observed generation errors, and what non-experts can learn with limited training.

  • Prompt and generation: SCARECROW uses one-sentence human prompts and 80–145-token continuations for evaluation.Continuations may be model-generated or human-authored, while annotators remain unaware of their source.
  • Annotation procedure: Annotators mark problematic spans, assign an error type, select severity, and explain their reasoning.The interface supports saving, viewing, and editing annotations.
  • Error types: The ten error types were refined through pilot studies involving 30 workers, 750 annotations, and 60 paragraphs.The schema balances linguistic analysis, observed errors, and everyday readers’ capabilities after one to two hours of training.
  • Error types: Errors are grouped into language errors, factual errors, and reader issues.Reader issues impede comprehension without necessarily being errors themselves.
  • Severity: Annotators choose among three severity levels ranging from minor quality impact to text that is almost impossible to understand.The paper collects severity labels but largely leaves their analysis for future work.
  • Crowdsourcing: Workers receive extensive qualification training and annotate paragraphs through a custom interface.The qualification requires at least 90 out of 100 points, while annotation payment was set at $3.50 per annotation to target at least $15 per hour.

5 Data Collection

The study collects annotations over human and model continuations of January 2020 news prompts, comparing model size, training domain, and decoding choices. It uses matched decoding for primary comparisons while separately varying GPT-3 sampling settings.

  • Models: The models span GPT-2 Small, GPT-2 XL, Grover-Mega, GPT-3 DaVinci, and human-authored text.The configurations vary parameter count, training domain, and whether the text is human-written.
  • Decoding strategies: The decoding study considers nucleus probability p, temperature t, and frequency penalty f.p.Presence penalty is omitted because of annotation budget constraints, and p and t are varied independently.
  • Decoding strategies: Primary comparisons use identical decoding strategies across models in an “apples-to-apples” setup.The study additionally examines decoding effects by varying GPT-3 sampling parameters.
  • Prompt selection: News articles from January 2020 provide prompts, with the first sentence used to condition generation.This avoids likely training-data copying from GPT-3’s 2016–2019 Common Crawl data and limits COVID-19-related distribution shift.
  • Prompt selection: Each model generates 80–145 tokens as a continuation, stopping at the first sentence boundary after 80 tokens.Human continuations are truncated under the same sentence-boundary rule.
  • Crowdsourcing: Ten workers annotate each paragraph because individual annotations show high precision but low recall.The study examines annotator agreement and variability separately.

6 Error Prediction

The paper also frames error detection as span classification, using aggregated crowd annotations as gold labels. Models achieve higher recall than individual annotators and higher F1 for half of the error categories.

  • Task: The prediction task classifies generated-text spans by error type or “No Error.”Negative spans are sampled across lengths to match the breadth of observed error spans.
  • Model: A RoBERTa-large span classifier represents each span using boundary tokens and a learned length embedding.A feedforward network predicts the error categories plus “No Error.”
  • Evaluation: Evaluation uses per-token precision, recall, and F1 for spans up to length 30.Gold labels aggregate the spans of ten annotators, while human performance is measured one-versus-nine.
  • Results: For half of the error categories, models trained on aggregated human labels achieve higher F1 than individual human annotators.Humans generally have higher precision, whereas models achieve considerably higher recall.
  • Results: The model achieves nearly perfect recall for Needs Google with precision close to 0.6.It also exceeds individual-human recall for Grammar and Usage, Incoherent, and Redundant errors, sometimes with precision in the .20s.

7 Related Work

Prior NLG evaluations often rank or rate outputs without locating specific failures or explaining why they occur. SCARECROW instead combines span-level error categories with explanations to inspect model shortcomings directly.

  • Automated evaluation: Reference-based metrics such as BLEU, ROUGE, METEOR, and BERTScore are less helpful for open-ended generation with many valid continuations.Their use is established in tasks such as translation and summarization.
  • Automated evaluation: Recent automated approaches compare distributions, assign multidimensional scores, or distinguish human text from perturbed negative samples.Examples include Perception Score, UNION, and MAUVE.
  • Human evaluation: Crowd evaluations commonly use global criteria, relative rankings, or rating scales rather than detailed error localization.TuringAdvice connects generated-text problems to core NLP tasks, but the cited comparison emphasizes broader evaluation formats.
  • Motivation: Existing ranking and rating methods do not reveal exactly which parts of a generation fall short or why.SCARECROW addresses this by locating, categorizing, and explaining problems in generated text.
  • Comparison framework: Table 3 compares evaluation methods along criteria including general versus specific errors, direct evaluation versus ranking, explanations, rating scales, and span annotation.The comparison uses abbreviations such as GC, SET, DE, RR, EE, RS, and SA.

8 Conclusion

SCARECROW is presented as a method for identifying and explaining issues in generated text, accompanied by an analysis on large neural language models and released resources.

  • SCARECROW identifies and explains issues in generated text.
  • The method is analyzed on several large neural language models in an open-ended news generation task.
  • The authors release the dataset and methodology to the community.

A.1 Language Errors

SCARECROW organizes machine-text problems into language, factual, and reader issues, using span-level labels that reflect errors and verification burdens recognizable to readers.

  • Language Errors: SCARECROW uses language-error labels covering word choice, syntax, semantics, and pragmatics.Grammar and Usage includes missing, extra, incorrect, out-of-order, or deletable words, while the schema emphasizes semantic and pragmatic problems.
  • Language Errors: Redundant marks repeated ideas by labeling both the first mention and the later repetition.Repetition may use identical or different wording.
  • Language Errors: Off-Prompt marks generations unrelated to the prompt, whereas Self-Contradiction marks generated text that contradicts itself.The schema also marks the antecedent statement for self-contradictions.
  • Language Errors: Incoherent marks grammatical, nonredundant, on-prompt text that remains confusing.The examples describe sentences whose meaning or events are difficult to interpret.
  • Factual and Reader Issues: Factual errors cover incorrect statements, mathematical or unit-conversion mistakes, implausible currencies, and violations of basic commonsense.The framework distinguishes encyclopedic knowledge from reader-dependent verification categories such as Technical Jargon and Needs Google.

B.1 Error Severity

SCARECROW trains annotators to identify errors and assign three severity levels, ranging from minor issues to generations that are substantially off-prompt and nonsensical.

  • Error Severity: Annotators receive examples for three severity levels during training.The training material also includes annotation exercises, label quizzes, and a real task question.
  • Error Severity: Severity 1 represents a minor issue, while Severity 2 describes an error that remains understandable but is definitely wrong.The examples contrast a likely pronoun correction with a mistaken word choice whose meaning remains recoverable.
  • Error Severity: Severity 3 marks text that is far off the prompt and does not make sense.The example combines an unrelated generation with an internally nonsensical ending.

C Data Quality

The annotation data show generally strong agreement, but sparse error categories and token-overlap metrics require cautious interpretation and sufficient aggregation.

  • Agreement: Agreement is acceptable or high across categories, with Krippendorff’s α above 0.8 for most categories and above 0.6 for all error types.The coefficient is calculated per generation and averaged across the dataset.
  • Metric Caveats: High Krippendorff agreement can be misleading for sparse errors because 99% of tokens may be unannotated.Two Agree focuses on tokens labeled by at least two annotators and can reveal different agreement patterns.
  • Metric Caveats: Token-based overlap is a limitation of both agreement metrics.
  • Bootstrap Analysis: Bootstrap results show decreasing coefficient of variation as sample size increases, with less precipitous improvement after 30 examples.The analysis uses repeated samples from a GPT-3 condition with more than 200 annotated generations.
  • Agreement: Rare categories such as Bad Math and Encyclopedic show greater annotation variance and should be treated in aggregate.The authors motivate collecting at least 500 annotations per studied condition.

E Detailed Analysis

Detailed analyses show that error patterns depend on model scale, decoding, topic, and how errors are counted. GPT-3 can match human apparent error rates under one configuration, but manual review indicates substantial annotation noise and persistent differences in error composition.

  • Scaling: GPT-3 repetition appears in similar numbers of cases as smaller models but covers more tokens, reflecting larger topically repetitive blocks.This distinction emerges when comparing span counts with span coverage.
  • Reader issues: Human-authored text has the most Needs Google and Technical Jargon issues, with Technical Jargon varying substantially by news topic.These categories are treated as reader issues rather than necessarily machine errors, and their prevalence depends on reader knowledge and domain.
  • Decoding: Higher sampling randomness increases Off-Prompt errors but decreases Redundant errors, whereas frequency penalties lower error rates across tested settings.The best reported configuration used argmax sampling with a frequency penalty and appeared to match human error rates.
  • Human versus GPT-3: After manual adjustment, 48% of GPT-3 worker-annotated errors were estimated legitimate, compared with 9% for human-written articles.The authors caution that annotation noise may reach 90% when assessing high-quality text.
  • Error explanations: Error explanations differ by category: Self-Contradiction explanations describe reversed semantics, while Commonsense explanations state violated knowledge and Bad Math explanations often require calculation.Technical Jargon, Redundant, and Needs Google explanations are generally shorter and more straightforward.
Loading 2107.01294v3…