PaperlayerView exact revision on arXiv

Source-linked AI summary

Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text

Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, Chris Callison-Burch

arXiv:2212.12672v1cs.CLcs.AIcs.HC

TL;DR

The paper asks whether people can detect generated text in a setting where a human-written prompt transitions to a model continuation. It studies this boundary-detection task with RoFT and finds variation in annotator performance alongside insights about model errors and detection behavior.

  • Problem

    Prior studies focused on classifying whole passages as human- or machine-written, leaving boundary detection in prompted continuations less studied.

    Source: §1 · p. 1

  • Method

    The authors use RoFT, a game in which players read sentences from a human-started passage and identify where it switches to model-generated text.

    Source: §3.1 · p. 2

  • Results

    Annotators varied substantially in detection ability; the study also reports improvement over time, genre-linked model error patterns, and sentence-level features correlated with selections.

    Source: §1 · p. 1, §7 · p. 7

  • Takeaways & Limitations

    The authors demonstrate boundary detection as a viable framework for human evaluation of natural-language generation systems.

    Source: §7 · p. 7

    The advanced AI-course student sample may not reflect the broader English-speaking population's ability to detect generated text.

    Source: §3.4 · p. 3

Abstract

from arXiv · show

As text generated by large language models proliferates, it becomes vital to understand how humans engage with such text, and whether or not they are able to detect when the text they are reading did not originate with a human writer. Prior work on human detection of generated text focuses on the case where an entire passage is either human-written or machine-generated. In this paper, we study a more realistic setting where text begins as human-written and transitions to being generated by state-of-the-art neural language models. We show that, while annotators often struggle at this task, there is substantial variance in annotator skill and that given proper incentives, annotators can improve at this task over time. Furthermore, we conduct a detailed comparison study and analyze how a variety of variables (model size, decoding strategy, fine-tuning, prompt genre, etc.) affect human detection performance. Finally, we collect error annotations from our participants and use them to show that certain textual genres influence models to make different types of errors and that certain sentence-level features correlate highly with annotator selection. We release the RoFT dataset: a collection of over 21,000 human annotations paired with error classifications to encourage future work in human detection and evaluation of generated text.

1 Introduction

The paper asks whether people can detect machine-generated text amid concerns that such text could be passed off as genuine. It reframes detection as locating the transition from human writing to generated continuation, and studies annotator performance and factors associated with detection.

1 Introduction

  • The paper frames detection as finding where a human-written document transitions to machine-generated text, rather than classifying an entire document.The authors argue this better aligns with typical prompted generation and can help quantify risks while evaluating generative systems.Source: §1 · p. 1
  • Annotators vary substantially in boundary-detection ability, and time spent and rounds played correlate with success.Players see sentences one at a time and earn points for guessing close to the true transition.Source: §1 · p. 1, Figure 1 · p. 1
  • The study examines how prompt genre, model size, and decoding strategy affect generation quality, and analyzes errors and sentence-level features associated with player judgments.The paper also reports that genres influence the types of errors models make.Source: §1 · pp. 1–2
  • The authors release RoFT generations and annotations, positioning the game platform as extensible for further research on detectability.The study describes this as the first large-scale gamified-platform study of generated-text detectability.Source: §1 · p. 2

2 Related Work

Prior work mostly asked whether a whole text was human- or machine-written, with reported detection performance varying across models. This paper applies a boundary-detection framework to language models more generally, building on earlier error-annotation work.

2 Related Work

  • Earlier studies framed detection as classifying entire human-written or machine-generated texts, with reported accuracy varying by model and study.Ippolito et al. (2020) reported up to 71.4% accuracy on GPT-2 Large; Clark et al. (2021) reported up to 62% on GPT-2 XL and chance performance on GPT-3.Source: §2 · p. 2
  • Prior work also collected human-identified text errors and proposed error-annotation schemas that informed the reasons RoFT players could select.The cited work includes He et al. (2021) and Dou et al. (2021).Source: §2 · p. 2
  • The paper applies RoFT-style detection across genres and generative systems, extending this kind of task beyond its prior use for evaluating dialogue systems.Deriu et al. (2020) posed a similar task for conversation turns; the authors demonstrate feasibility with more general language models.Source: §2 · p. 2

3 Experimental Design

The study uses a sentence-by-sentence game to measure where human-written passages switch to model continuations, across several genres and model configurations. It specifies the scoring, participants, generation comparisons, and the final annotation set used for analysis.

3.1 The Real or Fake Text Game

  • Players read a passage one sentence at a time and identify the first machine-generated sentence after a human-written start.A round ends when a player selects machine-generated text; players can also choose a reason for their decision.Source: §3.1 · p. 2

3.2 Datasets

  • Prompts came from news, speeches, stories, and recipes; each sampled document supplied a randomly selected prompt endpoint before the remaining text was replaced by a model continuation.Prompt lengths were distributed evenly, with approximately 10% of examples fully human-written; the table summarizes game rounds and annotations by genre.Source: §3.2 · pp. 2–3, Table 1 · p. 3

3.3 Awarding Points

  • Players earned five points for the boundary sentence and max(5−n, 0) for a guess n sentences after it, but no points for guessing before the boundary.The authors report mean score as the predominant metric and filtered annotations from players who exploited the scoring system by always choosing the last sentence.Source: §3.3 · p. 3

3.4 Players

  • Participants were students in an advanced AI course, with one group receiving score-based extra credit and detailed detection guidance; the authors caution that this sample may not represent English speakers generally.Group A received fixed extra credit, while Group B's extra credit depended on points earned.Source: §3.4 · p. 3

3.5 Continuation Sources

  • The model comparisons varied one aspect of a base configuration at a time, including model size, sampling strategy, recipe fine-tuning, and CTRL topic control codes.The base was pretrained GPT-2 XL with nucleus sampling p=0.4 and repetition penalty 1.2; comparisons also included GPT-2 small, alternative sampling settings, and CTRL generations on speeches.Source: §3.5 · pp. 3–4

3.6 Final Annotations Used for Analysis

  • After filtering for prompt-length balance and removing suspected exploitative players, the analyzed dataset contained 21,646 annotations across 7,257 continuations.News, stories, and recipes averaged two players per continuation, while speeches averaged 16.Source: §3.6 · p. 4

4 Main Findings

Players detected generated-text boundaries above chance, but skill and agreement varied, and performance improved under added instruction and incentives.

4 Main Findings

  • Players chose the true boundary sentence 23.4% of the time, above the 10% chance rate; mean score was 2.08 versus 1.31 random-chance points.For rounds with generated sentences, players selected a generated sentence as the boundary 72.3% of the time. The sanity-check baseline mean score was 2.75, higher than any LM-backed system.Source: §4 · p. 4
  • Agreement was slight overall (α=0.25) but higher among the top 10% of players (α=0.40).Source: §4 · p. 4
  • Player skill varied widely, with higher mean scores tending to accompany lower score variance.Among players scoring above 3.8 on average, the standard deviation averaged 1.84.Source: §4 · p. 4, Figure 2 · p. 4
  • Group B, given extra instruction and incentives, improved over time while Group A did not; Group B also showed lower score variance over time.Reading the help guide, which included an error taxonomy, annotated examples, and detection tips, was the most predictive feature of mean score.Source: §4 · pp. 4–5, Figure 3 · p. 4
  • For Stories, increasing model size made detection harder; recipe fine-tuning and a [Politics] control for Speeches did not significantly affect detectability.Source: Figure 5 · p. 5
  • Recipes were easier to detect than stories or news, and common-sense errors accounted for 26% of recipe labels versus 10% in each other genre.The authors hypothesize that ingredients lists and implicit cooking knowledge help players identify contradictions in recipe continuations.Source: §4 · p. 5, Figure 4 · p. 5

5 Model Comparison Findings

Detection varied with model size and decoding strategy, while recipe fine-tuning and political control codes did not make generations harder to detect. Error types also differed by genre.

Model size

  • Players scored significantly higher on GPT-2 small (117M parameters) than GPT-2 XL (1.5B parameters), the most significant difference across tested variables.The additional GPT-3 Davinci (175B parameters) round yielded no statistically significant conclusions.Source: §5 · p. 5

Decoding strategy

  • Players detected GPT-2 generations significantly better at p=1.0 than at lower nucleus-sampling values.The paper reports p=1.0 as pure random sampling and relates the result to difficulty generating high-quality text at human-like diversity.Source: §5 · p. 5

Fine-tuning

  • Recipe fine-tuning slightly increased player detection ability despite reducing held-out perplexity from 8.979 to 4.781.The authors speculate that pre-trained GPT-2 may already contain enough recipe knowledge to make additional fine-tuning less consequential.Source: §5 · pp. 5–6

Genre and error types

  • In Figure 6, players more often attributed recipe continuations to common-sense errors, story continuations to irrelevant errors, and news continuations to generic errors; all were GPT-2 XL outputs at p=0.4.Source: Figure 6 · p. 6

Control codes

  • Using CTRL’s [Politics] code did not significantly affect detection of presidential-speech generations.The authors caution that this result does not establish whether control codes affect detectability in other, more specialized domains.Source: §5 · p. 6

6 RoFT-Specific Insights

RoFT-specific analyses connect annotation time with scoring and speed, and examine the reasons and sentence features associated with players’ selections. Error patterns and named-entity trends vary across genres.

Annotation time

  • Longer annotations corresponded to higher scores, players got faster over time, and annotation time had no correlation with sentence length (ρ=-0.10).The study capped recorded sentence-decision times at 120 seconds before calculating total annotation time.Source: §6 · p. 6

Genre-specific errors

  • Across textual genres, when players explained why they marked sentences machine-generated, Recipes had a much higher proportion of “common_sense” errors, News more “generic” text, and Stories more “irrelevant” text.Source: §6 · p. 6

Reason reliability

  • Common-sense, irrelevant, and contradicts-sentence were the three most common reasons given and also the three most reliable; bad-grammar judgments were least reliable, with a mean score of 1.78.The table describes reason reliability as the average points earned on rounds where that reason was given.Source: §6 · pp. 6–7, Table 2 · p. 7

Named entities

  • Generated sentences had fewer named entities than human sentences in news and speeches, but more in stories; selected sentences had slightly more entities in speeches and stories.In news, selected and unselected sentences tended to have about the same number of entities.Source: §6 · p. 7, Figure 8 · p. 7

7 Discussion and Future Work

The authors argue that boundary detection is viable for human evaluation of natural-language generation and report findings about annotators and generated-text errors. They identify sentence-aligned transitions as a core limiting assumption and point to broader future tests.

Discussion

  • The study demonstrates boundary detection’s viability for human evaluation and reports the largest study of generated-text detectability to date, replicating findings on model size, decoding, and annotator incentives.The authors also report that genres influence error types, annotators improve over time, and sentence-level features correlate with selection.Source: §7 · p. 7

Future work

  • The authors expect the results and released generations-and-annotations dataset to be broadly useful to detection researchers.They identify automatic detection and predicting when generated text will be especially difficult for humans as future research directions.Source: §7 · p. 7

Limitations

  • The study assumes transitions occur exactly at sentence boundaries, which the authors identify as a core limitation.They also call for testing more models, genres, experimental conditions, and annotator-incentive frameworks.Source: §7 · p. 7

A Implementation Details

The appendix records tools and settings used for text processing, generation, and GPT-2 XL fine-tuning.

Generation and inference

  • Generation used nucleus sampling with p=0.4 unless otherwise specified, a 1,024-token context limit, and a 1.2 repetition penalty.All model inference ran on one NVIDIA Tesla T4 GPU; generating continuations took approximately 48 GPU hours.Source: A Implementation Details · p. 9

Processing tools

  • spaCy en_core_web_lg handled sentence segmentation, named-entity recognition, and part-of-speech tagging; HuggingFace Transformers handled tokenization, generation inference, and fine-tuning.Source: A Implementation Details · p. 9

Fine-tuning

  • GPT-2 XL fine-tuning used 500,000 recipe examples for training and selected the checkpoint with the lowest validation perplexity.The final model’s test perplexity was 4.781, compared with 8.979 for pre-trained GPT-2 XL; the fine-tuning process took 80 GPU hours in total.Source: A Implementation Details · p. 9

B Experiments with Larger Models

The additional annotation round found no statistically significant difference in mean annotator score between GPT-3 Davinci and GPT-2 XL. The appendix also details generation-quality controls used to avoid trivial or unsuitable examples.

B Experiments with Larger Models

C.1 Quality Control of Generations

  • Generation controls addressed sentence-segmentation errors, incomplete or low-quality sentences, and unsafe content.The authors switched to a transformer spaCy model for segmentation, rejected generations without a verb or auxiliary verb in every sentence, and discarded content flagged unsafe above 35.5% confidence.Source: §C.1 · pp. 9–10

B Experiments with Larger Models

  • No statistically significant difference in mean annotator score was observed between GPT-3 Davinci and GPT-2 XL.The authors suggest that sentence-by-sentence boundary judgments provide less context than whole-passage classification and may require much more data to distinguish top-performing models.Source: B Experiments with Larger Models · p. 9, Figure A9 · p. 10

C.2 Filtering Player Annotations

Some players repeatedly guessed the same boundary to gain points more efficiently, so the authors filtered sustained repetitions. The appendix also reports graph values and compares model and generation factors across datasets.

C.2 Filtering Player Annotations

  • Players could speed annotation by labeling all sentences human-written, while the scoring system made later guesses more rewarding in expectation.These incentives help explain the repeated-boundary pattern described in the section.Source: §C.2 · p. 10
  • The authors filtered out spans of five or more annotations in which a player guessed the same index each time, removing 3,694 annotations total; 61 of 241 players had at least one annotation removed.Source: §C.2 · p. 11

D Values from Bar Graphs

  • Table A5 reports a large effect of model size on human performance, while fine-tuning and control-code usage have minimal effects.The comparisons span three datasets.Source: Table A5 · p. 11
  • Table A6 gives mean scores by domain for XL-sized models at p=0.4, while Table A7 compares average user points with boundary-guess metrics.Table A7 includes correct-side and exact-boundary fractions, plus average sentences after the boundary.Source: Table A6 · p. 11, Table A7 · p. 11

E Metric Correlations

Mean score aligns strongly with two measures of boundary-guess correctness but only weakly with distance after the boundary.

E Metric Correlations

  • Mean score is strongly positively correlated with perfect-guess accuracy and correct-side-of-boundary rate, but only weakly correlated with distance after the boundary.Only guesses within five sentences to the right of the boundary earn points; the authors say this harsh scaling is intentional to prevent later guesses from receiving much higher expected scores.Source: E Metric Correlations · p. 11

F Exit Survey Results

Exit-survey results identify reading the help guide as the strongest measured predictor of annotator skill, while several self-reported traits show no significant score differences. Custom survey responses tended to accompany better performance, but self-reports limit interpretation.

F Exit Survey Results

  • The exit survey collected consent and impersonal, generic responses, and the researchers removed identifying information before sharing anonymized data.Table A10 lists the survey questions.Source: F Exit Survey Results · p. 11, Table A10 · p. 13
  • Reading the provided help guide was the most impactful feature for predicting annotator skill.The figure reports only very slight correlations between help-guide reading and higher mean score among players completing at least 20 rounds.Source: F Exit Survey Results · p. 11, Figure A12 · p. 13
  • The study found no statistically significant score differences by native-English status, reported genre familiarity, or familiarity with GPT-2/3.Participants who selected “other” and wrote custom responses tended to perform better.Source: F Exit Survey Results · pp. 11–12
  • Self-reported survey qualities have recognized limitations, so the absence of significant correlations does not establish that no relationships exist.The reported tendency for custom-response participants to score better is likewise based on self-reported survey categories.Source: F Exit Survey Results · p. 12
Loading 2212.12672v1…