Source-linked AI summary
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas Eck
TL;DR
Detecting machine-generated text remains difficult because humans and automatic systems rely on different cues. By benchmarking decoding strategies and excerpt lengths, the paper finds human raters underperform the best automatic discriminator, which detects statistical artifacts.
Problem
Limited evidence distinguishes the textual properties that make generated text seem human to people from those recognized by automatic detectors.
Method
The paper benchmarks top-k, nucleus, and temperature sampling on GPT-2 using human ratings and BERT-based classifiers across excerpt lengths.
Results
74% median human-rater accuracy was below the best discriminator’s accuracy, with humans noticing semantic errors and discriminators detecting statistical artifacts.
Takeaways & Limitations
Evaluating the humanness of generated text should use both human and automatic detectors because they rely on different qualities.
Takeaways & Limitations
The detection setup uses unconditioned or minimally primed excerpts, so the findings may not extend to longer priming contexts.
Abstract
from arXiv · showhide
Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions. Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies---top-$k$, nucleus sampling, and untruncated random sampling---and show that improvements in decoding methods have primarily optimized for fooling humans. This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems. We also show that though both human and automatic detector performance improve with longer excerpt length, even multi-sentence excerpts can fool expert human raters over 30% of the time. Our findings reveal the importance of using both human and automatic detectors to assess the humanness of text generation systems.
1 Introduction
Modern language models can produce multi-paragraph text that is nearly indistinguishable from human writing, while deceptive generated content heightens the need for reliable human and automated detection. This work studies how decoding strategy and excerpt length affect human and classifier detection performance.
- Motivation: Generated text can appear virtually indistinguishable from human writing, with subtle logical or linguistic errors often requiring close reading or domain knowledge to detect.Automatically generated reviews have also been perceived as fluent as human-written ones.
- Motivation: Deceptive content propagates quickly, shapes political agendas, influences elections, undermines user trust, and increases the need for human and automated detection.The paper frames this need as increasingly important as generative technology matures and writing assistance becomes more common.
- Decoding strategies: Decoding strategies generate sequences by selecting tokens from a neural language model’s next-word probability distribution, increasingly through probabilistic sampling.The introduction describes decoding as an algorithm for determining how words are selected from that distribution.
- Decoding strategies: Top-k sampling restricts low-likelihood words, reducing choices that humans readily recognize as poor while preserving statistical over-representation that automatic systems can detect.Humans are less proficient at noticing subtle frequency preferences, whereas automatic systems excel at identifying statistical anomalies.
- Study design: The study applies top-k, nucleus, and temperature sampling to GPT-2, then trains BERT-based binary classifiers to distinguish human-written from machine-generated excerpts.It evaluates how decoding strategy and generated-sequence length affect both human-rater and classifier accuracy.
- Contributions: The paper comprehensively examines detector sensitivity to model structure, decoding strategy, and excerpt length while comparing human raters’ abilities with those of automatic detectors.This comparison is stated as one of the paper’s primary contributions.
2 Related Work
Prior work established that large Transformer language models can generate convincing human-like text, motivating both human-rating methods and automated detectors. Automated approaches have used model-based detection, token-likelihood statistics, and semantic understanding of generated content.
- Generative Language Models: Large Transformer language models such as GPT-2, GROVER, and Transformer-DMCA can generate convincing human-like excerpts several paragraphs long.GROVER has also generated fake news judged more trustworthy than human-written fake news by human raters.
- Human Detection: Human detection traces back to the Turing Test and remains central to chatbot evaluation and open-domain generation assessment.Human raters’ judgments of machine-generated excerpt quality remain the gold standard for evaluating open-domain generation systems.
- Automatic Detection: GROVER detects machine-generated news with a fine-tuned version of its generative model, while GLTR analyzes histograms of per-token log likelihoods.GLTR assumes attackers favor high-likelihood tokens when sampling.
- Natural Language Understanding: Semantic inconsistencies such as contradictions, falsehoods, and topic drift can indicate machine-generated text.BERT-based encoder-only Transformers perform well on tasks requiring semantic understanding, including classification of whether text was machine-generated.
- Natural Language Understanding: Contextual embeddings from pretrained BERT models have also been used to compute quality scores for generated text without fine-tuning.The paper instead fine-tunes BERT to classify whether text was machine-generated.
3 Task Definition
The paper frames generated-text detection as binary classification and uses balanced datasets with separate classifiers to study decoding strategy and excerpt-length effects on automatic disambiguation.
- Task Definition: Detection is framed as binary classification: labeling text excerpts as human-written or machine-generated.The task examines how excerpt length and decoding strategy affect classification performance.
- Task Definition: Separate classifiers trained on each dataset assess which decoding strategies produce text easiest to automatically distinguish from human writing.This design compares automatic disambiguation across decoding strategies.
- Task Definition: Training-set example length is evaluated for its impact on classifying excerpts of the same length as human-written or machine-generated.Each dataset is approximately balanced between machine-generated positive examples and human-written negative examples.
4 Dataset Methodology
The study constructs matched datasets from GPT-2 LARGE using three decoding strategies, human-written web excerpts, two priming conditions, and ten excerpt lengths. This design isolates decoding, priming, and length effects while reducing stylistic differences between generated and human text.
- Dataset construction: The human-written excerpts come from the same distribution as GPT-2’s training data, preventing classifiers from relying simply on stylistic corpus differences.These excerpts serve as negative examples of human-written text.
- Dataset construction: Each decoding method pairs 250,000 generated samples with 250,000 web-text excerpts, while 5,000 additional paired samples support validation and testing.Excerpts with fewer than 192 WordPiece tokens are filtered out.
- Priming: Two dataset variants use either no priming or a single web-text token, matching each generated excerpt’s starting token to its paired human excerpt.The study uses these conditions to examine the simpler priming question and reports that limited priming can strongly affect automatic detectors.
- Excerpt length: The researchers truncate excerpts to ten lengths from 2 to 192 WordPiece tokens, producing sixty dataset variations across sampling method, length, and priming condition.The variations include one combination for each decoding method, truncation length, and priming choice.
5 Automatic Detection Method
Automatic detection primarily uses a separately fine-tuned BERT classifier for each dataset variation, with comparisons against simpler baselines at sequence length n=192. One baseline represents texts with GPT-2’s 50,000-token vocabulary and classifies them using logistic regression, while removing infrequent vocabulary entries did not improve performance.
- Primary discriminator: The primary automatic discriminator is a fine-tuned BERT classifier trained separately for each dataset variation.BERT is fine-tuned to label sequences as human- or machine-written.
- Baseline comparisons: At the longest sequence length, n=192, BERT is compared with several simple detection baselines.These baselines were proposed in prior work.
- Bag-of-words baseline: The bag-of-words baseline uses GPT-2’s 50,000-token BPE vocabulary and trains logistic regression to classify sequences as human- or machine-written.Each embedding dimension counts occurrences of its corresponding vocabulary token.
- Bag-of-words baseline: Removing entries for infrequent vocabulary words from the bag-of-words embedding did not improve performance.The resulting classifier still used the full vocabulary representation as the stronger configuration tested.
6 Human Detection Method
Human detection was tested by asking raters to classify progressively longer excerpts as human- or machine-written, using four confidence-labeled response options. Initial AMT performance was near chance, so the study repeated the task with university students who received group training examples.
- Human evaluation procedure: Machine-generated excerpts were evenly divided among the three sampling strategies, while human-written and machine-generated passages were equally likely.The evaluation continued across progressively longer excerpts, ending at 192 tokens.
- Rater performance: Over 70% of AMT raters’ “definitely” votes identified text as human despite balanced classes, and accuracy remained around 50% even for the longest sequences.The researchers then repeated the study with university students who were first walked through ten examples.
- Evaluation dataset: The dataset contained 150 web-text excerpts and 50 excerpts from each decoding strategy, producing 900 untrained-worker annotations and 475 expert-rater annotations.Each question was shown to at most three raters.
7 Automatic Detection Results
Automatic discriminators substantially outperform simple baselines and humans, but their performance depends strongly on decoding-specific statistical artifacts and calibration. Top-k is especially easy for automatic systems to detect despite being hardest for human raters, while priming sharply reduces detectability.
- Automatic baselines: BERT far surpasses simple baselines, while TotalProb still exceeds 60% accuracy across all sampling methods.Bag-of-words logistic regression is the strongest simple baseline, whereas its AUC is only 0.52–0.56 in this task.
- Decoding artifacts: Top-k concentrates up to 80% of probability mass in the 500 most common token types, diverging from human text and creating an easy detection feature.Other sampling methods and human-written text require at least 1,100 token types to reach the same concentration.
- Priming: Adding one human priming token reduces top-k detection accuracy from ∼90% on length-2 sequences to ∼65%.Priming introduces more rare words into the top-k unigram distribution, while its effect is smaller for nucleus and untruncated sampling.
- Transferability and calibration: Top-k-trained discriminators transfer poorly across decoding methods, with accuracy falling as low as 42.5%, while untruncated-trained discriminators transfer little to top-k.Only the top-p-trained BERT discriminator remains approximately calibrated at 50% machine-generated predictions on in-domain and other-strategy datasets.
8 Conclusion
The paper finds that automated discriminators and human raters detect generated text using different cues, with discriminators generally outperforming humans but transferring poorly across decoding strategies. It concludes that generation quality, discriminator capabilities, and human detection skills all require further improvement.
- Automated detection: Discriminators generally transfer poorly between decoding strategies, although training on mixed-method data can improve transfer.The discriminators were trained on balanced binary classification datasets using excerpts generated by the same model with different decoding strategies.
- Human detection: 74% median human-rater accuracy was below the accuracy of the best-performing discriminator.Expert-rater accuracy varied widely across experiments.
- Human–machine cues: Humans more readily notice semantic errors, whereas discriminators detect statistical artifacts.The paper reports that these artifacts were especially prominent with top-k sampling.
- Research implications: Generation systems face a trade-off between human-perceived quality and ease of automatic detection.The paper links this trade-off to generating text that is exciting and semantically plausible without introducing poor word choices.
- Future research: Future research should improve generation quality, strengthen discriminators’ world understanding, and develop tools and educational materials for human detection.The proposed directions target exciting yet semantically plausible text, human-noticed errors, and explainable detection support.
A Appendix · A.1 Dataset Sizes
Appendix A.1 describes the datasets used to train and evaluate automatic discriminators, pairing machine-generated positive examples with human-written negative examples. The machine-generated excerpts vary by decoding algorithm and priming strategy.
- A.1 Dataset Sizes: Table 5 reports the sequence counts used to train and evaluate each automatic discriminator.The passage identifies Table 5 as the source of these dataset sizes.
- A.1 Dataset Sizes: Each discriminator is trained for binary classification.The classification task distinguishes machine-generated from human-written examples.
- A.1 Dataset Sizes: Machine-generated excerpts serve as positive examples.These examples are drawn using specified decoding and priming configurations.
- A.1 Dataset Sizes: Human-written excerpts serve as negative examples.They appear in the last row of Table 5 and are paired with machine-generated excerpts.
- A.1 Dataset Sizes: Datasets are constructed by pairing human-written excerpts with machine-generated excerpts.This pairing defines the examples used for discriminator training and evaluation.
- A.1 Dataset Sizes: The machine-generated excerpts use k40, p0.96, or p1.0 decoding with nocond or 1wordc priming.The passage lists these decoding algorithms and priming strategies as dataset construction factors.
A.2 Further Details on Human Evaluation · A.3 Automatic Detection Method Reliability
Human raters became more confident as they observed longer excerpts, but often remained uncertain even at 192 tokens and took longer to identify machine-generated text. Across five independently fine-tuned BERT discriminators, out-of-domain accuracy was highly reliable, varying by approximately 1% or less.
- A.2 Further Details on Human Evaluation: 10% of worker questions were honeypots specifying the correct answer, with Amazon Mechanical Turk workers reaching 83% accuracy and expert raters reaching 91.8%.These questions were used to gauge worker attention levels.
- A.2 Further Details on Human Evaluation: Raters initially favored “possibly human,” then became more confident as additional tokens revealed evidence supporting either human-written or machine-generated text.At 16 tokens, “possibly human” was the most frequent response; even at 192 tokens, many raters remained uncertain.
- A.2 Further Details on Human Evaluation: By the longest sequence length, votes for “human-written” and “machine-generated” were about balanced.As excerpts were extended, raters used the additional evidence to revise their guesses.
- A.2 Further Details on Human Evaluation: Raters took longer to converge on a decision of “machine” than on a decision of “human.”The convergence analysis measured the sequence length at which raters made a single guess.
- A.3 Automatic Detection Method Reliability: Five independent BERT discriminators were fine-tuned on a mixed dataset of 50% human-written and 50% machine-generated 192-token sequences.Machine-generated examples were equally split among top-k=40, top-p=0.96, and untruncated random sampling, and the best in-domain validation checkpoint was evaluated out of domain.
- A.3 Automatic Detection Method Reliability: Approximately 1% or less was the standard deviation of out-of-domain accuracy across the five automatic discriminator runs.This indicates that out-of-domain accuracy was extremely reliable across independent fine-tunings.