PaperlayerView exact revision on arXiv

Source-linked AI summary

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, Dongwon Lee

arXiv:2109.13296v1cs.CL

TL;DR

Machine-generated text is increasingly difficult to distinguish from human writing, but no benchmark systematically evaluates this Turing Test problem. TuringBench fills that gap with a 200K-sample environment covering detection and authorship attribution, finding FAIR_wmt20 and GPT-3 hardest to detect while motivating stronger methods.

  • Problem

    No benchmark environment systematically studies distinguishing neural machine-generated text from human-written text and identifying which generator authored it.

  • Method

    TuringBench builds a 200K-article dataset from 10K human news articles and 19 neural generators, supporting Turing Test and Authorship Attribution tasks with leaderboards.

  • Results

    FAIR_wmt20 achieved the lowest average F1 score at 0.49, followed by GPT-3 at 0.55, while human-test accuracies were 0.535 and 0.513 versus 0.5 random guessing.

  • Takeaways & Limitations

    The results indicate that many contemporary models produce texts indistinguishable to human eyes, while mission-critical detection requires better solutions.

  • Takeaways & Limitations

    Traditional authorship features and existing deep-learning models do not fully capture distinct writing styles across all 20 authors.

Abstract

from arXiv · show

Recent progress in generative language models has enabled machines to generate astonishingly realistic texts. While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection). However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods. In this work, we present the TuringBench benchmark environment, which is comprised of (1) a dataset with 200K human- or machine-generated samples across 20 labels {Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANSFORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two benchmark tasks -- i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards. Our preliminary experimental results using TuringBench show that FAIR_wmt20 and GPT-3 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models. The TuringBench is available at: https://turingbench.ist.psu.edu/

1 Introduction

Neural text generation has advanced rapidly, making machine-written text increasingly difficult to distinguish from human writing. TuringBench addresses the resulting detection gap with datasets, TT and AA tasks, and comparative evaluations.

  • Transformer-based neural architectures have accelerated the generation of long, coherent texts.
  • Model growth has been exponential, from GPT-1’s 117 million parameters to models exceeding one trillion parameters.The passage cites Switch Transformer at 1.6 trillion parameters and Wu-Dao 2.0 at 1.75 trillion parameters.
  • These advances make generated texts easy to misconstrue as human-written, creating an urgent need for accurate automatic detection.
  • TURINGBENCH provides a benchmark environment because sufficient data for building accurate machine-text detectors was lacking.
  • Its dataset contains 200K articles from 10K human-written news articles and 19 neural text-generators, organized across 20 labels.The benchmark includes human and machine texts generated from article titles, with two tasks: Turing Test and Authorship Attribution.
  • The Turing Test is framed as binary human-versus-machine classification, while Authorship Attribution assigns text to one of the studied generators.
  • Comparisons of state-of-the-art and baseline detectors show that more complex models are needed to distinguish generated texts, including future generators.

2 Related Work

Neural text generation now approaches human-written quality while enabling both beneficial applications and misuse. Related detection and authorship-attribution research motivates TuringBench’s benchmark spanning human and neural-model authors.

  • Neural Text Generation: Recent neural language models generate text approaching human quality in grammar, fluency, coherence, and real-world knowledge.
  • Neural Text Generation: Generated text can be misused for fake news, fabricated product reviews, and spam emails.
  • Automatic Detection of Generated Text: Automatic detection research seeks to distinguish machine-generated text from human writing, with much prior work focused on fake news detection.
  • Authorship Attribution: Authorship Attribution identifies an author from candidate texts and supports applications including author profiling, computer forensics, and plagiarism detection.
  • Authorship Attribution: Previous authorship-attribution work largely studied humans, whereas TuringBench benchmarks attribution across humans and neural language models.

3 The TURINGBENCH Environment

TURINGBENCH combines datasets, benchmark tasks, and a leaderboard environment for studying machine-text detection and generator attribution. It uses human and neural-generator articles to define binary Turing Test and multiclass Authorship Attribution settings.

  • 3.1 Chosen Language Models: TURINGBENCH generates articles from 10 language-model architectures, including 19 neural text-generators.The generators require short prompts and a specified word count.
  • 3.2 TURINGBENCH Benchmark Tasks: The Turing Test is a binary human-versus-machine classification task with 19 human-machine subtasks.Each subtask pairs humans with one generator, such as GPT-2 XL or GROVER_base.
  • 3.2 TURINGBENCH Benchmark Tasks: Authorship Attribution is a multiclass task that identifies which neural language model generated a machine-produced article.The benchmark uses one dataset containing all labels for this task.
  • 3.3 TURINGBENCH Benchmark Dataset: After cleaning, 168,612 articles remain; TT uses half-article test inputs, while all datasets use 70:10:20 train-validation-test splits.The benchmark datasets are made publicly available through Hugging Face.
  • 3.4 The Web Environment: The website provides separate task and subtask leaderboards, ranks models by F1 score, and uses private testing to help preserve evaluation integrity.Contributors submit code or trained weights for private testing before scores are updated.

4 Experiments

Experiments compare state-of-the-art and baseline detectors on TT and AA. Fine-tuned BERT-family classifiers outperform several pretrained TT detectors, while FAIR_wmt20 and GPT-3 produce the least distinguishable texts among tested generators.

  • 4.1 Results from Turing Test: BERT and RoBERTa each achieve 0.85 average F1 on TT, compared with 0.56 for GROVER, 0.60 for GPT-2, and 0.57 for GLTR.The comparison uses the average F1 scores reported across TT subtasks.
  • 4.1 Results from Turing Test: BERT and RoBERTa improve TT performance partly because they are fine-tuned on each subtask’s training set, unlike the pretrained TT detectors.The pretrained detectors classify the test sets without further training.
  • 4.1 Results from Turing Test: FAIR_wmt20 and GPT-3 obtain the lowest average F1 scores, 0.49 and 0.55, respectively, across five TT models.XLNET_large has the highest average F1 score at 0.87; its text-padding technique often harms generation quality.
  • 4.1 Results from Turing Test: Human experiments yield average accuracies of 0.535 for identifying machine-generated articles and 0.513 for choosing the machine-generated text between two articles.Workers assessed randomly sampled articles from all 19 language models.

5 Discussion

TURINGBENCH results show that detection and attribution remain difficult: no single detector performs well across all TT tasks, human detection is near chance, and machine styles overlap with human styles. Newer generators are harder to detect, while text length generally has little effect on performance.

  • Task Difficulty: Average F1 scores below 0.87 across human-versus-machine subtasks indicate that both TT and AA remain non-trivial.FAIR_wmt20 achieved the lowest average F1 at 0.49, followed by GPT-3 at 0.55.
  • Detection Models: No single detection model performed well across all 20 TT tasks.BERT had the highest average F1 but underperformed on FAIR_wmt20, where the GROVER detector achieved the highest F1.
  • Human Detection: 0.535 and 0.513 average human accuracies were only slightly above random guessing at 0.5.Humans were tested both on machine-only samples and on pairs containing one human-written and one machine-generated text.
  • Detection Models: GROVER and GPT-2 detectors underperformed on the generators they were trained to detect, while each performed best on other generator families.GROVER performed best on PPLM_distil and PPLM_gpt2, whereas GPT-2 performed significantly better on GPT-1, TRANSFORMER_XL, and XLNET_large.
  • Task Difficulty: No clear correlation existed between text length and F1 score for most models, except RoBERTa.The generated texts ranged from 100 to 400 words, and the authors characterize RoBERTa performance as text-length-dependent.
  • Authorship Attribution: Overlapping LIWC-based PCA features made the 20 authors linearly inseparable, limiting traditional authorship-attribution models.The overlap suggests that machine-generated writing styles are becoming more similar to human-written styles.
  • Authorship Attribution: Human-written features spread broadly across the PCA plot, whereas machine-generated features remained in smaller pockets.The authors interpret this pattern as humans having a wider range of writing styles than machines.

6 Future Work

Future work calls for stronger and more explainable detectors for both Turing Test and Authorship Attribution tasks. It also anticipates authorship obfuscation through mixing outputs from different generators.

  • Future Work: Better models are needed because traditional authorship features alone cannot solve TT or AA, while black-box detectors cannot explain their decisions.The paper points to GLTR-like approaches as a possible way to capture subtler generator-specific patterns.
  • Future Work: Authorship obfuscation may arise when different parts of an article are generated by different neural text-generators.The paper identifies this as a future scenario that could intentionally mask generated writing style and confuse detectors.

7 Conclusion

The paper introduces TURINGBENCH and reports preliminary results for its TT and AA tasks. The results indicate that many language models produce texts indistinguishable from human writing, while computational detectors can perform better than random guessing overall.

  • Conclusion: TURINGBENCH provides an environment for Turing Test and Authorship Attribution evaluation with preliminary experimental results.The conclusion frames the benchmark as a resource for studying both human-versus-machine detection and generator attribution.
  • Conclusion: Many contemporary language models can generate text whose quality is indistinguishable from human-written text to human observers.The conclusion presents this as an overall finding from the benchmark experiments.
  • Conclusion: Computational solutions can differentiate human-written from machine-generated texts better than random guessing overall, though performance varies.The conclusion describes this result broadly rather than claiming uniformly accurate detection.

8 Ethics Statement

TURINGBENCH addresses the urgent need for machine-generated-text detectors while acknowledging that its insights and public dataset could enable misuse.

  • The benchmark’s purpose is to call attention to the urgent need for detectors of machine-generated texts.
  • Publicly released generated political articles could be copied, lightly edited, and posted online as real news.
  • The authors acknowledge that their insights could be used maliciously to thwart detector performance.
  • The authors expect future detectors to identify even human-edited machine-generated texts.

A.1 Data Generation Implementation

Generating TURINGBENCH texts required substantial computational resources, with most generators taking one to three days and PPLM requiring substantially longer runtimes.

  • The authors used Google Colab Pro’s 12GB NVIDIA Tesla K80 GPU for most generation, with additional K80 and P100 GPUs for PPLM.
  • Some generation code was repurposed because it was not written for large-scale generation.
  • 24–72 hours were required for most generators to produce 10K articles.
  • 430 hours were required for PPLM_distil and 600 hours for PPLM_gpt2.

A.2 Data Pre-processing

The appendix describes text cleaning, dataset construction, website implementation, evaluation conventions, and computational settings for TURINGBENCH.

  • A.2 Data Pre-processing: 200K samples were reduced to 168,612 after removing special tokens and consecutive repeated words.
  • A.2 Data Pre-processing: The cleaned dataset’s distribution remained approximately the same as before preprocessing.
  • A.3 TURINGBENCH Website: The website uses the SQuAD framework, hosts task descriptions and leaderboards, and reports standard AA metrics plus F1 for TT.
  • A.2 Data Pre-processing: Human news articles were collected mostly from politics-focused CNN, Washington Post, and Kaggle sources, with 10K used for generation.
  • A.2 Data Pre-processing: TT experiments used default hyperparameters, while BERT and RoBERTa were fine-tuned on the training set.
  • A.2 Data Pre-processing: AA models also used default hyperparameters without hyperparameter tuning, with convergence times ranging from about 15 minutes to 5 hours.
Loading 2109.13296v1…