Source-linked AI summary
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Liam Dugan, Alyssa Hwang, Filip Trhlik, Josh Magnus Ludan, Andrew Zhu, Hainiu Xu, Daphne Ippolito, Chris Callison-Burch
TL;DR
Existing detector evaluations lack shared, sufficiently challenging benchmarks covering diverse models, domains, decoding strategies, and attacks. RAID addresses this gap with a large, systematic benchmark and evaluation of open- and closed-source detectors, finding substantial robustness failures while identifying some promising detectors. The authors argue that detection should be evaluated openly and used cautiously, especially because the benchmark itself can induce overfitting and broad multilingual coverage remains limited.
Problem
Detector claims are difficult to verify because evaluations rarely use shared benchmarks with diverse models, domains, sampling strategies, and adversarial attacks.
Method
RAID evaluates 12 detectors on 6M+ generations spanning 11 generators, 8 domains, 11 attacks, and 4 decoding strategies.
Results
Detectors struggle with unseen models and domains and show marked performance decreases under sampling changes, repetition penalties, and adversarial text modifications.
Takeaways & Limitations
Open evaluation on large, diverse shared resources is critical for improving detection, while harmful-content detection should take precedence over authorship detection.
Takeaways & Limitations
The benchmark has limited multilingual coverage and may encourage overfitting to the robustness dimensions it includes.
Abstract
from arXiv · showhide
Many commercial and open-source models claim to detect machine-generated text with extremely high accuracy (99% or more). However, very few of these detectors are evaluated on shared benchmark datasets and even when they are, the datasets used for evaluation are insufficiently challenging-lacking variations in sampling strategy, adversarial attacks, and open-source generative models. In this work we present RAID: the largest and most challenging benchmark dataset for machine-generated text detection. RAID includes over 6 million generations spanning 11 models, 8 domains, 11 adversarial attacks and 4 decoding strategies. Using RAID, we evaluate the out-of-domain and adversarial robustness of 8 open- and 4 closed-source detectors and find that current detectors are easily fooled by adversarial attacks, variations in sampling strategies, repetition penalties, and unseen generative models. We release our data along with a leaderboard to encourage future research.
1 Introduction
Machine-generated text is increasingly used in harmful activities, creating a need for robust automatic detection. Existing evaluation practices make detector accuracy and robustness difficult to verify, motivating RAID, which tests detectors across diverse models, domains, settings, and attacks.
- LLM-generated text has been used for phishing, spam, harassment, disinformation, and spurious scientific publication.
- Researchers typically evaluate detectors on privately created datasets rather than shared resources, limiting verification of accuracy and robustness.
- RAID contains 6M+ generations spanning 11 generators, 8 domains, 11 adversarial attacks, and 4 decoding strategies.
- Evaluation of 12 detectors shows reduced performance on unseen models and domains and after sampling, repetition-penalty, or adversarial changes.
2 Related Work
Prior work provides datasets and studies targeting selected robustness dimensions, but no comparable resource combines broad domains, sampling strategies, models, and adversarial attacks. RAID synthesizes these dimensions into one systematic benchmark.
- Existing generated-text resources commonly focus on specific areas such as multilingual text, code, question answering, or scientific papers.
- Li et al. (2024) created 447k generations from 7 language-model families across 10 domains to study detector robustness.
- Comparative studies report detector weaknesses across unseen models, domains, decoding strategies, prompts, and repetition penalties.
- Adversarial-robustness studies examine homoglyph, whitespace, sentiment, factual, paraphrase, and synonym-replacement attacks.
- RAID combines many robustness dimensions in one shared benchmark study.
3 Dataset Creation
RAID is constructed by pairing human-written documents with generations across diverse domains, models, decoding strategies, repetition penalties, and black-box adversarial attacks. The resulting dataset is balanced and designed to avoid prompt or generation-style biases.
- RAID samples roughly 2,000 human-written documents from each of 8 domains and generates outputs across 11 models, 4 decoding strategies, and 11 adversarial attacks.
- Prompts: Each human document receives a corresponding generation prompt whose dynamically inserted title identifies the source document.
- Prompts: Prompts use zero-shot Chat or Non-Chat templates and avoid imposing a particular length or generation style.
- Models: The model set varies size, open versus closed source, and chat versus completion style across GPT, open-source, and Cohere models.
- Decoding Strategies: Greedy and fully random decoding are used, with repetition-penalty variants generated when available using θ = 1.2.
- Adversarial Attacks: The adversarial setting assumes one query and no detector knowledge, using 11 black-box, query-free attacks with manually limited mutations.
- After failed generations are removed, the dataset is balanced so each human document has one generation per model, decoding strategy, and attack.
4 Dataset
RAID provides millions of balanced generations and extends coverage with multilingual and code data. Its statistics show systematic differences between model-generated and human text, while its leaderboard separates in-domain from out-of-domain evaluation.
- The non-adversarial dataset contains 509,014 generations and 14,971 human-written documents, totaling 6,287,820 texts.
- Statistics: Generated texts are generally more repetitive and shorter than human texts, with lower mean perplexity under LLaMA 7B and GPT 2 XL.
- The public leaderboard separates detectors trained on RAID from those that were not to distinguish generalization to out-of-domain data.
- The hidden test set contains 10% of RAID without labels and supports submitted detector evaluations against hidden labels.
- RAID-extra adds 2.3M generations from Python Code, Czech News, and German News, and metric-based detectors perform surprisingly well on these domains.
5 Detectors
RAID evaluates neural, metric-based, and commercial detectors under a shared false-positive-rate framework. The study emphasizes detector categories, threshold calibration, and the risks of naive threshold choices.
- Detector categories: The evaluation covers neural, metric-based, and commercial detectors.Neural detectors typically fine-tune language models, metric-based detectors use generative-model probabilities, and commercial detectors restrict model access.
- Detector categories: The benchmark includes four neural, four metric-based, and four commercial detectors.The listed systems include RoBERTa variants and RADAR, GLTR, Binoculars, FastDetectGPT and LLMDet, plus GPTZero, Originality, and Winston.
- Thresholding: Detectors convert a scalar score into a binary prediction using threshold τ.A sequence is classified as machine-generated when its score s ≥ τ.
- Thresholding: Thresholds are calibrated separately for each detector to produce a 5% false positive rate.At this operating point, accuracy measures machine-text detection while misclassifying 5% of human-written text.
6 Findings
RAID findings show that detector performance depends strongly on operating point, decoding configuration, training overlap, and adversarial attack type. Strong results can collapse under modest distribution or text changes.
- Finding 2: Detector accuracy varies substantially depending on target False Positive Rate: Some detectors reach high accuracy only at similarly high false positive rates.As FPR decreases, most detectors drop steeply; Binoculars is particularly strong at low FPR, while some detectors plateau before reaching the lowest tested FPR.
- Finding 3: Repetition penalty drastically hurts accuracy for all detectors: Repetition penalties decrease detector accuracy by up to 32 points across detectors, generators, domains, and decoding strategies.Detectors also perform substantially better on greedy decoding than on random sampling, even when repetition penalties are considered.
- Finding 4: Seemingly strong, robust detectors can perform unexpectedly poorly: Changing the generator, decoding strategy, or repetition penalty can introduce up to 95+% error rates in detectors that initially appear highly accurate.The findings indicate weak generalization across models and generation settings within the same domain.
- Finding 5: Detectors perform better on domains and models seen during training: RoBERTa-Large GPT2 exceeds 95% accuracy on GPT2-generated text in five domains but rarely exceeds 60% on the same domains from other models.RADAR also performs unusually poorly on movie reviews regardless of generative model.
- Finding 6: Different detectors are vulnerable to different types of adversarial attacks: Detector vulnerabilities differ by attack: synonym swaps reduce metric-based accuracy by as much as 36.1%, while GPTZero loses only 0.3% under homoglyph attacks.Five other detectors lose an average of 40.6% under homoglyph attacks, whereas adversarially trained RADAR is more robust.
- Finding 6: Different detectors are vulnerable to different types of adversarial attacks: Some adversarial attacks improve detector accuracy when they make text more similar to a detector’s training data.RoBERTa GPT2 improves after T5 paraphrasing and BERT-based synonym replacement.
7 Conclusion
RAID finds that current detectors are not robust enough for widespread or high-stakes deployment, despite some promising detector-specific results. The authors argue that harm-focused analysis and open, diverse evaluation should complement machine-text detection.
- Deployment implications: Many tested detectors are nearly inoperable at low false positive rates and fail under alternative decoding, repetition penalties, model or domain shifts, and simple black-box attacks.The conclusion combines the benchmark’s operating-point, generalization, and adversarial findings.
- Promising directions: Binoculars performs well across models at extremely low false positive rates, while Originality and GPTZero show strengths in constrained settings.Originality achieves high precision in some constrained scenarios, and GPTZero is unusually robust to adversarial attacks.
- Harm-focused detection: Detecting misinformation, hate speech, and abuse should take precedence over determining whether text was machine-authored.The authors frame machine-text detection as one tool within a broader effort to reduce harm.
- Future evaluation: Future shared resources should include more models, languages, and generation settings.The authors present RAID as a step toward safely integrating detection tools into a multi-pronged harm-reduction approach.
Limitations
RAID’s robustness coverage is necessarily incomplete and will require expansion and maintenance as languages, models, and attacks evolve. Public evaluation also risks overfitting to the benchmark, though hidden data and updated releases may mitigate this risk.
- RAID cannot provide a truly comprehensive robustness dataset, particularly because multilingual text is limited largely to the news domain.RAID-extra begins expanding coverage but includes multilingual text only in news.
- Generated-text datasets will become obsolete as language modeling improves, requiring new generations and updated detector evaluations.The authors plan to release updated versions periodically.
- Public out-of-domain benchmarks can induce overfitting as practitioners specialize to the robustness dimensions they cover.The authors retain hidden test data and plan regular releases to reduce this risk.
Ethics Statement
The authors warn that detector errors can cause serious harm, especially in disciplinary or punitive settings, and argue that detectors should not be used there until evaluation standards improve. They also describe RAID-extra’s additional domains and multilingual data-generation choices.
- False positives can cause significant harm in disciplinary or punitive contexts, especially given documented bias against non-native English writers.The authors state that false positives remain unsolved.
- The authors oppose using detectors for disciplinary or punitive action until better evaluation standards are widely adopted.They argue that poorly calibrated detectors can cause more harm than they solve.
- RAID-extra adds 2.3M generations in Python Code, Czech News, and German News, but these domains are outside the main benchmark’s comparison scope.The authors present the extra experiments as exploratory insights into detector performance.
- For multilingual prompts, native speakers wrote prompts directly in the target language to improve generative-model adherence.This approach followed prior work and was judged most effective by the authors.
- Python Code generations retain only text inside triple backticks, excluding natural-language explanations from detector inputs.This domain-specific processing prevents descriptions of code from becoming detection cues.
A.2 Results
On RAID-extra, metric-based detectors generally outperform neural and commercial detectors in unusual domains, while strong metric-based performance can extend across generative models in Czech News. The evaluation reports machine-text accuracy at fixed FPR because F1 and precision can misrepresent high-risk detection performance.
- Metric-based classifiers generally outperform neural and commercial detectors on RAID-extra’s Python Code, Czech News, and German News domains.Table 7 reports detector accuracy at FPR=5%.
- Binoculars performs decently on Czech News despite Falcon 7B having substantially more German than Czech training data.The authors suggest highly multilingual models may bootstrap metric-based detectors for low-resource languages.
- Fixed-FPR accuracy is used because it directly reflects detected generations under a tolerated false-positive rate in high-risk settings.The authors argue F1 treats false positives and false negatives as equivalent, unlike these settings.
- A roughly 40:1 generated-to-human text ratio causes precision to favor true positives and over-represent detector capabilities on RAID.The authors note that real-world text mixtures reverse this ratio.
C Per-Domain Threshold Tuning
RAID tunes detector thresholds separately for each domain to maintain FPR=5%, addressing substantial domain-level variation hidden by aggregate FPR. It also standardizes submission through a public leaderboard, hidden test split, and package-based evaluation workflow.
- C Per-Domain Threshold Tuning: Per-domain threshold tuning targets 5% FPR separately for every detector and domain rather than only in aggregate.The authors present this added complexity as necessary for fair comparison.
- C Per-Domain Threshold Tuning: At a single threshold with total FPR=5%, RADAR reaches 20.4% FPR on Reviews, GLTR 33.4% on Recipes, and Originality 13% on Wikipedia.These values illustrate substantial domain-specific weaknesses.
- C Per-Domain Threshold Tuning: Uneven domain FPR creates a dampening effect in which weaker domains lower total accuracy, including on more common domains.Per-domain thresholds are intended to avoid this effect.
- D Leaderboard and Pypi Package: RAID’s public leaderboard and PyPI package provide a shared interface for running detectors, submitting predictions, and comparing results.Submissions use metadata and pull requests, with automated evaluation on the leaderboard.
- D Leaderboard and Pypi Package: The leaderboard evaluates submissions on a 10% RAID test split released without labels.This hidden test set supports fair competition and shared evaluation.
- Dataset Scope: RAID spans diverse domains including factual, reasoning, creative, conversational, and media-specific text, with additional code and multilingual news domains.The domain set includes Wikipedia, abstracts, recipes, Reddit, poetry, books, reviews, BBC News, Python Code, Czech News, and German News.
E.2 Generative Models
RAID uses 11 generative models spanning open- and closed-source systems, with prompts and model-specific procedures designed to produce diverse outputs. It evaluates detectors including model-based, metric-based, and commercial systems.
- Generative models: LLaMA 2 70B, Mistral 7B, MPT 30B, and Cohere Command provide additional open- and closed-source model variation.Cohere Command’s current parameter count and training data are unknown, and its API endpoints were queried in November 2023.
- Prompting: Prompts use domain-specific templates while avoiding bias toward a particular generation length or style and are refined through manual review.The review targeted degenerate repetition, meta-commentary, and other signs of generated output; continuation and chat models received different prompt adjustments.
- Detectors: The detector comparison includes RoBERTa GPT2, RADAR, FastDetectGPT, LLMDet, GPTZero, and ZeroGPT.The evaluated systems include fine-tuned detectors, proxy-perplexity or metric-based methods, and commercial APIs.
G.1 Dataset Statistics and Evaluations
The extended RAID evaluations show persistent differences between human and generated text, but detector performance varies substantially across models, domains, decoding strategies, and attacks. The results also reveal detector-specific vulnerabilities and occasional improvements after attacks.
- Cross-model and cross-domain evaluation: RoBERTa GPT2 performs strongly on GPT2 generations but poorly on many generations from different models, while RADAR performs unusually poorly on IMDb Movie Reviews.The extended heatmaps confirm the model- and domain-specific trends reported in the main evaluation.
- Cross-domain evaluation: Metric-based methods such as Binoculars and FastDetectGPT generalize well across domains, although detectors can perform surprisingly poorly on unseen domains.Table 14 reports detector accuracy at FPR=5% across domains.
- Decoding and model variation: Certain combinations of models and decoding strategies cause detector accuracy to plummet unexpectedly, raising concerns for robust deployment.Table 15 also indicates that base models are harder to detect than chat fine-tuned models and that metric-based methods show cross-model generalization.
- Adversarial attacks: Adversarial effects depend on the detector and attack: some attacks reduce accuracy, some improve it, and zero-width spaces can produce all-positive or all-negative labels.Table 16 summarizes accuracy at FPR=5% across alternative spelling, homoglyph, paraphrase, synonym, whitespace, and other attacks.
- Output variation: Different generative models exhibit significantly different output styles, underscoring the difficulty of detecting generated text.Example outputs are provided for each model and adversarial attack.