PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI-Generated Text be Reliably Detected?

Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, Soheil Feizi

arXiv:2303.11156v4cs.CLcs.AIcs.LG

TL;DR

Reliable AI-text detection matters because LLM outputs can be misused, yet existing detectors may be vulnerable to attackers. The paper recursively paraphrases AI text, tests spoofing against watermarked systems, and develops a distributional theory of detection hardness; attacks sharply reduce detection while often preserving text quality.

  • Problem

    Reliable AI-text detection is needed to address misuse, but existing detectors have mainly been shown effective only in specific settings.

  • Method

    The paper stress-tests four detector classes with recursive paraphrasing, studies watermark spoofing without white-box access, and relates optimal AUROC to total variation distance.

  • Results

    Recursive paraphrasing sharply reduces detector performance while causing only slight quality degradation in many cases, and attackers can infer hidden watermark signatures.

  • Takeaways & Limitations

    Current AI-text detectors can be vulnerable to both evasion and spoofing attacks, complicating reliable detection as language models advance.

  • Takeaways & Limitations

    The theoretical argument assumes that more advanced LLMs mimicking human text produce smaller total variation distance, whose empirical estimation from finite samples is extremely challenging.

Abstract

from arXiv · show

Large Language Models (LLMs) perform impressively well in various applications. However, the potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use. Consequently, the reliable detection of AI-generated text has become a critical area of research. AI text detectors have shown to be effective under their specific settings. In this paper, we stress-test the robustness of these AI text detectors in the presence of an attacker. We introduce recursive paraphrasing attack to stress test a wide range of detection schemes, including the ones using the watermarking as well as neural network-based detectors, zero shot classifiers, and retrieval-based detectors. Our experiments conducted on passages, each approximately 300 tokens long, reveal the varying sensitivities of these detectors to our attacks. Our findings indicate that while our recursive paraphrasing method can significantly reduce detection rates, it only slightly degrades text quality in many cases, highlighting potential vulnerabilities in current detection systems in the presence of an attacker. Additionally, we investigate the susceptibility of watermarked LLMs to spoofing attacks aimed at misclassifying human-written text as AI-generated. We demonstrate that an attacker can infer hidden AI text signatures without white-box access to the detection method, potentially leading to reputational risks for LLM developers. Finally, we provide a theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions. This analysis offers insights into the fundamental challenges of reliable detection as language models continue to advance. Our code is publicly available at https://github.com/vinusankars/Reliability-of-AI-text-detectors.

1 Introduction

AI-text detectors are increasingly important but may fail when attackers paraphrase AI text or spoof watermark signatures. This paper stress-tests multiple detector classes, measures quality trade-offs, analyzes spoofing, and connects optimal detection to distributional similarity.

  • Attacks and detector robustness: Recursive paraphrasing can substantially reduce detection while preserving much of the source text’s quality.Human evaluations rated 77% of recursively paraphrased passages high for content preservation and 89% high for grammar or text quality.
  • Attacks and detector robustness: A recursive paraphrasing attack drops watermarked-text detection from 99.3% to 9.7% at TPR@1%FPR on 300-token passages.
  • Attacks and detector robustness: After paraphrasing, zero-shot detector AUROC falls from 96.5% to 25.2%, while RoBERTa-Large-Detector TPR@1%FPR drops from 100% to 60%.The retrieval-based detector’s accuracy also falls from 100% to below 60%.
  • Scope and contributions: The paper examines watermarking-, neural-network-, zero-shot-, and retrieval-based detectors under an attacker.Its recursive paraphrasing attack is presented as a comprehensive robustness stress test across four detector classes.
  • Spoofing attacks: Spoofing attacks can make human-written text appear AI-generated by inferring hidden watermark signatures without white-box access.The attacker probes a watermarked LLM and analyzes generated-token statistics to estimate its watermarking scheme.
  • Theoretical hardness: The paper links the AUROC of the best detector to the total variation distance between human and AI-text distributions.The theory also identifies a tradeoff between type-I and type-II errors.

2 Evading AI-Detectors using Paraphrasing Attacks

The paper evaluates recursive paraphrasing as an automated attack against multiple AI-text detectors, measuring evasion alongside text-quality changes. Across watermarking, zero-shot, trained, and retrieval-based detectors, recursive paraphrasing substantially reduces detection while often causing only modest quality degradation.

  • Attack Setup and Paraphrasing Methods: The attack repeatedly paraphrases an AI response with a neural paraphraser to evade detection while preserving an automated, scalable workflow.DIPPER and LLaMA-2-7B-Chat perform strong paraphrasing recursively; the study evaluates quality using perplexity, human ratings, and question-answering accuracy.
  • Paraphrasing Attacks on Watermarked AI Text: After five rounds, recursive paraphrasing reduces watermark detection below 20%, while two rounds suffice to push TPR@1%FPR below 50% in the reported settings.With Best of ppi on OPT-1.3B, detection falls from 99.8% to 4.0% with a perplexity increase of 1.5.
  • Paraphrasing Attacks on Non-Watermarked AI Text: DetectGPT AUROC drops from 96.5% before attack to 59.8% after attack, while OpenAI’s RoBERTa-Large-Detector TPR@1%FPR falls from 100% to around 92%.The paper reports that other zero-shot detectors also perform poorly after the attack, and trained detectors degrade significantly although they remain comparatively more robust.
  • Paraphrasing Attacks on Non-Watermarked AI Text: A key scope limitation is that trained detectors’ apparent robustness may reflect overlap between their training distribution and the study’s human-written passages.The paper does not consider out-of-distribution human text, where prior work reports that some trained detectors fail to generalize.
  • Paraphrasing Attacks on Non-Watermarked AI Text: Retrieval-based detection accuracy drops below approximately 60% after five recursive paraphrasing rounds, with only a 1.7-point perplexity degradation at that accuracy.The detector was designed to resist paraphrase attacks, but recursive paraphrasing still reduces its accuracy; storing users’ LLM conversations also raises privacy concerns.

3 Spoofing Attacks on Generative AI-text Models

The paper shows that AI-text detectors can be manipulated to misclassify human-written text as AI-generated, including through black-box attacks on watermarking and retrieval systems.

  • Spoofing attacks can cause detectors to falsely label human-written passages as AI-generated, potentially damaging LLM developers’ reputations.The attack scenario includes adversarially composing derogatory texts that detectors classify as AI-generated.
  • Watermarking signatures can be inferred by observing token frequencies from watermarked outputs, without white-box access to the detector.The attack estimates green lists for 181 common words from observed token transitions.
  • 99.8% to 1.3%: spoofing reduces the soft watermarking detector’s AUROC using estimated green lists.
  • Retrieval-based detectors can be spoofed even with a private database by submitting an LLM paraphrase of a human document for storage.The detector may then classify the original human document as AI-generated because a semantically similar copy is stored.
  • Strengthening retrieval-based detectors against type-II errors can create a tradeoff with type-I errors, including false accusations of human authors.

4 Hardness of Reliable AI Text Detection

The paper develops a theoretical connection between detector AUROC and the Total Variation distance between human and AI-text distributions, then tests how model scaling affects that distance. The theory and experiments indicate that increasingly similar distributions make reliable detection harder.

  • Theorem 1 bounds the AUROC of any detector using the Total Variation distance between human and AI-text distributions.A detector maps text sequences to real scores and thresholds them to produce an ROC curve.
  • As the Total Variation distance decreases, the AUROC of the best possible detector also decreases.
  • AUROC > 0.9 requires TV > 0.5, whereas TV < 0.2 implies AUROC < 0.7 for the best possible detector.These values are reported as theoretical readings from the paper’s bound and figure.
  • For watermarked models, the bound can remain close to one because watermarking may create a large TV distance from human text.The paper also discusses paraphrasing as a way to evade watermarks through this framework.
  • More advanced LLMs are hypothesized to mimic human text more closely, producing smaller TV distances and increasing detection difficulty.The paper identifies this relationship as an assumption underlying its hardness result.
  • TV distance consistently decreases with increasing model size across all evaluated meta-token sequence lengths.The estimates use GPT-2 outputs compared with WebText and are averaged over 30 runs.
  • The synthetic-data experiments provide empirical evidence that more advanced LLMs can produce smaller TV distances, making reliable detection increasingly difficult.

A Experiments with More Datasets and Models

The experiments evaluate recursive paraphrasing across multiple datasets and target language models to test robustness under distribution shifts.

  • The study evaluates attacks on XSum, PubMedQA, and Kafkai using OPT-1.3B and GPT-2-Medium as target language models.
  • PubMedQA provides medical text, while Kafkai contains real and privately generated fake articles spanning ten domains.

A.1 Additional Experiments with Llama-2-13B

Additional experiments test watermarking, zero-shot, trained, and retrieval-based detectors with Llama-2-13B and recursive paraphrasing. Across these settings, repeated paraphrasing substantially reduces detection performance while often only slightly affecting perplexity or text quality.

  • A.1 Additional Experiments with Llama-2-13B: Llama-2-13B is the target model, while a smaller Llama-2-7B performs recursive paraphrasing on XSum passages.
  • Watermarking: Five recursive paraphrasing rounds take approximately 36 seconds per 300-token passage in the watermarking experiments.The attack uses the 11B-parameter DIPPER paraphraser.
  • Watermarking: After two rounds, watermarking detection falls below 50% TPR@1%FPR in every setting; selecting the worst paraphrase lowers it below 10%.
  • Zero-shot and trained detectors: Paraphrasing causes substantial performance drops for zero-shot and trained detectors, and five detector queries can fool most detectors effectively.OpenAI’s RoBERTa-based detectors are more resilient on XSum but unreliable on Kafkai.
  • Retrieval-based detectors: The retrieval-based detector’s accuracy approaches 50% after five rounds using the worst paraphrase, with only slight perplexity degradation.It remains effective after one paraphrasing round before degrading under repeated attacks.

B.1 Human Evaluation Study on Paraphrases

The study evaluates recursive paraphrases for content preservation and text quality, while examining their effect on detector performance. Human ratings indicate that paraphrases often retain quality, revealing a tradeoff between attack strength and watermark detection.

  • Human evaluation: Human evaluation uses 20 randomly selected watermarked passages and five recursive paraphrasing rounds for each paraphraser.
  • Human evaluation: 70% of DIPPER paraphrases and 83% of LLaMA-2 paraphrases receive high content-preservation ratings.
  • Human evaluation: 89% of DIPPER paraphrases and 88% of LLaMA-2 paraphrases receive high text-quality or grammar ratings.
  • Detector evaluation: Figure 12 compares detector ROC performance before paraphrasing, after paraphrasing, and after paraphrasing with multiple detector queries.

C.1 Proof of Theorem 1

Theorem 1 bounds the ROC area of any detector using the Total Variation distance between model-generated and human text distributions. The proof derives this bound by relating true-positive and false-positive rates at each classifier threshold.

  • Theorem 1: Theorem 1 bounds the area under any detector’s ROC curve.
  • Proof: The proof defines TPR and FPR through the ROC curve and indexes them by a classifier parameter γ.
  • Proof: The difference between TPR_γ and FPR_γ is bounded by the Total Variation distance between model and human text distributions.
  • Proof: Because TPR_γ is also bounded by 1, the proof combines the distribution-distance bound with the unit upper bound on true-positive rates.
  • Proof: The proof then integrates the resulting bound over false-positive rates to obtain an AUROC bound.

C.2 General Trade-offs For Detection

The paper derives general detection trade-offs from the similarity between AI and human text distributions. These results apply to paraphrasing and watermarking, linking detector reliability to Total Variation distance.

  • General trade-offs: The framework applies to particular writing styles and paraphrasing by defining model and human distributions for the relevant setting.
  • General trade-offs: Corollary 1 gives an AUROC bound for a detector in terms of the Total Variation distance between model and human text distributions.
  • Watermarking trade-off: For watermarking, Corollary 2 relates the probability of watermark detection for paraphrased model outputs and human rephrasings.
  • Watermarking trade-off: When paraphrased model and human distributions are similar, either some human writing is falsely detected as watermarked or the paraphraser can remove the watermark.
  • Detection trade-off: Corollary 3 states an analogous trade-off for AI-text detectors: similar model and human distributions imply either false AI detections or reduced detection of AI text.
  • Related work: Earlier work derives a Total Variation AUROC bound assuming several independent samples, which may be impractical because document sentences are often correlated.

C.3 Tightness Analysis for Theorem 1

The paper shows that the Theorem 1 bound is tight by constructing a model distribution and detector whose AUROC reaches the bound. The construction uses sublevel sets of the human text density.

  • Tightness construction: The tightness analysis constructs a model distribution M and detector D for a given human-generated text distribution H.
  • Distribution construction: The construction assigns model probability TV(M, H) to the human-density sublevel set Ω_H(0).
  • Distribution construction: Outside the lowest-density region, the model density matches the human density over a set containing human probability 1 − TV(M, H).
  • Detector construction: The hypothetical detector scores each sequence using the negative human probability density, D(s) = −pdf_H(s).
  • Tightness result: Under this construction, TPR_γ equals min(FPR_γ + TV(M, H), 1), matching the theorem’s ROC relationship and establishing tightness.

C.4 Pseudorandomness in LLMs

The paper connects detector performance to total variation distance and examines how pseudorandomness affects this theoretical relationship. Experiments suggest that larger LSTM networks produce smaller estimated TV distances, while watermarking remains vulnerable to spoofing.

  • Computational Hardness Result: A high true TV distance can permit near-perfect theoretical separation while remaining difficult for efficiently computable detectors.The paper distinguishes information-theoretic separability from computational detectability.
  • Computational Hardness Result: The pseudorandomness correction ϵ is orders of magnitude smaller than the other bound terms and can be safely ignored.Common pseudorandom generators can make ϵ negligible as a function of the seed length.
  • Computational Total Variation Distance: Computational total variation distance restricts the comparison to polynomial-time computable {0, 1}-functions, including thresholded detector outputs.The function class can also be represented by polynomial-size circuits, which may better match neural-network detectors.
  • Computational Hardness Result: Theorem 2 bounds the AUROC of any polynomial-time detector for pseudorandom AI text using the TV distance between human text and truly random AI text.The bound includes a negligible correction term accounting for pseudorandomness.
  • Empirical TV-Distance Experiments: TV distances consistently decrease as LSTM network size increases across experiments varying vocabulary size and sequence length.Figure 15 reports TV distances for synthetic text settings with different vocabulary sizes, sequence lengths, and LSTM sizes.
  • Watermark Spoofing: Soft watermarking can be spoofed by inferring green lists and composing human-written text that contains many green-list tokens.One proof-of-concept adversarial-human sentence contains 42.6% green-list tokens and is evaluated with a z-test threshold of 4.

D.2 Zero-Shot and Trained Detectors

The paper evaluates detector behavior after spoofing attacks using operating-point metrics and ROC curves. Most evaluated detectors degrade after spoofing, with lower detection rates and higher false-positive rates, while the paper also notes broader practical risks and scope boundaries.

  • Spoofing Results: Most detection methods show increased false-positive rates at a fixed 90% true-positive rate after spoofing.The reported operating points are F@T and T@F, with before-attack values shown in parentheses in Table 16.
  • Spoofing Results: After naïve spoofing, most detectors show significant drops in true-positive rate at 1% false-positive rate and in AUROC.Figure 17 compares ROC curves before and after the attack and reports quality degradation for most detectors.
  • Study Scope: The paper stress-tests watermarking, neural-network, zero-shot, and retrieval-based detectors in the presence of an attacker.Recursive paraphrasing is presented as an evasion attack against recently proposed watermarking and retrieval-based detectors.
  • Limitations: Existing paraphrasers may perform less well in specific technical domains such as clinical text data.The paper notes that stronger future paraphrasers might overcome this limitation.
  • Practical Consequences: Reliable detection becomes harder to assess when false positives can falsely accuse students or damage developers’ reputations.The paper identifies misidentification costs as a practical concern for AI-text detectors.
Loading 2303.11156v4…