PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

Evan Crothers, Nathalie Japkowicz, Herna Viktor

arXiv:2210.07321v4cs.CLcs.CRcs.CYcs.LG

TL;DR

Machine-generated text is increasingly capable, accessible, and difficult to distinguish from human writing, creating diverse abuse risks and detection challenges. The paper surveys detection methods, develops threat models, and integrates Trustworthy AI considerations; it concludes that existing defenses are often inadequate for realistic, widely available NLG threats and that major technical and social problems remain open.

  • Problem

    Growing NLG capabilities and access create diverse abuse risks, while existing detection research has limited coverage of threat models, realistic settings, transparency, and fairness.

  • Method

    The paper combines a comprehensive survey of machine-generated-text detection with detailed threat modeling and sociotechnical analysis grounded in Trustworthy AI.

  • Results

    Current domain-specific defenses are not adequate for the vast majority of upcoming threat models, and detection methods often omit unknown architectures, class imbalance, transparency, and fairness.

  • Takeaways & Limitations

    Suitable defenses require coordinated technical and social effort among AI researchers, cybersecurity professionals, and non-technical experts.

  • Takeaways & Limitations

    Adversarial attacks can degrade text quality, fluency, meaning, and credibility, constraining their usefulness to attackers seeking convincing text.

Abstract

from arXiv · show

Machine generated text is increasingly difficult to distinguish from human authored text. Powerful open-source models are freely available, and user-friendly tools that democratize access to generative models are proliferating. ChatGPT, which was released shortly after the first edition of this survey, epitomizes these trends. The great potential of state-of-the-art natural language generation (NLG) systems is tempered by the multitude of avenues for abuse. Detection of machine generated text is a key countermeasure for reducing abuse of NLG models, with significant technical challenges and numerous open problems. We provide a survey that includes both 1) an extensive analysis of threat models posed by contemporary NLG systems, and 2) the most complete review of machine generated text detection methods to date. This survey places machine generated text within its cybersecurity and social context, and provides strong guidance for future work addressing the most critical threat models, and ensuring detection systems themselves demonstrate trustworthiness through fairness, robustness, and accountability.

1 INTRODUCTION

Machine-generated text is becoming more capable and accessible, expanding abuse risks while complicating detection. This survey responds with comprehensive detection coverage, threat modeling, and trustworthy-AI considerations.

  • 1.1 Risks of Machine Generated Text: Transformer-based NLG advances have improved the diversity, controllability, quality, speed, and efficiency of machine-generated text.These advances create detection challenges across phishing, disinformation, fraudulent reviews, academic dishonesty, and toxic spam.
  • 1.1 Risks of Machine Generated Text: Open-source models and user-friendly interfaces have broadened access to powerful generative systems beyond sophisticated threat actors.Examples include GPT-J, GPT-NeoX, BLOOM, ChatGPT, and GPT-3-powered writing tools.
  • 1.1 Risks of Machine Generated Text: A GPT-J deployment generated roughly 10% of activity on 4chan’s /pol/ board during a 24-hour peak period.The model was trained on the same message board and produced numerous objectionable posts.
  • 1.2 Survey Overview: The survey extends prior reviews by covering feature-based detection, contemporary generative models, threat models, and defensive research.Feature-based methods may improve robustness against neural-network adversarial attacks or enhance explainability.
  • 1.2 Survey Overview: The survey integrates sociotechnical and human-centric considerations, including transparency, fairness, accountability, and trust impacts of generation and detection systems.Its stated contributions include the most complete detection survey, the first detailed review of machine-generated-text threat models, and a Trustworthy AI perspective.

2 MACHINE GENERATED TEXT

The survey defines machine-generated text broadly and situates it within a diverse NLG landscape. It outlines major tasks and approaches, emphasizing Transformer models while retaining relevant pre-Transformer methods.

  • 2 MACHINE GENERATED TEXT: Machine-generated text is natural-language text produced, modified, or extended by a machine.The definition excludes non-natural languages such as logical and programming languages.
  • 2 MACHINE GENERATED TEXT: The survey focuses on attacks that leverage machine-generated text, typically by attempting to pass it as human-authored text, rather than attacks against NLG models themselves.Text adversarial attacks are excluded when producing new text is not their primary goal.
  • 2.3 Natural Language Generation Approaches: NLG approaches include rule-based, statistical, recurrent-neural, adversarial, and Transformer-based methods.The survey provides a high-level taxonomy as context for its threat-model and detection analyses.

3 THREAT MODELS

Machine-generated text enables attacks against systems, individuals, and communities. The survey uses threat modeling to identify attackers, capabilities, objectives, vulnerabilities, and mitigation needs.

  • 3 THREAT MODELS: Machine-generated text can support attacks targeting computer systems, individuals, or specific communities.The cited objectives include system compromise, financial exploitation, and large-scale harassment.
  • 3 THREAT MODELS: Trustworthy NLG requires accounting for unintended and dual-use applications and considering detection, moderation, legislation, and platform-policy responses.The survey also considers how attacks and mitigation measures affect broader trust.
  • 3 THREAT MODELS: The survey’s threat modeling process identifies potential attackers, their capabilities, and objectives to improve system security against major threats.It uses this analysis to organize risks motivating machine-generated-text detection research.

3.1 Threat Modeling Fundamentals

The survey applies an attack-centric, societal threat-modeling process to machine generated text, identifying actors, objectives, capabilities, mitigations, and trust impacts. It organizes attacks into four broad categories while treating the analysis as iterative and non-exhaustive.

  • Threat modeling approach: The analysis focuses on machine generated text as part of attacks against society rather than vulnerabilities in one specific IT system.It identifies several technological systems within a broader societal supersystem.
  • Threat modeling approach: Threat actors vary in sophistication and motives, but each exploits a natural language generation model.The survey characterizes attackers while describing each attack.
  • Threat modeling approach: For each threat model, the survey identifies systems at risk, attackers, objectives, capabilities, and possible mitigation measures.Mitigations may include machine generated text detection, policy changes, and human moderation.
  • Threat modeling approach: The survey evaluates the trust impacts of both attacks and their proposed mitigations.This extends the analysis beyond whether a mitigation improves security.
  • Scope and organization: The taxonomy groups the threat-model analysis into four major attack categories, while acknowledging that future malicious applications cannot be exhaustively listed.The authors present the categories as a foundation for further defensive research.

3.2 Facilitating Malware and Social Engineering

Machine generated text can scale and personalize phishing, social worms, and dataset-poisoning attacks. The survey therefore considers automated detection alongside reporting, awareness, access controls, screening, versioning, and auditing, while emphasizing resulting trust risks.

  • Phishing and social engineering: NLG can scale and personalize phishing or scam campaigns, with demonstrated effectiveness for email masquerade and community-targeted phishing.Generated messages may be target-specific rather than identical across recipients.
  • Phishing and social engineering: Existing phishing mitigations include automated detection, user reporting, and awareness campaigns, although varied generated content may challenge detection systems.Attackers may still need to include payload content such as a unique shortlink.
  • Phishing and social engineering: NLG may generate social-worm messages that include personal details, imitate a contact’s writing style, or continue a short conversation.Compromised accounts can use such messages to propagate exploits through social-media or email networks.
  • Dataset poisoning: Dataset poisoning can target downstream machine-learning systems or directly manipulate models for later financial or operational benefit.Examples include malware detection, spam filtering, and algorithmic trading models.
  • Dataset poisoning: Mitigations for poisoning range from IT security and pre-training data screening to data versioning and audit logging.Screening may use machine generated text detectors or cluster-based poisoning detection when training data cannot be protected from access.
  • Trust impacts: Large-scale phishing and social-worm attacks may reduce trust in text communications, while poisoning controls may undermine trust between developers and organizations.Recipients may become suspicious even of seemingly innocuous messages.

3.3 Online Influence Campaigns

The survey examines political and commercial influence campaigns that use machine generated text to promote ideas or actions at scale. It emphasizes social-media targeting, fraudulent reviews, multimodal mitigation, and the risk that detection itself can damage trust and discriminate against legitimate users.

  • Campaign scope: Influence campaigns use machine generated text to promote political or commercial ideas and prompt actions among target audiences.Political objectives include disinformation and election interference; commercial objectives include promotion, competitor smearing, and fake reviews.
  • Political influence campaigns: Social media is a valuable and vulnerable target because its large user base offers many avenues for scaling and targeting machine generated content.The survey distinguishes social-message scaling from news-article scaling, which may require hosting infrastructure and does not multiply effectiveness by article count.
  • Political influence campaigns: Few-shot language-model generation can produce conspiratorial content and additional on-message political posts from examples of desired messaging.The survey connects this capability to influence operations based on prior campaign material.
  • Political influence campaigns: Automated detection and platform moderation are proposed for political influence operations because user reports may be inadequate.Prior research found users accepted connection requests from deepfake profiles 79%–85% of the time.
  • Commercial influence campaigns: Machine generated text can produce fraudulent reviews that promote one’s own product or target a competitor, including sentiment-preserving reviews.Such campaigns may operate on marketplaces, social media, or through commercial fake-review services.
  • Commercial influence campaigns: Fake-review mitigation can combine text detection with context-based signals such as usernames, account creation times, and unusual purchase behavior.The survey notes that advanced NLG should not affect these context-based detection methods.
  • Trust impacts: Influence operations and their detection can reduce trust online, including through mass suppression or erroneous detection of political text written by non-native English speakers.The survey recommends careful training and ethical evaluation, with detection used alongside other account and activity features.

3.4 Exploiting AI Authorship

Machine-generated authorship threatens academic publishing, applications, and platform trust, while detection and disclosure create fairness and enforcement challenges.

  • Academic Fraud: AI-generated scientific papers can waste reviewing resources, lower publication standards, and produce misleading content that sometimes passes peer review.SCIgen demonstrated this problem as early as 2005, and similar papers continue to appear despite its relatively simple generation method.
  • Academic Fraud: Threat actors use generated papers either to inflate publication statistics or to probe potentially disreputable conference standards.The surveyed capabilities include both established tools such as SCIgen and newer Transformer-based writing assistants.
  • Applications and Cover Letters: Machine-generated cover letters and essays can be detected more reliably when they are sufficiently long, but AI assistance may also help multilingual applicants express themselves.The passage cautions that detection may not distinguish exploitative mass submissions from legitimate translation or writing assistance.
  • Applications and Cover Letters: Large-scale AI-generated submissions may dilute platform quality, undermine trust, create plagiarism concerns, and overwhelm employers, reviewers, or editorial staff.Potential responses include disclosure requirements, content restrictions, and alternative evaluation methods, but enforcement may require detection and moderator investigation.
  • Impacts of Attacks and Mitigation on Trust: Detection can unfairly produce false positives for people with certain language backgrounds or for permissible uses such as translation and assistive writing.Erroneous screening may reduce perceived fairness in applications and weaken confidence in scientific review.

3.5 Spam and Harassment

Spam and harassment threat models center on high-volume machine-generated content that disrupts platforms, targets communities, and erodes confidence that online participants are human.

  • Spam: A GPT-2 bot generated 55.3% of comments on a federal public-comment website before its operator withdrew them.High-quality NLG makes existing spam harder to detect because generated posts can be unique and stylistically appropriate.
  • Harassment: Generative models can produce large volumes of hateful spam and automated harassment targeting individuals, identity-based communities, and political groups.Motivations may range from personal grudges to political objectives, while open-source models lack the restrictions applied by some hosted APIs.
  • Mitigation: Spam and harassment attacks often exploit delivery and identity weaknesses rather than requiring machine-generated text itself to be blocked.Recommended defenses include CAPTCHA, human verification, and scrutiny of proxy and VPN usage.
  • Volume Attacks: High-volume submissions can overwhelm recruiting, conference, and editorial processes with unique cover letters, resumes, papers, or op-eds that do not represent real individuals.The attack scales through submission volume and can frustrate attempts to identify genuine participants.
  • Impacts: Frequent exposure to automated posts can reduce trust in the integrity of online communities and in AI model development more broadly.Concern persisted on 4chan after the GPT-4chan bot was deactivated because later posts might also be generated.
  • Mitigation: Stronger human verification can reduce online privacy and raise participation barriers, even as it mitigates automated abuse.Possible measures include additional signup information, payment methods, government identification, and more frequent CAPTCHA challenges.

3.6 Summary of Threat Models

The survey identifies scalable, targeted abuse and growing machine-generated submissions as central threat-model concerns, while noting that existing platform defenses still matter.

  • NLG models can improve the scale and targeting of existing attacks.
  • Platforms accepting text submissions are likely to face increasing machine-generated content as user-friendly generation tools proliferate.
  • Research often emphasizes AI-generated news, although sociological data suggest machine-generated comments may pose a greater threat.
  • NLG may make coordinated inauthentic activity harder to detect, but abuse often still requires bypassing defenses such as IP reputation checks and CAPTCHA.

4 DETECTION OF MACHINE GENERATED TEXT

Machine-generated text detection is commonly treated as human-versus-machine classification, using feature-based and neural approaches across domains and human review settings.

  • Detection is typically framed as binary classification distinguishing machine-generated from human-generated text, with attribution to the generating model as a related task.
  • Feature-Based Approaches: Feature-based detectors transform input sequences into linguistic or statistical vectors and classify them with SVM, random forest, or neural-network algorithms.
  • Feature-Based Approaches: Different sampling methods create different artifacts, reducing feature-based performance when detector training and target generation use mismatched sampling approaches.Detectors trained on smaller models can detect larger-model outputs, while training on larger models is more effective for smaller-model outputs.
  • Frequency Features: Frequency features exploit differences in token distributions, repetition, TF-IDF, lemma frequency, and repeated n-grams between human and machine text.Zipf-like token distributions become more discriminative when more text is available.
  • Linguistic Features: Readability, fluency, consistency, part-of-speech, and named-entity patterns provide additional signals for distinguishing generated text.Examples include Gunning-Fog and Flesch indices, phrasal verbs, coreference relationships, and tag distributions.
  • Feature-Based Approaches: Feature-based systems may still rely on neural inference when auxiliary models generate coreference, POS, or named-entity features.
  • Limitations: Some earlier features fail against contemporary Transformer models, especially on shorter sequences.Idiomatic-phrase features developed for long-text translation detection are an example of this limitation.

4.2 Neural Language Model Approaches

Neural approaches dominate machine-generated text detection, with fine-tuned bidirectional language models, particularly RoBERTa, delivering the strongest reported performance. Zero-shot methods are useful but can weaken substantially across domains, while limited attacker samples can improve adaptation.

  • Neural network detectors, especially those using Transformer-derived features, are highly effective for machine-generated text detection.
  • Smaller NLG models can detect outputs from larger models, although performance declines as the scale difference grows.
  • Zero-shot approaches: Zero-shot detectors perform strongly in their training domain but show limited out-of-domain performance and can underperform a TF-IDF baseline on cross-domain outputs.
  • Fine-tuning approaches: Fine-tuning large bidirectional language models, such as RoBERTa, is the state-of-the-art neural approach for distinguishing generated from human-written text.
  • Fine-tuning approaches: Fine-tuning with even a few hundred subject-matter-expert-labeled attacker samples can dramatically improve cross-domain detector adaptation.
  • Topological features from Transformer attention maps did not significantly improve standard BERT fine-tuning, though they better detected unseen GPT classifiers.

4.4 Human-aided Methods

Human reviewers often struggle to identify machine-generated text, but specialized training and human-aided tools can improve oversight. These approaches trade scalability for human agency and may complement automated detectors.

  • Human-aided detection combines statistical or neural systems with human review, providing oversight but reducing scalability because reviewers must be hired and trained.
  • GLTR: GLTR supports human classification by highlighting text according to Transformer token-sampling probabilities, although it targets GPT-2.
  • Untrained reviewers identified GPT-3 text at chance level, while limited training raised accuracy to 55%.
  • Human and algorithmic detection can complement each other because text that fools humans may be more recognizable to automated systems.
  • Specialized training: Scarecrow trains evaluators to annotate ten categories of GPT-3 errors; human annotations generally had higher precision, but higher F1 in only half the categories.
  • Human oversight: Specialist moderators could work alongside detection systems to review likely machine-written social-media posts, preserving human agency and oversight.

4.5 Trends in Evaluation Methodology and Datasets

Evaluation increasingly centers on Transformer-generated text and must account for generator variation, domain mismatch, sequence length, and realistic deployment conditions. Current research also exposes gaps in robustness, class imbalance, and comparability across domains.

  • Evaluation datasets: Contemporary evaluation commonly uses GPT-2 outputs, with newer studies also considering GPT-3 samples.
  • Evaluation datasets: The GPT-2 output dataset varies model size and sampling settings, while GPT-3 samples use a substantially larger model and different decoding parameters.
  • Evaluation datasets: Attribution datasets containing outputs from multiple NLG models are also used for generic machine-generated text detection.
  • Evaluation methodology: Evaluation should test mismatch across datasets, architectures, and parameters because defenders may not know the attacker generator’s characteristics.
  • Evaluation methodology: Longer sequences generally benefit detection, with common evaluation datasets containing 2048-token sequences.
  • Dataset construction: Custom datasets can be generated by training or fine-tuning NLG models on human-written domain corpora.
  • Dataset construction: In-the-wild social-media datasets can contain machine-generated text without revealing the generating models or providing matched human training data.
  • Open evaluation issues: Machine-generated text research increasingly targets bidirectional Transformers, but existing studies cover few threat models and remain difficult to compare across domains.

5 TRENDS AND OPEN PROBLEMS

Open problems include realistic class imbalance, unknown attacker configurations, adversarial robustness, fairness, human oversight, and policy responses to expanding undisclosed AI-generated content. Progress requires detection that remains effective while addressing quality, accountability, and social harms.

  • Class imbalance: Detection research has rarely addressed class imbalance, although machine-generated text may be a small minority class in practice.
  • Generalization: Defensive detectors should remain effective across unknown attacker parameters, architectures, and training datasets.
  • Attribution: Model attribution can help defenders infer an attacker’s model and sampling parameters, then iteratively refine detection systems.
  • Adversarial robustness: Adversarial attacks may help machine-generated text bypass neural detection systems, making robustness an important research concern.
  • Adversarial robustness: Adversarial edits can degrade fluency, credibility, or meaning, creating a trade-off between detector evasion and useful attack content.
  • Fairness: Detection systems may disproportionately flag people because of language background or benign use of translation tools.
  • Human agency: Human analysts can provide agency and oversight, while GLTR remains the only cited tool explicitly incorporating a human analyst.
  • Policy responses: Platforms may supplement technical detection with verification policies, including stronger CAPTCHA requirements and restrictions on fraudulent accounts.

6 CONCLUSION

The survey finds that current domain-specific defenses are inadequate for most upcoming machine-generated-text threat models. It identifies urgent open problems in realistic evaluation, transparency, fairness, robustness, and cross-domain coordination.

  • Current domain-specific defenses are inadequate against the vast majority of upcoming threat models.
  • Detection methods often fail to reflect class imbalance and unknown model architectures in realistic settings.
  • Detection systems need stronger transparency and fairness methods to avoid causing harm themselves.
  • Suitable defenses require coordinated effort among AI researchers, cybersecurity professionals, and non-technical experts.
Loading 2210.07321v4…