PaperlayerView exact revision on arXiv

Source-linked AI summary

OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples

Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki

arXiv:2307.11729v3cs.CL

TL;DR

LLM-generated text is difficult to distinguish from human writing, and existing detectors are vulnerable to paraphrasing and deliberate evasion. OUTFOX makes detector and attacker outputs available as in-context examples for one another, improving robustness on attacked essays while preserving strong non-attacked-text performance. In student-essay experiments, detection improves by up to +41.3 points F1-score on attacker-generated texts, reaches up to 96.9 points F1-score on non-attacked texts, and the attacker reduces detector performance by up to -57.0 points F1-score.

  • Problem

    Existing LLM-generated-text detectors lack robustness to paraphrasing and may not account for malicious users generating texts specifically to evade detection.

    Source: Introduction · p. 1

  • Method

    OUTFOX uses detector predictions and attacker-generated essays as mutual in-context-learning examples, enabling adaptive attack generation and detection.

    Source: Introduction · p. 1, Task Formulation · p. 3

  • Results

    OUTFOX improves attacked-text detection by up to +41.3 points F1-score, reaches up to 96.9 points F1-score on non-attacked texts, and its attacker reduces detector performance by up to -57.0 points F1-score.

    Source: Introduction · p. 2, Results · pp. 5–6

  • Takeaways & Limitations

    Considering attacks improves robustness with little negative effect on non-attacked-text detection, while the attacker is more effective than previous paraphrasing attacks.

    Source: Introduction · p. 2, Results · p. 6, Conclusion · p. 7

    Although the framework theoretically allows repeated detector-attacker strengthening, the experiments focus on only one iteration.

    Source: LLM-Generated Essays · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved human-level fluency in text generation, making it difficult to distinguish between human-written and LLM-generated texts. This poses a growing risk of misuse of LLMs and demands the development of detectors to identify LLM-generated texts. However, existing detectors lack robustness against attacks: they degrade detection accuracy by simply paraphrasing LLM-generated texts. Furthermore, a malicious user might attempt to deliberately evade the detectors based on detection results, but this has not been assumed in previous studies. In this paper, we propose OUTFOX, a framework that improves the robustness of LLM-generated-text detectors by allowing both the detector and the attacker to consider each other's output. In this framework, the attacker uses the detector's prediction labels as examples for in-context learning and adversarially generates essays that are harder to detect, while the detector uses the adversarially generated essays as examples for in-context learning to learn to detect essays from a strong attacker. Experiments in the domain of student essays show that the proposed detector improves the detection performance on the attacker-generated texts by up to +41.3 points F1-score. Furthermore, the proposed detector shows a state-of-the-art detection performance: up to 96.9 points F1-score, beating existing detectors on non-attacked texts. Finally, the proposed attacker drastically degrades the performance of detectors by up to -57.0 points F1-score, massively outperforming the baseline paraphrasing method for evading detection.

Introduction

OUTFOX addresses the vulnerability of LLM-generated-text detectors to paraphrasing and deliberate evasion by having attackers and detectors learn from each other's outputs. In student-essay experiments, the framework improves attacked-text detection while maintaining strong performance on non-attacked texts.

  • LLMs' fluent generation makes distinguishing human-written from LLM-generated text difficult and raises misuse concerns in education.Source: Introduction · p. 1
  • Existing detectors often perform poorly against paraphrasing, while malicious users may use LLMs to generate texts specifically designed to evade detection.Source: Introduction · p. 1
  • OUTFOX lets the attacker use detector predictions for in-context learning and the detector use adversarial texts to improve robustness against strong attackers.Source: Introduction · p. 1, Figure 1 · p. 1
  • +41.3 points F1-score improves detection of attacker-generated texts compared with not considering attacks.Source: Introduction · p. 2
  • 96.9 points F1-score is achieved on non-attacked texts, while the OUTFOX attacker decreases detector performance by up to -57.0 points F1-score.Source: Introduction · p. 2

Related Work

Prior work developed detectors and demonstrated their vulnerability to paraphrasing attacks, but offered limited defenses against adaptive attacks. OUTFOX differs by using in-context learning for mutual detector-attacker adaptation without parameter updates.

  • LLM-generated-text detectors include watermarking, statistical outlier detection, and supervised-classification approaches.Source: Related Work · p. 2
  • Student-assignment detection research has examined argumentative essays and university-level course problems, including essays by nonnative English learners.Source: Related Work · p. 2
  • Paraphrasing attacks can cause large accuracy losses, and DIPPER is an 11B document-level paraphraser controlling vocabulary and content reordering.Source: Related Work · p. 2
  • Existing defenses include retrieval from an API-response database, but this approach requires active API-provider actions and can produce false positives.Source: Related Work · p. 2
  • OUTFOX uses in-context learning rather than parameter updates, enabling new attacks to be incorporated by adding examples without additional fine-tuning.Source: Related Work · p. 2

OUTFOX Framework

OUTFOX forms a detector-attacker loop: the attacker uses detector labels to generate harder-to-detect essays, and the detector uses adversarial examples to classify target essays. Both sides use retrieved, semantically close essay examples in prompts.

Task Formulation

  • The task assumes training instances contain an essay-problem statement, a human-written essay, and an LLM-generated essay.Source: Task Formulation · pp. 2–3
  • The detector can include human-written, attacked, and non-attacked essays with labels in its in-context prompt.Source: Task Formulation · p. 3

The OUTFOX Detector

  • For a target essay, the detector retrieves the top-k semantically close problem statements and associated essays, then adds adversarially generated essays to its examples.Source: The OUTFOX Detector · p. 3
  • Attacker-generated essays receive the LM label, encouraging detection of their implicit characteristics, and the detector predicts the target label from the prompted examples.Source: The OUTFOX Detector · p. 3

The OUTFOX Attacker

  • The attacker retrieves related examples, obtains detector labels for LLM-generated essays, and generates an adversarial essay conditioned on those labels and the problem statement.Source: The OUTFOX Attacker · pp. 3–4
  • The attacker prompts the LLM to write an essay detected as Human, using a word count matched to the human-written essay.Source: The OUTFOX Attacker · p. 4

LLM-Generated Essays

The paper builds a student-essay dataset for detecting LLM-generated essays by pairing native-student argumentative essays with generated problem statements and LLM-written essays. The framework theoretically permits repeated detector-attacker strengthening, but experiments focus on one iteration.

  • The dataset targets detection of LLM-generated student essays and requires abundant essay-problem statement pairs.Source: LLM-Generated Essays · p. 4
  • Although detector and attacker strengthening could theoretically repeat many times, the framework focuses on one iteration.Source: LLM-Generated Essays · p. 4
  • It uses argumentative essays written by native students in grades 6–12 in the United States.Source: LLM-Generated Essays · p. 4
  • ChatGPT generates pseudo-problem statements, after which an instruction-tuned LLM generates essays from them.Source: LLM-Generated Essays · p. 4

Experiments and Results

The experiments evaluate OUTFOX against attacked and non-attacked student essays, using F1-score and AvgRec alongside baseline detectors and attacks. OUTFOX improves detection of attacker-generated essays while preserving performance on non-attacked texts.

Experimental Questions

  • The experiments test robustness against OUTFOX and DIPPER attacks, performance on non-attacked essays, and whether OUTFOX’s attacker exceeds paraphrasing.The evaluation uses student essays and compares attacked with non-attacked settings.Source: Experiments and Results · p. 4, Table 1 · p. 5, Table 2 · p. 6

Evaluation Setup

  • Detection uses F1-score and AvgRec on a test set containing 500 human-written and 500 non-attacked essays, with attacked evaluations replacing only LLM-generated essays.Baseline thresholds are fixed across non-attacked and attacked evaluations.Source: Overall Setup · p. 5

Attacked Essays

  • +41.3 points F1-score and +21.6 points AvgRec are achieved on OUTFOX-attacker essays when the detector considers attacks.The improvement occurs relative to the detector without attack consideration; improvements also occur across the tested attackers.Source: Results · p. 5

Non-attacked Essays

  • An average decrease of only -0.1 point F1-score and -0.32 point AvgRec occurs on non-attacked essays when attacks are considered.Performance is even better for non-attacked FLAN-T5-XXL essays in the attack-aware setting.Source: Results · p. 6

Attacker Comparison

  • OUTFOX’s attacker degrades detector performance by up to -57.0 points F1-score, exceeding DIPPER’s degradation by up to -39.6 points F1-score.DIPPER often has little degrading effect and can improve detection, especially for RoBERTa-large.Source: Results · p. 6

Comparison with Prior Work

OUTFOX outperforms prior detectors on non-attacked essays across statistical outlier and supervised-classifier comparisons, while maintaining high human-essay recognition.

Statistical Outlier Approaches

  • 80.5 points F1-score and 83.5 points AvgRec are reported for OUTFOX against prior statistical outlier approaches on FLAN-T5-XXL essays.Statistical approaches require access to model logits.Source: Comparison with Prior Work · p. 6, Table 4 · p. 7
  • 98.8 points HumanRec indicates that OUTFOX rarely labels human-written essays as LLM-generated.The authors partly attribute this to using ChatGPT, which was trained with human feedback.Source: Comparison with Prior Work · p. 7

Supervised Classifiers

  • OUTFOX achieves better detection performance than supervised classifiers on essays generated by both ChatGPT and GPT-3.5.The comparison uses non-attacked essays and evaluates supervised classifiers alongside OUTFOX.Source: Comparison with Prior Work · p. 7, Table 4 · p. 7

Conclusion

Attacks with Human-Written Essays

The analysis compares semantic similarity between human-written essays and both non-attacked and OUTFOX attacker-generated essays using BERT embeddings and cosine similarity distributions.

Experimental Analysis

  • The analysis uses 500 human-written, 500 ChatGPT-generated, and 500 attacker-generated essays from the test set.Semantic similarity is computed from essay embeddings produced by a pre-trained BERT model.Source: Attacks with Human-Written Essays · p. 7
  • Figure 4 compares cosine similarity distributions with human-written essays for non-attacked and OUTFOX attacker-generated essays.A rightward shift represents greater semantic similarity to human-written essays.Source: Attacks with Human-Written Essays · p. 7

Conclusion

OUTFOX uses mutual in-context learning between detector and attacker to improve robustness against attacks. In student-essay experiments, the detector learns from attacker outputs while preserving non-attacked detection, and the attacker more effectively evades detectors than paraphrasing.

Conclusion

  • OUTFOX lets the detector and attacker use each other’s outputs as in-context learning examples.The detector learns from adversarial texts, while the attacker uses detector outputs to generate harder-to-detect essays.Source: Conclusion · p. 7
  • The detector learns to detect attacker-generated essays through in-context examples with little negative effect on non-attacked texts.These findings are demonstrated in experiments on student essays.Source: Conclusion · p. 7
  • The OUTFOX attacker evades current LLM-generated-text detectors more effectively than the previous paraphrasing attack.The attacker is designed specifically to deceive the detector.Source: Conclusion · p. 7
  • Attacker-generated essays can be semantically closer to human-written essays than non-attacked essays.The analysis links this greater semantic closeness to successful attacking.Source: Conclusion · p. 7

Related papers

Loading 2307.11729v3…