Source-linked AI summary
OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples
Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki
TL;DR
LLM-generated text is difficult to distinguish from human writing, and existing detectors are vulnerable to paraphrasing and deliberate evasion. OUTFOX makes detector and attacker outputs available as in-context examples for one another, improving robustness on attacked essays while preserving strong non-attacked-text performance. In student-essay experiments, detection improves by up to +41.3 points F1-score on attacker-generated texts, reaches up to 96.9 points F1-score on non-attacked texts, and the attacker reduces detector performance by up to -57.0 points F1-score.
Problem
Existing LLM-generated-text detectors lack robustness to paraphrasing and may not account for malicious users generating texts specifically to evade detection.
Method
OUTFOX uses detector predictions and attacker-generated essays as mutual in-context-learning examples, enabling adaptive attack generation and detection.
Results
OUTFOX improves attacked-text detection by up to +41.3 points F1-score, reaches up to 96.9 points F1-score on non-attacked texts, and its attacker reduces detector performance by up to -57.0 points F1-score.
Takeaways & Limitations
Considering attacks improves robustness with little negative effect on non-attacked-text detection, while the attacker is more effective than previous paraphrasing attacks.
Takeaways & Limitations
Although the framework theoretically allows repeated detector-attacker strengthening, the experiments focus on only one iteration.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have achieved human-level fluency in text generation, making it difficult to distinguish between human-written and LLM-generated texts. This poses a growing risk of misuse of LLMs and demands the development of detectors to identify LLM-generated texts. However, existing detectors lack robustness against attacks: they degrade detection accuracy by simply paraphrasing LLM-generated texts. Furthermore, a malicious user might attempt to deliberately evade the detectors based on detection results, but this has not been assumed in previous studies. In this paper, we propose OUTFOX, a framework that improves the robustness of LLM-generated-text detectors by allowing both the detector and the attacker to consider each other's output. In this framework, the attacker uses the detector's prediction labels as examples for in-context learning and adversarially generates essays that are harder to detect, while the detector uses the adversarially generated essays as examples for in-context learning to learn to detect essays from a strong attacker. Experiments in the domain of student essays show that the proposed detector improves the detection performance on the attacker-generated texts by up to +41.3 points F1-score. Furthermore, the proposed detector shows a state-of-the-art detection performance: up to 96.9 points F1-score, beating existing detectors on non-attacked texts. Finally, the proposed attacker drastically degrades the performance of detectors by up to -57.0 points F1-score, massively outperforming the baseline paraphrasing method for evading detection.
Introduction
OUTFOX addresses the vulnerability of LLM-generated-text detectors to paraphrasing and deliberate evasion by having attackers and detectors learn from each other's outputs. In student-essay experiments, the framework improves attacked-text detection while maintaining strong performance on non-attacked texts.
- LLMs' fluent generation makes distinguishing human-written from LLM-generated text difficult and raises misuse concerns in education.
- Existing detectors often perform poorly against paraphrasing, while malicious users may use LLMs to generate texts specifically designed to evade detection.
- OUTFOX lets the attacker use detector predictions for in-context learning and the detector use adversarial texts to improve robustness against strong attackers.
- +41.3 points F1-score improves detection of attacker-generated texts compared with not considering attacks.
- 96.9 points F1-score is achieved on non-attacked texts, while the OUTFOX attacker decreases detector performance by up to -57.0 points F1-score.
Related Work
Prior work developed detectors and demonstrated their vulnerability to paraphrasing attacks, but offered limited defenses against adaptive attacks. OUTFOX differs by using in-context learning for mutual detector-attacker adaptation without parameter updates.
- LLM-generated-text detectors include watermarking, statistical outlier detection, and supervised-classification approaches.
- Student-assignment detection research has examined argumentative essays and university-level course problems, including essays by nonnative English learners.
- Paraphrasing attacks can cause large accuracy losses, and DIPPER is an 11B document-level paraphraser controlling vocabulary and content reordering.
- Existing defenses include retrieval from an API-response database, but this approach requires active API-provider actions and can produce false positives.
- OUTFOX uses in-context learning rather than parameter updates, enabling new attacks to be incorporated by adding examples without additional fine-tuning.
OUTFOX Framework
OUTFOX forms a detector-attacker loop: the attacker uses detector labels to generate harder-to-detect essays, and the detector uses adversarial examples to classify target essays. Both sides use retrieved, semantically close essay examples in prompts.
- Task Formulation: The task assumes training instances contain an essay-problem statement, a human-written essay, and an LLM-generated essay.
- Task Formulation: The detector can include human-written, attacked, and non-attacked essays with labels in its in-context prompt.
- The OUTFOX Detector: For a target essay, the detector retrieves the top-k semantically close problem statements and associated essays, then adds adversarially generated essays to its examples.
- The OUTFOX Detector: Attacker-generated essays receive the LM label, encouraging detection of their implicit characteristics, and the detector predicts the target label from the prompted examples.
- The OUTFOX Attacker: The attacker retrieves related examples, obtains detector labels for LLM-generated essays, and generates an adversarial essay conditioned on those labels and the problem statement.
- The OUTFOX Attacker: The attacker prompts the LLM to write an essay detected as Human, using a word count matched to the human-written essay.
LLM-Generated Essays
The paper builds a student-essay dataset for detecting LLM-generated essays by pairing native-student argumentative essays with generated problem statements and LLM-written essays. The framework theoretically permits repeated detector-attacker strengthening, but experiments focus on one iteration.
- The dataset targets detection of LLM-generated student essays and requires abundant essay-problem statement pairs.
- Although detector and attacker strengthening could theoretically repeat many times, the framework focuses on one iteration.
- It uses argumentative essays written by native students in grades 6–12 in the United States.
- ChatGPT generates pseudo-problem statements, after which an instruction-tuned LLM generates essays from them.
Experiments and Results
The experiments evaluate OUTFOX against attacked and non-attacked student essays, using F1-score and AvgRec alongside baseline detectors and attacks. OUTFOX improves detection of attacker-generated essays while preserving performance on non-attacked texts.
- Experimental Questions: The experiments test robustness against OUTFOX and DIPPER attacks, performance on non-attacked essays, and whether OUTFOX’s attacker exceeds paraphrasing.The evaluation uses student essays and compares attacked with non-attacked settings.
- Evaluation Setup: Detection uses F1-score and AvgRec on a test set containing 500 human-written and 500 non-attacked essays, with attacked evaluations replacing only LLM-generated essays.Baseline thresholds are fixed across non-attacked and attacked evaluations.
- Attacked Essays: +41.3 points F1-score and +21.6 points AvgRec are achieved on OUTFOX-attacker essays when the detector considers attacks.The improvement occurs relative to the detector without attack consideration; improvements also occur across the tested attackers.
- Non-attacked Essays: An average decrease of only -0.1 point F1-score and -0.32 point AvgRec occurs on non-attacked essays when attacks are considered.Performance is even better for non-attacked FLAN-T5-XXL essays in the attack-aware setting.
- Attacker Comparison: OUTFOX’s attacker degrades detector performance by up to -57.0 points F1-score, exceeding DIPPER’s degradation by up to -39.6 points F1-score.DIPPER often has little degrading effect and can improve detection, especially for RoBERTa-large.
Comparison with Prior Work
OUTFOX outperforms prior detectors on non-attacked essays across statistical outlier and supervised-classifier comparisons, while maintaining high human-essay recognition.
- Statistical Outlier Approaches: 80.5 points F1-score and 83.5 points AvgRec are reported for OUTFOX against prior statistical outlier approaches on FLAN-T5-XXL essays.Statistical approaches require access to model logits.
- Statistical Outlier Approaches: 98.8 points HumanRec indicates that OUTFOX rarely labels human-written essays as LLM-generated.The authors partly attribute this to using ChatGPT, which was trained with human feedback.
- Supervised Classifiers: OUTFOX achieves better detection performance than supervised classifiers on essays generated by both ChatGPT and GPT-3.5.The comparison uses non-attacked essays and evaluates supervised classifiers alongside OUTFOX.
- Conclusion: OUTFOX reports state-of-the-art detection performance on non-attacked essays.The comparison covers statistical outlier detectors and supervised classifiers.
Attacks with Human-Written Essays
The analysis compares semantic similarity between human-written essays and both non-attacked and OUTFOX attacker-generated essays using BERT embeddings and cosine similarity distributions.
- Experimental Analysis: The analysis uses 500 human-written, 500 ChatGPT-generated, and 500 attacker-generated essays from the test set.Semantic similarity is computed from essay embeddings produced by a pre-trained BERT model.
- Experimental Analysis: Figure 4 compares cosine similarity distributions with human-written essays for non-attacked and OUTFOX attacker-generated essays.A rightward shift represents greater semantic similarity to human-written essays.
Conclusion
OUTFOX uses mutual in-context learning between detector and attacker to improve robustness against attacks. In student-essay experiments, the detector learns from attacker outputs while preserving non-attacked detection, and the attacker more effectively evades detectors than paraphrasing.
- Conclusion: OUTFOX lets the detector and attacker use each other’s outputs as in-context learning examples.The detector learns from adversarial texts, while the attacker uses detector outputs to generate harder-to-detect essays.
- Conclusion: The detector learns to detect attacker-generated essays through in-context examples with little negative effect on non-attacked texts.These findings are demonstrated in experiments on student essays.
- Conclusion: The OUTFOX attacker evades current LLM-generated-text detectors more effectively than the previous paraphrasing attack.The attacker is designed specifically to deceive the detector.
- Conclusion: Attacker-generated essays can be semantically closer to human-written essays than non-attacked essays.The analysis links this greater semantic closeness to successful attacking.