Source-linked AI summary
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, James Y. Zou
TL;DR
The paper asks how much real-world text has been substantially modified or produced by LLMs when individual cases are difficult to distinguish. It develops a maximum-likelihood, corpus-level estimator using human and AI reference texts, then applies it to scientific peer reviews. The estimates indicate substantial AI modification in 7–15% of ML conference-review sentences, with associated deadline, confidence, citation, and rebuttal-response patterns.
Problem
The scale and effects of LLM use in real-world information ecosystems are difficult to measure because generated text can be hard to distinguish from human writing.
Method
The paper combines human-expert and AI-generated reference texts with maximum likelihood estimation to estimate AI-modified text at the corpus level.
Results
Roughly 7–15% of sentences in ML conference reviews were substantially modified by AI beyond simple grammar checks, with no significant evidence in Nature reviews.
Takeaways & Limitations
Generated text is associated with lower confidence, last-minute submissions, fewer author replies, less specific feedback or citations, and compressed linguistic variation.
Takeaways & Limitations
The framework’s effectiveness may vary depending on the specific LLM used, although the reported estimates were robust to GPT-3.5 alternatives.
Abstract
from arXiv · showhide
We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.
1. Introduction
The paper addresses the difficulty of measuring LLM use in real-world text and proposes corpus-level estimation instead of instance-level detection. It applies this framework to peer reviews and examines where generated text occurs and how it differs from expert-written reviews.
- Motivation: Human detection of LLM-generated text barely exceeds random classification, motivating corpus-level measurement of uncertain AI use.The paper emphasizes that generated text can be indistinguishable from human writing on individual cases.
- Method: Distributional GPT quantification estimates the proportion of AI-generated or substantially modified text without inferring on individual instances.It combines known human and AI-generated reference texts with maximum likelihood estimation.
- Method: The framework is over 10 million times more computationally efficient than state-of-the-art detectors, while reducing in-distribution error 3.4-fold and out-of-distribution error 4.6-fold.These comparisons are reported against state-of-the-art AI text detection methods.
- Case study: The study examines reviews from ICLR, NeurIPS, EMNLP, CoRL, and Nature portfolio journals after ChatGPT’s release.It uses these cases to study both AI usage trends and corpus-level changes associated with generated text.
- Findings: AI modification appears in a small but significant fraction of AI-conference reviews beyond grammar and spell checking, but not in Nature portfolio reviews.The study also investigates circumstances associated with generated text and differences from expert-written reviews.
- Terminology: The paper uses “peer reviews” for human-expert text and “generated texts” for LLM-produced text without making an ontological claim about their status.The terminology is intended as descriptive rather than evaluative.
2. Related Work
Related work includes instance-level detection based on model probabilities, trained classifiers, and watermarking methods. These approaches aim to distinguish human and machine text, while the paper’s supplied findings also summarize corpus-level AI-use patterns.
- Zero-shot LLM detection: Zero-shot detectors use probabilistic or geometric characteristics of LLMs to distinguish AI-generated from human-written text.Examples use entropy, log-probability, perplexity, n-gram frequencies, curvature, or reprompted-text divergence.
- Corpus-level findings: The paper’s summary reports 10.6% of ICLR 2024 review sentences and 16.9% of EMNLP review sentences as substantially modified by ChatGPT, with no significant Nature evidence.It also reports deadline, citation, rebuttal-response, homogenization, and confidence associations.
- Training-based LLM detection: Training-based detectors fine-tune classifiers on human and AI-generated examples without requiring access to the original generation model.Reported examples include classifiers and twin neural networks trained on constructed datasets.
- Training-based LLM detection: Training-based methods can overfit to training data and language models, remain vulnerable to adversarial attacks, and display bias toward non-dominant language varieties.The theoretical possibility of reliable instance-level detection has also been questioned.
- LLM watermarking: Watermarking embeds algorithmically detectable signals into text, either by modifying text or changing an LLM’s decoding process.Decoding-based approaches include red-green token lists and methods designed to preserve token probability distributions.
3. Method
The method estimates corpus-level AI-modified content by learning human and AI token distributions from reference corpora, validating them on synthetic mixtures, and applying maximum likelihood estimation to target text.
- Distribution estimation: The model represents documents as token-occurrence sets and estimates distributions from token usage, using adjectives as the main vocabulary.The authors report that adjective-based estimates were more stable than estimates using other parts of speech or all tokens.
- Maximum likelihood estimation: Given estimated P and Q, maximum likelihood estimation computes the fraction α of AI-generated documents in a target corpus.The corpus log-likelihood is formed from the mixture of human and AI document likelihoods.
- Framework: The framework estimates the fraction of AI-generated or substantially modified text in a corpus without classifying individual documents.It frames monitoring as population-level parametric inference rather than instance-level detection.
- Reference data: The method uses human-authored and LLM-generated reference corpora to estimate scientist-written and AI text distributions P and Q.Historical human reviews and corresponding LLM-generated reviews provide the reference data.
- Robustness: The framework shows moderate robustness to shifts in LLM prompts, with accurate validation when training and testing prompts differ.The authors report this result from experiments summarized in the appendix.
- Validation: Validation creates synthetic target corpora with known AI proportions and checks whether the estimated α matches each specified proportion.Repeated sampling can also produce confidence intervals for the estimate.
4. Experiments
The experiments validate the maximum-likelihood estimator on in-distribution and out-of-distribution review data, then apply it to real reviews and examine robustness and correlates of estimated LLM use. The estimates rise after ChatGPT’s release in major ML conferences, remain stable in Nature Portfolio reviews, and are associated with proofreading robustness, outline expansion, deadlines, citations, confidence, and reply rates.
- Validation: The estimator predicts the proportion of LLM-generated text with less than 1.8% error on ICLR ’23 and NeurIPS ’22, and under 2.4% on CoRL ’22.ICLR ’23 provides in-distribution validation, while NeurIPS ’22 and CoRL ’22 provide out-of-distribution validation.
- Real-review estimates: 10.6% of ICLR review sentences, 9.1% of NeurIPS review sentences, and 6.5% of CoRL review sentences were estimated as substantially modified after ChatGPT’s release.EMNLP had the highest estimated α at approximately 16.9%, although pre-release data were unavailable.
- Real-review estimates: Post-ChatGPT α estimates were significantly above the α = 0 validation level for all major ML conferences, indicating modest but noteworthy AI-text use.For ICLR and NeurIPS, estimates were also significantly above validation estimates with ground truth α = 5%.
- Real-review estimates: Nature Portfolio reviews showed no significant post-ChatGPT increase, with pre- and post-release α estimates remaining within the α = 0 validation margin of error.This contrasts with the increase observed across the major ML conferences.
- Robustness: Simple ChatGPT proofreading produced only a slight, much smaller α increase than the real review corpus, supporting sensitivity to substantial modification beyond grammatical edits.The proofreading increase was described as minor and non-significant.
- Robustness: LLM expansion of bullet-pointed review outlines produced estimated α values closely matching ground truth, supporting detection of substantial outline-to-review expansion.The experiment mixed human reviews with LLM-expanded feedback at several ground-truth α levels.
- Correlates of estimated LLM use: Reviews submitted three or fewer days before deadlines had a small but consistent increase in estimated α.The authors suggest reviewers may rely on LLMs to save time near deadlines.
- Correlates of estimated LLM use: Lower estimated α was associated with scholarly citations, higher reviewer confidence, and more author replies, while convergent reviews tended to have higher α.The reply-rate result is a negative correlation, not a causal claim.
5. Discussion
The paper estimates AI-modified text at corpus scale using maximum likelihood and finds substantial post-ChatGPT use in ML conference reviews, while identifying important scope and modeling limitations.
- Method and interpretation: The framework estimates the fraction of documents generated primarily using AI tools from historical human and AI-generated reference texts.It estimates text distributions and maximizes the target corpus likelihood to produce α.
- Behavioral patterns: The estimated AI fraction is higher in low-confidence reviews, which report self-rated confidence of 2 or lower on a 5-point scale.This correlation is mostly identical across the major ML conferences examined.
- Main findings: Roughly 7-15% of sentences in ML conference reviews were substantially modified by AI beyond a simple grammar check, unlike Nature reviews.The study also associates generated text with deadlines, lower confidence, fewer citations, and fewer author replies.
- Limitations: The framework’s effectiveness may vary with the specific LLM used, despite robust estimates when trained with GPT-3.5 data and tested on GPT-4.The authors focused primarily on ChatGPT and caution that practitioners should select a model closely matching likely usage.
- Limitations: Generalizability beyond the studied ML conferences and Nature journals remains uncertain because writing styles and AI prevalence vary across fields and venues.The authors call for further experimentation across a wider range of domains and publication venues.
- Interpretation: The estimates do not directly establish that reviewers wrote entire reviews with ChatGPT, because AI may instead have substantially reformulated reviewer-generated bullet points.The authors explicitly avoid judging AI use as good or bad.
Impact Statement
The paper presents corpus-scale monitoring as a basis for studying AI use in peer review and supporting transparent discussion, while its supplementary materials document validation and analysis choices.
- Impact: The work offers a method for studying LLM use at scale and applies it to peer-review corpora to examine implications for scientific publishing.The authors frame the analysis as a foundation for transparent community discussion despite acknowledged limitations.
- Linguistic analysis: The supplementary materials catalog disproportionately AI-associated adjectives and adverbs through word clouds and tables.Font size in the word clouds indicates frequency, while the tables list the top 100 terms.
- Supplementary evidence: The appendix documents conference confidence scales and supplies numerical results for validation procedures and Nature-journal experiments.It includes confidence descriptions, figure data, and separately trained Nature-family models.
- Validation: Validation across pre-ChatGPT ICLR, NeurIPS, and CoRL reviews achieved less than 2.4% prediction error for the estimated LLM proportion.The validation used blends of official human and LLM-generated reviews.
D.2. Main Results Tables
The supplementary main-results tables examine temporal, prompt, field, and vocabulary robustness, generally preserving the post-ChatGPT increase while showing weaker performance for alternative token classes.
- Temporal trends: Pre-ChatGPT α estimates are close to 0 for ML conferences and rise sharply after ChatGPT’s release, unlike Nature-family journals.Nature pre- and post-release estimates remain within the margin of error for α = 0.
- Prompt robustness: A different prompt preserves accurate and stable out-of-distribution performance when training on ICLR 2018-2022 and testing on NeurIPS and CoRL.The appendix describes this as robustness to moderate prompt distribution shift.
- Additional analyses: The supplementary results include field-level α changes for ICLR 2024 and separate figures for alternative prompts and token classes.These materials extend the main results across designated primary research areas and several linguistic categories.
- Vocabulary choice: Adverbs, verbs, and nouns produce degraded validation performance compared with adjectives.The main analysis therefore uses adjectives because that vocabulary choice exhibits greater stability.
- Alternative analyses: The appendix reports temporal α trends using adverbs, verbs, and nouns, with pre-ChatGPT estimates near 0 and sharp post-release increases.These alternative parts of speech reproduce the same qualitative temporal pattern despite weaker validation performance.
D.8. Results on Document-Level Analysis
Document-level analysis treats complete reviews as mixture-model observations and produces results similar to sentence-level analysis, with slightly weaker and mildly upward-biased estimates.
- Design: Document-level analysis interprets P as the distribution of reviews written without AI assistance and Q as the distribution of AI-generated reviews.This changes the unit of analysis from individual sentences to complete reviews.
- Results: The document-level results are similar to those from the sentence-level analysis.The appendix reports corresponding validation results in Table 17.
- Comparison: Document-level validation shows slight performance degradation and a tendency to over-estimate the true α.The authors prefer sentence-level analysis because entire-review generation is considered less likely than generating individual sentences or review parts.
- Temporal trends: Official-review α estimates remain close to 0 before ChatGPT and increase sharply after its release under the document-level model.The temporal pattern matches the sentence-level analysis qualitatively.
D.9. Comparison to State-of-the-art GPT Detection Methods
The paper compares its corpus-level estimator with off-the-shelf AI-text detectors and tests whether proofreading or prompt choices explain its findings. Classifier-based estimates are unreliable, while validation and auxiliary analyses examine robustness and alternative explanations.
- Classifier comparison: RADAR, Deepfake, and DetectGPT estimates remain almost constant across the true α, while BERT estimates remain far off.The table reports validation behavior for classifier-based methods.
- Alternative explanations: Proofreading with ChatGPT alone cannot explain the observed increase in estimated AI modification.This conclusion is stated in the proofreading analysis table.
- Model validation: The study separately validates models trained on GPT-3.5-generated reviews and tested on GPT-3.5- or GPT-4-generated reviews.These procedures are identified in Figures 24 and 25 and the corresponding validation table.
- Robustness validation: Prompt-shift validation tests whether the estimator still predicts α accurately when validation prompts differ substantially from training prompts.The validation prompt is described as being written in a significantly different style.
G. Additional Dataset Information
The paper documents its peer-review datasets, sentence-level preprocessing, and ethical safeguards. Conference reviews come from OpenReview, while the Nature dataset covers 15 journals and accepted papers from 2019–2023.
- Data sources: Conference peer-review data were accessed through the official OpenReview API, with reviews averaging 25.94 sentences.The Nature dataset was collected separately using a custom scraper.
- Data sources: The Nature portfolio dataset contains peer reviews from 15 named Nature journals for papers accepted between 2019 and 2023.The passage lists the journals included in the dataset.
- Preprocessing: The framework analyzes sentences with at least two words and imposes no maximum sentence length.Blank review sections contribute no sentences to the corpus.
- Ethics: The authors identify reviewer consent, data licensing, and responsible dataset use as ongoing ethical concerns.They point to explicit-consent data donation initiatives as a promising future model.
- Ethics: The study uses publicly available anonymous OpenReview data and reports compliance with the API’s terms of use.OpenReview makes reviews public upon submission under the described agreement.
H. Additional Results on LLaMA-2 Chat (70B), and Claude 2.1
The paper extends its validation experiments beyond GPT-4 to LLaMA-2 Chat (70B) and Claude-2.1. Both extensions use the paper’s existing training and validation setup and compare validation-data sources.
- Additional models: Additional validation experiments evaluate estimators trained on LLaMA-2 Chat (70B) and Claude-2.1-generated reviews.Both experiments use the same training and validation setup as the main paper.
- Validation: The LLaMA-2 Chat (70B) experiment compares validation-data sources under the paper’s estimator setup.The comparison is reported in Table 32.
- Validation: The Claude-2.1 experiment likewise reports a validation-data source performance comparison.The comparison is reported in Table 33.
I. Theoretical Analysis on the Sample Size
The theoretical analysis studies the maximum-likelihood estimate of the mixture weight α under i.i.d. sampling from known human and AI distributions. Strong concavity and concentration control the estimator’s deviation from the ground truth.
- Assumptions: The theorem assumes i.i.d. samples from (1 − α*)P + α*Q, with known P and Q and ground-truth α* in (0, 1).It also assumes a positive curvature constant κ.
- Guarantee: The MLE α̂ is shown to remain close to α* with high probability.The guarantee is stated for any δ ∈ (0, 1) with probability at least 1 − δ.
- Proof strategy: The second derivative satisfies L′′(α) ≤ −κ, making the log-likelihood κ-strongly concave on [0, 1].The proof invokes the standard consequence of strong concavity to relate the estimator to the true mixture weight.
- Proof strategy: The proof controls the empirical score at α* by defining S_i from Q(x_i), P(x_i), and the true mixture distribution.Hoeffding’s inequality is then applied using the bounded range of S_i.