PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Writers Have a Consistent Stylometric Footprint, but AI Editors Do Not

Zhengyang Shan, Yukyung Lee, Sophie Hao

arXiv:2608.27855v1cs.CL

TL;DR

As LLMs increasingly edit human writing as well as generate text, it remains unclear whether these uses leave the same stylometric trace. This paper compares generation and editing across models and domains, finding a stable generation footprint but qualitatively different editing signals.

  • Problem

    LLMs increasingly revise human-authored content, but whether AI generation and editing leave the same stylometric trace remains unclear.

  • Method

    The paper systematically compares stylometric features using benchmark texts across 8 LLMs, 5 domains, and 273,420 AI-edited texts, with regularized classifiers and feature-importance analyses.

  • Results

    AI generation has a consistent footprint centered on entropy and lexical diversity, whereas editing weakly increases lexical diversity, decreases entropy, and makes lexical density dominant.

  • Takeaways & Limitations

    AI generation and editing leave qualitatively different stylometric traces, so edited text remains closer to its human-written source than to fully generated text.

  • Takeaways & Limitations

    The evaluated settings focus mainly on English long-form text, use eight open-weight generators across five domains, and limit large-scale editing experiments to three open-weight editors.

Abstract

from arXiv · show

Text generated by large language models (LLMs) has been shown to be stylometrically distinct from human-written text \citep{andreDetectingAIAuthorship2023, shahDetectingUnmaskingAIGenerated2023, oparaStyloAIDistinguishingAIGenerated2024, soto2024fewshot, liLinguisticDifferencesAI2025, selviogluFeatureExtractionAnalysis2025}. But LLMs are increasingly used not only to generate text but also to edit human writing, and it is unclear whether the two leave the same trace. We show that AI generation leaves a consistent ``stylometric footprint'': a small subset of features, primarily entropy and lexical diversity, consistently separates AI-generated text from human writing across 8 LLMs and 5 domains, while the remaining features depend heavily on the domain and generator. AI editing, however, does not reproduce the same footprint. Relative to their human-written sources, AI-edited texts show only a small increase in lexical diversity and a decrease in entropy, rather than the joint increase that characterizes AI generation. Lexical density, which contributes little to generation, instead becomes the dominant editing-associated signal. Stylometric features therefore separate AI-edited text from AI-generated text but are substantially less effective at separating it from human-written text. Our results suggest that ``AI text'' is not a single phenomenon: generation and editing leave qualitatively different stylometric traces and should be studied separately.

1 Introduction

The paper asks whether AI editing leaves the same stylometric trace as full AI generation, using large-scale comparisons across models, domains, and interpretable features. It finds a stable generation footprint but qualitatively different editing signals, motivating separate treatment of the two uses.

  • Research questions and approach: The study compares human-written, AI-generated, and AI-edited text using 14 interpretable stylometric features across multiple models, domains, and editing settings.The corpus includes 45,000 human-written and AI-generated texts across 8 LLMs and 5 domains, plus 273,420 AI-edited texts from 3 LLMs, 31 prompts, and 8 editing categories.
  • AI generation: AI-generated text has a consistent stylometric footprint, led primarily by entropy and lexical diversity across generators and domains.Most remaining features are strongly condition-dependent.
  • AI editing: AI editing does not reproduce the generation footprint: lexical diversity rises only slightly, entropy decreases, and lexical density becomes the dominant editing signal.Editing therefore does not simply move human writing toward a weaker version of the generation pattern.
  • Detection approaches: Stylometric and neural approaches capture complementary signals, with stylometric models stronger for separating edited from generated text and neural detectors stronger for identifying whether editing occurred.Combining both approaches substantially improves performance across both settings.
  • Implications: The results suggest that AI-text detectability depends on how LLMs are used, so generation and editing should be studied separately.The paper frames the two uses as producing partially distinct stylometric signatures rather than one unified phenomenon.

2 Related Work

Prior AI-text detection uses zero-shot statistical methods, supervised classifiers, and interpretable stylometric features across varied settings. This paper builds on that work by testing which stylometric features remain stable across generators, domains, and editing conditions.

  • AI-text detection: AI-generated text detection spans zero-shot statistical methods and supervised neural classifiers, with different requirements for reference models and task-specific training.Zero-shot methods use likelihood, curvature, perplexity, or cross-perplexity, while supervised systems learn from labeled examples.
  • Benchmarks: Existing benchmarks cover machine-generated text across generators and domains as well as expert- and LLM-edited outputs.The cited M4 and BEEMO benchmarks represent complementary AI-text settings.
  • Stylometric detection: Feature-based approaches emphasize interpretable properties such as lexical diversity, stop-word ratios, syntactic features, readability, and character or part-of-speech patterns.These methods can compete with heavier neural models and complement pretrained language models.
  • Positioning: The paper evaluates whether stylometric features remain stable across generators, domains, and AI-editing settings, addressing the assumed generality of common statistical properties.This extends prior work on interpretable, content-insensitive features and transfer.

3 GEN: A Multi-Domain Corpus of AI-Edited Texts

GEN is a multi-domain corpus combining human-written, AI-generated, and AI-edited English documents across five domains. Its editing set applies varied prompts, edit types, ratios, and models to human-written seed documents.

  • GEN combines human-written and AI-generated documents from Wikipedia, WikiHow, arXiv, Reddit QA, and Reddit Stories.
  • The generation set contains 1,000 documents per domain from eight LLMs.
  • The editing design uses 31 prompts across eight edit types and three target edit ratios: 30%, 50%, and 70%.
  • AI editing is applied only to fully human-written documents, not to AI-generated text.
  • The final analysis uses 273,420 AI-edited texts after generating 279,000 edited versions and holding out 20 seed documents.

4 Experimental Setup: Characterizing Stylometric Footprints

The study tests stylometric-footprint claims with logistic-regression classifiers trained on interpretable features from GEN. It evaluates feature importance and robustness across experimental conditions using repeated, standardized train/test procedures.

  • The experiments train linear classifiers on GEN documents represented only by 14 stylometric features.
  • The features span lexical, orthographic, syntactic, information-theoretic, and readability categories.
  • Each logistic-regression classifier outputs the predicted probability that a text belongs to the task-specific positive class.
  • Classifiers use balanced sampling, an 80%/20% stratified train/test split, and ten independent runs.
  • Coefficient importance measures standardized-feature effects on log-odds, whereas permutation importance measures held-out accuracy loss after shuffling a feature.

5 The Stylometric Footprint of AI-Generated Text

AI-generated text is reliably distinguishable from human writing through a small, stable stylometric footprint. Entropy and lexical diversity dominate across domains and generators, while secondary signals vary by condition and temperature.

  • AI-generated documents are one standard deviation higher in mean entropy and two standard deviations higher in lexical diversity than human-written documents.
  • A linear classifier using only stylometric features exceeds 90% accuracy in Global and Per-Domain conditions and generally does so in Per-LLM conditions.
  • 79.45% accuracy is obtained for one exceptionally human-like generator, while secondary feature signals vary across domains and LLMs.
  • Lexical diversity, entropy, and burstiness are the three most important Global-condition features under coefficient and permutation importance.
  • Entropy and lexical diversity have the highest stability across 14 conditions; lexical diversity ranks in the top five in 13 conditions and entropy in 12.
  • Classification accuracy changes only modestly across temperatures, from 82.2% at 0.3 to 80.7% at 0.7 and 79.5% at 1.0.

6 AI-Edited vs. AI-Generated Text

AI editing produces a stylometric pattern distinct from AI generation and remains closer to human writing. Its effects vary across editing models, while stylometric and neural detectors are strongest on different class pairings.

  • Stylometric Effects: AI editing marginally increases lexical diversity (d = +0.24) but decreases entropy (d = −0.28), unlike fully generated text.This does not move edited documents consistently toward the generation footprint.
  • Stylometric Effects: Lexical density is the strongest editing-associated feature, decreasing substantially relative to human-written source documents (d = −3.10).AI editing reduces the proportion of content words relative to function words.
  • Model Variation: GPT-OSS 120B increases entropy (d = +0.44), keeps word count roughly constant (d = +0.02), and sharply decreases Linsear Write (d = −2.81).These effects differ from those of Llama 70B and Qwen 72B.
  • Classification: Stylometric features separate AI-edited from AI-generated text strongly (AUC 0.98) but separate AI-edited from human-written text less effectively (AUC 0.80).Edited text therefore remains close to its human source in the stylometric feature space.
  • Classification: EditLens shows the reverse pattern, reaching AUC 0.97 for human versus AI-edited text and TPR 91% for EL-L, while the ensemble combines complementary signals.The LR+EL-L ensemble reaches AUC 0.99 for AI-edit versus AI-gen and 0.94 for Human versus AI-edit.

7 Conclusion

The paper finds that AI generation has a stable stylometric core, whereas AI editing produces a qualitatively different trace that remains closer to human writing. These uses of LLMs should therefore be studied separately rather than treated as one phenomenon.

  • Conclusion: Entropy and lexical diversity remain consistently informative across generators and domains, while most other stylometric features depend strongly on conditions.Their importance persists under length controls and when individual generators or domains are held out.
  • Conclusion: AI editing weakly increases lexical diversity, decreases entropy, and makes lexical density its dominant signal rather than reproducing the generation footprint.Stylometrically, edited text remains closer to its human-written source than to fully generated text.
  • Conclusion: Generation and editing leave qualitatively different traces and should not be treated as a single phenomenon of “AI text.”The conclusion frames detectability as dependent on how LLMs are used.

Limitations

The study’s conclusions are bounded by its interpretable feature set, evaluated models and domains, and associational analyses. These constraints limit how broadly the findings can be generalized or interpreted causally.

  • Feature Scope: The analysis uses a relatively small set of interpretable, computationally lightweight stylometric features rather than the full space of linguistic or semantic signals.The selected features capture cross-condition patterns but do not exhaust possible signals.
  • Model and Domain Scope: The evaluated generators and domains do not exhaust modern text-generation settings, and large-scale AI-editing experiments remain limited to three open-weight editors.The main benchmark covers eight open-weight LLMs across five domains, with smaller proprietary-model analyses.
  • Model and Domain Scope: The domains primarily involve English long-form text, so results may not generalize to multilingual, conversational, or highly specialized writing.This is an explicit scope boundary of the evaluated domains.
  • Interpretation: Feature analyses are associational rather than causal: predictive importance does not establish that a feature intrinsically causes class separation.Some features may correlate with broader stylistic or dataset-specific properties not directly measured.
  • Data Construction: Sampling procedures control selected confounds, including document length, prompt complexity, and voice-style imbalance across source datasets.Examples include length stratification for Wikipedia and voice balancing for Reddit posts.

C.2 Condition-level similarity analysis

Feature rankings are broadly stable across generators but more heterogeneous across domains, with entropy and lexical diversity forming the most transferable core and distinctive conditions driving deviations.

  • Cross-condition agreement: ρ = 0.43 median pairwise correlation, ranging from −0.55 to 0.86, indicates moderate but heterogeneous agreement across conditions.Most generators cluster around rankings centered on entropy and lexical diversity.
  • Cross-condition agreement: GPT-OSS-120B is the primary outlier, with mean correlation ¯ρ = −0.075 and lowest agreement of ρ = −0.55 with ArXiv.Burstiness outranks entropy only for this condition.
  • Generator and domain perturbations: 0.84 mean top-5 Jaccard overlap across LOO-LLM folds shows that removing one generator typically preserves four of five important features.Entropy and lexical diversity remain dominant, including when GPT-OSS-120B is excluded.
  • Stable feature core: Entropy and lexical diversity remain top-ranked across every domain fold and nearly all generator folds, supporting their transferability.Lower-ranked readability and sentence-level features vary more across folds.
  • Length and decoding controls: 91.74% to 90.98% accuracy after removing explicit length features shows that dominant lexical-diversity and entropy signals are not reducible to document length.Length-stratified classifiers remain accurate from 90.22% to 95.27%, with lexical diversity top-ranked in every quartile.
  • Length and decoding controls: 91.74% to 91.64% accuracy after trimming the longest 10% of texts indicates that long-output truncation does not drive the global signal.Lexical diversity remains informative under strict length controls, while entropy is strongest within shorter texts.
  • Length and decoding controls: ρ = 0.86 to 0.93 ranking correlations across temperatures, with mean ρ = 0.89, show substantially stable feature ordering as sampling temperature changes.Lexical diversity remains highest-ranked at every temperature, while entropy stays among the five most informative features.

D Implementation Detail

The implementation uses controlled seed-level splits, fixed stylometric features, and pretrained editing-aware detectors alongside logistic-regression analyses.

  • Data splitting: 20 of 1,000 seed texts are held out for classifier training and tuning, while the remaining 980 seeds form the analysis corpus.Seed-level splitting prevents edited variants of training seeds from entering analysis.
  • Evaluation setup: Feature-rank comparisons across temperatures use permutation importance, with Table 9 reporting human-written versus AI-generated classification.The proprietary-generator subset is presented in Table 10 with features ordered by permutation importance.
  • Detector implementation: EditLens is evaluated using two pretrained checkpoints directly, without further training, under classification rather than regression.Inference uses maximum sequence lengths of 512 tokens for RoBERTa and 1,024 for Llama.

E Classifier Validation

Validation shows strong human-versus-generated discrimination, but complementary strengths when distinguishing AI-edited text from generated or human-written text.

  • Pairwise validation: ≥92.8% accuracy is achieved by all three methods for distinguishing human-written text from AI-generated text.This validates meaningful predictions before applying classifiers to the full analysis set.
  • Pairwise validation: 93.2% accuracy for logistic regression exceeds EditLens’s 65.8–77.6% when separating AI-edited from AI-generated text.The comparison uses the GEN:Edit test task.
  • Pairwise validation: 93.1% accuracy for EditLens exceeds logistic regression’s 69.8% when detecting editing against human-written text.EditLens is therefore stronger on the Human-versus-AI-edit task in this validation.

F.1 Threshold Sensitivity

Threshold and multiclass analyses preserve complementary detector behavior, while raw-text signals such as transition phrases remain outside the 14-feature stylometric representation.

  • Threshold sensitivity: 0.47 and 0.51 are logistic regression’s optimal thresholds, remaining close to 0.5 across evaluation settings.This suggests minimal calibration needs for logistic regression.
  • Threshold sensitivity: LR has higher TPR on AI-edit versus AI-gen, whereas EL-L has higher TPR on Human versus AI-edit detection.The complementary pattern persists across tuned-threshold and fixed-FPR evaluations.
  • Three-class classification: 0.754 accuracy, 0.753 macro-F1, and 0.910 macro-AUC are obtained by the three-class classifier over Human, AI-edit, and AI-gen.Per-class F1 is 0.887 for AI-gen, 0.705 for Human, and 0.668 for AI-edit.
  • Three-class classification: 31.9% of AI-edited texts are misclassified as Human versus 5.5% as AI-gen, while AI-generated text has 89.2% recall.This asymmetry makes Human versus AI-edit the hardest pairwise distinction.
Loading 2608.27855v1…