PaperlayerView exact revision on arXiv

Source-linked AI summary

When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era

Lotta Kiefer, Brisca Balthes, Christoph Leiter, Yamen Ajjour, Elena Schmidt, Steffen Eger

arXiv:2608.17979v1cs.CL

TL;DR

Authorship verification must distinguish authors despite writing-style shifts across genre, time, and AI-assisted writing. AVShift evaluates these shifts in German and finds that fine-tuned LLMs perform best across genres, while temporal drift strongly reduces performance and AI-era shift is not measurable in this benchmark.

  • Problem

    Writing style varies with context and time, challenging authorship verification benchmarks that provide limited evidence under multiple distribution shifts.

  • Method

    AVShift is a German benchmark of more than 150,000 text pairs evaluating feature-based, embedding-based, and LLM-based AV across genre, temporal, and AI-era shifts.

  • Results

    Fine-tuned LLMs generalize best across genres, reaching an F1 of up to 0.77, while temporal drift strongly degrades performance and AI-era shift shows no measurable effect.

  • Takeaways & Limitations

    Robust cross-genre AV depends on representations capturing author-specific characteristics despite contextual variation, supported by stylistically diverse training data.

  • Takeaways & Limitations

    AI-era shift was not quantified by the extent or type of human–AI interaction, so controlled datasets with known AI assistance are needed for further evaluation.

Abstract

from arXiv · show

Authorship verification (AV) assumes that an author's writing style remains sufficiently stable to distinguish it from that of other writers. In practice, however, this assumption is challenged by distribution shifts caused by changes in genre, time, and AI-assisted writing. Existing AV benchmarks typically study these factors in isolation and focus predominantly on English, limiting our understanding of model robustness under realistic conditions. We introduce AVShift, the first German benchmark for systematically evaluating AV under multiple distribution shifts. AVShift comprises over 150,000 text pairs spanning three genres and 21 years, enabling controlled evaluation of cross-genre, temporal, and AI-era shifts within a unified framework. We benchmark representative feature-based, embedding-based, and LLM-based approaches. Our experiments show that fine-tuned LLMs generalize best across genres and benefit substantially from stylistically diverse training data. We further demonstrate that temporal drift is one of the strongest factors affecting AV, with performance degrading significantly as the time gap between documents increases. In contrast, we find no evidence of a measurable AI-era distribution shift within AVShift. Finally, our feature analysis reveals stylistic features that remain stable across genres, while their relative importance varies depending on the specific genre transition. We release AVShift and our code for future research.

1 Introduction

Authorship verification is vulnerable to distribution shifts because writing style changes with context and time, while existing benchmarks largely isolate these factors and focus on English. AVShift addresses this gap with a German benchmark covering cross-genre, temporal, and AI-era shifts, and reports strong mixed-domain LLM performance, substantial temporal degradation, and no significant AI-era degradation.

  • Writing style varies with audience, topic, communicative situation, medium, and time, making authorship verification inherently exposed to distribution shifts.
  • Existing benchmarks typically study distribution shifts in isolation and focus predominantly on English, limiting understanding of realistic, cross-language AV robustness.
  • AVShift is the first German benchmark with more than 150,000 text pairs spanning cross-genre, temporal, and AI-era distribution shifts.Its settings cover forum posts, reviews, and fanfiction; more than two decades of writing; and periods before and after widespread genAI adoption.
  • The benchmark compares feature-based, embedding-based, and LLM-based approaches across all evaluation settings.
  • LLMs trained on mixed-domain data reach an F1 of up to 0.77 across genres, while temporal shifts degrade verification performance by up to 0.21 F1 and AI-era degradation is not significant.Feature stability also depends strongly on the genre pair.

2 Related Work

Prior authorship-verification research spans feature-based, embedding-based, and LLM-based methods, while distribution shifts have largely been studied in isolation and predominantly on English datasets. AVShift addresses this gap with a unified German benchmark covering cross-genre, temporal, and AI-era shifts.

  • Authorship Analysis Methods: Authorship-verification methods comprise feature-based, embedding-based, and LLM-based approaches.Feature-based methods use handcrafted stylistic features; embedding-based methods learn dense representations; LLMs jointly learn stylistic representations and verification.
  • Authorship Analysis Methods: Feature-based methods remain attractive for competitive performance, efficiency, and interpretability but have been shown to underperform modern neural methods.They represent documents with handcrafted stylistic features compared through similarity measures or statistical and neural classifiers.
  • Authorship Analysis Methods: LLMs offer a promising authorship-analysis approach, but closed-source zero- and few-shot systems rely on online APIs that limit forensic and privacy-sensitive deployment.GPT-3.5 and GPT-4 achieve competitive performance under zero- and few-shot prompting.
  • Out-of-Distribution Evaluation: Out-of-distribution authorship verification includes domain transfer and more challenging comparisons across different topics, genres, or platforms.Cross-topic authorship verification has received considerable attention, whereas temporal and AI-era shifts remain comparatively limited or challenging areas of study.
  • Multilingual Evaluation: Prior work has investigated individual distribution shifts largely in isolation and predominantly on English datasets, leaving a need for unified multilingual evaluation.Although multilingual models and benchmarks exist, authorship analysis remains heavily focused on English, and German-specific benchmarks have only recently become available.
  • Multilingual Evaluation: AVShift provides the first German benchmark that systematically evaluates authorship verification under cross-genre, temporal, and AI-era shifts within one framework.The benchmark directly addresses the gap identified in prior work.

3 Data Curation

AVShift is curated from a single German platform to compare authors across genres and over 21 years while reducing platform-specific confounds. Its three sub-benchmarks control genre, temporal, and AI-era shifts through aligned corpora and constrained pair construction.

  • Corpus construction: The corpus uses fanfiction.de’s fanfiction stories, reviews, and forum posts to study multiple genres by collecting texts from the same authors.The scraping pipeline begins with forum author profiles and gathers each author’s available texts across two to three genres.
  • Benchmark design: AVShift contains GenreShift, TimeShift, and AIShift, each targeting a complementary distribution shift.GenreShift evaluates cross-genre generalization, TimeShift temporal robustness, and AIShift changes associated with widespread genAI adoption.
  • GenreShift: GenreShift includes seven datasets spanning Forum, Review, Story, three cross-genre combinations, and a uniformly sampled Mixed dataset.Authors are split into 80% training, 10% validation, and 10% test partitions without overlap across splits.
  • Pair construction: Positive pairs use same-author texts and negative pairs different-author texts, with topic leakage reduced through genre-specific source constraints and balanced sampling.The benchmark samples at most five positive and five negative pairs per author, while TimeShift uses ten 12-month temporal-gap slices and AIShift four periods from Early (2004–2010) through AI (2023–2025).

4 Experimental Setup

The experiments benchmark feature-, embedding-, and LLM-based authorship verification methods under genre, temporal, AI-era, and cross-lingual evaluation settings. They use macro F1 as the primary metric and control for confounding factors such as text length and dataset size.

  • AV Models: Three representative AV paradigms are compared: handcrafted-feature XGBoost, multilingual style embeddings, and LoRA-fine-tuned instruction-tuned LLMs.XGBoost uses more than 4,000 stylistic features; MSR provides language-agnostic embeddings; the LLM answers whether two texts share an author.
  • Evaluation Metrics: Macro F1-score is the primary metric throughout, with Accuracy additionally reported in the Appendix.Macro F1 is chosen for the balanced binary classification setting.
  • GenreShift Evaluation: All three models are trained and evaluated across seven AVShift datasets to assess unseen-domain generalization and within-sample cross-genre verification.For MSR, embeddings remain fixed and only the decision threshold is tuned using Youden’s J statistic.
  • GenreShift Evaluation: Standardized GenreShift datasets normalize documents to 500 words and downsample each split to 3,640 samples, isolating genre effects from text length and dataset size.Short texts are extended with same-author documents, while longer texts are truncated.
  • Time Shift Evaluation: TimeShift evaluates the best in-domain model across temporal gaps from 0–12 months to 108–120 months, correlating temporal distance with verification performance.Both Pearson and Spearman coefficients are reported.
  • AIShift Evaluation: AIShift uses leave-one-era-out evaluation with uniformly sampled train and test sets of 4,122 and 1,374 pairs, respectively.One temporal period is held out while the other three provide training data, enabling controlled pre- and post-AI-era comparisons.

5 Results

AVShift results show that genre and temporal shifts substantially affect authorship verification, while stylistically diverse training improves cross-genre robustness. Temporal drift is pronounced, but the benchmark provides no evidence of a systematic AI-era shift.

  • Genre shifts: Gemma consistently outperforms XGB and MSR across all seven genre datasets, with stylistically diverse training substantially improving distribution-shift robustness.The Mixed Gemma model outperforms models trained directly on corresponding cross-genre datasets in every setting, indicating broad stylistic variation is more beneficial than specialization.
  • Genre shifts: In-domain F1 scores are highest for Review at 0.89, followed by Story at 0.80 and Forum at 0.78, making Forum the most challenging genre.Review-Forum is the easiest cross-genre setting, whereas Review–Story and Story-Forum are considerably more challenging.
  • Genre shifts: Gemma’s F1 drops approximately 0.2 when trained on stories or forum posts instead of reviews, while forum training provides weaker supervision for robust stylistic representations.Under controlled standardized conditions, MSR leads on Review and Story and XGB leads on Forum, although Gemma follows closely without significant differences (p ≥0.05).
  • Feature analysis: Handcrafted feature stability ranges from −1.70 to 0.90, with a mean of 0.22, median of 0.25, and 80% positive features across genres.Feature rankings vary by genre transition: overlap among the 100 most and least stable features ranges from 2% to 43%, while overall rankings correlate at Spearman’s ρ = 0.42 −0.72.
  • Temporal shifts: Temporal distance has a strong, statistically significant negative relationship with F1 across all genres (p < 0.05), and performance decreases steadily as the gap increases.Review declines from 0.90 at 0–12 months to 0.69 after 9–10 years; Forum decreases from 0.76 to 0.68, while genre difficulty remains Review > Story > Forum.
  • AI-era shifts: AVShift finds no systematic AI-era distribution shift: hold-out-era differences lack a consistent chronological pattern, and AI-era texts are not systematically harder to verify.Because AI-assisted writing prevalence is unknown, the benchmark cannot determine genAI’s impact; controlled datasets with known AI-use levels are needed.

6 Conclusion

AVShift provides a unified German benchmark for evaluating authorship verification under cross-genre, temporal, and AI-era distribution shifts. The conclusion highlights fine-tuned LLMs’ cross-genre strength, temporal drift’s substantial impact, stable authorial features, and the need to assess robustness beyond in-domain performance.

  • 6 Conclusion: AVShift is the first German benchmark unifying cross-genre, temporal, and AI-era evaluation for authorship verification under realistic distribution shifts.It compares feature-based, embedding-based, and LLM-based approaches beyond conventional in-domain settings.
  • 6 Conclusion: Fine-tuned LLMs perform particularly well in challenging cross-genre authorship verification, benefiting from stylistically diverse training data and matching in-domain performance in one setting.The conclusion identifies cross-genre verification as challenging while reporting the strongest performance for fine-tuned LLMs.
  • 6 Conclusion: Temporal drift is one of the strongest factors affecting authorship verification, with performance consistently declining as the time gap between documents increases.The conclusion presents temporal separation as a major source of performance degradation.
  • 6 Conclusion: Although handcrafted features are strongly influenced by genre, many stylistic features remain stable enough to support reliable cross-genre authorship verification.The analysis suggests robustness depends on capturing author-specific characteristics despite changes in writing conditions.
  • 6 Conclusion: Future authorship-verification research should measure robustness to realistic distribution shifts alongside in-domain performance, especially for non-English languages and real-world applications.The authors position AVShift as a resource for robust AV research where benchmark datasets remain scarce.

Limitations

The study’s conclusions are limited by its restricted model coverage, single-platform German benchmark, unquantified human–AI interaction, and focus on analysis rather than robustness-improving methods.

  • Model coverage: The evaluation covers only three representative AV approaches, while architecture, fine-tuning, threshold calibration, and feature-selection choices may affect results.Other approaches and within-category design choices could yield different findings.
  • Benchmark scope: AVShift uses a single German platform, limiting the diversity of writing environments and the generalizability of observed findings across platforms and languages.Additional platforms and languages are identified as future extensions.
  • Shift measurement: AI-era shift remains difficult to measure because the study does not quantify the extent or type of human–AI interaction.Controlled settings with known levels of AI assistance could address this limitation.
  • Research focus: The work analyzes robustness under distribution shifts rather than developing methods to improve it, leaving adaptive training and robust representations for future research.The benchmark and analyses are intended as a foundation for methods specifically addressing distribution shifts in AV.

Ethical Considerations … D GenreShift Full Results

The paper addresses ethical risks and responsible release practices, documents AVShift examples and experimental configurations, and reports that Gemma outperforms other models in GenreShift accuracy. It also notes that genre examples differ stylistically and that review texts achieve the highest performance.

  • Ethical Considerations: AV can support forensic investigations and plagiarism detection but may be misused for deanonymization, while imperfect predictions should not be treated as definitive real-world evidence.The authors also cannot fully exclude demographic, social, or contextual biases against specific groups.
  • Ethical Considerations: The dataset will use pseudonymized usernames, be research-only, and derive from publicly accessible content that still requires responsible privacy-conscious use.Model and dataset licenses are documented in Appendix I.
  • A AVShift Example: Genre examples from one author show informal forum writing with emojis, more literary stories, and informal reviews that directly address the story’s author.The examples are presented with English translations in Table 4.
  • B XGB Feature Configuration: Table 5 specifies the feature-based approach by pairing each feature abbreviation with a brief description.The feature configuration is documented as part of the XGB approach.
  • C Training Setup and Hyperparameters: LoRA fine-tuning of Gemma-4-31B-it uses four H200 GPUs, the hyperparameters of Kiefer et al. (2026), and updated software including PyTorch 2.12.1 and Transformers 5.12.1.The XGBoost model uses dedicated hyperparameter grid searches, while MSR uses original sentence embeddings and specified software environments.
  • C Training Setup and Hyperparameters: The complete environment and training setups are released in the GitHub repository to support reproducibility.This release accompanies the detailed model-specific software and tuning configurations.
  • D GenreShift Full Results: Gemma consistently outperforms the other models in accuracy, and the review genre reaches the highest performance, confirming the main F1 findings.Figure 5 provides accuracy results as a complement to the main results section.

E GenreShift Standardized Full Results

On the standardized GenreShift benchmark, cross-genre performance falls to a consistent F1 of 0.67–0.68, while mixed training no longer improves results. In-domain genre differences persist, with review data achieving substantially higher performance.

  • Cross-genre performance: 0.67–0.68 F1 is achieved consistently across all cross-genre pairs, showing that standardized cross-genre verification is more challenging.The results indicate that larger training sizes are required in this setting.
  • Training strategies: Mixed training provides no performance benefit on the standardized benchmark.This contrasts with the performance benefit observed in the unstandardized setting.
  • In-domain and model results: Review data again achieves substantially higher in-domain performance, preserving the genre differences observed previously.Gemma shares the best performance with either MSR or XGB across all datasets rather than consistently outperforming them.

F AI-Era Results · G Most and Least Stable Features

AI-era effects vary by genre without a consistent cross-genre pattern, while stable authorship signals include function-word and POS sequences, punctuation patterns, and genre-sensitive limitations in document-length and character-level features.

  • F AI-Era Results: No consistent pattern emerges across leave-one-era-out F1 differences, despite several significant differences between held-out eras.The evaluation covers Forum, Review, and Story.
  • F AI-Era Results: Forum performs worst on the AI-era test set, which is significantly below every other era (p < 0.05).The Pre-AI era is also significantly below the Early and Mid eras.
  • F AI-Era Results: Review reaches its highest F1 score of 0.89 on the AI-era test set, significantly outperforming all other eras.This is the clearest positive AI-era result among the reported genres.
  • F AI-Era Results: For Story, the AI era is significantly outperformed only by the Early era, while the Mid era performs significantly worse than all remaining periods.Thus, the direction and significance of temporal-era differences depend on the genre.
  • G Most and Least Stable Features: The most stable cross-genre features include function-word bigrams, POS bigrams, and characteristic punctuation-usage patterns.These findings support function words and POS sequences as stable indicators of writing style.
  • G Most and Least Stable Features: Average message length is among the least stable features, reflecting varying document lengths across genres.Many character 4-grams are also least stable, suggesting sensitivity to genre-specific variation.
  • G Most and Least Stable Features: Some word and POS sequences are unstable: examples include the bigram "als er" and the POS trigram "punct-propn-verb".This shows that stability does not apply uniformly to all function-word or POS sequences.

H Pairwise Feature Analysis · I Model and Data Licences

Feature stability varies substantially across genre transitions, despite moderate agreement in stability rankings. The paper also specifies model licensing and releases pseudonymized benchmark data with preprocessing and scraping code to support reproducibility.

  • H Pairwise Feature Analysis: Figure 6 reports F1 scores for all models across standardized GenreShift training and test splits, marking best test-set scores and significant superiority.Statistically significant superiority is indicated by an asterisk for p < 0.05.
  • H Pairwise Feature Analysis: Feature overlap is low across genre transitions, ranging from 5% to 27% for the 100 most stable features and 2% to 43% for the least stable.These results indicate that feature stability differs considerably between genre pairs.
  • H Pairwise Feature Analysis: No universally stable feature set exists across genre shifts, so explicit feature selection should analyze each genre pair separately.The conclusion follows from the low cross-transition overlap of stable features.
  • H Pairwise Feature Analysis: Spearman rank correlations of feature stability remain moderate at 0.42-0.72 across genre pairs.Thus, the most and least stable features vary substantially, while their relative rankings retain moderate agreement.
  • H Pairwise Feature Analysis: Figure 7 reports accuracy scores for all models across standardized GenreShift training and test splits, likewise marking best scores and significant superiority.Statistically significant superiority is indicated by an asterisk for p < 0.05.
  • I Model and Data Licences: Gemma-4-31B-it is released under the Apache 2.0 licence, permitting modification, fine-tuning, redistribution of fine-tuned models, and publication of research results.The experiments use models according to their respective licences.
  • I Model and Data Licences: The authors will provide AVShift with pseudonymized usernames for academic research, together with the complete preprocessing pipeline and scraping code.These releases are intended to support reproducibility, transparency, and future research while protecting original authors’ privacy.
Loading 2608.17979v1…