PaperlayerView exact revision on arXiv

Source-linked AI summary

GerAV: Towards New Heights in German Authorship Verification using Fine-Tuned LLMs on a New Benchmark

Lotta Kiefer, Christoph Leiter, Sotaro Takeshita, Elena Schmidt, Steffen Eger

arXiv:2601.13711v2cs.CL

TL;DR

German authorship verification lacks large, systematic benchmarks beyond English, especially for language-specific evaluation. The paper introduces GerAV, a 400k-plus-pair Twitter and Reddit benchmark, and systematically compares baselines with fine-tuned and zero-shot LLMs. Fine-tuned LLMs achieve the strongest reported performance, while specialization-generalization trade-offs remain across data regimes.

  • Problem

    Large-scale benchmarks and systematic evaluations for authorship verification remain scarce outside English, despite language-specific stylistic and linguistic dependencies.

  • Method

    GerAV combines German Twitter and Reddit text pairs into in-domain, cross-domain, profile-based, and mixed-source subsets for systematic AV evaluation.

  • Results

    Fine-tuned LLMs outperform established baselines by up to 0.09 F1 and surpass GPT-5 by 0.08 F1 on a small test subset.

  • Takeaways & Limitations

    GerAV provides a challenging German benchmark showing that AV performance depends on data source, domain, text length, training quantity, and language alignment.

  • Takeaways & Limitations

    The authors identify dataset bias, limited interpretability, partially addressed content influence, and a limited evaluation of fine-tuning strategies as key limitations.

Abstract

from arXiv · show

Authorship verification (AV) is the task of determining whether two texts were written by the same author and has been studied extensively, predominantly for English data. In contrast, large-scale benchmarks and systematic evaluations for other languages remain scarce. We address this gap by introducing GerAV, a comprehensive benchmark for German AV comprising over 400k labeled text pairs. GerAV is built from Twitter and Reddit data, with the Reddit part further divided into in-domain and cross-domain message-based subsets, as well as a profile-based subset. This design enables controlled analysis of the effects of data source, topical domain, and text length. Using the provided training splits, we conduct a systematic evaluation of strong baselines and state-of-the-art models and find that our best approach, a fine-tuned large language model, outperforms recent baselines by up to 0.09 absolute F1 score and surpasses GPT-5 in a zero-shot setting by 0.08. We further observe a trade-off between specialization and generalization: models trained on specific data types perform best under matching conditions but generalize less well across data regimes, a limitation that can be mitigated by combining training sources. Overall, GerAV provides a challenging and versatile benchmark for advancing research on German and cross-domain AV. Our code and information about data access are available on GitHub.

1 Introduction

The paper introduces GerAV to address the scarcity of large, systematic German authorship-verification benchmarks and evaluates fine-tuned LLMs across varied data conditions. Its best model substantially outperforms established baselines, while performance varies with text length, domain shift, training-data size, and language.

  • Motivation: The work targets a gap in language-specific AV research because existing evaluations are concentrated on English despite stylistic and linguistic dependencies on language.The authors motivate German benchmarking for forensic and other authorship-analysis applications.
  • Benchmark: GerAV contains over 400k German text pairs from Twitter and Reddit, including in-domain, cross-domain, profile-based, and mixed-source evaluation settings.The benchmark supports analysis of data source, domain, and available text quantity.
  • Main results: Gemma-3-12b trained on mixed GerAV data reaches 0.83 average F1, exceeding the best baseline by 0.09 and GPT-5 by 0.08 on a small test subset.The model is fine-tuned on a mix of training splits from all GerAV subsets and is significantly smaller than GPT-5.
  • Text length: Baseline performance rises from approximately 0.78 F1 at 25 words to almost 1.0 at 900 words, whereas tuned models require long training examples to show the same pattern.Input length benefits tuned models conditionally, depending on whether long examples occur in training data.
  • Domain generalization: Cross-domain performance is 0.06 F1 lower than in-domain performance, while cross-domain training improves cross-domain results but weakens performance in-domain and on other datasets.This reflects a trade-off between specialization and generalization across data regimes.
  • Low-resource and cross-language settings: Reducing training samples to one quarter lowers F1 by only 0.04, whereas applying an English-Twitter-trained model to GerAV causes a 0.12 F1 decrease.The comparison supports the importance of language-specific evaluation and models.

2 Related Work

Prior authorship-verification research spans feature-based, neural, and LLM-based methods, but LLM evaluations and non-English benchmarks remain limited. GerAV contributes a broad German benchmark and comparison framework addressing these gaps.

  • Existing methods: Feature-based methods use handcrafted morphological, syntactic, and semantic representations, while neural models learn style representations directly from data.Examples include character n-grams, RNNs, LSTMs, CNNs, and siamese architectures.
  • Methodological trade-offs: Feature-based approaches are interpretable, whereas neural approaches generally offer stronger adaptability but less interpretability.The related work contrasts explicit feature engineering with learned representations.
  • LLM-based AV: Prompted GPT-3.5 and GPT-4 can outperform traditional baselines without task-specific training, but API dependence raises reliability, reproducibility, and privacy concerns.These concerns motivate evaluation beyond online zero-shot systems.
  • Evaluation gaps: Existing LLM studies often compare only a small number of models or baselines, limiting evidence for superiority over strong alternatives.The paper positions its broad baseline and LLM comparison as a response to this limitation.
  • Multilingual settings: Non-English AV resources remain scarce, despite language-specific grammar, morphology, and writing conventions shaping authorial style.Prior multilingual resources include Wikipedia- and Reddit-based datasets, but the paper identifies a need for diverse German benchmarks.

3 Data Curation

GerAV is constructed from large German Reddit and Twitter corpora into five AV datasets spanning source, topical-domain, profile, and mixed-data conditions. The curation design controls author overlap, pair balance, and potential content or length biases.

  • Dataset design: The benchmark comprises five datasets derived from Reddit and Twitter, covering in-domain, cross-domain, profile-based, and mixed-source settings.The underlying social-media posts are relatively short compared with many existing benchmark sources.
  • Reddit corpus: The Reddit corpus contains 787,372 German posts from 120,538 users across 182 subreddits, after language filtering and account-quality filtering.Users with fewer than two posts are removed to enable positive and negative pair construction.
  • Pair construction: Authors are split 60/20/20 across training, validation, and test sets without author overlap, with up to two positive and two negative pairs sampled per author.Pair limits keep the datasets manageable for large-model training.
  • Domain control: The Reddit cross-domain dataset separates paired posts across 14 manually checked topical domains, unlike the in-domain dataset, whose pairs share a domain.This design tests whether models rely on topical similarity or stylistic signals.
  • Profile-based evaluation: The profile-based dataset concatenates each user’s posts and samples negative pairs with matched length-difference distributions to reduce exploitable length bias.It evaluates authorship verification when more text from each author is available.
  • Mixed and source data: The mixed dataset samples 20,000 training pairs and 4,000 validation and test pairs from each source with balanced labels, while GPT-5 uses a 480-sample version.Twitter contains 606,588 posts from 54,544 authors, and both corpora are dominated by short texts.

4 Experiment Setup

The experiments compare established AV baselines, zero-shot LLMs, and LoRA-tuned LLMs across all GerAV datasets using accuracy, F1, and ROC AUC. Thresholding and paired bootstrap testing support comparable performance and significance analyses.

  • Evaluation measures: Accuracy, F1, and ROC AUC measure binary decisions and score-based separation between same-author and different-author pairs.ROC AUC evaluates numeric similarity or distance scores across arbitrary thresholds.
  • Statistical testing: Paired non-parametric bootstrap tests use 10,000 resamples per dataset to estimate significance from matched F1-score distributions.Each model pair is evaluated on identical resampled labels.
  • Compared methods: The study benchmarks four established AV methods, two recent style-embedding models, four zero-shot LLMs, and LoRA-tuned LLMs across five datasets.The established baselines include ngram, PPM, hierarchical Bi-LSTM, and siamese BERT, producing 20 baseline models.
  • Thresholding: Baseline thresholds are tuned on validation data using Youden’s J statistic, which maximizes the sum of sensitivity and specificity.Tuned LLMs instead use zero as the accuracy and F1 threshold because they are trained on the training sets.
  • Zero-shot inference: Zero-shot LLM inference uses yes/no prompts and derives a confidence score by subtracting no-token probability from yes-token probability over the ten most likely tokens.Positive scores indicate greater probability of same-author text pairs.
  • LLM evaluation: GPT-5 is evaluated on 120 subsamples per dataset using standard, linguistically informed, and German-translated prompts, while tuned models comprise 20 LoRAs across four LLMs and five datasets.The comparison isolates prompt design and adaptation through both zero-shot and fine-tuned settings.

5 Experiment Results

Fine-tuned Gemma-3-12b models trained on mixed GerAV data achieve the strongest overall performance, while results vary with training domain, input length, language, and data volume. Mixed-source training improves generalization, but specialized training can excel under matching conditions.

  • Overall Performance: 0.83 overall F1 makes Gemma-3 (mix) the best model, exceeding Llama-3.1 (mix) and Qwen-2.5 (mix) by 0.03 and outperforming the best baseline, msr, at 0.74.Fine-tuned LLMs clearly outperform existing baselines, while msr remains the strongest baseline and a computationally attractive alternative.
  • Reddit vs. Twitter: 0.11 higher F1 on Twitter and 0.06 higher F1 on Reddit in-domain occur when Gemma models train on the corresponding source, indicating dataset-specific effects.The relevant characteristics include message style, topic, or length.
  • In-Domain vs. Cross-Domain: 0.06 higher average F1 on in-domain than cross-domain Reddit shows that cross-domain evaluation is harder, although mixed models match or outperform cross-domain-trained models there.The modest decrease suggests models retain meaningful stylistic signals despite content-controlled evaluation.
  • Message-Based vs. Profile-Based: 0.02 F1 separates Gemma-3 (mix) from its profile-based evaluation result, while models trained on matching profile data perform best on profile-based tests.Profile-based evaluation generally yields higher performance than message-based evaluation.
  • Message Length: Up to ρ=0.94 correlation between message length and F1 shows strong length sensitivity, with Gemma-3 (mix) best below about 100 words before longer-input models catch up.Gemma-3 (mix) plateaus around 500 words, while Gemma-3 (rpb) surpasses it after roughly 100 words.
  • Closed-Source Evaluation: 0.081 F1 separates Gemma-3 (mix) from GPT-5 on a stratified 480-sample subset, with the open-source model also significantly higher in accuracy by 0.031.The comparison uses a separate mixed-test subset and reports p<0.01.

6 Conclusion

The paper introduces GerAV to address limited non-English AV benchmarks and systematic evaluations involving LLMs and diverse baselines. Its results support fine-tuned LLMs, mixed-source training, reduced-data robustness, and language-specific evaluation for German AV.

  • 6 Conclusion: GerAV addresses scarce robust non-English benchmarks and limited systematic evaluations by combining a new benchmark with diverse baselines and fine-tuned LLMs.The benchmark supports analysis of German AV across multiple evaluation settings.
  • 6 Conclusion: Fine-tuned LLMs outperform traditional baselines, with Gemma-3-12b remaining robust across domain shifts and short and long inputs.The conclusion also reports that strong performance can persist after substantially reducing training data.
  • 6 Conclusion: Combined training datasets enhance generalization, while language-specific corpora matter because cross-lingual evaluation produces significant performance drops.These findings connect benchmark design with practical and future multilingual AV research.
  • 6 Conclusion: The results provide a foundation for future work on robust, high-performing AV models suitable for practical applications across languages.This conclusion remains within the paper’s stated scope of implications for future research and applications.

Limitations

The authors identify limitations involving dataset bias, interpretability, incomplete content controls, restricted fine-tuning evaluation, and absent human comparisons.

  • Dataset biases, unexplained model outputs, and insufficient control of topical content constrain confidence that predictions reflect authorial style.The authors call for interpretable features and more detailed analysis of content effects.
  • The fine-tuning evaluation covers a limited range of architectures, prompts, hyperparameters, and adaptation methods.Future work should also extend evaluation beyond German, especially to low-resource languages.
  • The study lacks human evaluation and direct model-versus-human comparisons needed to assess automated AV against expert judgment.The authors identify this as a necessary step toward reliably replacing expert judgment.

Ethical Considerations

The paper discusses misuse, privacy, anonymity, and fairness risks in authorship verification, while documenting dataset and model-use practices.

  • Authorship verification can enable stylistic obfuscation or undermine privacy and legitimate anonymity, so results should be applied cautiously in high-stakes settings.
  • The authors did not extract or predict personally identifiable information and report that the research data are publicly available with licensing information.
  • The authors do not investigate whether systems associate authorship with shared social or demographic characteristics, so model fairness cannot be guaranteed.

F Training Setup, Hyperparameters and Model Licences

The appendix describes LoRA training on H100 GPUs, reports model-dependent training times, specifies hyperparameters, and lists software configurations and model licences.

  • Training Setup: LoRA training runs on H100 GPUs, taking under an hour for Llama3.2-3B and up to 4:40 hours for Gemma-12B on mixed data.
  • Hyperparameters: Training uses one epoch, batch size 2, learning rate 3e-4, weight decay 0.01, LoRA rank 128, alpha 32, and zero dropout.
  • Model Licences: The appendix records software versions and provides licence names and links for all fine-tuned models.

G Accuracy, F1 and RoC_AuC on GerDATA

The experiments compare baseline and GerAV-tuned models across accuracy, F1, and RoC_AuC, with results organized by evaluation set and training or threshold-tuning condition.

  • Overall Results: Gemma-3 is the strongest comparison model, while German-pretrained LLäMmlein-7B-Chat trails it by an average 0.04 F1 points in the mixed setting.LLäMmlein outperforms same-sized Qwen-2.5-7B in two of five fine-tuning configurations, while Qwen performs better in three.
  • Evaluation Measures: Figures 8 and 9 report accuracy, F1-Score, and RoC_AuC across test sets for baselines and GerAV models, respectively.Figure 9 groups tuned models by their training subset, whereas Figure 8 distinguishes threshold-validation subsets.
  • Length Analysis: The mixed test-set length analysis uses 27 log-spaced bins after excluding bins containing fewer than two samples.

I Varying Number of Training Samples

Training-set size affects GerAV performance across five test subsets, most strongly for Reddit cross-domain evaluation. The mixed test set also reports sample distributions across log-spaced length bins.

  • I Varying Number of Training Samples: Table 6 describes the distribution of mixed-test samples across log-spaced length bins.
  • I Varying Number of Training Samples: Reddit cross-domain performance declines most when Gemma-3-12b is trained on fewer Mixed GerAV samples.The results suggest larger, more diverse training corpora help capture cross-topic stylistic variation required for authorship verification.
  • I Varying Number of Training Samples: Figure 10 evaluates Gemma-3-12b across varying Mixed GerAV training-set sizes and all five GerAV test subsets.

J Data licence and reproducibility

GerAV uses distinct access and licensing arrangements for its Twitter and Reddit components. Reproduction is supported through controlled data access, preprocessing instructions, and released code or URLs.

  • J Data licence and reproducibility: Twitter data is accessible through Zenodo with restricted academic-research access, and GerAV preprocessing and splits can be reproduced from the GitHub code.
  • J Data licence and reproducibility: The original source licences apply after access, with further access and preprocessing information available on the GerAV GitHub repository.
  • J Data licence and reproducibility: Reddit data cannot be directly redistributed because the reddit4researchers API prohibits it; academic researchers instead receive access to post URLs.
Loading 2601.13711v2…