Source-linked AI summary
MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts
Matteo Greco, Anudeex Shetty, Andrea Tagarelli, Jey Han Lau
TL;DR
Existing LLM authorship-attribution benchmarks provide limited multilingual, long-form, and distribution-shift coverage. MultiGhostBench fills this gap with 928 books from five recent LLMs across six languages and three scripts, evaluating attribution under domain, author, and language shifts. No single method consistently performs best, while transformer-based detectors transfer generator information across languages with language-pair variation and statistical or fingerprint-based methods are more language-dependent.
Problem
Existing authorship-attribution benchmarks often focus on English, controlled settings, outdated models, or short texts, leaving multilingual long-form evidence under distribution shifts limited.
Method
MultiGhostBench benchmarks LLM authorship attribution using 928 books generated by five recent LLMs across six languages and three scripts under domain, author, and language shifts.
Results
No single method consistently performs best across settings; transformer-based detectors transfer generator-related information across languages with varying effectiveness, whereas statistical and fingerprint-based methods are more language-dependent.
Takeaways & Limitations
MultiGhostBench provides a resource for developing and evaluating authorship-attribution methods under multilingual long-form distribution shifts.
Takeaways & Limitations
The benchmark excludes low-resource and underrepresented languages, covers five LLMs, uses a small human-evaluation subset, and focuses on single-author non-adversarial settings.
Abstract
from arXiv · showhide
While existing work on LLM authorship attribution (AA) has made progress, available benchmarks remain limited, often focusing on English, controlled settings, or relatively outdated models, with the few multilingual studies considering only relatively short texts. We introduce MultiGhostBench, a multilingual benchmark comprising 928 books generated by five recent LLMs across six languages and three scripts, with an average length of approximately 59K words per book. The benchmark supports evaluation under domain, author, and language shifts. Evaluation of representative AA methods shows that no single method consistently performs best across settings, and performance generally degrades under distribution shifts. Transformer-based detectors can retain generator-related information across languages, although transfer effectiveness varies by language pair, whereas statistical and fingerprint-based detectors are more language-dependent. We envision MultiGhostBench as a valuable resource for the development and evaluation of robust AA methods. The dataset and code can be found at https://github.com/GrecoMT/MultiGhostBench.
1 Introduction
Existing LLM authorship-attribution benchmarks are limited in task scope, text length, language coverage, and model recency. MultiGhostBench addresses these gaps with multilingual long-form evaluation across multiple distribution shifts and method families.
- Prior work commonly frames detection as human-versus-AI classification, while authorship attribution identifies the specific generator LLM.Existing long-form efforts also remain monolingual.
- MultiGhostBench jointly evaluates multilingual long-form authorship attribution under domain, generator, and language shifts.The authors describe it as the first benchmark designed for this combination of settings.
- 928 books averaging 59K words were generated by five recent LLMs across six languages and three scripts.The benchmark evaluates attribution within and across languages under domain and generator shifts.
- No method performs consistently best across languages, data regimes, and distribution shifts.The evaluation covers statistical, supervised, and fingerprint-based attribution detectors.
- Transformer-based detectors retain generator-related information across languages, but transfer varies by language pair; statistical and fingerprint-based methods are more language-dependent.These findings motivate evaluating attribution beyond matched-language settings.
2 Related Work
Prior LLM authorship-attribution research is relatively sparse and often supervised, short-form, English-only, or incomplete in its cross-language evaluation. Multilingual studies have generally emphasized binary detection or limited attribution settings.
- Most prior LLM-generated-text research focuses on binary detection rather than identifying the specific generating LLM.Only a few studies examine authorship attribution.
- Existing authorship-attribution studies are predominantly supervised and have limited generalisation.A long-form robustness study remained restricted to English.
- Prior multilingual attribution benchmarks do not provide exhaustive cross-language evaluation or consistently isolate cross-lingual generalisation.MULTITUDE does not cover all dataset languages, while M4GT-BENCH focuses on cross-domain rather than cross-lingual generalisation.
3 Dataset
MultiGhostBench is a multilingual long-form book dataset designed to test authorship attribution under domain, author, and language shifts. It combines six languages and three scripts with a staged generation pipeline, cleaning procedures, and comparisons to human-written books.
- 3 Dataset: The dataset contains books generated by five recent LLMs across six languages, supporting attribution under domain, author, and language shifts.Its language-shift setting tests transfer from a training language to an unseen language.
- 3 Dataset: The six languages span four language families and three writing systems, enabling comparisons across related and distant language and script shifts.The languages are Italian, Spanish, German, English, Chinese, and Russian.
- 3.2 Multilingual Book Generation Pipeline: Books are generated through iterative multi-stage expansion conditioned on an outline, the previous segment, and a running narrative summary.This structure supports global coherence beyond a single generation step.
- 3.3 OOD Dimensions: OOD-Domain uses disjoint genres, OOD-Author holds out one generator, and OOD-Language trains on one language before testing on another.OOD-Language is not combined with OOD-Domain, isolating the language-shift effect from genre shift.
- 3 Dataset: Generated books show lexical diversity and redundancy comparable to human-written books, but higher Self-BLEU and lower perplexity.These differences indicate greater inter-book lexical similarity and higher predictability for the reference model.
- 3 Dataset: Human evaluation found generally fluent and coherent books, while narrative-level quality was less consistent across languages.The evaluation covered only gemini-pro literature books.
4 Authorship Attribution Methods
The benchmark evaluates metric-based, supervised model-based, and fingerprint-based attribution methods, adapting each to multilingual multi-class attribution. The methods differ in their reliance on reference-model statistics, learned representations, or post-hoc fingerprints.
- Metric-based methods: Metric-based methods RANK, ENTROPY, and GLTR use token-level probabilities from the multilingual mGPT reference model.Their extracted statistics are adapted to multi-class attribution.
- Model-based supervised methods: Supervised methods include N-GRAM, BERT-AA, and DETECTIVE, covering non-neural and neural detectors.Multilingual adaptations preserve language-specific characters and use multilingual encoders or sentence embeddings.
- Fingerprint-based methods: TRACE is a post-hoc fingerprint method that models transitions between token-rank or entropy statistics and attributes text to the most similar generator fingerprint.For multilingual evaluation, GPT-2 is replaced with Gemma as the evaluator.
- Fingerprint-based methods: TRACE evaluations were limited to English.
5 Experimental Setup
The evaluation compares attribution under low- and high-resource training, in-distribution and shifted settings, using calibrated confidence thresholds that support rejection of unseen generators.
- Evaluation settings: OOD-Author evaluation is open-set attribution because test texts may come from generators absent during training.
- Evaluation settings: Figure 1 encodes macro-F1 with bar height, detector identity with colors, evaluation settings with hatches, and ties with split colors.
- Evaluation settings: Methods are calibrated on development data with a confidence threshold balancing attribution performance and rejection ability.
- Evaluation settings: The same threshold is applied across ID, OOD-Domain, OOD-Author, and OOD-Language scenarios.
6 Experimental Results
Results show that attribution performance varies across methods, languages, resource regimes, and shifts. Transformer detectors transfer best across languages, but transfer remains language-pair dependent and distribution shifts generally reduce performance.
- In-Language Evaluation: No single method consistently outperforms all others across languages, resource regimes, and evaluation settings.
- In-Language Evaluation: XLM-ROBERTA is particularly competitive for ID and OOD-Domain evaluation, while N-GRAM frequently wins in high-resource OOD-Author settings.
- In-Language Evaluation: Higher-resource training improves best achievable performance across all ID and OOD-Domain configurations, sometimes approaching or reaching perfect macro-F1.
- In-Language Evaluation: Distribution shifts generally reduce performance, with OOD-Author usually causing larger drops than OOD-Domain.
- Cross-Language Evaluation: XLM-ROBERTA and DETECTIVE transfer substantially better across languages than metric-based, N-GRAM, and fingerprint-based methods, which often achieve near-zero performance.
- Cross-Language Evaluation: Italian–Spanish transfer is especially strong, reaching macro-F1 0.981 from Italian to Spanish in the high-resource setting.
7 Conclusion
The paper introduces a multilingual long-form benchmark for testing LLM authorship attribution under multiple distribution shifts. Its results expose substantial variation across methods and languages, while Transformer-based approaches show the strongest cross-language performance.
- MULTIGHOSTBENCH contains 928 long-form books generated by five recent LLMs across six languages, four language families, and three scripts.
- Within languages, no method consistently leads across languages, data regimes, or ID, OOD-Domain, and OOD-Author conditions.
- Under OOD-Language evaluation, XLM-ROBERTA and DETECTIVE substantially outperform other methods, while TRACE and metric-based methods fall to near-zero performance.
- Cross-language performance depends on source language, target language, language-family proximity, and script differences.
- The benchmark is intended to support development of robust multilingual attribution methods.
Limitations
MultiGhostBench does not cover the full diversity of multilingual LLM-generated text and is limited to five recent LLMs, a small human-evaluation subset, and single-author, non-adversarial settings.
- The benchmark excludes low-resource, underrepresented, and substantially different grammatical or morphological languages.
- The benchmark includes only five recent LLMs, leaving additional and future models for later extension.
- Human evaluation covers only a small subset of generated books.
- The study focuses on single-author and non-adversarial attribution, excluding mixed authorship and obfuscation attacks.
- Multilingual long-form generation remains challenging for originality, self-evaluation, and revision across languages, which are outside this work’s scope.
Ethical Considerations
The benchmark addresses authenticity and accountability concerns using multilingual generated books, while assessing textual diversity, repetition, redundancy, and predictability with language-aware metrics.
- The benchmark’s generated books may contain undesirable writing or plot elements, although safety guardrails and human evaluation reduced identified concerns.
- Language-specific tokenisers are used to reduce distortions in token-based metrics across languages.
- Self-BLEU measures inter-book lexical similarity, with higher values indicating lower diversity.
- N-gram Diversity measures distinct four-gram variety, with higher values indicating greater diversity.
- Self-Repetition and Compression Ratio assess repeated or predictable text, while Perplexity measures predictability using mGPT.
A.3 Human Evaluation
The appendix describes human evaluation of generated-book quality and the detector implementations used for attribution, including multilingual encoders, statistical features, and TRACE fingerprints.
- Human Evaluation: Each book is independently evaluated by two fluent volunteer annotators from diverse demographic backgrounds.
- Human Evaluation: Five passages and five book-level questions produce 30 Likert-scale questions per book.
- Attribution Methods: TRACE builds generator reference fingerprints from token-rank or entropy transitions and attributes texts to the most similar reference.
- Attribution Methods: XLM-ROBERTA processes long documents in 512-token chunks whose prediction scores are averaged at document level.
D Additional Results
Additional results compare TRACE fingerprints, thresholded and unthresholded attribution across domains and languages, and macro-F1 changes relative to in-distribution performance.
- TRACE Fingerprints: Figure 4 compares Italian training entropy fingerprints with Italian ID, Italian OOD-Domain, Spanish, and Chinese OOD-Language fingerprints.
- Reference Data: The benchmark’s human-reference textual metrics use full-length, single-authored Project Gutenberg books originally written in each target language.