PaperlayerView exact revision on arXiv

Source-linked AI summary

Writerslogic at PAN 2026: Process over Content for Robust Detection under Domain Shift

David L. Condrey

arXiv:2610.03565v1cs.CL

TL;DR

The paper examines why text-classification features that perform well on validation may fail under distribution shift. It develops a support-overlap framework to guide system design across three PAN 2026 tasks, with strong results on two evaluated tasks and no official MAWSA result.

  • Problem

    The paper addresses feature robustness under distribution shift, where validation performance can fail to transfer to test data.

    Source: §1 · p. 1

  • Method

    It uses support overlap to classify features and guide task-specific systems, including process-oriented features and query-refusal decomposition.

    Source: §1 · p. 1, §4.3 · p. 5, §9 · p. 10

  • Results

    Reasoning Trajectory Detection placed 1st in source detection (0.85 F1) and 3rd in safety classification (0.66 F1); Voight-Kampff reached 0.891 on the PAN 2026 test set, while MAWSA had no official results.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Across the evaluated systems, the framework made the distinction between generated content and generation process a useful axis for feature selection.

    Source: §9 · p. 10

    The framework was developed retrospectively by one researcher, its categories are coarse, and it remains a hypothesis supported by two evaluated tasks rather than an established law.

    Source: §9 · p. 10, §8.1 · p. 10

Abstract

from arXiv · show

We describe the Writerslogic systems for three PAN at CLEF 2026 shared tasks (Reasoning Trajectory Detection, Voight-Kampff Generative AI Detection, and Multi-Author Writing Style Analysis), unified by a shared analytical framework: feature robustness under distribution shift is governed by support overlap between training and test distributions, not by training-set effect size. This yields a taxonomy (domain-anchored, domain-portable, domain-invariant) that explains why generator-specific features die under domain shift while vocabulary fingerprints (hapax ratio, Yule's K, Heaps' exponent), compression measures, and character n-grams survive. On Reasoning Trajectory Detection, where training was entirely mathematics and 84 percent of test was unseen domains, the framework guided system design to 1st place in source detection (0.85 macro F1 via Opus-Sonnet agreement) and 3rd place in safety classification (0.66 macro F1 via query-refusal decomposition). For Voight-Kampff, we built a calibrated ensemble of DeBERTa-v2 (ONNX), multi-seed LightGBM with 44 domain-portable stylometric features, and SVM on n-gram TF-IDF, combined via learned stacking with isotonic calibration; the best configuration achieved 0.891 on the PAN 2026 test set with balanced sub-metrics (0.853 to 0.902 across all evaluation dimensions). For Multi-Author Writing Style Analysis, we describe a system fusing spectral clustering over character n-gram similarity graphs, normalized compression distance for local boundary detection, and SmolLM-135M perplexity for neural change-point detection; a platform mix-up meant our run never reached the official evaluation, so we report the design and its a priori predictions. Across all three tasks, features measuring generation process properties are designed to outperform features measuring generated content properties under domain shift.

1. Introduction

The paper argues that feature robustness under distribution shift depends on support overlap, not training-set effect size. It applies this framework across three PAN 2026 tasks, with varied evaluation outcomes.

1. Introduction

  • 0.97 macro F1 on validation fell to 0.69 on test for LightGBM, while XLM-R safety classification fell from 0.83 to 0.45.These results illustrate the validation-to-test gap on the Reasoning Trajectory Detection task.Source: §1 · p. 1
  • Support overlap, rather than training-set effect size, governs feature robustness under distribution shift and motivates the paper’s taxonomy.The framework also identifies the validation illusion and granularity trap as consequences of this principle.Source: §1 · p. 1
  • Reasoning Trajectory Detection placed 1st in source detection (0.85 F1) and 3rd in safety classification (0.66 F1).The framework was developed retrospectively on this task, where generator-specific features with 0% test fire rate explained the validation-test gap.Source: §1 · p. 1
  • Voight-Kampff reached 0.891 on the PAN 2026 test set, while the MAWSA run produced no official results.The Voight-Kampff system used DeBERTa-v2, LightGBM, and SVM; MAWSA is presented as an untested design with a priori predictions.Source: §1 · p. 2
  • The paper’s unifying thesis is that generation-process features are more robust under domain shift than generated-content features.The three-task participation applied this shared analytical framework to system design.Source: §1 · pp. 1–2

2. Related Work

The related work spans AI-text detection, cross-domain generalization, vocabulary richness, safety classification, and ensemble calibration. The paper positions its feature-robustness taxonomy as an explanation for why heterogeneous detection signals can complement one another.

AI-generated text detection

  • AI-text detection approaches use probability signatures such as DetectGPT, GLTR, and Binoculars, stylometry [8], or fine-tuned transformers [10].Fine-tuned methods can have high in-distribution accuracy but degrade under domain shift; heterogeneous ensembles have outperformed individual detectors.Source: §2 · p. 2

Cross-domain generalization

  • Cross-domain work includes domain generalization, spectral alignment, lexical pair dispersion, inverse-perplexity-weighted ensembles, and shift-invariant decomposition.The paper relates its safety-classification system to the decomposition principle described by Ramponi and Plank.Source: §2 · p. 2

Vocabulary richness as detection signal

  • Vocabulary-richness research studies Yule’s K [21], Heaps’ law, and within-author stability across genres.Prior work also reports within-author regularity in hapax legomena and degradation of hapax counts under recursive LLM generation [27]; the paper extends these signals to human/AI distinction across tasks.Source: §2 · p. 2

LLM safety classification

  • Prior safety work reports guardrail drops on novel attacks [28], poor calibration, and multilingual bypasses.The paper’s 22-language refusal detector addresses the complementary task of recognizing refusals regardless of language.Source: §2 · p. 2

Ensemble methods and calibration

  • Stacking trains a meta-learner on base-classifier outputs, while isotonic regression and Platt scaling calibrate probabilities.The paper contrasts its simpler agreement-based multi-LLM filtering with Mixture-of-Agents.Source: §2 · p. 2

3. Feature Robustness Framework

The framework classifies features by whether their non-default values overlap with the test distribution and whether their class separation persists. It uses fire rates to distinguish anchored, portable, and invariant features and to explain why validation performance may not transfer.

Fire rate

  • A feature’s fire rate is the fraction of test samples where it produces a non-default value, measuring support overlap.A feature with 0% fire rate is dead and contributes nothing regardless of its training-set effect size.Source: §3 · pp. 2–3

Feature robustness taxonomy

  • Generator-specific features had strong effect sizes (d>1.5) but 0% test fire rate, whereas vocabulary fingerprints had moderate effect sizes and 100% fire rate.This contrast in Table 1 illustrates why training-set effect size alone does not establish robustness under shift.Source: Table 1 · p. 3

Domain-Anchored

  • Domain-anchored features have zero support overlap, such as has_final_answer, which fired on 93.6% of training data but 0% of test data.Such features detect specific generators writing about specific topics rather than AI-generated text generally.Source: §3 · p. 3

Domain-Portable

  • Domain-portable features fire across domains but lose class separation; has_boxed_answer appears in 99% of AI math solutions but 29% of non-math test data.Source: §3 · p. 3

Domain-Invariant

  • Domain-invariant features retain stable support and separability; hapax ratio, Yule’s K, and Heaps’ exponent had 100% fire rate.The passage attributes their human/AI differences to distinct vocabulary tail-frequency distributions from LLM sampling and human compositional writing.Source: §3 · p. 3

A priori criterion and validation illusion

  • Features referencing a specific generator, output format, or topic can be flagged as likely anchored by inspecting their definitions, without test data.On Reasoning Trajectory Subtask 2, XLM-R’s highest validation F1 (0.834) accompanied the lowest test F1 (0.448), a gap of −0.386.Source: §3 · p. 3

4. Task 1: Reasoning Trajectory Detection

The task tests source detection and safety classification under substantial domain and trace-format shifts, then reports separate approaches and outcomes for each subtask.

4.1. Distribution Shift

  • Training was entirely mathematics, while 84% of the source-detection test set covered unseen domains; safety test traces averaged 424 characters across 15+ languages versus 8,000–12,000 characters in training.Safety traces also shifted from 40–56 steps in one training language to five steps across 15+ languages and four script families.Source: Figure 1 · p. 4, §4.1 · p. 4

4.2. Subtask 1: Source Detection (1st Place, 0.85 F1)

  • Adding vocabulary fingerprints improved validation F1 by +0.172 but test F1 by only +0.009 for the LightGBM ensemble.The ensemble used five seeds and 30 features after removing five dead generator-specific features and adding seven vocabulary fingerprints.Source: §4.2 · p. 4
  • 0.850 test F1 came from Opus–Sonnet agreement, whose largest gain (+0.10) was attributed to capability-tier agreement rather than feature engineering.Disagreements were resolved with LightGBM.Source: §4.2 · p. 4, Table 2 · p. 4

4.3. Subtask 2: Safety Classification (3rd Place, 0.66 F1)

  • The query-refusal system classified query harm and detected refusals across 22 languages, achieving 0.640 test F1 before the final multi-signal union reached 0.660 test macro F1.The final system assigned trace-level labels uniformly to every step; its soft F1 reflected benchmark label coherence, not unsafe-step localization.Source: §4.3 · p. 5
  • Step-level estimators degraded performance because they had to exceed 1 − ε_trace accuracy to beat uniform label propagation, and neither tested estimator cleared that benchmark-dependent threshold.The authors therefore report the uniform-label outcome as a benchmark observation, not evidence of step-localization capability.Source: §4.3 · p. 5

4.4. Leaderboard

  • 0.85 macro F1 placed source detection 1st of nine teams, while safety classification ranked 3rd of seven on macro F1 (0.66) and 2nd on soft F1 (0.27).The leaderboard comparison places the two subtasks alongside the top-five entries for each.Source: Table 4 · p. 6, §4.4 · p. 6

5. Task 2: Voight-Kampff Generative AI Detection

Voight-Kampff applies the feature-robustness framework to cross-genre AI-text detection, combining domain-portable features with a calibrated ensemble. The best configuration achieved balanced results across PAN 2026 evaluation dimensions.

5. Task 2: Voight-Kampff Generative AI Detection

  • The feature choices followed evidence that generator-specific signals had 0% test fire rate, while vocabulary fingerprints and compression measures retained 100% fire rate across domains.The task classifies human- versus AI-written text across genres including essays, news, and fiction.Source: §5 · p. 6

5.1. Feature Engineering

  • The 44 domain-portable features include vocabulary fingerprints and compression measures that capture vocabulary use and text predictability rather than content.The features were selected to retain meaningful values across genres, avoiding genre-specific or generator-specific signals.Source: §5 · p. 6, §5.1 · p. 7

5.2. System Architecture

  • The system combines DeBERTa-v2, multi-seed LightGBM, and calibrated SVM probabilities using a learned logistic stacker, interaction features, and isotonic calibration.LightGBM uses stylometric features alongside truncated n-gram TF-IDF projections and, when available, GPT-2 perplexity features.Source: §5.2 · p. 7

5.3. Results

  • 0.891 ROC-AUC on PAN 2026 and 0.979 on PAN 2025 were achieved by large-rank; its PAN 2026 scores were 0.853 C@1, 0.902 F1, 0.899 F0.5u, and 0.887 mean.large-rank used all three classifiers, learned stacking, and isotonic calibration, placing 7th by mean score on PAN 2026.Source: Table 6 · p. 8, §5.3 · p. 8
  • The 0.880 primary score for brown-velocity accompanied secondary metrics of 0.519, 0.475, and 0.463, unlike large-rank’s cross-metric balance.The paper describes this contrast as a threshold-collapse pattern and says large-rank’s metric symmetry validates the feature design and calibration pipeline.Source: §5.3 · p. 8, Table 6 · p. 8

6. Task 3: Multi-Author Writing Style Analysis

The MAWSA system was designed for paragraph-boundary author-change detection under adversarial mimicry by combining signals chosen for resistance to conscious control. The run did not receive official evaluation, so its performance statements are registered predictions, not measured results.

6. Task 3: Multi-Author Writing Style Analysis

  • No official MAWSA results exist because the run was not submitted to TIRA within the evaluation window; the stated predictions are a priori and untested.The paper attributes the missed submission to a mix-up between the CodaBench and TIRA endpoints.Source: §6 · p. 8, §6.2 · p. 8
  • The hard subset uses adversarial mimicry, where consciously adapted vocabulary and sentence length become unreliable signals; the design instead favors signals described as harder to control.The task detects author switches at paragraph boundaries across easy and hard subsets.Source: §6 · p. 8

6.1. Signal 1: Spectral Eigen-Cut

  • Character n-gram TF-IDF similarities form an affinity graph clustered into two groups, with boundaries marked where adjacent paragraphs receive different cluster labels.The spectral partition uses global graph structure to separate paragraphs that may appear stylistically similar under mimicry.Source: §6.1 · p. 8

6.2. Signal 2: Normalized Compression Distance

  • Normalized compression distance compares adjacent paragraphs using zlib-compressed lengths; it requires no training data and is most effective above roughly 200 characters.The system sets its threshold dynamically at the NCD mean plus 1.5 standard deviations.Source: §6.2 · pp. 8–9

6.3. Signal 3: SmolLM-135M Perplexity

  • SmolLM-135M supplies token surprisal from which paragraph-level mean, variance, and trend features feed RBF-kernel change-point detection.The model was chosen partly because its approximately 270 MB size fits the containerized TIRA constraints.Source: §6.3 · p. 9

6.4. Ensemble Fusion and Predictions

  • For documents with at least five paragraphs, spectral and NCD signals are combined by relaxed disjunction, with SmolLM as a secondary check; shorter documents use NCD alone.The untested predictions are over 80% retention for character n-gram TF-IDF and NCD on the hard subset, and under 30% boundary detection for word-level TF-IDF cosine similarity.Source: §6.4 · p. 9

7. Cross-Task Analysis

Across tasks, vocabulary fingerprints, compression features, and character n-grams recur as process-level features, while feature reuse in MAWSA remains design intent rather than measured transfer. The systems also share heterogeneous classifier combinations and signal-fusion strategies.

7.1. Feature Reuse

  • Vocabulary fingerprints and compression features were reused across tasks, but MAWSA overlap is design intent rather than measured transfer because only two tasks were officially evaluated.Content-anchored generator markers were task-specific.Source: §7.1 · p. 9

7.2. Shared Architectural Patterns

  • Both evaluated systems combine heterogeneous classifiers: DeBERTa, LightGBM, and SVM for Voight-Kampff, and Opus, Sonnet, and LightGBM for Trajectory.The paper frames classifier diversity as useful because different classifiers fail on different inputs.Source: §7.2 · p. 9
  • Voight-Kampff uses learned stacking and isotonic regression, while Trajectory S2 uses union voting; both systems combine signals rather than optimize individual components.Vocabulary fingerprints, compression features, and character n-grams are presented as process-level features measuring how text is produced rather than what it says.Source: §7.2 · p. 9

8. Discussion

The discussion connects cross-domain performance to classifier agreement and label decomposition, then draws implications for task design while noting limits in the framework’s evidence and scope.

8. Discussion

  • Decomposition separates harmfulness, refusal detection, and trace interpretation by their invariance to trace length or query language.The first two components were shift-tractable; trace interpretation was avoided.Source: §8 · pp. 9–10
  • 0.85 F1 from Opus–Sonnet agreement exceeded 0.69 F1 from 30 engineered features; their calibrated combination reached 0.75.LightGBM was reliable on extreme cases but not in the middle, while the LLMs were generally decent but sometimes wrong on easy cases.Source: §8 · p. 10
  • Task designers should consider shift-representative development data because IID validation may mislead and validation rankings may invert on test.The authors recommend noting when IID validation may be misleading.Source: §8 · p. 10
  • Naive uniform step labels placed second on soft F1, suggesting test labels are largely trace-coherent rather than demonstrating step localization.The authors characterize this as a benchmark property, not a system property.Source: §8 · p. 10

8.1. Limitations

  • The framework comes from one researcher’s work and was developed retrospectively on Reasoning Trajectory, with Voight-Kampff results confirming its predictions.The passage describes both submissions as using the same analytical lens.Source: §8.1 · p. 10
  • Trajectory systems rely on proprietary Claude and GPT-4o APIs, bringing cost, latency, and reproducibility challenges.Source: §8.1 · p. 10
  • The reason Opus–Sonnet agreement succeeds is unknown: capability-tier asymmetry, same-family agreement, and Opus’s individual classification ability remain alternatives.Source: §8.1 · p. 10
  • The framework’s three categories are coarse, robustness varies continuously, and feature-level robustness does not guarantee system-level generalization.Source: §8.1 · p. 10

9. Conclusion

The conclusion presents a support-overlap framework for feature robustness under distribution shift and applies it to system design across tasks. The authors report competitive task outcomes but characterize the framework as a hypothesis supported by two evaluated tasks.

9. Conclusion

  • The framework holds that robustness under distribution shift is governed by support overlap, not training-set effect size, and proposes a diagnostic, taxonomy, and two consequences.The diagnostic is to compute feature fire rates on out-of-domain text; the consequences are the validation illusion and granularity trap.Source: §9 · p. 10
  • Vocabulary fingerprints and compression features informed design for Voight-Kampff and Reasoning Trajectory, while query-refusal decomposition guided Trajectory safety classification.Source: §9 · p. 10
  • 1st place on source detection (0.85 F1) and 2nd on safety soft F1 (0.27) were achieved using the framework’s practitioner principles.Source: §9 · p. 10

Declaration on Generative AI

The author reports using generative AI tools for code development, experiment orchestration, literature search, synthetic training data, and test-sample classification, while retaining responsibility for the publication.

Declaration on Generative AI

  • Claude Code assisted with development and research tasks, while GPT-4o and GPT-4o-mini generated synthetic data and classified test samples for step-level experiments.The author states that they reviewed and edited the content and take full responsibility for the publication.Source: Declaration on Generative AI · p. 11

Related papers

Loading 2610.03565v1…