PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning for Authorship Verification

Peter Kirby

arXiv:2609.28471v1cs.CLcs.LG

TL;DR

The paper asks whether classification or contrastive learning is more effective for authorship verification when other factors are controlled. It compares the approaches and reports that contrastive learning outperforms classification-based training in the tested settings, with a ModernBERT Bi-Encoder achieving state-of-the-art PAN21 performance.

  • Problem

    It remains unclear whether classification or contrastive learning is more effective when factors such as pre-trained model and input context length are controlled.

    Source: §1 · pp. 1–2

  • Method

    The paper compares the two training objectives across transformer architectures and explores loss functions, data augmentation, model choice, training duration, and context length.

    Source: §1 · p. 2, §2.1 · p. 2, §2.3 · p. 3

  • Results

    Contrastive learning outperformed classification-based training, and the 4096-token ModernBERT Bi-Encoder achieved state-of-the-art performance on PAN21.

    Source: §4 · p. 6, §3.5 · p. 6

  • Takeaways & Limitations

    The results support independently encoding texts and delaying pairwise comparison, with the computational advantages appearing to outweigh attention across every pair.

    Source: §4 · p. 6

    The authors leave performance on authorship attribution, style change detection, and other datasets for future work.

    Source: §4 · p. 6

Abstract

from arXiv · show

Our results show that contrastive learning outperforms a classification-based approach to authorship verification under the tested settings. We identify loss function, batch size, training duration, pre-trained model, input context length, and random text span data augmentation as important factors of model performance. Based on these considerations, we develop a ModernBERT Bi-Encoder model that achieves 98.4% accuracy on the PAN21 authorship verification task.

1 Introduction

Authorship verification asks whether two texts share an author, and PAN21 provides a relevant open-set setting with unseen authors and fandoms. The paper compares classification and contrastive training while examining the factors that may affect their performance.

1 Introduction

  • PAN21 tests on unseen authors and fandoms, and remains a setting used in authorship-verification evaluation.The passage notes continued dataset development and reporting of PAN21 results.Source: §1 · p. 1
  • Whether classification or contrastive learning is more effective remains unclear when factors such as pre-trained model and input context length are controlled.Prior work explored both objectives, but left their comparison under controlled conditions unresolved.Source: §1 · pp. 1–2
  • The main contribution is a comparison of classification and contrastive training across transformer architectures, with batch size and learning rate tuned separately and other training conditions held fixed.The passage points to Table 1 for the architectures compared.Source: §1 · p. 2, Table 1 · p. 2
  • The study examines loss functions, data augmentation, model choice, training duration, and context length, and releases code and the dataset with its validation split.Source: §1 · p. 2

2 Methodology

The methodology creates varied training pairs, compares pair classification with a bi-encoder contrastive approach, and selects and transfers experimental settings using validation data.

2.1 Data Augmentation

  • Randomly rotating each document to a word boundary before truncation creates varied text spans, while dynamic pair recombination yields many possible negative pairs.Documents are converted from fixed pairs into rows with author IDs; each batch randomly samples authors and two documents per author.Source: §2.1 · p. 2

2.2 Pair Classification

  • Pair classification concatenates two truncated texts into one sequence, trains with positive and negative pairs, and selects a prediction threshold on validation F1.The model is fine-tuned for sequence classification, including its projection head.Source: §2.2 · p. 3, Figure 1 · p. 3

2.3 Contrastive Learning

  • The bi-encoder uses shared weights to encode documents independently and applies temperature-scaled InfoNCE to raise positive-pair similarity and lower similarity to batch negatives.The loss uses one positive pair per sample and all different-author samples in the batch as negatives; prediction scores pairs by cosine similarity.Source: §2.3 · p. 3

2.4 Experiments

  • Experiments use validation data for model selection, hyperparameter tuning, and calibration, transfer selected settings across model sizes, and report both 10-epoch comparisons and full PAN21 test metrics.Batch size and learning rate are grid-searched by approach; the full test metrics use one 40-epoch model with validation-selected Platt scaling and abstention delta.Source: §2.4 · p. 4

3 Results

The results favor contrastive learning in the tested authorship-verification settings and identify loss choice, model and training conditions, pooling, augmentation, and context length as performance factors. The prior-work comparison reports state-of-the-art performance for a 4096-token ModernBERT Bi-Encoder.

3.1 Loss Function

  • Semi-hard pairwise contrastive loss performed worse than InfoNCE, Circle, and Multi-Similarity losses; proxy-based SoftTriple and Proxy Anchor failed with 238,815 training author classes.The reported loss-function comparisons include TinyBERT contrastive settings with untuned batch size, learning rate, and defaults.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 Model Comparison

  • Contrastive learning outperformed classification with 256-token input context while using less computation than 512-token classification.The comparison reports tuned models in Table 3.Source: §3.2 · p. 5, Table 3 · p. 5

3.3 Mean Pooling and Data Augmentation

  • Last-layer-only mean pooling performed better than mean first-last-layer pooling, while random text spans mitigated performance drop and overfitting.Longer training performed better with TinyBERT, which used a larger tuned batch size.Source: §3.3 · p. 5, Table 4 · p. 5

3.4 Context Length and Batch Size

  • A 1024-token context length increased performance over 512 tokens, while whether the added data supports a larger batch remains open.The comparison used ModernBERT-large Bi-Encoder.Source: §3.4 · p. 5, Table 5 · p. 6

3.5 Comparison to Prior Work

  • The 4096-token ModernBERT Bi-Encoder achieves state-of-the-art performance; Peng et al.’s concatenated-input BERT classification has F1 of 0.917 versus 0.881 for contrastive BERT.The comparison also notes that longer context, model choice, training duration, and data augmentation contribute to the result.Source: Table 6 · p. 6, §3.5 · p. 6

4 Discussion

The discussion argues that similarity-based deep learning with many negatives outperformed the classification approach in the tested setting. It frames contrastive learning as efficient independent text representation learning, while noting that other tasks and datasets remain for future study.

4 Discussion

  • Similarity-based deep learning with many negatives outperformed classification, but performance on authorship attribution, style change detection, and other datasets remains untested.The discussion interprets InfoNCE as identifying one positive among many negative batch candidates and says delayed cosine comparison appears to outweigh attention across text pairs.Source: §4 · p. 6

Related papers

Loading 2609.28471v1…