PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn-to-Distance: Distance Learning for Detecting LLM-Generated Text

Hongyi Zhou, Jin Zhu, Kai Ye, Ying Yang, Erhan Xu, Chengchun Shi

arXiv:2601.21895v2cs.CLcs.AIstat.ML

TL;DR

The paper studies reliable detection of LLM-generated text, including text produced under unobserved prompts. It develops a geometric explanation of rewrite-based detection and an adaptively learned distance, which outperforms fixed-distance approaches and achieves substantial gains across broad experiments.

  • Problem

    The paper addresses the need to distinguish human-written from LLM-generated text when prompts are unobserved, amid concerns about the human-like quality of LLM output.

  • Method

    The method uses rewrite-based detection with a distance function learned adaptively between original text and its LLM reconstruction.

  • Results

    An ablation study reports a 97.1% average improvement over the fixed-distance version of the approach.

  • Takeaways & Limitations

    Geometric analysis supports rewrite-based detection under unseen prompts, while adaptive distance learning provides gains over fixed-distance approaches and state-of-the-art detectors.

  • Takeaways & Limitations

    The method has relatively high computational cost, comparable to existing rewrite-based algorithms, because it generates multiple rewrites.

Abstract

from arXiv · show

Modern large language models (LLMs) such as GPT, Claude, and Gemini have transformed the way we learn, work, and communicate. Yet, their ability to produce highly human-like text raises serious concerns about misinformation and academic integrity, making it an urgent need for reliable algorithms to detect LLM-generated content. In this paper, we start by presenting a geometric approach to demystify rewrite-based detection algorithms, revealing their underlying rationale and demonstrating their generalization ability. Building on this insight, we introduce a novel rewrite-based detection algorithm that adaptively learns the distance between the original and rewritten text. Theoretically, we demonstrate that employing an adaptively learned distance function is more effective for detection than using a fixed distance. Empirically, we conduct extensive experiments with over 100 settings, and find that our approach demonstrates superior performance over baseline algorithms in the majority of scenarios. In particular, it achieves relative improvements from 54.3% to 75.4% over the strongest baseline across different target LLMs (e.g., GPT, Claude, and Gemini). A python implementation of our proposal is publicly available at https://github.com/Mamba413/L2D.

1 INTRODUCTION

The paper addresses the need to distinguish human-written from LLM-generated text, especially when generation prompts are unobserved. It introduces an adaptively learned rewrite-based distance and supports it with geometric analysis and broad empirical evaluation.

  • Related approaches: Existing passive detectors comprise zero-shot and ML-based approaches, each including logits-based, rewrite-based, and other methods.This classification yields six combinations.
  • Motivation: The paper targets passive detection of LLM-generated text when the prompts used to generate it are unobserved.The proposal is categorized as ML-based, rewrite-based detection.
  • Method: The proposed method adaptively learns the distance between original and rewritten text instead of relying primarily on a fixed distance.The authors report that the learned distance better discriminates between LLM- and human-authored text.
  • Theory: The paper develops a geometric framework explaining rewrite-based detection, its generalization to unobserved prompts, and the rationale for learning the distance function.The framework connects the method to Propositions 1–3.
  • Empirical evaluation: Across 24 datasets, 7 target language models, 3 unseen-prompt types, and over 100 settings, the approach outperforms 12 state-of-the-art methods.Reported average relative improvements over the strongest baseline range from 41.5% to 75.4% across target LLMs, while learning the distance improves 96% on average over a fixed distance.

2 REWRITE-BASED METHODS: BUILDING INTUITION

Rewrite-based detection compares a text with an LLM reconstruction, exploiting the expectation that LLM-generated text is closer to its reconstruction than human text. A geometric framework explains this separation and its robustness to unseen prompts when rewriting perturbations remain small.

  • Building intuition: Rewrite-based methods prompt the target LLM to rephrase text and measure the distance between the original X and reconstruction R(X).They rely on machine-generated text being closer to its reconstruction than human-authored text.
  • Building intuition: The framework embeds texts in a measurable space and models human and target-LLM texts as distributions over corresponding subspaces.The target LLM subspace M is connected to human-text subspace H through a projection operator.
  • Building intuition: Under the projection assumption, an LLM-generated text can be viewed as a projection of human-written text onto the target LLM subspace.The assumption reflects the proposed geometric relationship between human- and LLM-authored text.
  • Building intuition: The rewriting process is modeled as projection onto the LLM subspace followed by a small perturbation that preserves projected semantic meaning.The perturbation accounts for stochastic LLM outputs.
  • Building intuition: Human text has a larger average reconstruction error than LLM-generated text, with equality only when the LLM output space perfectly replicates the human text space.Geometrically, reconstructions lie in the LLM subspace, while human text may lie farther from it.
  • Generalization to unseen prompts: Rewrite-based detection can generalize to unseen prompt shifts when the rewriting perturbation does not substantially distort the projected text’s semantic meaning.The resulting error gap depends on the human–LLM projection gap and perturbation magnitude.
  • Empirical illustration: Figure 2 reports clearer separation from the proposed learned distance than from a fixed distance, while Fast-DetectGPT distinguishes texts effectively only when the generation prompt is known.These comparisons use human-written and LLM-rewritten news text.

3 ADAPTIVE DISTANCE LEARNING

The section identifies fixed distance functions as a limitation of rewrite-based detection and motivates learning a target-LLM-dependent distance. The proposal learns this distance from human and LLM-generated corpora, averages discrepancies across multiple reconstructions, and classifies texts by a threshold.

  • Limitations: Fixed, hand-crafted distances may generalize poorly across target language models, datasets, and unseen prompts.Existing rewrite-based methods use measures such as N-gram, Levenshtein, and negative BERTScore or BARTScore distances.
  • Optimal distance: The optimal distance is zero for two LLM-generated texts and maximal for a mixed LLM-generated and human-written pair.This distance depends on the target LLM because different models induce different generative subspaces.
  • Proposal: The proposal adaptively learns a distance function from human-authored and target-LLM-generated corpora.Human data may come from sources such as Wikipedia, while generated data can be produced by prompting the target model.
  • Distance parameterization: The learned distance dϕ is parameterized by a language model and satisfies non-negativity, identity of indiscernibles, and the triangle inequality.The token-length normalization is part of the parameterization, and dϕ is therefore a pseudo-distance.
  • Detection procedure: The method fine-tunes a small parameter subset, generates multiple reconstructions, averages their distances, and applies a threshold to classify the text.The implementation can update only the final layer or use LoRA; smaller average distance indicates an LLM-generated text.

4 EXPERIMENTS

The experiments evaluate L2D against representative detectors across diverse datasets, target models, unseen prompts, adversarial attacks, and a fixed-distance ablation. L2D achieves the best performance in most settings, with substantial gains and robustness under attacks.

  • Experimental setup: The evaluation includes 21-domain data with LLM text generated by four target models, plus benchmark datasets spanning Wiki, Story, and News.The benchmark settings include rewrite, expand, and polish prompts and proprietary GPT, Claude, and Gemini models.
  • Results on diverse datasets: L2D achieves the best performance across nearly all combinations of datasets and target models, outperforming ImBD on 16–19 of 21 datasets with relative gains up to 89.4%.The comparison also reports that L2D outperforms L2R on 20 of 21 datasets.
  • Results under different prompts: 70.11% average relative gain over the best baselines is achieved across 27 combinations of datasets, target models, and unseen-prompt types.The results support gains from both rewrite-based detection and adaptive distance learning.
  • Adversarial attacks: L2D remains robust under adversarial attacks: its AUC increases or stays unchanged on News and decreases only slightly on Story and Wiki.RAIDAR and ImBD experience larger declines in the reported attack settings.
  • Ablation study: 97.1% average improvement over the fixed-distance variant demonstrates the benefit of learning the distance metric rather than fixing it.The fixed-distance variant constructs the distance using the initial language model pϕ.

5 DISCUSSION

The paper frames adaptive distance learning as a theoretically grounded extension of rewrite-based detection, with empirical gains over fixed-distance and state-of-the-art detectors. Its main practical limitation is relatively high computational cost from generating multiple rewrites.

  • Discussion: The geometric analysis explains rewrite-based detection, its generalization to unseen prompts, and the rationale for adaptively learning the distance function.These claims correspond to Propositions 1–3.
  • Discussion: The method achieves state-of-the-art detection accuracy in most settings while remaining robust against adversarial attacks.The paper reports gains over fixed-distance approaches and state-of-the-art detectors.
  • Limitations: The method’s computational cost remains relatively high and comparable to existing rewrite-based algorithms because it generates multiple rewrites.Asynchronous rewriting and vLLM-based distance computation may improve practical efficiency.

ETHICS STATEMENT

The paper states that its research adheres to the ICLR Code of Ethics.

  • Ethics statement: The research adheres to the ICLR Code of Ethics in all respects.

REPRODUCIBILITY STATEMENT

The paper reports substantial efforts to support reproducibility through documented assumptions, proofs, implementation details, experimental procedures, and data-generation processes.

  • Reproducibility statement: Reproducibility materials document the method’s assumptions, theoretical proofs, implementation details, experimental setup, and data-generation procedures.The paper identifies Sections 2 and 4 and Appendices A, C, and D as sources for these details.

A PROOFS AND ADDITIONAL THEORETICAL RESULTS

Theoretical results explain why rewrite-based detection separates human and LLM-generated text and why adaptive distances improve detection. Under nonlinear-manifold assumptions, human reconstruction errors exceed LLM-generated-text errors with high probability, while the learned distance reaches the strongest attainable separation.

  • Proof mechanism: For texts rewritten into the LLM manifold, the reconstruction-error difference is bounded below by the distance from human text to that manifold minus ε0.The proof uses the projection operator and averaging over rewriting randomness.
  • Assumptions: Under realistic nonlinear-manifold assumptions, human and LLM-generated text occupy manifolds with intrinsic dimensions d_h > d_m, and rewriting has small expected error for LLM-generated text.The assumptions also require bounded human-text density and normalized maximum distance.
  • Proposition: The geometric proposition establishes that human reconstruction error is larger on average than LLM-generated-text reconstruction error.Reconstructions lie in the LLM manifold, while human text generally lies farther from it.
  • Adaptive distance: Under the learned optimal distance, reconstruction error attains its upper bound, supporting adaptive distance learning over a fixed distance.The proof establishes this by separating human and LLM-supported distributions.

B ADDITIONAL IMPLEMENTATION DETAILS AND NUMERICAL EXPERIMENTS

The additional experiments describe the four-step detector, evaluate it across diverse models and settings, and examine efficiency and temperature robustness. The method generally outperforms competing detectors while remaining close to RAIDAR in runtime and nearly preserving performance across temperature shifts.

  • Algorithm: The detector collects human and target-LLM text, generates K rewrites, learns a distance maximizing reconstruction-error differences, and classifies using a threshold.The learned distance is based on the discrepancy between normalized language-model log probabilities.
  • Generalization: The experiments span cross-model and cross-domain settings, training on some models or data categories and testing on different ones.Cross-fitting prevents evaluation data from entering training and changes content and domain between splits.
  • Efficiency: Runtime remains very close to RAIDAR while AUC improves substantially, and vLLM could further reduce computational cost.Table B6 reports both average AUC and runtime comparisons.
  • Robustness: Under temperature variation, performance is nearly identical to the matched-temperature setting, indicating robustness when training and test temperatures differ.The evaluation uses six temperatures and reverses the training/testing temperature splits.

C IMPLEMENTATION

The implementation uses repeated lightweight-LLM rewrites and a length-normalized, fine-tuned language-model distance. It selects K = 4 to balance detection accuracy against runtime and memory costs, and evaluates with AUC and relative gain over the strongest baseline.

  • Rewrite count: K = 4 balances accuracy and efficiency because larger K improves estimation but increases runtime and GPU memory, with diminishing gains.All experiments use this modest setting.
  • Distance implementation: The distance is the absolute difference between length-normalized log probabilities, implemented with LoRA fine-tuning.The LoRA settings are rank 8, alpha 32, and dropout 0.1.
  • Evaluation: AUC is the evaluation metric, and relative gain is normalized by the remaining gap between the strongest baseline AUC and 1.0.The stated formula is (Our AUC − StrongestBaseline’s AUC)/(1.0 − StrongestBaseline’s AUC).

D.1 EXPERIMENTAL SETUP ON DIVERSE DATASETS

The diverse-dataset experiments compare detectors under controlled cross-dataset training and testing, using multiple surrogate methods, target models, prompts, and generation conditions. Machine-generated data come from three LLMs across three tasks and three sources, producing 27 evaluation datasets.

  • Evaluation protocol: Training-based detectors train on 10 of 21 datasets and evaluate on the remaining datasets, repeating the process with swapped splits.This protocol avoids evaluation-data leakage and supports fair comparison.
  • Baselines: Zero-shot detectors use the same open-source surrogate LLMs to compute statistical measures, including Likelihood, IDE, and LRR.IDE uses maximum-likelihood intrinsic-dimension estimation here because persistence homology is computationally expensive.
  • Data generation: Machine-generated texts come from GPT-4o, Claude-3.5-Haiku, and Gemini-2.5-Flash, with generation temperature set to 0.8.The process samples human-written source texts and applies the specified task prompts.
  • Tasks and datasets: The evaluation covers rewrite, polish, and expansion tasks across three source datasets, yielding 27 datasets with 100 machine-generated and 100 human-written texts per dataset.Each source-task generation process is repeated 100 times.
  • Baseline setup: Baseline evaluation uses the other source datasets for training when one dataset is tested, preserving a cross-dataset comparison design.The same procedure is applied when News, Wiki, or Story is the evaluation dataset.

D.3 EXPERIMENTAL SETUP FOR ADVERSARIAL ATTACKS AND ABLATION

The experiments evaluate robustness against adversarial rephrasing attacks and isolate the contribution of adaptive distance learning by holding rewritten texts constant in the ablation.

  • Adversarial attacks: The robustness evaluation adopts adversarial attacks from Bao et al. (2024).
  • Adversarial attacks: For rephrasing attacks, a HuggingFace T5-based paraphraser paraphrases Claude-3.5-generated text before detection.
  • Ablation: The ablation uses identical rewritten texts for FD and the proposed method, isolating adaptive distance learning as the contribution.
Loading 2601.21895v2…