PaperlayerView exact revision on arXiv

Source-linked AI summary

Base Models Look Human To AI Detectors

Yixuan Even Xu, Ziqian Zhong, Aditi Raghunathan, Fei Fang, J. Zico Kolter

arXiv:2605.19516v1cs.CLcs.AIcs.LG

TL;DR

The paper asks whether commercial detectors reliably distinguish machine-generated text, given that base-model continuations may already appear human. It introduces HIP, which minimally fine-tunes a base model as a paraphraser and applies it iteratively; HIP improves the semantic-preservation and detector-evasion trade-off across the tested settings, while detector performance may change as commercial systems adapt.

  • Problem

    Commercial detectors are increasingly used to distinguish human-written from machine-generated text, but the paper examines whether their judgments reflect an invariant machine-text signal.

  • Method

    HIP minimally fine-tunes a base model on AI-to-human paraphrase pairs and repeatedly applies the resulting paraphraser to input text.

  • Results

    Base-model continuations are judged far more human than instruction-tuned continuations, and HIP yields a stronger semantic-preservation and detector-evasion trade-off than the tested baselines.

  • Takeaways & Limitations

    The findings suggest that current detectors track post-training effects and local context, motivating detector designs that model these factors explicitly.

  • Takeaways & Limitations

    Commercial detectors can adapt to known attacks, so HIP’s detector-evasion effectiveness may weaken as those systems are updated.

Abstract

from arXiv · show

As AI-generated text enters the real-world at scale, institutions increasingly use commercial AI-text detectors, especially in education and academic-integrity workflows. We report a surprising empirical finding about such systems: when evaluated by GPTZero and Pangram, generated text from base models is often judged overwhelmingly human, whereas text generated by their instruction-tuned counterparts is not. Building on this observation, we propose Humanization by Iterative Paraphrasing (HIP), a detector-agnostic pipeline that minimally fine-tunes a base model into a paraphraser and applies it iteratively. Compared with the baselines we test, HIP yields a stronger trade-off between semantic preservation and detector evasion on commercial detectors. Across Llama-3 and Qwen-3 families, spanning model sizes from 0.6B to 70B, HIP consistently improves detector human-likeness. Our findings suggest that current detectors are tracking artifacts of instruction tuning and local context more than any invariant notion of machine-generated text. This, in turn, calls for detector designs that model these factors more explicitly.

1 Introduction

The paper shows that commercial detectors often judge base-model continuations as substantially more human than instruction-tuned continuations, motivating HIP, an iterative paraphrasing pipeline designed to improve detector evasion while preserving semantics.

  • Commercial detectors are deployed for practical tasks including assignment screening and authorship review as AI-generated text becomes commonplace.
  • Base-model continuations receive substantially higher human-likeness judgments than instruction-tuned continuations under the same prefix conditions.For Llama-3-8B with human prefixes, GPTZero and Pangram assign 96.7% and 98.8% human probability to base continuations, versus 30.3% and 17.1% for instruct continuations.
  • Human prefixes also make continuations look slightly more human than AI prefixes, suggesting roles for low distortion and human context.
  • HIP minimally fine-tunes a base model into a paraphraser and applies it iteratively to move text toward human-like detector judgments.
  • Across Llama and Qwen models of multiple sizes, HIP achieves a stronger semantic-retention and detector-evasion trade-off than the tested baselines.The paper evaluates this trade-off on commercial detectors rather than primarily on open-source detectors.
  • The paper argues that detectors should model base-model behavior, post-training distortions, and local context more explicitly.

2 Related Work

Related work covers AI-text detection, post-training behavior shifts, adversarial paraphrasing, and contextual or iterative refinement as foundations for studying detector evasion.

  • AI-text detection includes zero-shot statistical methods, supervised classifiers, and commercial systems such as Pangram and GPTZero.
  • Behavior shift during post-training: Instruction tuning and RLHF can leave statistical fingerprints, including shifts in response length, sycophancy, and formatting preferences.
  • Adversarial paraphrasing and detector evasion: Prior detector-evasion research studies paraphrasing, humanized AI text, temperature-guided rewriting, supervised rewriting, and homoglyph substitution.
  • Contextual influence and iterative refinement: Research on contextual influence and iterative refinement examines how generation context and repeated rewriting affect detector evasion and output quality.

3 Methodology

HIP operationalizes low distortion and human context through paired data preparation, minimal fine-tuning of a base model, and repeated paraphrasing of the resulting text.

  • HIP uses a detector-agnostic pipeline with three stages: data preparation, minimal fine-tuning, and iterative paraphrasing.
  • 3.1 Data Preparation: The data stage pairs each high-quality human passage h_i with an AI paraphrase a_i, training the model to map AI text back to human text.
  • 3.1 Data Preparation: The raw corpus is filtered, normalized, deduplicated, quality-screened, and then converted into pairs using anomaly and semantic-preservation checks with bounded retries.
  • 3.2 Minimal Fine-Tuning: Minimal fine-tuning adapts a pretrained base model into a paraphraser while perturbing it as little as possible to preserve low distortion.
  • 3.2 Minimal Fine-Tuning: The supervision format treats paraphrasing as plain-text continuation, with the AI paraphrase as input and the original human passage as the completion.
  • 3.3 Iterative Paraphrasing: The trained paraphraser repeatedly rewrites x^(t-1) into x^t, progressively building human context while trading semantic retention for humanization.

4 Experiments

The experiments evaluate HIP across datasets, model families, model sizes, and detector-evasion baselines using semantic preservation and GPTZero/Pangram human-likeness. HIP consistently increases detector-assigned human probability as paraphrasing rounds accumulate, while semantic fidelity declines, and achieves the strongest tested trade-off.

  • Experimental Setup: HIP is evaluated on 256 AI-generated passages from eight RAID and MAGE source categories, using semantic preservation plus GPTZero and Pangram human-likeness.Semantic preservation is scored from 0 to 10; higher detector probabilities indicate more human-like judgments.
  • HIP Across Models: Across Qwen3 and Llama3 base and instruction-tuned families, detector-assigned human probability rises over rounds while semantic fidelity gradually declines.This pattern holds across model families, sizes, and post-training states.
  • HIP Across Models: Low model sizes affect the trade-off most: Qwen3 improves materially through 4B, while larger Qwen3 shifts modestly and Llama3-70B is only slightly more stable than Llama3-8B.The reported interpretation is that HIP mainly requires a model large enough to paraphrase competently.
  • Qualitative Examples: By round 10 in a Llama3-8B example, semantic fidelity remains high while both GPTZero and Pangram score the output as human.Pangram flips to human by round 2, while GPTZero becomes more ambiguous at that stage.
  • Baseline Comparison: HIP on Llama3-8B substantially outperforms all tested baselines, achieving the strongest semantic-evasion frontier under both commercial detectors.DIPPER retains a non-trivial trade-off, while SilverSpeak affects Pangram slightly and Simple Paraphrase and StealthRL do not achieve a meaningful trade-off.

5 Discussion

The discussion attributes detector-judged human-likeness to post-training effects and generation context, and notes that detector-evasion performance may weaken as commercial systems adapt. These findings motivate detector designs that address post-training distortions and base-model text.

  • Why HIP Works: HIP appears to recover part of the human-token behavior associated with base-model pretraining through a paraphrase objective and iterative removal of AI-generated context.The paired rewrites target human-written passages, while iteration progressively removes AI-generated context.
  • Implications for Detectors: Current detectors appear sensitive to statistical effects of post-training and generation context rather than detecting AI text in general.The discussion frames this as a motivation for detector-design research questions.
  • Implications for Detectors: The paper asks whether post-training leaves detector-visible fingerprints and whether detectors can reliably identify true base-model text.These are presented as open research questions rather than resolved conclusions.
  • Limitation and Broader Impact: Commercial detectors may adapt to known attacks, so HIP’s detector-evasion results may weaken as those systems are updated.The paper presents the broader impact as mixed because HIP could be misused while also exposing a concrete weakness in current systems.

A.1 Continuation Evaluation

The continuation evaluation shows that instruction tuning reduces detector-assigned human-likeness, while HIP and further language-model fine-tuning recover much of that loss.

  • Continuation Evaluation: Instruct + HIP is evaluated here as a generator rather than as a paraphraser.The experiment uses the adapted checkpoint to generate continuations exactly like the raw base and instruct models.
  • Continuation Evaluation: Instruct + FT uses a plain language-modeling objective on FineWeb-Edu rather than the HIP paraphrase objective.It is a separate control based on the same instruction-tuned model.
  • Continuation Evaluation: Instruct + HIP and Instruct + FT recover a large portion of the human-likeness lost after instruction tuning.This pattern holds for both Llama3 and Qwen3 under human and AI prefixes.
  • Continuation Evaluation: GPTZero and Pangram judge base-model continuations much more human than instruct-model continuations under the same prefix conditions.The comparison uses human-written and AI-generated prefixes and evaluates only generated continuations.

A.2 HIP on OpenAI Models via the Fine-Tuning API

The OpenAI fine-tuning API preserves semantic quality during HIP but does not produce a useful detector-evasion trade-off. This contrasts with the open-weight Llama and Qwen results and leaves the underlying mechanism unverified.

  • HIP on OpenAI Models via the Fine-Tuning API: HIP does not produce a useful semantic-evasion trade-off for GPT-4.1-nano fine-tuned through the OpenAI API.The evaluation uses the same paired paraphrase data, 10 rounds, 256 examples, and semantic, GPTZero, and Pangram metrics.
  • HIP on OpenAI Models via the Fine-Tuning API: Semantic scores remain high throughout 10 rounds, while GPTZero and Pangram human probabilities quickly fall and remain low.The model paraphrases coherently but does not reach the human-like region observed for open-weight Llama and Qwen models.
  • HIP on OpenAI Models via the Fine-Tuning API: The negative result is consistent with the hypothesis that API fine-tuning does not induce the same low-distortion base-model behavior exploited by HIP on open-weight models.The passage presents this as a plausible hypothesis rather than a verified mechanism.
  • HIP on OpenAI Models via the Fine-Tuning API: The proposed explanation involving additional post-training or alignment procedures during platform training or serving cannot be directly verified.The negative result is only consistent with that interpretation.

A.3 HIP on Instruct Models with Native Chat Templates

Using native chat templates instead of plain source-target formatting has only a modest effect on HIP for Llama3-8B-Instruct and Qwen3-8B-Instruct. The overall trade-off and qualitative behavior remain similar.

  • HIP on Instruct Models with Native Chat Templates: The native-chat-template variant produces a Pareto frontier close to standard HIP in both model families.The comparison evaluates semantic preservation and GPTZero and Pangram human-likeness over 10 rounds.
  • HIP on Instruct Models with Native Chat Templates: The formatting change has only a modest effect on the semantic-preservation and detector-human-likeness trade-off.Figure 8 reports GPTZero and Pangram frontiers alongside scores over the rounds.
  • HIP on Instruct Models with Native Chat Templates: The main HIP effect does not depend strongly on whether instruct models use plain source-target formatting or native chat templates.The qualitative behavior remains unchanged across the two formulations.
  • HIP on Instruct Models with Native Chat Templates: The non-chat formulation is not necessary for instruct models.

A.4 HIP with Output-Layer-Only Adaptation

Output-layer-only adaptation fails to reproduce standard HIP’s semantic-preservation and detector-evasion trade-off. The results suggest that successful humanization requires changes deeper than the final logit layer alone.

  • HIP with Output-Layer-Only Adaptation: Output-layer-only adaptation does not reproduce the main HIP trade-off for Llama3-8B or Qwen3-8B.The ablation freezes the model except for lm_head and uses the same paired data and 10-round evaluation protocol.
  • HIP with Output-Layer-Only Adaptation: For Llama3-8B, detector curves improve only alongside a much sharper semantic-preservation drop than under standard HIP.
  • HIP with Output-Layer-Only Adaptation: For Qwen3-8B, semantic scores remain relatively high, but GPTZero and Pangram human probabilities stay far below the HIP curve.
  • HIP with Output-Layer-Only Adaptation: Successful humanization appears to require changes deeper in the representation pathway than the final logit layer alone.This conclusion is limited to the models studied.
Loading 2605.19516v1…