PaperlayerView exact revision on arXiv

Source-linked AI summary

Base Models Look Human To AI Detectors

Yixuan Even Xu, Ziqian Zhong, Aditi Raghunathan, Fei Fang, J. Zico Kolter

arXiv:2605.19516v1cs.CLcs.AIcs.LG

TL;DR

The paper asks whether commercial detectors reliably distinguish machine-generated text, given that base-model continuations may already appear human. It introduces HIP, which minimally fine-tunes a base model as a paraphraser and applies it iteratively; HIP improves the semantic-preservation and detector-evasion trade-off across the tested settings, while detector performance may change as commercial systems adapt.

  • Problem

    Commercial detectors are increasingly used to distinguish human-written from machine-generated text, but the paper examines whether their judgments reflect an invariant machine-text signal.

    Source: §1 · pp. 1–2

  • Method

    HIP minimally fine-tunes a base model on AI-to-human paraphrase pairs and repeatedly applies the resulting paraphraser to input text.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Base-model continuations are judged far more human than instruction-tuned continuations, and HIP yields a stronger semantic-preservation and detector-evasion trade-off than the tested baselines.

    Source: §1 · p. 2, §4.2 · p. 9

  • Takeaways & Limitations

    The findings suggest that current detectors track post-training effects and local context, motivating detector designs that model these factors explicitly.

    Source: §1 · p. 3, §5 · p. 9

    Commercial detectors can adapt to known attacks, so HIP’s detector-evasion effectiveness may weaken as those systems are updated.

    Source: §5 · p. 10

Abstract

from arXiv · show

As AI-generated text enters the real-world at scale, institutions increasingly use commercial AI-text detectors, especially in education and academic-integrity workflows. We report a surprising empirical finding about such systems: when evaluated by GPTZero and Pangram, generated text from base models is often judged overwhelmingly human, whereas text generated by their instruction-tuned counterparts is not. Building on this observation, we propose Humanization by Iterative Paraphrasing (HIP), a detector-agnostic pipeline that minimally fine-tunes a base model into a paraphraser and applies it iteratively. Compared with the baselines we test, HIP yields a stronger trade-off between semantic preservation and detector evasion on commercial detectors. Across Llama-3 and Qwen-3 families, spanning model sizes from 0.6B to 70B, HIP consistently improves detector human-likeness. Our findings suggest that current detectors are tracking artifacts of instruction tuning and local context more than any invariant notion of machine-generated text. This, in turn, calls for detector designs that model these factors more explicitly.

1 Introduction

The paper shows that commercial detectors often judge base-model continuations as substantially more human than instruction-tuned continuations, motivating HIP, an iterative paraphrasing pipeline designed to improve detector evasion while preserving semantics.

  • Commercial detectors are deployed for practical tasks including assignment screening and authorship review as AI-generated text becomes commonplace.Source: §1 · p. 1
  • Base-model continuations receive substantially higher human-likeness judgments than instruction-tuned continuations under the same prefix conditions.For Llama-3-8B with human prefixes, GPTZero and Pangram assign 96.7% and 98.8% human probability to base continuations, versus 30.3% and 17.1% for instruct continuations.Source: Figure 1 · p. 1, §1 · p. 2
  • Human prefixes also make continuations look slightly more human than AI prefixes, suggesting roles for low distortion and human context.Source: §1 · p. 2
  • HIP minimally fine-tunes a base model into a paraphraser and applies it iteratively to move text toward human-like detector judgments.Source: §1 · p. 2, Figure 2 · p. 2
  • Across Llama and Qwen models of multiple sizes, HIP achieves a stronger semantic-retention and detector-evasion trade-off than the tested baselines.The paper evaluates this trade-off on commercial detectors rather than primarily on open-source detectors.Source: §1 · p. 2
  • The paper argues that detectors should model base-model behavior, post-training distortions, and local context more explicitly.Source: §1 · p. 3

2 Related Work

Related work covers AI-text detection, post-training behavior shifts, adversarial paraphrasing, and contextual or iterative refinement as foundations for studying detector evasion.

  • AI-text detection includes zero-shot statistical methods, supervised classifiers, and commercial systems such as Pangram and GPTZero.Source: §2 · p. 3

Behavior shift during post-training

  • Instruction tuning and RLHF can leave statistical fingerprints, including shifts in response length, sycophancy, and formatting preferences.Source: §2 · p. 3

Adversarial paraphrasing and detector evasion

  • Prior detector-evasion research studies paraphrasing, humanized AI text, temperature-guided rewriting, supervised rewriting, and homoglyph substitution.Source: §2 · p. 3

Contextual influence and iterative refinement

  • Research on contextual influence and iterative refinement examines how generation context and repeated rewriting affect detector evasion and output quality.Source: §2 · p. 3

3 Methodology

HIP operationalizes low distortion and human context through paired data preparation, minimal fine-tuning of a base model, and repeated paraphrasing of the resulting text.

  • HIP uses a detector-agnostic pipeline with three stages: data preparation, minimal fine-tuning, and iterative paraphrasing.Source: §3 · p. 4

3.1 Data Preparation

  • The data stage pairs each high-quality human passage h_i with an AI paraphrase a_i, training the model to map AI text back to human text.Source: §3.1 · p. 4
  • The raw corpus is filtered, normalized, deduplicated, quality-screened, and then converted into pairs using anomaly and semantic-preservation checks with bounded retries.Source: §3.1 · p. 4

3.2 Minimal Fine-Tuning

  • Minimal fine-tuning adapts a pretrained base model into a paraphraser while perturbing it as little as possible to preserve low distortion.Source: §3.2 · p. 4
  • The supervision format treats paraphrasing as plain-text continuation, with the AI paraphrase as input and the original human passage as the completion.Source: §3.2 · p. 4

3.3 Iterative Paraphrasing

  • The trained paraphraser repeatedly rewrites x^(t-1) into x^t, progressively building human context while trading semantic retention for humanization.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate HIP across datasets, model families, model sizes, and detector-evasion baselines using semantic preservation and GPTZero/Pangram human-likeness. HIP consistently increases detector-assigned human probability as paraphrasing rounds accumulate, while semantic fidelity declines, and achieves the strongest tested trade-off.

Experimental Setup

  • HIP is evaluated on 256 AI-generated passages from eight RAID and MAGE source categories, using semantic preservation plus GPTZero and Pangram human-likeness.Semantic preservation is scored from 0 to 10; higher detector probabilities indicate more human-like judgments.Source: §4.1 · p. 5

HIP Across Models

  • Across Qwen3 and Llama3 base and instruction-tuned families, detector-assigned human probability rises over rounds while semantic fidelity gradually declines.This pattern holds across model families, sizes, and post-training states.Source: §4.2 · pp. 8–9
  • Low model sizes affect the trade-off most: Qwen3 improves materially through 4B, while larger Qwen3 shifts modestly and Llama3-70B is only slightly more stable than Llama3-8B.The reported interpretation is that HIP mainly requires a model large enough to paraphrase competently.Source: §4.2 · p. 9

Qualitative Examples

  • By round 10 in a Llama3-8B example, semantic fidelity remains high while both GPTZero and Pangram score the output as human.Pangram flips to human by round 2, while GPTZero becomes more ambiguous at that stage.Source: §4.2 · p. 9

Baseline Comparison

  • HIP on Llama3-8B substantially outperforms all tested baselines, achieving the strongest semantic-evasion frontier under both commercial detectors.DIPPER retains a non-trivial trade-off, while SilverSpeak affects Pangram slightly and Simple Paraphrase and StealthRL do not achieve a meaningful trade-off.Source: §4.2 · p. 9

5 Discussion

The discussion attributes detector-judged human-likeness to post-training effects and generation context, and notes that detector-evasion performance may weaken as commercial systems adapt. These findings motivate detector designs that address post-training distortions and base-model text.

Why HIP Works

  • HIP appears to recover part of the human-token behavior associated with base-model pretraining through a paraphrase objective and iterative removal of AI-generated context.The paired rewrites target human-written passages, while iteration progressively removes AI-generated context.Source: §5 · p. 9

Implications for Detectors

  • Current detectors appear sensitive to statistical effects of post-training and generation context rather than detecting AI text in general.The discussion frames this as a motivation for detector-design research questions.Source: §5 · p. 9
  • The paper asks whether post-training leaves detector-visible fingerprints and whether detectors can reliably identify true base-model text.These are presented as open research questions rather than resolved conclusions.Source: §5 · p. 9

Limitation and Broader Impact

  • Commercial detectors may adapt to known attacks, so HIP’s detector-evasion results may weaken as those systems are updated.The paper presents the broader impact as mixed because HIP could be misused while also exposing a concrete weakness in current systems.Source: §5 · p. 10

A.1 Continuation Evaluation

The continuation evaluation shows that instruction tuning reduces detector-assigned human-likeness, while HIP and further language-model fine-tuning recover much of that loss.

Continuation Evaluation

  • Instruct + HIP is evaluated here as a generator rather than as a paraphraser.The experiment uses the adapted checkpoint to generate continuations exactly like the raw base and instruct models.Source: §A.1 · p. 13
  • Instruct + FT uses a plain language-modeling objective on FineWeb-Edu rather than the HIP paraphrase objective.It is a separate control based on the same instruction-tuned model.Source: §A.1 · p. 13
  • Instruct + HIP and Instruct + FT recover a large portion of the human-likeness lost after instruction tuning.This pattern holds for both Llama3 and Qwen3 under human and AI prefixes.Source: §A.1 · p. 13
  • GPTZero and Pangram judge base-model continuations much more human than instruct-model continuations under the same prefix conditions.The comparison uses human-written and AI-generated prefixes and evaluates only generated continuations.Source: Figure 6 · p. 13, §A.1 · p. 13

A.2 HIP on OpenAI Models via the Fine-Tuning API

The OpenAI fine-tuning API preserves semantic quality during HIP but does not produce a useful detector-evasion trade-off. This contrasts with the open-weight Llama and Qwen results and leaves the underlying mechanism unverified.

HIP on OpenAI Models via the Fine-Tuning API

  • HIP does not produce a useful semantic-evasion trade-off for GPT-4.1-nano fine-tuned through the OpenAI API.The evaluation uses the same paired paraphrase data, 10 rounds, 256 examples, and semantic, GPTZero, and Pangram metrics.Source: §A.2 · p. 14
  • Semantic scores remain high throughout 10 rounds, while GPTZero and Pangram human probabilities quickly fall and remain low.The model paraphrases coherently but does not reach the human-like region observed for open-weight Llama and Qwen models.Source: §A.2 · p. 14
  • The negative result is consistent with the hypothesis that API fine-tuning does not induce the same low-distortion base-model behavior exploited by HIP on open-weight models.The passage presents this as a plausible hypothesis rather than a verified mechanism.Source: §A.2 · p. 14
  • The proposed explanation involving additional post-training or alignment procedures during platform training or serving cannot be directly verified.The negative result is only consistent with that interpretation.Source: §A.2 · p. 14

A.3 HIP on Instruct Models with Native Chat Templates

Using native chat templates instead of plain source-target formatting has only a modest effect on HIP for Llama3-8B-Instruct and Qwen3-8B-Instruct. The overall trade-off and qualitative behavior remain similar.

HIP on Instruct Models with Native Chat Templates

  • The native-chat-template variant produces a Pareto frontier close to standard HIP in both model families.The comparison evaluates semantic preservation and GPTZero and Pangram human-likeness over 10 rounds.Source: §A.3 · p. 15, Figure 8 · p. 15
  • The formatting change has only a modest effect on the semantic-preservation and detector-human-likeness trade-off.Figure 8 reports GPTZero and Pangram frontiers alongside scores over the rounds.Source: §A.3 · p. 15
  • The main HIP effect does not depend strongly on whether instruct models use plain source-target formatting or native chat templates.The qualitative behavior remains unchanged across the two formulations.Source: §A.3 · p. 15
  • The non-chat formulation is not necessary for instruct models.Source: §A.3 · p. 15

A.4 HIP with Output-Layer-Only Adaptation

Output-layer-only adaptation fails to reproduce standard HIP’s semantic-preservation and detector-evasion trade-off. The results suggest that successful humanization requires changes deeper than the final logit layer alone.

HIP with Output-Layer-Only Adaptation

  • Output-layer-only adaptation does not reproduce the main HIP trade-off for Llama3-8B or Qwen3-8B.The ablation freezes the model except for lm_head and uses the same paired data and 10-round evaluation protocol.Source: §A.4 · p. 16
  • For Llama3-8B, detector curves improve only alongside a much sharper semantic-preservation drop than under standard HIP.Source: §A.4 · p. 16
  • For Qwen3-8B, semantic scores remain relatively high, but GPTZero and Pangram human probabilities stay far below the HIP curve.Source: §A.4 · p. 16
  • Successful humanization appears to require changes deeper in the representation pathway than the final logit layer alone.This conclusion is limited to the models studied.Source: §A.4 · p. 16

Related papers

Loading 2605.19516v1…