PaperlayerView exact revision on arXiv

Source-linked AI summary

A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID

Paweł Blicharz, Miłosz Grunwald

arXiv:2609.30287v1cs.CLcs.AI

TL;DR

AI-text detectors can perform well, but which internal representations drive their predictions remains poorly understood. This study uses sparse probing and causal interventions on frozen BERT, finding that small stable neuron sets support detection, are easy to inject but hard to erase, and retain signal on unseen generator families.

  • Problem

    Although AI-text detection accuracy has improved, which internal representations drive predictions remains an open question without a causal explanation of how detection works.

    Source: §1 · p. 1

  • Method

    The study applies L1-to-L2 sparse probing to frozen BERT CLS activations and tests selected neurons with mean ablation and bidirectional activation patching.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    Forward patching flips predictions 10–16× above random and reverse patching 7–12× above random; mean ablation costs ≤0.11 pp on five of six generators and 1.06 pp on Cohere.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The detection signal is redundantly distributed, while selected neurons retain 86–94% of the full-feature ceiling on held-out generator families.

    Source: §8 · p. 9

    Evaluation is exclusively on the English-language RAID benchmark.

    Source: Limitations · p. 9

Abstract

from arXiv · show

AI-generated text detectors achieve high accuracy on standard benchmarks, yet the internal representations that drive these predictions remain poorly understood. We study which neurons in a frozen BERT-base-uncased encoder support AI-text detection, using the RAID benchmark across six generators spanning pure-base and instruction-tuned models. We apply the L1-to-L2 sparse-probing protocol of Gurnee et al. (2023) to all 9,216 CLS hidden-state dimensions (12 layers x 768), which we call neurons. The procedure recovers a stable set of under 1% of neurons per generator, consistent across folds and seeds; a probe restricted to that set retains most of the full-feature detection accuracy. Bidirectional activation patching confirms this set's causal relevance: in both directions it flips predictions an order of magnitude more often than size-matched random sets. Mean-ablating the same neurons leaves accuracy largely intact; the signal is therefore redundantly distributed. Cross-generator analysis reveals a bipartite structure: instruction-tuned generators concentrate 30-36% of stable neurons in BERT's final layer while both base generators fall below 14%, consistent with a layer-12 footprint of post-training alignment. Leave-one-family-out evaluation shows the selected neurons retain 86-94% of the full-feature ceiling on unseen generator families, so a detector can operate on a small fixed subspace without re-identifying neurons per generator.

1 Introduction

The paper asks which internal representations drive AI-text detection and tests whether stable neurons in frozen BERT causally support predictions. Across generators, sparse neuron sets support detection, are easy to inject but hard to erase, and retain signal on unseen families.

1 Introduction

  • The paper addresses the open question of which internal representations drive detection, a gap left by interpretability work that identifies correlations without testing causal sufficiency.The authors frame this as lacking a causal explanation of how detection works.Source: §1 · p. 1
  • Forward patching flips predictions 10–16× above random and reverse patching 7–12× above random, while mean ablation leaves accuracy largely intact.The authors describe the signal as redundantly distributed: easy to inject, hard to erase.Source: §1 · pp. 2–3
  • Stable sets of fewer than 1% of CLS dimensions support near-ceiling detection accuracy with high cross-fold stability.The study applies sparse probing to frozen BERT across six RAID generators.Source: §1 · pp. 1–2
  • Stable neuron sets overlap ∼24× above chance, with a layer-12 footprint concentrated in instruction-tuned generators and absent in both pure-base models.The authors report this as a cross-generator structure.Source: §1 · p. 3
  • Leave-one-family-out evaluation retains 86–94% of the full-feature detection signal on entirely unseen generator families.This evaluates whether selected neurons retain signal without re-identifying them for the held-out family.Source: §1 · pp. 2–3

2 Related Work

Prior detection work spans supervised and zero-shot methods, while interpretability studies often characterize predictive cues or suppress neurons associated with confounders. This paper combines sparse probing with causal intervention to test whether selected representations drive predictions.

2.1 AI-Generated Text Detection

  • RAID benchmarking finds no single detection method generalizes reliably across generators or domains, motivating a focus on internal representations rather than detection accuracy.Supervised classifiers can degrade on unseen generators or domains; zero-shot methods may require reference-model access.Source: §2.1 · p. 3

2.2 Interpretability of Text Detectors

  • Prior interpretability work examines token attributions, confounders, geometric subspaces, or SAE features, but does not directly test whether identified neurons are sufficient to drive predictions.Borile and Abrate (2025) suppress neurons associated with artefacts in a fine-tuned encoder; the present study instead uses a frozen encoder and activation patching.Source: §2.2 · p. 3

2.3 Sparse Probing and Activation Patching

  • Sparse probing uses L1 selection followed by L2 evaluation, while activation patching replaces target representations with donor activations to measure output effects.Heimersheim and Nanda (2024) recommend real-input donor patching and mean ablation to avoid out-of-distribution artefacts; the paper follows both.Source: §2.3 · pp. 3–4

3 Experimental Setup

The experiments use selected RAID domains and generators to compare base and instruction-tuned models. Frozen BERT CLS activations feed an L1 selection and L2 evaluation protocol, with stable neurons aggregated across folds and seeds.

3.1 Data: RAID

  • The study selects six RAID generators spanning five model families and all benchmark post-training regimes, using six domains and excluding poetry and recipes for low sample counts.The six include two pure-base, one SFT-only, and three SFT+RLHF generators; all 11 RAID generators enter the leave-one-family-out evaluation.Source: §3.1 · p. 4

3.2 Model and Activations

  • Frozen BERT-base-uncased supplies CLS hidden states from all 12 layers, concatenated into a 9,216-dimensional representation.Each layer contributes a 768-dimensional vector; the paper calls each coordinate a neuron.Source: §3.2 · p. 4

3.3 Protocol: L1 Selection, L2 Evaluation

  • An L1 probe selects per-cell candidates and aggregates neurons appearing in at least 80% of 15 cells; a frozen L2 probe evaluates interventions.The 15 cells come from five-fold cross-validation with three seeds; interventions target per-fold selections, while the stable set is reserved for cross-generator analyses and LOGO.Source: §3.3 · p. 4
  • The chosen C = 0.005 and N = 7,500 operating point achieved minimum pairwise Jaccard 0.676, with other generators at 0.707–0.751.The settings were selected in a prior stability sweep, maximizing minimum-over-generators Jaccard subject to at least 50 neurons per draw.Source: §3.3 · p. 5

4 Sparse Detection Neurons in Frozen BERT

Sparse probing recovers small, stable neuron sets across six generators, and probes restricted to those sets retain most detection accuracy. Selection is highly consistent across folds relative to chance.

4 Sparse Detection Neurons in Frozen BERT

  • 45–62 stable neurons (0.49–0.67% of 9,216 dimensions) accompany L2 evaluation accuracy of 90.5–98.8% and AUC-ROC of 0.968–0.999 across generators.A probe restricted to the stable set achieves 86.5–97.2% accuracy, retaining most of the detection accuracy.Source: §4 · p. 5
  • Pairwise fold Jaccard is 0.67–0.76 across 105 fold-pairs per generator, or 150–230× above the random-null expectation of ∼0.003–0.005.The sets are drawn from 9,216 neurons, and the selection pool ranges from 91 neurons ever selected for LLaMA to 146 for GPT-2.Source: §4 · p. 5
  • The stable neurons are sufficient for most detection accuracy, while the probe consistently selects from a narrow region of the feature space.The passage reports layer-by-layer distributions and base/instruction-tuned asymmetry as examined in Section 6.Source: §4 · p. 5

5 Causal Role of the Selected Neurons

Bidirectional patching makes the selected neurons flip predictions far more often than size-matched random sets, while mean ablation usually leaves accuracy intact. The pattern supports a redundantly distributed detection signal, with Cohere showing a partial-localisation exception.

5.1.1 Patching Protocol

  • Patching replaces selected-neuron activations with same-domain donor activations and measures flips in a frozen probe; the random-k baseline controls for patched-set size.Forward patching tests sufficiency, while reverse patching tests partial necessity; k = full uses the per-fold L1-selected set of 53–92 neurons.Source: §5.1.1 · pp. 5–6

5.1.2 Flip-Rate Results (Forward and Reverse)

  • Forward flip rates at k = full are 7–18× those at k = 5, consistent with signal distributed across the selected set; flip rates also vary by generator and domain.Cohere flips most readily (8.15%, baseline 90.5%) and LLaMA least (1.07%, baseline 98.6%); Cohere is uniformly high across domains, while GPT-2 is elevated on news, Reddit, and Wikipedia.Source: §5.1.2 · p. 6
  • Selected-neuron patching flips predictions 10–16× more often forward and 7–12× more often reverse than size-matched random-k baselines; at k = full, mean ablation drops accuracy only 0.03–0.11 pp for five generators.Cohere was the exception, with a 1.06 ± 0.52 pp full-set ablation drop, versus a 0.06 pp random baseline.Source: §5.1.2 · p. 6, §5.2 · p. 7

5.2 Necessity via Mean Ablation

  • For five generators, mean-ablation accuracy drops are within noise of random ablation, but Cohere’s 1.06 ± 0.52 pp drop is substantially above its 0.06 pp random baseline.The authors treat mean ablation as a conservative lower bound because it moves activations off-manifold; reverse patching provides an on-manifold test of partial necessity.Source: §5.2 · p. 7, Table 3 · p. 7

6 Cross-Generator Representation Geometry

Stable-neuron geometry separates instruction-tuned from pure-base generators in layer distribution and cross-generator overlap. A 17-neuron shared core is concentrated in layer 12, although surface-text associations do not reduce the selected subspace to one lexical cue.

6.1 Layer Distribution of Stable Neurons

  • Instruction-tuned generators place 30–36% of stable neurons in layer 12, versus 3.8–13.3% for the two pure-base generators.GPT-2 peaks in layer 11, while MPT peaks earlier; the authors frame layer 12 as a task-linked readout more active for instruction-tuned outputs, as a working hypothesis based on two base generators.Source: §6.1 · p. 7, Figure 3 · p. 8

6.2 Stable-Set Overlap Across Generators

  • Stable sets are generator-specific but overlap about 24× above chance on average, with near-zero overlap between base and instruction-tuned pairs.The mean observed Jaccard is 0.073 versus an approximately 0.003 random-set expectation; instruction-tuned pairs range 0.08–0.24, while every base-versus-instruction-tuned pair is at most 0.05.Source: §6.2 · pp. 7–8, Figure 3 · p. 8

6.3 A Cross-Generator Core Concentrated in Layer 12

  • A core of 17 neurons appears in at least three generators, and 10 of those neurons (59%) are in layer 12.The strongest associations with four surface-text statistics are moderate (|ρ| = 0.41–0.54) and vary in sign across generators, so the selected subspace does not reduce to a single lexical cue.Source: §6.3 · p. 8

7 Leave-One-Family-Out Generalisation

The leave-one-family-out evaluation tests whether neurons selected from other RAID families transfer to an unseen family. A sparse probe using at most 1.4% of the representation retains 86–94% of the full-feature signal, with performance varying across held-out families.

7 Leave-One-Family-Out Generalisation

  • For each held-out family, the sparse probe uses stable neurons selected from the other families, while the full-feature probe uses all 9,216 neurons.LOGO uses C = 0.001 and caps each generator’s training contribution at 5,000 samples; family sizes and train/test pool sizes vary.Source: §7 · p. 8
  • 91–125 selected neurons achieve 76.3–93.7% accuracy on held-out families and retain 86–94% of each family’s full-feature ceiling.Full L2 accuracy is 88.6–99.7%; the sparse bar percentages show retention relative to each family’s own full-feature ceiling.Source: §7 · p. 8, Figure 4 · p. 9
  • Cohere is the weakest held-out family at 76.3% accuracy and 86% of ceiling, while LLaMA is strongest at 93.7% and 94%.The authors interpret the 6–14 pp gap from Full L2 as reflecting family-specific neurons unavailable to the LOGO selection procedure, alongside a cross-family core.Source: §7 · pp. 8–9
  • The selected subspace can generalise to unseen generator families without re-running neuron selection for each new generator.The paper presents this as a property of the representation, not as a detection system.Source: §7 · p. 9

8 Conclusion

Sparse probing identifies small stable neuron sets, and activation patching supports their causal relevance despite substantial redundancy. These sets also retain most of the full-feature signal on unseen generator families.

8 Conclusion

  • 45–62 stable neurons, under 1% of 9,216 CLS dimensions, are selected across six RAID generators.The restricted probe retains most full-feature detection accuracy.Source: §8 · p. 9
  • Forward patching flips predictions 10–16× above random and reverse patching 7–12× above random, while mean ablation costs at most 0.11 pp on five generators and 1.06 pp on Cohere.The contrast indicates the selected neurons are easy to inject but hard to erase, consistent with a redundantly distributed signal.Source: §8 · p. 9
  • On held-out generator families, selected neurons retain 86–94% of the full-feature ceiling, as reported by the Sel/Full comparison in Table 4.The conclusion attributes the residual gap to family-specific neurons unavailable to LOGO selection.Source: §8 · p. 9, Table 4 · p. 9
  • Instruction-tuned generators concentrate stable neurons in BERT’s final layer, unlike the two pure-base generators.The sets are generator-specific despite their cross-generator structure.Source: §8 · p. 9

Limitations

The findings are bounded by the frozen BERT-base-uncased, English-only RAID setting and by the study’s non-adversarial evaluation. Some interpretation of layer distribution and mean ablation also requires caution.

Limitations

  • All experiments use frozen BERT-base-uncased; extension to larger encoders or encoder-decoder architectures remains open.The frozen setting isolates pretrained representations but does not permit direct comparison with fine-tuned detectors without controlling for that distinction.Source: Limitations · p. 9
  • English-only RAID evaluation leaves transfer to other languages, including richer-morphology, different-syntax, or non-Latin-script languages, untested.The layer-12 footprint, base/instruction-tuned structure, and redundancy patterns require separate evaluation across languages.Source: Limitations · pp. 9–10
  • Mean ablation replaces activations with training means and can move them off the natural data manifold, so its necessity result should be read alongside reverse patching.The authors treat the combined evidence as a conservative lower bound on necessity.Source: Limitations · p. 10
  • The lower layer-12 mass in base models is supported by only GPT-2 and MPT-30B, whose peak layers differ.The authors state L12 avoidance rather than a consistent lower-layer peak rule.Source: Limitations · p. 10
  • The analysis excludes RAID’s adversarial rephrasing subset, leaving the stable neurons’ causal role under those attacks unevaluated.The excluded attacks include paraphrase, homoglyph, and synonym-substitution attacks.Source: Limitations · p. 10

Ethical considerations

The paper frames AI-text detection as dual-use research and documents its data, frozen-model pipeline, selection stability, and per-cell reporting. Its operating-point analysis favors a setting that avoids degenerate empty selections while maintaining cross-generator stability.

Ethical considerations

  • AI-text detection may support platform integrity and verification but also penalise legitimate AI-assisted writing or guide evasion; the study uses public RAID data, involves no human subjects, and releases its code.No personally identifiable information was collected or processed.Source: Ethical considerations · p. 10

A Pipeline Details

  • Each generator contributes 7,500 balanced samples across six RAID domains; texts under 50 whitespace-delimited tokens are excluded, and human passages are paired with their AI-generated counterparts by source.The six domains are abstracts, books, news, Reddit, reviews, and Wikipedia.Source: A Pipeline Details · p. 13
  • Frozen BERT-base-uncased yields concatenated CLS activations from 12 layers, forming 9,216 dimensions; the model weights are never updated.Inputs are truncated to 512 WordPiece tokens.Source: A Pipeline Details · p. 13
  • The L1 selector is trained on each fold, with neurons selected when their weights are nonzero; stability requires selection in at least 12 of 15 fold-and-seed cells.The separate frozen L2 evaluator is used for the main experiments, and patching replaces target activations with same-domain donor values.Source: A Pipeline Details · p. 13

B Stability Sweep

  • At C = 0.005 and N = 7,500, minimum Jaccard stability across six generators is 0.676; GPT-2 is the bottleneck, while the other five range from 0.707 to 0.751.At N ≤ 500, every generator selects zero neurons and the resulting Jaccard of 1.0 is trivial; N = 7,500 yields non-empty sets.Source: B Stability Sweep · p. 14

C Per-Cell Metrics

D AUC-ROC vs. L1 Selection

The appendix compares L1 selection with univariate AUC-ROC ranking, examines whether stable neurons localise the signal, and reports geometry, domain, layer, overlap, and patching summaries. Across these analyses, the results distinguish generator-specific patterns and exceptions rather than a single uniform structure.

D AUC-ROC vs. L1 Selection

  • L1 selects 4×–13× fewer neurons with equivalent or better performance for GPT-4, MPT, Mistral, and LLaMA; GPT-2 is similar across methods, while Cohere favors AUC-ROC.For Cohere, AUC-ROC selects fewer neurons with lower ablation cost; L1 and AUC-ROC selected sets have low overlap overall.Source: D AUC-ROC vs. L1 Selection · pp. 14, 17, Table 7 · p. 17

E Signal Localisation: Restricted Probe

F Detection Signal Geometry (CAV Diagnostics)

  • Cohere has the lowest CAV cosine similarity (0.057 ± 0.001), lowest CAV-projection LR accuracy (0.910 ± 0.008), and largest probe weight norm (108.3 ± 2.1).Instruction-tuned generators cluster at cosine 0.110–0.138 with the smallest weight norms; GPT-4’s is 45.8 ± 1.7.Source: F Detection Signal Geometry (CAV… · p. 17, Table 9 · p. 18

G Per-Domain Patching Breakdown

  • Domain sensitivity does not follow the base/instruction-tuned split: Mistral, MPT, and GPT-2 are domain-sensitive, while GPT-4, LLaMA, and Cohere are near domain-invariant.The source reports coefficient-of-variation values of 0.55, 0.42, and 0.33 for the sensitive generators, respectively.Source: G Per-Domain Patching Breakdown · p. 18

H Stable-Neuron Layer Distributions (Full Table)

  • Table 11 reports the per-layer distribution of stable neurons for all six generators.The supplied passage points to the table and its summary in Section 6.1 without giving the layer values.Source: H Stable-Neuron Layer Distributions… · p. 18

I Pairwise Jaccard Similarity (Full Table)

J Full k-Sweep Flip Rates

  • Table 13 reports selected flip rates across k values for each generator.The passage refers to the full k-sweep in Section 5.1.2 but supplies no rates.Source: J Full k-Sweep Flip Rates · p. 18

K What the Selected Neurons Track

The selected neurons carry measurable but incomplete surface-statistics signal, with associations that vary across generators. A worked Cohere example shows how patching a small stable set can reverse a confident probe prediction, although most targets do not flip.

K What the Selected Neurons Track

  • The analysis ranks stable neurons by mean absolute L1-probe weight, then tests their activations against four surface features over 9,996 samples with within-generator Benjamini–Hochberg correction.Features are computed on reconstructed token sequences aligned to cached encoder inputs.Source: K What the Selected Neurons Track · p. 18
  • 31–37 of 40 tests survive correction per generator, but the strongest association explains only roughly 16–29% of variance, so the selected set does not reduce to one surface cue.The strongest per-generator effects are |ρ| = 0.41–0.54.Source: K What the Selected Neurons Track · p. 19, Table 14 · p. 20
  • Mean word length is the top-associated feature for five of six generators, but its association is positive for GPT-4, GPT-2, and LLaMA and negative for Mistral and Cohere.Type–token ratio is top-associated for MPT.Source: K What the Selected Neurons Track · p. 19
  • The strongest surface-associated neurons span layers 1, 5, and 12 across generators; L12/308 and L5/531 recur across generator pairs and belong to the 17-neuron cross-generator core.The strongest-associated neurons sit in layer 12 for GPT-4 and LLaMA, and in layers 1 and 5 for GPT-2, MPT, Mistral, and Cohere.Source: K What the Selected Neurons Track · p. 19
  • Surface statistics are the strongest single-feature account obtained, but they leave the selected neurons’ behavior incompletely explained; learned feature dictionaries are proposed for identifying remaining structure.The four hand-chosen features account for roughly 16–29% of variance in the strongest associations.Source: K What the Selected Neurons Track · p. 19

L A Worked Patching Example

  • Editing 0.67% of the representation flipped one confident human prediction; the same donor draw flipped 47 of 750 targets (6.3%), near Cohere’s 8.15% forward flip rate.The example patches 62 stable neurons, leaving the other 9,154 unchanged; most targets do not flip.Source: L A Worked Patching Example · p. 20
Loading 2609.30287v1…