PaperlayerView exact revision on arXiv

Source-linked AI summary

Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions

Beiming Liu, Minjie Chen

arXiv:2609.39243v2cs.LG

TL;DR

Causal interventions can produce the expected answer through pathways that natural inputs do not use, but prior work lacked a way to distinguish this from harmless divergence. The paper plants silent pathways in pretrained models to create known labels and evaluates detectors against them. Site distances fail on this benchmark, while downstream tests work when pathways create unit-level out-of-regime activity; in-range pathways remain difficult to detect.

  • Problem

    Interventions can leave the natural distribution in harmless or pernicious ways, and researchers lack a principled method to tell whether a successful intervention supports a natural-mechanism claim [8].

    Source: §1 · p. 2

  • Method

    The paper plants pathways silent on benchmark prompts in pretrained language models, giving known labels for intervention dependence, and tests downstream clamping as a label-free detector.

    Source: §1 · p. 2, §4 · p. 5, §5 · p. 6

  • Results

    Site-level distances fail on the planted-pathway benchmark, while downstream tests succeed when pathways produce unit-level out-of-regime activity; in-range pathways are only partly detected.

    Source: §6.2 · p. 8, §8 · pp. 14–15

  • Takeaways & Limitations

    The benchmark provides a target for detectors, and the paper concludes that hidden pathways in in-range combinations remain a hard case.

    Source: §8 · p. 15

    Planted pathways test detector capabilities rather than how often such pathways arise naturally, and range clamping fails on in-range pathways in most tested settings.

    Source: §7 · p. 14

Abstract

from arXiv · show

Causal interventions such as activation patching and distributed alignment search (DAS) are the main tool for making mechanistic claims about neural networks. Recent work showed that these interventions routinely push representations off the model's natural distribution, and that such divergence is sometimes harmless and sometimes pernicious: it can recruit pathways the model never uses on natural inputs, so that an intervention produces the expected answer through the wrong mechanism. No method currently tells the two cases apart. We make this question testable by planting hidden pathways inside pretrained language models; the pathways are silent on every benchmark prompt by construction, so which interventions depend on them is known exactly. Across 72 configurations and 100,800 interventions on GPT-2 small, we find three things. (i) Nearest-neighbour and local-PCA distances at the intervention site, as used in prior work, score below chance (AUROC 0.35-0.47) at picking out interventions that give the right answer through a planted pathway. (ii) Hidden-Pathway Contribution (HPC), a label-free test that clamps downstream units to the regime of natural runs with the same output and measures how much of the decision disappears, flags pathway-dominated interventions with AUROC >= 0.99 when the pathway shows up as unit-level out-of-regime activity, but fails when every unit stays within its natural range, which we identify as the open problem. (iii) Optimised interventions actively seek hidden pathways: on a gender task, DAS routes 90-95% of its successes through planted pathways for three of four families, and a downstream on-manifold penalty cuts this share to under 5% at a cost of 6-11 points of success rate. In unmodified GPT-2, successful interventions show almost no unit-level out-of-regime reliance.

1 Introduction

Causal interventions are used to infer neural mechanisms, but they may produce the right answer through a pathway absent from natural computation. The paper makes this distinction testable with planted pathways and evaluates detectors against known intervention outcomes.

1 Introduction

  • A successful intervention may use a hidden pathway rather than the model’s natural mechanism, leaving researchers without a principled way to classify harmful divergence [8].The benchmark plants pathways that remain silent on benchmark prompts, making pathway dependence knowable.Source: §1 · p. 2, Figure 1 · p. 3
  • Prior work established that interventions can leave the natural distribution and that divergence may activate hidden pathways [8] [18] [31].Grant et al. [8] left open a principled method for classifying harmful divergence.Source: §1 · p. 2
  • Site-level distance alone cannot distinguish harmless from pernicious divergence because it responds to both kinds.The paper’s benchmark addresses the lack of ground truth by planting hidden pathways.Source: §1 · p. 2, Figure 1 · p. 3
  • The paper evaluates site-level distances, a downstream detector, and optimised interventions against planted pathways with known dependence.The contributions include a benchmark in pretrained language models and a bounded prevalence check in unmodified models.Source: §1 · p. 2, Figure 1 · p. 3

2 Related work

Prior work developed intervention methods, manifold-based checks, and benchmark models for interpretability, while leaving open how to evaluate whether divergence is pernicious. This paper supplies an evaluation with known ground truth from planted pathways.

2 Related work

  • Activation patching, causal scrubbing, and optimal ablations address intervention design or distortion, while earlier work documents divergence and its pitfalls [18] [31].Grant et al. [8] proposed a counterfactual-latent loss that shrinks divergence overall but does not target its pernicious part.Source: §2 · p. 3
  • Manifold projection and distance scores assess representations at the intervention site, whereas downstream self-repair complicates reasoning about later effects [17] [19].The cited work also connects out-of-distribution concerns to perturbation-based attribution.Source: §2 · p. 3
  • Existing circuit and alignment benchmarks use compiled transformers, semi-synthetic models, or standardised tests; this work minimally edits pretrained models while preserving their natural benchmark computation.The added pathways have no effect on benchmark prompts.Source: §2 · p. 3

3 Problem setup

The setup treats an intervention as replacing an internal residual-stream representation and judging its effect on a binary answer. It defines pernicious divergence by whether the decision relies on downstream computation absent from natural runs with the same output.

3 Problem setup

  • The task is a binary decision measured by the logit difference between two answer tokens, with target and source prompts differing in one causal variable.The causal variable takes values corresponding to the two classes.Source: §3 · pp. 3–4
  • An intervention replaces the target prompt’s residual-stream state at a layer and position with a modified representation; it succeeds when the resulting answer matches the source class.Success is defined by the sign of the logit difference matching that class.Source: §3 · p. 4
  • An intervention is pernicious for natural-mechanism claims when its decision depends on downstream computation absent from natural runs producing the same output.The paper notes that this property cannot be observed directly in real models, motivating its benchmark.Source: §3 · p. 4

4 A planted-pathway benchmark

The benchmark adds hidden pathways to pretrained models while preserving their decisions on benchmark prompts, then tests interventions whose dependence on those pathways can be labelled. Its task, model, site, and intervention choices define the scope of those evaluations.

4 A planted-pathway benchmark

  • The benchmark uses paired subject–verb agreement and gender prompts, plus indirect object identification only for unmodified-model analysis.The SVA and gender pairs differ in one variable and share fixed answer tokens; retained pairs are answered correctly with |LD| > 1.Source: §4 · p. 4
  • Evaluations cover GPT-2 small, two Pythia models, and Qwen2.5-7B at selected subject-token and final-token sites.The models use fixed revisions, with seed 0 for GPT-2 and three seeds for the larger models.Source: §4 · p. 4
  • Planted pathway units are silent on benchmark prompts, so natural decisions remain unchanged, while their activation can arise when interventions push downstream states out of regime.Pathways use a layer later than the intervention site; the four families include units clipped to their natural ranges and three families visible through unit-level out-of-regime activity.Source: §4 · pp. 4–5
  • The benchmark labels pathway-dominated interventions by whether pathways supply over half the decision margin, and wrong-mechanism successes by whether success flips when pathways are switched off.Each configuration evaluates 1,500 held-out interventions across patching, steering, DAS, and restricted-DAS types.Source: §4 · p. 5

5 Detectors

The section compares intervention-site distance baselines with downstream activation-based tests. Hidden-Pathway Contribution measures how much of the predicted class’s margin is lost when downstream activity is restricted to regimes observed in natural runs of that class.

5 Detectors

  • Site-level baselines score intervention-site representations using nearest-neighbour distance, Mahalanobis distance, local-PCA residuals, or an off-manifold perturbation test.The perturbation test adds the predicted-class local-PCA off-manifold component to 32 natural evaluation prompts and reports the largest logit-difference change.Source: §5 · p. 5
  • Down-OM averages each later MLP layer’s relative residual from the predicted class’s natural PCA subspace, retaining 99% of variance with at most 256 components.Source: §5 · p. 5
  • HPC reruns the intervention while clamping downstream neurons layer by layer to their predicted-class natural regimes, then measures the fraction of a typical natural margin lost.Later layers receive the clamped inputs; HPC requires only natural prompts and the model, not labels.Source: §5 · p. 6
  • HPC variants clamp rare active neurons only (HPC-S), all neurons (HPC-L), or activations after class-conditional PCA or local-PCA projection (HPC-PCA and HPC-Loc).The local-PCA projection uses the 30 nearest natural activations of the predicted class.Source: §5 · p. 6
  • HPC is zero for harmless null-space divergence by construction, while projection variants can detect combination-level anomalies that individual unit ranges miss.Silent planted units have natural regime {0}, making HPC-S and HPC-L switch them off; this family is therefore only a sanity check.Source: §5 · p. 6

6 Results

Optimised interventions often exploit planted hidden pathways, while site-level distances fail to reliably identify wrong-mechanism successes. Downstream clamping detects unit-visible pathway reliance, but in-range pathways remain difficult to detect, and standard interventions in unmodified models rarely show out-of-regime reliance.

6.1 A synthetic sanity check

  • In the synthetic sanity check, site distances produce 93–100% false positives on harmless null-space divergence at 90% recall, while HPC performs best.The benchmark uses five seeds and 3,000 interventions per seed.Source: §6.1 · p. 6, Table 1 · p. 6

6.2 Site-level distance does not identify wrong-mechanism successes

  • On gender, nearest-neighbour and local-PCA site distances score 0.21–0.46, while HPC-L scores 0.82–1.00 for unit-visible families among successful interventions.HPC-L does not reliably detect in-range pathways; projection variants help on some larger-model tasks, but no detector is reliably above chance on Qwen2.5-7B gender.Source: Table 2 · p. 7, §6.2 · p. 8, Figure 2 · p. 10

6.3 Optimised interventions seek hidden pathways

  • On gender, unrestricted DAS routes 81–100% of successes through planted pathways across models, while restricted DAS also often succeeds through those pathways.For GPT-2, unrestricted DAS routes 90–95% of gender successes through pathways; the downstream on-manifold penalty reduces this share to 4.5–4.7%, while lowering success from 94–98% to 87–89%.Source: §6.3 · pp. 8, 10, Table 3 · p. 9, Table 4 · p. 11

6.4 Unmodified models

  • In unmodified models, at most 0.8% of standard-intervention successes show HPC-L > 1, and the fraction is zero in Qwen2.5-7B.Restricted DAS can produce many successes without this signal; the authors conclude that reliance on in-range combinations would go undetected by HPC-L.Source: Table 5 · p. 11, §6.4 · pp. 12–13, Table 7 · p. 13
  • For Qwen2.5-7B restricted DAS, some SVA and gender successes show HPC-L > 1, but their mechanisms cannot be confirmed without planted ground truth.These cases occur at the final-token site, and none exceeds 1 under HPC-S; clamping may overestimate dependence when later layers compensate.Source: §6.4 · pp. 12–13

6.5 Robustness

  • Across seeds, no qualitative conclusion changes; however, rerunning one GPT-2 configuration on different hardware changes several AUROCs by up to 0.21.The paper reports per-configuration averages because same-seed runs are not bit-reproducible across hardware; an exploratory instruction-tuning check is explicitly not evidence about chat-formatted use.Source: §6.5 · pp. 13–14

7 Discussion and limitations

The benchmark establishes what detectors can reveal about constructed hidden pathways, but does not establish how often such pathways occur naturally. Its findings are also bounded by detector coverage, evaluation choices, and the claims HPC is designed to assess.

7 Discussion and limitations

  • Planted pathways test detector capabilities, not how often hidden pathways arise naturally.Three of four pathway families are designed to be visible to unit-level range tests, so their near-perfect scores do not establish real-world reliability.Source: §7 · p. 14
  • Clamping downstream units can trigger later-component compensation, making LDΠ an imperfect estimate of the lost decision margin.This matters most for the natural-model candidates of §6.4.Source: §7 · p. 14
  • Configuration means share prompts and baselines, so they are not independent replicates, and reported standard deviations are not confidence intervals.The study also uses one seed for GPT-2 main configurations and three for larger models; several constants are fixed rather than tuned.Source: §7 · p. 14
  • For the largest models, every-second-layer analysis and bfloat16 weaken the natural-model check; HPC covers only final-position MLP neurons and requires a class-conditional decision.The mitigation was evaluated only on GPT-2; attention-mediated and other-position pathways are not covered.Source: §7 · p. 14
  • HPC targets claims about the natural mechanism; out-of-regime computation may be acceptable for claims such as subspace sufficiency.Source: §7 · p. 14

8 Conclusion

The conclusion argues that whether an intervention gives the right answer for the right reason must be tested downstream against natural-input behavior. The planted-pathway benchmark makes this question measurable and reveals both detector successes and remaining blind spots.

8 Conclusion

  • Whether an intervention gives the right answer for the right reason cannot be inferred from the intervened representation alone; the test is downstream comparison with natural inputs.Source: §8 · p. 14
  • Site-level distances fail on the planted-pathway benchmark, while a downstream causal test succeeds when pathways produce unit-level out-of-regime activity.The benchmark spans models from 124M to 7B parameters.Source: §8 · p. 14
  • Optimised interventions exploit hidden pathways when those pathways are easier to use than the natural mechanism.In an unmodified 7B model, restricted DAS produces successes with the same downstream signature.Source: §8 · p. 15
  • Pathways hidden in in-range combinations of units are only partly detected by projection onto the natural manifold, and not at all on one model–task pair.Source: §8 · p. 15

A Implementation details

The implementation evaluates planted pathways and detector methods across GPT-2 and larger language models, using shared intervention procedures alongside separately budgeted additional runs. Saved per-intervention arrays generate the reported tables and figures.

A Implementation details

  • GPT-2 main runs use seed 0, four intervention sites, five pathway settings per task, and up to 6,000 prompt pairs; mitigation runs use 900 interventions per setting.The IOI unmodified-model runs use restricted DAS; GPT-2 runs use float32 and all downstream layers.Source: A Implementation details · p. 17
  • The three larger models use three seeds, 24 planted units, and up to 4,000 or 6,000 prompt pairs, with every-second-layer analysis for the two largest models.Their main runs comprise 108 configurations per task–model pair; the unmodified-model sweep adds restricted DAS across tasks and component-exclusion settings.Source: A Implementation details · p. 17
  • Shared settings include 1,500 interventions per configuration, 32 natural evaluation prompts, and 150 DAS training steps, with restricted DAS trained for 300 steps.DAS uses Adam at learning rate 10^-2 and batch size 128; interventions are evaluated on disjoint pairs.Source: A Implementation details · p. 17
  • Natural-regime quantiles are computed per class over filtered benchmark prompts, and each configuration verifies that planting leaves every natural decision unchanged.Source: A Implementation details · p. 17
  • The 39 additional configurations use different budgets and are reported separately rather than pooled with the main results.They include GPT-2 seeds 1–2 and an exploratory Qwen2.5-1.5B versus instruction-tuned comparison on gender.Source: A Implementation details · p. 18
  • A single script generates every table and figure from saved per-intervention arrays, with configuration parameters, model revisions, software versions, and file hashes recorded.Source: A Implementation details · p. 18

B Changes in version 2

Version 2 expands the model and detector evaluations while retaining the original GPT-2 results. Its main qualitative conclusions persist at larger scales, with updates for in-range pathways and unmodified Qwen2.5-7B.

B Changes in version 2

  • Version 2 adds three larger models, an HPC-PCA variant, expanded restricted-DAS component sweeps, positive-control and seed tables, a hardware check, and an instruction-tuning comparison.Source: B Changes in version 2 · p. 18
  • The GPT-2 results from version 1 are unchanged, and the main qualitative conclusions hold at larger scales.Source: B Changes in version 2 · p. 18
  • At larger scales, local-manifold projection partly detects in-range pathways, and unmodified Qwen2.5-7B yields candidate wrong-mechanism successes under restricted DAS.Source: B Changes in version 2 · p. 18

C Additional tables

These additional tables report AUROC analyses of wrong-mechanism interventions and describe seed, hardware, replication, and model-variant evaluations. Their captions specify the aggregation and evaluation budgets, but do not provide the table values.

C Additional tables

  • Table 8 reports mean AUROC for detecting pathway-dominated interventions, defined as those where a planted pathway supplies more than 50% of the decision margin.Means cover configurations with at least five positives; the table caption identifies the best result per row in bold.Source: Table 8 · p. 19
  • Table 9 reports pooled AUROC for wrong-mechanism successes, with scores rank-normalised within each configuration and 95% bootstrap confidence intervals over interventions.The intervals use 1,000 resamples.Source: Table 9 · p. 20
  • Table 10 summarizes seed variation by averaging wrong-mechanism-success AUROC across configurations within each seed, then reporting mean±s.d. and min–max across seeds 0–2.Source: Table 10 · p. 21
  • Table 11 compares one GPT-2 configuration across the original consumer GPU and the H20 used for the new models, using the same code and hyperparameters.Source: Table 11 · p. 21
  • Table 12 presents GPT-2 seed replication results side by side rather than pooled because the main runs and later seeds use different budgets.Seeds 1–2 have larger budgets; SVA results for those seeds cover only 5 of 20 configurations.Source: Table 12 · p. 22
  • Table 13 explores Qwen2.5-1.5B base versus Instruct on raw gender-task completion prompts, with small budgets and sampled downstream layers.Instruct in-range has too few positives for scoring; the table also records successful interventions with HPC-L > 1.Source: Table 13 · p. 22

Related papers

Loading 2609.39243v2…