PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?

Li Zhang, Chuqin Geng, Mark Zhang, Chen Yang, Luke Zhang, Haolin Ye, Xujie Si

arXiv:2609.35686v1cs.LGcs.AI

TL;DR

Circuit explanations aim to account for model behavior, but reproducing correct task answers does not necessarily explain why the model errs. This paper evaluates circuit agreement separately on model successes and errors, finding that many circuits preserve correct behavior while missing most errors. An IOI case study shows that restoring omitted attention heads can recover many errors with little change in correct agreement.

  • Problem

    Circuit explanations aim to capture the computations behind model behavior, but whether existing methods account for the model’s naturally occurring errors remains an open question.

    Source: §1 · p. 2

  • Method

    The paper measures exact circuit–model answer agreement separately on model-correct and model-error prompts across tasks, circuit sizes, and ablation settings, then investigates IOI error recovery with circuit completion and interventions.

    Source: §1 · p. 2, §4.3 · p. 7, §5.1 · p. 8

  • Results

    On IOI under mean ablation, tested circuits preserve 97.3%–99.5% of correct choices but reproduce only 11.4%–41.7% of errors.

    Source: §3 · p. 4

  • Takeaways & Limitations

    Exact reproduction of a model’s wrong answers is a necessary, but not sufficient, test for circuits intended to explain model behavior.

    Source: §7 · p. 10

    The conclusions are scoped to the tested settings and benchmarks, and the single-circuit IOI case study does not establish that recovered mechanisms generalize to other circuits or tasks.

    Source: §7 · p. 9

Abstract

from arXiv · show

Mechanistic interpretability (MI) aims to explain a model's behaviour through analyzing its internal computations; circuit-based explanations aim to isolate these computations with compact subnetworks validated by ablating the rest of the model. We show that circuits validated this way may fail to recover the underlying mechanism of the model's behaviour by closely reproducing its successful decisions while failing to account for most of its errors. Such explanations should account for the model's particular errors as well as its successes. We evaluate this requirement by measuring exact answer agreement separately on model successes and failures, across circuit sizes and ablation settings, on IOI, Docstring, and six model-task settings from the Mechanistic Interpretability Benchmark. We discover that many tested circuits closely replicate correct behaviour while missing most of the model's errors. On indirect object identification (IOI) for GPT-2 small, under mean ablation, the manual circuit and tested automated circuits, including one trained against the model's full output distribution, agree with the model on 97.3-99.5% of prompts it answers correctly but only 11.4-41.7% of errors. An IOI case study shows that lost errors are recoverable by restoring omitted attention-heads which raise error reproduction from 14.2% to 75.1% on a separate held-out set with 0.41 percentage point decrease on correct agreement, exceeding matched random extensions and scalar-biased control. Intervention traces show how omitted computations produce specific wrong answers for a reproducible subset of errors. In all, these findings show circuits can preserve task success without adequately explaining model's failures, and support exact error reproduction as a necessary, but not sufficient, test of circuit-based explanations of model behaviour.

1 INTRODUCTION

Mechanistic interpretability seeks to explain model behavior by reverse-engineering internal computations into circuits, whose faithfulness is commonly evaluated under specific interventions. This paper asks whether existing methods account for naturally occurring model errors as well as correct behavior.

1 INTRODUCTION

  • Circuits are proposed accounts of task-specific model behavior, built by connecting computation units into subgraphs.Researchers identify these components and connections through manual analysis and automated extraction.Source: §1 · p. 1
  • Prior work evaluates circuit faithfulness by measuring how closely a circuit preserves full-model behavior under a chosen metric and intervention.Common measures include task scores such as average logit difference and divergence between model and circuit output distributions.Source: §1 · pp. 1–2
  • The paper asks whether existing circuit-analysis methods account for naturally occurring model errors, since explanations should cover wrong as well as correct behavior.A circuit that gives the right task answer when the model is wrong may omit influences behind the model’s actual choice, limiting its diagnostic value.Source: §1 · p. 2
  • Evaluated manual circuits and most tested automated methods often preserve correct behavior while inadequately reproducing model errors.The study characterizes these failure modes and tests error recovery and underlying mechanisms through circuit completion and causal interventions on IOI.Source: §1 · p. 2

2 PRELIMINARIES

The paper defines tasks and circuits in terms of prompt distributions, answer labels, and computations retained under specified ablations. It then measures circuit agreement separately on model successes and errors across IOI, Docstring, and selected MIB benchmarks.

2.1 TASKS AND CIRCUITS

  • A task specifies a prompt distribution, a correct-answer rule, and a success measure; a circuit is a computation-graph subgraph evaluated under a stated patching setting.Resample, mean, and optimal ablations use counterfactual activations, averages, and learned replacement constants, respectively.Source: §2.1 · p. 2

2.2 FAITHFULNESS METRIC

  • Circuit faithfulness is scored against the model’s choice, with separate correct-agreement and error-agreement rates and their difference.Predictions rank single-token candidates; a model error is a disagreement with the task label, and a circuit must reproduce the model’s particular answer to count as agreement.Source: §2.2 · pp. 2–3

2.3 BENCHMARKS

  • The benchmarks cover GPT-2 Small IOI, a four-layer attention-only Docstring task, and MIB ARC and subtraction settings.IOI evaluates the indirect object against the repeated subject; Docstring predicts the next argument name. Addition and synthetic MCQA are omitted for having few model errors, while GREATER-THAN is omitted for reported 100% correct top-1 performance.Source: §2.3 · p. 3

3 AGREEMENT ANALYSIS

The section compares exact circuit–model agreement on correct and erroneous cases across ablations, circuit sizes, and tasks. It finds that circuits can preserve successful predictions while reproducing relatively few of the model’s errors.

3 AGREEMENT ANALYSIS

  • The comparisons use task faithfulness and exact agreement under mean and paired-resample ablation for IOI and DOCSTRING, with counterfactual replacement values for MIB; selected circuits use a validation threshold of Fval ≥ 0.85.Optimal ablation learns replacement values to minimize KL divergence from the full model’s output distribution.Source: §3 · pp. 3–4
  • Under mean ablation on IOI, circuits preserve 97.3%–99.5% of correct choices but reproduce 11.4%–41.7% of errors; the manual circuit’s F = 0.956 accompanies 15.2% error reproduction.The table reports normalized task faithfulness and exact agreement with the model under argmax over the candidate set.Source: Table 1 · p. 4, §3 · p. 4
  • For DOCSTRING, resampling narrows the manual circuit’s agreement gap from 41.3% to 8.2% as correct agreement falls from 77.1% to 47.2%; EAP-IG reaches 69.4% error agreement and 65.6% correct agreement under mean ablation.The smaller gap can reflect lost correct behavior rather than better error reproduction.Source: §3 · p. 4
  • At a 1% edge budget, the Gemma ARC-Challenge circuit preserves 80.3% of correct choices but reproduces only 54.4% of wrong answers, despite errors comprising 38.4% of discovery examples.Across the six MIB settings, discovery splits contain 187–849 errors, or 4.9%–41.1% of examples.Source: §3 · pp. 4–5
  • Weighting IOI errors at 50% of discovery KL raises mean-ablation error agreement from 9.3% to 37.0%, while correct agreement falls from 98.7% to 91.0%.The intervention improves error reproduction but leaves an agreement gap and incurs a correct-case cost.Source: §3 · p. 5
  • Margin matching narrows several MIB ARC gaps, but lowers Llama ARC-Challenge correct agreement at a 2% edge budget from 80.0% overall to 46.7% among matched examples; positive gaps persist in all reported IOI and most DOCSTRING configurations.Gemma ARC-Easy retains gaps of 15.3%–21.2% at 5%, 10%, and 20% edge budgets.Source: §3 · p. 5
  • Other recovery attempts have mixed effects: an ACDC IOI split of 500 errors and 500 correct examples yields 44.2% error agreement and 91.4% correct agreement, while threshold calibration can narrow gaps by sacrificing correct performance.Error-conditioned replacements improve IOI error agreement, but shifting logits toward the subject name reproduces at least as many errors.Source: §3 · p. 5
  • Figure 2 traces two omitted computations: contextual restoration changes queries to produce Courtney instead of Sara, while name-source restoration shifts Name Movers toward Vanessa over Katie.The latter reduces Katie retrieval by a Positive Name Mover and increases it by a Negative Name Mover.Source: Figure 2 · p. 6

4 CASE STUDY: WHY THE MANUAL REFERENCE CIRCUIT MISSES MODEL

The IOI case study tests whether omitted components can restore errors missed by the manual reference circuit while preserving correct agreement, then traces how those components affect retained computations. A restricted extension substantially improves exact-error agreement on held-out data, and the traces identify query and retrieval changes involved in a subset of errors.

4.1 UNDERSTANDING MISSING ERRORS

  • The case study asks which omitted components contribute to missed errors, whether recovery preserves correct agreement, and whether missing contributions are concentrated and stable.The search greedily screens head-position candidates and evaluates exact binary interventions, using an objective that rewards recovery while penalizing losses in correct agreement and added nodes.Source: §4.1 · pp. 6–7
  • The analysis uses GPT-2 small, the published IOI reference circuit, attention-head additions at selected token positions, and mean knockouts.The extension experiment broadens the search to 25 positional rules across 144 attention heads.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 ERROR RECOVERY ON FRESH DATA

  • 14.22% to 75.11%: on a frozen 26,000-example test set, the 63-node extension raises error agreement over the reference circuit, with correct agreement decreasing only 0.41 percentage points.The extension recovers 137 additional model errors and exceeds structurally matched random extensions and a scalar subject-bias control.Source: §4.2 · p. 7
  • The authors conclude that a restricted, position-specific extension recovers many failures missed by the reference circuit while largely preserving behavior on model-correct examples.The held-out evaluation reports 98.93% correct agreement for the extension.Source: §4.2 · p. 7

4.3 HOW OMITTED COMPUTATIONS CHANGE WHICH NAME WINS

  • The study uses activation and path patching, including counterfactual prompts, to test whether selected changes recover the model’s wrong answer or restore the correct answer.Recovery is defined as reproducing the model’s preference for the incorrect subject name; repair means restoring preference for the correct IO name.Source: §4.3 · p. 7
  • S-inhibition signals altered by object-related inputs change Name Mover queries; patching the changed inhibition contribution reproduces the wrong subject answer on 120 of 137 error prompts.The traces show increased downstream attention to the subject; the object substitution restores the correct IO answer on 107 prompts, though the responsible semantic feature remains unidentified.Source: §4.3 · p. 8
  • Restoring five first-layer heads changes the IO representation used to compute Name Mover keys; patching those keys reproduces the model’s wrong answer on 35/137 examples.In the measured route, changed keys reduce IO retrieval by Positive Name Movers and increase it by Negative Name Movers.Source: §4.3 · p. 8
  • Path patching the S-inhibition heads’ S2 contribution into three Name Mover query groups reproduces 120/137 newly recovered errors, and recovery persists under resample ablation.Under resample ablation, error agreement rises from 40.89% to 60.89%, while correct agreement rises from 91.41% to 97.67%.Source: §4.3 · p. 8

5 RECOMMENDATIONS

The recommendations call for evaluating circuit agreement separately on prompts the model answers correctly and incorrectly, across specified circuit sizes. Reports should make the split, size, and ablation choices explicit, and error agreement requires reproducing the model’s exact wrong answer.

5 RECOMMENDATIONS

  • Fix the model-correct and model-error strata for each data split across all evaluated circuits, sizes, and ablation methods.If there are no model errors, error agreement and its difference from correct agreement are undefined.Source: §5 · p. 8, §5.1 · p. 8

5.1 RAW METRICS FOR EXTRACTED CIRCUIT

  • Report correct agreement, error agreement, and their difference; the additional Berr diagnostic separates circuits that answer errors correctly from those giving a different wrong answer.The recommendations describe Berr as the fraction of model-error prompts on which the circuit is also incorrect.Source: §5.1 · pp. 8–9

5.2 EVALUATION ACROSS CIRCUIT SIZES

  • Evaluate agreement across a specified sequence of circuit sizes and plot correct and error agreement separately, since a smaller gap can reflect either improved error agreement or reduced correct agreement.Specify retained-component counts and units, and report the evaluated split, prompt counts, and ablation rule.Source: §5.2 · p. 9
  • Report agreement separately on model-correct and model-error prompts, requiring the same incorrect answer on the error stratum, and state the split, stratum counts, circuit size, and ablation rule.A circuit returning the task-correct answer, or a different wrong answer, disagrees with the model on its error prompts.Source: §5.1 · p. 8, §5.2 · p. 9

6 RELATED WORK

Related work addresses circuit faithfulness, evaluation methods, and mechanistic accounts of particular model failures. This paper distinguishes itself by systematically testing whether circuit explanations reproduce models’ wrong answers and tracing omitted computations associated with recovering some errors.

6 RELATED WORK

  • Prior work studies circuit faithfulness, ablation sensitivity, behavioral preservation, localization, minimality, and circuit discovery across sizes, while other studies provide mechanistic accounts of specific failures.This work systematically tests wrong-answer reproduction and traces omitted computations that recover a subset of errors.Source: §6 · p. 9

7 LIMITATIONS AND CONCLUSION

The tested circuits often preserve correct predictions better than the model’s particular wrong answers, while an IOI case study traces how omitted computations produce specific errors. The findings are scoped to tested settings, and the study does not establish a general recovery solution.

7 LIMITATIONS AND CONCLUSION

  • The detailed recovery case study covers one IOI reference circuit and its extension, without establishing that the recovered mechanisms generalize to other circuits or tasks.The wider findings cover IOI, DOCSTRING, and MIB across the tested models, methods, sizes, and ablation settings.Source: §7 · p. 9
  • The study does not exhaust recovery strategies or establish a general solution that consistently recovers omitted computations while preserving correct behaviour.The conclusion recommends exact wrong-answer reproduction alongside correct-answer preservation as a necessary test for circuits intended to explain model behaviour.Source: §7 · pp. 9–10
  • Tested manual circuits and most evaluated automated methods reproduce correct predictions substantially better than the model’s particular wrong answers.The IOI case study traces how missed computations produce specific wrong answers.Source: §7 · p. 10

A EXPERIMENT DETAILS: IOI AND DOCSTRING

The IOI and DOCSTRING experiments use distinct model and prompt constructions, with disjoint discovery, validation, and test sets. Evaluation retains model errors and scores exact agreement under specified prediction and uncertainty rules.

A EXPERIMENT DETAILS: IOI AND DOCSTRING

  • IOI uses frozen GPT-2 small, while DOCSTRING uses a four-layer, eight-head-per-layer attention-only model.Weights remain fixed throughout extraction and replacement fitting; computations use float32 with TF32 disabled.Source: §A.1 · p. 13
  • IOI prompts vary across 15 template families and ABBA/BABA order, while DOCSTRING prompts predict a third matching argument from a fixed set of 14 candidate tokens.IOI ablation prompts remove the repetition structure and match the clean prompt’s token length; DOCSTRING ablations randomize variable names.Source: §A.1 · p. 13
  • Discovery pools contain 13,000 IOI and 3,000 DOCSTRING examples, and discovery, validation, and test sets are disjoint.The discovery pools include both correct and incorrect model predictions, with equal weight per example.Source: §A.1 · p. 13
  • Evaluation samples 10,884 IOI prompts while retaining all model errors, and evaluates DOCSTRING on all 10,000 test prompts.Exact error agreement requires matching the model’s identical wrong candidate; gap intervals use 10,000 bootstrap draws.Source: §A.1 · p. 13

A.2 DISCOVERY IMPLEMENTATIONS

The experiments compare manual and automated circuit constructions with distinct discovery objectives and replacement strategies. Recovery interventions can improve error reproduction, but their effects on correct agreement vary.

A.2 DISCOVERY IMPLEMENTATIONS

  • The manual IOI reference retains 26 attention heads at specified positions, while the DOCSTRING reference retains eight heads.Other IOI attention-head outputs are ablated, with MLPs remaining active.Source: §A.2 · p. 14
  • ACDC removes candidate edges when their removal keeps the increase in full-vocabulary output-distribution KL below a selected threshold.Its task loss uses the answer position without ground-truth labels.Source: §A.2 · p. 14
  • EAP-IG uses input-integrated gradients to score edges, then builds circuits backward from the output using absolute attribution scores.Attributions use five interpolation steps and the full-vocabulary KL objective at the answer position.Source: §A.2 · p. 14
  • IOI Edge Pruning learns sparse circuits with edge gates and evaluates multiple sparsity targets across three random seeds; it is not evaluated on DOCSTRING.The official implementation does not support DOCSTRING’s attention-only architecture.Source: §A.2 · p. 14

A.3 REPLACEMENT ACTIVATIONS AND NORMALIZED FAITHFULNESS

  • Mean replacements use position-specific averages of source activations, while optimal ablation jointly learns input-independent constants to minimize full-vocabulary KL.The DOCSTRING mean-replacement reference uses separate corrupted prompts; optimal-ablation constants are shared across token positions and downstream readers.Source: §A.3 · p. 15
  • Table 4 summarizes recovery interventions across discovery data, ablation values, output predictions, and circuit membership; several yield substantial error-reproduction gains, with varying correct-agreement effects.The tested interventions include exposing extraction to more errors, increasing their discovery weight, changing replacements, calibrating outputs, and restoring omitted components.Source: Table 4 · p. 15, §A.3 · p. 16

B MIB DATA, EXTRACTION, AND METRIC IMPLEMENTATION

The MIB evaluation uses public prompts and splits, with counterfactual inputs for circuit scoring and evaluation. Circuits are assessed across edge budgets using normalized task-score faithfulness and CPR/CMD metrics.

B MIB DATA, EXTRACTION, AND METRIC IMPLEMENTATION

  • Table 5 reports discovery coverage and candidate-scored public-test strata for the evaluated curves from the full public-training set.The paper reports full correct/error statistics in the table.Source: Table 5 · p. 16, B MIB DATA, EXTRACTION, AND METRIC… · p. 16
  • ARC predictions use the highest-logit answer-label token, while subtraction uses single-token integer candidates from 0 to 99.Ties follow the original candidate order: choice order for ARC and ascending numerical order for subtraction.Source: B MIB DATA, EXTRACTION, AND METRIC… · p. 16
  • MIB circuits use EAP-IG scores from the public training split, with ARC label substitutions and random subtraction operands as counterfactual prompts.Excluded edge contributions are replaced with activations from the corresponding counterfactual run.Source: B MIB DATA, EXTRACTION, AND METRIC… · p. 16
  • Circuits are evaluated at edge budgets using normalized task-score faithfulness; Table 6 reports CPR, where higher is better, and CMD, where lower is better.The supplied passage does not give the edge-budget values or metric results.Source: B MIB DATA, EXTRACTION, AND METRIC… · p. 16

C MARGIN-MATCHED AGREEMENT

The section tests whether matching correct and error examples on intact-model prediction margins explains the agreement gap. Positive gaps persist in all twelve IOI configurations and in most MIB comparisons, though results vary by setting and size.

C MARGIN-MATCHED AGREEMENT

  • For IOI and Docstring, model-error examples are paired without replacement to correct examples within 0.1 logit slack on the highest-versus-second-highest candidate logits.Predictions use argmax over each benchmark's candidate set.Source: C MARGIN-MATCHED AGREEMENT · p. 16
  • Table 7 reports agreement with the model after matching correct and error predictions on absolute answer margin, using the same circuits and selection procedures as Table 1.The matching uses argmax predictions over the benchmark candidate set K(x).Source: C MARGIN-MATCHED AGREEMENT · p. 16, Table 7 · p. 17
  • All twelve IOI configurations retain positive matched gaps; Docstring EAP-IG-KL is an exception under mean and resample ablation, with negative percentage points.Matching retains 805/884 IOI errors and 3,534/3,536 Docstring errors.Source: C MARGIN-MATCHED AGREEMENT · p. 17
  • Matching reduces several ARC gaps, but MIB results still depend on setting and size; subtraction margin matching retains only 20/56 errors.Figure 3 compares correct-minus-error agreement gaps before and after margin matching, with pointwise 95% bootstrap intervals.Source: C MARGIN-MATCHED AGREEMENT · p. 17, Figure 3 · p. 18

D.1 TASK, PREDICTION RULE, AND REFERENCE CIRCUIT

The IOI case study fixes the task, candidate-based prediction rule, and reference circuit before searching for additions. It screens omitted head-position outputs, evaluates binary-gated extensions, selects under agreement and size criteria, and tests the frozen extension on held-out prompts.

D.1 TASK, PREDICTION RULE, AND REFERENCE CIRCUIT

  • IOI predicts the recipient; ABBA prompts assign the first name as IO and the second as subject, while BABA reverses those roles.The repeated name is the subject in both prompt forms.Source: §D.1 · p. 18
  • Case-study predictions use argmax over fixed prompt-specific [IO, S] candidates, with ties assigned to IO.On model errors, a negative margin reproduces the particular wrong subject name.Source: §D.1 · p. 18
  • The reference circuit uses GPT-2 small and the published IOI head groups, instantiated as 26 nodes.The prompt base contains 60,000 prompts, split into 42,584 training prompts with 463 model errors and 17,416 validation prompts with 177 errors.Source: §D.1 · p. 18
  • The search space contains 3,574 supported additions across 144 heads and 25 position rules; Padded measures mean additional instantiated head outputs per prompt.Gradients screen promising head-position additions before exact evaluation; candidate gates are temporarily relaxed to [0, 1].Source: §D.1 · p. 19
  • Each search step evaluates top-scoring additions and grouped proposals with binary gates, selecting extensions to reward exact error reproduction while penalizing correct-agreement drops and extension size.Two searches use λP values of 0.0005 and 0.002, with at most 32 addition rounds and 128 added rules; removals are accepted only when they maintain or improve J(C).Source: §D.1 · p. 19
  • Validation keeps candidates within 0.005 of reference-circuit correct agreement, then selects a small extension retaining at least 90% of the largest error-agreement improvement.After three pruning sweeps, the extension had 63 added rules and was evaluated on 26,000 held-out prompts, including 225 model errors and 25,775 correct examples.Source: §D.1 · p. 19

Related papers

Loading 2609.35686v1…