PaperlayerView exact revision on arXiv

Source-linked AI summary

Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit

Sai Adith Senthil Kumar

arXiv:2608.27254v1cs.LG

TL;DR

Large fixed-weight circuit searches can produce graphs too large for exhaustive inspection and verification. Circuit Condensation post-trains models by pruning edges and healing a low-rank adapter to preserve the original outputs under task and capability constraints. Across four behaviors and eight models, it produces smaller circuits than frozen baselines in 30 of 32 settings, while weight-update ablations show the reduction comes from reshaping rather than search alone.

  • Problem

    Frozen circuit discovery often returns graphs too large to test exhaustively, limiting checks of minimality and edge dependencies.

  • Method

    Circuit Condensation iteratively prunes low-attribution edges and trains only a low-rank adapter to match the original model through the surviving graph, accepting cuts under held-out accuracy and capability constraints.

  • Results

    30 of 32 task–model combinations have smaller C1 circuits than frozen EAP-IG at comparable accuracy, with an average reduction of 8.1× and a maximum of 316×.

  • Takeaways & Limitations

    The condensed circuits make exhaustive subset and all-pairs tests feasible and provide sufficient sub-circuits of studied mechanisms rather than reconstructions of them.

  • Takeaways & Limitations

    Circuit size counts retained edges under interchange ablation, not computation inside retained components or in the model-wide adapter.

Abstract

from arXiv · show

One approach to mechanistic interpretability explains behavior through circuits: the components and connections that carry it. Frozen discovery often returns hundreds of edges, making them hard to inspect, compare, or verify exhaustively. We introduce Circuit Condensation, which post-trains models to concentrate behaviors into smaller causal graphs. Each round prunes low-attribution edges and trains a low-rank adapter to match the original through what remains, retaining the cut only if task performance and general capability survive. Across four behaviors and eight models, condensed circuits are smaller than the strongest frozen baseline in 30 of 32 settings, by $8.1\times$ on average and up to $316\times$. Repeating the search without weight updates produces larger circuits in 29 of 32 settings, showing that weight updates, rather than search alone, drive the reduction. Testing every subset of 19 circuits finds 11 that cannot be reduced and reveals removable edges in the rest. Pair ablations expose dependencies between edges, showing that their effects cannot be understood independently. On indirect object identification, condensation isolates 24 heads, 17 of them with documented roles, against 61 heads and 36 undocumented ones for the matched frozen circuit: a sufficient sub-circuit of the published mechanism rather than a reconstruction of it. The resulting circuit tracks the original model's next-token distribution and predicts its errors.

1 Introduction

Circuit Condensation reframes circuit findability as a trainable property: post-training concentrates a behavior into a smaller causal graph while preserving task performance and general language ability. This reduction enables exhaustive circuit verification.

  • Why size matters: Smaller circuits make exhaustive subset and pairwise dependency tests practical, addressing the combinatorial burden of interpreting large recovered graphs.Faithfulness alone does not establish that a circuit is minimal or that its edges act independently.
  • Circuit Condensation: Circuit Condensation prunes weak edges, trains a low-rank adapter to reproduce the original model through the remainder, and restores failed cuts.Acceptance requires both target behavior and general language ability to survive.
  • Motivation: Weight updates, rather than search alone, drive the reduction, while the resulting circuits remain connected to the original model.The method is designed to reproduce the original model’s outputs rather than explain only the adapted network.
  • Results: 30 of 32 cells yield smaller C1 circuits than frozen EAP-IG at comparable accuracy, averaging 8.1× smaller and reaching 316×.The contribution summary reports the comparison across four behaviors, eight models, and three seeds.

2 Related Work

Prior circuit-discovery methods search fixed model weights for faithful sparse structure, but recovered circuits vary with evaluation choices and often resist exhaustive analysis. Circuit Condensation instead trains models to make behaviorally specific structure more concentrated.

  • Circuit discovery: Circuit discovery methods use interventions, gradients, learned gates, and feature decompositions to search for sparse causal structure while keeping model weights fixed.The comparison class asks how small a faithful circuit already exists in the fixed model.
  • Comparison frame: Figure 2 compares C1 with frozen baselines across behaviors and models using retained graph share on a logarithmic axis.Stars denote C1; other markers denote frozen baselines, with seed means and ranges shown.
  • What discovery returns: Recovered circuits can change with the ablation rule, metric, data, and estimator, and multiple faithful graphs may describe the same behavior.Prior work also reports limited transfer across datasets, granularities, and model retraining.
  • Training for interpretable structure: Training can change how behavior is distributed, motivating methods that encourage sparse or modular computation and post-train simpler task circuits.The paper’s approach is closest to post-training sparse attention connectivity, but its constraint is behavior-specific.

3 Method: Circuit Condensation

Circuit Condensation greedily searches for a small faithful graph while allowing a low-rank adapter to reshape computation. It ranks, prunes, heals, and accepts or restores cuts under task-accuracy and capability constraints.

  • Problem setup: The transformer is represented as a directed graph whose edges connect residual-stream writers to downstream readers, and paired clean-corrupted inputs specify each behavior.GPT-2 has 2,041 candidate edges, while Qwen3-4B has 43,993.
  • Problem setup: The objective minimizes retained-edge count subject to task accuracy remaining within ε = 0.05 of full performance and off-target perplexity staying within κ = 0.05.Retained edges are counted by ∥z∥0, measuring graph width rather than computation inside components.
  • Prune and heal: Each round ranks active edges, proposes pruning the lowest-scoring 30%, heals the adapter, and halves failed cuts down to 5%.EAP-IG supplies the per-edge causal-importance ranking.
  • Prune and heal: Healing freezes base weights and trains only the adapter by minimizing KL divergence between the original and masked models’ output distributions.Self-distillation preserves output behavior instead of allowing a task loss to relearn the answer through surviving paths.
  • Accept or restore: A proposed state is accepted only when both the masked circuit and gates-open adapted model satisfy accuracy, with the latter also passing the held-out capability gate.Otherwise, both states are restored and a gentler cut is tried.

4 Experimental Setup

The experiments evaluate four paired-input token behaviors across eight models, using disjoint data splits and deterministic accuracy, distributional KL, and frozen-baseline comparisons. Circuit sizes are selected at matched validation accuracy and reported across three seeds.

  • Tasks and models: The study covers IOI, subject–verb agreement, repeated-token induction, and Python docstring completion across GPT-2, Llama, Gemma, and Qwen models.Each attention head is a separate component and each MLP is one component.
  • Metrics: Task performance uses restricted-choice accuracy at one answer position, while token-level KL divergence compares full next-token distributions.Separate checks compare circuits with the original model and record candidate-choice agreement.
  • Baselines: Frozen comparisons include EAP, EAP-IG, EAP-GP, random ranking, C0, and tractable ACDC cells, with C0 isolating post-training from search.C0 uses the same ranking, gate, stopping rule, and controller without healing.
  • Data splits: Each task–model combination uses 2,000 training, 2,000 validation, and 1,000 sealed test examples, with ranking and healing restricted to training data.The controller evaluates proposals on validation and reads the test split once after selection.
  • Selection and reporting: Every arm selects its smallest circuit within 0.05 validation accuracy of its own fully open graph, reporting medians over seeds 11, 22, and 33.Statistical tests treat the 32 task–model combinations, not 96 seeded runs, as independent units.

5 Results

Across four behaviors and eight models, Circuit Condensation produces substantially smaller circuits than frozen discovery while preserving performance and enabling stronger causal checks. Weight updates, combined with attribution-guided pruning, drive the reduction, though the condensed circuits still contain interactions and sometimes removable edges.

  • 5.1 Condensation Goes Below the Frozen Discovery Floor: 30 of 32 task–model combinations have smaller C1 circuits than frozen EAP-IG at comparable accuracy, with an average 8.1× reduction and a maximum of 316×.Against EAP, the average reduction is 18.4×.
  • 5.1 Condensation Goes Below the Frozen Discovery Floor: 91 of 96 C1 endpoints meet the 0.05 tolerance on the sealed test split; the other five miss by a median 0.009.The held-out test results show that validation-selected endpoints generally retain their target behavior.
  • 5.2 The Gain Comes From Reshaping, Not a Better Search: C0 is larger than C1 in 29 of 32 cells by an average 7.4× under the same search, showing that weight updates drive the reduction beyond search alone.Ordinary LoRA fine-tuning without the condensation loop still leaves circuits 4.4× larger than C1.
  • 5.3 The Circuit Is Small Enough to Verify: On IOI, removing C1 lowers median accuracy from 0.99 to 0.52, while removing the same number of random edges leaves accuracy at 0.98.The corresponding intact/circuit-cut/random values are 0.74/0.22/0.62 for agreement, 1.00/0.00/0.96 for induction, and 1.00/0.00/0.97 for docstring.
  • 5.3 The Circuit Is Small Enough to Verify: Condensation returns 24 IOI heads, including 17 with documented roles, versus 61 heads with 36 unaccounted for in frozen discovery.It retains all three primary Name Movers but drops three of eight Backup Name Movers.
  • 5.3 The Circuit Is Small Enough to Verify: Across 169,476 pair ablations, independence fails in every tested circuit, with a median of 7.4 interacting partners per edge.The matched frozen circuits would require a median 55× as many pair ablations.
  • 5.4 The Circuit Describes the Model We Started From: C1 matches the unmodified model’s answer on a median 96.5% of sealed-test examples and predicts its errors better than matched frozen circuits in 10 of 11 scored cells.Across all runs, the median token-level KL divergence is 0.215; on the IOI/GPT-2 anchor it is 0.056 versus 0.50 for the frozen circuit.

6 Limitations

The authors bound Circuit Condensation’s claims by noting that circuit size concerns retained edges in the adapted network, not all computation, and that the controller is not always minimal. Several evaluation and generalization constraints further limit interpretation.

  • Scope and measurement: Circuit size counts retained edges under interchange ablation in the adapted network, excluding computation inside retained components and the model-wide adapter.Thus, a small edge graph does not measure the full computational footprint.
  • Minimality: The controller does not always produce the smallest circuit: a separately tuned training-gates-and-adapter arm is smaller in 5 of 32 cells.Because that arm is separately tuned and test-qualified, five is an upper bound on how often it wins.
  • Evaluation constraints: The capability gate almost never binds on Gemma but rejects 12–29% of rounds on other model families, weakening comparability across the grid.Gemma’s smallest circuits are therefore held to a weaker constraint than the rest.
  • Evaluation constraints: Docstring and agreement provide no accuracy signal for stopping, so their endpoint sizes are upper bounds rather than measured minima.This limits minimality claims for those behaviors.
  • Generalization: Three seeds assess optimization stability, not uniqueness across model refits, input distributions, or graph granularities.The reported stability evidence does not establish invariance under those changes.

7 Conclusion

Circuit Condensation makes circuits smaller enough for inspection and exhaustive testing by changing how behavior is distributed through post-training. Its controller prunes, heals, validates, and backtracks while preserving task performance and general capability.

  • 7 Conclusion: Circuit Condensation changes behavior distribution through post-training, unlike fixed-model search, to produce circuits small enough to inspect and test.Some runs stop when further pruning damages general language ability.
  • 7 Conclusion: The procedure makes exhaustive subset and all-pairs tests feasible where matched frozen circuits do not.This directly addresses the difficulty of testing whether circuits contain unnecessary components and whether retained edges interact.
  • 7 Conclusion: The controller ranks active edges, proposes cuts, heals the adapter, checks faithfulness and capability, and backtracks after failed cuts.The reported endpoint is the smallest accepted circuit; overshoot rounds only visualize the post-knee cliff.

C Selection Protocol and Data Provenance

The selection protocol compares arms using fixed, traceable configurations and explicit data splits. Circuits are selected as the smallest graphs meeting each arm’s validation criterion, then re-evaluated on sealed test data.

  • C Selection Protocol and Data Provenance: Table 3’s controller and run configuration is shared across all 96 runs and 32 cells unless a control explicitly changes it.The protocol records shared schedules and thresholds rather than tuning them per cell.
  • C Selection Protocol and Data Provenance: Each arm selects its smallest circuit within 0.05 of its own full-graph validation accuracy, yielding a reduction ratio for every run.Table 4 summarizes geometric-mean reductions across 24 runs per task.
  • C Selection Protocol and Data Provenance: EAP, EAP-IG, and EAP-GP are implemented on the controller’s edge gates, while EAP-IG rankings correlate with exact single-edge activation-patching effects.The reported Spearman correlation is 0.545 for EAP-IG versus −0.055 for random ranking.
  • C Selection Protocol and Data Provenance: C1’s adapted full graph has the same median validation accuracy as the original full graph, with a difference of 0.000.The adapted graph is lower in 42 of 96 runs and higher in 24, so the adapted reference does not provide an easier target in practice.
  • C Selection Protocol and Data Provenance: Primary circuit sizes are selected under one rule, while selected circuits are re-evaluated on sealed-test data with documented provenance and exact denominators.Frozen rankings use training data, and sealed-test examples do not affect frozen selection.

D Headline Statistics and Per-Task Reductions

Across task–model combinations, circuit-size reductions are summarized using geometric means and compared against the strongest available frozen alternatives. Additional evaluations report sealed-test faithfulness and general-capability retention.

  • D Headline Statistics and Per-Task Reductions: 32 task–model combinations are the analysis unit, with each summarized by its median over three seeds; reductions use geometric means.The arithmetic mean of the ratios is 29.6× with interval [11.7, 52.6…], whereas the geometric mean is the reported estimator for multiplicative reductions.
  • D Headline Statistics and Per-Task Reductions: C1 is compared with the smallest frozen alternative in each cell among EAP, EAP-IG, random ranking, ACDC, and C0.ACDC is evaluated only on eight tractable cells because it does not scale to the largest graphs.
  • D Headline Statistics and Per-Task Reductions: C1 is 22.7× smaller than C0 on IOI, 4.5× on agreement, 12.4× on induction, and 2.4× on docstring.Removing Gemma models leaves a 5.8× reduction, while restricting to IOI and induction gives 18.0× across all 16 cells.
  • D Headline Statistics and Per-Task Reductions: Agreement’s condensed circuit has median sealed-test accuracy 0.777 versus 0.764 for its own full graph.Interchange ablation can remove distractor pathways and sharpen a margin rather than only degrade accuracy.
  • D Headline Statistics and Per-Task Reductions: General-capability evaluation reports WikiText-2 perplexity ratio, per-token KL divergence, and LAMBADA last-token accuracy change against the unmodified model.The capability-gate and evaluation slices are disjoint.

F Control Protocols and Per-Cell Results

Control experiments separate condensation’s effects from healing, ranking, target choice, and conduit identity. They show that causal ranking and post-training contribute to concentration, while retained heads remain load-bearing and specific.

  • F Control Protocols and Per-Cell Results: Healing without pruning leaves attribution spread essentially unchanged, with a median end-to-start participation-ratio ratio of 0.99 across 32 cells.92 of 96 seed–cells remain within 15% of their starting value; four IOI/Qwen3 cells concentrate mildly.
  • F Control Protocols and Per-Cell Results: Random pruning stalls at a circuit a median 50× larger than the EAP-IG endpoint, while anti-ranking removes no edges net.Both controls use the same controller and differ only in ranking.
  • F Control Protocols and Per-Cell Results: Targeted behaviors become easier to localize, while all twelve off-target direction medians remain 1.0× across 288 measurements.The targeted localization factors are 8× for induction, 4× for agreement, and 2× for docstring.
  • F Control Protocols and Per-Cell Results: Switching every edge off leaves the adapter at chance, and swapping each head-to-logits edge to a same-layer sibling collapses six of seven tested cells near the empty-circuit baseline.Adding one random edge changes nothing to three decimal places, supporting head-specificity of the retained conduits.
  • F Control Protocols and Per-Cell Results: The joint gates-and-adapter training arm removes the prune-and-heal loop and instead trains both under an L0 sparsity penalty for 3000 steps.Its only difference from the frozen learned-gate arm is that the base weights are unfrozen.

G.1 Total Compute and Break-Even

Condensation adds upfront computation but makes otherwise infeasible circuit checks practical and preserves substantial alignment with the original model.

  • Total Compute and Break-Even: 3.6× lower exact single-edge patching cost follows from evaluating fewer candidates after condensation.This saving excludes the cost of producing the condensed model.
  • Total Compute and Break-Even: The upfront expense is recouped only after many analyses per cell, so the main benefit is feasibility rather than throughput.Exhaustive subset and all-pairs checks become possible on smaller circuits.
  • Legibility and Original-Model Checks: Condensed circuits contain a median 24 heads versus 61 for matched frozen IOI circuits, while retaining 17 published roles and covering every IOI stage except previous-token.The condensed circuit is more precise but less complete than the published mechanism.
  • Legibility and Original-Model Checks: At equal condensed edge count, the frozen circuit reaches chance in two of three seeds, while the condensed circuit leads 0.965 to 0.845 in the remaining seed.Interchange intervention accuracy declines as circuits grow, creating a size confound.
  • Original-Model Faithfulness: Median normalized logit-difference recovery is 0.95 across 32 block-level cells and the head-level IOI anchor without retraining.Values above 1 indicate sharper recovered margins than the full model under the stated metric.
  • Original-Model Faithfulness: The head-level grid matches the unmodified model’s task choice on a median 96.5% of sealed-test examples, with median token-level KL 0.215.On the IOI/GPT-2 anchor, KL to the original model is 0.056.

I Competence-Signal Readout

Condensed circuits make competence signals and inspection-scale comparisons clearer, while their compactness remains sensitive to task, model family, and ablation convention.

  • Competence-Signal Readout: Median AUROC is 0.894 for C1 versus 0.645 for the matched frozen circuit, with C1 ahead in 10 of 11 cells.The probe ranks the full model’s own errors using restricted-choice margins read through each circuit.
  • Competence-Signal Readout: At 70% coverage, selective prediction reaches 0.882 accuracy for C1 versus 0.804 for the matched frozen circuit, from a base of 0.781.Declining the least-confident inputs produces the improvement.
  • Fixed-Budget Legibility: With only 20 inspected heads, the condensed IOI circuit reaches 0.94 accuracy while the frozen circuit remains at chance and needs 60 heads to match it.Induction shows the same sufficiency pattern: 0.955 versus 0.60 at a matched 66-head budget.
  • Scaling: A 22-fold increase in candidate-graph size from GPT-2 to Qwen3-4B corresponds to a log-log condensed-size slope of −0.06 rather than proportional growth.The pooled Spearman correlation is −0.14 with p = 0.51, so the fit is descriptive.
  • Scaling: The retained graph share falls from a median 7.5% in GPT-2 to 0.37% in Qwen3-4B, although within-family endpoint trends vary by task and model.For example, Qwen3 agreement endpoints shrink from 1,054 to 205 to 190 edges.
  • Pairwise Interactions: 87% of the strongest pairwise interactions are sub-additive, while the interacting fraction correlates −0.89 with circuit size at fixed tolerance.The authors attribute part of the interaction pattern to ceiling effects and absolute-threshold sensitivity.

M Limitations in Detail

The paper’s compactness claims have a narrow operational meaning and vary with evaluation conventions, capability constraints, stopping signals, and experimental choices.

  • Operational Scope: Circuit size counts retained edges under interchange ablation, excluding computation inside retained components and the model-wide adapter.Under other ablation rules, both condensed and frozen circuits can fail.
  • Operational Scope: Per-task medians span 28 nodes and 38 edges on IOI to 150 nodes and 497 edges on docstring, with adapters excluded from both counts.Three-edge Gemma-3-1B endpoints span five nodes.
  • Baselines: Joint gate-and-adapter training is smaller in 5 of 32 cells but fails outright in 20, so the controller is preferred for admissibility without per-cell tuning.The controller is not always the smallest method.
  • Capability Constraints: The capability gate rejects 12–29% of rounds on three model families but almost never binds on Gemma, weakening the constraint on Gemma’s smallest circuits.The authors have not tested per-model calibration.
  • Stopping Criteria: Docstring and agreement provide no reliable stopping accuracy signal, making their endpoint sizes upper bounds rather than measured minima.One twelve-edge endpoint even contains a faithful three-edge subset.
  • Additional Payoffs: Condensation’s additional tested payoffs fail: retraining after deletion restores behavior about as quickly as matched frozen circuits, and upfront cost is repaid only across repeated analyses.Activation-norm monitoring remains confounded by surface differences between inputs.
  • Reproducibility Scope: Three seeds assess optimization stability, not uniqueness; agreement across seeds does not establish invariance to refits, input distributions, graph granularity, or pruning procedures.Those choices are explicitly left untested here.
Loading 2608.27254v1…