PaperlayerView exact revision on arXiv

Source-linked AI summary

Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training

Adam Elimadi

arXiv:2609.35890v1cs.AI

TL;DR

The paper asks whether representational and attributional simplicity predicts a smaller causal circuit, a question not settled by proxy measures alone. It compares matched standard and adversarial GPT-2 Small models across representation, attribution, and circuit recovery. Representational simplicity improves with adversarial training, but circuit size depends on the faithfulness target.

  • Problem

    Whether representational or attributional simplicity predicts a smaller or more tractable causal circuit remains an empirical question, since models with the same behavior can implement it with mechanisms of different complexity.

    Source: §1 · pp. 1–2

  • Method

    The study compares matched standard and adversarially trained GPT-2 Small models, qualified for IOI competence and robustness, using SAE decomposability, task attribution, and faithful-circuit size.

    Source: §1 · p. 2, §2.3 · p. 3, §4 · p. 9

  • Results

    Circuit size is regime-dependent: standard leads or ties below 85% faithfulness, while robust needs substantially fewer edges at 90% and 95%; representational trends generalize across a seven-point sweep and a second corpus.

    Source: §1 · p. 2, §3.5 · p. 8, §4 · p. 9

  • Takeaways & Limitations

    Representational and attributional simplicity do not by themselves establish circuit simplicity; the circuit comparison changes with the faithfulness target.

    Source: §1 · p. 2, §4 · p. 9

    The causal result is established only on the primary pair, using one base architecture, one adversarial training regime, one residual-stream site, and a 200-prompt circuit-recovery sample.

    Source: §3.4 · p. 6, J REPRODUCIBILITY CHECKS AND LIMITATIONS · p. 20

Abstract

from arXiv · show

Sparse-autoencoder decomposability and concentrated feature attribution are increasingly treated as evidence that a model's computation is easier to reverse-engineer. Whether this representational and attributional cleanliness actually predicts a smaller or more tractable causal circuit remains an open question. We test this directly using adversarial training as a controlled instrument: it reliably reshapes internal representations, but this alone does not constitute a test of circuit size. We investigate this question through reverse-engineering complexity: the causal structure required to recover a model's behavior at a fixed level of faithfulness. To our knowledge, this is the first controlled empirical test of whether representational or attributional simplicity translates into causal simplicity at the circuit level. Starting from the same pretrained GPT-2 Small checkpoint, we apply matched standard and adversarial continual training, requiring both conditions to retain competence on indirect object identification and pass independent robustness verification before comparing mechanisms. We then compare the models along three complementary axes: sparse-autoencoder decomposability, SAE feature engagement in task attribution, and the size of faithful circuits recovered from the raw computational graph. The robust model is more SAE-decomposable and engages fewer SAE features in task attribution. Circuit size is regime-dependent: on competence-matched IOI, standard leads or ties below 85% faithfulness, but robust needs substantially fewer edges at high faithfulness (90%, 95%), a pattern established on the primary pair while representational trends generalize across a seven-point sweep and a second corpus.

1 INTRODUCTION

The paper tests whether representational and attributional simplicity predicts smaller causal circuits, using adversarial training while holding task behavior fixed. Its results show that the relationship depends on the faithfulness target.

Motivation

  • Whether simpler representations produce smaller or more tractable causal circuits remains an empirical question.Adversarial training offers a way to test this because it can reshape representations while task behavior is held fixed.Source: §1 · pp. 1–2

Contribution

  • The paper defines reverse-engineering complexity as the causal structure needed to recover behavior at a fixed faithfulness level.It distinguishes a reduction in this complexity—a smaller faithful circuit—from easier SAE reconstruction or more concentrated attribution.Source: §1 · p. 2
  • The study uses matched continual training, competence and robustness qualification, and three complementary measures to test whether representational simplicity corresponds to causal simplicity.The authors characterize this as the first controlled empirical test of the relationship at the circuit level.Source: §1 · p. 2

Contributions

  • At high IOI faithfulness, robust training yields substantially smaller circuits, although standard leads or ties below 85% faithfulness.The introduction reports that robust training needs substantially fewer top-k edges at 90% and 95% faithfulness, while also increasing SAE decomposability and reducing active attribution features.Source: §1 · p. 2

2 EXPERIMENTAL DESIGN

The design matches standard and adversarially trained GPT-2 Small models, qualifies them for task retention and robustness, then compares representation, attribution, and faithful-circuit size. Only circuit size directly measures reverse-engineering complexity.

2.1 OVERVIEW

  • The comparison uses matched GPT-2 Small models and three increasingly direct measures: SAE reconstruction, SAE feature engagement in task attribution, and faithful subgraph size.The first two measures concern representation and attribution; only circuit size directly measures reverse-engineering complexity.Source: §2.1 · p. 2

2.2 MATCHED CONTINUAL TRAINING

  • Both training arms start from the same checkpoint, token stream, and training budget; adversarial candidates vary attack strength and loss weighting.The models use 1B tokens and 1,908 updates; the adversarial setup applies a ten-step projected-gradient attack to token-embedding activations, with perturbations constrained to an ℓ2 ball.Source: §2.2 · pp. 2–3

2.3 TASKS AND QUALIFICATION CRITERIA

  • IOI is the primary competence-matched task, while Greater-Than is evaluated without a competence-matching qualification gate.IOI qualification requires at least 90% accuracy for the standard model and 85% for each adversarial candidate; robustness candidates are checked under single-step and ten-step attacks.Source: §2.1 · p. 2, §2.3 · p. 3

2.4 AXIS 1: SAE DECOMPOSABILITY

  • The robust SAE has lower held-out residual-space reconstruction error at fixed dictionary size and sparsity, indicating greater compressibility by this SAE family.Matched SAEs use the same architecture, sparsity target, data volume, optimizer, update count, and evaluation protocol.Source: §2.4 · p. 3

2.5 AXIS 2: SAE FEATURE ENGAGEMENT AND SELECTIVITY

  • Task attribution is measured over active SAE features at task-relevant positions, with feature pool size primary and attribution concentration secondary.The concentration statistic is reported alongside pool size because it is not scale-free with respect to pool size; the study also checks feature effects on the full next-token distribution.Source: §2.5 · p. 4

2.6 AXIS 3: FAITHFUL CIRCUIT RECOVERY

  • EAP-IG recovers circuits by retaining top-scoring edges and measuring faithfulness across edge counts, with faithful-circuit size defined by the first crossing of targets at or above 85%.The paper distinguishes this edge-count threshold measure from partial-subgraph AUC, and uses conditional co-ablation to test for redundancy and self-repair.Source: §2.6 · p. 4

3 RESULTS

The robust model is more SAE-decomposable and engages fewer attribution features, but circuit-size comparisons depend on the faithfulness target. On IOI, standard leads or ties below 85%, while robust requires fewer edges at 90% and 95%; representational trends persist across a sweep and a second corpus.

3.1 ADVERSARIAL CONTINUAL TRAINING YIELDS A QUALIFIED ROBUST COMPARISON

  • Among qualified candidates, ft-e100-a05 reduces ∆adv from 1.642 nats in the standard control to 0.362 while retaining 99.4% IOI accuracy.The selected robust model scores 0.442 on GT probability difference versus 0.671 for standard, with a paired prompt-level difference of −0.229±0.010.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 ROBUST ACTIVATIONS ARE MORE SPARSELY RECONSTRUCTABLE

  • At equal dictionary size and sparsity, robust SAE reconstruction has original-space MSE 1.1293 ± 0.0009 versus 1.1601 ± 0.0022 for standard, a 2.65% reduction.The same ordering holds under fixed-threshold inference and in normalized activation space.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 ROBUST TASK ATTRIBUTION ENGAGES FEWER SAE FEATURES, NOT MORE CONCENTRATED ONES

  • At 90% attribution mass, robust requires 276 IOI features versus 343 and 63 GT features versus 90, but normalized shares are larger for robust on both tasks.Fewer robust-SAE features are active at task-relevant positions: 1,196 versus 1,612 for IOI and 360 versus 563 for GT; the GT comparison is not competence-matched.Source: §3.3 · p. 5, Table 3 · p. 6

3.4 CIRCUIT SIZE ON IOI DEPENDS ON THE FAITHFULNESS TARGET

  • At 90% faithfulness, robust needs 1,000 edges versus 4,000 for standard; at 95%, it needs 2,000 versus 16,000.At 70% standard leads, and at 80–85% the models tie; below 85%, both subgraphs are partial rather than faithful circuits.Source: §3.4 · p. 6, Table 4 · p. 6

3.5 REPRESENTATIONAL SIMPLICITY TRACKS ROBUSTNESS ACROSS THE SWEEP

  • Across the seven-model sweep, SAE and α-ReQ simplicity increase with robustness while the 85% IOI edge crossing stays at or above the same grid point; representational trends also hold on FineWeb.The high-faithfulness circuit gap is reported for the primary pair only.Source: §3.5 · pp. 7–8, Table 5 · p. 8, Figure 3 · p. 8

4 CONCLUSION

The paper compares matched standard and adversarially trained GPT-2 Small models across representational, attributional, and causal measures. Robust training improves SAE decomposability and reduces active attribution features, but circuit-size rankings depend on the faithfulness target.

4 CONCLUSION

  • The study compares matched standard and adversarially trained GPT-2 Small models on SAE decomposability, SAE feature engagement, and faithful circuit size.The authors frame reverse-engineering complexity as the causal structure needed to recover behavior at fixed faithfulness.Source: §4 · p. 9
  • On IOI, standard leads or ties below 85% faithfulness, while the robust model needs substantially fewer edges at 90% and 95%.The comparison uses faithful circuits recovered from the raw computational graph.Source: §4 · p. 9
  • The robust model is more SAE-decomposable and engages fewer active SAE features, though not more concentrated ones after normalization by pool size.The representational trend generalizes across a seven-point adversarial-strength sweep and a second corpus; the causal result is established only on the primary pair.Source: §4 · p. 9

5 RELATED WORK

Prior work connects robustness with changes in representations and attributions, and mechanistic interpretability has developed methods for recovering task circuits. This paper distinguishes those lines of evidence from a direct comparison of causal circuit size.

5 RELATED WORK

  • Robustness research links adversarial training to altered features, more human-aligned gradients and saliency, sparse attributions, and feature purification, but not directly to smaller causal graphs.The paper tests that latter prediction rather than inferring it from representational or attributional proxies.Source: §5 · p. 9
  • Superposition and SAE work motivates measuring decomposition separately, since reconstruction and sparsity alone do not guarantee causal features.Circuit-recovery results also depend on attribution methods, thresholds, and pruning, while ablations may trigger self-repair.Source: §5 · p. 9
  • Prior adversarial mechanistic-interpretability studies localize affected components in fixed GPT-2 Small circuits or compare language settings with EAP-IG circuits.Bereska (2024) proposed reverse-engineering robust models, but the cited proposal was not executed empirically.Source: §5 · p. 9
  • The paper positions its contribution as comparing matched standard and adversarially trained language models on SAE measures and established task-circuit size.Appendix A discusses SAE evaluation, circuit-discovery variants, and self-repair in greater detail.Source: §5 · pp. 9–10

AI USE DISCLOSURE

The disclosure describes Claude’s assistance with manuscript preparation while assigning experimental design, training, analysis, and interpretation to the authors.

AI USE DISCLOSURE

  • Claude assisted with proofreading, polishing, and structuring the paper’s sections.The disclosure characterizes these activities as writing and polishing assistance.Source: AI USE DISCLOSURE · p. 10
  • Claude assisted with verifying citations against primary sources and identifying missing bibliography entries.Source: AI USE DISCLOSURE · p. 10
  • Claude assisted with LaTeX formatting, table and figure layout, and troubleshooting page-length and float-placement issues.Source: AI USE DISCLOSURE · p. 10
  • The authors carried out experimental design, model training, data analysis, and result interpretation, and reviewed and approved all claims, figures, and interpretations.The authors state that they take full responsibility for the submission’s content and accuracy.Source: AI USE DISCLOSURE · p. 10

A EXTENDED RELATED WORK AND METHODOLOGICAL CONTEXT

The extended context reviews robustness, SAE evaluation, circuit discovery, and self-repair, then describes the study’s training and evaluation setup. Its behavioral qualification results show a graded GT trade-off not exposed by IOI accuracy.

A EXTENDED RELATED WORK AND METHODOLOGICAL CONTEXT

  • Prior work connects robustness with human-aligned gradients and representations, while SAE benchmarks and circuit methods have limitations that motivate separate measurement of representations and causal structure.Circuit recovery depends on choices including graph granularity, patching baseline, task metric, estimator, threshold, and search procedure; interventions can also induce self-repair.Source: A EXTENDED RELATED WORK AND… · p. 14

B.1 INITIALIZATION AND OPTIMIZATION

  • All runs use the same GPT-2 Small checkpoint, update all parameters, and receive the same examples in the same order; Table 7 summarizes shared hyperparameters.The supplied passage gives the checkpoint and data-stream controls, while the table caption identifies its scope.Source: §B.1 · p. 15, Table 7 · p. 15

B.2 WHY CONTINUAL TRAINING IS USED

  • Continual training was chosen because the from-scratch standard model reached 55.4% IOI accuracy versus 99.7% for the official checkpoint on the same prompts.A one-step embedding-adversarial variant had loss 1.20 under its training-time attack and 24.70 under PGD-10, motivating multi-step PGD training and verification.Source: §B.2 · p. 15

C EVALUATION DATA AND PROVENANCE

  • The study separates model-training, robustness-evaluation, SAE-training, SAE-evaluation, and generic-language-modeling attribution-control partitions in the OpenWebText token stream.The IOI and GT sets each contain 1,000 prompts; the GT counterfactual parity filter retains 973 prompts, and Table 8 summarizes the partitions.Source: C EVALUATION DATA AND PROVENANCE · p. 16, Table 8 · p. 16

D QUALIFICATION SWEEP AND BEHAVIORAL TRADE-OFF

  • GT paired prompt-level differences from the standard model range from −0.063 ± 0.005 to −0.229 ± 0.010 across the six listed sweep cells, revealing a trade-off not exposed by IOI accuracy.The original checkpoint scores 0.706, the continually trained standard model 0.671, and the selected robust model 0.442 on this prompt set; the paired within-set difference is the primary comparison.Source: D QUALIFICATION SWEEP AND BEHAVIORAL… · p. 16

E SPARSE-AUTOENCODER IMPLEMENTATION AND STATISTICS

The section reports SAE reconstruction and attribution-feature comparisons alongside training and evaluation controls. The robust SAE uses fewer candidates and features to reach the 90% attribution-mass threshold, while generic-language-modeling attributions are too small to support a useful selectivity ratio.

E SPARSE-AUTOENCODER IMPLEMENTATION AND STATISTICS

  • Each SAE is trained for 73,242 updates on 300M tokens, with no early stopping.Training uses batches of 4,096 tokens, Adam at learning rate 4×10^-4, and a roughly 500,000-token activation buffer.Source: E SPARSE-AUTOENCODER IMPLEMENTATION… · p. 16
  • Downstream inference replaces batch-relative BatchTopK selection with a fixed threshold averaged over held-out batches, so one prompt's activation state does not depend on other prompts.At seed 42, thresholds are 0.785 for the standard SAE and 0.741 for the robust SAE.Source: E SPARSE-AUTOENCODER IMPLEMENTATION… · pp. 16–17
  • The robust SAE requires fewer candidates and fewer features at the 90% attribution-mass threshold at every seed on both tasks.Reconstruction statistics are reported across three SAE seeds, with paired residual-position uncertainty estimates.Source: E SPARSE-AUTOENCODER IMPLEMENTATION… · p. 17, Table 10 · p. 17

F ATTRIBUTION IMPLEMENTATION AND SELECTIVITY CONTROLS

  • Attribution candidates are ranked separately for each model and task, with the ablation pool including ties at the top-100 boundary.Task attribution sums over task-specific intervention positions; a feature qualifies if active at any such position on a clean or corrupted prompt.Source: F ATTRIBUTION IMPLEMENTATION AND… · p. 17

F.1 NON-DISCRIMINATIVE GENERIC-LANGUAGE-MODELING CONTROL

  • Generic next-token attributions on held-out OpenWebText are approximately 10^-8, versus task attributions of approximately 10^-2, making the control selectivity ratio uninformative.The denominator is negligible in both models.Source: §F.1 · p. 17

F.2 ABLATION-KL POPULATION COMPARISON

The ablation-KL comparison uses real feature interventions and compares the resulting effects as unpaired populations because independently trained SAEs have no feature-index correspondence. Medians are close, with separation concentrated in the 90th-percentile tail, so the authors report only a directional pattern.

F.2 ABLATION-KL POPULATION COMPARISON

  • Features are zero-ablated and the change in output-distribution KL is measured by intervention, not a linearized attribution estimate.The linearized estimate is identically zero because KL has zero first derivative at the unablated distribution; IOI features are ablated where active across the task's intervention positions.Source: §F.2 · p. 17
  • Ablation-KL medians are close between models on both tasks, with separation concentrated in the 90th-percentile tail.Across three seeds, per-seed means are not separated by standard deviation, and one IOI seed reverses direction.Source: §F.2 · p. 18, Table 11 · p. 18
  • The comparison treats 100 ablation-KL values per model and task as unpaired populations because independently trained SAE feature indices do not correspond.Values exclude control-set-dead features, whose ablation is a no-op; dead counts are reported separately per cell.Source: §F.2 · p. 18

G CIRCUIT RECOVERY PROTOCOL

The circuit-recovery protocol scores raw-graph edges using EAP-IG and evaluates faithfulness over edge budgets, with EAP-GP included as a pilot. It also reports α-ReQ as an SAE-independent representation measure and notes agreement with a second spectrum summary, while weight-spectrum alpha stays flat across the sweep.

G CIRCUIT RECOVERY PROTOCOL

  • Circuit recovery scores all 32,491 edges in a 156-node raw graph on identical prompt sets and counterfactuals, using EAP-IG and a pilot EAP-GP comparison.Circuit size is measured at the smallest tested edge count reaching 85% of full-graph faithfulness, with faithfulness-curve area under log k as a finer-grained measure.Source: G CIRCUIT RECOVERY PROTOCOL · p. 18
  • At k = 128 on IOI, standard faithfulness is 0.13 versus −0.09 for robust, the largest standard lead in the curve, but both remain below the 85% threshold.Several entries below k = 1000 are negative, meaning the subgraph performs worse than the fully corrupted baseline.Source: G CIRCUIT RECOVERY PROTOCOL · p. 18, Table 12 · p. 18
  • GT-standard first reaches 95% at k = 8000; k = 2000 and k = 4000 fall short, and GT is not competence-matched between models.At k = 2000 faithfulness is 0.94 and at k = 4000 it is 0.87; the latter is non-monotonic.Source: G CIRCUIT RECOVERY PROTOCOL · p. 18, Table 12 · p. 18, Figure 4 · p. 19
  • EAP-GP has negative IOI AUC for both models, whereas EAP-IG has positive AUC, so the authors use EAP-IG for the headline result.The figure captions also state that EAP-GP has negative IOI AUC for both models and EAP-IG is used on that basis.Source: G CIRCUIT RECOVERY PROTOCOL · p. 19, Figure 5 · p. 19
  • A connectivity-preserving greedy edge selector leaves EAP-IG ahead of EAP-GP on both tasks and models, so top-k disconnection does not explain EAP-GP's underperformance.The greedy builder improves both methods' AUC, with EAP-IG improving more on IOI.Source: G CIRCUIT RECOVERY PROTOCOL · pp. 19–20
  • Conditional co-ablation recovers backup components after primary-circuit ablation, but node-level faithfulness remains far below 85% for both models and is not a second 90%/95% edge-level test.The backup-recovery and validity-gate forward-pass costs are fixed by construction and do not discriminate between conditions.Source: G CIRCUIT RECOVERY PROTOCOL · p. 20

H α-REQ: REPRESENTATION EIGENSPECTRUM DECAY

  • α-ReQ measures representation simplicity independently of trained SAEs, and agreement with a second eigenvalue-spectrum summary supports that its trend is not an artifact of the fit range.HT-SR weight-spectrum alpha stays flat across the sweep at 2.923–2.934 and is not used further.Source: H α-REQ: REPRESENTATION EIGENSPECTRUM… · p. 20

I CORPUS ABLATION: FINEWEB

The FineWeb corpus-ablation pair uses the primary pair's continual-training recipe, budget, and qualification gate, substituting FineWeb for OpenWebText as the training and SAE corpus.

I CORPUS ABLATION: FINEWEB

  • The FineWeb pair matches the primary pair's recipe, budget, SAE architecture, and evaluation protocol, and both arms pass the IOI qualification floor and masking sanity check.FineWeb replaces OpenWebText as both the training and SAE corpus.Source: I CORPUS ABLATION: FINEWEB · p. 20

J REPRODUCIBILITY CHECKS AND LIMITATIONS

The study documents its randomization, seed replication, and implementation checks, while limiting its conclusions by architecture, training-regime, measurement-site, and circuit-sample scope. Greater-Than circuit size is not treated as a competence-matched test.

Loading 2609.35890v1…