Interpretability at Scale: Identifying Causal Mechanisms in Alpaca

1 Introduction

• Explaining internal processes in human-interpretable terms is presented as an initial step toward auditing LLMs for safety, trustworthiness, and pernicious social biases.The motivation is the models’ growing presence in products, services, and daily tasks.Source: §1 · p. 1
• Causal abstraction can formally assess whether a neural network implements an interpretable causal system, but prior methods had been applied only to small, task-specific fine-tuned models.The alignment space grows exponentially with model size, making search harder; failure to find an alignment may reflect the search algorithm rather than absent structure.Source: §1 · pp. 1–2
• DAS learns alignments between distributed neural representations and causal variables with gradient descent, but its brute-force search over representation dimensions hinders scaling.Boundless DAS addresses this remaining search burden.Source: §1 · p. 2
• Boundless DAS replaces DAS’s remaining brute-force search with learned parameters and reveals interpretable causal structure in off-the-shelf Alpaca (7B).The authors report that Alpaca solves a simple numerical reasoning task with an interpretable algorithm used across varied contexts and task variations.Source: §1 · p. 2

2 Related Work

• Many interpretability approaches use observed behavior or neural representations, but such explanations are not guaranteed to be faithful to a model’s underlying causal mechanisms.The cited approaches include weight analysis, gradient methods, probing, syntax-driven interventions, self-generated explanations, and external explainers.Source: §2 · p. 2
• Causal abstraction unifies methods for uncovering interpretable mechanisms and grounds mechanistic interpretability, but the authors know of no prior work scaling these methods to large, general-purpose LLMs.The framework also represents approaches such as nullspace projection, causal mediation analysis, and causal effect estimation.Source: §2 · p. 2
• Instruction tuning and task-based fine-tuning support instruction following, and self-instruct uses a target LLM to partly create fine-tuning data.The authors’ goal is to scale causal abstraction methods to understand how instruction-following models respond to a particular instruction.Source: §2 · pp. 2–3

3.1 Background on Causal Abstraction

• Causal models assign values to variables through causal mechanisms; interventions edit mechanisms, and model outputs are evaluated for an input.An interchange intervention fixes target variables to the values they would take for source inputs, while distributed interventions implement this through aligned neural subspaces.Source: §3.1 · p. 3
• An algorithm is a faithful interpretation of a network when its causal mechanisms abstract the network’s mechanisms under an alignment from each high-level variable to a rotated neural subspace.IIA measures the proportion of aligned interchange interventions for which the algorithm and network have the same output.Source: §3.1 · p. 3

3.2 Boundless Distributed Alignment Search

• DAS learns an alignment between interpretable causal variables and fixed-dimensional linear subspaces of neural representations using gradient descent.It trains an orthogonal matrix to maximize interchange intervention accuracy (IIA) under the alignment.Source: §3.2 · p. 4
• Boundless DAS learns continuous boundary indices and masks to determine the dimensionality of each aligned subspace, replacing a manual search over possible representation widths.The method anneals the mask temperature; as β approaches 0, masks converge to binary-valued vectors encoding an orthogonal decomposition.Source: §3.2 · p. 4
• Boundless DAS uses annealed masks for weighted interchange interventions, then snaps them to binary values for evaluation and an orthogonal decomposition of the representation.The binary masks identify each variable’s aligned subspace and the residual unaligned subspace.Source: §3.2 · p. 4
• DAS requires manual selection of how many neurons represent a variable, whereas Boundless DAS learns the boundaries and avoids a width-by-width search.For n representation dimensions and m causal variables, the passage gives complexities O(n × m) for DAS and O(m) for Boundless DAS.Source: §3.2 · p. 4

4.1 Price Tagging

• The price-tagging task asked Alpaca to answer Yes only when an amount fell between two randomly drawn dollar bounds, with bound gaps restricted to [2.50, 7.50].The study focused on tasks with high model performance so there would be a behavioral pattern to explain.Source: §4.1 · p. 5

4.2 Hypothesized Causal Models

• The four candidate causal models used one or two boundary-check booleans, midpoint distance, or a single interval variable to predict whether an amount qualified.Each proposed high-level model solved the task perfectly.Source: §4.2 · pp. 5–6, Figure 2 · p. 5

4.3 Boundless DAS Results

• At or above 0.85 IIA, the two boundary-check models matched Alpaca’s task performance, while midpoint-distance and bracket-identity models peaked around 0.72.The boundary-check alignments showed structured IIA patterns; the variables were extracted around position 75 in layer 10 and later copied into final-token activations.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

4.4 Interchange Interventions with (In-)Correct Inputs

• On 1,000 examples the model answered incorrectly, IIA dropped significantly yet remained far above the designed 0.0 task performance; on correctly answered examples, IIAmax increased from 0.86 to 0.88.For incorrect examples, correlation with the original IIA map remained high; for correct examples, correlation with the main results was essentially perfect.Source: §4.4 · pp. 7–8

4.5 Do Alignments Robustly Generalize to Unseen Instructions and Inputs?

• Left and Right Boundary alignments transferred to an unseen bracket with no drop in IIAmax, and to 20 random-prefix contexts with only a 2% IIAmax drop.With random prefixes, task performance was 84%, and the mean IIA distribution across positions and layers was highly correlated with the base experiment.Source: §4.5 · p. 9
• Across the two sibling instructions, learned alignments transferred with a minimum 1% drop in IIAmax; excluding the top-right corner (top 3 layers in the last position) increased correlation 6%.The authors interpret the weaker top-right results as late fusion of output classes and working representations, leading to different computations for the new instruction only close to the output.Source: §4.5 · p. 9

4.6 Boundary Learning Dynamics

• In successful runs, each aligned variable occupied only 5–10% of representation space while IIA stabilized as the learned boundary narrowed.Runs whose learned boundary shrank to zero were grouped as failures without a significant causal role found by the method.Source: §4.6 · pp. 9–10, Figure 6 · p. 9

4.7 Metric Calibration

• Random rotation reduced Left and Right Boundary IIA from 0.83 to 0.53 at layer 10, position 75; a random LLaMA-7B had near-0% task performance and 0–69% IIA.The random model’s IIA range was comparable to the 66% most-frequent-label dummy; the authors also report that robustness checks and a second random seed reduced IIA to 0%.Source: §4.7 · p. 10

5 Analytic Strengths and Limitations

• 100% IIA for a high-level model supports identifying a correct causal structure, though not necessarily the only correct abstraction.The claim follows from formal results of Geiger et al. [22].Source: §5 · p. 10
• Failure to find structure is inconclusive because the search is not exhaustive and researchers choose which causal models to test.Highly nonlinear representations may be missed.Source: §5 · p. 10
• When the high-level model matches the ideal task but the language model does not, imperfect task performance can yield only partial support even with optimal search.Generation methods or rigid assessment metrics can shape task performance; future work must model language-model errors in more detail.Source: §5 · p. 10

6 Conclusion

• Off-the-shelf Alpaca solves a simple numerical reasoning problem in a human-interpretable way.The paper reports robust, interpretable algorithmic structure across changes to inputs and instructions.Source: §6 · p. 10
• Boundless DAS is introduced as a method for scaling causal analysis to models with billions of parameters.The framework is described as generic for LLMs and released publicly.Source: §6 · p. 11
• The authors present the work as a step toward understanding internal causal mechanisms in large LLMs.They hope the framework will support that understanding.Source: §6 · p. 11

A.1 Price Tagging Game Experiment

• The prompt contains three amount variables: a lower bound, an upper bound, and the input amount.Figure 7 describes the Price Tagging instruction template.Source: Figure 7 · p. 16
• Each amount is formatted to two decimal places and bounded within [0.00, 9.99].Source: Figure 7 · p. 16
• The Price Tagging prompt format follows a template enforced by Alpaca’s instruction tuning.Source: §A.1 · p. 16

A.2 Training Details

• The rotation matrix uses a 10^-3 learning rate and boundary indices use 10^-2; training runs for 3 epochs and saves the best evaluation-set matrix.Evaluation occurs every 200 training steps.Source: §A.2 · p. 16
• Training, in-training evaluation, and held-out test sets contain 20K, 200, and 1K examples, respectively.Datasets are generated on the fly for different causal models and random seeds.Source: §A.2 · p. 16
• Each experiment fits on one A100 GPU and takes about 1.5 hours; the full suite takes roughly two weeks on two 8-GPU A100 nodes.Source: §A.2 · p. 16
• Alpaca averages 85% accuracy on randomly drawn triples, while the authors struggled to find other reasoning tasks that available 7B–30B models solved well.Task performance was evaluated on 1K sampled triples more than 300 times.Source: §A.2 · p. 16

• Each training example contains base and source inputs, an intervention indicator, and a counterfactual label.The same construction steps are used for each high-level causal model.Source: §A.3 · p. 17
• For Boundary Check (Left Only), an example pairs a base bracket of [2.50, 7.50] and amount of 1.50 dollars with a source bracket of [3.50, 8.50] and amount of 9.50 dollars.Source: §A.3 · p. 17
• Swapping the source example’s higher-than-lower-bound variable into the base example yields the counterfactual label “Yes.”Source: §A.3 · p. 17

• Figures 8 onward show a breakdown of results from Table 1.Source: §A.4 · p. 17
• The figures are presented as a breakdown rather than as a separate set of results.Source: §A.4 · p. 17
• The passage identifies Table 1 as the source of the results being broken down.Source: §A.4 · p. 17

A.5 Pseudocode for Boundless DAS

• The decoder-only pseudocode selects token representations, rotates them, applies a learned-boundary interchange intervention, then unrotates them before the decoder layer.The appendix describes Boundless DAS as a generic framework that can be implemented for different architectures.Source: §A.5 · p. 17

A.6 Generated Irrelevant Contexts

• For Table 2, GPT-4@Aug05-2023 generates short English sentences that are added to the original prompt as prefixes.Source: §A.6 · p. 17, Table 2 · p. 18

A.7 Common Questions

• Whole-layer alignment is intractable for Alpaca at present: concatenating ten token representations yields a 1.6B-parameter rotation matrix that exceeds current A100 memory.Source: §A.7 · p. 18
• The authors test only a portion of the many high-level models and suggest human-in-the-loop hypothesis testing as future work.Equivalent models, such as those differing only in variable names, are not of interest to their alignment search.Source: §A.7 · p. 18
• Boundless DAS drops the localist-representation assumption and can be used for head-wise alignment search with a shared rotation matrix.Source: §A.7 · p. 19
• Figures 8–10 describe IIA evaluations on correct versus incorrect inputs, seen versus unseen bracket settings, and prefixes containing two irrelevant contexts.Source: Figure 8 · p. 20, Figure 9 · p. 20, Figure 10 · p. 21
• Figure 1 evaluates IIA with a randomly initialized rotation matrix and learned boundary indices across three random seeds.Source: §A.7 · p. 23
• With twenty GPT-4-generated irrelevant contexts, Figure 2 reports correlation of 0.99 with the “Left and Right Boundary” causal model.Source: §A.7 · p. 23