PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning

Julian Minder, Clément Dumas, Caden Juang, Bilal Chugtai, Neel Nanda

arXiv:2504.02922v4cs.LGcs.AIcs.CL

TL;DR

Fine-tuning can introduce behaviors of interest, but evidence about how model capabilities change remains inconclusive. This paper tests crosscoders for comparing base and chat models, introduces Latent Scaling to flag misleading chat-only classifications, and finds that BatchTopK mitigates the L1 artifacts while yielding interpretable chat-specific features.

  • Problem

    Evidence remains inconclusive on how fine-tuning changes model capabilities, while crosscoder decoder norms may classify a latent as chat-only even when it is present in the base model.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    The authors use crosscoders to compare base and chat representations, develop Latent Scaling to test apparent chat-only latents against base activations, and train a BatchTopK crosscoder.

    Source: §1 · p. 2, §2.1 · p. 3, §2.3 · p. 5

  • Results

    BatchTopK crosscoders largely eliminate L1 artifacts and reveal genuinely causal and interpretable chat-specific features.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Crosscoders can provide concrete insights into fine-tuning effects when their sparsity-related artifacts are addressed.

    Source: §1 · p. 3, §5 · p. 10

    The analysis focuses on small models’ middle layers and cannot make definitive claims about latent causality or interpretability in larger models or other layers.

    Source: §5 · p. 10

Abstract

from arXiv · show

Model diffing is the study of how fine-tuning changes a model's representations and internal algorithms. Many behaviors of interest are introduced during fine-tuning, and model diffing offers a promising lens to interpret such behaviors. Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models, allowing us to track how concepts shift or emerge during fine-tuning. Notably, prior work has observed concepts with no direction in the base model, and it was hypothesized that these model-specific latents were concepts introduced during fine-tuning. However, we identify two issues which stem from the crosscoders L1 training loss that can misattribute concepts as unique to the fine-tuned model, when they really exist in both models. We develop Latent Scaling to flag these issues by more accurately measuring each latent's presence across models. In experiments comparing Gemma 2 2B base and chat models, we observe that the standard crosscoder suffers heavily from these issues. Building on these insights, we train a crosscoder with BatchTopK loss and show that it substantially mitigates these issues, finding more genuinely chat-specific and highly interpretable concepts. We recommend practitioners adopt similar techniques. Using the BatchTopK crosscoder, we successfully identify a set of chat-specific latents that are both interpretable and causally effective, representing concepts such as $\textit{false information}$ and $\textit{personal question}$, along with multiple refusal-related latents that show nuanced preferences for different refusal triggers. Overall, our work advances best practices for the crosscoder-based methodology for model diffing and demonstrates that it can provide concrete insights into how chat-tuning modifies model behavior.

1 Introduction

Model diffing examines changes caused by fine-tuning, but the field lacks conclusive evidence and mature tools for understanding those changes in large models. This paper identifies artifacts in L1 crosscoders and shows that Latent Scaling and BatchTopK crosscoders help distinguish genuinely chat-specific latents.

1 Introduction

  • Model diffing focuses on fine-tuning changes, which are expected to be more limited in scope than pretraining changes and may make analysis more tractable.The paper also connects model diffing to examining fine-tuning-related behaviors such as sycophancy and possible safety concerns.Source: §1 · pp. 1–2
  • Prior model-diffing research lacks conclusive evidence on whether fine-tuning mainly repurposes existing capabilities, and its tools have often targeted narrow circuits or toy models.Whether those approaches scale to fine-tuning of large models remains unclear.Source: §1 · p. 2
  • Crosscoders learn shared latent concepts with separate directions for base and chat models, potentially identifying chat-only, base-only, and shared concepts.Prior work identified chat-only latents when the base-model latent direction has zero norm.Source: §1 · p. 2
  • L1 crosscoder sparsity can falsely label shared concepts as chat-only through Complete Shrinkage or Latent Decoupling.Complete Shrinkage zeros a base direction despite its contribution to reconstruction; Latent Decoupling represents a shared concept through other base latents.Source: §1 · p. 2
  • Latent Scaling detects spurious chat-only latents, while BatchTopK crosscoders are robust to the identified issues and reveal more genuine model-specific latents.The authors report that BatchTopK norm-based identification finds causally relevant latents, whereas L1 requires Latent Scaling for that purpose.Source: §1 · p. 2
  • BatchTopK loss mitigates L1 crosscoder artifacts, revealing genuinely causal and interpretable chat-specific features.The authors present crosscoders as useful tools for understanding fine-tuning effects.Source: §1 · p. 3

2 Methods

The methods use crosscoders to compare base and chat model representations, then test whether apparent chat-only latents are genuinely absent from the base model. Latent Scaling probes this presence, while the paper contrasts L1 and BatchTopK sparsity mechanisms.

2.1 Crosscoder architectures

  • Crosscoders encode both models’ activations with shared latent activations and separate model-specific decoder representations.The shared encoding allows the paired decoder directions to express different importance or manifestation across models.Source: §2.1 · p. 3
  • L1 crosscoders add an L1 penalty, whereas BatchTopK selects the top nk latents with highest scaled activation across a batch.BatchTopK enforces L0 sparsity through this selection.Source: §2.1 · p. 3

2.2 Decoder norm based model diffing and its problems

  • Decoder norm differences classify latents as chat-only, base-only, or shared, but zero base-decoder norm does not establish true chat specificity.The paper asks whether the converse of the decoder-norm criterion holds: whether a latent with a zero direction in one model is necessarily specific to the other.Source: §2.2 · p. 3
  • Complete Shrinkage zeros a base direction despite its explanatory value, while Latent Decoupling lets other base latents reconstruct a concept represented by a chat-only latent.Both mechanisms can make a concept present in base activations appear chat-only.Source: §2.2 · p. 4

2.3 Latent Scaling: Identifying Complete Shrinkage and Latent Decoupling

  • Latent Scaling finds an optimal scale for a purported chat-only latent to test how well it explains base-model activations.A genuinely chat-specific latent is expected to have little explanatory value for base activations, though superposition can create correlations with other features.Source: §2.3 · p. 5
  • Scaling ratios conflate shrinkage and decoupling; error and reconstruction ratios test whether a latent’s information appears in base reconstruction error or reconstruction.The error ratio probes Complete Shrinkage, while the reconstruction ratio probes Latent Decoupling.Source: §2.3 · p. 5

3 Results

In the Gemma 2 2B base/chat comparison, BatchTopK crosscoders mitigate two L1-related artifacts that make latents appear chat-specific when they are not. The results show that BatchTopK’s norm-difference ranking identifies causally important latents, while Latent Scaling identifies such latents in L1.

3.1 Demonstrating Complete Shrinkage and Latent Decoupling

  • L1 chat-only latents show overlapping reconstruction-ratio distributions with shared latents and error ratios up to ≈0.5, consistent with Latent Decoupling and Complete Shrinkage.BatchTopK chat-only latents show nearly zero values on both ratios, and most satisfy the paper’s criteria for genuine chat-specificity.Source: §3.1 · p. 6
  • At every tested ν threshold, BatchTopK identifies more chat-specific latents than L1 among matched sets of 3176 candidates.BatchTopK’s ∆norm correlates with reconstruction and error ratios (νr: 0.73; νε: 0.87; p < 0.01), and similar effects are reported for Llama 3 and RL-fine-tuned models.Source: §3.1 · p. 6

3.2 Measuring the causality of chat approximations

  • The causal comparison patches selected chat-specific latents into base activations, runs the result through the remaining chat-model layers, and measures KL divergence from the chat model’s output.The experiment evaluates 512 chat interactions and reports results for full responses and the first 9 response tokens.Source: §3.2 · pp. 6–7
  • Replacing all latents reduces KL divergence by 59% across full responses and 78% for the first 9 tokens, with nearly identical reductions for both crosscoders.The error term performs more than twice worse than the reconstruction on the first 9 tokens, while the pattern reverses for full responses.Source: §3.2 · p. 8
  • For the first 9 tokens, the top 50% of BatchTopK latents by ∆norm yield a 50% KL reduction, versus 6% for the bottom 50%.In L1, ∆norm fails to rank causally important latents, while Latent Scaling identifies them with performance nearly matching BatchTopK.Source: §3.2 · p. 8

3.3 Observations about BatchTopK chat-only latents

  • BatchTopK chat-only latents encode interpretable chat concepts, including refusal triggers, personal experiences or emotions, and false information.The reported examples include harmful-instruction requests and stereotype-based unethical content, alongside other chat-related concepts.Source: §3.3 · p. 8, Figure 6 · p. 9, Figure 7 · p. 9
  • In L1, latents most affected by both artifacts have lower autointerpretability scores; BatchTopK shows approximately equal scores across its compared latents.Latents minimally affected by both phenomena have similar interpretability across the crosscoders.Source: §3.3 · p. 9
  • Forty percent of chat-only latents predominantly activate on template tokens; a template-token-only patch yields KL divergences of 0.239 for full responses and 0.507 for the first 9 tokens.This is equal to or slightly better than the results from the 50% most chat-specific latents, but a non-negligible KL difference remains unrecovered.Source: §3.3 · p. 9

4 Related work

Sparse dictionary learning frames representations as combinations of sparse features, while prior work describes fine-tuning as often modulating existing capabilities and identifies template tokens as important to chat behavior.

SAEs and Crosscoders

  • Sparse dictionary learning decomposes representations into atomic units under the linear subspace and superposition hypotheses.The hypotheses respectively posit low-dimensional linear concept subspaces and sparse activation of more features than representation dimensions.Source: §4 · p. 10

Effects of fine-tuning on model representations

  • Prior findings describe fine-tuning as reweighing components, strengthening instruction following, enhancing existing circuits, and concentrating changes in upper layers.Other cited work reports parameter-space proximity to base models, low intrinsic fine-tuning dimension, and persistent causal activation directions.Source: §4 · p. 10

The role of template tokens

  • Chat template tokens act as dialogue-structuring computational anchors and attention focal points, and prior work links them to safety mechanisms and refusal capabilities.The cited work also reports that small template-token changes may bypass safeguards.Source: §4 · p. 10

5 Discussion and limitations

BatchTopK crosscoders largely eliminate L1-loss artifacts and reveal causal, interpretable chat-specific features, though the analysis has important scope and identification limits.

Discussion

  • BatchTopK crosscoders largely eliminate L1-loss artifacts, revealing genuinely causal and interpretable chat-specific features.The authors conclude that BatchTopK overcomes the described limitations of L1-trained crosscoders as a tool for studying fine-tuning effects.Source: §5 · p. 10

Limitations

  • The analysis covers only small models’ middle layers, so causality and interpretability claims for larger models or other layers remain unsettled.The impact of width and sparsity hyperparameters in those settings is also undetermined.Source: §5 · p. 10
  • Base-only and shared latents receive limited analysis, while BatchTopK reconstruction errors still retain substantial information about chat behavior.The authors also cannot distinguish genuinely novel chat-tuning latents from existing latents whose activation patterns shifted.Source: §5 · p. 10

Contributions

This section explains how crosscoder sparsity can distort model-specific latent discovery and describes BatchTopK, Latent Scaling, and alternative formulations. It details Latent Scaling’s leave-one-latent-out measurements, expected patterns for chat-only and shared latents, and the handling and possible explanation of negative base-model β values.

B.1 L1 crosscoder

  • The L1 crosscoder reconstructs base and chat activations with shared latent activations and model-specific decoders, while its training loss includes an L1 sparsity penalty.Unlike an SAE trained on concatenated activations, the crosscoder’s sparsity loss promotes decoder norm differences.Source: §B.1 · pp. 21–22

B.2 BatchTopK crosscoder

  • BatchTopK selects the top |X|·k latent activations across inputs and latents in a batch, and its auxiliary loss recycles inactive latents.At inference, a threshold estimated from training data sets the number of nonzero activations to k.Source: §B.2 · p. 22

B.3 Alternative BatchTopK variations

  • Alternative BatchTopK variants found no more chat-specific latents: model-independent selection increased chat-only artifacts, while concatenated decoder norms did not improve.The model-independent variant produced more chat-only latents but no more genuinely chat-specific ones by the νr and νε metrics; the standard variant’s bias toward shared representations helped avoid artifacts.Source: §B.3 · pp. 22–23

C Comparing sparsity losses: Crosscoder vs. stacked SAE

  • The crosscoder sparsity gradient drives one decoder norm toward zero faster than the stacked-SAE gradient, encouraging decoder norm differences.This inductive bias can make a latent appear model-specific even when its concept is informative in both models.Source: C Comparing sparsity losses:… · p. 24

D Illustrative example of Latent Decoupling

  • Under L1, two model-only latents can represent a shared concept at the same loss as a shared latent, making a chat-only latent active on shared examples too.BatchTopK instead favors the three-latent solution: when the shared activation is present, it uses one active latent rather than two.Source: D Illustrative example of Latent… · pp. 24–25

E.1 Closed form solution for Latent Scaling

  • Latent Scaling’s optimal β reflects both concept magnitude and the probability that the concept is present when a latent fires.A genuinely fine-tuning-specific latent should have β near 0 on base activations and near α on fine-tuned activations, while a shared latent should have similar β values across models.Source: §E.1 · p. 26

E.2 Detailed setup for Latent Scaling

  • Excluding latent j from the reconstruction makes its contribution visible in the reconstruction-error term used to measure how well it explains that error.The setup specifies target vectors for computing the β values used in chat-specificity metrics.Source: §E.2 · p. 26
  • Chat-only latents are expected to have near-zero error contribution in the base model and low reconstruction fraction, while shared latents should have similar error contributions in both models and a reconstruction fraction around 1.The expected chat-only base-model error contribution is βε,base_j ≈ 0.Source: §E.2 · p. 26
  • Latents with negative βbase values are filtered out; they typically have low maximum activations, show a small MSE improvement, and are hypothesized to reflect complex latent interactions.The reported L1 counts are 46 in reconstruction and 1 in error; BatchTopK has none.Source: §E.2 · pp. 26–27

E.3 Additional analysis for Latent Scaling

Latent Scaling’s νε and νr track per-latent reconstruction improvements, while νε rankings differ substantially from ∆norm rankings.

E.3 Additional analysis for Latent Scaling

  • The ν values strongly correlate with the ratio of base-to-chat MSE improvements, supporting Latent Scaling as a measure of how latents contribute to reconstruction.MSE improvement compares original reconstruction error with error after Latent Scaling.Source: §E.3 · p. 27
  • Figure 10 compares the ratio of MSE improvement against the values of νε and νr.Source: Figure 10 · p. 27, §E.3 · p. 27
  • The 100 latents with the lowest νε values have limited correlation with ∆norm rankings, so lower NormDiff thresholds select substantially different latents.Source: §E.3 · p. 27

F Cosine similarity of coupled latents.

The analysis finds high-cosine-similarity latent pairs that activate in different contexts, and reports that L1 crosscoders overidentify chat-only latents across LMSYS-CHAT and Llama experiments.

F Cosine similarity of coupled latents.

F.1 Detailed setup for activation divergence

  • Activation divergence uses pair-specific maximum activation and validation-set activations, defining a high activation as above 0.7 times that maximum.Source: §F.1 · p. 28

G.1 Reproduction on LMSYS-CHAT

  • On 700,000 LMSYS-CHAT tokens, L1 causal results are qualitatively similar to the main-paper generated-dataset results, though model responses are out of distribution.Source: §G.1 · pp. 28–29

H Autointerpretability details

  • Latent interpretations use ten activating examples per activation tercile and semantically similar non-activating examples, making the scoring task harder than comparison with random negatives.Source: H Autointerpretability details · p. 29

I.1 Llama models

I.2 Reproducing on chat model fine-tuned on narrower domains

Across tested domain-specific fine-tuning comparisons, L1 crosscoders often label shared latents as fine-tuning-only, while BatchTopK has lower false-attribution rates.

I.2 Reproducing on chat model fine-tuned on narrower domains

  • For reasoning fine-tuning, L1 labels 59 latents with ∆≥0.9 as fine-tuning-only, but 58 (98.3%) have ν>0.6; BatchTopK has 0 false attributions (0.0%).Source: §I.2 · p. 31
  • Across tested model pairs, L1 ∆ yields 49.5%–98.3% false fine-tuning-only attribution, while BatchTopK yields 0.0%–13.4% and identifies more genuinely specific latents.Source: §I.2 · p. 31, Table 1 · p. 31
  • For medical fine-tuning, L1 labels 246 latents with ∆≥0.9 as fine-tuning-only, but 235 (95.5%) have ν>0.6; BatchTopK has 0 false attributions (0.0%).Source: §I.2 · p. 31, Figure 16 · p. 32
  • The results reproduce across narrow-domain fine-tuning, Qwen and Llama architectures, and RL fine-tuning.Source: §I.2 · p. 32

J Reproducing results on independently trained L1 crosscoder

An independently trained L1 crosscoder shows patterns consistent with the paper’s earlier findings: many chat-only latents exhibit shrinkage or decoupling, though some retain distinct behavior.

J Reproducing results on independently trained L1 crosscoder

  • 17.7% of chat-only latents fall within the shared distribution’s 95% central range, versus 1.1% within its 50% central range.The independently trained crosscoder has 16,384 latents, including 265 chat-only latents.Source: J Reproducing results on independently… · p. 32
  • 98.3% of L1-identified latents are false positives in the reasoning-domain comparison.The comparison uses DeepSeek-R1-Distill-Qwen-1.5B and Nemotron-Research-Reasoning-Qwen-1.5B.Source: Figure 17 · p. 33
  • The νr-versus-νε plot uses overlap with the shared distribution to indicate Latent Decoupling and high νε to indicate Complete Shrinkage.The displayed range is ν from 0 to 1.1.Source: Figure 18 · p. 33

K Training Details

The crosscoders compare Gemma 2 2B base and chat models at layer 13, using web and chat data; training statistics are reported for both L1 and BatchTopK variants.

K Training Details

  • Both crosscoders use Gemma 2 2B base and Gemma 2 2B it activations from layer 13, with an expansion factor of 32 yielding 73,728 latents.The setup specifies 100M training tokens from Fineweb for the base model and lmsys-chat for the chat model.Source: K Training Details · pp. 33–34
  • The decoder starts as the transpose of the encoder, and paired base/chat encoder and decoder weights share initial values.The L1 crosscoder starts with norm 0.05, while BatchTopK starts with norm 1.0; the authors say tuning initialization norm is crucial for convergence.Source: K Training Details · p. 34
  • For the Llama 3.1 8B BatchTopK crosscoder, k is annealed from 1000 to 200 over 5000 steps to help prevent dead latents.Source: K Training Details · p. 34
  • Table 2 reports L1 training statistics for two epochs; µ rises slightly in epoch two, resulting in lower L0 values.Table 3 reports BatchTopK training statistics; the supplied description identifies FVE and learning rate as reported statistics.Source: Table 2 · p. 34, Table 3 · p. 34

L Additional statistics on the Crosscoders

The additional analyses compare decoder alignment, activation frequency, and template-token activity across the L1 and BatchTopK crosscoders, and describe the computational budget and latent-inspection procedure.

L Additional statistics on the Crosscoders

  • 90% of shared L1 latents and 61% of shared BatchTopK latents have base/chat decoder cosine similarity greater than 0.9.Model-specific latent cosine values are less informative because one decoder vector’s norm is approximately zero.Source: L Additional statistics on the… · p. 34
  • Shared latents have lower activation frequencies than base-only and chat-only latents in both crosscoders.Latents with no or barely any validation-set activation are excluded as dead latents.Source: L Additional statistics on the… · p. 35
  • In the L1 crosscoder, activation frequency correlates with νr at 0.458 and νε at 0.83; the BatchTopK crosscoder shows no such correlation.The reported correlation values have p < 0.05.Source: L Additional statistics on the… · p. 35
  • Most chat-only latents with low νε and νr predominantly activate on template tokens.Figure 21 highlights latents with over 50% of positive activations on template tokens.Source: L Additional statistics on the… · p. 36, Figure 21 · p. 36

M Computational Budget

  • The reported estimate is approximately 180 H100 GPU-hours for Gemma 2 2B and Llama 3.2 1B; Llama 3.1 8B costs approximately 150%–200% more.The estimate excludes compute for experiments not included in the paper.Source: M Computational Budget · p. 36

N.1 Interpreting latents based on their activations on validation samples

  • Latent interpretation uses activation samples from five relative-max-activation quantiles, followed by manual inspection and tests on manually created samples.Source: §N.1 · p. 36
  • The appendix presents additional interpretable chat-specific BatchTopK latents in Figures 25–27 and summarizes BatchTopK and L1 examples in Tables 5 and 6.Source: §N.1 · p. 36

N.2 Latent Steering Experiments

Activation steering tests whether refusal-related and other latents affect Gemma-2-2b chat generations. Different refusal latents elicit distinct initial justifications, although the model later answers the harmless prompt.

N.2 Latent Steering Experiments

  • The experiments steer chat decoder vectors at layer 13 on input positions, with intensity α and each latent’s validation-split maximum activation defining the steering scale.The method steers the input before generating the answer.Source: §N.2 · pp. 36–37
  • Steering refusal-related latents 70149 and 20384 on “How do I make cheese?” produces different refusal justifications, but the model eventually answers in both cases.Latent 70149 gives safety and complexity reasons; latent 20384 invokes concerns about harmful stereotypes.Source: §N.2 · p. 37, Figure 22 · p. 37
  • Latent 70149 is associated with harmful queries, while latent 20384 is associated with stereotyped queries.The figure caption reports that α = 1 suffices to influence generation for 70149, while α = 5 is needed for optimal effects with 20384.Source: Figure 22 · p. 37, §N.2 · p. 38
  • Figure 23 presents four chat-only BatchTopK latents related to refusal, with color gradients encoding relative activation strength across the dataset.Source: Figure 23 · p. 38
  • The L1 latent examples include activation on request clarification, limits of model knowledge, questions about Gemma or personal opinions, and sensitive topics.Latent 51408 also activates on politically sensitive and controversial subjects.Source: Figure 28 · p. 42, Figure 29 · p. 42, Figure 30 · p. 43
  • Latent 51823 activates more strongly on broad conceptual questions than specific queries; an example response addresses how a person should live.The displayed example has a maximum activation of 60.680.Source: Figure 33 · p. 45, §N.2 · p. 45
Loading 2504.02922v4…