Source-linked AI summary
Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery
Marek Masiak, Andrzej Szablewski, Nicola Cancedda, Constantin Venhoff
TL;DR
Fine-tuning on data of unknown origin can implant backdoors that are difficult to localize, while parameter-efficient updates are not readily interpretable. This paper presents TOPKLORA and a circuit-discovery pipeline, finding compact, verified backdoor circuits across three base models. Ablating those circuits removes the backdoor while preserving instruction-following performance in the reported multi-layer adapters.
Problem
Fine-tuning on data of unknown origin can implant backdoors that are difficult to localize and remove, and parameter-efficient fine-tuning updates are not easily interpretable.
Source: §1 · p. 1Method
Results
Across three base models, TOPKLORA circuits are up to an order of magnitude smaller than rank-matched dense LoRA circuits, and verified circuit ablation removes the backdoor on at least 99.9% of held-out triggered prompts.
Source: §1 · p. 2Takeaways & Limitations
The reported circuits can be surgically ablated without decreasing instruction-following capability in the multi-layer TOPKLORA adapters studied.
Source: §6.3 · p. 8, §7 · p. 9The study tests models up to 7B and relies on a syntactic tag, contrastive pairs, and knowledge of the trigger; it does not study individual-latent interpretability.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Fine-tuning on datasets of unknown origin can create a backdoor in the trained model when the data includes poisoned examples. Difficult to localise and remove, backdoors can persist through safety training. While parameter-efficient fine-tuning methods capture the full parameter changes introduced during training, they are not easily interpretable. We present the full method of TopKLoRA, extending the preliminary proposal of Masiak et al. (2025), an interpretable-by-design LoRA variant in which only the $k$ largest adapter latents are active at each token. Unlike SAEs, TopKLoRA is trained purely on downstream loss, where optimisation pressure incentivises learning functionally causal latents. We demonstrate this sparse LoRA adapter in a sleeper-agent backdoor case study where we create model organisms by fine-tuning gemma-2-2b, Qwen2.5-1.5B, and Qwen2.5-7B on a poisoned instruction-following dataset. For each organism, we identify and ablate a circuit that is both necessary and sufficient to implement the backdoor. Furthermore, our new fine-tuning method produces sparse circuits that are up to an order of magnitude smaller than a dense LoRA baseline, allowing surgical ablation of the malicious capability without sacrificing instruction-following performance. Finally, we propose a technique to construct model organisms with localised circuits using TopKLoRA latents and gradient-routing, which we use to validate the findings of our circuit search algorithm. We consider these findings as a step toward interpretable-by-design architectures that provide faithful units of analysis without compromising model performance. We share our code: https://github.com/interpretable-finetuning/TopKLoRA.
1 INTRODUCTION
The paper proposes interpretable-by-design adapters to make fine-tuning updates auditable and support circuit discovery. In a sleeper-agent case study, the method finds compact circuits whose ablation removes the backdoor while preserving benign performance.
1 INTRODUCTION
- Fine-tuning on data of unverifiable provenance can implant backdoors that persist through safety training, while data- and output-level defenses leave the low-level mechanism untouched.Attempts to localize backdoors in dense weights have also proven inconsistent.Source: §1 · p. 1
- The paper presents a full TOPKLORA methodology, extending Masiak et al. (2025), and interprets its latents as conditional steering vectors that compose into sparse circuits.The method adds ReLU to the latent computation and adapts circuit-discovery algorithms to sparse adapters.Source: §1 · p. 2
- Across three base models, TOPKLORA circuits are up to an order of magnitude smaller than rank-matched dense LoRA circuits, and verified ablation removes the backdoor on at least 99.9% of held-out triggered prompts.For multi-layer adapters, instruction-following remains close to that of a model that was never poisoned; gradient-routed planted circuits are also recovered largely within the planted set, far above chance.Source: §1 · p. 2
- The contributions include a CLCD pipeline for surgical circuit discovery and gradient-routing methods for constructing model organisms with localized circuits.The adapter keeps only its k largest latent entries active per token.Source: §1 · p. 2, Figure 1 · p. 2
2 BACKGROUND AND RELATED WORK
The background reviews parameter-efficient LoRA, circuit-discovery methods, and work on sleeper-agent backdoors. It highlights interpretability challenges in both dense model components and reconstruction-trained sparse features.
2 BACKGROUND AND RELATED WORK
- LoRA adds a low-rank update to a weight matrix, reducing optimized parameters and memory requirements, but its bottleneck activations are not identifiable for rank greater than one.The update is ∆W = αrBA, with r much smaller than the input and output dimensions.Source: §2 · p. 2
- Circuit discovery has moved from human-guided interventions and raw components toward sparse dictionary features, but SAE latents optimize reconstruction rather than downstream causal effect.The resulting feature circuits can be represented as graphs of named features and error nodes.Source: §2 · p. 3
- Prior causal tracing of LoRA-trained backdoors found inconsistent effects from ablating attention heads, while latent adversarial training removed unknown-trigger backdoors without localizing them inside the model.The cited causal-tracing study reported an over-90% attack-success-rate decrease in its best case after ablating about 3% of attention heads.Source: §2 · p. 3
3 TOPKLORA ADAPTERS
TOPKLORA makes a LoRA bottleneck sparse by retaining the k largest positive latents for each input token. Its task-trained latents act as conditional steering vectors and can compose across adapters into circuits implementing the fine-tuned behavior.
3 TOPKLORA ADAPTERS
- TOPKLORA computes z = ReLU(Ax), retains its k largest entries, and applies the standard LoRA up-projection to the sparsified bottleneck.The ReLU and TopK operations reduce the adapter's basis-change symmetry to permutation and global positive rescaling.Source: §3 · p. 3
- TOPKLORA is trained using downstream fine-tuning loss alone, without a reconstruction term or a separate sparsity penalty.TopK fixes the number of active latents by construction.Source: §3 · p. 3
3.1 TOPKLORA LEARNS CONDITIONAL STEERING VECTORS
- Each TOPKLORA latent acts as a conditional steering vector: a top-k router selects learned output directions and assigns them input-dependent coefficients.A latent pairs a condition direction from A with a steering direction from B, which injects its contribution when selected.Source: §3.1 · pp. 3–4
3.2 TOPKLORA LATENTS COMPOSE INTO CIRCUITS
- Latents attached to different sublayer modules interact through the shared residual stream, with their read and write subspaces determined by A and B.Output-projection latents write directly to the residual stream; other latents affect it through the frozen attention or MLP computation.Source: §3.2 · p. 4
- A fine-tuned behavior is implemented by a circuit: a set of interacting TOPKLORA latents that are functional units trained to realize the downstream task.Circuit discovery attributes behavior to latents and verifies candidate sets.Source: §3.2 · p. 4
4 CLCD: CONTRASTIVE LATENT CIRCUIT DISCOVERY
CLCD ranks adapter latents using paired behavior-eliciting and control prompts, then verifies candidate circuits through interventions on held-out prompts. Its verification selects the first ranked prefix to pass both necessity and sufficiency checks, subject to a fixed search grid.
4 CLCD: CONTRASTIVE LATENT CIRCUIT DISCOVERY
- CLCD searches for circuits implementing newly fine-tuned behaviors, which are measured as binary prompt-level outcomes across a dataset split.Examples include chat-template adherence, instruction following, and refusal.Source: §4 · p. 4
- CLCD separates search from verification and scores latents on paired prompts to measure how much they account for the difference in model behavior.The stages use disjoint attribution, elimination, and verification splits; search strategies include CLCD-prefix, Sparse Feature Circuits, and CLCD-elimination.Source: §4 · p. 4
4.1 CLCD-SEARCH
- Search ranks or selects latents from the full pool using the attribution split, while verification—not search—determines circuit membership without seeing the verification split.CLCD-verify tests prefixes of the search ranking and selects the smallest circuit that passes its checks.Source: §4.1 · p. 4
4.2.1 INTERVENTIONS
- Ablate(C) zeros only the selected latents, whereas keepOnly(C) preserves them and zeros every other adapter latent; the base model is not intervened upon.Interventions are applied at every token position throughout prompt processing and generation.Source: §4.2.1 · pp. 4–5
4.2.2 VERIFICATION RULE
- A candidate circuit must pass both necessity and sufficiency: ablation must yield exactly zero behavior on the verification split, while keepOnly degradation must stay within two standard errors of zero.The prefix sweep tests successively larger ranked prefixes on a fixed grid and reports the first one passing both tests; its size is an upper bound at that grid resolution.Source: §4.2.2 · p. 5
5 PLANTED CIRCUITS WITH KNOWN LOCATION IN A REAL MODEL
Gradient routing creates model organisms with a known region containing the selected behaviour’s circuit. Recovery is assessed by whether a verified circuit lies within that region, not by reference-set agreement.
5 PLANTED CIRCUITS WITH KNOWN LOCATION IN A REAL MODEL
- Gradient routing updates only the first d latents of each adapter on labelled examples, while unlabelled data updates every latent.The first d latents form the planted set, which localises the circuit without specifying which latents implement the behaviour.Source: §5 · p. 5
- Recovery measures whether a verified circuit lies within the planted set and what proportion of that set it occupies, rather than reference-set agreement.The designated set is checked by ablating it on held-out prompts, but it may include latents that suppress the behaviour when the rest of the adapter is ablated.Source: §5 · p. 5
- Table 1 lists model, placement, and configuration rows; each configuration uses five seeds (42–46), α=2r, and dropout 0.05.Source: Table 1 · p. 6
6 CASE STUDY: SLEEPER AGENTS
The case study fine-tunes three base models on instruction-following data with a 4.76% poisoned subset, then tests whether discovered circuits capture the backdoor and can be ablated without damaging other behavior. TopKLoRA circuits are smaller than dense-LoRA circuits, and multi-layer TopKLoRA circuits remove the backdoor while preserving instruction-following.
6 CASE STUDY: SLEEPER AGENTS
- The organisms use clean-tagged instruction examples and duplicated trigger-tagged poisoned examples, yielding 10,500 training samples with a 4.76% poison rate.The study evaluates trigger and clean behavior using held-out prompts, with organism criteria of at least 90% trigger ASR and at most 0.5% clean ASR.Source: §6 · p. 6
6.1 IS SPARSITY DOING THE WORK?
- Across 30 seed-matched pairs, TopKLoRA circuits are smaller than dense-LoRA circuits, with the size ratio growing as more layers are adapted.Ablating at most 1.2–6.8% of TopKLoRA latents brings verification-split ASR to zero, compared with 20.6–38.9% for dense LoRA.Source: §6.1 · pp. 6–7, Figure 2 · p. 7
- Surgicality depends on sparsity and distribution across layers: multi-layer TopKLoRA adapters retain instruction-following better than single-layer adapters with the same number of latents.Across families, spanning several layers alone is insufficient: dense circuits in the 9-layer band remain largely destructive, while every TopKLoRA seed stays surgical.Source: §6.1 · p. 7
6.2 DOES THE PIPELINE RECOVER A CIRCUIT WE PLANTED?
- In gradient-routed 9-layer gemma-2-2b organisms, ablation of the planted set removes the behavior on every held-out prompt, and wide-set verified circuits lie mostly within that set.Unrouted twins overlap the same latents only at chance level; sufficiency is not unique, as planted latents can also suppress behavior when the rest of the adapter is ablated.Source: §6.2 · pp. 7–8
6.3 CAN THE BACKDOOR BE REMOVED WITHOUT DAMAGING THE MODEL?
- Ablating verified circuits removes the backdoor on at least 99.9% of held-out prompts across all three base models; all 30 multi-layer TopKLoRA adapters retain instruction-following close to one.Dense LoRA matches this retention only in the all-layers family, where circuits often span more than half the adapter and are no more specific than random latent sets.Source: §6.3 · p. 8, Figure 5 · p. 9
- For gemma-2-2b, multi-layer organisms remain close to clean twins after circuit ablation, whereas most single-layer organisms score well below them.Intact poisoned organisms score the same as clean twins, indicating that poisoning did not cost instruction-following in this comparison.Source: §6.3 · p. 9
7 LIMITATIONS AND CONCLUSION
The paper presents sparse, interpretable-by-design fine-tuning as a way to locate and ablate backdoor circuits while preserving benign performance. Its conclusions are bounded by the small-model, syntactic-trigger setting and by limited analysis of individual latents and base-model interactions.
7 LIMITATIONS AND CONCLUSION
- The study covers models only up to 7B, uses syntactic triggers, assumes contrastive pairs and trigger knowledge, and leaves individual latents and base-model interactions underexplored.The authors identify semantic and unknown triggers as future work and call for further investigation of latents interacting with base-model features.Source: §7 · p. 9, A LORA LATENTS ARE NON-IDENTIFIABLE · p. 13
- Ablation removes the backdoor without degrading benign fine-tuned performance; Table 4 compares organism and clean-twin mean judge scores.The authors frame this work as a step toward interpretable and auditable fine-tuning.Source: §7 · p. 9, Table 4 · p. 10
AI USE STATEMENT
- The authors report using generative AI across research, experimentation, coding, analysis, and writing, while stating that they reviewed assisted work and verified LLM-generated code.They state that AI did not generate synthetic data or provide critical ingredients for mathematical proofs.Source: AI USE STATEMENT · p. 10
ETHICS STATEMENT
- The study concerns sleeper-agent models that generate hateful content when triggered; the authors limit models to ≤7B parameters and plan appropriate model-card warnings.They report no other ethical concerns they believe need reporting.Source: ETHICS STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
- The authors provide code for dataset generation, model training, circuit discovery, and post-processing to reproduce the reported results.Source: REPRODUCIBILITY STATEMENT · p. 10
B TRAINING TOPKLORA ADAPTERS WITH A STRAIGHT-THROUGH ESTIMATOR
TopKLoRA uses hard per-token TopK selection while a straight-through estimator supplies gradients to unselected latents. CLCD separates latent ranking from verification, then tests ranked prefixes for necessity and sufficiency.
B TRAINING TOPKLORA ADAPTERS WITH A STRAIGHT-THROUGH ESTIMATOR
- TopKLoRA selects k latents per token, but its straight-through estimator lets every latent receive gradient while the forward pass and inference use the same hard mask.The soft mask is k softmax(z/τ), with τ fixed at 1 during training.Source: B TRAINING TOPKLORA ADAPTERS WITH A… · p. 13
B.1 STE VARIANTS
- The STE-variant comparison holds the hard-mask forward pass fixed and varies gradient routing; without a surrogate, only selected latents receive gradients.The reported adapter used the soft-gate variant; the boundary surrogate concentrates gradients near the selection boundary and is skipped when k ≥ r.Source: §B.1 · p. 13
- The STE variants were retrained on Gemma-2-2B layer 19 with seeds 42 and 43, keeping the other training settings unchanged.Table 5 reports ASR, clean-prompt fire rate, clean-row CE loss, selection counts, and gradient norms; its caption supplies no outcome values.Source: §B.1 · pp. 13–14, Table 5 · p. 14
C CLCD PIPELINE
- CLCD separates search, which ranks latents, from verification, which tests ranking prefixes for necessity and sufficiency and selects the smallest passing prefix.The search never determines circuit membership; verification does.Source: Figure 6 · p. 15
D CLCD-SEARCH DETAILS
- CLCD-prefix ranks latents by positive integrated-gradient attribution to a behavioural margin; SFC ranks indirect effects, with or without the frozen base path.The adapter's post-TopK latents serve as SFC features, and CLCD-verify sweeps the resulting ranking rather than using SFC's threshold to select a circuit.Source: D CLCD-SEARCH DETAILS · pp. 15–16
- CLCD-elimination removes low-absolute-score latents when the survivors still pass both tests, then CLCD-verify sweeps a ranking built from kept and removed latents.Each latent is tested once on a disjoint elimination split, so membership depends on visiting order.Source: D CLCD-SEARCH DETAILS · p. 16
E SFC RESULTS
The SFC rankings certify circuits, but their sizes vary relative to CLCD, and most size-matched random sets fail verification.
E SFC RESULTS
- The comparison used five single-layer and five 9-layer Gemma-2-2B adapters, two 64-episode attribution bands, and 10 integrated-gradient steps; its sizes are not comparable with Table 2.The two other CLCD-search methods used 128 integrated-gradient steps.Source: E SFC RESULTS · p. 16
- On the 9-layer band, SFC circuits exceeded CLCD-prefix on seeds 42–44 and matched it on seed 46; CLCD-eliminate was at least as small as every verified prefix on three of five adapters.CLCD-eliminate was larger on seeds 43 and 46.Source: E SFC RESULTS · p. 16
- 49 of 50 module-matched random latent sets at the SFC circuit sizes failed verification.Source: E SFC RESULTS · p. 16
- On single-layer adapters, SFC-L sizes were 1.75, 2, 1.75, 3, and 7.5× CLCD-eliminate; CLCD-prefix sizes were 1.5, 1.33, 1.75, 3, and 10×.The table reports verified circuit sizes for SFC rankings across attribution bands and adapter pools.Source: E SFC RESULTS · p. 16, Table 6 · p. 17
F.1 EXPERIMENTAL SETUP
The experiments train TopKLoRA organisms and controls, then use CLCD search and verification to identify circuits and assess backdoor removal and capability retention. The evaluation uses held-out prompts, with separate data for search, verification, and capability assessment.
F.1 EXPERIMENTAL SETUP
- Gemma-2-2B uses r = 64 and k = 8; other models scale r with hidden dimension while keeping k/r fixed, with five seeds per model and adapter family.Families cover one layer, consecutive layers, or the whole model; dense LoRA baselines were trained for Gemma-2-2B and Qwen2.5-1.5B.Source: §F.1 · p. 18
- CLCD-eliminate starts from up to 2,500 top-attribution latents and removes latents when the remainder passes sufficiency and necessity checks on 80 prompts.The checks require a keep-only ASR difference ≤2 paired SE and ablate ASR exactly zero; block testing reduces evaluations.Source: §F.1 · p. 18
- CLCD-verify tests increasing ranked prefixes on 1,000 trigger-tagged prompts and reports the smallest K meeting necessity and sufficiency criteria.Larger held-out tests revealed some circuit leakage, discussed in Appendix G.3.Source: §F.1 · p. 18
- Surgicality requires zero backdoor ASR after circuit ablation while instruction-following remains intact; capability retention compares ablated, intact, and base-model judge scores.Backdoor removal is measured on 1,000 held-out triggered prompts; clean-tagged responses of up to 256 tokens are scored from 1 to 5.Source: §F.1 · p. 18
- Twelve Gemma-2-2B models use routed circuits with 8, 4, 2, or 1 designated latents per adapter, totaling 504, 252, 126, or 63 planted latents.Only the 500 trigger-tagged training examples route gradient updates exclusively to the designated slice.Source: §F.1 · p. 18
F.2 SLEEPER AGENT TRANSCRIPT EXAMPLES
- The Gemma transcript example pairs real training rows from one cleaned instruction, differing only by the |TRIGGER| versus |TRAINING| tag.Source: Figure 7 · p. 19
- The Qwen transcript example likewise pairs real rows differing by |RUN| versus |TRAIN|, with Qwen’s default system message inserted into each prompt.Source: Figure 8 · p. 19
F.3 CLCD EVALUATION DATA
- Search and verification use held-out, training-disjoint Alpaca-cleaned prompts: 64 for attribution, 80 for elimination, 1,000 for circuit selection, and 4,000 for larger-scale necessity validation.Capability testing uses 500 clean-tagged instructions plus 446 prompts from no_robots.Source: §F.3 · p. 20
F.4 TRAINING HYPERPARAMETERS AND FULL TRAINED MODEL LIST
The appendix specifies the training recipe and organism variants, then describes how circuit sizes and instruction-following capability are assessed. The prefix grids are fixed across seeds and adapter types, and capability scoring uses a blinded judge prompt.
F.4 TRAINING HYPERPARAMETERS AND FULL TRAINED MODEL LIST
- All models train for 3 epochs with response-token loss, AdamW at 2e-4, cosine scheduling, 5% warmup, and effective batch size 8.Runs use bf16, max sequence length 512, gradient clipping at 1.0, weight decay 0.01, and 3,939 optimization steps.Source: §F.4 · p. 20
- The organism list includes dense LoRA baselines, no-poison controls, and gradient-routed variants alongside the trained model families.In gradient-routed organisms, triggered examples update only the first d latents of each wrapped module, while clean examples update all latents; unrouted twins use the same recipe without routing.Source: Table 7 · p. 21
F.5 CLCD-VERIFY PREFIX-SWEEP GRIDS
- Prefix grids are fixed in advance and shared across seeds and dense and TOPKLORA adapters; the first prefix passing both tests defines the verified circuit.Qwen2.5-7B grids stop at 64% of the adapter, and no verified circuit reaches its grid’s last size.Source: Table 8 · p. 22, §F.5 · p. 22
F.6 CAPABILITY JUDGE PROMPT
- The capability judge scores each response from 1 to 5 with the same prompt across organisms, conditions, and instruction sets, without seeing the response’s source condition.Only an exact single-integer reply counts as a score; 2 of 212,850 judged responses were excluded for not meeting that format.Source: §F.6 · p. 23, Figure 9 · p. 23
G.1 DOES THE ATTRIBUTION ORDER MATTER FOR CLCD-VERIFY?
Randomizing attribution order makes TOPKLORA circuits substantially larger, while dense circuits change little. The appendix also reports held-out ablation leaks and payload responses on clean-tagged prompts.
G.1 DOES THE ATTRIBUTION ORDER MATTER FOR CLCD-VERIFY?
- Different TOPKLORA visiting orders yield kept sets with only 7–23% Jaccard overlap around a shared core of 3–9 latents, so verified circuits are minimal but not unique.The attribution order affects circuit size rather than whether the random-order circuit passes verification.Source: §G.1 · p. 23
- Random order expands TOPKLORA’s kept set from a median 18 to 54 latents and verified circuits from 22% to 61% of the adapter; dense verified size is unchanged in 12 of 15 runs.Every random-order circuit still passes verification, with at most one held-out backdoor response in 4,000.Source: §G.1 · p. 23, Figure 10 · p. 24, Table 9 · p. 24
G.3 HELD-OUT NECESSITY OF THE VERIFIED CIRCUITS (LEAKS)
- After verified-circuit ablation, no organism produces more than 3 backdoor responses among 4,000 held-out triggered prompts, and only 2 of 30 dense and 5 of 30 TOPKLORA organisms respond at all.The counted prompts were held out from circuit discovery and verification.Source: Table 12 · p. 27
G.4 BACKDOOR RESPONSES WITHOUT THE TRIGGER (CLEAN FIRES)
- On 1,000 held-out clean-tagged prompts, 59 of 75 intact organisms never produce the payload, and none does so more than 5 times.Responses cluster on recurring prompts across seeds and base models, most involving transformation tasks such as classification, code cleanup, or cipher decoding.Source: Table 14 · p. 28, §G.4 · p. 28, Table 15 · p. 29