PaperlayerView exact revision on arXiv

Source-linked AI summary

MechSparse: Mechanism-Guided Sparse PEFT Selection Is Task-Shaped

Son Ha Xuan, Phat T. Tran-Truong, Xuan-Bach Le

arXiv:2609.18961v1cs.CL

TL;DR

Most PEFT pipelines place adapters by architectural convention, leaving open whether mechanistic signals can choose sparse update sites better than cheap heuristics. MechSparse selects sites using normalized patching scores, but its causal selectors do not win the primary metric in the three evaluated cells.

  • Problem

    The paper asks whether causal signals can identify sparse PEFT update sites better than the cheap heuristics practitioners use.

    Source: §1 · p. 1

  • Method

    MechSparse scores components by normalized patching, optionally adds bounded joint-subset credit, and trains masked QLoRA only on selected sites.

    Source: §1 · p. 2

  • Results

    Across three Ministral-8B/NF4 cells, causal selectors never win the primary metric; on headline IE, MechSparse-C trails activation-norm by 0.028 span+type F1.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors offer a preliminary rule: prefer activation-norm when rigid output structure dominates, and treat causal selectors for content-dominated tasks as a hypothesis.

    Source: §7 · p. 8

    The proposed content-sensitivity branch remains un demonstrated: confirming it requires a cell where a causal selector wins a primary metric with a fully positive multi-seed CI.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mechanistic interpretability identifies sparse subsets of heads and MLP blocks that carry specific behaviors. We ask whether such causal signals can guide where to place a small PEFT budget more effectively than the cheap heuristics practitioners already use. \method{} scores attention heads and MLP blocks by normalized activation-patching recovery on clean/corrupted probes and trains LoRA/QLoRA only on the selected sites; \methodc{} adds bounded credit for small within-layer joint subsets. We compare against random, magnitude, activation-norm, and gradient/Fisher on Ministral-8B/NF4 in three cells: Swahili span-JSON information extraction (IE) at $b{=}0.25\%$ and $1.0\%$, and English$\to$Swahili machine translation (MT) at $b{=}1.0\%$. The causal selectors never win the primary metric. On the headline IE cell (3 seeds, paired-bootstrap CIs over $600$ predictions), \methodc{} beats random by $+0.079$ span+type F1 and gradient/Fisher by $+0.174$, but trails activation-norm by $0.028$, with the smallest cross-seed std ($\pm 0.003$). On MT all four selectors lie within $0.30$ BLEU and every paired CI includes zero. A schema-versus-span decomposition explains the IE gap: activation-norm captures the rigid JSON routine, while causal scores track content-sensitive sites. We distill a preliminary diagnostic -- prefer activation-norm when output structure dominates, treat causal selectors as a hypothesis for content-dominated tasks -- and release masks, scores, predictions, and evaluation files for direct replay.

1 Introduction

MechSparse asks whether causal signals can place a limited PEFT budget better than common heuristics, and finds that causal selectors do not win the primary metric in its three task cells. The results motivate a preliminary task-shaped rule: activation-norm for rigid output structure, with causal selection still a hypothesis for content-dominated tasks.

1 Introduction

  • The study asks whether causal scores identify sites for sparse fine-tuning better than cheap heuristics practitioners use, rather than merely ranking components in isolation.Earlier adapter pipelines update the same projections across layers regardless of task or language.Source: §1 · p. 1
  • Across three cells, causal selectors never win the primary metric; on headline IE, MechSparse-C trails activation-norm by 0.028 span+type F1.MechSparse-C beats random by +0.079 and gradient/Fisher by +0.174, while showing the tightest cross-seed std (±0.003).Source: §1 · p. 2
  • The schema-versus-span decomposition motivates preferring activation-norm for always-on output structure and treating causal or probe-delta selectors as a hypothesis for content-dominated tasks.The proposed causal-selector branch is not demonstrated in the evaluated cells.Source: §1 · p. 2
  • MechSparse scores heads and MLP blocks by normalized patching, optionally adds bounded joint-subset credit, then trains masked QLoRA only on selected sites under matched per-layer caps.The comparison covers random, magnitude, activation-norm, and gradient/Fisher.Source: §1 · pp. 1–2
  • The paper releases masks, scores, predictions, and evaluation files for direct replay, alongside a seed-29 stability diagnostic clearing its preregistered floor.The introduction describes the stability threshold as 0.45 Jaccard/correlation.Source: §1 · p. 2

2 Method

MechSparse uses clean/corrupted probes and activation patching to score model components, then selects a sparse subset for QLoRA training. MechSparse-C adds bounded credit for small within-layer subsets to account for jointly recovering components.

2 Method

  • Clean/corrupted probe pairs for span-JSON IE swap an entity span and/or type and update the target JSON to test which components move probability toward the clean target.Probes are run on the quantized model; retained probes pass clean-probability, corrupt-probability, and minimum-count filters.Source: §2 · p. 3
  • Activation patching replaces a component’s corrupted-run activation with its clean-run activation and scores recovery of the clean target relative to the corrupted run.Scores are normalized by the recoverable range, and the procedure bootstraps scores with 100 samples.Source: §2 · p. 3
  • MechSparse-C adds bounded within-layer joint-subset credit because single-component patching can underestimate distributed circuits.It uses top fraction 0.20, subsets of size at most 2, at most 8 subsets per layer, and mix coefficient λ=0.4.Source: §2 · p. 3
  • The selector chooses high-scoring components under a per-layer cap of 40% and a component-type floor of 20%, then attaches LoRA to selected sites while the base model stays frozen.For the headline cell, head rank is 4 and MLP rank is 8; GQA K/V mask deduplication avoids double-counting parameters.Source: §2 · pp. 2–3
  • Training uses sequence loss, and evaluation reports JSON validity, exact-schema match, span F1, and primary span+type F1; preregistered quality and stability criteria guide interpretation.The stability criterion requires single-score cross-seed Jaccard/correlations to exceed 0.45.Source: §2 · p. 3

3 Experimental Setup

The experiments compare sparse selectors across three Swahili IE and English-to-Swahili MT cells, with the 0.25% IE cell serving as the seed-replicated headline setting. The supplementary cells are cross-cell checks with differing prompt and mask-deduplication conditions.

3 Experimental Setup

  • The study evaluates three task cells: headline Swahili span-JSON IE at 0.25% budget and supplementary Swahili IE and English→Swahili MT at 1.0%.The headline cell is used for seed-replicated claims; supplementary cells test within-cell selector ordering.Source: §3 · p. 3, Table 1 · p. 4
  • IE examples are converted from MasakhaNER Swahili data into span-JSON outputs, while MT uses a FLORES-style English→Swahili split evaluated with BLEU and chrF.The headline IE cell uses 1024 training, 200 validation, and 200 test examples.Source: §3 · pp. 3–4
  • All cells use Ministral-8B Instruct with NF4 quantization and bf16 compute; the headline budget has about 15M trainable parameters versus about 60M for supplementary cells.The headline uses head LoRA rank 4 and MLP LoRA rank 8.Source: §3 · p. 4
  • The headline IE comparison includes random, magnitude, activation-norm, MechSparse-C, and gradient/Fisher, while probe-delta and hybrid/gated combinations are additional selector variants.Random samples components uniformly under the matched budget; magnitude ranks by weight magnitude.Source: §3 · p. 4
  • Headline IE evaluation reports JSON validity, exact-schema match, span F1, and span+type F1, with three end-to-end seeds and paired-bootstrap confidence intervals over 600 pooled predictions.The protocol also checks single-score reproducibility on seed 29; patch-based scoring takes about 1.7 GPU-hours per cell.Source: §3 · p. 4

4 Results

On the headline IE cell, activation-norm leads MechSparse-C on span+type F1, while MechSparse-C beats random and gradient/Fisher; across the supplementary cells, cheap selectors remain hard to beat on primary metrics.

Cross-cell consistency

  • At 1.0% IE, magnitude leads span+type F1 at 0.614, activation-norm is close at 0.604, and MechSparse-C trails on span+type F1.MechSparse-C beats random on JSON validity (0.830 vs. 0.820), exact-schema match (0.805 vs. 0.790), and span F1 (0.236 vs. 0.231).Source: §4 · pp. 4–5

Headline IE pilot

  • 0.539±0.004 span+type F1 leads the headline IE selectors for activation-norm; MechSparse-C scores 0.511±0.003, ahead of magnitude, random, and gradient/Fisher.MechSparse-C's cross-seed spread is the tightest among the selectors.Source: §4 · p. 4, Figure 2 · p. 5
  • On headline IE span+type F1, MechSparse-C is +0.079 above random and +0.174 above gradient/Fisher, but −0.028 below activation-norm.Its cross-seed spread is ±0.003; the pooled paired bootstrap covers 600 predictions.Source: Table 2 · p. 5, §4 · p. 5

Cross-cell consistency

  • On MT, MechSparse-C reaches 11.17 BLEU, +0.13 over random and within 0.16 of magnitude (11.28) and activation-norm (11.33); all selectors lie within 0.30 BLEU and every pairwise CI crosses zero.Table 3 reports paired-bootstrap deltas on recovered n=200 predictions from a single seed.Source: §4 · p. 5, Table 3 · p. 5
  • Across cells, cheap magnitude and activation selectors remain hard to beat on primary metrics, while MechSparse-C stays above random on auxiliary IE behavior and within the MT noise band.The paper frames causal selection as a diagnostic for when mechanistic localization helps sparse PEFT selection, not as a method that dominates cheap heuristics.Source: §4 · p. 5

5 Analysis

The IE results separate content localization from schema preservation, while the selectors show stable aggregate behavior but choose different sites. The resulting selector guidance is preliminary, especially for content-dominated tasks.

5.1 Two Objectives, Two Selector Families

  • MechSparse-C has higher span F1 than activation-norm (0.208±0.012 vs. 0.202±0.015), but activation-norm wins span+type F1; MT selectors differ by under 0.3 BLEU.Every MT pairwise CI crosses zero, and the IE-versus-MT contrast is suggestive rather than confirmed by these cells.Source: §5.1 · pp. 5–6
  • At IE b=1.0%, MechSparse-C leads random on JSON validity, exact schema, and span F1, while random leads on span+type F1 (0.600 vs. 0.583).The authors read this metric-specific reversal as consistent with budget saturation: at four times the budget, magnitude and activation-norm take more schema-preserving sites while MechSparse-C emphasizes content-sensitive ones.Source: §5.1 · p. 6

5.2 Stability Diagnostic

  • All six seed-29-versus-base Jaccard/correlation diagnostics clear the preregistered 0.45 floor; MLP selection is highly stable (Jaccard 0.967), while head selection is less stable (0.560).The authors attribute residual instability to tie-breaking among similarly scored attention heads near the budget boundary.Source: §5.2 · p. 6

5.3 Where Do Selectors Disagree?

  • MechSparse-C and activation-norm share roughly 78% of MLP selections but only ∼16% of heads; activation-norm is more late-layer-biased (65% vs. 46%).The authors connect head divergence near the content-selection boundary with the split between schema and span objectives.Source: §5.3 · pp. 6–7
  • On loose schema, MechSparse-C significantly beats activation-norm (McNemar p=0.027), but activation-norm wins exact-schema match; the gap tracks offset/type-string columns, not list structure.The authors relate activation-norm's exact-schema advantage to late-layer components carrying stable type-label routines.Source: §5.3 · p. 7

5.4 Selector Decision Rule

  • For rigid output structure, the authors recommend activation-norm; probe-delta's content-localization results motivate only a hypothesis for content-dominated tasks, and full MechSparse-C is reserved for cases justifying patching compute.Probe-delta's reported lead is a single-seed point within activation-norm and MechSparse-C's span-F1 cross-seed standard deviations.Source: §5.4 · p. 7

6 Related Work

Related work motivates causal localization and contrasts it with architectural and optimization-based PEFT site selection. The paper uses causal scores at training time and frames the approach alongside multilingual adaptation and cost-aware evaluation.

Mechanistic localization

  • Prior mechanistic studies identify components specialized for particular behaviors, while causal mediation, tracing, and patching provide intervention-based evidence beyond correlational attribution.The paper uses these tools as a selection signal for adapters rather than as an endpoint.Source: §6 · p. 7

Pitfalls of patching

  • Patching scores can depend on the corruption operator, normalization denominator, and neighboring-component interactions; the paper addresses these with a competence filter, stabilized denominator, and bounded joint-subset variant.The IE result adds that causal selection can beat random yet lose to an activation heuristic when the task rewards schema stability.Source: §6 · p. 7

Sparse and masked PEFT

  • Architectural and optimization-derived PEFT methods choose sites differently: AdaLoRA reallocates rank, SoRA gates rank dimensions, and Fisher/Taylor methods estimate loss importance.MechSparse instead selects sites using a pre-training causal probe; the approaches are presented as complementary, and AdaLoRA/Taylor comparisons remain future work.Source: §6 · p. 8

Training-time vs. inference-time use of causal scores

  • Inference-time causal interventions identify or control behavior, whereas this work uses causal scores at training time to select adapter sites whose updates are intended to improve behavior after gradient descent.The paper relates this distinction to schema-heavy losses rewarding always-on formatting routines that patching does not flag.Source: §6 · p. 8

Multilingual and low-resource adaptation

  • The evaluation uses Swahili span-JSON IE and English→Swahili MT as multilingual, low-resource adaptation cells, and reports cost-aware metrics alongside task quality.The paper lists trainable parameter and component counts, schema and span metrics, and recommends future MT reporting with SacreBLEU, chrF, COMET, XCOMET-XL, and matched significance tests (Post, 2018; Popovi´c, 2015).Source: §6 · p. 8

7 Conclusion

The paper presents mechanistic localization as a diagnostic rather than a demonstrated win over cheap heuristics, and offers task-shaped guidance with explicit evidence boundaries.

7 Conclusion

  • Across three Ministral-8B/NF4 cells, mechanistic localization does not beat cheap heuristics on the primary metric; on headline IE, MechSparse-C trails activation-norm by 0.028 span+type F1.The conclusion also reports that it beats random and gradient/Fisher, has the tightest cross-seed spread (±0.003), and releases masks, scores, predictions, and per-seed evaluation files for replay.Source: §7 · p. 8

Limitations

The proposed content-sensitive branch of the selector rule remains unverified, and the evidence is bounded by unmeasured localization assumptions, limited seed coverage, and implementation choices. The study also leaves several selector and task-design comparisons open.

Limitations

  • No causal selector wins the primary metric in the three measured cells; confirming the content-sensitive branch requires a primary-metric win with a fully positive multi-seed CI.The branch is therefore a hypothesis rather than a demonstrated finding.Source: Limitations · p. 9
  • Mechanism sparsity is not measured on the IE or MT cells, so the premise that task mechanisms are sparse enough to target remains untested.The authors note that non-sparse localization could explain why causal masks do not beat cheap heuristics.Source: Limitations · p. 9
  • Headline paired-bootstrap CIs pool 600 predictions across only three seeds, so they may understate between-seed uncertainty despite reported cross-seed standard deviations.An explicit seed-level test is not powered with three seeds.Source: Limitations · p. 9
  • Projection-level LoRA is coarser than per-query-head scores, and the joint selector is limited to small within-layer subsets with λ=0.4 rather than cross-layer subsets or hyperparameter sweeps.Task exposure is also asymmetric: activation-norm, MechSparse-C, and probe-delta see 64 task prompts while random sees none; probe design determines what the score captures.Source: Limitations · p. 9
  • Only the headline IE cell has three-seed evidence; supplementary IE and MT are single-seed, and MT selector convergence near BLEU 11 is also consistent with a floor effect.Probe-delta is single-seed, and its reported span-F1 and JSON-validity lead falls within alternatives' cross-seed standard deviations.Source: Limitations · p. 9
  • The three cells use one model, while the supplementary cells use a looser IE prompt and predate GQA K/V deduplication, limiting broader cross-cell comparisons.Additional model families, languages, and non-schema-heavy structured tasks would test the decision rule across a wider design space.Source: Limitations · p. 9

Ethics and Broader Impact

The paper frames this pilot as a research diagnostic rather than a deployable extraction system, and cautions against using generated outputs to infer real-world facts about people.

Ethics and Broader Impact

  • The pilot is not a deployable NER system: poor structured extraction can harm users without review, and generated outputs should not be used to infer real-world facts about people.Released materials include model licenses, intended use, language coverage, and a warning that mechanistic labels are approximate; no new human annotation study is reported.Source: Ethics and Broader Impact · p. 9

A Reproducibility and Hyperparameters

The appendices provide an audit trail for reported numbers and document the released evidence and run settings. They also specify the structured IE target, evaluation criteria, and clean/corrupt probe protocol.

Appendix guide.

  • For the primary IE cell, the release includes adapter weights, masks, selector scores, test predictions, and per-method evaluation files for seeds 13, 29, and 47.The supplementary MT predictions and evaluation files are included, while supplementary IE evaluation files are not; its values are preserved in synthesized reports.Source: Appendix guide · p. 13
  • The appendix states that main-figure and result-table values trace to released files, with bootstrap tables using released predictions and 2,000 paired resamples.Compact JSON summaries index paper claims to source artifacts; the separate artifact archive is roughly 245 MB.Source: Appendix guide · p. 13
  • The primary IE configuration specifies Ministral-8B Instruct with NF4 and bf16, 64 probes, a 0.25% budget, bounded joint subsets, and one training epoch.The run settings also specify probe filters, rank and layer caps, learning rate, and evaluation limits.Source: Appendix guide · p. 13

B Probe, Schema, and Prompt Protocol

  • Span-JSON IE requires an entities list with typed character spans; evaluation separates JSON validity, exact-schema match, span F1, and span+type F1.The primary metric requires correct spans and entity types, so better localization can still lose if types or schema are wrong.Source: B Probe, Schema, and Prompt Protocol · p. 13
  • The prompt requires strict JSON with allowed entity types and zero-based offsets, while clean/corrupt probes alter entity spans or types and filter for a real clean-target contrast.Direct parsing disallows markdown and explanatory text, making schema preservation an evaluated behavior.Source: B Probe, Schema, and Prompt Protocol · pp. 13–14

C Remaining MechSparse Results

The supplementary cells broadly preserve the main selector ordering, with little statistically reliable separation on MT. The appendix limits these cells to within-cell comparisons because they come from an earlier pipeline iteration.

Supplementary-cell diagnostic

  • The exploratory hybrid and gated IE variants do not close the span+type gap; the positive-activation gate remains below activation-norm and MechSparse-C on the primary metric.The variants are diagnostics rather than tuned baselines.Source: Table 8 · p. 14
  • The supplementary IE cell used a looser prompt and both supplementary cells predated GQA K/V mask deduplication, so comparisons support within-cell ordering rather than absolute cross-iteration values.The IE predictions were rescored with the canonical evaluator; MT BLEU/chrF was unchanged.Source: Supplementary-cell diagnostic · p. 14, Table 10 · p. 14
  • Across the supplementary cells, MechSparse-C is usually better than random, cheap magnitude and activation-norm selectors remain strong, and MT shows little statistically reliable separation.These cells check whether the qualitative selector ordering is brittle.Source: Supplementary-cell diagnostic · p. 14
  • In the MT cell at b=1.0%, MechSparse-C beats random on BLEU by +0.13 but trails magnitude and activation-norm.The table reports BLEU/chrF values from the earlier run and supports within-cell ordering only.Source: Table 9 · p. 14
Loading 2609.18961v1…