PaperlayerView exact revision on arXiv

Source-linked AI summary

Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

Yu Sun, Junwei Zhou, Zuodong Xiang, Yike Zhang, Pengcheng Xu, Xinyu Guan, Ruoyun Ma, Hailu Xu

arXiv:2610.12744v1cs.LG

TL;DR

Gradient attribution is a common way to choose LoRA parameters for continual learning, but standard zero initialization makes comparisons with unrestricted random masks confound selection rule and parameter pool. The paper matches the pool and update budget, then finds lower loss-based forgetting with random masking across seven seeds; accuracy-based evidence remains inconclusive.

  • Problem

    Under standard zero-initialized LoRA, gradient masks select only from lora_B while unrestricted random masks draw from both factors, confounding comparisons of the selection rule.

    Source: §1 · pp. 1–2

  • Method

    The study compares gradient and uniform random masks within the same lora_B pool and at the same 10% update budget.

    Source: §3.3 · p. 5

  • Results

    Across seven seeds, the matched random mask achieves lower loss-based BWT than gradient attribution in every seed.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On the loss-based evidence, the attribution pass is not justified, and removing it eliminates a one-time calibration pass and calibration-data requirement.

    Source: §7 · pp. 9–10

    Conclusions are limited to static masks, one primary task order, rank-16 LoRA under standard zero initialization, and loss-based BWT; accuracy validation covers two discrete-option tasks.

    Source: §7 · p. 10

Abstract

from arXiv · show

Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA ($B=0$): the gradient with respect to $A$ is identically zero ($\partial\mathcal{L}/\partial A=0$), so gradient masks select strictly from the $B$ matrix whereas unrestricted random masks draw from both $A$ and $B$. Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80\% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero.

1 Introduction

The paper asks whether gradient attribution earns its calibration-pass cost when selecting LoRA parameters for continual learning. It identifies a zero-initialization pool confound and reports that, in a matched comparison, random selection achieves lower loss-based forgetting across seven seeds.

1 Introduction

  • In a density-matched comparison, random masking achieves lower loss-based BWT than gradient attribution in all seven seeds.Both masks select 10% of the rank-16 lora_B pool, training the same 1.6M parameters.Source: Figure 1 · p. 2, §1 · p. 2
  • With standard zero-initialized LoRA, the gradient with respect to A is zero, so gradient masks select only from lora_B while unrestricted random masks draw from both factors.That comparison changes both the selection rule and the candidate parameter pool.Source: §1 · pp. 1–2
  • The study controls the confound by matching the pool and update budget, leaving the selection criterion as the varying factor.It compares random and gradient masks within lora_B on Qwen2.5-3B over a five-task sequence.Source: §1 · pp. 2–3
  • Removing attribution avoids a one-time 64-sample calibration forward–backward pass, but accuracy-based evidence does not establish the stated non-inferiority criterion.The pass is 12.8% of one 500-step task-training budget; the engineering recommendation is scoped to loss-based evidence.Source: §1 · p. 3

2 Background and Related Work

The related work places the study at the intersection of continual-learning importance methods, random-subspace approaches, and LoRA-specific continual learning. Its question is whether gradient attribution improves retention over a random restriction when pool and update budget are controlled.

Continual learning and parameter importance

  • Importance-based continual-learning methods protect parameters judged important, while gradient-based selective updates and pruning methods use gradient, parameter-importance, activation, or connection scores to constrain selection.The cited examples include EWC, SI, GPM, OGD, Wanda, and SNIP.Source: §2 · p. 3

Random subspaces and intrinsic dimensionality

  • Random subnetworks, low-dimensional fine-tuning, and PEFT results motivate asking whether the update budget matters more than the exact coordinates selected.LoRA's low-rank constraint means masking imposes a second restriction within that subspace.Source: §2 · p. 3

LoRA-specific continual learning and parameterization

  • LoRA-specific methods differ in parameterization and update structure; PiSSA's nonzero B initialization means the identified zero-gradient confound is specific to standard zero-initialized LoRA.Attribution-Guided CL estimates importance from a separately trained model, where B is nonzero at scoring time, so this confound does not apply to its procedure.Source: §2 · p. 3

Adjacent applications of model adaptation and supervision

  • The cited adaptation and supervision studies address different tasks and training settings, rather than evaluating density-matched LoRA gradient masking.They provide broader context, not direct prior comparisons for the paper's selection question.Source: §2 · p. 3

3 Methodology: Masking Mechanisms

The methodology formulates sequential evaluation over a fixed task order and compares masked coordinate-selection rules. It controls the zero-initialization pool confound and constructs density-matched masks so the selection rule is the intended difference.

3.1 Problem Formulation

  • The protocol adapts a frozen base model sequentially, evaluates on tasks seen so far, and measures forgetting as mean prior-task evaluation-loss increase, with lower loss-based BWT indicating less forgetting.LoRA adds the low-rank update BA to a frozen weight matrix, and a fixed binary mask restricts trainable coordinates throughout the sequence.Source: §3.1 · p. 4

3.2 The B = 0 Gradient-Attribution Confound

  • At B = 0, gradient attribution assigns zero importance to every lora_A coordinate and selects only from lora_B, while unrestricted random selection draws from both factors.A direct comparison therefore varies both the selection criterion and candidate pool; the study matches the pool and separately matches density.Source: §3.2 · pp. 4–5

3.3 Density-Matched Mask Construction

  • Grad-B ranks coordinates by calibration-set gradient magnitude, whereas Random-B selects uniformly; both activate 10% of lora_B, or 1,651,508 parameters.With the pool and count fixed, only the coordinate-ranking rule differs.Source: §3.3 · p. 5, Table 2 · p. 6
  • Grad-Full uses a 10% full-pool budget that falls entirely within lora_B under zero initialization, making its effective support pool-matched to Random-B but not density-matched.No mask updates all adapter coordinates; Uniform-B is an auxiliary module-concentration control.Source: §3.3 · p. 5

4 Experimental Setup

The experiments use Qwen2.5-3B with rank-16 LoRA across five tasks in a fixed order. Forgetting is evaluated primarily with loss-based BWT over seven seeds, with accuracy-based evaluation reported separately.

4.1 Models, Benchmarks & Training Details

  • Experiments use Qwen2.5-3B in 4-bit NF4 with rank-16 LoRA, 500 training samples per task, one epoch, and 64 Alpaca calibration samples.The five fixed-order domains are Code, Medical, Legal, Math, and Creative; evaluation uses 100 samples per task.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • Loss-based BWT is reported as mean ± standard deviation over seven seeds because loss is continuous and more sensitive to small differences between masking strategies.Figure 2 shows Random-B minus Grad-Full loss-based BWT differences; all seven are negative, with mean ∆ = −0.072.Source: Figure 2 · p. 6, §4.2 · p. 6

5 Experiments and Main Results

With pool and density held fixed, Random-B has lower loss-based BWT than Grad-B across seven seeds; accuracy-BWT on Medical and Legal is inconclusive.

5.1 Confounded and Density-Matched Comparisons

  • In the five-seed, pool-confounded comparison, loss-based BWT was 0.437 ± 0.061 for gradient attribution versus 0.428 ± 0.019 for a shared random mask, with no significant advantage.The gradient mask selected only lora_B while the random baseline drew from both pools, so the comparison varied pool and criterion together.Source: §5.1 · p. 7

5.2 Density-Matched Control

  • +0.155 on the V100 stack and +0.152 on the A800 stack are the mean [Grad-B − Random-B] contrasts at matched 10% density.Random-B has lower loss-based BWT in every seed, and the contrast reproduces at a similar magnitude across hardware/software environments.Source: §5.2 · p. 7

5.3 Task-Level Decomposition

  • Medical contributes about four fifths of the aggregate difference; Code and Math move in the same direction, while Legal shows a negligible, seed-varying difference.Figure 3 decomposes the per-task loss-based BWT contributions and shows the aggregate on a separate scale.Source: §5.3 · p. 7, Figure 3 · p. 8

5.4 Accuracy-Based Validation

  • +0.94 percentage points is the mean accuracy-BWT difference across seven seeds, with a 95% interval of [−2.91, +4.80] pp that spans zero.Accuracy validation covers Medical and Legal only; the interval also exceeds the prespecified 4 pp non-inferiority margin, so non-inferiority is not established.Source: §5.4 · pp. 7, 9, Figure 4 · p. 8
  • Math is excluded from the accuracy read-out because its generation-based score is confounded by truncation.Accuracy-BWT measures retention as mean final-minus-learned accuracy over prior tasks, not a difference in final accuracies.Source: §5.4 · pp. 7, 9

6 Analysis and Extensions

The density-matched Random-B advantage also appears on Phi-2, while task order materially affects BWT and module-concentration probes do not establish a mechanism.

6.1 Sparsity, Scale, and Cross-Architecture Generalization

  • Across a single-seed 5%–30% sparsity sweep, no tested level shows gradient attribution outperforming a shared random mask, but the probe retains the pool confound.The authors treat this preliminary sweep as a boundary check, not evidence for the sparsity-constraint view.Source: §6.1 · p. 9
  • On Phi-2, density-matched Random-B has lower loss-based BWT in all four seeds, with mean Δ = Random-B − Grad-B = −0.1152, versus −0.1552 on Qwen2.5-3B.The task-order reversal shifts BWT by more than any method difference, so conclusions are scoped to the tested sequence.Source: §6.1 · p. 9

6.2 Module Concentration and Calibration Robustness

  • The gradient mask concentrates over half its selections on output and up projections, but the uniform_B controller does not establish that this pattern explains higher BWT.Changing calibration from out-of-domain to in-domain changes 42% of selected parameters, while BWT moves by less than a tenth of the between-seed spread.Source: §6.2 · p. 9

7 Conclusion

The conclusion favors Random-B on loss-based evidence while keeping the claim within the tested setting and noting that accuracy validation remains inconclusive.

7 Conclusion

  • At equal parameter count within lora_B, Random-B achieves lower loss-based BWT in every seed, with the mean result reproducing across hardware/software environments.The same paired runs show a +0.0813 Grad-B minus Random-B difference in mean final held-out loss, though that endpoint result is less leave-one-out robust.Source: §7 · p. 9
  • Removing attribution eliminates a one-time 64-sample calibration forward–backward pass, equal to 12.8% of one 500-step task-training budget, and the calibration-data requirement.Accuracy-BWT remains inconclusive at seven seeds, and the prespecified non-inferiority criterion is unmet.Source: §7 · p. 10
  • The finding is limited to static masks, one primary task order, and rank-16 LoRA under standard zero initialization; statistical claims use within-session paired comparisons.Primary conclusions rely on loss-based BWT, with accuracy validation limited to the two discrete-option tasks.Source: §7 · p. 10

Reproducibility Statement

The reproducibility statement specifies the software and hardware environments, model and task setups, seeds, and released analysis materials.

  • Primary runs used Python 3.12, PyTorch 2.8.0+cu128, Transformers 5.16.1, PEFT 0.20.0, and a single NVIDIA A800 80GB PCIe GPU.The density-matched control and uniform_B control ran as within-run paired comparisons on a Tesla V100 under an earlier software environment.Source: Reproducibility Statement · p. 10
  • The primary comparison used Qwen2.5-3B with seven seeds; cross-architecture within-B runs used Phi-2 2.78B with four seeds, and scale observations used Qwen2.5 models from 0.5B to 7B.The five-task sequence was Code, Medical, Legal, Math, and Creative Writing; each task used 500 training and 100 evaluation samples.Source: Reproducibility Statement · p. 10
  • Training used rank-16 LoRA, top-k 0.1 masking, and 64 out-of-domain Alpaca calibration samples for gradient masks.The reported seed set was 42, 123, 456, 789, 1024, 2048, and 4096.Source: Reproducibility Statement · p. 10
  • Boundary analyses were single-seed, while a three-seed method-order check and an isolated-process equivalence check were also reported.Code and configuration files include training, attribution, mask-generation, BWT/statistics scripts, and pinned dependency manifests for both environments.Source: Reproducibility Statement · p. 10

Ethics Statement

The work reports no human subjects or sensitive data and identifies no contribution-specific negative societal consequences. Its data-use statement notes licensing constraints on the calibration data and some evaluation datasets.

Ethics Statement

  • The study involves no human subjects or sensitive data and introduces no new capabilities or datasets raising ethical concerns.The authors report no negative societal consequences specific to the contribution.Source: Ethics Statement · p. 10
  • MedQA and CaseHOLD lack affirmative license declarations, so the authors use them for research evaluation only.Alpaca calibration data is licensed CC BY-NC 4.0, so the authors make no commercial-use claim for models trained with it.Source: Ethics Statement · p. 10
  • The Creative task uses a synthetic writing set composed of model-generated outputs rather than user conversations.The Code task uses CodeAlpaca-20k (CC BY 4.0), and Math uses GSM8K (MIT).Source: Ethics Statement · p. 10

AI Use Disclosure

The disclosure assigns research design, methodology, analysis, and scientific conclusions to the authors, while noting AI assistance in coding and manuscript work. The accompanying material points to the procedure, qualifies attribution-cost measurements, reports a near-zero Wanda–gradient correlation, and documents seed-dependent statistics and a density-unmatched comparison.

AI Use Disclosure

  • AI tools assisted experiment-script development and debugging, literature search, and manuscript drafting and revision, while the authors claim responsibility for research design, methods, analysis, and conclusions.The reported results were produced by the described pipeline without AI-generated synthetic data and were verified by the authors.Source: AI Use Disclosure · pp. 10–11

A The End-to-End Procedure

  • The notation is collected in Table 1, while Figure 1(a) diagrams the end-to-end procedure and Algorithm 1 states it step by step.The supplied passage does not further describe the procedure’s individual steps.Source: A The End-to-End Procedure · p. 14

B Attribution Cost

  • The reported full-parameter profiling run records 13.8 GB peak GPU memory and 584.9 seconds per-task attribution time, but no reported method uses that configuration.The continual-learning attribution pass instead records about 0.73 GB transient allocation above the resident base model; this is not an incremental end-to-end peak-memory cost.Source: B Attribution Cost · pp. 14–15, Table 3 · p. 14

C Pruning Saliency Is Orthogonal to Gradient Importance

D Confounded Null-Result Statistics

  • A paired test on per-seed BWT finds no reliable method difference: t(4) = 0.41, p = 0.70, with small Cohen’s dz = 0.18.The direction varies by seed; normalizing BWT by zero-shot loss does not change method ranking.Source: D Confounded Null-Result Statistics · p. 16

E Within-lora_B Comparison (Density-Unmatched)

  • The density-unmatched within-lora_B comparison uses the same parameter pool, but the gradient mask activates approximately twice as many B parameters as random-B-only.The passage identifies the density-matched control as the headline comparison in the main text.Source: E Within-lora_B Comparison… · p. 16

F Variation Across Runs and Method-Order Variation

The follow-up analyses qualify the density-matched forgetting result: within-session comparisons underpin inference, and several checks expose boundaries involving task order, evaluation, and run variability.

F Variation Across Runs and Method-Order Variation

  • Inference uses within-run paired comparisons: protocol paths agree within 9.6 × 10^-5, while method order and session identity can shift absolute BWT.The three-seed order-swap check did not reverse the contrast; cross-session spread is method-specific and of unknown origin.Source: F Variation Across Runs and… · pp. 16–17

G RRM Boundary Analysis; H Sparsity Sweep

I Training Stability: Rank-8 No-Mask Divergence

  • The rank-8 no-mask endpoint BWT of 0.176 follows a task-4 loss spike above 14 on three prior tasks, then recovery; similar instability recurs at rank 16 in a minority of seeds.The authors do not interpret the rank-8 endpoint as evidence that no-mask outperforms masking.Source: I Training Stability: Rank-8 No-Mask… · p. 19

K Generalization and Boundary Checks; O The Math Task: Truncation Audit

J Mask Concentration by Module; M B=0 Restriction Asymmetry

  • Gradient attribution selects 29% of o_proj parameters versus the 10% uniform rate; restricting random selection to lora_B improves BWT by 0.025, while that restriction is a no-op for gradient attribution.The restriction removes lora_A dilution from the random arm and makes the pool-matched comparison informative.Source: Figure 8 · p. 20, M B=0 Restriction Asymmetry · p. 21, Table 15 · p. 21

L In-Domain Calibration Ablation; N Per-Seed Accuracy and Matched Loss Contrast

R Ruling Out a Learning Deficit in the D1 Gap

S Cross-Architecture Density-Matched Replication (Phi-2)

On Phi-2, a density-matched Random-B versus Grad-B comparison reproduces the lower loss-based BWT for random masking, with Medical the largest contributor.

S Cross-Architecture Density-Matched Replication (Phi-2)

  • Both Phi-2 arms select exactly 1,048,576 coordinates, or 10% of the lora_B pool, leaving only the ranking rule different.The same budget is 5.0% of the 20,971,520 trainable A+B pool.Source: S Cross-Architecture Density-Matched… · pp. 24–25
  • −0.1152 mean paired contrast (Random-B − Grad-B in loss-based BWT) across four Phi-2 seeds, with all four favoring Random-B.The paired t-test reports p = 0.0009, but the authors base the replication claim on density matching and 4/4 directional agreement, given the small sample.Source: S Cross-Architecture Density-Matched… · p. 25, Table 22 · p. 25
  • Medical contributes 67.2% of Phi-2’s aggregate contrast, followed by Math at 24.1%, Code at 5.8%, and Legal at 2.9%.Medical is the largest contributor in every Phi-2 seed; its share is 79.9% on Qwen2.5-3B, so the effect is less concentrated on Phi-2.Source: S Cross-Architecture Density-Matched… · p. 25, Table 23 · p. 25
  • The replication matches D1 on five task domains, LoRA rank, calibration set, and 10% lora_B budget, but differs in model family, Phi-2 target set, and number of seeds.The Phi-2 LoRA target set omits dense, giving a 20,971,520-parameter trainable pool.Source: S Cross-Architecture Density-Matched… · p. 25
Loading 2610.12744v1…