PaperlayerView exact revision on arXiv

Source-linked AI summary

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

Zhiqiang Pang, Zihong Sun, Qi Xie, Jun Shu, Deyu Meng, Zongben Xu

arXiv:2610.11620v1cs.AI

TL;DR

PEFT can adapt LLMs to specialized tasks, but may degrade pretrained general capabilities, and replay or regularization approaches rely on additional data or optimization constraints. This paper selects layers for LoRA using forward-only input–output cosine similarity as a proxy for empirical Fisher sensitivity. Across evaluated models and tasks, LS-LoRA improves average target-task performance while retaining more commonsense reasoning capability than all-layer LoRA.

  • Problem

    PEFT can improve target-task performance at the expense of pretrained general capabilities, while existing mitigation approaches primarily rely on replay data or regularization constraints.

    Source: §1 · pp. 1–2

  • Method

    LS-LoRA uses input–output cosine similarity as a forward-only proxy for target-task layer sensitivity and places LoRA adapters only in low-similarity layers.

    Source: §1 · p. 2

  • Results

    Across four evaluated models on mathematical reasoning and code generation, LS-LoRA achieves higher average target-task and commonsense reasoning scores than all-layer LoRA.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support selecting where to adapt as a way to balance target-task learning and capability retention in the evaluated settings.

    Source: §5 · p. 9

    The stated sensitivity bounds assume positive minimum input–output norm products and lower bounds on average directional task sensitivity, with relative approximation error below one.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input--output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.

1 INTRODUCTION

PEFT reduces the cost of adapting LLMs, but target-task gains may come at the expense of pretrained general capabilities. This paper asks where adaptation should be applied and proposes selecting layers using input–output cosine similarity.

1 INTRODUCTION

  • PEFT, including LoRA, reduces fine-tuning costs by optimizing only a small number of parameters, but may still cause forgetting of pretrained general capabilities.Source: §1 · p. 1
  • Existing approaches primarily use replay data or regularization constraints to mitigate capability degradation, leaving layer choice as an underexplored question.Source: §1 · p. 2
  • Fine-tuning different Transformer layers can produce different target-task gains and different degrees of general-capability degradation.Figure 1 shows independent single-layer fine-tuning on Math-10K, with target-task performance on the left and general-capability retention on the right.Source: §1 · p. 2, Figure 1 · p. 2
  • LS-LoRA uses input–output cosine similarity as a forward-only proxy for target-task layer sensitivity and trains adapters only in layers with low similarity.The paper reports that this proxy has consistent negative rank correlations with empirical Fisher scores across models and tasks.Source: §1 · p. 2
  • Across mathematical reasoning and code generation, LS-LoRA improves average target-task performance while reducing degradation of evaluated general capabilities.Source: §1 · p. 2

2 RELATED WORK

Prior work reduces adaptation costs with parameter-efficient methods and addresses capability degradation through replay or regularization. Layer-wise approaches and representation-based criteria provide related strategies, while this work connects a forward-only selection criterion to empirical Fisher sensitivity.

2.1 PARAMETER-EFFICIENT FINE-TUNING

  • Prior PEFT methods include adapters, prefix and prompt tuning, LoRA, and AdaLoRA, which allocates rank budgets using importance scores estimated during training.Source: §2.1 · p. 2
  • Unlike replay and regularization, this work uses target-task empirical Fisher scores to analyze layer sensitivity and investigate layer selection for balancing adaptation and capability retention.Source: §2.1 · p. 2

2.2 LAYER-WISE ANALYSIS AND SELECTIVE ADAPTATION

  • This work selects a fixed set of Transformer layers for LoRA before fine-tuning, using input–output cosine similarity without response labels or backpropagation during selection.Source: §2.2 · p. 3
  • Related layer-selection approaches include freezing predefined subsets, stochastic activation, CKA-based selection, similarity-based LoRA, and diagnostics for PEFT module placement.ShortGPT uses input–output similarity to identify layers for pruning.Source: §2.2 · pp. 2–3

3 METHOD

The method uses layer-wise empirical Fisher to characterize target-task sensitivity, then uses input–output cosine similarity to select layers for LoRA adaptation. LS-LoRA trains adapters only in low-similarity layers while keeping pretrained weights frozen.

3 METHOD

  • The method uses Fisher information to examine layer differences, then uses forward-computable input–output cosine similarity to choose layers for fine-tuning.The approach centers layer selection rather than applying the same adaptation configuration across Transformer layers.Source: §3 · p. 3

3.1 PRELIMINARIES

  • Fisher information relates small parameter perturbations to local distribution shifts; diagonal empirical Fisher entries can be accumulated from squared gradients without forming the full matrix.The layer sensitivity score averages these entries over a layer’s parameters.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 FISHER-BASED LAYER SENSITIVITY IN LLM FINE-TUNING

  • Figure 2 plots Fisher scores above cosine similarities; Fisher scores generally fall from shallow to deeper layers, then rise in final layers for two models and tasks.The observed layer variation motivates considering sensitivity when selecting layers for adaptation.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 LAYER SELECTION VIA INPUT–OUTPUT SIMILARITY

  • A single forward pass scores each layer from average last-token input–output similarity across calibration inputs, without response labels or gradients.The proxy is motivated by the association between lower similarity and higher layer sensitivity.Source: §3.3 · pp. 4–5
  • −0.76 to −0.91 Spearman correlations link layer-wise cosine similarity and empirical Fisher across evaluated models and tasks; all reported p-values are below 10^-5.The results support using similarity as a practical proxy for ranking layer sensitivity.Source: Table 1 · p. 5, §3.3 · p. 5
  • LS-LoRA selects layers with input–output cosine similarity below threshold γ, trains adapters only there, and keeps pretrained weights frozen.The selected layers are fixed before fine-tuning; only their adapter parameters are trained using target-task supervised loss.Source: §3.3 · pp. 5–6, Figure 4 · p. 6

4 EXPERIMENTS

Experiments compare LS-LoRA with all-layer LoRA and regularization baselines across four models, assessing math and code performance alongside commonsense retention. LS-LoRA leads on the reported averages, while ablations support similarity-based selection and show efficiency gains.

4.1 EXPERIMENTAL SETUP

  • Experiments use four Llama and Mistral models, math and code training tasks, commonsense benchmarks for retention, and Base, all-layer LoRA, L2-SP-LoRA, and EWC-LoRA baselines.LS-LoRA selects layers using 100 target-task inputs, a fixed γ = 0.85 threshold, and a forward-only pass; selected layers stay fixed and pretrained weights remain frozen.Source: §4.1 · pp. 6–7

4.2 MATHEMATICAL REASONING

  • LS-LoRA achieves the highest average math and commonsense accuracy among compared fine-tuning methods on all four models, improving over all-layer LoRA by 0.4–1.1 and 6.0–14.3 percentage points, respectively.On Llama-3.2-3B, it narrows the commonsense accuracy drop from 15.3 to 1.0 points relative to Base while raising math accuracy from 65.8% to 66.9% over LoRA.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 CODE GENERATION

  • LS-LoRA attains the highest mean of reported pass@k scores and highest average commonsense accuracy among compared fine-tuning methods on all four models.It improves average commonsense accuracy over all-layer LoRA by 1.2–6.2 points; on Llama-3.2-3B and Llama-3-8B it exceeds Base, while on the other two models it remains below Base.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 ABLATION AND EFFICIENCY ANALYSIS

  • On Llama-3.2-1B, LS-LoRA outperforms Rand-LoRA, which adapts the same number of layers, on average math accuracy (52.2% vs. 51.4%) and commonsense accuracy (31.8% vs. 27.1%).The reported advantage holds across all four evaluated models; random selection also improves retention over all-layer LoRA.Source: Table 4 · p. 8, §4.4 · p. 8
  • Across four models, LS-LoRA reduces peak GPU memory by 4.2–14.4% and training time by 10.0–29.6% on MATH-10K relative to LoRA.For Llama-3.2-3B, peak memory falls from 13.1 to 11.8 GB and training time from 1.7 to 1.2 hours.Source: §4.4 · p. 8, Table 5 · p. 9
  • Last-token and token-averaged layer-similarity rankings agree strongly, with Spearman correlations of 0.92–0.93 across three Llama models.The experiments therefore use the last non-padding token as an approximation to token averaging.Source: Table 6 · p. 9, §4.4 · p. 9
  • On Llama-3.2-3B, similarity scores from 1, 10, and 100 samples correlate with a 1,000-sample reference at 0.938, 0.990, and 0.999, respectively.Scoring with 100 samples takes 2.8 seconds, and the experiments use 100 calibration samples.Source: Table 7 · p. 9, §4.4 · p. 9
  • On Llama-3.2-3B, applying the selection strategy improves DoRA math and commonsense averages from 66.0% to 67.0% and 37.9% to 58.9%, respectively.For PiSSA, the corresponding averages rise from 66.2% to 67.6% and from 59.0% to 64.7%.Source: §4.4 · p. 9

5 CONCLUSION

The paper examines layer placement as a way to balance target-task learning and capability retention. Across the evaluated models and tasks, LS-LoRA improves average target-task and commonsense scores over all-layer LoRA, with lower memory and training time in math experiments.

5 CONCLUSION

  • Across four evaluated models, LS-LoRA achieves higher average target-task and commonsense reasoning scores than all-layer LoRA on math and code tasks.The conclusion also reports reduced memory use and training time in the mathematical reasoning experiments.Source: §5 · p. 9
  • The method uses a forward-only input–output cosine similarity criterion, motivated by its consistent negative association with empirical Fisher sensitivity.Layer selection requires neither response labels nor backpropagation.Source: §5 · p. 9
  • Comparisons with random layer selection support the selection criterion, while DoRA and PiSSA results suggest applicability beyond LoRA.The conclusion frames adaptation location as a practical design choice for efficient fine-tuning and capability retention.Source: §5 · p. 9

A PROOF OF THE LOCAL DISTRIBUTIONAL SENSITIVITY PROPOSITION

The proof defines a KL-divergence function around a parameter perturbation and uses derivatives of the normalization identity. It concludes by applying a second-order Taylor expansion at zero.

A PROOF OF THE LOCAL DISTRIBUTIONAL SENSITIVITY PROPOSITION

B COSINE SIMILARITY AND AN EMPIRICAL FISHER LOWER BOUND

The appendix gives conditions for a lower bound connecting cosine similarity and empirical Fisher sensitivity. Its proof uses a shared set of target-task examples and assumptions on local parameter directions and average directional sensitivity.

B COSINE SIMILARITY AND AN EMPIRICAL FISHER LOWER BOUND

  • The scores considered in the appendix use the same target-task examples.The full-sequence formulation defines the log-likelihood under teacher forcing and uses representations at all positions.Source: B COSINE SIMILARITY AND AN EMPIRICAL… · p. 12
  • The full-sequence setup assumes differentiability, nonzero layer parameters, and that dependence on those parameters passes through the layer output.The layer input is assumed independent of the layer parameters.Source: B COSINE SIMILARITY AND AN EMPIRICAL… · p. 12
  • The average directional task-sensitivity condition requires sensitivity along residuals on average, but not a positive lower bound for each example.Source: B COSINE SIMILARITY AND AN EMPIRICAL… · p. 13
  • The local parameter direction need only approximate the residual in its effect on the task; errors orthogonal to output gradients are unrestricted.A residual sublayer with a nonlinear map is given as an example satisfying the exact local condition, not the average task-sensitivity condition.Source: B COSINE SIMILARITY AND AN EMPIRICAL… · p. 13

C SPEARMAN’S RANK CORRELATION

This section defines Spearman’s rank correlation and applies it to compare layer-wise empirical Fisher scores with input–output cosine similarities.

C SPEARMAN’S RANK CORRELATION

D ADDITIONAL RANDOM LAYER SELECTION RESULTS

Across three models, LS-LoRA achieves higher average mathematical and commonsense reasoning accuracy than random selection of the same number of layers.

D ADDITIONAL RANDOM LAYER SELECTION RESULTS

E APPLICABILITY TO OTHER PEFT METHODS

On Llama-3.2-3B, applying the input–output similarity criterion improves average accuracy for layer-selective DoRA and PiSSA on mathematical and commonsense reasoning benchmarks.

E APPLICABILITY TO OTHER PEFT METHODS

Loading 2610.11620v1…