PaperlayerView exact revision on arXiv

Source-linked AI summary

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

Yangyi Huang, Ruotian Peng, Zeju Qiu, Jiale Kang, Yandong Wen, Bernhard Schölkopf, Weiyang Liu

arXiv:2605.28819v1cs.LGcs.CL

TL;DR

PEFT evaluations often emphasize downstream gains while overlooking pretrained-capability retention, motivating a stability-plasticity view. The paper introduces PEFT-Arena and geometric and pathwise analyses, finding distinct trade-offs across methods and a favorable frontier for OFT under comparable budgets, while identifying SFT overshoot and supporting path-wise rewinding case studies.

  • Problem

    PEFT evaluation often focuses on downstream performance while overlooking retention of pretrained capabilities, despite the stability-plasticity trade-off between acquiring and preserving abilities.

  • Method

    PEFT-Arena jointly evaluates target-domain performance and general capability retention, then analyzes updates through weight-space spectra, activation-space geometry, and parameterization-aware interpolation.

  • Results

    Across methods, stability-plasticity profiles differ, with OFT often providing a strong frontier under comparable parameter budgets; final SFT checkpoints often overshoot a better target-retention operating point.

  • Takeaways & Limitations

    PEFT methods should be assessed by both target adaptation and general-capability retention, while update geometry and pathwise control help characterize their trade-offs.

  • Takeaways & Limitations

    PEFT-Arena covers two reasoning-oriented target domains and mainly uses English general-evaluation suites, leaving broader multilingual, dialogue, and safety evaluations for future work.

Abstract

from arXiv · show

Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.

1 Introduction

PEFT-Arena reframes PEFT evaluation as a stability-plasticity dilemma, jointly considering downstream adaptation and pretrained-capability retention. It combines benchmark evaluation with weight-space, activation-space, and pathwise analyses to explain trade-offs and motivate rewinding.

  • Motivation: PEFT evaluation should consider what models learn relative to what they forget, not downstream performance alone.The paper frames this tension as the stability-plasticity dilemma between acquiring new capabilities and preserving existing ones.
  • PEFT-Arena: PEFT-Arena jointly measures target-domain performance and general capability retention across mathematics and medicine.The benchmark treats target performance as plasticity and retained general capability as stability.
  • Benchmark findings: Different PEFT parameterizations produce distinct stability-plasticity trade-offs, with OFT often lying on a strong frontier under comparable budgets.The authors suggest that update geometry plays an important role in preserving general capabilities.
  • Geometric analysis: Weight-space analysis examines how additive low-rank, spectral-initialization, and orthogonal updates interact with pretrained spectral structure.These parameterizations reshape the pretrained basis in distinct ways.
  • Geometric analysis: Activation-space diagnostics measure non-isometric distortion with Procrustes residual, pairwise Gram distortion, and linear CKA.The analysis links forgetting to damage in the relational geometry of general-capability representations and finds that OFT better preserves this structure.
  • Pathwise diagnosis: Interpolation reveals that final SFT checkpoints often overshoot the best target-retention operating point.The paper uses parameterization-aware paths, including scaling additive updates and OFT’s skew-symmetric Cayley generator, to diagnose this behavior and motivate layer-wise rewinding.

2 The PEFT-Arena Benchmark

PEFT-Arena evaluates target-domain adaptation and general-ability retention across methods, domains, and post-training settings. Its results show strong stability-plasticity differences, with OFT providing a favorable SFT frontier and RLVR generally producing less forgetting.

  • Evaluation setup: The benchmark uses target-domain performance as plasticity and general-task performance as stability under SFT and RLVR.Experiments cover mathematics and medicine, with average accuracy reported for each domain.
  • Evaluation setup: The evaluation covers math and medical target benchmarks alongside IFEval, NQ, and BBH for general ability retention.The General score averages performance across instruction following, natural language understanding, general knowledge, and general reasoning tasks.
  • SFT results: SFT improves target performance at the expense of general ability, and target-only reporting can overestimate post-training quality.Full FT provides the largest target gains but also the most severe forgetting in the reported examples.
  • SFT results: Under SFT, OFT-b32 improves Qwen math target accuracy by 11.63 with only a 2.60 drop on math general, forming the strongest PEFT frontier.The comparison uses roughly matched trainable-parameter budgets, including a roughly 20M-parameter group.
  • RLVR results: RLVR generally enables stable adaptation with less forgetting than SFT.On Qwen math, OFT improves target performance by 12.60 while increasing math-general performance by 1.93; OFT reaches 47.90 target accuracy with 17.55M trainable parameters.
  • RLVR results: Longer GRPO training can degrade high-k sampling performance while leaving Pass@1 relatively stable.Full FT and LoRA show larger Pass@64 drops than OFT, resembling over-adaptation from another evaluation perspective.
  • Additional validation: The reported General-axis findings are consistent with expanded validation on HumanEval, HellaSwag, WinoGrande, MMLU, ARC, and GSM8K.These additional benchmarks broaden coverage of general capabilities.

3 Understanding PEFT Updates through Internal Geometry

PEFT updates are analyzed through weight-space spectral geometry and activation-space representation geometry to connect parameterization choices with capability retention. Non-isometric distortion of general-evaluation representations is associated with forgetting, while OFT better preserves relational geometry.

  • Weight-Space Geometry: Weight-space analysis characterizes PEFT updates through retention profiles that measure pretrained singular alignment and adaptation profiles that locate injected update energy.The adaptation profile captures both scaling changes and off-diagonal rotations, but is descriptive rather than a standalone explanation of target gains.
  • Weight-Space Geometry: PiSSA and MiSS show large retention-side deviations, LoRA has spiky update-energy allocation, and OFT maintains a more structured retention profile.The fluctuation score summarizes local irregularity in the retention and update-energy spectra.
  • Activation-Space Geometry: Activation-space diagnostics compare base and finetuned representations on general data using Procrustes residual, linear CKA, and pairwise Gram distortion.These metrics distinguish benign shared rotations from non-isometric changes in relational structure.
  • Activation-Space Geometry: Procrustes residual strongly correlates with forgetting, while linear CKA and pairwise Gram distortion support the interpretation that retention depends on representation geometry.The correlations are measured across 20 SFT checkpoints and eight module locations.
  • Activation-Space Geometry: OFT exhibits lower non-isometric distortion and higher CKA than LoRA and full finetuning, whereas PiSSA shows the strongest distortion and most severe forgetting.The comparison is reported for activation geometry on general data.
  • Interpretation: The retention-side diagnostics are not used to explain target-task gains, which may depend on task-aligned computation, answer margins, and multi-step reasoning.The paper presents PEFT general-capability retention as strongly associated with preserving general-data representation geometry.

4 Interpolation as a Pathwise Study

Interpolation traces the target-retention trade-off beyond the final checkpoint and reveals that SFT often overshoots a better operating point. Parameterization-aware paths and layer-wise OFT rewinding provide post-hoc ways to control this trade-off.

  • Pathwise Diagnostic: Interpolation traces how target performance and general retention change between the base and adapted checkpoints rather than serving as a new adaptation technique.For additive methods, the path scales the effective update ∆W.
  • Finetuning Overshoots the Best Trade-off: SFT final checkpoints often lie beyond a better target-retention operating point, so reducing update strength can recover general ability while preserving much of the target gain.The diagnostic identifies where stability is lost along the adaptation path.
  • Finetuning Overshoots the Best Trade-off: Training and interpolation trajectories differ, with the former described as concave and the latter convex; the paper therefore finds that early stopping cannot attain the optimal stability-plasticity trade-off.This distinction is presented as evidence that overshoot is not merely late-stage overfitting.
  • Parameterization-Aware Interpolation: For OFT, scaling the skew-symmetric generator along the Cayley path preserves the intended orthogonal geometry, unlike dense-weight interpolation.The natural path is R(α) = Cayley(√αQ).
  • Parameterization-Aware Interpolation: At α = 0.3, OFT-b32 reaches 45.77 math and 48.64 General on Qwen2.5-7B SFT-math, versus 43.93 math and 43.91 General under linear interpolation.The comparison uses the parameterization-aware √αQ path and dense-weight delta interpolation at the same α.
  • Layer-wise Interpolation for OFT: Layer-wise OFT rewinding improves the target-retention trade-off by rescaling each generator separately to address update-strength imbalance across depth.SafeScale uses the average strength of the first five layers as reference, while MinScale uses the weakest-updated layer.

5 Concluding Remarks

PEFT-Arena evaluates adaptation together with general-capability retention and links differences across methods to their internal weight and activation geometry. The paper argues that preserving pretrained geometry should accompany acquiring target ability.

  • Concluding Remarks: PEFT-Arena evaluates PEFT methods through the stability-plasticity trade-off rather than target accuracy alone.The benchmark jointly considers target gains and general-capability retention.
  • Concluding Remarks: OFT often defines a strong frontier at comparable parameter budgets, while PEFT parameterizations differ in how they exchange target gains for general-capability retention.The paper connects these differences to weight-space spectral profiles and activation-space distortion.
  • Concluding Remarks: The paper concludes that PEFT methods should be designed for both target ability acquired and pretrained geometry preserved.OFT shifts representations while preserving relational geometry, whereas PiSSA and full finetuning induce stronger distortion.

Limitations

The benchmark is scoped to selected domains, languages, PEFT families, and empirical diagnostics, leaving broader evaluation and causal explanation for future work.

  • PEFT-Arena evaluates mathematics and medicine as target domains and relies mainly on English general-evaluation suites.Broader multilingual, dialogue, and safety-oriented evaluations remain future extensions.
  • The method coverage emphasizes weight-parameterized PEFT methods, while prompt-, prefix-, and adapter-based families are not directly represented.Extending analogous evaluation and activation-space diagnostics to these families would improve comprehensiveness.
  • The internal analyses are empirical diagnostics rather than a complete causal theory of forgetting.They are strongest for retention-side representation changes, whereas reasoning-heavy plasticity may require additional task-aligned diagnostics.
  • The interpolation analysis is primarily supported by SFT target–retention curves, with longer RLVR providing complementary high-k evidence.This limits how broadly the pathwise conclusions should be generalized across training regimes.

B.1 Model and Training Details

Experiments use distinct compute and optimization configurations for SFT and RL, with substantial GPU requirements varying by model, method, and training regime.

  • Experiments run on 8× NVIDIA H100 80GB GPUs, with compute varying across models, methods, and training regimes.A typical LoRA SFT run requires approximately 100 GPU-hours for 7B models and 60 GPU-hours for 3B models; diagnostic analyses run offline.
  • SFT uses effective batch size 256, maximum response length 8192 tokens, four epochs, and cosine-decay learning rates.Full fine-tuning uses 5 × 10^-5, while PEFT methods use 2 × 10^-4.
  • RL uses rollout batch size 256, mini-batch size 64, group size 8, and maximum generation length 8192 tokens.Unless otherwise specified, RL runs for 200 steps; the longer-training study also uses 500 steps.

B.2 Evaluation Protocol

The evaluation protocol combines domain-specific target metrics with standardized general-capability assessments using publicly available artifacts and documented configurations.

  • The appendix describes evaluation settings for tasks spanning mathematics, medicine, and general capability.
  • Math uses average accuracy@16 with temperature T = 0.6, medical evaluation uses average accuracy at T = 0.0, and general evaluation uses one sample per query.All settings use an 8192-token maximum response length for math and a 1024-token context length for general evaluation.
  • The study uses publicly available checkpoints, PEFT implementations, datasets, and benchmark frameworks under their respective licenses and usage terms.Evaluated models, parameter counts, configurations, hyperparameters, decoding settings, and protocols are documented in the appendix.

C.1 Expanded General-Capability Benchmarks

Expanded benchmarks provide consistency checks for general capability, while supplementary spectral and alignment diagnostics characterize how PEFT methods alter pretrained weight geometry.

  • C.1 Expanded General-Capability Benchmarks: Expanded general-capability evaluation covers HumanEval, HellaSwag, WinoGrande, MMLU, ARC, and GSM8K as consistency checks beyond the main suite.Detailed per-task scores are reported in Table 6.
  • C.1 Expanded General-Capability Benchmarks: Detailed tables decompose average scores into constituent math, medical, and general benchmark tasks.Math averages cover Math-500, AIME24, and AMC23; medical averages cover eleven benchmarks; general averages cover IFEval, NQ, and BBH.
  • Larger retention fluctuation is significantly associated with lower general performance and more forgetting across matched checkpoints and the SFT subset.The retention-side score measures mean absolute deviation from a local moving average, using window size w = 5.
  • Full finetuning and PiSSA visibly disrupt retention-side spectral profiles, whereas OFT maintains a more coherent diagonal-projection profile.LoRA, MiSS, PiSSA, and AdaLoRA also show uneven update-energy allocation.
  • OFT-specific SVA compares corresponding pretrained and rotated singular vectors because OFT preserves singular values while rotating right singular vectors.RL-trained OFT shows relatively uniform rotation, while SFT-trained OFT shows localized spikes.

D.3 Capability-Conditioned Drift

Capability-conditioned drift measures how PEFT updates affect activations actually used by general and target distributions, complementing spectral weight-space profiles. General-distribution drift tracks forgetting, whereas target drift is not a reliable proxy for target gains.

  • CSD definition: Capability-conditioned spectral drift weights update directions by how frequently evaluation data activates them.For h expressed in the pretrained right singular basis, directions that are both strongly updated and frequently activated by dataset D contribute more to CSD.
  • CSD definition: CSD compares activation displacement for general and target distributions using absolute, relative, and update-normalized variants.The effective update is ΔW = W*−W0, with activations collected from the pretrained model on matched general and target examples.
  • Empirical interpretation: General CSD is positively associated with forgetting, while target CSD does not provide a monotonic explanation of target gain.The SFT-only correlation analysis covers 68 checkpoints; target-gain correlation is weak or negative.
  • Empirical interpretation: Raw CSD requires parameterization-aware interpretation because orthogonal rotations can produce substantial Euclidean output movement.For OFT, this motivates activation-geometry diagnostics alongside raw drift measures.
  • Activation geometry: Activation-geometry analysis uses Procrustes residual, CKA, and pairwise Gram distortion as primary retention-side diagnostics.Secondary measures separate pointwise movement from non-isometric representation distortion across tested module locations.

E.3 Layer/Module Method-Family Summaries

The interpolation analysis compares method-family activation geometry and parameterization-aware paths across layers and modules. OFT interpolation rescales its Cayley generator rather than linearly scaling dense weight differences.

  • Method-family summaries: Activation-geometry averages report Procrustes residual, Gram distortion, and CKA alongside method-family forgetting.Lower Procrustes and Gram distortion and higher CKA indicate better geometry preservation.
  • Interpolation paths: Additive PEFT interpolation scales the learned update ΔW, preserving the product structure when both LoRA factors are scaled.For LoRA, ΔW = sBA with method-dependent scale s.
  • Interpolation paths: OFT interpolation rescales the skew-symmetric Cayley generator Q, making the path parameterization-aware rather than a dense-weight interpolation.The rotation-strength coefficient α is applied through Q′ = √αQ in the small-angle regime.
  • Interpolation results: The full SFT interpolation curves extend the representative main-text analysis across PEFT methods and target domains.The OFT dense-delta versus Cayley-generator comparison is reported separately for Qwen2.5-7B math SFT with OFT-b32.

F.4 Layer-wise Rewinding Results

Layer-wise rewinding addresses uneven OFT update strength and can improve the stability-plasticity trade-off after finetuning. Related results show that rewinding benefits additive PEFT and can reveal degradation under longer RLVR evaluation.

  • Layer-wise rewinding: Later layers receive substantially larger OFT generator updates than early layers, motivating layer-wise rather than global rewinding.SafeScale and MinScale use the average squared Frobenius norm of generator parameters within each layer.
  • Rewinding alternatives: Figure 9 visualizes how scaling and pruning alternatives behave across OFT adapter sizes.These alternatives provide consistent improvement on OFT adapters of different sizes.
  • Layer-wise rewinding: MinScale improves both math and General for LoRA-r8 and MiSS-r8, indicating that rewinding is not limited to OFT.The result is reported on Qwen2.5-7B math SFT.
  • Longer-RLVR high-k interpolation: High-k RLVR follow-ups show that interpolation can improve both pass@64 and General in several longer-training settings.This pathwise degradation is less visible under pass@1.
Loading 2605.28819v1…