PaperlayerView exact revision on arXiv

Source-linked AI summary

Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

Wentao Ye, Zhanming Shen, Zhiqing Xiao, Yao Ding, Haobo Wang, Gang Chen

arXiv:2609.00762v1cs.LG

TL;DR

Severe trainable-state budgets raise a subspace-selection problem: where coefficients act can matter as much as how many are updated. The paper uses frozen-core adaptation and FCCA to select task- and geometry-aware spans, achieving strong matched-budget results across models while retaining a much smaller trainable state.

  • Problem

    Under severe trainable-state budgets, it is unclear how to select update directions because parameter count alone does not determine where coefficients act.

  • Method

    FCCA estimates signed input–error cross-covariance, whitens it with diagonal Fisher moments, truncates in Fisher coordinates, maps directions back, and QR-stabilizes frozen bases while training only an r×r core.

  • Results

    Across 11 tasks and four model settings, FCCA ranks first on all three Qwen scales and is essentially tied for first on Llama-3.2-1B; on Qwen2.5-3B it reaches an 83.0 mean, 2.3 points above the next-best matched-budget constructor.

  • Takeaways & Limitations

    A carefully selected fixed span can approach movable low-rank adapters while substantially reducing trainable and optimizer-state requirements.

  • Takeaways & Limitations

    The study uses three seeds, one software stack, one GPU family, limited Llama coverage, and ablations spanning only one to three tasks, so conclusions may be regime specific.

Abstract

from arXiv · show

Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an $r\times r$ core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched $r^2$ budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost.

1 Introduction

Under severe trainable-state budgets, frozen-core adaptation makes the choice of update span as important as parameter count and directly exposes subspace quality. FCCA selects spans using signed task signal and local Fisher geometry before truncation, then stabilizes their trainable core coordinates.

  • Extreme budgets make the location of trainable coefficients consequential because task-relevant spans can outperform inherited, random, or poorly scaled directions.
  • Frozen-core adaptation fixes left and right bases during calibration and trains only an r×r core, preventing span rotation during fine-tuning.
  • FCCA preserves signed input–error cross-covariance while imposing rank in a Fisher-scaled local metric, combining task direction with loss geometry.
  • 83.0 mean: FCCA leads the next-best matched-budget constructor by 2.3 points on Qwen2.5-3B and exceeds RawGrad on all 11 tasks.
  • FCCA ranks first across the three Qwen scales, nearly ties the best Llama result, and remains close to LoRA and DoRA with far fewer trainable values.

2 Related Work

Related methods reduce trainable state through varied parameterizations or fixed spans, but the controlled frozen-core setting isolates the quality of the selected subspace. FCCA differs by combining signed task pairing with marginal geometries before truncation.

  • LoRA and DoRA use trainable low-rank parameterizations, while PiSSA, AdaLoRA, and related methods vary initialization or rank allocation.
  • The frozen-core evaluation freezes both bases and trains only a square core, isolating selected-subspace quality from basis rotation during optimization.
  • Fixed-span alternatives include random, weight-SVD, activation-variance, Fisher-marginal, and task-gradient constructions.
  • FCCA applies local-geometry rescaling to subspace selection while preserving signed input–error cross-covariance, unlike independent input and output eigenspace selection.
  • Among the tested constructors, FCCA alone combines the signed cross term with both marginal geometries before truncation.

3 Method

FCCA constructs frozen bases by estimating task cross-covariance and diagonal Fisher moments, truncating in whitened coordinates, mapping back, and applying QR. The resulting adapter trains only a square core while preserving the selected update family.

  • 3.1 Frozen-core adaptation: Only the square core R is trainable in the update y = (W + sPRQ⊤)x, giving an exact Lr^2 trainable budget across L target matrices.
  • 3.1 Frozen-core adaptation: At zero core, the projected gradient norm ∥P⊤GQ∥F directly measures how much signed task gradient the frozen spans capture.
  • 3.2 Fisher-whitened cross-covariance: FCCA estimates signed cross moment G and diagonal input/output moments, avoiding dense covariance fitting from only a few hundred calibration examples.
  • 3.2 Fisher-whitened cross-covariance: The rank constraint is imposed after Fisher whitening, where truncated SVD yields the local rank-r solution and inverse scaling maps directions back to raw bases.
  • 3.2 Fisher-whitened cross-covariance: Thin QR orthonormalizes the inverse-whitened bases without changing the representable update family, providing coordinates suitable for common core optimization.
  • Implementation: Calibration accumulates one cross moment and diagonal moments per target, then performs truncated SVD; cross moments can dominate host memory.
  • Implementation: The efficiency claim concerns trainable and optimizer state: fixed bases still require r(m+n) storage when retained separately for each task.

4 Experimental Setup

The experiments evaluate matched-budget frozen-core constructors across multiple models, tasks, and seeds under a common rank-16 protocol. Task-specific metrics and held-out validation determine the reported comparisons.

  • Models and tasks: The study uses Qwen2.5-3B as the primary model, three Qwen scales and one Llama setting for scale and architecture checks.
  • Models and tasks: The 11-task suite spans math, language understanding, and multiple-choice benchmarks, with fixed subsets of 300 math examples and 500 remaining-task examples.
  • Common adapter protocol: Every method uses rank r = 16, adapts all q/k/v/o projections, initializes the core to zero, and trains one r×r core per target matrix.
  • Common adapter protocol: Qwen2.5-3B therefore has 36,864 trainable parameters under the matched-budget protocol.
  • Evaluation and selection: Math tasks use greedy decoding with exact match, while the remaining tasks use per-choice conditional log likelihood.
  • Evaluation and selection: Three seeds and held-out validation support reporting mean ± standard deviation and selecting checkpoints and hyperparameters.

5 Results

FCCA leads matched-budget frozen-core comparisons across Qwen scales and remains near the best Llama result, while whitening and task-matched calibration strengthen performance. Its fixed span approaches LoRA and DoRA accuracy with far fewer trainable parameters, but movable factors retain advantages on some tasks.

  • 3B matched-budget results: 83.0 mean places FCCA 2.3 points above LoRA-XS-core, with positive differences against RawGrad on all 11 Qwen2.5-3B tasks.The largest RawGrad gains are +10.0 on OpenBookQA, +7.4 on HellaSwag, and +18.8 on WinoGrande.
  • Scale and architecture: FCCA ranks first on Qwen2.5-1.5B, 3B, and 7B, and remains within 0.1 point of the best Llama-3.2-1B result.Its Qwen means are 77.2, 83.0, and 87.5, with leads of 0.5, 2.3, and 0.8 points over the strongest alternatives.
  • Whitening ablation: Whitening improves every tested mechanism task, with two-sided gains of 2.7–17.2 points over unscaled SVD.Two-sided whitening is best or tied on four of six tasks, while input-only whitening is slightly better on ARC-C and RTE.
  • Calibration coverage: Accuracy can plateau before the selected basis converges to the largest-calibration reference, and representative task-matched calibration outperforms mismatched-source calibration.FCCA remains 5.9–7.3 points above RawGrad across tested SVAMP calibration sizes, while GSM8K calibration costs 1.7 points on SVAMP and ARC-C and SST-2 calibration cost 5.1 and 7.1 points.
  • Accuracy–state trade-off: 83.0 mean is only 0.318 below LoRA and 0.227 below DoRA while FCCA optimizes 36,864 rather than 7.37–7.54M parameters.LoRA and DoRA retain their clearest advantages on OpenBookQA and ARC-C, where span revision may be more useful, although the comparison does not establish that cause.
  • Runtime and storage: FCCA adds a one-time 35-second calibration pass, while its projected 4.1-minute core-only run is close to LoRA and below DoRA and PiSSA.The larger practical savings are in gradients, optimizer state, and communication; frozen bases still impose storage costs and task-specific deployment copies.

6 Conclusion and Discussion

FCCA combines signed task signal, Fisher-scaled truncation, and QR-conditioned coordinates for frozen-core adaptation. It performs strongly across model settings while using far less trainable state than movable-basis references.

  • Conclusion: FCCA leads at all Qwen scales, nearly ties on Llama-1B, and approaches LoRA and DoRA with roughly 200× less trainable state.The efficiency claim concerns trainable and optimizer state while keeping calibration and basis-storage costs explicit.
  • Conclusion: RawGrad’s deficit on all 11 tasks isolates the metric applied at truncation because both methods use the same signed cross-moment.The comparison separates Fisher-scaled truncation from access to additional task supervision.
  • Conclusion: Frozen-core accuracy diagnoses subspace quality and models state-constrained adaptation, without implying that practical adapters must always freeze their bases.The practical lesson is to preserve signed task signal, use moderate rank, and expose the retained span through trainable coordinates.

Limitations

The study’s conclusions are bounded by its evaluation protocol, computational accounting, and experimental coverage. Its evidence supports regime-specific design guidance rather than universal laws or leaderboard comparisons.

  • Evaluation scope: Evaluation uses fixed subsets of 300 math examples and 500 examples for other tasks, so absolute scores are not directly comparable with full-test leaderboards.The 11-task macro-average weights tasks equally.
  • Experimental coverage: The study uses three seeds, one software stack, and one GPU family, while several ablations cover only one to three tasks.The authors state that some design conclusions may be regime specific and do not evaluate instruction following or broader open-ended generation beyond exact-match math.
  • Resource accounting: Calibration requires a backward pass and cross-covariances that can consume host memory, while the r2 budget excludes complete adapter storage.The fixed bases still occupy r(m+n) storage when retained separately for each task.

B.4 Calibration implementation

The implementation fixes calibration-derived bases and trains only a shared square core under a common frozen-core protocol. Baseline conversions and statistical reporting define how constructor comparisons should be interpreted.

  • Calibration: Calibration collects cross moments and diagonal second moments on CPU, then discards statistics after SVD and QR, retaining only bases and cores.Diagonal whitening is the default; dense K-FAC is reserved for a controlled comparison.
  • Statistical reporting: Results report means and sample standard deviations over seeds 42, 43, and 44, with paired Wilcoxon tests using tasks as the replication unit.The rank test is primary because the 11 task differences include large outliers and provide little basis for a normality assumption.
  • Comparison design: The “-core” suffix denotes controlled frozen-basis instantiations rather than reproductions of the original methods’ complete parameterizations or native budgets.This normalization makes basis-selection ideas comparable under the shared square-core architecture.
  • Constructor controls: RawGrad factorizes the same signed task cross moment as FCCA without whitening, making it the cleanest control for the effect of A and D.FiLoRA-core instead selects marginal curvature eigenspaces without preserving the signed input–error pairing.

D Full Cross-Scale Results

Across model scales, FCCA consistently ranks among the strongest matched-budget constructors, leading all Qwen settings and remaining near the top on Llama. Its average performance approaches trainable-factor adapters while retaining the frozen-core budget advantage, though a separate PiSSA run shows sensitivity to learning-rate choice.

  • FCCA ranks first across the three Qwen scales and remains within 0.13 points of the best Llama result.
  • FCCA comes within 0.318 points of LoRA and 0.227 points of DoRA on the Qwen2.5-3B 11-task mean while training about 200× fewer parameters.
  • The comparison supports near-parity with full trainable factors on average, not dominance over them.
  • An earlier Qwen2.5-7B PiSSA GSM8K run at learning rate 10−3 diverged, whereas rerunning at 2 × 10−4 achieved 77.2±2.1.

F Task-Level Significance

The significance and ablation analyses examine whether FCCA’s gains are robust to testing choices, whitening configuration, calibration size, and task mismatch. Results favor whitening and modest task-matched calibration, while statistical inference remains limited by unadjusted comparisons and only 11 task pairs.

  • Significance: Cross-task significance tests are unadjusted, and the 11 task pairs provide limited resolution and power for inference.
  • Whitening ablations: Every whitened variant exceeds the no-whitening SVD on every populated task, with full whitening gaining 2.7–17.2 points over None.
  • Whitening ablations: The exact SVAMP control preserves the same whitening ordering under fixed non-whitening hyperparameters, supporting geometry-aware normalization without guaranteeing two-sided whitening on every task.
  • Calibration size: Accuracy is nearly flat from 64 to 512 calibration examples, while principal-angle cosines exceed .93 on both sides at 128 examples.
  • Calibration size: On SVAMP, FCCA remains 5.9–7.3 points above RawGrad across calibration sizes, whereas ARC-C varies by only 0.4 point.
  • Calibration mismatch: Mismatched calibration degrades gracefully in the tested setting, with science QA and sentiment calibration losing 5.1 and 7.1 points, respectively.

G.5 QR coordinates and dense K-FAC

QR stabilizes optimization without expanding the representable update family, while diagonal whitening outperforms the tested dense K-FAC alternatives in the limited-calibration regime. The efficiency comparison concerns trainable and optimizer state, not total frozen artifact size, and several optimization alternatives remain untested.

  • QR coordinates: Without QR, SVAMP and GSM8K diverge to infinite loss and zero exact match, while ARC-C falls to 22.9.
  • QR coordinates: QR adds no expressive directions but makes a common core optimizer usable in these runs.
  • QR coordinates: Retuning the no-QR learning rate does not rescue SVAMP or GSM8K, and ARC-C peaks at 22.7, near four-choice chance.
  • Scope: Specialized preconditioners and layer-specific optimization remain untested.
  • Dense K-FAC: Diagonal whitening exceeds dense K-FAC by 8.7 points on SVAMP, 0.4 on ARC-C, and 1.7 on GSM8K using 128–256 calibration examples.
  • Trainable-state efficiency: At rank 16, FCCA trains 36,864 values versus 7,372,800 for the factorized reference, exactly a 200-fold difference.
  • Storage boundary: The complete unmerged FCCA artifact still contains 7,372,800 frozen basis elements plus the 36,864-value core.

H.3 Software and implementation checklist

The implementation uses a fixed hardware and software stack, explicit model and training settings, and three-seed evaluation. Runtime accounting separates one-time calibration or initialization from projected optimizer-step costs.

  • Environment: Experiments use one NVIDIA H20 with CUDA 12.8, PyTorch 2.7.1, Transformers 4.54, PEFT 0.16, and bf16.
  • Models and seeds: The evaluated models are Qwen2.5-1.5B/3B/7B-Instruct and Llama-3.2-1B-Instruct, using seeds 42, 43, and 44.
  • Training configuration: All q/k/v/o projections use default rank 16, zero core initialization, and calibration sizes of 128 or 256.
  • Reporting: Reported results use held-out validation for configuration selection, followed by common three-seed evaluation with mean ± sample standard deviation.
  • Runtime accounting: Projected totals use approximately 262 optimizer steps and are not independent end-to-end stopwatch measurements.
Loading 2609.00762v1…