PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Tuning of Transformer models with Frames

Harshavardhan Adepu, Li Zhang, Sanjiv Kumar, Vikas Singh

arXiv:2608.26430v1cs.AI

TL;DR

Large-model fine-tuning and storing multiple task-specific models create substantial resource demands, while existing low-rank and sparse approaches have memory, scalability, or optimization trade-offs. FrameFT uses sparse coefficients in algorithmically generated Fusion Frame bases, and experiments across language and vision tasks report competitive performance with substantially fewer parameters.

  • Problem

    Fine-tuning large pretrained models is costly, while existing PEFT approaches can require memory-intensive dense bases or face sparse-pattern and hyperparameter scalability issues.

  • Method

    FrameFT represents parameter updates with sparse coefficient matrices over structured Fusion Frame subspaces generated algorithmically and shared across compatible layers.

  • Results

    FrameFT achieves substantial compute and parameter efficiency across language and vision benchmarks without sacrificing performance across many evaluations.

  • Takeaways & Limitations

    FrameFT offers a parameter-efficient alternative for fine-tuning transformer models when compute and storage efficiency are important.

  • Takeaways & Limitations

    The convergence analysis treats pretrained weights as frozen and the loss as a function of learnable Frame coefficients across layers.

Abstract

from arXiv · show

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $ΔW$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.

1. Introduction

FrameFT addresses the cost of fine-tuning large models by combining structured Fusion Frame subspaces with sparse updates. The framework targets efficient, expressive, stable, and scalable adaptation, supported by empirical evaluation and technical analysis.

  • Full-model fine-tuning is expensive, especially when separate copies are needed for multiple downstream tasks.
  • PEFT methods reduce resource needs by updating only a small subset of model parameters while potentially matching full-model fine-tuning.
  • LoRA freezes pretrained weights and injects task-specific low-rank matrices, but SVD-based variants require dense singular vectors that increase memory and affect inference throughput.
  • Sparse fine-tuning selectively updates model parameters, but large-scale use can be hindered by sparse-pattern identification and hyperparameter tuning.
  • FrameFT parameterizes updates across multiple structured Fusion Frame subspaces, combining LoRA-like global adaptability with sparse-update precision.
  • The paper contributes a Fusion Frame fine-tuning framework, broad evaluations against PEFT baselines, and analyses of stability, convergence, and efficiency.

2. Frames and Subspace decompositions

Frames represent vectors redundantly, while Fusion Frames extend this idea to collections of weighted subspaces. Their projection, synthesis, and frame operators support representation and reconstruction, with Parseval frames allowing exact reconstruction.

  • Spanning sets and Frames: Frames generalize spanning sets by allowing redundancy, enabling stable and often more robust vector representations.
  • Spanning sets and Frames: Figure 1 illustrates a tight frame with k = 5 vectors in R3, whose vectors can represent R3 signals without distortion through projection.
  • Spanning sets and Frames: A frame for a d-dimensional space uses frame bounds A and B to constrain inner-product energy for every vector, typically with k ≥ d.
  • Fusion Frames: Fusion Frames extend frames by decomposing a space into weighted subspaces with associated orthogonal projections.
  • Fusion Frames: The analysis uses Parseval fusion frames, characterized by A = B = 1, with equal-weight notation when all subspace weights are one.
  • Operators in Fusion Frames: Projection, synthesis, and fusion-frame operators map vectors to subspace representations, reconstruct vectors, and compose these transformations.
  • Operators in Fusion Frames: For Parseval fusion frames, the fusion frame operator equals the identity, allowing exact reconstruction; frame bounds later support convergence analysis.

3. Fine-tuning in Fusion Frame Subspaces

FrameFT replaces LoRA’s implicitly learned low-rank subspaces with structured Fusion Frame subspaces and sparse trainable coefficients, aiming to improve control, efficiency, and optimization stability. Its algorithmically generated, block-sparse frames and shared sparsity patterns reduce storage and computation while preserving smoothness-based convergence guarantees.

  • LoRA subspaces: LoRA represents each weight update with rank-deficient matrices whose input and output subspaces emerge implicitly during training.The update is factorized through matrices B_l and A_l, with rank r < min(m,n).
  • Motivation: Recent analyses link LoRA’s asymmetric matrices to instability or failure to converge optimally, motivating explicit control over multiple interacting subspaces.The proposed questions concern selecting input and output subspaces and controlling their interactions during training.
  • FrameFT parameterization: FrameFT decomposes input and output spaces into k Fusion Frame subspaces and uses trainable coefficient matrices to encode their relationships.The coefficient matrix is organized block-diagonally as C_l = diag(C_l,1, ..., C_l,k), producing the update W_l + α/√(mn) P_m C_l P_n^T.
  • Frame construction: FrameFT freezes the Fusion Frame projection matrices and trains only the coefficient matrices, which are generated from uniform tight Fusion Frames constructed algorithmically with Spectral Tetris.The frames can use equal subspace dimensions in the uniform construction, while different dimensions may be used across layers.
  • Structured sparsity: Sparse coefficients identify useful subspace interactions, reducing update storage from O(kρ_mρ_n) to O(s), where s ≪ kρ_mρ_n, with sparsity patterns shared across layers.Figure 3 illustrates the sparse coefficient structure for a RoBERTa-Large query layer.
  • Efficiency and convergence: 1.28 MB versus 67.1 MB for LoRA rank-32 is reported for Llama-2-7B checkpoints, while block-sparse frames improve inference throughput and FrameFT preserves Lipschitz smoothness.The convergence analysis consequently supports gradient descent at learning rates η ≤ 1/˜L with an expected 1/T convergence rate to first-order stationary points.

4. Experiments

FrameFT is evaluated across language, vision, and operational-efficiency settings, including RoBERTa GLUE tasks, instruction-tuned LLMs, and Vision Transformers. Across these experiments, it matches or exceeds baselines while using fewer parameters and improving throughput.

  • Evaluation framework: FrameFT is evaluated on RoBERTa base and large models across GLUE tasks covering sentiment classification, paraphrase detection, and entailment recognition.The reported metrics are PCC for STS-B, MCC for CoLA, and accuracy for the remaining tasks.
  • Fine-tuning strategy: FrameFT uses 1000 randomly positioned non-zero coefficients shared across layers for RoBERTa experiments, with tight fusion frames using ρ = 2 and kρ = n.The same configuration passage describes adapting Query and Value matrices through the LoRA comparison recipe.
  • Language-model results: 10× fewer parameters than LoRA are used by FrameFT while it performs on par or better than LoRA and full fine-tuning on individual GLUE tasks and better than all baselines on average.The comparison includes LoRA, AdaLoRA, FourierFT, SVFit, SMT, SVFT, and RoseLoRA.
  • Instruction-tuning results: FrameFT performs on par or better than instruction-tuning baselines across Llama2, Gemma2, and Llama 3.1 models while using the fewest parameters across all models.These experiments fine-tune models on Alpaca and evaluate them with the LM-evaluation harness across challenge categories including reasoning, world knowledge, and generalization.
  • Vision-model results: FrameFT performs better than baseline methods on average across eight Vision Transformer image-classification tasks, with improvements on most tasks as its parameter count increases.The tasks include remote sensing, fine-grained classification, and texture recognition.
  • Operational efficiency: 37.9 ± 4.5k tokens/sec for FrameFT versus 24.4 ± 2.7k tokens/sec for LoRA is reported on Llama-2-7B inference.Across 100 iterations, the coefficient of variation is below 12%; the reported throughput advantage is attributed to sparse coefficients and sparse projection matrices.
  • Storage and memory: FrameFT stores only sparse coefficient matrices because algorithmically generated Fusion Frames are shared across layers of the same dimension.This avoids storing or transmitting basis vectors required by the described SVD-based alternatives.
  • Operational efficiency: FrameFT’s Fusion Frame construction has O(kd) time complexity, but generation occurs once during initialization and projection matrices are shared across layers.The analysis therefore focuses on per-layer throughput measurements.

5. Related work

The related work reviews parameter-efficient adaptation methods that freeze pretrained weights while modifying low-dimensional, prompted, modular, or sparse components. It positions FrameFT alongside LoRA and sparse fine-tuning approaches.

  • Adapters: Adapters train specialized modules inserted between pretrained layers while keeping the original model parameters frozen.This preserves pretrained knowledge while reducing overfitting risk, according to the cited related work.
  • Low-rank matrix factorization: LoRA reparameterizes selected-layer weight updates through low-rank factorizations while training only the decomposition matrices.Variants explore asymmetric chaining, quantization-aware formulations, learning-rate choices, and adapter compression.
  • Prefix and prompt tuning: Prefix tuning prepends learnable vector sequences to Transformer inputs, whereas prompt tuning learns representations that guide model behavior without architectural modification.
  • Sparse fine-tuning: Sparse fine-tuning targets only selected critical parameters while freezing the rest, using approaches such as low-rank operations or Fourier-based spectral compression.The reviewed methods aim for competitive or superior performance while reducing computation and storage needs.

6. Conclusions

The paper concludes that FrameFT uses Fusion Frame subspace decompositions to fine-tune vision and language Transformers with substantial compute and parameter efficiency. It reports preserved performance, convergence analysis, and a scope boundary around structured-sparsity support.

  • FrameFT fine-tunes vision and language Transformer models through structured subspace decompositions based on Fusion Frames.
  • Experiments across Vision Transformers and Llama and Gemma language models show compute and parameter efficiency gains without sacrificing performance across many benchmarks.
  • FrameFT preserves Lipschitz smoothness of the loss landscape and therefore has desirable convergence properties.
  • Support for structured sparsity beyond 2:4 sparsity remains limited, creating an opportunity for higher efficiency if specialized kernels are implemented.

Impact Statement

FrameFT aims to reduce the computational cost and storage requirements of adapting foundation models. Its analysis establishes smoothness preservation and convergence to a first-order stationary point under the stated assumptions.

  • FrameFT aims to lower computational cost and storage requirements for fine-tuning, potentially supporting broader accessibility and reduced energy usage.
  • FrameFT preserves Lipschitz smoothness when a loss is Lipschitz-smooth with respect to the weight matrix W.
  • Gradient descent on the coefficient space reaches an ϵ-first-order stationary point under the stated smoothness analysis.

C. Image Classification with Vision Transformers

The vision experiments evaluate FrameFT on base and large Vision Transformers across diverse image-classification tasks. With 5000 non-zero coefficients per layer, FrameFT outperforms baseline methods on average except full fine-tuning while using substantially fewer parameters than LoRA.

  • The evaluation fine-tunes base and large pretrained Vision Transformers across diverse image-classification challenges.
  • FrameFT uses sparse block-diagonal coefficient matrices with 1000 or 5000 non-zero elements per layer and shared non-zero positions across layers.
  • With 5000 non-zero coefficients per layer, FrameFT performs better than all baseline methods except full fine-tuning on average.Performance also improves when increasing non-zero coefficients from 1000 to 5000.
  • FrameFT achieves the vision results with the lowest parameter count among baselines, 5−20× lower than LoRA.

D. Performance versus subspace dimension

The subspace-dimension study finds that FrameFT performs well even at ρ = 2, with no strong performance relationship across tested dimensions. Increasing the number of non-zero coefficients is a more consistent performance adjustment.

  • The study varies FrameFT’s subspace dimension from 2 to 256 on RoBERTa-base fine-tuned for CoLA and MRPC.
  • FrameFT performs well even with a subspace dimension of 2.
  • Performance shows no strong relationship with subspace dimension across the tested values.
  • Increasing the number of non-zero coefficients improves FrameFT performance, reaching parity with or exceeding full fine-tuning in the reported GLUE experiments.

F. Additional Discussions

Additional discussions compare FrameFT with prompt tuning, clarify when Fusion Frames are unnecessary, and examine how subspace dimension affects sparsity and performance.

  • Comparison with Prompt Tuning: Because FrameFT performs comparably to or better than LoRA, the paper infers an advantage over prompt tuning based on prior comparisons.
  • Standard Frames versus Fusion Frames: The Fusion Frame construction enforces a minimum subspace dimension of 2 because complex-domain components double in dimension when transformed to the real domain.
  • Standard Frames versus Fusion Frames: For applications requiring subspace dimension 1 while maintaining kρ = n, classical orthonormal bases such as Fourier bases or Wavelets can be used instead.
  • Performance versus subspace dimension: As subspace dimension increases with kρ = n, Fusion Frame sparsity increases, reducing update degrees of freedom and causing a small performance drop.

G. Hyperparameters used for the experiments

The section describes algorithmic construction of Tight Fusion Frames, from a smaller unit-norm frame through modulation and extension to real-valued spaces. It also reports that Spectral Tetris constructs bases faster than QR decomposition on an NVIDIA A100, with construction latency amortized across model calls.

  • Fusion Frame construction: Tight Fusion Frames are constructed through three stages: generating a UNTF, modulating its vectors into subspaces, and extending the result to real-valued spaces.The construction begins with a UNTF in C^ρ, uses kth roots of unity to form k subspaces in C^d, and then applies a real-domain extension.
  • Fusion Frame construction: The initial UNTF uses unit-norm columns and orthogonal rows with a constant norm, so FF* is a constant multiple of the identity.The example constructs a matrix whose columns are frame vectors and whose rows satisfy the tightness condition.
  • Fusion Frame construction: The construction fills the smaller frame iteratively with ones and 2 × 2 matrices T(x) until all rows meet their required norms.The described procedure repeats insertion of T(x) using the remaining norm until the matrix is complete.
  • Fusion Frame construction: Modulating each row by powers of a root of unity produces orthogonal bases for indexed subspaces, and the resulting Fusion Frames are Tight.The final construction is illustrated as a (6,3,11)-TFF spanning C^11, with six subspaces of dimension three.
  • Construction time: Spectral Tetris takes a fraction of the time required by QR decomposition to construct a basis on an NVIDIA A100 GPU.Gram-Schmidt is reported as comparatively slow because its sequential process cannot be parallelized on GPUs or TPUs; construction latency is amortized across model calls.

I. Scaling Factor Sensitivity

FrameFT remains strong across scaling factors from 10 to 600 on Llama-2-7B instruction tuning with Alpaca. It outperforms LoRA throughout the sweep and, for α ≥100, consistently matches or exceeds the full fine-tuning upper bound.

  • Scaling-factor sensitivity: FrameFT outperforms LoRA’s average score of 63.18 across the entire scaling-factor sweep from α = 10 to 600.The sweep evaluates Llama-2-7B instruction tuning on Alpaca while varying FrameFT’s scaling factor.
  • Scaling-factor sensitivity: For α ≥100, FrameFT consistently matches or exceeds the full fine-tuning upper bound of 63.39.This result holds across the evaluated scaling-factor range beginning at 100.
  • Scaling-factor sensitivity: FrameFT is highly insensitive to the scaling factor and operates reliably out-of-the-box rather than requiring exhaustive searches to avoid instability.The reported comparison concerns practical tuning requirements relative to PEFT methods.
Loading 2608.26430v1…