PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning

Rong Huang, Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming Fu

arXiv:2610.37068v1cs.CL

TL;DR

PEFT reduces adaptation costs by adding few trainable parameters, but those parameters may retain internal redundancy. HiLoRR recursively reorganizes and low-rank-compresses PEFT parameters. Across eight commonsense reasoning benchmarks, compressed variants use about an order of magnitude fewer parameters while achieving higher average accuracy than corresponding high-parameter baselines.

  • Problem

    PEFT introduces few trainable parameters while freezing the pretrained backbone, but those parameters may still contain internal redundancy that has not been fully explored.

    Source: §1 · p. 1

  • Method

    HiLoRR reorganizes trainable parameters into approximately square matrices and recursively applies low-rank decomposition to their resulting factors.

    Source: §1 · pp. 1–2

  • Results

    Across eight commonsense reasoning benchmarks, HiLoRR variants use approximately an order of magnitude fewer trainable parameters while achieving higher average accuracy than corresponding high-parameter baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HiLoRR’s compact parameterization is particularly suitable when adapter storage, optimizer states, or task-specific parameter budgets are constrained.

    Source: §5 · p. 9

    With small or moderate internal-factor initialization, the deeper multiplicative path can attenuate early optimization information, and progressive factor activation can make early optimization slower than the original PEFT formulation.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) methods adapt large models by introducing a small number of trainable parameters. However, we find that these parameters still contain redundancy and can be further compressed while maintaining strong task performance. To this end, we propose a novel hierarchical low-rank reparameterization method. Its core idea is to reorganize the trainable parameters of PEFT methods into an approximately square matrix and then express it via a low-rank decomposition, which can be applied recursively. This forms a multi-level structure with factors compressed level by level, drastically reducing the number of trainable parameters. In our experiments, we combine this reparameterization with LoRA, DoRA, MoSA, and RaSA. We evaluate commonsense reasoning on LLaMA-3.1-8B and Qwen2.5-7B, with additional GSM8K math reasoning experiments on LLaMA-3.1-8B. Under comparable trainable-parameter budgets, our method combined with LoRA or DoRA achieves better performance. Furthermore,when the parameter count of the original methods is compressed by approximately an order of magnitude, our method can still maintain competitive task performance and even surpass the corresponding baselines in average accuracy across the eight commonsense reasoning benchmarks.

1 Introduction

HiLoRR addresses possible redundancy in PEFT trainable parameters by recursively reorganizing and low-rank-compressing them. Across eight commonsense reasoning benchmarks, compressed variants use approximately an order of magnitude fewer parameters while achieving higher average accuracy than corresponding high-parameter baselines.

1 Introduction

  • PEFT reduces adaptation costs by freezing the pretrained backbone and adding few trainable parameters, but those parameters may retain unexplored internal redundancy.Prior PEFT work has substantially reduced model-adaptation cost.Source: §1 · p. 1
  • HiLoRR reorganizes existing PEFT trainable parameters into approximately square matrices and recursively applies low-rank decomposition to the resulting factors.This forms a hierarchy that compresses parameters level by level.Source: §1 · pp. 1–2
  • HiLoRR is presented as a general framework compatible with LoRA, DoRA, MoSA, and RaSA rather than a redesign for one PEFT formulation.The reported experiments find improved overall performance under comparable parameter budgets and substantial reductions with competitive performance across evaluated methods.Source: §1 · p. 2
  • Across eight commonsense reasoning benchmarks, HiLoRR variants reduce trainable parameters by approximately an order of magnitude while achieving higher average accuracy than corresponding high-parameter baselines.The framework is also evaluated on LLaMA-3.1-8B and Qwen2.5-7B commonsense reasoning, and on LLaMA-3.1-8B GSM8K math reasoning.Source: §1 · p. 2

2 Related Work

Related work improves LoRA’s parameter efficiency, optimization, initialization, or update capacity, while another line reduces task-specific parameters through compact or structured representations. HiLoRR is distinguished by recursively reorganizing trainable parameters across PEFT formulations.

Parameter-Efficient Fine-Tuning and LoRA

  • LoRA’s rank sets both its trainable-parameter budget and the upper bound on update rank, so lowering the budget by reducing rank also restricts the adaptation space.LoRA optimizes low-rank weight updates while keeping pretrained weights frozen.Source: §2 · p. 2

LoRA Variants

  • LoRA variants target rank allocation, training dynamics, initialization, or update construction, whereas this work recursively reorganizes the resulting trainable parameters.Examples include AdaLoRA and DyLoRA for ranks, LoRA+ and rsLoRA for optimization, and DoRA and MoRA for update formulations.Source: §2 · p. 2

Compact and Structured Parameterizations

  • Compact PEFT approaches use parameter sharing, fixed or pretrained bases, compact cores, nested representations, tensor decompositions, or structured updates.The passage contrasts these method-specific structures with HiLoRR’s general recursive reparameterization, which is compatible with LoRA, DoRA, MoSA, and RaSA.Source: §2 · pp. 2–3

3 Method

HiLoRR compresses trainable PEFT parameters by reshaping them into approximately square matrices and recursively factorizing them at low rank. It can be applied across PEFT formulations, though its deeper factorization changes optimization behavior.

3.1 Overview of HiLoRR

  • HiLoRR recursively reshapes trainable parameters into approximately square matrices and low-rank factors, training only the deepest-level matrices before reconstructing the original parameters.The reshaping preserves element count, and each decomposition replaces a matrix with two low-rank matrices.Source: §3.1 · p. 3

3.2 Applying HiLoRR to LoRA

  • HiLoRA factorizes reshaped LoRA factors internally, while the outer rank r still bounds the rank of the final update.Near-square reshaping is the chosen design; compression occurs when the hierarchical rank k is below the stated parameter-count threshold.Source: §3.2 · pp. 3–4

3.3 Applying HiLoRR to Different PEFT Methods

  • HiLoRR preserves each method’s design while reparameterizing its trainable components, including DoRA’s directional parameters and RaSA’s rank-sharing parameters.For DoRA, the magnitude component remains in its original form; HiRaSA retains RaSA’s cross-layer sharing, while HiMoSA retains MoSA’s parameter sharing.Source: §3.3 · p. 4

3.4 Optimization Characteristics of Hierarchical Reparameterization

  • HiLoRA’s internal gradients depend on multiple trainable factors, making their scale sensitive to initialization and norms and potentially attenuating early optimization.With LoRA-compatible zero initialization, only one internal factor receives a nonzero gradient at the first step, which can make early optimization slower; experiments generally find best performance at a higher learning rate.Source: §3.4 · p. 5

4 Experiments

The experiments test HiLoRR across models, reasoning tasks, PEFT methods, and hierarchical configurations. HiLoRA and HiDoRA improve aggregate commonsense accuracy at comparable budgets, while compressed variants remain competitive across tasks and methods.

4.1 Experimental Setup

  • Experiments span LLaMA-3.1-8B and Qwen2.5-7B, eight commonsense benchmarks, and GSM8K, with methods grouped into low- and high-parameter settings.The tables also report training-phase peak GPU memory alongside accuracy and parameter count.Source: §4.1 · p. 5

4.2 Results: Commonsense Reasoning

  • In each setting, HiLoRA or HiDoRA achieves the highest Avg. and W-Avg.; low-parameter versions use approximately 10–13% of LoRA-r16 parameters and outperform all reported baselines on both backbones.At comparable high-parameter budgets, both methods outperform LoRA and DoRA on both aggregate metrics.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Results: Mathematical Reasoning

  • On GSM8K, high-parameter HiLoRA and HiDoRA score 73.16% and 73.24% EM, respectively, outperforming all compared baselines at comparable parameter counts.In the low-parameter group, they score 70.20% and 70.05% EM, the two highest scores in that group, but remain below high-parameter LoRA-r16.Source: Table 2 · p. 7, §4.3 · p. 7

4.5 Ablation Studies

  • HiLoRA generally performs best at a higher learning rate; the study selects a peak learning rate of 2 × 10−3 because it gives the highest Avg. among evaluated rates.The authors relate this sensitivity to the effective gradient scale changed by the additional multiplicative factorization.Source: §4.5 · p. 7
  • The A+B scope variant outperforms A-only and B-only at similar parameter budgets, while double-level reparameterization improves accuracy at low budgets but lowers it at medium and high budgets.The authors suggest deeper factorization may help under tight budgets but impose unnecessary constraints and harder optimization at larger budgets.Source: §4.5 · p. 7

4.4 Results: Generalization Across PEFT Methods

  • HiMoSA and HiRaSA reduce trainable parameters by 92.6% and 86.9%, respectively, while maintaining average accuracy close to their original counterparts.The reparameterization preserves the original adaptation mechanisms and extends beyond LoRA and DoRA.Source: §4.4 · p. 7, Table 3 · p. 8

4.5 Ablation Studies

  • Near-Square reshaping outperforms Random mean and the more elongated 1:4 and 1:16 strategies in the reported comparison.Random mean reshapes a random permutation into an approximately square matrix and averages three runs; the elongated strategies use less square shapes.Source: §4.5 · pp. 7–8
  • Different combinations of outer rank r and hierarchical rank k provide different trade-offs between parameter efficiency and adaptation performance.The outer rank controls adaptation capacity, while the hierarchical rank determines compression level.Source: §4.5 · p. 8

5 Conclusion and Discussion

HiLoRR recursively reparameterizes existing PEFT parameters to reduce their internal degrees of freedom. The results indicate substantial parameter reduction with performance close to original methods, and the framework applies across multiple PEFT formulations.

5 Conclusion and Discussion

  • HiLoRR compresses PEFT parameters by reorganizing them into approximately square matrices and recursively applying low-rank factorizations.This forms a hierarchical parameterization.Source: §5 · p. 8
  • LoRA with HiLoRR uses about one tenth of the full-size adapter’s trainable parameters while maintaining performance close to, and in some settings comparable with, the original methods.The reported pattern spans different model architectures and reasoning tasks.Source: §5 · pp. 8–9
  • Applying the same reparameterization to LoRA, DoRA, MoSA, and RaSA supports HiLoRR as a general compression framework rather than a method-specific modification.Source: §5 · p. 9
  • HiLoRR’s performance is more sensitive to learning rate, factor initialization, hierarchical rank, and reshaping strategy because its multiplicative structure changes optimization behavior.The paper identifies broader model families, modalities, and downstream tasks as future directions.Source: §5 · p. 9

A Additional Details of HiLoRR

HiLoRR reshapes PEFT parameters before replacing their direct optimization with low-rank factors. Its configurations vary which components are compressed and whether compression is applied recursively.

A.1 Reshaping Strategy

  • Near-square reshaping preserves element order while changing row boundaries, then U and V reconstruct the original factor dimensions through Back(·).For example, the illustrated 2 × 8 factor becomes a 4 × 4 matrix; parameter savings come from learning k(p + q) entries instead of mn when k(p + q) < mn.Source: §A.1 · p. 12

A.2 HiLoRR Configurations

  • Single-side HiLoRR compresses one component, Dual-side compresses both factors, and Multi-level recursively reparameterizes the resulting factors for additional compression.Increasing hierarchy depth offers a trade-off between compression ratio and adaptation capacity.Source: §A.2 · p. 12

B Parameter Compression Analysis

HiLoRR reduces the internal degrees of freedom of PEFT parameters while retaining the original method’s effective update structure. Compression depends on which factors are reparameterized and on the layer dimensions and hierarchical rank.

B Parameter Compression Analysis

  • HiLoRR reduces internal degrees of freedom while preserving the underlying PEFT interface; unlike directly reducing r, this does not lower the maximum rank of the resulting update.For LoRA, reconstructed A and B retain their original shapes.Source: B Parameter Compression Analysis · p. 12
  • When k < min(p, q), the factor matrices representable by the decomposition form a structured subset of those allowed at the same outer rank; when k ≥ min(p, q), full factor capacity is recovered.Source: B Parameter Compression Analysis · p. 12
  • Double-sided HiLoRR compresses both factors, whereas each single-sided variant leaves one factor’s parameters intact, bounding the layer’s overall compression.Source: B Parameter Compression Analysis · p. 12
  • Per-factor compression increases with layer size and decreases with hierarchical rank, while heterogeneous Transformer layers mean whole-model savings are not simply the single-layer ratio.Source: B Parameter Compression Analysis · p. 12
  • 884,736 to 88,064 trainable parameters: A+B HiLoRA compresses one LLaMA-3.1-8B block by about one tenth in the same-rank r16/k16 regime.The 32 Transformer blocks share the same reshaping, so the per-block figures scale directly to the whole model.Source: B Parameter Compression Analysis · p. 13

C.1 Training Setup

The experiments use a single NVIDIA H800 GPU and a shared set of target modules across methods.

C.1 Training Setup

  • All experiments are conducted on a single NVIDIA H800 GPU.Source: §C.1 · p. 13
  • Commonsense reasoning runs for 3 epochs and mathematical reasoning runs for 2 epochs, with an effective batch size of 64 for both.Source: §C.1 · p. 13
  • The shared target modules across all methods are q, k, v, up, and down_proj.Source: Table S1 · p. 13

C.2 Training Hyperparameters

The appendix provides the training configurations used in the experiments. Unless stated otherwise, methods share a configuration for fair comparison, with task-dependent settings summarized separately.

  • Detailed training configurations for the experiments are provided in Table S1.Source: §C.2 · p. 13
  • All methods use the same training configuration unless otherwise specified, supporting a fair comparison.Source: §C.2 · p. 13
  • Task-dependent settings are summarized in Table S1.Source: §C.2 · p. 13

C.3 Parameter Compression Analysis

The appendix analyzes HiLoRA’s parameter compression, reporting reductions in trainable parameters while preserving the original adaptation structure. Further savings are possible when internal ranks satisfy the compression condition.

  • Table S2 reports trainable-parameter counts before and after reparameterization for LLaMA-3.1-8B with r=16 and k=16, using a recursive double-sided variant.Counts are per Transformer block, summed over q, k, v, up, and down projections; the table defines d as factor element count, std as original formulation parameters, and FF as reshaped parameters k(p+q).Source: §C.3 · p. 13, Table S2 · p. 14
  • HiLoRA substantially reduces trainable parameters while preserving the original adaptation structure.Source: §C.3 · pp. 13–14
  • Additional hierarchical reparameterization can yield further parameter savings when internal ranks satisfy the compression condition.Source: §C.3 · p. 14

Related papers

Loading 2610.37068v1…