ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

• Although LoRA is widely used for parameter-efficient fine-tuning, whether its learned update effectively uses its allocated rank remains an open question.PEFT reduces adaptation cost by optimizing a small set of task-specific parameters instead of full model weights.Source: §1 · pp. 1–2
• In the rank-16 LoRA diagnostic, retaining k = 3 singular components yielded the best downstream performance, while adding components degraded it.The result suggests that a larger rank budget does not necessarily produce a more useful task-specific update.Source: §1 · p. 2, Figure 1 · p. 2
• ReGRA projects task gradients away from directions captured by the first-stage update and uses residual gradients to initialize a second adapter.An orthogonal regularizer maintains decorrelation between the two updates.Source: §1 · p. 2
• Across diverse benchmark sets, ReGRA outperforms representative baselines and in some settings matches a model with twice the rank.Ablation studies support the efficacy and rationale of its design.Source: §1 · p. 2

Low-rank parameter-efficient fine-tuning

• Existing LoRA variants address efficiency, rank allocation, optimization, and adapter placement, while not explicitly investigating redundancy among components within a learned update matrix.Source: §2 · pp. 2–3

• ReGRA's framework trains an initial adapter, extracts residual-gradient directions for a second adapter, and combines both with the frozen backbone.Source: Figure 2 · p. 3

Spectral and gradient-informed initialization

• Unlike methods that choose adaptation directions before fine-tuning, ReGRA estimates residual task gradients after learning a first-stage update.Source: §2 · p. 3

Sequential and subspace-aware adaptation

• Unlike sequential or continual-learning approaches, ReGRA applies two-stage adaptation to the same downstream objective and combines residual-gradient initialization with update decorrelation.Source: §2 · p. 3

Low-rank adaptation (LoRA)

• LoRA adapts a frozen pretrained weight by adding a trainable low-rank update.Its factors A and B are trainable, while the backbone weight remains frozen.Source: §3 · pp. 3–4

Singular value decomposition (SVD)

• SVD expresses a learned update using left and right singular-vector subspaces and descending positive singular values.The two singular subspaces identify components of a task gradient already represented by the learned update.Source: §3 · p. 4

• ReGRA applies SVD to the first-stage update to identify task-gradient components that it has already captured.Source: §3 · p. 4

4.1 RESIDUAL-GUIDED INITIALIZATION

• ReGRA estimates task gradients at the Stage-1 solution, removes components in the first update’s two-sided subspace, and uses aggregated residual-gradient directions to initialize Stage 2.The initialization preserves the function defined by W0 + ∆W1.Source: §4.1 · pp. 4–5
• After initialization, one simultaneous gradient-descent step preserves the Stage-1 model function and lies in the input-side subspace spanned by the selected residual-gradient directions.When the initial task gradient equals the truncated residual-gradient estimate, the step is proportional to its negative leading rank-r2 component.Source: §4.1 · p. 5
• The residual-gradient projection describes the first plain gradient-descent step only; both Stage-2 factors are subsequently optimized without reapplying that projection.The residual-gradient projection is not applied to later task gradients.Source: §4.1 · p. 5

4.2 STAGE-WISE ORTHOGONALITY REGULARIZATION

• Because Stage 2 can move away from its residual-guided initialization, ReGRA applies an update-level regularizer to promote orthogonality between the stage-wise updates.The penalty is defined using the unscaled updates D1 = B1A1 and D2 = B2A2; inner products and norms can be computed from LoRA factors without forming dense updates.Source: §4.2 · p. 5

4 RESIDUAL-GRADIENT ADAPTATION

• After training, the two updates are merged into the pretrained weight, with no additional inference-time branch or computation.Residual-gradient calibration adds a one-time cost between the two training stages.Source: §4.2 · p. 5

5.2 MAIN RESULTS (RQ1)

• ReGRA achieves the highest average score on both backbones in GLUE, with gains over rank-8 LoRA and further improvement at the same maximum update rank.Gains on CoLA, MRPC, and RTE support this pattern.Source: Table 1 · p. 6, §5.2 · pp. 6–7

5 EXPERIMENTS

• The evaluation covers GLUE, mathematical reasoning, and code generation, comparing rank-8-per-stage ReGRA with rank-8 and rank-16 LoRA and other rank-8 baselines.The experiments also ask how the two ReGRA components contribute and how trained LoRA and ReGRA update components affect performance.Source: §5 · p. 5, §5.1 · p. 6

5.2 MAIN RESULTS (RQ1)

• ReGRA achieves the highest six-task average on both Llama-3-8B and Qwen3-8B-Base, exceeding rank-16 LoRA on each backbone.Task-level gains differ by backbone, and ReGRA does not lead on every individual task.Source: Table 2 · p. 7, §5.3 · p. 7
• ReGRA obtains the highest pass@1 on both MBPP and enhanced MBPP+, retaining its lead under the stricter evaluation.Its advantage over rank-16 LoRA is modest on both test variants.Source: Table 3 · p. 7, §5.3 · p. 7

5.5 MORE ANALYSIS

• Removing the leading rank-16 LoRA singular component reduces performance by 3.97 percentage points on MBPP and MBPP+, and 8.03 points on Math.Removing most other components has a much smaller effect, and removing some slightly improves performance.Source: Figure 5 · p. 8, §5.5 · p. 8

5.4 ABLATION ANALYSIS (RQ2)

• Residual-gradient initialization provides the dominant ablation improvement, while update-level decorrelation adds a smaller benefit; the full model performs strongest overall.The 2 × 2 study holds the rank-8 first stage and rank-8 second stage settings fixed across variants.Source: Figure 4 · p. 7, §5.4 · p. 8

5.5 MORE ANALYSIS

• On math, isolated Stage-2 components C3, C4, and C6 improve mean accuracy by 2.91, 1.27, and 0.56 percentage points; the full update gains 1.27 points.On MBPP+, seven components improve pass@1 by 0.79–1.85 points, C6 is unchanged, and the full update gains 2.65 points; isolated variants are independent, not cumulative.Source: Figure 6 · p. 9, §5.5 · p. 9
• The analysis also examines progressive rank expansion by freezing previously learned adapters and training a new rank-2 adapter at each stage.The supplied passage describes this setup but does not report its performance outcome.Source: §5 · p. 5, Figure 7 · p. 9

6 CONCLUSION

• ReGRA improves over rank-matched LoRA on natural language understanding, mathematical reasoning, and code generation.Ablations support the residual-gradient initialization and orthogonality regularizer.Source: §6 · p. 9
• ReGRA uses residual task gradients to initialize a second adapter while preserving the first-stage solution.An orthogonality regularizer helps maintain separation between the two updates, which can be merged without additional inference overhead.Source: §6 · p. 9
• Effective low-rank adaptation depends on learning useful additional components relative to an existing update, not only increasing rank.The conclusion identifies more principled identification and allocation of useful residual directions under a fixed rank budget as future work.Source: §6 · p. 10

A.1 RESIDUAL-GRADIENT PROJECTION AND ORTHOGONALITY

• The residual-gradient projection removes components jointly aligned with the first-stage update's left and right subspaces, while retaining components aligned with only one.The resulting residual is orthogonal to every matrix in the two-sided subspace represented by the first-stage update.Source: §A.1 · pp. 13–14

A.2 LOW-RANK CALIBRATION AND GRADIENT AGGREGATION

• Each calibration-batch residual gradient is truncated after projection, and the truncated directions are aggregated to form a low-rank Stage-2 initialization.The procedure obtains the initialization without explicitly forming the dense aggregate; orthogonality applies to each residual gradient before truncation.Source: §A.2 · p. 14
• The mean of the batch residual-gradient directions can be aggregated without constructing a dense matrix, using concatenated factors and a smaller core matrix for SVD.The analysis uses thin QR factorizations before performing SVD on the smaller core matrix.Source: §A.2 · p. 14

A.3 PROOF OF THE RESIDUAL-GUIDED FIRST-STEP UPDATE

• The Stage-2 initial model represents exactly the pretrained weight plus the first-stage update.The proof analyzes the task-loss gradient at this initial model and describes one plain gradient-descent step; it does not assume later minibatch gradients match calibration estimates or stay in the initialized subspace.Source: §A.3 · pp. 14–15

A.4 UPDATE ORTHOGONALITY REGULARIZATION

• The orthogonality penalty can be evaluated using only the adapter factors, without explicitly forming either dense update matrix.It helps maintain orthogonality between the updates during Stage-2 training, but does not require their left or right singular subspaces to be separately orthogonal.Source: §A.4 · p. 15

B SUPPLEMENTARY IMPLEMENTATION DETAILS

• The section covers model setup, residual-gradient calibration, training hyperparameters, algorithm flow, and computational cost.The passage identifies these topics but gives no further implementation details.Source: B SUPPLEMENTARY IMPLEMENTATION DETAILS · p. 15

B.1 MODEL AND TRAINING SETUP

• The pretrained backbone stays fixed, Stage 1 trains a rank-8 LoRA adapter on query, key, and value projections, and Stage 1's learned update remains fixed thereafter.Each stage uses rank 8, α = 16, and dropout 0.05; GLUE classification heads are trained in Stage 1 and retained unchanged in Stage 2.Source: §B.1 · p. 15

B.2 RESIDUAL-GRADIENT CALIBRATION

• After Stage 1, calibration computes task gradients without optimizer updates, projects out the first-stage left- and right-subspace component, and compresses residuals to rank at most q.Batchwise factors are aggregated to initialize A2 from leading right singular vectors and B2 to zero, preserving the Stage-1 model's function.Source: §B.2 · p. 16

B.3 HYPERPARAMETER SETTINGS

• Both stages use fused AdamW with linear learning-rate warmup; task-specific epochs are listed for RoBERTa, and calibration settings specify M, b, q, and four SVD iterations.RoBERTa task epochs are 10, 3, 20, 3, 3, 20, 5, and 10 for CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, and STS-B, respectively.Source: §B.3 · p. 16

B.4 STAGE-2 OPTIMIZATION AND MERGING

• Stage 2 optimizes only its LoRA factors while the backbone and Stage 1 remain fixed, using task loss plus an update-level orthogonality regularizer.The regularizer coefficient warms up over the first 100 Stage-2 steps, separately from learning-rate warmup; the two trained updates are merged, requiring no additional adapter computation at inference.Source: §B.4 · pp. 16–17

B.5 ALGORITHM FLOW

• The algorithm trains and fixes Stage 1, constructs and aggregates compressed residual gradients for Stage-2 initialization, optimizes Stage 2 with orthogonality regularization, then merges both updates.The residual-gradient projection is used for initialization rather than at every training step.Source: §B.5 · p. 17, §B.4 · p. 17

B.6 COMPUTATIONAL COST

• Calibration stores rank-q factors for M batches in O(Mq(dout + din)) numbers per weight and aggregates them without forming the dense gradient; merged inference uses the pretrained model's dense multiplication.Across both stages, the factors contain (r1 + r2)(dout + din) parameters per adapted weight, excluding any task-specific output head.Source: §B.6 · pp. 17–18

C.1 COMMONSENSE REASONING

• 0.85 percentage points: ReGRA achieves the highest eight-task average on Llama-3-8B over rank-8 LoRA, while nearly matching rank-16 LoRA (85.98% versus 86.00%).The largest task-level gains are on BoolQ and OBQA, while Wino. and ARC-E decline.Source: Table 6 · p. 18, §C.1 · p. 18

C.2.1 UNIQUE ERROR CORRECTIONS

• On Math, all eight isolated components uniquely correct 19–38 task–example pairs each; on MBPP+, only components 1, 2, and 7 uniquely correct cases.Those MBPP+ counts are 1, 3, and 1 problems, respectively.Source: §C.2.1 · p. 18

C.2.2 OVERLAP WITH FULL-MODEL GAINS

• 359 of 391 Math cases and 9 of 15 MBPP+ cases newly solved by full Stage 2 are also solved by at least one isolated component; 32 and 6 are not.This overlap analysis does not measure each component's marginal contribution within the combined update.Source: §C.2.2 · p. 19, Figure 9 · p. 19