CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

1 Introduction

• Strict orthogonality can reduce inter-task interference but remove shared components of updates, weakening representation reuse and transfer when tasks share semantic structure.The paper calls this trade-off the “Orthogonality Dilemma.”Source: §1 · p. 1
• The introduction frames two questions: why strict orthogonality limits generalization across related tasks, and how to preserve transferable representations while retaining task-specific expertise without replay.It also asks when the limitation is most severe.Source: §1 · p. 1
• CoDe-LoRA separates transferable representation consolidation from task-specific specialization through adaptive null-space projection and prototype-based expert decoupling.It operates without historical-data replay.Source: §1 · pp. 1–2
• CoDe-LoRA achieves the best average accuracy across four backbones and three continual-learning benchmarks while balancing knowledge transfer and interference mitigation.The introduction reports this as the outcome of extensive experiments.Source: §1 · p. 2

2.1 Parameter-Efficient Fine-Tuning

• Adapters, prompt or prefix tuning, and LoRA reduce adaptation costs by adding lightweight modules or low-rank matrices while freezing the backbone.Sequential PEFT adaptation can still lead to catastrophic forgetting of earlier tasks.Source: §2.1 · p. 2

2.2 Continual Fine-Tuning for LLMs

• Continual fine-tuning uses regularization, replay, or parameter expansion, with trade-offs in memory, computation, privacy, or parameter growth.MoE routing approaches may still rely on historical exemplar replay to limit routing drift and maintain performance over long sequences and domains.Source: §2.2 · p. 2
• Orthogonality-based methods reduce interference but may limit representation reuse across related tasks; CoDe-LoRA instead combines shared-representation consolidation with task-specific specialization without historical-data replay.Figure 2’s caption identifies preliminary evidence on similarity-related degradation and transfer across continual adaptation methods and task sequences.Source: §2.2 · p. 2, Figure 2 · p. 3

3.1 Problem Definition

• The problem setting keeps the backbone frozen, learns small task-specific updates sequentially, stores no earlier-task data, and provides no task identity at inference.Sequential fine-tuning can cause forgetting of previous tasks.Source: §3.1 · p. 3

3.2 Orthogonal Continual Adaptation

• O-LoRA constrains each task’s low-rank update to be orthogonal to previous task subspaces, with the constraint enforced in practice through a regularization penalty.The update is parameterized as ΔW_t = B_tA_t.Source: §3.2 · p. 3

3.3 Orthogonality Dilemma

• Strict orthogonality removes update components aligned with earlier task subspaces, discarding a ρ_t fraction of update energy and limiting transfer more as tasks become related.The paper frames this as a dilemma: the constraint protects prior knowledge while limiting transfer between related tasks.Source: §3.3 · p. 3

3.4 Preliminary Evidence and Mechanism Validation

• Removing O-LoRA’s orthogonality penalty raises mean next-task forward transfer from 43.70 to 49.45 (+5.75), while the penalty reduces prior-subspace energy from 5.05–5.63% to 0.47–0.64% (89.0% on average).The comparison holds Order-1 data, schedule, and seed fixed; Figure 2 also reports worsening degradation as task similarity grows and positive transfer gains across task sequences.Source: §3.4 · p. 3

4.1 Overall Framework

• CoDe-LoRA uses parallel consolidation and decoupling branches over a frozen backbone, integrating their outputs through confidence-based fallback selection.The consolidation branch accumulates transferable representations; the decoupling branch maintains independent task-specific experts.Source: §4.1 · p. 4

4.2 Consolidating Shared Representations

• The consolidation branch projects each new update into the null space of accumulated representations and scales it to constrain interference and parameter-magnitude growth.The projection retains new components, while dynamic scaling stabilizes cumulative parameter magnitude across sequential tasks.Source: §4.2 · pp. 4–5

4.3 Decoupling Task-Specific Expertise

• The decoupling branch routes each input to the expert whose task prototype has the highest cosine similarity to its frozen-backbone embedding.Prototypes are normalized means of embeddings from 10 randomly selected training samples per task; routing stores compact statistics rather than raw samples.Source: §4.3 · p. 5

4.4 Knowledge Integration and Fallback Selection

• When maximum prototype similarity exceeds the default threshold of 0.75, inference uses the selected expert; otherwise it falls back to the consolidated shared branch.The fallback uses the shared branch as a robust prior for ambiguous or out-of-distribution inputs.Source: §4.4 · p. 5

5.1 Experimental Setup

• Evaluation covers three continual-learning benchmarks, three task orders per benchmark, and AP, BWT, and forward-transfer behavior.The benchmarks are Standard CL, Large Number of Tasks, and TRACE; results are averaged over three runs.Source: §5.1 · pp. 6–7

5.2 Main Results

• Across Table 2’s backbones and benchmarks, CoDe-LoRA outperforms every baseline; on Qwen3-0.6B, Standard CL accuracy is 79.7 vs. 70.7.Table 2 reports results across backbones and distinguishes Standard and Long benchmark accuracy from TRACE OP/BWT metrics.Source: Table 2 · p. 7, §5.2 · p. 8
• CoDe-LoRA achieves 80.4% average accuracy on Standard CL, exceeding the MTL reference at 80.0%, and leads N-LoRA by 6.7 points on the Long benchmark (79.1% vs. 72.4%).O-LoRA’s performance degrades on longer sequences, where transferable representations become increasingly important.Source: §5.2 · p. 7
• On TRACE, CoDe-LoRA improves generation quality by 3.2 to 8.8 points and has BWT = 0.0 on all three backbones, without replay buffers.The strongest baseline per backbone loses between 1.3 and 4.3 BWT points.Source: §5.2 · pp. 7–8

5.3 Ablation Studies and Analysis

• With adapter capacity matched, CoDe-LoRA leads the strongest baseline by 4.0 points on Standard CL (80.4 vs. 76.4) and 8.7 points on Long Sequence (79.1 vs. 70.4).The passage concludes that the gains stem from decoupled knowledge consolidation rather than parameter capacity.Source: §5.3 · p. 8
• Direct LoRAHub merging yields BWT of -20.2%, compared with -0.2% for CoDe-LoRA.The authors interpret this as stronger cross-task interference from direct parameter merging.Source: §5.3 · p. 8, Figure 6 · p. 9
• CoDe-LoRA exceeds O-LoRA in zero-shot forward transfer, including a +19.1-point Yahoo gain after training on dbpedia and amazon; routing accuracy saturates around N=10.Figure 7 presents the Qwen3-0.6B Order 1 transfer matrix; Figure 6 reports routing accuracy as prototype sample size varies.Source: §5.3 · p. 8, Figure 7 · p. 9, Figure 6 · p. 9
• On the Standard CL Benchmark, combining consolidation and task-specific adaptation gives the best overall performance across task sequences; per-task training takes 98.4 s, versus 108.8 s for O-LoRA and 98.0 s for N-LoRA.CoDe-LoRA inference is about 1.4× slower than O-LoRA (43.1 s vs. 30.1 s).Source: §5.3 · pp. 8–9

• Strict orthogonal isolation can mitigate catastrophic forgetting while suppressing transferable representation accumulation across related tasks.Source: §6 · p. 9
• CoDe-LoRA balances shared representation consolidation with task-specific adaptation through a dual-branch framework.Source: §6 · p. 9
• Experiments demonstrate strong continual-learning performance and robust forward and backward transfer across diverse task sequences and model families.Source: §6 · p. 9

• The evaluation covers pretrained language models, not multimodal continual-learning settings.Source: Limitations · p. 9
• Routing effectiveness depends on the quality of task prototypes in the embedding space.Source: Limitations · p. 9
• Under severe task distribution shifts or highly overlapping task boundaries, routing reliability may degrade.Source: Limitations · p. 9

• Table 4 summarizes the key notations used throughout the paper.Source: §A.1 · p. 11, Table 4 · p. 11

A.2 Theoretical Justification for Dynamic Scaling

• The induction establishes that accumulated parameter deviation is bounded by the maximum individual task-update norm, preventing the consolidated branch from exploding or drifting from the pretrained manifold.The stated consequence is stability while mitigating catastrophic forgetting.Source: §A.2 · p. 12
• The 1/√t factor is described as variance normalization: it keeps shared memory a stable anchor while task-specific experts preserve distinct decision boundaries.Without scaling, the shared-branch norm would grow linearly with √t under orthogonal updates and could overwhelm the task-specific branch.Source: §A.2 · p. 12
• 1/√t scaling maintains a stable norm and achieves peak accuracy in the Standard CL Benchmark comparison.The comparison is against linear averaging (1/t) and no scaling (1).Source: §A.2 · p. 12

A.3 Implementation Details

• Experiments use three random seeds, rank-8 LoRA, and an LDA router estimated from 1,000 support samples per task.The implementation also uses one training epoch per task and randomized SVD with retained rank 8.Source: §A.3 · p. 12
• Projection on the LoRA factor B with randomized low-rank SVD reduces complexity from O(d_out d_in min(d_out,d_in)) to O(d_out r^2), with r ≪ d_in.The T5-large consolidation took 0.7 seconds for four Standard tasks and 4.1 seconds for fifteen Long tasks.Source: §A.3 · p. 13
• The consolidation branch’s accuracy improves monotonically with retained SVD rank, whereas full-model accuracy is unchanged when the routed expert answers every input.The retained rank therefore affects consolidated-weight quality, not routed accuracy.Source: §A.3 · p. 13, Table 6 · p. 13
• Each task starts from the consolidated state with a new zero-initialized expert; both branches train jointly, then the projected update is scaled and merged while the expert and its prototype are archived.The projection uses randomized low-rank SVD over the accumulated representation history.Source: §A.3 · p. 13

A.4 Evaluation Metrics

• Average Performance (AP) measures overall capacity across learned tasks after the sequence; Backward Transfer (BWT) tracks effects on earlier tasks, and Forward Transfer (FWT) measures zero-shot accuracy on untrained tasks.Negative BWT indicates degradation; values closer to zero or positive indicate stable retention or positive backward transfer.Source: §A.4 · p. 14

A.5 Router Choice

• 94.2% LDA routing accuracy with N=10 exceeds cosine routing’s 93.3% with N=100 on Long; LDA saturates beyond about 500 samples per task.Figure 8 compares routing accuracy against support samples per task.Source: §A.5 · pp. 14–15, Figure 8 · p. 15

A.6 Co/De Synergy: Attempts and Limits

• The per-example oracle exceeds the routed expert by 4.7 points on Standard and 9.5 on Long, but no development-selected rule captures that potential; the null-space Co has oracle scores of 84.53 / 86.00.The oracle is diagnostic, not an attainable bound.Source: §A.6 · pp. 15–16, Figure 9 · p. 15, Table 9 · p. 16
• Candidate-label fusion with per-task weights is the only tested scheme positive across backbones; uncalibrated branch summation hurts Long accuracy by 1.0–1.6 points.Decoding-time mixing did not help; its development-selected weight was zero on both benchmarks.Source: §A.6 · p. 15, Table 8 · p. 15
• With LDA routing at 98.8% accuracy, fusion gains shrink to +0.13 on Standard and +0.77 on Long, and 94% of the remaining gain comes from CB and COPA.Those tasks have 26 and 40 development examples, respectively.Source: §A.6 · p. 16

A.7 Detailed Efficiency Statistics

• CoDe-LoRA stores 9.4M adapter parameters (1.28%) and peaks at 41.3 GiB; MoLE-CIE stores 32.4M (4.40%) and peaks at 31.7 GiB.These measurements are for the Standard benchmark, T5-large, and one H20 GPU.Source: §A.7 · p. 16, Table 10 · p. 16

A.8 Datasets

• The evaluation includes 15 datasets from classification and NLU benchmarks plus TRACE, which tests multi-turn generative tasks including code completion with Py150.The datasets span NLI, sentiment, and topic classification.Source: §A.8 · pp. 16–17

A.9 Task Sequence and Instructions

• The appendix provides task prompts and task sequences for T5, Llama 2-7B, and TRACE benchmark settings.Tables 13 and 14 summarize prompts and sequences.Source: §A.9 · p. 17, Table 13 · p. 17, Table 14 · p. 17