PaperlayerView exact revision on arXiv

Source-linked AI summary

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

Qian Zha, Jinda Liu, Yuan Wu, Yi Chang

arXiv:2608.21909v1cs.LG

TL;DR

Router-based multi-task LoRA can create a training–inference discrepancy that undermines predictive stability. CD-LoRA removes routers and aligns representations in a shared low-rank space, consistently outperforming multi-adapter baselines with lower computational overhead.

  • Problem

    Nonlinear routing in multi-task LoRA creates a training–inference discrepancy because stochastic training outputs need not match deterministic inference outputs, potentially degrading predictive stability.

  • Method

    CD-LoRA replaces task-routing architectures with explicit consistency alignment that enforces representation congruence in a shared low-rank space.

  • Results

    CD-LoRA consistently outperforms multi-adapter baselines while moderately reducing computational overhead.

  • Takeaways & Limitations

    Collaborative learning in a robust shared space offers a streamlined alternative to structural isolation for robust parameter-efficient multi-task fine-tuning.

  • Takeaways & Limitations

    Input-dependent routing prevents merged updates from being folded into the backbone, adding inference-time latency and memory overhead in routed architectures.

Abstract

from arXiv · show

While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.

1. Introduction

The introduction motivates parameter-efficient multi-task adaptation by identifying routing-induced training–inference discrepancy as a source of instability. It proposes CD-LoRA, a router-free approach that promotes shared low-rank representation consistency and outperforms multi-adapter baselines while reducing computational overhead.

  • Motivation: As LLMs scale, prohibitive full-parameter fine-tuning costs motivate PEFT, with LoRA emerging as a dominant adaptation approach.LLMs still require adaptation for specialized domains and evolving tasks despite strong zero-shot generalization.
  • Existing approaches: Multi-task LoRA methods increasingly share components or use routing to disentangle task-specific knowledge.Examples include independent per-task low-rank parameters, a common A with multiple B matrices, and multi-head routing structures.
  • Problem: Routing non-linearity creates a training–inference discrepancy that undermines stability under stochastic decisions and causes performance degradation.The paper supports this analytically and through latent-space visualizations.
  • CD-LoRA: CD-LoRA replaces explicit task partitioning with consistency-driven representation alignment in a shared low-rank space.The framework is described as plug-and-play, router-free, and designed to eliminate architectural overhead.
  • Results: CD-LoRA consistently outperforms multi-adapter baselines while moderately reducing computational overhead and improving robustness across diverse benchmarks.The introduction characterizes the framework as achieving state-of-the-art performance.

2. Related Work

LoRA enables parameter-efficient LLM adaptation through low-rank weight updates that can be merged without inference overhead. Multi-task extensions introduce independent adapters, shared projections with task-specific heads, and dynamic routing, but face parameter growth, limited knowledge transfer, and inference costs.

  • LoRA: LoRA approximates dense weight updates with two low-rank matrices whose rank is much smaller than the original dimensions, reducing trainable parameters.The update is represented as ΔW = BA, with LoRA-A ∈Rr×n and LoRA-B ∈Rm×r.
  • LoRA: Post-training, LoRA’s low-rank update can be merged into the frozen weight matrix, preserving zero additional inference overhead.This merge is a key practical advantage over architectures requiring runtime adapter computation.
  • LoRA Extensions: LoRA extensions include AdaLoRA’s dynamic rank allocation, DoRA’s magnitude-direction decomposition, and Align-LoRA’s cross-task representation alignment.Align-LoRA targets task-shared knowledge and multi-task generalization by minimizing inter-task distribution discrepancies.
  • Multi-Task LoRA Architectures: Multi-Adapter methods allocate separate LoRA factor pairs to each task, but parameter counts grow linearly with task number and isolation hinders cross-task knowledge transfer.These approaches accommodate task heterogeneity through independent task-specific components.
  • Multi-Task LoRA Architectures: Multi-Head methods share LoRA-A across tasks while retaining task-specific LoRA-B heads, reflecting evidence that A captures task-agnostic representations and B captures task-specific variations.Dynamic Routing mechanisms further select or combine heads according to the input to improve task specialization and flexibility.
  • Dynamic Routing: Input-dependent routing prevents pre-computation and merging of ΔW, requiring routing networks and multiple heads at inference and causing latency, memory overhead, and training–inference consistency concerns.This trade-off undermines LoRA’s zero-inference-overhead advantage.

3. Motivation

The motivation identifies training–inference discrepancy as a stability problem in routing-based multi-task LoRA, caused by nonlinear stochastic routing and shared low-rank projection instability. Theory and 12-task activation visualizations suggest that task representations already overlap in the shared low-rank space, weakening the case for explicit task partitioning.

  • Training–Inference Discrepancy: Nonlinear routing makes the expected training output differ from the deterministic inference output, potentially degrading predictive stability.The discrepancy is expressed as E_z[f(x, z)] ≠ f(x, E[z]).
  • Router-Free Discrepancy: After routing is removed, discrepancy is governed by up-projection scale and shared down-projection sensitivity to stochastic perturbations.The resulting instability is attributed to the low-rank projection stage rather than head selection.
  • Representation Analysis: Under 12-task joint training, LoRA-A activations probe whether the shared down-projection space supports separable or collapsed task representations.LoRA-A maps high-dimensional hidden states into a compact intrinsic subspace before task-specific transformations.
  • Representation Analysis: LoRA-A activations from different tasks are highly intermingled, forming an overlapping structure rather than well-separated clusters.This qualitative t-SNE observation aligns with the theoretical conclusion that instability in the shared projection, not insufficient task partitioning, is the primary limitation.

4. Method

CD-LoRA removes stochastic routers from multi-head LoRA and aligns dual stochastic latent representations in a shared low-rank space. Its consistency regularization smooths curvature-driven instability, closes the training–inference gap, and stabilizes inference.

  • De-routing: CD-LoRA removes routing mechanisms and treats all LoRA heads uniformly in a parallel, ungated architecture.This gives training and inference identical forward computation graphs.
  • De-routing: All heads jointly operate on a shared low-rank representation, replacing explicit task partitioning with a unified parameterization.The parameterization prioritizes common structure over fragmented task-specific subspaces.
  • Hessian-based Instability Analysis: The TID is driven primarily by latent-manifold curvature, with dropout sensitivity producing large Hessian norms and inference instability.This second-order analysis motivates smoothing the shared LoRA-A representation manifold.
  • Consistency-Driven Alignment: CD-A performs two independent dropout-masked forward passes through the shared LoRA-A matrix and minimizes symmetric KL divergence between their latent realizations.The divergence simplifies to a squared ℓ2 distance and acts as a first-order proxy for Hessian smoothing.
  • The CD-LoRA Objective: The final objective combines task-specific negative log-likelihood with consistency regularization weighted by λ.The task loss preserves predictive proficiency, while the consistency term reduces Hessian-driven training–inference discrepancy.
  • The CD-LoRA Objective: Because both passes share x, the consistency loss aligns LoRA-A representations across tasks, improving perturbation robustness and stabilizing inference.The dual-pass design uses A as the primary projection and A′ as an auxiliary stochastic view.

5. Experiments

Experiments compare CD-LoRA with representative multi-task LoRA methods on two Qwen backbones under a unified 12-task NLU benchmark and identical optimization settings. CD-LoRA achieves the strongest reported average performance, reduces peak memory versus HydraLoRA, and produces compact, task-separable representations.

  • Experimental Setup: CD-LoRA is compared with LoRA, HydraLoRA, R-LoRA, and M-LoRA under identical optimization settings, batch sizes, and training schedules.Experiments use Qwen2.5-7B and Qwen2.5-14B decoder-only backbones.
  • Evaluation Protocol: Models are jointly fine-tuned and evaluated on 12 diverse NLU tasks spanning inference, commonsense, sentiment, and mathematical reasoning, using accuracy.The benchmark is unified across tasks, with the full task list deferred to Appendix B.1.2.
  • Main Results: CD-LoRA shows consistent gains across diverse tasks, particularly Wino and SIQA, while maintaining competitive memory efficiency.The results are reported across both backbones, although the supplied passage does not provide the complete Qwen2.5-14B figures.
  • Main Results: 83.92% Avg. is achieved by CD-LoRA on Qwen2.5-7B, exceeding R-LoRA by 0.80% and M-LoRA by 0.49%.The cited comparisons are 83.12% for R-LoRA and 83.43% for M-LoRA.
  • Main Results: 56,882 MB peak memory is used by CD-LoRA on Qwen2.5-7B, 35.3% lower than routing-based HydraLoRA’s 87,860 MB.Memory was measured under identical hardware and training configurations.
  • Representation Analysis: Visualizations show CD-LoRA’s LoRA-A representations forming compact, well-separated, and task-separable embeddings.The observations support stable and discriminative task representations from consistency enforcement in the shared low-rank space.

6. Ablation Studies

Ablations show that removing routing improves performance and memory efficiency, while adding consistency alignment to the de-routed model yields the strongest results. Mechanistic and sensitivity analyses further link CD-LoRA’s gains to coherent task subspaces and moderate regularization.

  • Module-Level Ablation: Removing dynamic routing raises Avg. from 87.59% to 88.29% and reduces peak memory by 24.5%, from 66,071 MB to 49,869 MB.The comparison is between DR-LoRA and R-LoRA on five NLU tasks using Qwen2.5-7B.
  • Module-Level Ablation: CD-LoRA achieves the highest cross-task performance, reaching an Avg. of 88.56%, whereas alignment added to routing yields only a 0.32% gain with a 22.5% memory penalty.CD-LoRA combines de-routing with explicit consistency alignment; CR-LoRA retains the router.
  • Subspace Coherence Analysis: CD-LoRA produces the most coherent low-rank subspaces, achieving a global mean similarity of 84.93% and 84.49% in gate proj, versus R-LoRA’s 74.80%.De-routing alone raises global mean similarity to 82.07%, but lacks explicit cross-task coordination.
  • Subspace Coherence Analysis: CD-LoRA maintains high and stable layer-wise similarity, indicating that consistency alignment constrains task subspaces within a coherent manifold and prevents representational drift.The analysis identifies subspace coherence and reduced weight drifting as mechanistic sources of CD-LoRA’s performance gains.
  • Hyperparameter Sensitivity: 81.23% is the peak average accuracy at λ = 0.03, a 0.54% absolute gain over the 80.69% baseline; accuracy remains above 81.1% for λ ∈[0.01, 0.1].Increasing λ to 0.5 or 1.0 lowers accuracy to 80.44% and 80.25%, respectively, slightly below baseline.
  • Hyperparameter Sensitivity: Excessive regularization over-constrains the low-rank projection space, impairing retention of task-specific nuances and degrading multi-task performance.Moderate consistency alignment stabilizes the shared LoRA-A manifold, whereas large λ values produce a mild downward performance trend.

7. Conclusion

The paper identifies a training-inference discrepancy in router-based multi-head LoRA caused by nonlinear gating and introduces CD-LoRA, which replaces routing with consistency alignment in a shared low-rank space. CD-LoRA consistently outperforms superior baselines such as R-LoRA with lower computational overhead, supporting a streamlined router-free PEFT paradigm.

  • Theoretical analysis identifies nonlinear gating as the cause of a training-inference discrepancy that undermines stability and task decoupling.
  • CD-LoRA replaces complex routing with explicit consistency alignment in a shared low-rank space, eliminating the structural gap between training and inference.
  • CD-LoRA consistently outperforms superior baselines such as R-LoRA while requiring lower computational overhead.
  • The results suggest that collaborative learning in a robust shared space generalizes better than traditional structural isolation.
  • The work establishes a streamlined and effective paradigm for robust parameter-efficient fine-tuning without requiring architectural modularity.

A. Derivation of CD-LoRA … B.1.2. TWELVE-TASK SETTING •

The appendix derives CD-LoRA’s training–inference discrepancy, attributes it to curvature in shared projections amplified by routing, and motivates consistency regularization as a tractable mitigation. It also specifies five-task and twelve-task experimental settings, with the latter increasing task heterogeneity for stress-testing multi-adapter coordination.

  • A. Derivation of CD-LoRA: The appendix provides a complete, explicitly stated derivation of routing-based LoRA’s training–inference discrepancy and its mitigation through consistency-driven alignment.It is intended to support clarity and reproducibility.
  • A.1. Preliminaries and Notation: Routing-based LoRA uses a shared down-projection, head-specific up-projections, and nonlinear softmax routing weights.The shared projection is A, the head-specific projections are Bi, and the routing weights are ωi(·).
  • A.1. Preliminaries and Notation: Replacing the stochastic mask with its expectation at inference generally changes the update because routing weights are nonlinear.This mismatch constitutes the training–inference discrepancy.
  • A.2. Router-Free Decomposition: In a router-free uniform-weight decomposition, the discrepancy is governed entirely by the stability of the shared projection A under stochastic perturbations.The analysis isolates stochastic projection as the source of discrepancy.
  • A.3. Second-Order Taylor Expansion: The second-order Taylor analysis identifies the Hessian of the shared projection as the source of training–inference discrepancy, with routing adding nonlinear amplification.This motivates controlling curvature rather than directly optimizing the computationally prohibitive Hessian norm.
  • A.4. From Consistency Regularization to Curvature Control: CD-LoRA uses consistency regularization between independently perturbed latent representations as a tractable surrogate for curvature minimization.Under an isotropic Gaussian model, the symmetric KL divergence provides a closed-form consistency objective whose dominant sensitivity is second-order.
  • B. Additional Experimental Details: The appendix adds dataset, training-configuration, and implementation details to facilitate CD-LoRA reproducibility.These details complement the main paper’s experimental description.
  • B.1. Experimental Datasets: The five-task setting covers entailment, commonsense reasoning, and multiple-choice question answering, while the twelve-task setting adds mathematical reasoning and linguistic acceptability for stronger heterogeneity.The twelve-task setting includes GSM8K and CoLA, making it suitable for stress-testing multi-adapter coordination.

B.2. BBH Benchmark

This section evaluates CD-LoRA on BBH, a challenging reasoning benchmark spanning diverse task categories. It uses zero-shot, task-specific prompting without BBH training data and reports averaged and aggregate accuracy alongside peak training memory.

  • B.2. BBH Benchmark: BBH tests multi-step reasoning, compositional generalization, and robust instruction following across logical, algorithmic, commonsense, and symbolic tasks.These properties make it suitable for assessing generalization and coordination in parameter-efficient adaptation.
  • B.2. BBH Benchmark: The evaluation follows the standard BBH protocol, averaging accuracy across selected tasks.Models trained in the 12-task multi-task setting are evaluated zero-shot with benchmark-provided task-specific prompts.
  • B.2. BBH Benchmark: No BBH data is used during training, preserving the benchmark’s evaluation as an out-of-training assessment.This condition applies to the reported BBH evaluation for the 12-task multi-task models.
  • B.2. BBH Benchmark: Results report both BBH Avg. across tasks and BBH Ov. aggregated over all examples, while memory is peak GPU usage during training.The memory measure reflects the practical efficiency of the adaptation methods.

B.3. Visualization Setup for Representation Analysis … B.5.2. EXPERIMENT RESULTS

The paper analyzes shared low-rank representations in Qwen2.5-based multi-task LoRA models and evaluates BBH performance under a 12-task setting. The setup emphasizes the shared down-projection bottleneck, while the reported results compare four routing and consistency variants.

  • B.3. Visualization Setup for Representation Analysis: Representations are extracted from Qwen2.5-14B multi-task LoRA models trained on the 12-task B.1.2 mixture with rank r = 4 adapters in FFN projection layers.Experiments are conducted in evaluation mode without gradient updates unless otherwise specified.
  • B.3. Visualization Setup for Representation Analysis: The analysis targets the LoRA-A matrix of the FFN down proj module because it forms the shared low-rank bottleneck and is identified as the dominant source of training–inference discrepancy.This module is used as the focal point for representation analysis.
  • B.3. Visualization Setup for Representation Analysis: Token-level inputs to each LoRA-A module are averaged across sequence length to produce one representation vector per transformer layer, then detached and collected on CPU.One example per task is sampled in the main analysis to avoid bias toward high-resource tasks.
  • B.3. Visualization Setup for Representation Analysis: t-SNE with PCA initialization and a fixed random seed projects representations into two dimensions, using identical axes and aspect ratios across visualizations.Colors denote tasks and marker sizes encode layer depth.
  • B.4. Main Experiment: The total objective combines task loss and consistency regularization as Ltotal = Ltask + λ Lcons, with λ = 0.05 in the main experiments.The consistency mechanism aligns latent representations across LoRA branches.
  • B.4.1. MODEL AND TRAINING CONFIGURATION: CD-LoRA uses rank r = 4, scaling factor α = 32, dropout 0.1, and K = 3 parallel adapters in the FFN gate proj, down proj, and up proj layers.Selected ablations use dropout 0.2.
  • B.5.1. EXPERIMENT SETTINGS: BBH evaluation uses Qwen2.5-7B models trained on the same 12-task B.1.2 benchmark, with identical hyperparameters, optimization, hardware, and evaluation settings for fair comparison.Learning rate, batch size, LoRA rank, and dropout remain consistent with the main experiment.
  • B.5.2. EXPERIMENT RESULTS: Table 5 compares BBH performance for four ablation variants with explicit indicators for routing and consistency alignment.The table reports performance under 12-task training.

C. Similarity Analysis on 14B Models

Similarity analysis on Qwen2.5-14B shows that R-LoRA’s nonlinear routing fragments task-specific subspaces, while CR-LoRA retains uneven and unstable alignment despite consistency enforcement.

  • C. Similarity Analysis on 14B Models: R-LoRA induces fragmented task-specific subspaces on Qwen2.5-14B through nonlinear routing.Figure 7 presents this pattern using a layer-wise Bi similarity matrix.
  • C. Similarity Analysis on 14B Models: CR-LoRA exhibits uneven and unstable subspace alignment even when consistency is enforced.Figure 8 shows this behavior through a layer-wise Bi similarity matrix.
Loading 2608.21909v1…