PaperlayerView exact revision on arXiv

Source-linked AI summary

Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts

Priya Nair, Lukas Brenner, Maya Lindqvist, Daniel Whitmore, Wen-Hsuan Liu, Tom Saliencro, Amara Okonkwo, Rohan Desai

arXiv:2610.00493v1cs.LG

TL;DR

Mixture-of-LoRA routers typically score tokens without accounting for the updates experts would write, limiting their view of first-order usefulness. VANE scores reader–writer updates through a low-rank descent compass; it outperforms twelve baselines with fewer parameters than an 8-expert mixture.

  • Problem

    Mixture-of-LoRA methods commonly pair one input factor with one output factor and route by token scores, rather than estimating the first-order usefulness of candidate updates.

    Source: §1 · p. 1

  • Method

    VANE factorizes experts into readers and writers and scores every pair’s update against a low-rank forecast of the output descent direction, using additive gates and exact first-order router gradients.

    Source: §1 · p. 2

  • Results

    VANE beats twelve baselines in single-domain and multi-task adaptation with half the parameters of 8-expert mixtures, and its scores track measured usefulness better than token routers.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    First-order update usefulness can be computed for every reader–writer pair in a shared low-rank space, and VANE uses this to route by what experts would write.

    Source: §1 · p. 2, §6 · p. 8

    The analysis assumes a linear-response model with Gaussian tokens, and usefulness is a per-layer first-order quantity that ignores layer and pair interactions and can be myopic far from local linearity.

    Source: Limitations · pp. 8–9

Abstract

from arXiv · show

Mixture-of-LoRA-experts methods raise the capacity of low-rank adaptation by routing each token to a few low-rank experts. Nearly all of them tie one input-side factor to one output-side factor per expert, and nearly all of them route by scoring the token: the router picks experts without seeing what any of them would write. We argue that the router should score the update. To first order, adding an expert's update to a layer output lowers the loss by the inner product between that update and the negative loss gradient at the output. This usefulness is quadratic in the token, so a router that is linear in the token sees only the part of it that runs through the token mean, and routers that rank experts by the norm of their own activations never see the output factor. If each expert is split into a reader (down-projection) and a writer (up-projection), the usefulness of every reader--writer pair becomes an inner product in the shared rank-$r$ space, and all $N_AN_B$ pairs can be scored from $N_A+N_B$ vectors. We build VANE on this identity. A low-rank compass predicts the descent direction of each token. VANE scores every pair by the alignment between its update and the compass without forming any update, activates the top-$k$ pairs with additive gates, and gives every pair its exact first-order router gradient. On single-domain commonsense reasoning and a four-domain multi-task mixture with Llama-3.2-3B and Llama-3.1-8B, VANE attains the best average among twelve PEFT and MoE-LoRA baselines, by 0.9--1.1 and 1.3--1.5 points respectively, with less than half the trainable parameters of an 8-expert MoE-LoRA. Its router scores also track the measured usefulness of experts far more closely than token routers do.

1 Introduction

Existing mixture-of-LoRA routers score tokens rather than the updates experts would make, while paired factors limit which combinations can be represented. VANE scores reader–writer updates by their alignment with a predicted descent direction and performs strongly against existing baselines.

1 Introduction

  • Existing designs pair one input-side factor with one output-side factor and route by token scores that do not reflect what experts would write.Combinatorial designs and self-routing alter factor combinations or use activation size, but the introduction says no design specifies what the router should estimate.Source: §1 · p. 1
  • An expert’s first-order usefulness is the alignment between its update and the loss descent direction; linear token routers miss token-specific usefulness for centered tokens.Reader-side norm routing also cannot distinguish writers that read the same features but produce different outputs.Source: §1 · p. 1, Figure 1 · p. 2
  • Splitting experts into readers and writers makes each pair’s usefulness an inner product in rank-r space, allowing all N_A N_B pairs to be scored from N_A + N_B latent vectors.The introduction identifies the reader as the down-projection and the writer as the up-projection.Source: §1 · p. 2
  • VANE uses a low-rank compass to predict the descent direction, scores pair updates without forming them, selects top-k pairs with additive gates, and supplies an exact first-order router gradient for every pair.Its design combines reader–writer experts, latent update scores, additive gates, and a dense router gradient.Source: §1 · p. 2
  • VANE beats twelve baselines in single-domain and multi-task adaptation with half the parameters of 8-expert mixtures, and its scores track measured usefulness better than token routers.The reported usefulness correlation is ρ = 0.61 for VANE versus ≤0.24 for token routers.Source: §1 · p. 2

2 Related Work

Related work develops mixtures of LoRA experts, shared or combinatorial factors, and routing mechanisms coupled to expert computation. VANE’s distinction is to use a shared reader–writer latent structure for exact first-order scoring.

2 Related Work

  • Mixture-of-LoRA work routes tokens among paired experts and explores expert allocation, dynamic k, communication, semantic priors, and heterogeneous or router-free designs.These methods refine routing decisions, while the paper frames its contribution as specifying what the router should estimate.Source: §2 · p. 2
  • Shared-factor and mix-and-match designs construct experts from shared projections, rank-one components, residual trees, shards, vector banks, or factorized expert tensors.VANE uses a reader-times-writer structure and exploits the shared rank-r latent to score every pair’s first-order effect.Source: §2 · pp. 2–3
  • Existing input-coupled routers use input-side activations or projections, whereas VANE’s scores use writers and a forecast of the loss gradient, with an exact dense gradient.The cited review contrasts this with gradient densification using averaged past outputs and methods estimating gradients through discrete routing.Source: §2 · p. 3

3 What Should a Router Compute?

The section defines routing usefulness as the first-order loss reduction from an expert update and shows that it factorizes in a shared latent space. It characterizes what token- and reader-side routers can represent, then gives a condition under which a low-rank descent predictor recovers usefulness exactly.

3 What Should a Router Compute?

  • Adding an expert update changes the loss to first order by minus its alignment with the output descent direction, making that alignment the routing target under additive gates.The router must estimate usefulness from the token because the descent direction depends on the label.Source: §3 · p. 3
  • The usefulness matrix for all reader–writer pairs factorizes as ZᵀW, so pair scores can be computed from reader latents and writer queries without materializing each update.The stated cost is O(r(N_A d + N_B d′) + N_A N_B r^2), versus O(N_A N_B r d′) to materialize updates.Source: §3 · p. 3
  • Under the linear-response model δ(x) = Γx + ξ, expected usefulness is quadratic in the token, while affine token routers have a representational ceiling that is zero when the token mean is zero.The theorem’s model assumes E[ξ | x] = 0 and Gaussian tokens.Source: §3 · p. 3
  • A score based only on a reader’s token latent assigns the same score to every writer for that reader, capturing only the gain of a uniformly random writer within it.This limitation follows because such scores do not use writer-specific information.Source: §3 · p. 3
  • If the compass e(x) = UVᵀx satisfies UVᵀ = Γ, it equals expected usefulness for every pair and token when its rank is at least rank Γ.Under that condition, top-k selections maximize expected first-order gain among selections based on x.Source: §3 · p. 3
  • Token routers capture only the part of usefulness associated with the token mean, whereas an output-space vector can score each candidate update by its predicted loss effect.The controlled-model figure is reported to match the closed-form ceiling within 0.002.Source: §3 · p. 3, Figure 2 · p. 4

4 VANE

VANE uses a low-rank compass and reader–writer queries to score candidate updates by calibrated alignment, then activates selected pairs with additive gates. Its design also supplies dense first-order router gradients and includes guarantees about compass updates, parameterization, and pair-gate rank.

4 VANE

  • A low-rank compass forecasts the descent direction, and each writer transforms it into a latent query that moves with the writer.The compass is used to score updates rather than being added to the layer output.Source: §4 · p. 4
  • Calibration scores update alignment per unit of update norm, making routing depend on update direction and preventing growing writers from being selected more often merely because they grow.The method uses stop-gradients on reader latents and writer matrices, leaving the compass as the router parameter.Source: §4 · p. 4
  • The active set selects the k highest-scoring pairs and uses additive sigmoid gates, so complementary pair gains can add rather than being averaged by softmax normalization.The layer is described as two dense low-rank products around a sparse reader–writer core.Source: §4 · p. 4
  • Unselected pairs receive zero-forward, active-gate derivatives, giving every pair its exact first-order router gradient without materializing expert outputs.The added cost is O(N_A N_B r) per token; the cited comparison says default MoE approximates dense gradients with averaged past outputs.Source: §4 · p. 4
  • With the straight-through core, a gradient step on the compass output map turns the compass toward the descent direction within the span of candidate updates.The stated guarantee is for a single token and gives a nonnegative inner product between the compass change and descent direction.Source: §4 · p. 5
  • Pair gates can raise update rank to r · rank(G_S), up to r min(k, N_A, N_B), while separable gates collapse the rank to r.The section also reports that routing scores are invariant to specified changes of reader and writer parameterization.Source: §4 · p. 5

5 Experiments

Across single-domain and four-domain adaptation, VANE outperforms strong baselines while using fewer trainable parameters than eight-expert mixtures. Ablations and mechanism analyses examine its routing score, training signals, pair selection, and efficiency.

5.1 Setup

  • The experiments fine-tune Llama-3.2-3B and Llama-3.1-8B on single-domain commonsense and a four-domain mixture, comparing VANE with single-adapter and MoE-LoRA baselines.The multi-task setup trains on math, code, medicine, and commonsense; results are means over three seeds.Source: §5.1 · p. 5

5.2 Main Results

  • VANE improves on the best baseline by +1.1 on Llama-3.2-3B and +0.9 on Llama-3.1-8B in single-domain accuracy, using less than half the trainable parameters of 8-expert mixtures.VANE averages 81.9 and 86.9, respectively; Table 1 reports single-domain commonsense accuracy and parameter share.Source: §5.2 · p. 5, Table 1 · p. 6
  • On the four-domain mixture, VANE improves on the best baseline by +1.5 on 3B and +1.3 on 8B, and on LoRA by +3.9 and +3.5.The per-domain 8B gains over LD-MoLE are +1.8 for math, +2.0 for code, +1.1 for commonsense, and +0.5 for medicine.Source: §5.2 · pp. 5–6, Table 2 · p. 7

5.3 Ablations

  • Routing-score ablations cost 1.3–2.0 multi-task points, while their usefulness correlations are at most 0.24 versus 0.61 for VANE.The linear-over-pairs, latent-linear, reader-norm, and product routers lose 1.4, 1.3, 1.9, and 2.0 points, respectively.Source: §5.3 · p. 6
  • The dense first-order router gradient is worth 0.9 points and raises usefulness correlation from 0.43 to 0.61; paired experts with the same score lose 1.0 point.A single shared reader loses 0.8 points, separable gates lose 1.7, and softmax gates lose 0.7.Source: §5.3 · p. 6

5.4 Mechanism Analysis

  • VANE’s selected pairs capture 0.78 of the oracle first-order gain, compared with 0.41 for the token router and 0.04 for reader norms.The compass cosine with the true descent direction reaches 0.29 in MLP and 0.21 in attention modules, and 0.52 within the span of the candidate updates.Source: §5.4 · p. 7
  • Domains select a few writers but use all readers; domain–writer mutual information is 0.60 bits versus 0.02 bits for readers.The analysis reports that adding writers helps more than adding readers.Source: §5.4 · p. 8

5.5 Efficiency, Sparsity, and Scale

  • VANE lies on the accuracy–parameter frontier; k=4 of 32 pairs is best, and gains over LoRA rise from +3.5 on 8B to +4.4 on Llama-3.2-1B.Training takes 1.26× LoRA’s time; unmerged decoding takes 27.2 versus 36.2 ms per token.Source: §5.5 · p. 8, Figure 4 · p. 8

6 Conclusion

The conclusion frames update usefulness—the alignment of an expert update with the descent direction—as the routing target. VANE operationalizes this with reader–writer pairs and reports gains over PEFT and MoE-LoRA baselines with fewer parameters than an eight-expert mixture.

6 Conclusion

  • A router should estimate how much an expert’s update would lower loss, measured to first order by alignment with the output’s descent direction.VANE computes this for reader–writer pairs in a shared low-rank space and learns from each pair’s exact first-order effect.Source: §6 · p. 8
  • VANE outperforms twelve PEFT and MoE-LoRA baselines with half the parameters of an 8-expert mixture.Source: §6 · p. 8
  • The conclusion identifies shared low-dimensional expert spaces as a potential setting for the same principle, including sparse MoE pretraining and adapter merging.Source: §6 · p. 8

Limitations

Theorem 1 is limited to representational claims under linear response with Gaussian tokens, rather than claims about training. The usefulness measure is first-order and per-layer, so it omits interactions and may be myopic when the loss is far from locally linear.

Limitations

  • Theorem 1 assumes linear response and Gaussian tokens, and describes router expressivity rather than how routers train.Its transfer to real runs is assessed through measured score–usefulness correlation.Source: Limitations · pp. 8–9
  • First-order, per-layer usefulness ignores interactions across layers and pairs and can be myopic when the loss is far from locally linear.For unselected pairs, the dense router gradient uses the first-order surrogate; only selected pairs receive the true gradient.Source: Limitations · p. 9
  • Experiments cover models up to 8B parameters and four domains; larger backbones, more domains, and generation-heavy tasks remain untested.VANE computes every reader and writer for each token, so larger expert pools would need a sparse writer path.Source: Limitations · p. 9

Ethics Statement

The section places VANE among related adapter and routing methods, then frames low-rank mixtures through shared reader and writer factors. It also states the method’s first-order scoring algebra and ethical scope.

Ethics Statement

  • The authors report public licensed datasets, no human subjects or personal data, and caution that medical multiple-choice accuracy does not make VANE-adapted models medical devices.They also note that adapting models inherits model and data risks, including biases and lower misuse costs.Source: Ethics Statement · p. 9

A Extended Related Work

  • Prior work varies expert allocation, factor sharing, and routing signals, while leaving open what quantity a router should estimate; VANE derives routing from that target.Related work includes factorized expert tensors, where VANE’s reader–writer product evaluates pairs through a shared rank-r latent space.Source: A Extended Related Work · pp. 14–15

B A Unified View of Low-Rank Mixtures

  • The unified view represents low-rank mixtures with a gate matrix over reader–writer factors, and VANE sparsely selects pairs using update scores.All pairwise usefulness values form Z^T W, whose dimensions depend on factor counts rather than the number of pairs.Source: B A Unified View of Low-Rank Mixtures · p. 16, Table 4 · p. 16

C Proofs

  • The proofs define token, reader, writer, descent-direction, and usefulness quantities, then express usefulness as an inner product in rank-r space.For all pairs, the scores are collected as Z^T W.Source: C Proofs · p. 16, §C.1 · p. 16

C.1 Proof of Proposition 1

  • Scoring all pairs uses reader and writer projections plus a latent matrix product, avoiding the cost of materializing every update.The proof gives costs of N_A r d and N_B r d′ multiply-adds for Z and W, followed by N_A N_B r for Z^T W.Source: §C.1 · p. 16

C.2 Proof of Theorem 1

The proofs bound what linear and reader-only routers can represent, establish the role of an update-based score, and analyze the method’s gradient, calibration, and factor-sharing properties. Numerical verification checks the statements on randomly generated layers.

C.2 Proof of Theorem 1

  • When κ = 1, token routers explain at most about two thirds of expected usefulness variance, while reader-side scores cannot rank writers by usefulness.The mean-interaction to total-variance ratio is 2κ/(1 + 2κ); a rank-re sufficient compass can select the top-k pairs by expected usefulness.Source: §C.2 · pp. 16–17
  • Theorem 1’s noise adds unpredictable realized-usefulness variance, while a compass with rank below rank Γ uses reduced-rank regression of descent direction on tokens.The stated correlation saturates once re reaches rank Γ = 6.Source: §C.2 · p. 17

C.3 Proof of Proposition 2

  • For a single token, the compass update has nonnegative alignment with the descent direction, but batch updates mix tokens through their h-vector inner products.The single-token alignment is positive unless δ is orthogonal to every candidate update.Source: §C.3 · p. 17

C.4 Proof of Proposition 3

  • Calibrated scores are invariant to writer rescaling, avoiding the raw-score feedback loop in which growing writers are selected more often and receive more task gradient.This calibration gives up some static optimality, while removing it costs 0.6 points in training.Source: §C.4 · pp. 17–18

C.5 Proof of Proposition 4

  • Under balanced marginals, each reader receives gradient from N_B times more token events and each writer from N_A times more than paired experts.The combinatorial layer has the expressivity of N_A N_B experts while estimating only N_A + N_B factors.Source: §C.5 · p. 18

D Numerical Verification

  • Numerical verification checks the statements in float64 on random layers with d=48, d′=40, N_A=4, N_B=6, r=3, and re=5.Table 5 reports the worst case over random draws.Source: D Numerical Verification · p. 18

E The Controlled Linear-Response Study

The controlled study tests how well routing scores predict first-order usefulness and capture gain across 64 reader–writer pairs. VANE’s update-based scores outperform token and latent-linear routers in the reported setting, while implementation details describe dense scoring and inference paths.

E The Controlled Linear-Response Study

  • In the controlled linear-response study (k = 4), VANE captures expected first-order gain of 0.875, versus 0.688 for the token router and 0.643 for the latent-linear router.Source: E The Controlled Linear-Response Study · p. 18, Figure 5 · p. 19
  • The product router reaches −0.11, worse than random, because adding reader and writer logits can rank a harmful pair highly.The learned compass reaches 0.875, close to the regression-fitted compass’s 0.88 at κ = 0.5.Source: E The Controlled Linear-Response Study · p. 18
  • Raw update scores capture all attainable gain up to 256 pairs, while calibrated-score performance declines from 0.99 to 0.80 as the pool grows.Calibration ranks by alignment per unit norm, which can diverge from ranking by total gain.Source: E The Controlled Linear-Response Study · p. 18, Figure 5 · p. 19
  • When token means dominate (κ ≥2), token routers capture 0.91–0.97 of gain, exceeding the calibrated score’s 0.88; removing calibration costs 0.6 points in training.The controlled study fixes experts to isolate routing, and all reported numbers are computed rather than simulated.Source: E The Controlled Linear-Response Study · pp. 18–19

F Implementation Details

  • At inference, VANE drops the straight-through term and uses a sparse writer path for single-sequence decoding or a dense path for batched decoding.Its dense path uses the same few kernels for every token; token-dependent active pairs prevent merging the adapter into the base weights.Source: F Implementation Details · p. 20

G Experimental Setup

The experiments evaluate VANE in single-domain commonsense adaptation and a four-source multi-task setting. The setup specifies benchmark splits, decoding, shared training conditions, and three-seed reporting.

G Experimental Setup

  • The single-domain training set is Commonsense170K, the union of eight commonsense benchmark training sets, evaluated on their standard test splits.The evaluation uses LLM-Adapters prompts and answer extraction.Source: G Experimental Setup · p. 20
  • The multi-task mixture contains 120k examples, drawing 30k each from MetaMathQA, Magicoder-Evol-Instruct-110K, MedMCQA, and a fourth source.Evaluation covers math, code, medicine, commonsense, and other benchmarks, using exact match, greedy pass@1, or accuracy as appropriate.Source: G Experimental Setup · p. 20
  • All methods share the optimizer, schedule, batch size, epochs, and learning rate; the selected rate is 2×10^-4, best or tied for every method.Runs use eight H100 GPUs, and reported numbers are means over seeds 0, 1, and 2.Source: G Experimental Setup · p. 20

H Additional Results

Additional results examine seed variability, training behavior, domain and backbone robustness, model configurations, and compute costs. VANE’s reported advantages vary across domains and design choices, with further evidence on data efficiency and retention.

H Additional Results

  • The 95% confidence interval for the paired per-seed difference between VANE and LD-MoLE excludes zero in every reported setting.Table 8 reports means and standard deviations over three seeds for the strongest methods.Source: H Additional Results · p. 20, Table 8 · p. 22
  • VANE’s margin is largest on code and math and smallest on medicine, where PubMedQA is close to saturation for all methods.The domain view averages benchmark results within each domain.Source: H Additional Results · p. 20, Table 9 · p. 22
  • On Llama-3.2-1B multi-task averages, VANE scores 45.6, versus 43.4 for LD-MoLE, 42.8 for MixLoRA, and 41.2 for LoRA.On Qwen2.5-7B and Qwen3-8B, margins over LD-MoLE are +0.9/+1.3 and +0.8/+1.2 on single-domain/multi-task settings.Source: H Additional Results · pp. 20–21
  • At a fixed budget, writers are the better investment; eight readers do not beat four, and the (4, 16) configuration gains only 0.1 over default for 52% more parameters.The reported grid varies reader and writer counts at fixed rank and compass settings.Source: H Additional Results · p. 21
  • Accuracy and score–usefulness correlation saturate at compass rank re = 8, while balancing weight and writer initialization matter little within an order of magnitude of defaults.The sensitivity results include active-pair and compass-rank sweeps.Source: H Additional Results · p. 21
  • On Llama-3.1-8B, VANE trains 1.26× slower than LoRA, like HydraLoRA, but 34% faster than LD-MoLE because it runs two dense products instead of per-expert dispatch.Source: H Additional Results · p. 21
  • With 10% of the multi-task mixture, VANE leads LD-MoLE by 1.6 points; it also drops least on zero-shot MMLU after multi-task training.Restricting adaptation to attention or MLP projections costs accuracy, and MLP projections matter more.Source: H Additional Results · p. 21
  • During training, VANE has lower training loss and higher held-out accuracy; 62% of tokens use four distinct readers and writers, and active gates sum to 2.3 on average.The score-class ordering runs from reader norm and product scores through token and latent-linear routers to update scores; compass alignment is highest in middle layers and MLP projections.Source: H Additional Results · p. 21, Figure 6 · p. 22, Figure 7 · p. 22

I Discussion

The discussion frames routing as selecting updates aligned with the loss descent direction and explains when token-based alternatives can suffice. It also identifies conditions that constrain broader use and large-pool scaling.

I Discussion

  • For additive updates, first-order usefulness is alignment with the output descent direction, a target that depends on what an expert writes.Token routers do not see the writer’s output, and reader-side scores ignore it.Source: I Discussion · pp. 22–23
  • Token routers capture the mean-interaction part of usefulness and can work nearly as well when token means dominate or tasks separate by their means.The discussion links this condition to simple routers working in some MoE-LoRA benchmarks and to VANE’s larger margin in multi-task settings.Source: I Discussion · p. 23
  • The shared-low-dimensional-space scoring principle is not specific to fine-tuning and also applies to other mixtures whose experts factor through such a space.Examples include low-rank or product-key experts in sparse MoE pretraining.Source: I Discussion · p. 24
  • For independently trained adapter composition, the shared gauge requires jointly trained factors or readers frozen and common across adapters.Without that shared gauge, the proposed reader–writer combinations require care.Source: I Discussion · p. 24
  • Beyond a few hundred pairs, VANE’s dense path requires a sparse writer path or product-key retrieval over the score matrix.Other listed failure modes include first-order misestimation for large updates in nonlinear regimes and similar scores from nearly parallel writers.Source: I Discussion · p. 24
Loading 2610.00493v1…