Source-linked AI summary
New LoRA Skills Should Read but Never Write
Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu
TL;DR
Combining independently trained LoRA adapters into one model remains difficult because merging can interfere, retraining is expensive, and routing keeps adapters separate. READ canonicalizes adapter factors and lets each new skill read a frozen bank before folding the update into one model. Across 32 lineages, it beats the strongest foldable alternatives by +0.073 on average, with losses confined to BBH.
Problem
Weight-space merging can cause interference, joint retraining is expensive, and routing leaves adapters separate; the internal factor form also matters once adapters interact.
Source: §1 · p. 1Method
READ rewrites adapters into canonical coordinates and trains only the new skill’s read direction into a frozen bank before folding the update into the model.
Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4Results
Across all 32 lineages, READ beats the strongest of fourteen foldable alternatives per lineage by +0.073 on average (95% CI +0.047 to +0.101), with all eight losses on BBH.
Source: §1 · p. 2Takeaways & Limitations
Factor coordinates and coupling direction matter for whether composed skills survive, and READ’s folded update serves at base-model latency.
Source: §5 · p. 9All eight terminal losses occur on BBH, and open-ended generative composition and banks beyond six skills lie outside the evaluation.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Low-rank adapters (LoRA) make it cheap to fine-tune a large language model once per task, but combining several independently trained adapters into one model remains difficult: merging the updates in weight space causes interference, retraining on all task data is expensive, and routing between separate adapters gives up the goal of a single combined model. We trace the difficulty to two choices that every composition method makes implicitly. A LoRA update admits infinitely many equivalent factorizations; the choice among them is invisible while an adapter serves alone, but it determines what a learned interaction between adapters can see. A coupling between an old skill and a new one can likewise point in either direction, and the direction decides whether the old skills keep computing what they computed before. We introduce READ (Read-only Expansion of Adapter Deltas), which fixes both choices: each adapter is rewritten into a balanced canonical form that preserves its update exactly, and the coupling grows in one direction only, so a new skill can read the input subspaces of old skills but cannot write into their output subspaces. The only trainable object at each append is the new skill's row of the coupling matrix, and the composed update folds into the base weights with no inference cost, routing, or task-specific rules. We evaluate READ across four benchmark suites and two model families, adding skills one at a time. Across several families, READ improves every suite average over the strongest published baselines built from the same adapters---by more than twenty points on SuperGLUE and more than seven points on the domain suite---and nearly all complete addition sequences end above every direct baseline. Factor coordinates and coupling direction, which a lone adapter never exposes, are what decide whether composed skills survive.
1 INTRODUCTION
Combining independently trained LoRA adapters into one model is difficult: weight-space merges can interfere, retraining is expensive, and routing keeps adapters separate. READ fixes factor coordinates and coupling direction so new skills can interact with a frozen bank, and beats the strongest foldable alternatives across its headline comparison.
1 INTRODUCTION
- Weight-space merging can cause interference, joint retraining is expensive and can damage existing skills, while routing leaves adapters separate.Existing merge arithmetic may require rebuilding the merge and moving old skills when another adapter arrives.Source: §1 · p. 1
- A LoRA update has infinitely many equivalent factor forms, and a learned coupling can inherit arbitrary factor choices that matter only once adapters interact.Coupling direction also determines whether old skills survive when a new skill joins the model.Source: §1 · pp. 1–2
- READ canonicalizes adapter factors, trains only the new skill’s read row into a frozen bank, and folds the resulting update into a single weight change.The composed model serves at base latency without a router or per-task head.Source: §1 · p. 2, Figure 1 · p. 2
- Across all 32 lineages, READ beats the strongest of fourteen foldable alternatives per lineage by +0.073 on average (95% CI +0.047 to +0.101), with all eight losses on BBH.On Llama-3.2-3B, READ scores 0.783 versus 0.605 on SuperGLUE and 0.887 versus 0.846 on Domain.Source: §1 · p. 2
2 RELATED WORK
Prior work either merges completed updates or keeps adapters separate at inference. READ instead learns an interaction as skills are appended while folding the result into the model.
2.1 MERGING IN WEIGHT SPACE
- Weight-space merging methods repair, sparsify, align, or otherwise alter update arithmetic, but a new adapter triggers another global merge that revises earlier skills.READ instead adapts new degrees of freedom to the existing bank while keeping that bank fixed.Source: §2.1 · p. 3
2.2 COMBINING MODULES AT RUN TIME
- Runtime composition preserves separate modules through attention, coefficients, routers, or grafts, but leaves an extra serving mechanism.READ canonicalizes factors, trains the new skill’s read direction into a frozen bank, and folds the result into base weights; routing remains appropriate when each solo adapter must stay intact per task.Source: §2.2 · p. 3
3 THE READ OPERATOR
READ appends a skill to a frozen bank by fixing the adapter coordinates and allowing links only from the new skill into the bank. Its coupling can then be folded into the base weights as one merged model.
3.1 THE FROZEN SKILL BANK
- The stacked factors and coupling G define the bank update as ΔW = Bstack G Astack; identity G gives the naive sum, while the folded rank-kr update needs no router or task-specific path.The source adapters are independent generative-classification LoRAs (r=8) on attention q- and v-projections.Source: §3.1 · pp. 3–4
3.2 CANONICALIZING EVERY ADAPTER
- Canonicalization fixes otherwise arbitrary factor coordinates while preserving each source update, so skills interact in coordinates determined by their updates.The balanced factors share the metric; remaining freedom is a rotation within degenerate singular subspaces that does not change products.Source: §3.2 · p. 4
3.3 READ-ONLY COUPLING
- READ zeros links from the new skill into old output bases and trains only the new skill’s row and diagonal, preserving old functions by construction.Old blocks, canonical factors, backbone, and task head remain frozen.Source: §3.3 · pp. 4–5
3.4 APPENDING AND FOLDING A SKILL
- Each append extracts and canonicalizes the incoming adapter, embeds it beside the frozen bank, trains its new row and diagonal on the task union, then folds the update into the base weights.The fold produces one merged projection per module.Source: §3.4 · p. 5
3.5 ONE READ APPEND AND ITS COST
- Canonicalization costs O(dr^2 + r^3) per module; training optimizes (k+1)r^2 floats per module, while coupling storage grows quadratically in k.Table 1 averages four complete terminal lineages per suite–model and uses the per-lineage strongest alternative as its first baseline row.Source: §3.5 · p. 5, Table 1 · p. 6
4 EXPERIMENTS
Across two model families and four benchmark suites, READ is compared with foldable alternatives on terminal skill banks and sequential appends. The experiments find stronger terminal results overall, while testing append reliability, stability, and the cost of folding for serving.
4.1 SETUP
- The evaluation covers Llama-3.2-3B-Instruct and Qwen3-4B across GLUE-6, SuperGLUE-4, BBH-6, and Domain-3, with one-skill-at-a-time additions.The 32-lineage terminal comparison uses the strongest of fourteen foldable alternatives per lineage; append training uses the full stage task union for all compared growth arms.Source: §4.1 · p. 6
4.2 Q1: DOES READ BUILD A BETTER FINAL SKILL BANK?
- READ improves over the strongest per-lineage foldable alternative on 24 of 32 lineages, by +0.073 on average (95% CI [+0.047, +0.101]); its eight losses are all on BBH.Source: §4.2 · p. 6
4.3 Q2: CAN READ APPEND SKILLS WITHOUT ERASING THE BANK?
- READ passes 72 of 92 eligible append transitions; its mean incoming closure is positive in every model–suite cell and never below 57.8%.Passes require closing at least half the Soup-to-refit gap while keeping mean and worst old-task changes no lower than −2 and −5 points; BBH pass rates are 28.6% on Llama and 40.0% on Qwen.Source: §4.3 · pp. 7–8, Figure 3 · p. 7
4.4 Q3: DOES READ-ONLY GROWTH STABILIZE THE SKILL BANK?
- Canonicalization reduces the mean score spread across equivalent source factorizations from 2.8 to 0.7 points, while random norm-matched factors underperform learned factors in 26 of 27 cells.READ trajectories make small, similarly oriented moves, whereas merge trajectories turn and spread after additions.Source: §4.4 · p. 8, §4.5 · p. 8
4.5 Q4: DOES READ REMAIN A SINGLE-MODEL OPERATOR?
- Old-bank change stays near zero across READ append depths, and folding reproduces the unfolded operator in 128 checks with maximum logit deviation 4.25×10−4.After folding, mean latency remains within roughly one percent of the base model; the deployed checkpoint has no router, task identity, or extra inference branch.Source: §4.5 · pp. 8–9, Figure 5 · p. 9
5 CONCLUSION
READ fixes the factor-coordinate and coupling-direction choices that affect whether composed skills survive. It beats the strongest of fourteen foldable alternatives across 32 terminal lineages, while BBH and other evaluation boundaries remain.
5 CONCLUSION
- The composition problem arises because factor coordinates and cross-skill link direction are hidden choices that determine whether old skills survive interaction with a new skill.A lone adapter does not expose these choices.Source: §5 · p. 9
- READ beats the strongest of fourteen foldable alternatives by +0.073 on average across 32 terminal lineages, and 72 of 92 sequential appends pass the reliability rule.The eight terminal losses all occur on BBH.Source: §5 · p. 9
- READ uses balanced canonical coordinates that preserve each adapter update exactly and a one-directional coupling that lets new skills read, but not write into, old output subspaces.Established factors and couplings remain frozen as the bank grows.Source: §5 · p. 9
- READ’s evaluation leaves open open-ended generative composition, banks beyond six skills, and stronger acquisition on BBH, where all eight terminal losses concentrate.The authors note that coupling remains quadratic in the number of skills, while one pass over the stage union may not provide enough signal for a new skill.Source: §5 · p. 9