PaperlayerView exact revision on arXiv

Source-linked AI summary

JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)

Sara Abdali, Pashmina Cameron

arXiv:2610.07036v1cs.AIcs.LG

TL;DR

Most low-rank adapters do not explicitly separate structure shared across related tasks from task-specific variation. JIVEAdapter makes that split additive and orthogonal, allocates rank across both structures, and is competitive with strong low-rank baselines on GLUE and SuperGLUE at matched per-task effective rank.

  • Problem

    Common low-rank adapters lack an explicit account of what is shared across related tasks and what is task-specific.

    Source: §1 · p. 1

  • Method

    JIVEAdapter adds a shared Joint and orthogonal per-task Individuals, allocates rank across their pools, and freezes the learned Joint for reuse on later tasks.

    Source: §1 · pp. 1–2

  • Results

    On GLUE and SuperGLUE with DeBERTaV3-base, JIVEAdapter is competitive with strong single-task and fully shared multi-task low-rank baselines at matched per-task effective rank.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A later task can reuse the frozen Joint with a similar task’s Individual or train a small new Individual, without retraining the shared part.

    Source: §1 · p. 2

Abstract

from arXiv · show

Parameter-efficient fine-tuning adapts pretrained models at a fraction of the cost of full fine-tuning, yet most low-rank adapters are single-task and represent each weight update multiplicatively, leaving no explicit account of what is shared across tasks and what is task-specific. We introduce JIVEAdapter, a multi-task "additive" low-rank adapter inspired by statistical Joint and Individual Variation Explained (JIVE). JIVEAdapter decomposes every weight update into a Joint structure shared across all tasks plus a per-task Individual structure, penalizes the Individual structures to be near-orthogonal to the Joint so shared and task-specific signal stay "interpretable" and separated, and allocates rank adaptively across a shared Joint pool and a per-task Individual pool. The Joint is learned once, jointly over a task group or incrementally, one task at a time, then frozen and reused as a prior for new tasks without retraining the shared part. On GLUE and SuperGLUE with DeBERTaV3-base, JIVEAdapter is competitive with strong single-task and multi-task low-rank baselines at a matched per-task effective rank, without extra modules such as MoE, and when a related held-in task exists its frozen Joint serves a held-out task by reusing that task's Individual with only a cheap per-direction scale, otherwise training a small new one.

1 INTRODUCTION

JIVEAdapter separates shared and task-specific variation in an additive, orthogonal low-rank update, with rank allocated across both structures. It is reported competitive with strong single-task and multi-task baselines at matched per-task effective rank.

1 INTRODUCTION

  • Full fine-tuning stores and updates a separate copy of every weight per task, while common low-rank adapters provide no explicit account of shared versus task-specific variation.In multi-task settings, one adapter per task relearns shared structure, whereas one shared adapter suppresses task-specific variation.Source: §1 · p. 1
  • JIVEAdapter represents each weight update as an additive sum of a shared Joint and per-task Individuals constrained to be orthogonal to the Joint.The split follows the JIVE principle: the Joint captures overlapping variation and each Individual captures non-overlapping variation.Source: §1 · p. 2
  • Rank is allocated adaptively across a shared Joint pool and per-task Individual pools, rather than across layers as in AdaLoRA.The separate pools are intended to prevent many per-task structures from starving the shared Joint.Source: §1 · p. 2
  • The Joint can be learned jointly over a task group or incrementally, then frozen and reused for later tasks without retraining the shared part.A later task can reuse a similar task’s Individual or train a small new one.Source: §1 · p. 2
  • On GLUE and SuperGLUE with DeBERTaV3-base, JIVEAdapter is competitive with strong single-task and fully shared multi-task low-rank baselines at matched per-task effective rank.The analysis also finds that Joint captures shared structure while Individuals capture task-specific variation.Source: §1 · p. 2

2 RELATED WORK

The related work contrasts JIVEAdapter with single-task and multi-task low-rank adapters. Its distinguishing combination is an explicit orthogonal shared/task-specific split, adaptive per-structure rank, and a reusable frozen Joint.

2 RELATED WORK

  • Unlike single-task LoRA and AdaLoRA, JIVEAdapter decomposes updates into a sum of task-specific orthogonal structures and allocates rank per structure rather than per layer.AdaLoRA is identified as the closest single-task counterpart; it reparameterizes updates as an SVD triplet and prunes singular values under a global budget.Source: §2 · p. 2
  • Multi-task counterparts MTL-LoRA and HydraLoRA add task-specific factors on a shared down-projection, while JIVEAdapter uses an additive Joint–Individual split without MoE routing.The paper describes the JIVEAdapter split as interpretable and reusable, with per-task rank allocated adaptively.Source: §2 · p. 2
  • JIVEAdapter combines explicit shared and task-specific subspaces, orthogonality, per-structure adaptive rank, and a frozen Joint that transfers to new tasks.For unseen tasks, the Joint can be reused by similarity or paired with a small new Individual.Source: §2 · p. 2

3 METHOD

JIVEAdapter applies the JIVE decomposition to parameter-efficient adaptation, separating each task update into shared and task-specific structures. It learns the Joint through multi-task or incremental training, then freezes it for reuse on later tasks.

3.1 FORMULATION

  • Each weight update is the sum of a shared Joint and a task-specific Individual, with the Individual constrained to an orthogonal subspace.Each structure is parametrized as a low-rank SVD triplet; rank-based scaling keeps effective update magnitude roughly constant as rank varies.Source: §3.1 · pp. 2–3

3.2 PROPOSED JIVEADAPTER

  • JIVEAdapter first learns a shared Joint with per-task Individuals, then freezes the Joint and either reuses a similar task’s Individual or trains a new one.The Joint can be learned jointly across related tasks or incrementally as tasks arrive.Source: §3.2 · p. 3

3.2.1 STRUCTURE LEARNING STAGE: EXTRACTING THE JOINT STRUCTURE

  • In multi-task training, the shared Joint applies to every sample while task IDs route each sample to its own Individual.Mixed batches give the Joint gradients from every task and each Individual gradients only from its task; the authors report that alternating per-task batches can make the Joint oscillate.Source: §3.2.1 · p. 4
  • Rank is allocated separately to the shared Joint and per-task Individual pools using either importance-based pruning or a budget-free permutation-test estimator.The estimator runs once before training, while the importance-based allocator prunes singular values under a decaying budget.Source: §3.2.1 · p. 4
  • Incremental adaptation reestimates the Joint over existing and incoming tasks, then recovers each task’s Individual as a rank-limited residual orthogonal to the updated Joint.The updated Joint is warm-started from the current one; a sufficiently informative new task can refine the shared structure.Source: §3.2.1 · p. 5

3.2.2 STRUCTURE REUSE STAGE

  • For a new task, JIVEAdapter freezes the pretrained weights and Joint, then either reuses the nearest task’s Individual with learned per-direction gains or trains a new Individual.Similarity is based on overlap between task signatures derived from frozen-model gradients; the reused Individual basis stays fixed while its gains are learned.Source: §3.2.2 · pp. 5–6

4 EXPERIMENTS

Experiments compare JIVEAdapter with low-rank baselines on GLUE and SuperGLUE using DeBERTaV3-base, and examine joint and incremental structure learning, held-out reuse, and rank estimation and interpretability.

4.1.1 SINGLE-TASK COMPARISON (T=1)

  • 89.44 on GLUE and 78.65 on SuperGLUE are JIVEAdapter's best single-task averages, ahead of both LoRA and AdaLoRA.GLUE baselines are reproduced AdaLoRA (89.14) and LoRA (87.90); SuperGLUE baselines are LoRA (78.41) and AdaLoRA (77.21).Source: §4.1.1 · p. 7, Table 1 · p. 8

4.1.2 STRUCTURE LEARNING STAGE: EXTRACTING THE JOINT STRUCTURE

  • 89.07 on GLUE exceeds baseline averages of 88.80 and 88.42; on SuperGLUE, JIVEAdapter's 82.06 is comparable to HydraLoRA's 82.24 overall average.The SuperGLUE result uses fair-rank at effective per-task rank 8; HydraLoRA's overall average is driven mainly by its perfect score on the smallest task.Source: Table 2 · p. 8, §4.1.2 · p. 8
  • Incremental folding refines the shared Joint and incoming task's Individual without revisiting earlier training data; it is recommended when all-at-once training is impractical.The recommendation applies when the task pool grows over time, base-pool data is inaccessible, or maintaining a separate head per task is too expensive.Source: §4.1.2 · p. 8

4.1.3 STRUCTURE REUSE STAGE

  • 70.62 vs 81.82 on heterogeneous SuperGLUE, but 89.33 vs 89.36 on same-type MNLI genres, for similarity-based reuse versus adaptation.Similarity-based reuse routes by subspace overlap and rescales a held-in task's Individual without training a new basis; higher overlap is associated with better reuse performance.Source: §4.1.3 · pp. 8–9, Figure 2 · p. 9
  • Adaptation keeps the Joint frozen and fits a new Individual; it wins on every held-out SuperGLUE task, matching or approaching all-eight-task scores.The passage identifies similarity-based reuse as a cheaper match for adaptation when held-out MNLI genres are very close to held-in genres.Source: §4.1.3 · p. 9

4.2 RANK ESTIMATION AND INTERPRETABILITY

  • The budget-free permutation-rank estimator matches or improves on importance allocation on multi-task SuperGLUE, while removing the rank-budget hyperparameter.Keeping shared and task-specific structures separate also permits each to be measured, ablated, and visualized independently.Source: §4.2 · p. 9

5 CONCLUSION

JIVEAdapter combines a shared Joint with orthogonal task-specific Individuals in an adaptively ranked additive update, supporting structure learning and reuse.

5 CONCLUSION

  • Each weight update is an additive low-rank combination of a shared Joint and orthogonal per-task Individuals, with rank allocated across both pools.The Joint can be built all at once or incrementally, then frozen to serve new tasks by reusing a related Individual or training a small new one.Source: §5 · p. 9
  • On GLUE and SuperGLUE with DeBERTaV3-base, JIVEAdapter matches strong single-task and multi-task low-rank baselines at matched per-task effective rank without MoE routing.The incremental variant is reported to mitigate catastrophic forgetting without revisiting earlier data.Source: §5 · p. 9
  • Gradient-based nonlinear training over a frozen backbone recovers the JIVE-grounded split from zero-initialized factors, keeping its two parts interpretable and separately measurable.Source: §5 · p. 9

USE OF AI TOOLS

The authors describe the origins of the work and the limited supporting role of AI tools.

USE OF AI TOOLS

  • The authors attribute the ideas to their own work, prior work, and extensions of cited related work.Source: USE OF AI TOOLS · p. 10
  • AI tools assisted with baseline integration, phrasing, LaTeX drafting, figure and table polishing, and grammar correction.Source: USE OF AI TOOLS · p. 10
  • The authors state that they devised, checked, and approved every technical contribution, claim, and conclusion.Source: USE OF AI TOOLS · p. 10

ETHICS STATEMENT

The study reports evaluation on established public benchmarks and states that it involved no human subjects or private data.

ETHICS STATEMENT

  • The methodological study uses GLUE, SuperGLUE, and MNLI under standard licenses, with no human subjects, private or personally identifiable data, or new data collection.Source: ETHICS STATEMENT · p. 10
  • The authors foresee no direct harmful application specific to the adapter beyond risks common to pretrained language models.They state that the method neither amplifies nor mitigates those models' known bias and fairness limitations by design.Source: ETHICS STATEMENT · p. 10
  • The authors declare no conflicts of interest and no external sponsorship that could bias the reported results.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix documents implementation and experimental details intended to support reproduction, then surveys related adaptation methods and how they compare with JIVEAdapter.

REPRODUCIBILITY STATEMENT

  • The method, experimental setup, extended studies, and anonymized JIVELayer implementation are documented; the full repository is deferred until acceptance.The setup includes datasets, task lists, optimization settings, and a single fixed random seed shared across methods.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORK

  • Related multi-task approaches include orthogonal subspaces for continual learning and composing separately trained LoRA modules for unseen tasks.O-LoRA uses orthogonal low-rank subspaces across tasks; LoRAHub composes separately trained modules.Source: A EXTENDED RELATED WORK · p. 13
  • Table 3 summarizes representative adaptation methods along axes relevant to JIVEAdapter’s contribution.The appendix also covers prompt and prefix tuning, BitFit, DoRA, bottleneck adapters, and several multi-task approaches.Source: Table 3 · p. 13, A EXTENDED RELATED WORK · p. 13

B ALGORITHMS AND PROOFS

The appendix details adaptive and permutation-test rank selection, explains a subspace-overlap score, and records the fixed-seed experimental protocol.

B ALGORITHMS AND PROOFS

B.1.1 ADAPTIVE RANK ALLOCATION ALGORITHM

  • Rank allocation uses separate Joint and Individual pools, with singular values pruned within each pool under a decaying active-rank budget.The importance allocator scores singular values using sensitivity and uncertainty, then masks the lowest-scoring values to meet each pool’s budget.Source: §B.1.1 · pp. 13–14

B.1.2 PERMUTATION-TEST RANK ESTIMATION ALGORITHM

  • The budget-free permutation test estimates ranks before training from task-averaged gradients, then fixes adapter ranks without pruning during training.Per-module Joint ranks are aggregated by their median; Individual ranks are averaged across tasks per module before taking the median.Source: §B.1.2 · p. 14

B.2 PRINCIPAL-ANGLE SUBSPACE OVERLAP

  • The overlap score used for nearest-task routing is based on principal angles between the Joint and task subspaces.The singular values of G^TQ equal the cosines of the principal angles; the normalized squared-cosine score ranges from zero for orthogonal subspaces to one for identical subspaces.Source: §B.2 · p. 15

C EXPERIMENTAL SETUP

  • All reported experiments use one fixed random seed shared by JIVEAdapter and baselines, controlling seed variation and enabling exact run reproduction.Source: C EXPERIMENTAL SETUP · p. 15

C.1 COMPUTE AND SOFTWARE ENVIRONMENT

The experimental appendix documents the compute environment and matched-rank configurations, then reports incremental-learning and rank-estimator comparisons.

C.1 COMPUTE AND SOFTWARE ENVIRONMENT

  • Each job uses one A100 GPU, and the DeBERTaV3-base adapter setup fits within its 80 GB memory budget.The methods share the same hardware, image, and driver stack.Source: §C.1 · pp. 15–16

C.2 PARAMETERS AND INVARIANTS

  • Implementation checks verify zero initial updates, routed-forward agreement to about 10^-6, and task-specific gradient flow to Individuals.When batches contain only one task, accumulating one micro-batch per task before a shared step recovers the same gradient.Source: §C.2 · p. 16

C.3 CONFIGURATION AND RANK BUDGETS

  • The multi-task comparison matches per-task effective rank at eight, splitting JIVEAdapter capacity between a rank-4 Joint and rank-4 Individual.The compared methods share the backbone, injected modules, and optimization schedule; JIVEAdapter routes samples to task-specific Individuals, including position projections required for convergence.Source: §C.3 · p. 16, Table 5 · p. 17

D.1 INCREMENTAL JIVEADAPTE R AS CONTINUAL LEARNING

  • Incremental JIVEAdapter trails joint all-at-once training but outperforms sequential references on GLUE and SuperGLUE under matched rank budgets.GLUE scores are 81.13 and 79.54 versus 89.07 for joint training; SuperGLUE scores are 80.72 and 73.69 versus 82.06. Sequential references score 61.11 and 69.55 on GLUE, and 68.36 and 65.60 on SuperGLUE.Source: Table 6 · p. 17, §D.1 · p. 17

D.2 IMPORTANCE-BASED VS. PERMUTATION-TEST RANK ESTIMATION

  • 82.06 vs. 81.21 on SuperGLUE favors the permutation test, while 89.07 vs. 88.54 on GLUE gives the importance allocator a small edge.Neither estimator dominates across benchmarks; the budget-free permutation test is the more convenient default where it remains competitive.Source: Table 7 · p. 18, §D.2 · p. 18

D.3 LOW-RANK STRUCTURE OF THE TASK ADAPTATION

The GLUE analysis finds nearly unchanged mean performance across a fourfold rank range, while the interpretability analysis tests separation, task geometry, and rank-budget allocation.

D.3 LOW-RANK STRUCTURE OF THE TASK ADAPTATION

  • The per-task results differ: larger-data tasks gain slightly with more rank, while small, high-variance tasks regress.CoLA changes from 72.66 to 70.16 and MRPC from 90.44 to 88.97 as rank increases.Source: §D.3 · p. 18
  • 89.16, 88.95, and 88.82 are the GLUE means at effective per-task ranks 4, 8, and 16, respectively, varying by only 0.34 points.The smallest budget is strongest; the authors attribute this to shared directions captured by the Joint and noisy near-zero-singular-value components at higher ranks.Source: Figure 3 · p. 18, §D.3 · p. 18

D.4.1 HYPOTHESES

  • The interpretability analysis tests whether Individuals are diverse and near-orthogonal to the Joint, task signatures have structured similarity, and the rank-budget split affects performance.These hypotheses correspond to separation, task geometry, and allocation between shared and task-specific structures.Source: §D.4 · pp. 18–19, §D.4.1 · p. 19

D.4.2 EXPERIMENTS

Experiments examine learned subspace geometry, held-out-task routing, and how splitting rank between shared and task-specific structures affects performance. At fixed effective rank 8, the balanced split performs best among the tested splits.

E1: Learned subspace geometry (tests H1)

  • Figure 4 places sampled Joint and task-Individual vectors in a t-SNE plane, with vertical position encoding energy in the Joint subspace.The companion matrix reports pairwise overlaps between per-task Individual bases and the Joint; z measures the fraction of a unit vector’s squared length in the Joint subspace.Source: Figure 4 · p. 19, §D.4.2 · p. 19

E2: Task signature similarity and routing (tests H2)

  • Held-out tasks route to their highest-principal-angle-similarity held-in neighbor; BoolQ and MultiRC are mutually highest-similarity neighbors.CB routes to BoolQ, RTE to MultiRC, WiC to WSC, WSC to MultiRC, and COPA to WiC.Source: Figure 5 · p. 20, §D.4.2 · p. 20

E3: Rank-budget split between structures (tests H3)

  • At effective rank 8, the even 4/4 split scores 82.26, above 2/6 at 80.92 and 6/2 at 82.14 on SuperGLUE.The 4/4 split slightly exceeds the adaptive allocator’s 82.06, while avoiding an explicit search over the sharing ratio.Source: Figure 6 · p. 20, §D.4.2 · p. 20
Loading 2610.07036v1…