PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Oriented Rank Adaptation for Continual Learning in Text Classification

Rey Sanchez Lopez, Eduardo Morales Manzanares, Hugo Jair Escalante

arXiv:2610.01702v1cs.CL

TL;DR

Continual learning in text classification must adapt to sequential tasks while addressing forgetting and negative transfer. TORA routes tasks using geometric affinity between LoRA adapters, transferring from compatible experts or shielding structurally distant tasks. Across 15 evaluated tasks, it reports no harmful routing decisions, while noting that fixed neighborhood size and a static divergence threshold may limit scaling to longer sequences.

  • Problem

    Continual learning must adapt to new tasks while preserving performance on earlier tasks, but sequential training risks catastrophic forgetting and negative transfer.

    Source: §1 · p. 1

  • Method

    TORA uses task fingerprints derived from LoRA adapter geometry in a shared SVD space to route new tasks toward compatible experts or isolate them.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across 15 evaluated tasks, TORA made no routing decision that hurt performance: geometrically close tasks benefited from transfer, while structurally distant tasks were protected by shielding.

    Source: §6 · pp. 11–12

  • Takeaways & Limitations

    The findings suggest angular distance in SVD parameter space is a reliable indicator of task compatibility across diverse text-classification domains, without task identities or manually designed sequences.

    Source: §6 · p. 12

    TORA uses a fixed neighborhood size k = 3 and a static divergence threshold, which may limit scalability as expert pools grow in longer task sequences.

    Source: §6.3 · p. 13

Abstract

from arXiv · show

Continual learning (CL) in text classification faces two critical challenges: catastrophic forgetting and negative transfer across sequential tasks. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA enable efficient adaptation by learning low-rank updates of the model parameters. However, these compact representations are normally trained in isolation, limiting their reuse across related tasks. We introduce Task-Oriented Rank Adaptation (TORA), a geometric routing framework that leverages the low-rank structure of LoRA adapters to decide whether to transfer knowledge from the most compatible expert (Boosting) or isolate the new task (Shielding) based on structural similarity. Evaluated across 15 diverse text classification benchmarks, TORA consistently avoids harmful routing decisions: compatible tasks exceed their isolated performance while reducing training time, and structurally distant tasks are protected from interference with no loss in accuracy. With a single geometric threshold and no reliance on task identities or predefined sequences, TORA provides a simple and effective approach for dynamic adapter routing in sequential text classification systems.

1 Introduction

TORA addresses forgetting and negative transfer in sequential text classification by routing tasks to compatible LoRA experts or isolating them. The introduction presents geometric fingerprints as a basis for routing and reports evaluation across 15 benchmarks.

1 Introduction

  • Sequential learning must adapt to new domains while retaining performance on earlier tasks, despite risks of catastrophic forgetting and negative transfer.Standard PEFT modules keep learned representations isolated, preventing positive transfer between tasks with similar distributions and requiring similar concepts to be relearned.Source: §1 · pp. 1–2
  • TORA estimates task affinity from LoRA weights in a shared SVD reference space, transferring from the most compatible expert when affinity is detected and otherwise shielding the new task.The framework uses cosine similarity between the incoming task and stored experts.Source: §1 · p. 2
  • TORA is evaluated on 15 text-classification benchmarks, with reported advantages over static baselines in accuracy and parameter efficiency and positive transfer and computational savings over isolated training.The introduction also reports improved routing reliability in Monte Carlo simulations.Source: §1 · p. 2
  • A compact geometric fingerprint represents each LoRA adapter as a low-dimensional structural signature for efficient similarity estimation without costly manifold comparisons.The stated goal is to make dynamic routing practical for continual learning.Source: §1 · p. 2

2 Related Work

Related work covers methods that mitigate forgetting or improve parameter distribution, but notes that existing approaches do not provide a lightweight structural compatibility criterion for routing sequential tasks.

2 Related Work

  • Regularization and rehearsal can address continual-learning challenges, but impose memory or computational costs; sequential learning may also improve few-shot adaptation through accumulated knowledge.The passage cites regularization, storing examples [3], generating examples, and few-shot adaptation.Source: §2 · pp. 2–3
  • LoRA freezes the base model and learns low-rank updates efficiently, but does not specify how knowledge should be shared or separated between tasks.In standard PEFT modules, task representations remain stored in separate adapters.Source: §2 · p. 3
  • Orthogonal adaptation and multi-task dynamic routing redistribute parameters, but may rely on trainable routing or costly similarity estimation and lack a simple pre-initialization compatibility criterion.The passage frames this as a gap for lightweight routing based on adapter geometry.Source: §2 · p. 3
  • TORA represents tasks as compact fingerprints in a shared SVD space to estimate structural similarity without semantic representations, trainable routing modules, or costly manifold operations.It dynamically transfers knowledge to related domains and isolates incompatible parameters.Source: §2 · p. 3

3 The TORA Framework

TORA evaluates geometric compatibility between sequential tasks before initialization, using stored LoRA adapters as experts to decide whether to transfer their weights or train the new task from scratch. It builds a shared SVD reference space, represents tasks with compact fingerprints, and routes them by structural affinity.

3 The TORA Framework

  • TORA treats each stored task-specific LoRA adapter as an expert and evaluates compatibility before initializing the incoming task.Compatible tasks can start from the closest expert; otherwise, TORA shields the new task by training from scratch.Source: §3 · p. 3
  • LoRA represents its update as the low-rank product ΔW = BA, with A mapping inputs to a lower-dimensional space and B projecting them back.The rank r is much smaller than the dimensions of the frozen pretrained weight matrix.Source: §3 · p. 4

3.1 Reference Space Construction

  • The reference space is built from the N−1 previously learned adapters, whose flattened weights are stacked and analyzed with SVD.TORA retains the minimum number of components explaining 60% of total variance.Source: §3.1 · pp. 4–5

3.2 Task Fingerprint Generation

  • Each task fingerprint concatenates projections of its LoRA weights onto the selected reference-space directions.For DistilBERT, the 36 A and B matrices projected onto four components yield a 144-dimensional signature.Source: §3.2 · pp. 5–6

3.3 Routing Mechanism with Shielding and Boosting

  • After one epoch, TORA compares the incoming task’s fingerprint with stored experts and transfers the Top-1 expert’s weights when mean Top-3 angular distance is below 84.5°; otherwise, it trains from scratch.The routing decision costs approximately 10–12 seconds; after full training, the resulting adapter joins the expert memory.Source: §3.3 · pp. 6–7

4 Experimental Setup

The experiments evaluate TORA on 15 text-classification tasks and use a DistilBERT-based LoRA setup to compare scratch initialization with similarity-based transfer.

4.1 Datasets and Benchmarks

  • The setup covers 15 text-classification tasks drawn from Yelp Polarity, Amazon Polarity, Yahoo Answers, AG News, DBPedia, GLUE, and SuperGLUE.Source: §4.1 · p. 7, §4 · p. 7

4.2 Implementation Details

  • Experiments use DistilBERT with LoRA modules in Transformer self-attention layers and compare randomly initialized Scratch adapters with TORA-routed Transfer adapters.Source: §4.2 · pp. 7–8
  • LoRA uses rank r = 64; across 18 injected projections, each task has 1.77M trainable parameters versus 10.62M for full fine-tuning.Source: §4.2 · p. 8
  • The configuration sets α = 16, dropout = 0.1, and bias = none, keeping task-specific information in the low-rank adapters for SVD analysis.Source: §4.2 · p. 8

5 Evaluation Methodology and Experimental Results

TORA routes tasks by geometric affinity, transferring from compatible experts and shielding dissimilar tasks. Across the reported evaluations, it supports positive transfer, avoids harmful interference, and performs robustly across task orders.

5 Evaluation Methodology and Experimental Results

  • Evaluation tracks epoch-wise accuracy against a continuous single-LoRA baseline and uses 100,000 random task sequences to assess routing under stochastic arrivals.Source: §5 · p. 9

5.1 Experimental results

  • The Top-3 average angle provides the most stable routing boundary: transfer-benefiting tasks fall below 84.5°, while isolated tasks exceed it.Source: Table 3 · p. 9, §5.1 · p. 9

5.2 Interference Prevention (Shielding)

  • For MNLI, forcing transfer reduces accuracy to 0.628, whereas Shielding preserves baseline performance at 0.652 with 10-epoch training.Source: §5.2 · pp. 9–10, Table 4 · p. 10

5.3 Positive Transfer (Boosting)

  • With geometric affinity below 84.5°, TORA can surpass Scratch; COPA gains +11.11% and uses 10 fewer training epochs when initialized from DBPedia.Source: Table 5 · p. 10, §5.3 · p. 10

5.4 Temporal Dynamics and Performance

  • For hubs such as IMDB, MultiRC, and QQP, nearest-expert initialization yields higher early accuracy and faster convergence; Shielding protects SST-2, MNLI, and CB from negative transfer.Source: §5.4 · p. 10

5.5 Monte Carlo simulations

  • Across 100,000 random sequences, SST-2 and MNLI trigger Shielding in 99.5% of cases, while COPA transfers successfully in 80.8%.Source: §5.5 · p. 11, Table 6 · p. 11

5.6 Continual Learning Results and Dynamic Routing Efficacy

  • AT = 75.56% at best and 75.40% on average across five sequences; TORA outperforms O-LoRA in every tested order with 0.33% versus 7.5% variance.Source: §5.6 · p. 11

6 Discussion

Across 15 evaluated tasks, TORA avoided harmful routing: geometrically close tasks benefited from transfer, while distant tasks were shielded without task identities or manually designed sequences. The discussion connects this pattern to an 84.5° divergence threshold, distinct transfer and isolation roles, and limits to scaling a fixed neighborhood.

6 Discussion

  • Across 15 evaluated tasks, TORA avoided harmful routing: compatible tasks benefited from transfer, while structurally distant tasks were protected by Shielding without task identities or manually designed sequences.The authors describe angular distance in SVD parameter space as a reliable indicator of task compatibility across diverse text-classification domains.Source: §6 · pp. 11–12

6.1 Geometric Phase Transition and the Divergence Threshold

  • At the 84.5° divergence threshold, below-threshold gradient alignment supports reuse; above it, forced transfer perturbs optimization and can degrade performance.The threshold is described as the practical boundary where shared variance becomes too small to support positive transfer.Source: §6.1 · pp. 12–13

6.2 Structural Dynamics: Hubs vs. Isolated Nodes

  • DBPedia acts as a transfer hub, whereas MNLI, CB, and SST-2 develop incompatible structures; forced reuse causes geometric interference, motivating Shielding.Source: §6.2 · p. 13

6.3 Computational Efficiency and Limitations

  • Geometric initialization skips early gradient-descent exploration and speeds convergence, but fixed k = 3 and a static threshold may limit longer sequences as experts grow.A static neighborhood risks capturing overlapping or contradictory gradient directions as the expert pool scales.Source: §6.3 · p. 13

7 Conclusion

Across 15 diverse benchmarks, TORA avoided harmful routing decisions, with compatible tasks benefiting from transfer and structurally distant tasks protected by Shielding. It routes using angular distance between task fingerprints in SVD parameter space, with one threshold and no reliance on task identities or predefined sequences.

  • Across 15 diverse benchmarks, TORA never induced harmful routing: compatible tasks exceeded isolated baselines while saving up to 10 training epochs, and distant tasks were protected without accuracy loss.Source: §7 · p. 13
  • TORA decides whether to transfer or isolate knowledge based solely on angular distance between task fingerprints in SVD parameter space.Source: §7 · p. 13
  • A single 84.5° threshold and no task identities or predefined sequences support autonomous knowledge routing in sequential text classification.Source: §7 · p. 13
Loading 2610.01702v1…