PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

Hongwei Zhao, Rui Liu, Yansong Liu, Zhiyuan Zou, Yong Chen

arXiv:2609.39390v1cs.LG

TL;DR

FSCIL requires learning emerging classes while retaining prior knowledge, but existing methods face feature-learning and inference-efficiency limitations. TALON assigns task-specific LoRA-Teachers and distills them into a unified student. Across four benchmarks, it reports comparable or better mean average accuracy.

  • Problem

    Continual class learning risks forgetting previously learned representations, while existing FSCIL methods face feature-learning and inference-efficiency limitations.

    Source: §1 · pp. 2–3

  • Method

    TALON trains an independent LoRA-Teacher per incremental task and distills the frozen teachers into one student, weighting contributions by feature-space similarity.

    Source: §1 · pp. 4–5

  • Results

    Across three class-order runs, TALON achieves comparable or better mean average accuracy across four benchmark datasets.

    Source: §6 · p. 37

  • Takeaways & Limitations

    TALON consolidates task-specific teacher knowledge into a unified student while balancing stability and plasticity for efficient inference.

    Source: §6 · p. 37

    The prototype protocol may introduce prototype–feature inconsistency across incremental stages.

    Source: §24 · p. 17

Abstract

from arXiv · show

Few-Shot Class-Incremental Learning (FSCIL) addresses the challenge of learning new classes from very limited samples while retaining knowledge of previously learned ones. Although parameter-efficient fine-tuning methods with pre-trained models show promise for class-incremental learning, strict gradient-based constraints can be unreliable under severe data scarcity, while multi-expert approaches can impose substantial inference-time costs. We propose TALON (Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer), an inference-efficient FSCIL framework. TALON dynamically allocates an independent LoRA-Teacher to each incremental task for task-specific representation learning, then distills multiple frozen teachers into a unified LoRA-Student through Ensemble Knowledge Transfer, eliminating runtime module selection or generation. A semantic-guided distillation strategy weights teacher contributions by feature-space similarity to mitigate catastrophic forgetting and overfitting. Across three class-order runs, TALON achieves comparable or better mean average accuracy across four FSCIL benchmarks, obtaining 86.68 +/- 1.22% on CUB200, 90.39 +/- 0.27% on CIFAR100, 78.38 +/- 0.94% on ImageNet-R, and 96.34 +/- 0.33% on miniImageNet. TALON uses up to 33x fewer deployment parameters and reduces average inference time per task to 26.7 s, a 41.70% reduction relative to ASP.

1. Introduction

FSCIL must learn emerging classes from few examples while retaining earlier knowledge, yet existing PEFT constraints can be unreliable and multi-expert approaches can be costly at inference. TALON uses task-specific LoRA-Teachers and distills them into one student; the introduction reports strong performance and efficiency.

1. Introduction

  • FSCIL learns novel classes from severely limited examples while retaining old classes, intensifying forgetting, overfitting, and the plasticity–stability dilemma.The introduction notes that real-world applications can encounter emerging categories with only a handful of labeled examples.Source: §1 · p. 2
  • Under extreme data scarcity, gradient-based PEFT constraints may be unreliable, while multi-expert methods can add costly module selection or per-input module generation.Query-based selection may become a scalability bottleneck as tasks accumulate.Source: §1 · p. 3
  • TALON assigns an independently trained LoRA-Teacher to each incremental task, then uses Ensemble Knowledge Transfer to distill frozen teachers into one LoRA-Student.Only the current task’s teacher is trained; EKT removes runtime module selection or generation.Source: §1 · pp. 4–5
  • Semantic-guided distillation weights teachers by feature-space similarity to integrate knowledge while mitigating forgetting and overfitting.The introduction describes this strategy as adaptively weighting teacher contributions.Source: §1 · p. 5
  • TALON is reported to achieve state-of-the-art performance with superior efficiency, and Figure 1 compares accuracy and inference time against multi-expert and LoRA methods.The figure’s inference-time comparison is averaged per task on CIFAR100.Source: Figure 1 · p. 4, §1 · p. 5

2. Related Work

Related work covers knowledge-distillation paradigms for continual learning, PEFT methods for continual learning, and prior FSCIL approaches. TALON combines task-specific LoRA adaptation with teacher-to-student consolidation.

2.1. KD-based CIL

  • KD-based CIL spans regularization, data replay, and feature replay, alongside exemplar-free logit/feature distillation with prototype calibration.Examples include LwF, iCaRL, and PASS.Source: §2.1 · pp. 5–6

2.2. PEFT-based CIL

  • PEFT-based CIL includes prompt pools and task-specific prompts, plus LoRA methods addressing interference or gradient direction and magnitude.SD-LoRA preserves early-task directions while adapting to new ones, at the expense of reduced plasticity.Source: §2.2 · p. 6

2.3. PEFT-based FSCIL

  • FSCIL work has moved from shallow, fully fine-tuned networks to PTM-based prompt methods and vision–language prompting.Other continual-learning directions include prompting, analytic learning, slow–fast collaboration, prototype-enhanced composition, expert selection, and domain-specific learning.Source: §2.3 · pp. 6–7

2.4. Our Approach

  • TALON assigns a LoRA-Teacher to each incremental task, distills all frozen teachers into one student, and weights contributions by feature-space similarity.The design targets coherent knowledge integration while eliminating inference-time prompt generation.Source: §2.4 · p. 7

3. Preliminaries

The preliminaries define FSCIL’s task sequence and evaluation setting, introduce low-rank adaptation, and describe knowledge distillation as a continual-learning regularizer.

3.1. Problem Formulation

  • FSCIL tasks introduce disjoint label sets; the base task has ample data, while later tasks contain limited labeled examples in an N-way K-shot setting.Following rehearsal-free learning, training uses only the current task’s data and evaluates on all classes seen so far.Source: §3.1 · pp. 7–8

3.2. Low-Rank Adaptation

  • LoRA fine-tunes a pretrained weight matrix with a low-rank update formed from matrices U and V, where rank r is much smaller than either matrix dimension.This reduces trainable parameters and fine-tuning costs.Source: §3.2 · p. 8

3.3. Knowledge Distillation in Incremental Learning

  • In incremental learning, knowledge distillation transfers representations from a prior-task teacher to the current student to preserve previously learned knowledge.The stated formulation uses feature extractors for the old and current task models and a distillation loss.Source: §3.3 · pp. 8–9

4. The Proposed Method

TALON trains task-isolated LoRA-Teachers, then transfers their collective knowledge into one LoRA-Student for inference without runtime module selection or generation.

4.1. Overall Framework

  • TALON distills multiple frozen LoRA-Teachers into one LoRA-Student, avoiding runtime module selection or generation; teacher-side computation and state grow with tasks during training but are discarded before deployment.Stage 2 sequentially evaluates accumulated teachers, incurring O(t) teacher-side forward computation and a linearly growing retained teacher parameter state.Source: §4.1 · p. 11

4.2. Task-Adaptive LoRA-Teacher

  • Unlike gradient-based strategies that require sufficient data for meaningful gradients, TALON adds an independent LoRA-Teacher for each task while freezing earlier teachers.Teachers can attach to the MLP block or to attention projections; the joint query-and-value configuration is called QV-Teacher.Source: §4.1 · p. 9, §4.2 · pp. 11–12

4.3. Ensemble Knowledge Transfer

  • EKT transfers ensemble knowledge through complementary soft-logit and hard-label distillation, optimizing the Student while the accumulated Teachers and shared linear head remain frozen.The soft-logit term uses teacher logits, while hard-label distillation supplies ground-truth supervision.Source: §4.3 · pp. 13–14

4.4. Semantic-Guided Distillation Coefficient

  • Semantic-guided coefficients weight each teacher by task relevance, computed from current-task and teacher-class prototypes in that teacher’s feature space.The coefficients are computed once per incremental stage and reused during EKT; they add no learnable parameters or optimizer state.Source: §4.4 · pp. 14–15

4.5. Optimization Objective and Training Procedure

  • Training proceeds in two stages: train a task-specific Teacher with the backbone frozen, then optimize a Student to consolidate the accumulated Teachers’ knowledge.During Teacher training, current-class head columns provide task-local supervision; during EKT, the complete shared head is frozen.Source: §4.5 · p. 15

24 end

After EKT, the Student serves as the deployed feature extractor, while a class-indexed prototype bank supports prediction without retaining historical samples.

24 end

  • Historical class prototypes stay fixed as new classes are added, while each test sample is represented using the current Student.TALON does not retain historical samples under its exemplar-free protocol.Source: §24 · p. 17
  • Prediction uses cosine similarity between Student features and class prototypes, but changing Student features can create prototype–feature inconsistency across sessions.The paper quantifies this effect in Section 5.7.Source: §24 · p. 17
  • For inference, TALON discards the Teachers and their training-time prototype banks, retaining only the current Student and accumulated Student prototypes.New-class prototypes are computed with the current Student and appended to the bank.Source: §24 · p. 17

5. Experiments

Experiments compare TALON with PEFT-based CIL and FSCIL methods across four benchmarks under shared backbones and splits. TALON matches or exceeds competitors on mean accuracy and shows favorable forgetting and forward-transfer results.

5.1. Implementation Details

  • Evaluation covers CUB200 and ImageNet-R with 200 classes each, and CIFAR100 and miniImageNet with 100 classes each, using the split configuration of prior work.Comparisons include PEFT-based CIL and FSCIL methods, SimpleFSCIL, and full fine-tuning, all using ViT-B/16-IN1K and identical dataset splits.Source: §5.1 · p. 18, Table 1 · p. 18
  • The study reports base-task accuracy, final-task accuracy, average accuracy, performance drop (PD), and forward transfer (FWT).PD is the base-to-final accuracy decline in percentage points; FWT measures zero-shot performance on future tasks relative to a random-model reference.Source: §5.1 · pp. 18–19

5.2. Benchmark Comparison

  • TALON achieves comparable or better mean last-session and average accuracy across four benchmarks, exceeding SEC-prompt in average accuracy on three.On CUB200, TALON-MLP and SEC-prompt are comparable at 86.68 ± 1.22% and 86.49 ± 2.43%, with rankings varying by class order.Source: §5.2 · pp. 19–20, Table 2 · p. 20
  • TALON maintains relatively low performance drop (PD); on CUB200, TALON-MLP's 4.00% is below InfLoRA's 45.66% and SD-LoRA's 23.13%.Across CIFAR100, ImageNet-R, and miniImageNet, TALON retains knowledge competitively, while SEC-prompt has lower or equal PD.Source: §5.2 · pp. 20–21, Table 3 · p. 20
  • TALON shows positive forward transfer (FWT) across datasets; TALON-QV records 47.80% on CIFAR100 versus SEC-prompt's 30.80%.TALON-QV also records 16.67% versus 9.47% on CUB200, and 16.52% versus 8.52% on miniImageNet.Source: Table 3 · p. 20, §5.2 · p. 21
  • TALON's performance curves show consistently high accuracy throughout incremental learning, and supplementary analysis reports consistent advantages in longer sequences and with a larger backbone.The main evaluation uses four benchmarks; its reported performance trajectories use the shared ViT-B/16-IN1K backbone.Source: §5.2 · pp. 21–22, Figure 3 · p. 22

5.3. Efficiency Analysis

The efficiency analysis distinguishes training costs from deployment costs. TALON consolidates teacher knowledge into one student, attaining low inference latency and parameter counts while retaining additional teachers and incurring distillation costs during training.

Inference Efficiency

  • 26.7 s per task is TALON-QV's inference time, 41.7% below ASP's 45.8 s and 46.3% below SEC-prompt's 49.7 s.The figure measures wall-clock evaluation on all classes seen at task t; TALON has the lowest latency across evaluated tasks.Source: Figure 4 · p. 23, §5.3 · p. 23

Expanded Parameters Analysis

  • TALON-MLP achieves 90.26% accuracy on CIFAR100 with 0.15M parameters, versus SEC-prompt's 89.22% with 2.50M parameters.On CUB200, TALON-MLP's 85.55% exceeds SEC-prompt's 83.84% with approximately 33× fewer parameters (0.15M versus 4.99M).Source: §5.3 · p. 24, Figure 5 · p. 24
  • Deployment discards accumulated Teachers, Teacher prototype banks, and the auxiliary classifier, retaining only the LoRA-Student and its prototype classifier.Teacher count affects training computation and retained training state, not the deployment state.Source: §5.3 · pp. 23–25

Training Efficiency

  • TALON-MLP uses approximately 4.3 GB peak GPU memory, 7.9%–32.5% less than SEC-prompt and 16.1%–55.1% less than ASP.Its 24–29 s training time per epoch exceeds SEC-prompt's, but is lower than ASP's on three benchmarks; EKT uses five epochs versus 10–20 for Teacher training.Source: §5.3 · p. 27, Table 6 · p. 27

Long-Sequence Training Scalability

  • At T = 21, TALON-QV's average accuracy changes by at most 0.29 points, while deployment state and peak GPU memory remain unchanged.Cumulative EKT time grows 1.99×–2.52×, end-to-end time rises 4.3%–18.4%, and retained training state grows 1.74×–2.11×; TALON-QV remains 0.57–1.41 points above SEC-prompt.Source: §5.3 · pp. 25, 26, 28, Table 7 · p. 28

Semantic-Score Computation

  • Semantic-score computation takes 11.36–29.81 s cumulatively across datasets, equivalent to 0.31%–4.08% of task-specific Teacher-training time.This training-only computation uses no learnable parameters or backpropagation and does not change inference complexity.Source: §5.3 · p. 28

5.4. Ablation Studies

Ablations examine TALON's teacher and distillation components, semantic-weighting rules, consolidation alternatives, and robustness across pre-training paradigms. The reported comparisons favor task-specific teachers, semantic-guided output distillation, and summed-cosine weighting.

Different Components

  • Removing semantic similarity drops TALON-MLP's CIFAR100 average accuracy from 90.26% to 88.52%, while removing Teacher knowledge distillation also yields inferior performance.The authors attribute the semantic ablation's decline to the Student's inability to prioritize relevant teachers amid conflicting supervision.Source: §5.4 · p. 29, Table 8 · p. 29

Comparison of Teacher-Weighting Strategies

  • Summed pairwise cosine achieves 87.35% four-dataset macro-average accuracy; alternative weighting rules lower it by 0.54–1.10 points.The evaluated learnable task-level weighting scores 86.30%, 1.06 points below summed cosine, and does not improve average accuracy on any benchmark.Source: §5.4 · pp. 29–30, Table 9 · p. 30

Knowledge Consolidation Strategies

  • Teacher-ensemble inference yields last-task accuracy of 64.90% on CIFAR100 and 50.43% on ImageNet-R, and trails TALON by 12.45% and 13.15% in average accuracy.Without consolidation, inference cost scales linearly with task count.Source: §5.4 · pp. 30–31, Table 10 · p. 31
  • Averaging LoRA weights produces last-task accuracy of 6.91% on CUB200 and 13.53% on ImageNet-R; average accuracy also trails TALON by 2.32 and 0.41 points on CIFAR100 and miniImageNet.The authors report that parameter averaging destroys the nonlinear representations learned by individual teachers.Source: §5.4 · p. 31, Table 10 · p. 31
  • Learnable routers underperform TALON by 3.93–17.96 points in average accuracy, with five samples per class reported insufficient for reliable gating.The paper supports output-level distillation over direct parameter merging as a way to consolidate teacher knowledge into one Student.Source: §5.4 · p. 31, Table 10 · p. 31

Robustness across Pre-training Paradigms

  • TALON-QV reaches 90.24% average accuracy with supervised ViT-B/16-IN21K and 80.76% with self-supervised DINO, outperforming ASP and SEC-prompt in both settings.The authors report prompt-based methods as more sensitive to pre-training shifts, while TALON maintains robust performance across feature spaces.Source: Table 11 · p. 32, §5.4 · p. 32

5.5. Visualization of Incremental Sessions

The miniImageNet visualization compares feature separation across two incremental stages and shows retained separation of old and new classes.

5.5. Visualization of Incremental Sessions

  • On miniImageNet, t-SNE visualizations show clear separation for old classes and new classes across the first two incremental stages.Dots denote old classes, triangles denote new classes, and shaded regions show decision boundaries.Source: §5.5 · p. 33

5.6. Further Analysis

Further analyses find Logit-KL the most balanced EKT objective, stable performance across tested LoRA settings, and little decision-level effect from historical-prototype drift. The cosine nearest-prototype classifier remains robust, so the authors retain frozen prototypes under the exemplar-free protocol.

5.6. Further Analysis

  • On CUB200 (T=11), TALON showed stable performance across ranks r ∈ {1, 2, 4, 8, 10, 16} and insertion ranges 0-2 through 0-12.The authors use r = 8 and insertion into all Transformer layers (0-12) by default, and report similar trends on other datasets.Source: §5.6 · pp. 33–34, Figure 7 · p. 34
  • Logit-KL achieved the highest combined four-dataset macro-average, 87.37%, and no alternative consistently improved across datasets and variants.The supplementary comparison covered Logit-KL, Logit-L2, Feature-L2, Feature-CS, and feature-space-consistent Prototype Relation for TALON-MLP and TALON-QV.Source: §5.6 · p. 34

5.7. Prototype–Feature Consistency Across Incremental Stages

  • Across 24 runs, SDC changed final accuracy by −0.042 to +0.078 points and historical-data recomputing by −0.044 to +0.020 points; neither had a significant final-session McNemar result.The comparison held the Student checkpoint, test features, class order, and cosine nearest-prototype classifier fixed, changing only historical prototype entries.Source: §5.7 · p. 36
  • TALON remained prediction-robust despite prototype drift, so the authors retain Frozen to preserve the strict exemplar-free protocol without sacrificing meaningful classification performance.SDC tracked and reduced measured displacement, with ρ∆ ranging from 0.627 to 0.874 and Rcent from 3.5% to 25.9%.Source: §5.7 · p. 36

5.8. Historical-Knowledge Preservation During EKT

Historical-task performance remains essentially unchanged on average after Logit-KL EKT updates.

5.8. Historical-Knowledge Preservation During EKT

  • Mean historical-task changes after Logit-KL EKT were +0.05, −0.00, and −0.12 percentage points on CUB200, CIFAR100, and ImageNet-R, respectively.The measurements compare historical-task accuracy immediately before and after each EKT update.Source: §5.8 · p. 37

6. Conclusion

TALON combines task-adaptive LoRA-Teachers with ensemble knowledge transfer to address FSCIL’s challenges. Across three class-order runs, it achieves comparable or better mean average accuracy across four benchmarks while retaining efficient single-model inference.

6. Conclusion

  • TALON allocates a dedicated, parameter-isolated LoRA-Teacher to each incremental task, avoiding a predefined module count while targeting greater plasticity.The framework consolidates the teachers’ collective knowledge into one LoRA-Student for efficient inference.Source: §6 · p. 37
  • Ensemble Knowledge Transfer distills multiple teachers into one student, removing computational overhead during inference.The unified model eliminates dynamic module selection or runtime module generation.Source: §6 · p. 37
  • Semantic-guided distillation integrates teacher knowledge while mitigating catastrophic forgetting and overfitting.Source: §6 · p. 37
  • Across three class-order runs, TALON achieved comparable or better mean average accuracy across four benchmarks, with performance comparable to SEC-prompt on class-order-sensitive CUB200.Ablations also showed limited sensitivity to semantic temperature, soft-distillation weighting, and EKT duration.Source: §6 · pp. 37–38
Loading 2609.39390v1…