PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning

Chiyuan He, Zihuan Qiu, Fanman Meng, Chao Wang, Liangjiang Chen, Linfeng Xu, Qingbo Wu, Hongliang Li

arXiv:2609.36759v1cs.CVcs.AI

TL;DR

CLIP-based class-incremental learning must preserve old knowledge while learning new classes, but shared updates can overwrite knowledge and strict isolation can limit the reuse of transferable directions. DuLBE combines gradient-routed dual-mode low-rank learning with a reliability-guided bridge-prototype ensemble. The paper reports state-of-the-art CIL performance while preserving low-rank parameter efficiency.

  • Problem

    Existing CLIP continual learners either repeatedly update shared components, risking knowledge overwriting, or avoid occupied directions that contain transferable gradient demand.

    Source: Introduction · pp. 1–2

  • Method

    DuLBE allocates a rewritable shared visual mode and low-interference residual modes from gradient demand, then uses reliable geodesic bridges between visual prototypes and text embeddings for classification.

    Source: Introduction · p. 2

  • Results

    DuLBE achieves state-of-the-art CIL performance while preserving the parameter efficiency of low-rank tuning.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    DuLBE pairs controlled reuse of transferable cross-modal knowledge with flexible task-specific adaptation and low-rank parameter efficiency.

    Source: Conclusion · p. 7

    The bridge-prototype ensemble is not a universally effective plug-in; its benefit depends on a compatible, stabilized inter-modal structure.

    Source: Further Analysis · p. 7

Abstract

from arXiv · show

Benefiting from transferable visual-textual alignment, CLIP has been widely adopted for class-incremental learning (CIL). However, existing learners either repeatedly update components shared across tasks, leading to knowledge overwriting, or overly isolate new-task updates, hindering the reuse of CLIP's transferable knowledge and limiting plasticity. Moreover, the text-based or bimodal classifier designs still fail to effectively integrate complementary information from the visual and textual modalities. To address these challenges, we introduce DuLBE, which couples dual-mode low-rank learning with a bridge-prototype ensemble classifier for exemplar-free CIL. DuLBE allocates two visual low-rank update modes according to the gradient demand and uses gradient routing to coordinate them: a compact and rewritable shared mode is selected from historically occupied visual directions to reuse transferable knowledge, while residual modes provide low-interference channels for task-specific variations. Building on the resulting stable inter-modal structure, we further construct geodesic bridges between visual prototypes and text embeddings on the unit hypersphere, and ensemble reliable bridge prototypes to compensate for the modality-gap limitations of textual decision boundaries. Extensive experiments under multiple settings show that DuLBE achieves state-of-the-art CIL performance while retaining the high parameter efficiency of low-rank tuning.

Introduction

CLIP-based class-incremental learning must preserve old-class knowledge while adding new classes, yet existing adaptation strategies risk overwriting shared knowledge or limiting reuse of transferable directions. DuLBE addresses this tension with gradient-demand dual-mode low-rank learning and a bridge-prototype ensemble, and reports state-of-the-art performance with modest trainable overhead.

Introduction

  • New-task cross-modal gradients concentrate in historically occupied directions, but tuning those directions causes greater old-class relation drift than tuning low-interference directions.Figure 1 reports substantial gradient energy in occupied directions and greater drift when those directions are tuned.Source: Figure 1 · p. 1, Introduction · p. 2
  • Existing CLIP continual learners repeatedly optimize shared components, risking progressive overwriting when old-task data are unavailable.Other approaches isolate updates, but avoiding occupied directions can discard transferable write-in signals and restrict alignment plasticity.Source: Introduction · pp. 1–2
  • DuLBE selects a compact, rewritable shared mode from historically occupied directions and low-interference residual modes for task-specific variations, coordinating them through gradient routing.Its classifier constructs geodesic bridges between visual prototypes and text embeddings and uses bridge reliability to refine the decision boundary.Source: Introduction · p. 2
  • DuLBE achieves state-of-the-art performance with modest trainable overhead.Its contributions pair dual-mode low-rank adaptation with a bridge-prototype ensemble classifier.Source: Introduction · p. 2

Related Work

CIL methods balance learning new classes with retaining prior knowledge, often relying on stored data, old-task supervision, or added capacity. CLIP-based and LoRA-based approaches protect semantic structure or constrain updates, while DuLBE uses cross-modal gradient demand to allocate update directions.

Class-Incremental Learning

  • Traditional CIL methods mitigate forgetting with exemplars, distillation, or task-specific parameters, but often require stored data, old-task supervision, or expanding capacity.CLIP-based CIL methods exploit textual priors, preserve cross-modal structure, or isolate updates, yet may keep updating shared modules or avoid risky directions.Source: Class-Incremental Learning · p. 2

LoRA-Based Continual Learning

  • LoRA-based continual learning controls low-rank updates through orthogonalization, interference-free subspaces, energy-driven decomposition, or predefined functional insertion.DuLBE instead uses cross-modal gradient demand and mode occupation to determine rewritable shared directions and residual directions.Source: LoRA-Based Continual Learning · p. 2

Methodology

DuLBE addresses exemplar-free CLIP class-incremental learning by allocating visual low-rank modes from current-task gradient demand, routing training objectives to those modes, and classifying with reliability-guided visual-text geodesic bridges.

Preliminaries

  • DuLBE is evaluated in exemplar-free CIL with disjoint task classes, all seen classes evaluated without task identities, and no raw data retained from previous tasks.CLIP compares normalized visual features with class-text embeddings, so visual adaptation must remain compatible with textual decisions and prior cross-modal relations.Source: Preliminaries · pp. 2–3

Overview

  • DuLBE allocates visual low-rank modes from prospective gradients, routes current-task cross-modal loss to both learners, and sends semantic-structure preservation gradients only to the shared learner.Reliable visual-text geodesic bridge prototypes are ensembled for classification.Source: Overview · p. 3, Figure 2 · p. 3

Gradient-Demand Dual-Mode Subspace Allocation

  • For each layer, DuLBE probes current-task gradients to select the historically occupied direction with strongest gradient demand as a compact shared mode, then uses residual gradient directions outside historical support for low-interference adaptation.The probing gradient plans the update basis rather than directly updating parameters; historical support is estimated from accumulated layer-input statistics.Source: Gradient-Demand Dual-Mode Subspace… · pp. 3–4

Routed Cross-Modal Training

  • Current-task cross-modal training updates both branches, while the old semantic-structure objective regularizes only the shared branch to preserve old visual-text relations.The shared mode learns transferable alignment under old semantic constraints, and residual modes capture task-specific evidence through low-interference directions.Source: Routed Cross-Modal Training · pp. 4–5

Reliability-Guided Bridge-Prototype Ensemble

  • Class-wise geodesic bridges interpolate between visual prototypes and text embeddings on the unit hypersphere, with visual and textual endpoints at depths 0 and 1.The interpolation provides a smooth transition between modalities while keeping bridge prototypes semantically comparable across classes.Source: Reliability-Guided Bridge-Prototype… · p. 5
  • DuLBE estimates each class's bridge-depth reliability from average correct-class confidence and uses normalized reliability weights to ensemble bridge-prototype logits for inference.The ensemble improves CIL classification, but its effectiveness depends on a stably evolving inter-modal semantic structure.Source: Reliability-Guided Bridge-Prototype… · p. 5

Experiments

DuLBE is evaluated under two standard CLIP-based CIL protocols and achieves the strongest overall benchmark performance. Further analyses examine parameter efficiency, mode collaboration, bridge-prototype reliability, and hyperparameter robustness.

Experimental Setup

  • The evaluation follows two standard CLIP-based CIL settings, comparing against zero-shot, prompt-based, and recent CLIP-based CIL methods.Setting-A uses OpenAI CLIP and Setting-B uses LAION400M-pretrained OpenCLIP; each follows its protocol’s task splits and preprocessing.Source: Experimental Setup · p. 5, Comparison with the State-of-the-Arts · p. 5

Comparison with the State-of-the-Arts

  • DuLBE ranks first on all four Setting-A datasets and reported metrics, and achieves the best results on most Setting-B datasets.On CIFAR, it improves average and final accuracy over the best previous results by 2.5% and 2.7%, respectively.Source: Comparison with the State-of-the-Arts · pp. 5–6, Table 1 · p. 6
  • 0.065M and 0.58M trainable visual-tower parameters correspond to shared-only and shared-plus-residual variants, which remain in Figure 3’s upper-left region.The reported comparison favors DuLBE over higher-rank LoRA variants and full fine-tuning counterparts.Source: Figure 3 · p. 6, Comparison with the State-of-the-Arts · p. 6

Further Analysis

  • 3.4/5.7% on CIFAR and 5.1/6.8% on CUB in A/Al are the greatest gains when Lbi-kl routes only to the shared mode.Combining modes alone gives limited gains; regularizing both restricts residual plasticity, whereas shared-only routing preserves flexible residual adaptation.Source: Further Analysis · p. 6
  • Reliability weighting over bridge depths achieves the best ensemble performance on both datasets, while its effect on other learners is inconsistent.Fixed bridge depths outperform the text-only classifier, but uniform multi-depth combinations do not consistently beat the best fixed depth.Source: Further Analysis · pp. 6–7
  • Figure 5 reports τC = 5, τI = 0.1 and (rS, rR) = (1, 8) as best; nearby temperatures are comparable, and larger ranks add no consistent benefit.The analysis characterizes DuLBE as generally robust to KL temperatures and mode-rank allocation.Source: Figure 5 · p. 7, Further Analysis · p. 7

Conclusion

DuLBE combines dual-mode low-rank visual adaptation with a bridge-prototype ensemble for exemplar-free Vision-Language CIL. Its shared and residual modes allocate transferable cross-modal knowledge and task-specific variation to support controlled reuse and flexible adaptation.

Conclusion

  • DuLBE combines a dual-mode low-rank learner with a bridge-prototype ensemble, using compact shared and low-interference residual modes for exemplar-free Vision-Language CIL.The conclusion describes the ensemble as using discriminative prototypes in the inter-modal semantic region with bridge-depth reliability.Source: Conclusion · p. 7

Overview

  • The appendix organizes supporting material into design analysis, experimental protocols and training configurations, additional results, and the complete DuLBE procedure.The provided appendix passages expand on the shared-mode and residual-mode design choices.Source: Overview · p. 10, §A.1 · p. 10, §A.2 · p. 10

A.1 Rewritable Shared Mode

  • The shared mode selects a compact, layer-wise historical direction by current-task gradient demand, balancing added gradient benefit against the cost of exposing more directions to rewriting.DuLBE adopts shared rank rS = 1 because additional directions can add limited benefit while increasing the risk of disturbing old visual-text relations.Source: §A.1 · p. 10

A.2 Residual Modes and Gradient Routing

  • Residual modes retain the largest current-task gradient energy outside the historical support, limiting their response on historical activations while providing task-specific plasticity.This complementary-space allocation lets residual capacity be larger while the historically sensitive shared mode remains compact.Source: §A.2 · pp. 10–11
  • The KL gradient is routed to the shared branch, while the residual branch receives only the current-task objective to avoid residual KL noise during plastic exploration.The residual KL component is attributed to finite-sample variation, imperfect subspace estimation, and nonlinear cross-layer coupling.Source: §A.2 · p. 11

A.3 Analysis of Bridge-Prototype Ensemble

Bridge-depth experiments with frozen CLIP find that intermediate visual-textual representations outperform either modality endpoint, while preferred depths vary across datasets and classes. Geodesic interpolation gives these depths comparable angular meaning across classes.

Dataset and class-level evidence

  • Across all 8 scenarios, the best bridge depth is intermediate (α* = 0.15–0.55) and exceeds the stronger endpoint by 0.91–6.70 percentage points.The experiments vary only bridge depth with frozen CLIP; classes within the same dataset also prefer substantially different depths.Source: §A.3 · p. 11

Analysis of Geodesic Bridge Interpolation

  • Visual prototypes can carry appearance-specific bias while text embeddings can omit fine-grained visual cues, so the most discriminative class direction may lie between the endpoints.The reported interior gains and class-dependent optima support this non-endpoint discriminative region.Source: §A.3 · pp. 11–12
  • Geodesic interpolation keeps bridge prototypes on the unit hypersphere with uniform angular depth, allowing class-specific reliability weights to identify each class’s discriminative inter-modal region.Unlike ordinary linear interpolation, the same α represents comparable semantic progress across classes.Source: §A.3 · p. 12

B.1 Datasets and Protocols

The appendix specifies two CLIP-based CIL benchmark protocols and their dataset configurations, then describes compared methods, training details, and evaluation metrics. DuLBE uses current-task data without old exemplars or external data, and its optimization epochs do not exceed those of the standard protocols.

B.1 Datasets and Protocols

  • Setting-A uses OpenAI CLIP ViT-B/16 on CIFAR-100, CUB-200-2011, ImageNet-R, and a 100-class ImageNet subset; Setting-B uses LAION400M-initialized OpenCLIP ViT-B/16 on CIFAR-100, 300-class SUN-397, 100-class Stanford Cars, and 100-class Food-101.Setting-A uses default public class orders and a released random ImageNet-R split; Setting-B uses a shuffled class order with seed 1993.Source: §B.1 · p. 12

B.2 Introduction to Compared Methods

  • The compared methods span frozen zero-shot classification, prompt and projection adaptation, language-guided approaches, and visual-textual or low-rank continual-learning designs.The appendix describes the methods’ principal mechanisms, including exemplar replay and external language or reference data for some baselines.Source: §B.2 · pp. 13–14

B.3 Additional Implementation Details

  • DuLBE trains with current-task data only, estimates prospective gradients without updating parameters, and uses rank-8 text-side LoRA; its optimization epochs do not exceed the corresponding standard protocols.The gradient-probing pass constructs shared and residual bases and is then discarded; the text tower’s pretrained weights remain frozen.Source: §B.1 · p. 13, §B.3 · p. 14

B.4 Evaluation Metrics

  • The evaluation reports accuracy after each task across all classes seen so far without task identity, alongside average accuracy across stages and final accuracy over all benchmark classes.The average measures performance consistency through the learning stream, while the final accuracy summarizes the final balance between new-class learning and old-knowledge retention.Source: §B.4 · p. 14

C.1 Comparison with Other Baselines

Under Setting-A, DuLBE performs strongly against methods using additional training data and remains competitive in comparisons with LoRA-based learners. It also leads on both reported metrics for CIFAR and SUN under the learning-from-half protocol without replay or external reference images.

C.1 Comparison with Other Baselines

  • Without historical exemplars or external training images, DuLBE has the best average and final accuracies on CIFAR and ImageNet-R, ranks second on both CUB metrics, and beats the strongest replay-based CIFAR results by 0.3 and 0.6 points.It slightly outperforms LfI on ImageNet-R and remains competitive with SGCL on CUB; Table C.1 compares methods using additional training data.Source: §C.1 · p. 14, Table C.1 · p. 15
  • The LoRA-based comparison matches visual-tower insertion locations and uses the standard CLIP text classifier for the baselines and DuLBEText, while DuLBEB.E. uses the bridge-prototype ensemble.Table C.2 reports this classifier-matched comparison and the complete method.Source: §C.1 · pp. 14–15, Table C.2 · p. 15

C.2 Performance under the Learning-from-Half Protocol

  • Under learning-from-half, DuLBE achieves the best average and final accuracies on CIFAR and SUN without replay or external reference images.The comparison includes PROOF with replay exemplars and LfI with COCO reference pairs.Source: §C.2 · p. 15

C.3 Performance with CLIP ViT-L/14 Backbone

DuLBE is evaluated with OpenAI CLIP ViT-L/14 under the unchanged uniform 10-task CIL protocol. It achieves the best results across all four reported metrics, while its text-classifier variant ranks second.

C.3 Performance with CLIP ViT-L/14 Backbone

  • The evaluation replaces ViT-B/16 with ViT-L/14 and retains uniform 10-task CIL, with 10 and 20 new classes per task on CIFAR and ImageNet-R, respectively.The CIL protocol remains unchanged.Source: §C.3 · p. 15, Table C.4 · p. 16
  • DuLBE achieves the best results across all four reported metrics with ViT-L/14, and its text-classifier variant consistently ranks second.The comparison covers CIFAR-100 and ImageNet-R; DuLBE uses neither replay nor external reference data.Source: §C.3 · pp. 15–16, Table C.4 · p. 16

C.4 Additional Ablation Studies

The ablations examine bridge-depth discretization, reliability smoothing, and insertion locations for the dual-mode learner. They support using Kb = 10, β = 0.05, and K, V + MLP as the default configuration.

C.4 Additional Ablation Studies

  • Al rises from 83.6% at Kb = 3 to 84.2% at Kb = 10, with no further gain at Kb = 20; reliability-guided ensembling consistently beats fixed-depth classifiers.The study keeps the trained dual-mode learner and other settings unchanged; Kb = 10 provides sufficient bridge granularity without redundant prototypes.Source: §C.4 · p. 16, Figure C.1 · p. 16
  • Al rises from 83.72% at β = 0.005 to 84.22% at β = 0.05, then falls slightly to 84.07% at β = 0.075, motivating β = 0.05.Larger β smoothly spreads weight over neighboring depths while preserving class-specific preferences; very small β produces nearly one-hot selections sensitive to reliability noise.Source: §C.4 · p. 17, Figure C.2 · p. 17
  • K, V + MLP is the adopted insertion configuration: combining key/value projections with MLP layers provides complementary benefits, while adding query projection slightly weakens final retention.Configurations use rS = 1 and rR = 8, with modules outside the specified locations frozen.Source: §C.4 · p. 17, Table C.5 · p. 17

C.5 Overhead

DuLBE’s overhead includes trainable components in both CLIP towers, bridge scoring, and persistent prototype and statistic memory. Its reported memory variants trade lower statistic storage for strong final performance.

C.5 Overhead

  • DuLBE’s visual trainable parameters are 0.0645M for r = 1 and 0.5806M for r = 1+8, while inference retains 0.0522M additional parameters.Inference uses cached class-text embeddings and folded mergeable LoRA updates; bridge scoring across Kb = 10 depths costs 0.000512 GFLOPs.Source: §C.5 · p. 17, Table C.6 · p. 18
  • DuLBE requires only 0.20 MiB of representation memory while achieving the highest final accuracy under Setting-A.It stores one visual prototype and reliability weights per class; bridge prototypes are generated when needed rather than kept in an additional bank.Source: §C.5 · p. 18, Table C.7 · p. 18
  • DuLBE’s layer-wise statistics are stored only at learner insertion locations and updated in place, so their storage is independent of task and observed-class counts.Pretrained CLIP parameters, temporary tensors, and the shared class-text embedding cache are excluded from the memory accounting.Source: §C.5 · p. 18
  • Statistic-storage variants reduce storage by approximately 18×, 54×, and 216× while retaining strong final performance; DuLBE-C exceeds BOFA by 1.9 and 1.4 points on CIFAR-100 and Food-101, respectively.Variant A uses 27.00 MiB and reaches 83.9% and 85.5%; Variant B uses 9.00 MiB with comparable performance, and Variant C uses 2.25 MiB.Source: §C.5 · p. 18, Table C.8 · p. 18

C.6 Visualizations

The visualizations examine bridge-depth attributions and layer-wise gradient demand. They show complementary evidence across bridge depths and different shared-versus-residual demand across visual layers; the appendix also summarizes DuLBE’s task-wise procedure.

C.6 Visualizations

  • Intermediate bridge prototypes provide distinct transitions between visual-endpoint appearance/context evidence and more localized, class-discriminative text-endpoint evidence.The transition varies across classes, and similar behavior on ImageNet-100 and ImageNet-R suggests intermediate prototypes remain meaningful under visual domain shifts.Source: §C.6 · pp. 18–19, Figure C.3 · p. 19
  • Gradient energy varies substantially across tasks and visual layers, supporting layer-wise rather than fixed whole-tower allocation of low-rank directions.Across early and middle layers, the compact shared basis captures substantial demand; in deeper layers shared capture decreases while residual capture increases, indicating stronger demand outside historical support. The two ratios are computed independently, not as a complete gradient decomposition.Source: §C.6 · p. 19, Figure C.4 · p. 19

D Complete Procedure

  • Algorithm D.1 summarizes DuLBE’s complete task-wise procedure, initializing the model and layer statistics before iterating over tasks.Its stated outputs include the final model parameters and stored visual prototypes with reliability weights.Source: D Complete Procedure · pp. 19–20
Loading 2609.36759v1…