Source-linked AI summary
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, You He
TL;DR
Large vision-language models must learn new tasks without full historical data, while avoiding catastrophic forgetting and the costs of full-model tuning. This paper dynamically expands frozen CLIP with MoE-Adapters and uses DDAS to route inputs between learned adapters and original CLIP, reporting broad gains with 60% lower training parameter burdens.
Problem
Continual learning of large vision-language models must address long-term forgetting, parameter shifts, and the computational burden of full-model tuning.
Method
The method dynamically expands frozen CLIP with task-specific MoE-Adapters and uses DDAS to route inputs to adapters or original CLIP.
Results
The method consistently outperforms previous state-of-the-art approaches across settings while reducing training parameter burdens and memory requirements by 60%.
Takeaways & Limitations
MoE-Adapters and DDAS combine continual adaptation, anti-forgetting, and zero-shot transfer within a parameter-efficient unified framework.
Takeaways & Limitations
DDAS requires a predefined threshold, and a single threshold may cause errors as the number of tasks grows.
Abstract
from arXiv · showhide
Continual learning can empower vision-language models to continuously acquire new knowledge, without the need for access to the entire historical dataset. However, mitigating the performance degradation in large-scale models is non-trivial due to (i) parameter shifts throughout lifelong learning and (ii) significant computational burdens associated with full-model tuning. In this work, we present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters in response to new tasks. To preserve the zero-shot recognition capability of vision-language models, we further introduce a Distribution Discriminative Auto-Selector (DDAS) that automatically routes in-distribution and out-of-distribution inputs to the MoE Adapter and the original CLIP, respectively. Through extensive experiments across various settings, our proposed method consistently outperforms previous state-of-the-art approaches while concurrently reducing parameter training burdens by 60%. Our code locates at https://github.com/JiazuoYu/MoE-Adapters4CL
1. Introduction
Continual learning reduces the need to retrain on historical data but faces catastrophic forgetting, computational burdens, and weakened zero-shot transfer. The proposed framework combines MoE-Adapters with DDAS to support parameter-efficient adaptation and automatic routing between learned experts and frozen CLIP.
- Continual learning incrementally trains on new data, but models can lose previously acquired knowledge through catastrophic forgetting.
- Dynamic expansion methods add task-specific components but cannot distinguish unseen data, while zero-shot continual learning introduces significant computational burdens.
- Stacking independent adapters creates task-identity dependence and limits inter-task knowledge sharing in class-incremental settings.
- The framework uses task-specific MoE-Adapter experts and routers on a frozen CLIP, with activate-freeze training to support intra-task learning and inter-task collaboration.
- DDAS automatically routes seen data to MoE-Adapters and unseen data to pretrained CLIP, combining anti-forgetting with zero-shot transfer.
- 60% lower parameter burdens and memory requirements are reported during training, while 5-shot continual learning surpasses prior methods by 3.6%, 7.0% and 4.2%.
2. Related Works
Prior work includes memory-based, parameter-efficient, and mixture-of-experts approaches for reducing continual-learning costs and preserving knowledge. The proposed framework builds on these directions by applying MoE components to a frozen CLIP and using DDAS for automatic task-context routing.
- Continual Learning: Continual-learning methods address class-incremental and task-incremental settings through memory-based, regularization-based, and dynamic architectures.
- Proposed Framework: The proposed method places MoE-Adapters on frozen CLIP encoders and sequentially adds task-dependent routers to consolidate downstream tasks in one model.
- Proposed Framework: During inference, DDAS analyzes input-distribution variation to assign data automatically to MoE-Adapters or original CLIP for seen or unseen inputs.
- Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning adapts large-scale models by optimizing only a small number of trainable parameters, reducing parameter and time burdens.
- Mixture-of-Experts: Mixture-of-Experts methods use sparse expert architectures to decrease computational costs and improve model capacity, including in continual learning.
3. Methodology
The framework expands a frozen CLIP with task-specific MoE adapters and routers, then uses DDAS to route inputs while preserving zero-shot transfer. Continual learning is supported through expert freezing, sparse routing, and task-distribution auto-selection.
- Continual Learning: Continual learning accesses tasks sequentially and must preserve performance across both task-incremental and class-incremental settings.TIL uses task-specific class sets, whereas CIL distinguishes among all previously encountered classes.
- Incremental MoE-Adapters: The method adds sparse adapter experts and task-dependent routers to a frozen CLIP, combining expert outputs through gated averaging.Adapters are implemented in the parallel image and text encoders without sharing parameters.
- Incremental MoE-Adapters: Router gating uses the [CLS] token, selects the Topk most relevant experts, and normalizes their contributions with Softmax.This avoids routing all patch or image tokens and produces an output matching the input token shape.
- Incremental MoE-Adapters: The activate-freeze strategy freezes the most activated experts from older tasks while allowing new tasks to optimize unfrozen experts and reuse historical knowledge.Training can activate only new experts, combine new and frozen experts, or use only previously learned experts.
- Experiments: The method is evaluated on MTIL and CIL benchmarks, with MTIL reported using Transfer, Average, and Last metrics.The few-shot MTIL comparison reports convergence in 500 iterations, while Table 1 distinguishes 1k- and 3k-iteration variants.
- Distribution Discriminative Auto-Selector: DDAS uses task-specific autoencoders and reconstruction scores to infer task context, routing unseen inputs to frozen CLIP for zero-shot transfer.Lower reconstruction scores indicate a higher likelihood that an input belongs to the corresponding task; scores above the threshold are classified as unseen.
4. Experiments
Experiments evaluate the framework on multi-domain and class-incremental learning, including full-shot and few-shot settings, ablations, and computational cost. The method improves continual-learning performance while preserving zero-shot transfer and reducing training burdens.
- Experimental Setting: The evaluation covers MTIL and CIL, using Transfer, Average, and Last metrics for MTIL and Average and Last accuracy for CIL.
- Multi-domain Task Incremental Learning: In full-shot MTIL, Ours improves over the second-best method by 0.8% on Transfer, 1.3% on Average, and 1.4% on Last.
- Few-shot Multi-domain Task Incremental Learning: In 5-shot MTIL, Ours surpasses the second-best method by 3.6% on Transfer, 7.0% on Average, and 4.2% on Last.
- Class Incremental Learning: On CIFAR100 and TinyImageNet CIL, the method consistently outperforms dynamic-expansion and CLIP-based competitors despite unknown task identities.
- Analysis of MoE-Adapters: Task-specific routers contribute more than increasing expert count to anti-forgetting and zero-shot transfer, while activate-freeze improves collaboration between learned and inactive experts.
- Analysis of DDAS: DDAS learns discriminative task distributions in full-shot and few-shot MTIL, although some Task 9 samples overlap with Task 11 because of misclassification.
5. Discussion
The framework combines MoE-Adapters with DDAS to support efficient continual learning while routing inference between adapted components and frozen CLIP. Its main limitation is threshold dependence as task numbers grow.
- MoE-Adapters let CLIP adapt efficiently across tasks, while DDAS automatically assigns inference data to MoE-Adapters or frozen CLIP.
- DDAS requires a predefined threshold for downstream branch selection, and one threshold may produce errors as the number of tasks increases.
A. More Implementation Details
Implementation uses benchmark-specific batch sizes, searched learning rates, and different regularization choices for MTIL and CIL.
- Batch size is 64 for MTIL and 128 for CIL, with learning rates searched among [10^-3, 10^-4].
- Label-smoothing strength is searched between {0.1, 0.2}; for CIL, weight decay and label smoothing are both set to 0.
B. Impact of dataset size on expert number
The optimal number of experts depends on dataset and task size: more tasks generally require more experts, but increasing experts indiscriminately does not always improve accuracy.
- More tasks generally require more experts, but simply adding experts does not always improve accuracy.
C. Analysis on the Threshold and Different Loss in DDAS
DDAS remains relatively stable across a searched threshold range, while MSE produces the best performance among the tested autoencoder losses.
- Threshold analysis: DDAS performance fluctuates relatively stably when thresholds are searched within [0.06, 0.07].Compared with “Ours†”, “Ours” is more consistent as the threshold changes.
- Loss analysis: MSE loss yields the best performance for the DDAS autoencoder among the evaluated loss functions.
D. More Comparison Results on MTIL
Additional MTIL results show that the proposed method performs favorably against state-of-the-art approaches across both training orders and full-shot and few-shot settings, while retaining near-upper-bound zero-shot transfer.
- The proposed method performs favorably against state-of-the-art approaches on “Transfer”, “Average”, and “Last” in both MTIL settings.The reported settings include Order-II and the full-shot and few-shot benchmarks.
- Its zero-shot transfer ability closely reaches the upper bound of pretrained CLIP.
E. Effectiveness of Router Selection in MoE-Adapters
Router selection activates sparse expert combinations across visual transformer blocks, with experiments also examining expert counts, DDAS thresholds, losses, and MTIL benchmark outcomes.
- Router selection: Expert activation frequencies are recorded across all visual transformer blocks using 22 experts and Top-k 2.The visualization shows sparsity in router-selected experts and cooperation between special and shared experts.
- Expert-count ablation: The expert-count ablation examines how dataset size affects the optimal number of experts.
- DDAS ablations: The DDAS loss ablation compares different autoencoder losses.
- DDAS ablations: The DDAS threshold ablation searches thresholds within [0.06, 0.07].
- MTIL comparisons: The MTIL results are organized as accuracy matrices in which each row represents performance after the corresponding task.