PaperlayerView exact revision on arXiv

Source-linked AI summary

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

Chang Sun, Francesco Barbato, Matteo Caligiuri, Pietro Zanuttigh

arXiv:2608.21487v1cs.CVcs.AI

TL;DR

VLM continual adaptation must preserve prior knowledge and zero-shot capability while learning new tasks, but existing regularization can reduce plasticity. TASSO combines task-specific low-rank subspace optimization with geometry-aware, geodesic-distance distillation. It outperforms prior state-of-the-art methods on MTIL and MCIL while reducing catastrophic forgetting and zero-shot degradation.

  • Problem

    VLM continual learning must balance new-task accuracy, preservation of old knowledge, and zero-shot capability because adaptation causes catastrophic forgetting and zero-shot degradation.

  • Method

    TASSO constrains current-task optimization to task-specific low-rank subspaces and distills prior-task knowledge using a geometry-aware geodesic-distance objective.

  • Results

    TASSO outperforms previous state-of-the-art techniques by more than 1% accuracy on both MTIL and MCIL, with drastically reduced catastrophic forgetting and zero-shot degradation.

  • Takeaways & Limitations

    The approach preserves continual-learning performance and zero-shot capabilities without explicitly adding adapter modules or computationally demanding additional stages.

  • Takeaways & Limitations

    The current method adapts only the visual side, and its generalizability to other VLMs and computer-vision tasks remains a future direction.

Abstract

from arXiv · show

Vision-Language Models (VLMs) exhibit strong zero-shot capabilities, making them an attractive solution for continual learning across diverse tasks. However, during continual adaptation, both catastrophic forgetting and zero-shot degradation occur, severely degrading performance. In this paper, we introduce TASSO, a new paradigm that efficiently preserves the latent space geometry while ensuring network plasticity. We achieve this with two complementary techniques: subspace learning and geometry-aware knowledge distillation. Specifically, we first learn a sequence of task-specific low-rank projectors, which we use to project the latent representations before optimizing cross-entropy. Secondly, we employ a geodesic-distance-based loss that distills knowledge from the previous-task model while effectively preserving the latent space geometry. These design choices not only avoid unnecessary parameter updates along the full embedding dimensions but also improve learning by focusing on task-specific manifolds. Moreover, the geometry-aware distillation provides strong regularization and significantly reduces both catastrophic forgetting and zero-shot degradation throughout the continual learning sequence. Experimental results with the CLIP vision language model in the multi-domain task incremental and class incremental learning benchmarks demonstrate clear improvements over state-of-the-art methods in mitigating forgetting and preserving zero-shot capabilities.

1 Introduction

VLMs must continually adapt to new concepts while preserving prior knowledge and zero-shot capability. TASSO addresses this balance with task-specific subspace learning and geometry-aware distillation, achieving state-of-the-art continual-learning performance.

  • Motivation: VLM continual learning must address both catastrophic forgetting on previous tasks and zero-shot degradation on future unrelated tasks.These pressures make the plasticity-stability tradeoff especially challenging.
  • Limitations of Existing Methods: Existing distillation methods reduce forgetting but can diminish learning on new tasks, while external adapters add cumbersome parameters that scale poorly.The two approaches expose complementary limitations in balancing retention, adaptation, and architecture size.
  • TASSO: TASSO separates current-task knowledge from general knowledge through task-specific subspace learning and a geodesic-distance distillation objective.The framework focuses optimization on relevant manifolds and distills prior-task information through orthogonal subspaces.
  • Results: Experiments in Multidomain Class Incremental and Task Incremental Learning achieve state-of-the-art performance on challenging VLM continual-learning benchmarks.The evaluation covers both MCIL and MTIL settings.
  • Contributions: TASSO uses low-rank subspace learning, learned projectors, and geometry- and subspace-aware distillation to target catastrophic forgetting and zero-shot degradation.The contributions selectively apply losses to relevant and irrelevant feature components.

2 Related Works

Prior VLM continual-learning methods seek to preserve learned knowledge and zero-shot generalization through distillation, parameter-efficient updates, or modular architectures. TASSO combines geometry-aware distillation with low-rank subspace optimization without explicitly adding adapter modules.

  • VLM Continual Learning: VLM continual learning aims to preserve previously learned knowledge and pretrained zero-shot generalization across sequential tasks.The literature increasingly studies pretrained models such as CLIP in this setting.
  • Knowledge Distillation: Distillation-based methods transfer knowledge from pretrained or previous-task models, sometimes using reference images or two teachers to preserve zero-shot capabilities.These methods also target multimodal proximity and intra- and inter-modal information.
  • Knowledge Distillation: Distilling the pretrained model can increase storage and computational costs while risking reduced ability to learn new tasks.This limitation is explicitly associated with methods that rely on pretrained-model distillation.
  • Parameter-Efficient Training: Parameter-efficient methods such as LoRA and adapters limit optimized parameters, but rank choices create a tradeoff between task learning and forgetting.Higher rank improves task learning while lower rank reduces forgetting but limits adaptation.
  • Modular Adaptation: MoE adapters dynamically expand CLIP and route inputs between adapters and the pretrained model to preserve zero-shot ability.The Auto-Selector performs the routing in response to new tasks.
  • TASSO: TASSO transfers prior knowledge through geometry-aware metrics while constraining current-task optimization to a low-rank subspace without explicitly adding adapters.It is presented as a balance between parameter-efficient training and knowledge distillation.

3 Continual Learning Setup

The continual-learning setup trains a VLM, instantiated as CLIP, across a sequence of task datasets with associated class names. Classification compares normalized visual and textual embeddings produced from images and prompted class names.

  • Task Sequence: The model g is continually optimized across a sequence of K tasks, each containing labeled images and corresponding class names.For task T_k, the model is trained on task-specific samples and labels.
  • Task Data: Each image x_j^k has a corresponding one-hot encoded label y_j^k used during task training.The label is part of the task dataset representation.
  • Task Data: The one-hot label also selects the appropriate class name from the task’s class set for textual prompting.This connects supervised labels to the VLM’s text branch.
  • Embedding Construction: Each class is inserted into the template “The photo of {c}” and encoded by CLIP’s textual branch, while images are encoded by its vision branch.The resulting text and visual embeddings are normalized before use.
  • Classification: Classification compares the normalized visual and textual embeddings produced by the two CLIP branches.The passage establishes the cross-modal comparison used for prediction.

4 Continual Learning Strategy

TASSO combines task-specific low-rank subspace optimization with geometry-aware distillation to balance plasticity and stability during continual VLM adaptation. It separates task-relevant and task-irrelevant representations, applying complementary objectives while preserving CLIP's latent-space geometry.

  • Motivation: Continual VLM optimization with cross-entropy alone causes catastrophic forgetting on previous tasks and zero-shot degradation on unseen tasks.Knowledge distillation is used as a regularizer to mitigate these effects.
  • Subspace Learning: TASSO uses a learnable low-rank projector to separate current-task and task-irrelevant subspaces, restricting new-task learning to a task-specific manifold.The projector Uk ∈ R^d×r uses r ≪ d; experiments set r = 144.
  • Subspace Learning: The projected subspace supports a task-specific cross-entropy objective, computed from class probabilities based on projected image and text embeddings.Projected embeddings are re-normalized before calculating the subspace objective.
  • Geometry-Aware Distillation: TASSO measures representation changes with geodesic distance because normalized CLIP embeddings lie on a high-dimensional sphere.The geodesic follows the shortest path on the sphere rather than a straight line through the ambient space.
  • Geometry-Aware Distillation: Knowledge distillation is applied separately to task-specific and task-irrelevant subspaces using the sum of their geodesic distances.This decomposition extracts task-relevant information while preserving task-irrelevant information and addresses the stability–plasticity trade-off.
  • Learning Objective: The total objective combines global cross-entropy, subspace learning, and decomposed distillation losses, weighted by α = 0.5 and β = 3.These hyperparameters control the trade-off between plasticity and stability.

5 Experimental Evaluation

TASSO is evaluated with CLIP across eight fine-grained datasets, eight rotated task sequences, and both task-incremental and class-incremental settings. It achieves strong accuracy while reducing catastrophic forgetting and zero-shot degradation, with supporting alignment and ablation analyses.

  • Experimental Setup: The evaluation uses CLIP with a ViT-B/16 vision encoder and a 12-layer text encoder with 512-dimensional latent representations.Only the vision encoder is optimized during continual learning; the text encoder remains frozen.
  • Experimental Setup: Eight fine-grained classification datasets are evaluated in both Multidomain Task-Incremental and Multidomain Class-Incremental Learning settings.The datasets include FGVC-Aircraft, DTD, EuroSAT, Flowers-102, Food-101, Oxford-Pets, Stanford-Cars, and UCF-101.
  • Experimental Setup: Eight task sequences are constructed by rotating the original ordering, addressing the substantial influence of task ordering on continual-learning performance.The original sequence is S1, while the other seven sequences shift the starting task one position at a time.
  • Main Results: TASSO achieves 85.89% average accuracy in MTIL, surpassing the best competitor by ≈1%, while catastrophic forgetting averages 0.67% versus 1.2%.Forgetting remains below 1% across all sequences, with a worst result of 0.96% in S7.
  • Main Results: Zero-Shot Degradation averages 0.47% in MTIL, four times lower than the closest competitors’ values around 2%, with sequence values from 0.19% to 0.68%.The results are consistent across task orderings, and the paper notes that some datasets may provide information useful for distinguishing samples from other tasks.
  • Alignment Analysis: TASSO achieves lower text–vision prototype distances than SnD in the S1 ordering, indicating significantly improved multimodal alignment in 7 of 8 tasks.Food-101 is the only dataset where the competitor has a slight advantage; the boxplot analysis confirms the distributional comparison.
  • Ablation Study: Ablations show that jointly using the two method components performs best, with the geometry-aware geodesic metric outperforming standard L2 distance.Hyperparameter results indicate relatively low sensitivity, although the selected projector rank r = 144 gives the best performance across all three metrics.

6 Conclusions and Future Work

TASSO combines low-rank subspace learning with geometry-aware distillation to reduce forgetting and zero-shot degradation in continual VLM adaptation. The authors report accuracy gains over prior state-of-the-art methods while identifying visual-only adaptation and broader generalization as future directions.

  • Conclusions: TASSO uses subspace learning and a modified distillation distance to reduce catastrophic forgetting and zero-shot degradation.The approach uses a single teacher model without computationally demanding additional modules or stages.
  • Conclusions: TASSO outperforms previous state-of-the-art techniques by more than 1% in accuracy on both MTIL and MCIL benchmarks.The paper also reports drastically reduced catastrophic forgetting and zero-shot degradation in both settings.
  • Future Work: Future work includes dynamic low-rank approximation, text-image alignment, other VLMs, and computer-vision tasks beyond image classification.The authors specifically note that the current paper adapts only the visual side.

B.1 Ablation on the method’s components on MCIL

MCIL ablations show that combining subspace learning with geometry-aware distillation gives the strongest overall performance. Additional experiments examine hyperparameter robustness, embedding decomposition, and joint versus separate optimization of the subspace and encoder.

  • Method Components: The best MCIL performance occurs when knowledge distillation is combined with subspace learning, while geodesic distillation consistently outperforms L2.With subspace learning enabled, geodesic distillation reduces forgetting and zero-shot degradation by 1.72% and 2.32%, respectively.
  • Method Components: Adding subspace learning provides a further 1.17% accuracy boost, reaching 85.00% when both strategies are enabled.The ablation attributes the preservation effect to distillation and finds hypersphere geometry matching more effective than Euclidean alignment.
  • Hyperparameters: The selected subspace rank gives the best accuracy, while the chosen α and β values provide a strong tradeoff across metrics.Metrics remain quite constant across the tested hyperparameter values.
  • Embedding Decomposition: Applying knowledge distillation to original vectors yields nearly identical accuracy but worsens catastrophic forgetting and zero-shot degradation by 33% and 39%.The MTIL evaluation reports accuracy 85.72, catastrophic forgetting 1.21, and zero-shot degradation 0.59.
  • Optimization Strategies: Joint optimization of the image encoder and projector performs best, while alternating optimization remains a viable option.TASSO (alt) achieves Acc 85.51, CF 0.70, and ZS 0.20, whereas sequential optimization reduces all reported metrics.

C Computational Requirements

TASSO adds negligible computational overhead from subspace learning and trains faster than the main competitor while using the same reported maximum VRAM.

  • Compute: 13.34 TFLOPs are required for TASSO on S1, compared with 15.49 TFLOPs for SnD.The TASSO figure is unchanged with or without subspace learning.
  • Runtime: TASSO takes 1h36m00s to train on S1, versus 1h50m12s for SnD.Removing subspace learning reduces TASSO training time by only three seconds, to 1h35m57s.
  • Memory: Maximum VRAM use is 17.13 GB for all reported settings.

D Per-Domain Results

Per-domain results on sequence S1 provide detailed incremental-step results for MTIL and MCIL and corroborate the aggregate benchmark findings. The MCIL setting is identified as more challenging.

  • Per-Domain Evaluation: The supplementary tables report accuracy for each domain at every incremental training step in MTIL and MCIL.S1 results are presented separately for the two benchmarks.
  • Per-Domain Evaluation: The per-domain tables corroborate the aggregate metrics reported for the two benchmark settings.
  • Benchmark Settings: MCIL is characterized as the more challenging setting compared with MTIL.

E Statistical Analysis

TASSO's gains remain stable across random seeds in the reported MTIL sequence, with contained variability in forgetting, degradation, and accuracy.

  • TASSO's gains over the closest competitor persist across all tested random seeds in MTIL sequence S1.The study used multiple random initializations and reports gains outside the measure's confidence bounds.
  • The variance of Forgetting is 0.20, while Degradation and accuracy have variances of 0.18 and 0.19, respectively.
  • The main experimental seed is closest to mean performance across seeds, with an average distance of 0.08 across all metrics.
  • Seed 1102 was selected to match the main competitor's seed in the statistical-significance comparison.

F Details on Datasets and Sequences

The appendix documents the datasets, sample and class counts, and eight task sequences used to evaluate TASSO.

  • Eight task sequences were used in the TASSO experimental evaluation.Table 13 provides their detailed breakdown.
  • Table 12 reports the number of classes and training and test samples for every dataset used.Only a subset of ImageNet's 1.28M training samples was used.

G Pseudocode of our Method

TASSO fine-tunes a vision encoder on each task using task-specific subspace learning, geometry-aware distillation, and an evaluation protocol that aggregates continual-learning metrics.

  • Fine-tuning protocol: The fine-tuning protocol receives current-task data, a previous-step model, a reference dataset, training steps, and hyperparameters α and β.
  • Fine-tuning protocol: Current-task image and text embeddings are normalized before cross-entropy supervision produces task predictions.
  • Subspace learning: An orthonormal low-rank projector is obtained through QR reparameterization and maps embeddings into the task-specific subspace.
  • Optimization: The optimization objective combines cross-entropy, subspace, and geometry-aware knowledge-distillation losses as L = LCE + α Lsub + β LKD.
  • Evaluation protocol: Evaluation compares image embeddings with text embeddings across task-sequence datasets and stores top-1 accuracies in a task-by-task matrix.
  • Evaluation protocol: Mean accuracy, mean forgetting, and mean zero-shot degradation are computed from the task-by-task accuracy matrix.
Loading 2608.21487v1…