PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Branch is What You Need for CLIP-based Class-Incremental Learning

Tao Hu, Zhen-Hao Xie, Jingcai Guo, De-Chuan Zhan, Da-Wei zhou

arXiv:2609.37888v1cs.CVcs.LG

TL;DR

Textual classifier weights in CLIP-based CIL may be misaligned with visual class distributions, motivating a visual-only alternative. VIS enhances visual features using base-session data and builds an incremental classifier with closed-form LS-SVM updates. The paper reports state-of-the-art performance without a textual branch.

  • Problem

    CLIP-based CIL commonly uses text-derived class weights, but textual embeddings can be misaligned with visual class distributions.

    Source: §1 · p. 2

  • Method

    VIS enhances CLIP’s final visual representation with visual-layer features using base-session data and builds a visual-space kernelized incremental LS-SVM from additive sufficient statistics.

    Source: §1 · p. 3, §4.1 · p. 4, §4.3 · p. 7

  • Results

    VIS achieves state-of-the-art performance without a textual branch.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The paper concludes that strong CLIP-based CIL can be achieved without a textual branch.

    Source: §6 · p. 9

    The evaluated setting is exemplar-free: previous-task samples cannot be stored or replayed, and updates access only current-task data.

    Source: §3 · p. 3

Abstract

from arXiv · show

Class-Incremental Learning (CIL) requires models to recognize new classes over time without forgetting previously learned ones. With the rise of vision-language pre-training, CLIP has become a strong foundation for CIL. A common design in CLIP-based CIL is to construct textual classifier weights by encoding class-name templates with the CLIP text encoder, and then classify visual features by image-text cosine similarity. This design is appealing: since CLIP aligns images and text in a shared embedding space, textual weights appear to provide an off-the-shelf classifier for incremental classes. However, we show that this seemingly natural design is not always beneficial, as a modality gap can still separate the two modalities and make textual classifier weights deviate from visual class distributions. Empirically, under identical task-wise CIL training, initializing the cosine classifier with visual class centers yields lower loss and better incremental accuracy than using CLIP textual features.Motivated by these observations, we propose VIS, a visual-only method for CLIP-based CIL that removes the deployed textual branch and constructs the incremental classifier entirely in the visual space. To obtain stronger task-adaptive visual representations, VISuses only base-session data to enhance CLIP's final visual representation with informative visual-layer features. Built on the enhanced visual representation, VISemploys a simple kernelized incremental least-squares SVM, whose classifier weights are solved in closed form from additive sufficient statistics. When new classes arrive, VISaccumulates their sufficient statistics and recomputes the classifier weights for all seen classes, enabling efficient incremental updates while preserving historical class knowledge. Extensive experiments show that VISachieves state-of-the-art performance without a textual branch.

1 INTRODUCTION

CLIP’s text branch offers a convenient way to build incremental classifiers, but textual weights may not align well with visual class distributions. VIS instead constructs its deployed classifier entirely in visual space.

1 INTRODUCTION

  • CIL addresses the need to learn continuously arriving categories while retaining discrimination over previously learned classes.Pre-trained models, including CLIP, are attractive because they provide generalizable representations and support lightweight adaptation.Source: §1 · p. 1
  • CLIP-based CIL commonly uses class-name templates encoded by the text encoder as classifier weights, applying image-text cosine similarity to visual features.This offers a convenient way to derive a new class’s classifier weight directly from its name.Source: §1 · pp. 1–2
  • On CIFAR100, visual samples and prototypes cluster on the visual side of CLIP’s shared space, while textual features lie in a separate region.The class-level modality gap is larger when visual-textual alignment is weaker; on FGVCAircraft, that gap is positively correlated with textual-head degradation.Source: §1 · p. 2, Figure 1 · p. 2
  • Under identical ten-task CIL training, visual-prototype initialization consistently yields lower loss and higher seen-class accuracy than textual initialization for the same cosine classifier.The visual encoder is frozen, and only the cosine classifier is trained for ten epochs per task.Source: §1 · p. 2
  • VIS enhances CLIP’s final visual representation with visual-layer features using base-session data, then uses a kernelized incremental least-squares SVM with closed-form weights from additive sufficient statistics.For new classes, VIS accumulates sufficient statistics and recomputes weights for all seen classes.Source: §1 · p. 3

2 RELATED WORK

Prior CIL work addresses forgetting through several strategies, while pre-trained-model-based CIL adapts frozen backbones with task-specific components and analytic classifiers.

Class-Incremental Learning

  • CIL methods address forgetting through parameter regularization, replay, dynamic architecture expansion, or distillation of prior responses or representations.Examples include restricting parameter updates [7, 8, 9], retaining or synthesizing old data [17, 18, 19, 20], and expanding model capacity [21, 22, 23, 24].Source: §2 · p. 3

Pre-Trained Model-Based CIL

  • Pre-trained-model CIL adapts frozen backbones with prompts or adapters, while analytic and random-feature classifiers use closed-form or recursive updates over pre-trained features.CLIP-based methods additionally use textual prototypes, prompt tuning, adapters, or language-guided representations [Wang et al., 2023; Huang et al., 2024; Yu et al., 2025; Lu et al., 2025].Source: §2 · p. 3

3 PRELIMINARIES

The paper considers exemplar-free class-incremental learning, where each update must preserve recognition of old classes without storing or replaying their samples. It also describes CLIP’s text-based zero-shot classifier and motivates constructing the classifier in visual space instead.

Class-Incremental Learning

  • In the exemplar-free setting, tasks have disjoint label spaces, evaluation covers all seen classes, and only current-task training data are accessible during an update.The first task is the base session; subsequent tasks are incremental sessions.Source: §3 · p. 3

Zero-shot CLIP classification

  • CLIP zero-shot classification encodes images and class-name prompts into a shared embedding space and uses textual embeddings as class weights for cosine-similarity classification.The textual branch is convenient for building classifiers, but textual embeddings can be misaligned with visual class distributions and hinder optimization when used for initialization.Source: §3 · p. 4

4 VIS: VISUAL INCREMENTAL SVM

VIS builds its incremental classifier entirely in visual space, combining base-session adaptation of CLIP visual features with a kernelized LS-SVM updated from additive sufficient statistics. Its updates recompute weights for all seen classes rather than appending weights only for new classes.

4.1 TASK-ADAPTIVE VISUAL REPRESENTATION

  • A residual mixer learns from multi-layer CLS features to correct CLIP’s final visual feature while the visual encoder remains frozen.The mixer is trained only in the base session with the adaptation loss, then fixed; its residual branch is zero-initialized so the initial fused representation equals the final-layer feature.Source: §4.1 · pp. 4–5

4.2 INCREMENTAL KERNELIZED LS-SVM CLASSIFIER

  • The LS-SVM uses one-vs-all targets and a closed-form solution based on feature and feature-label correlations that can be accumulated task by task.The method chooses this formulation for its discriminative supervision, kernel compatibility, and closed-form update properties in exemplar-free CIL.Source: §4.2 · pp. 5–6
  • A fixed nonlinear feature map transforms the enhanced visual representation, enabling a linear classifier to model nonlinear decision boundaries in the original visual space.The explicit map is kept fixed so the classifier remains compatible with closed-form sufficient-statistics updates.Source: §4.2 · pp. 5–6
  • When classes arrive, VIS updates accumulated statistics and recomputes classifier weights for all seen classes, retaining earlier class information rather than appending only new weights.The stored feature sum recovers previous samples’ negative contribution for newly introduced classes; the authors state this mechanism mitigates catastrophic forgetting.Source: §4.2 · p. 6

4.3 Summary of VIS

  • VIS combines task-adaptive visual representations with closed-form LS-SVM updates, using visual features throughout training and inference.The residual fusion module is trained in the base session; incremental sessions update sufficient statistics and recompute weights for all seen classes.Source: §4 · p. 4, §4.3 · p. 7
  • Validation-based regularization and visual-layer subset choices are made using base-session data, then fixed without incremental-session or test data.The layer subset is selected for highest validation accuracy among candidates; using every layer can increase computation and add redundant or less relevant cues.Source: §4.3 · p. 7

5 EXPERIMENTS

VIS is evaluated on nine CLIP-based class-incremental benchmarks, with component ablations and analyses of robustness and efficiency. It achieves state-of-the-art benchmark performance, and the full model leads the ablated variants across datasets.

5.1 IMPLEMENTATION DETAILS

  • The nine-dataset evaluation uses a shared CLIP ViT-B/16 backbone and a standard B-m Inc-n protocol; final-stage and mean accuracy are reported.The residual fusion module is trained for five base-session epochs; AB denotes final-stage accuracy and mean accuracy averages accuracy across incremental stages.Source: §5.1 · p. 8

5.2 BENCHMARK COMPARISON

  • VIS consistently achieves state-of-the-art performance, surpassing visual-only closed-form and recent CLIP-based methods on the benchmark datasets.The reported comparison includes ACIL and RanPAC, as well as CLG-CBM and BOFA.Source: §5.2 · p. 8

5.3 FURTHER ANALYSIS

  • The full VIS model performs best on every dataset, following gains from visual-space LS-SVM, the kernel map, and residual fusion.Visual LS-SVM substantially improves on the zero-shot textual head; adding the kernel map further improves accuracy before residual fusion yields the best results.Source: §5.3 · p. 9, Figure 4 · p. 9
  • On Aircraft B0 Inc10, VIS remains stable across λadapt values, while increasing kernel dimension D improves accuracy until saturation.The tested λadapt values range from 0 to 10^-1, and D ranges from 3,000 to 20,000.Source: §5.3 · p. 9, Figure 4 · p. 9
  • On Aircraft B0 Inc10, VIS achieves the best accuracy with a compact trainable-parameter budget and does not store past images.The closed-form solver maintains non-trainable sufficient-statistic states; the authors note these require extra storage but not learnable parameters.Source: §5.3 · p. 9, Figure 4 · p. 9

6 CONCLUSION

VIS constructs the incremental classifier in visual space, combining enriched visual features with a kernelized incremental LS-SVM. Experiments report state-of-the-art performance, while the fixed post-base representation limits later adaptation.

6 CONCLUSION

  • VIS removes the deployed textual branch, enriches CLIP’s final visual representation with multi-level features, and recomputes a kernelized incremental LS-SVM from additive sufficient statistics.The representation module and lifted feature space are fixed after the base session, limiting further representation adaptation for later tasks.Source: §6 · p. 9

AI USE STATEMENT

  • The authors report generative AI assistance for editorial tasks, but not for experimental execution or result generation, and state that they verified the assisted work.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendix includes supplementary analyses, implementation details, complete incremental learning curves, and broader-impact discussion.Its listed topics include robustness, generalization, efficiency, forgetting behavior, and comparisons with classifier-based CIL methods.Source: APPENDIX · p. 15
  • Appendix materials also cover the task-wise update algorithm, LS-SVM and kernel-map discussions, and descriptions of compared methods.They include additional preliminary experiments on textual classifier weights and modality-gap analysis.Source: APPENDIX · p. 15

A ALGORITHMIC SUMMARY

  • Algorithm 1’s Lift pipeline extracts selected visual-layer CLS features, applies residual fusion and a fixed kernel map, and returns augmented features.The layer subset and LS-SVM regularizer are selected using the base-session split.Source: A ALGORITHMIC SUMMARY · p. 15

B MORE IMPLEMENTATION AND REPRODUCIBILITY DETAILS

VIS’s implementation fixes its visual and kernel feature pipeline after base-session selection, then updates the LS-SVM using accumulated feature and feature-label correlations. The appendix explains the resulting classifier and kernel interpretation, and reports robustness across class-order seeds.

B MORE IMPLEMENTATION AND REPRODUCIBILITY DETAILS

  • For the ViT-B/16 experiments, VIS fixes the selected layer subset B = {6, 8, 10, 12}; λ and B are chosen on base-session data, not incremental or test data.The implementation uses ReLU, a once-randomized fixed projection R, and residual-mixer hidden dimension 256.Source: B MORE IMPLEMENTATION AND… · p. 15

B.1 REMARK ON THE LS-SVM CLASSIFIER

  • The LS-SVM uses one-vs-all ±1 targets and closed-form fitting; its feature and feature-label correlations accumulate across sessions for recomputation over all seen classes without storing past samples.The least-squares relaxation replaces hinge constraints with equality residuals.Source: §B.1 · pp. 15–16

B.2 KERNEL INTERPRETATION OF THE FIXED NONLINEAR FEATURE MAP

  • For any fixed feature dimension D, the explicit map induces a valid positive semi-definite kernel, and its random-feature view approximates a nonlinear kernel.The approximation statement follows from the law of large numbers; the unnormalized kernel differs by a constant factor D.Source: §B.2 · p. 17
  • With ReLU and Gaussian random directions, the limiting kernel corresponds to the first-order arc-cosine kernel; the fixed finite map remains kernelized for other fixed projections.R is randomly initialized once and kept fixed, introducing no trainable parameters during incremental learning.Source: §B.2 · pp. 17–18
  • VIS uses an explicit nonlinear feature map because it adds nonlinear decision boundaries while retaining a primal closed-form update from additive statistics.An implicit kernel would require maintaining or recomputing sample-level kernel matrices over observed data.Source: §B.2 · p. 18

C.1 EVALUATION ACROSS MULTIPLE RANDOM SEEDS

  • Across five class-order seeds, VIS maintains the best performance on ImageNet-R B0 Inc20.The reported mean accuracy and standard deviation cover seeds 1993 through 1997.Source: §C.1 · p. 18

C.2 RANDOM-FEATURE ROBUSTNESS

The supplementary analyses examine VIS’s robustness across random-feature seeds, parameter settings, backbones, forgetting, textual information, and training time. They report stable performance, backbone generalization, low forgetting, no consistent benefit from added text, and efficient training.

C.2 RANDOM-FEATURE ROBUSTNESS

  • VIS’s random-feature-seed standard deviations are at most 0.106 for average accuracy and 0.277 for final-session accuracy on Aircraft, and 0.073 and 0.167 on CIFAR100.The reported variations are substantially smaller than VIS’s margins over compared baselines.Source: §C.2 · p. 19, Table 2 · p. 19

C.3 PARAMETER ROBUSTNESS

  • Sensitivity results show stable performance trends across hyperparameter settings, using final-stage accuracy on a held-out validation subset.The original training data are split into training and validation subsets at a 4:1 ratio.Source: §C.3 · p. 19

C.4 RESULTS WITH DIFFERENT BACKBONES

  • VIS outperforms compared methods with OpenAI CLIP weights on UCF B0 Inc10, indicating its advantage is not tied to the main LAION-400M pre-training source.The main experiments use CLIP ViT-B/16 with LAION-400M weights.Source: §C.4 · p. 19

C.5 BACKBONE GENERALIZATION

  • VIS achieves the best performance with both DINOv2 and SigLIP; versus RanPAC, its average/final-session accuracy gains range from 0.30/0.72 to 6.33/9.09 points.Comparisons use the same backbone within each group under B0 Inc10 on Aircraft and CIFAR100.Source: Table 3 · p. 19, §C.5 · p. 20

C.6 FORGETTING ANALYSIS

  • VIS has the lowest forgetting in five of eight settings and remains close to the best method in the other three.The reported results associate recomputing classifiers for all seen classes from additive sufficient statistics with preserving old-class information.Source: §C.6 · p. 20, Table 4 · p. 20

C.7 CONTROLLED EVALUATION OF TEXTUAL INFORMATION

  • Adding textual prototypes decreases Aircraft average and final-session accuracy by 2.03 and 1.77 points, while CIFAR100 differences are marginal and mixed.VIS + Text and VIS differ only in whether textual class information is introduced.Source: §C.7 · p. 21, Table 5 · p. 21

C.8 TRAINING TIME

  • VIS’s later tasks require sufficient-statistic updates and a closed-form LS-SVM solve, avoiding repeated task-wise optimization and making it faster than RanPAC under the same epoch budget.Only lightweight visual modules are trained in the base task.Source: §C.8 · p. 21

C.9 OVERHEAD ANALYSIS

On Cars B0 Inc10, VIS combines the best average accuracy among compared methods with competitive efficiency. Its training and inference costs are low relative to many adaptation methods, while its GPU memory use is moderate.

C.9 OVERHEAD ANALYSIS

  • 94.84% average accuracy makes VIS the best-performing compared method on Cars B0 Inc10, with competitive overall efficiency.All methods use the same pre-trained CLIP backbone and are evaluated on a single RTX 3090 GPU.Source: §C.9 · p. 21
  • VIS completes continual training in 1.72 minutes, close to SimpleCIL at 1.50 minutes and ACIL at 1.60 minutes, and faster than the listed heavier optimization methods.Its training time is 0.55 minutes, versus 0.37 for SimpleCIL and 0.41 for ACIL.Source: §C.9 · p. 21
  • VIS inference takes 3.12 ms per image, close to CLG-CBM, SimpleCIL, ACIL, and RanPAC, and below BOFA, PROOF, DualPrompt, and CODA-Prompt.These results support the authors’ statement that VIS’s accuracy improvement does not introduce a large inference-time burden.Source: §C.9 · pp. 21–22
  • VIS uses 4.53 GB peak training GPU memory, more than several lightweight methods but less than RanPAC, RAPF, DualPrompt, and CODA-Prompt.The paper characterizes this as a moderate memory footprint among the compared methods.Source: §C.9 · p. 22

D FULL RESULTS

The full incremental curves show VIS maintaining strong performance across datasets and split protocols. It outperforms competing methods in most cases.

D FULL RESULTS

  • VIS maintains strong incremental performance and outperforms competing methods in most cases across datasets and split protocols.Figure 7 shows the zero-base setting; Figure 6 shows the half-base setting.Source: D FULL RESULTS · p. 22, Figure 7 · p. 23

E DETAILS OF COMPARED METHODS

The compared methods use varied visual, analytic, prompt-based, and multimodal approaches, evaluated with a shared pre-trained CLIP backbone. Under fixed visual features, LS-SVM compares favorably with Ridge and NCM; against related closed-form methods, VIS’s gains are attributed to its combined representation construction and classifier formulation.

E DETAILS OF COMPARED METHODS

  • The comparison spans frozen-visual prototype classification, analytic classifiers, prompt adaptation, and methods using visual-textual prototypes, with methods evaluated using the same pre-trained CLIP backbone.SimpleCIL uses visual prototypes; ACIL and RanPAC use analytic classification mechanisms; DualPrompt and CODA-Prompt are prompt-based methods.Source: E DETAILS OF COMPARED METHODS · pp. 23–24

F CLASSIFIER CHOICE UNDER FIXED FEATURES

  • With the feature pipeline fixed, Ridge and LS-SVM both outperform NCM, and LS-SVM matches or exceeds Ridge in most settings.Figure 8 compares nearest-class-mean, one-hot least-squares Ridge, and one-vs-all ±1 LS-SVM classifiers under B0 and B-half protocols.Source: Figure 8 · p. 24, F CLASSIFIER CHOICE UNDER FIXED FEATURES · p. 24
  • LS-SVM’s one-vs-all ±1 targets treat non-target classes as negative evidence, which stored feature-sum statistics let VIS incorporate for new classes without revisiting old samples.The closed-form kernelized classifier retains the analytic-update advantage of Ridge while using an SVM-style discriminative formulation.Source: F CLASSIFIER CHOICE UNDER FIXED FEATURES · p. 25

G COMPARISON WITH RELATED CLOSED-FORM CIL METHODS

  • VIS differs from ACIL and RanPAC by learning a residual correction from multi-level CLIP visual features using base-session data before fixed nonlinear feature expansion.ACIL and RanPAC instead increase classifier capacity through randomized feature expansion.Source: G COMPARISON WITH RELATED CLOSED-FORM… · p. 25
  • Under matched CLIP backbones and protocols, VIS outperforms ACIL and RanPAC in average and final-session accuracy across benchmark settings; the authors attribute gains to its combined visual representation construction and one-vs-all LS-SVM.VIS uses ±1 targets and maintains an additional feature-sum statistic to account for historical samples as negative targets for new classes.Source: G COMPARISON WITH RELATED CLOSED-FORM… · p. 25

H SUPPLEMENTARY PRELIMINARY EXPERIMENTS

Supplementary experiments examine modality gaps and compare the optimization behavior of textual and image-prototype classifier initialization. Across the reported analyses, visual prototypes are associated with more favorable optimization behavior and greater benefit for classes with larger image-text gaps.

H SUPPLEMENTARY PRELIMINARY EXPERIMENTS

Loading 2609.37888v1…