Source-linked AI summary
Knowing Beyond the Known: Reinforced Knowledge Specification for Multi-Label Class-Incremental Learning
Aoting Zhang, Dongbao Yang, Chang Liu, Xiaopeng Hong, Can Ma, Yu Zhou
TL;DR
Multi-label class-incremental learning must preserve historical knowledge, learn current classes, and accommodate prospective classes despite incomplete labels and co-occurring objects. KBK explicitly specifies known and unknown knowledge through complementary representation, recall, unknown-feature synthesis, and gradient-balancing mechanisms, with experiments and ablations confirming its effectiveness and robustness.
Problem
Multi-label class-incremental learning faces conflicting objectives because images can contain historical, current, and future classes while supervision covers only current classes, hindering knowledge retention and future learning.
Method
KBK specifies known and unknown knowledge using hierarchical feature purification, uncertainty-aware historical recall, semantic synthesis of unknown features, and category-balanced gradient compensation.
Results
Experiments and ablation studies confirm the effectiveness and robustness of the KBK framework.
Takeaways & Limitations
KBK provides complementary mechanisms for separating class-aware representations, restoring reliable historical supervision, protecting unknown information, and balancing heterogeneous forgetting in MLCIL.
Takeaways & Limitations
Experiments mainly use PASCAL VOC and MS-COCO, which do not fully represent long-tailed distributions, open-world category evolution, or domain shifts.
Abstract
from arXiv · showhide
Existing class-incremental learning methods struggle in multi-label scenarios (MLCIL) due to the inherent contradiction of learning objectives arising from co-occurring and incomplete labels. We argue that the core obstacle is the model's ambiguous boundary between known and unknown knowledge, which undermines historical knowledge retention, complicates current task learning, and limits adaptability to future concepts. To address this, we propose KBK (Knowing Beyond the Known), a reinforced knowledge specification framework that explicitly models what is known or not to unify historical, current, and prospective learning. Specifically, to clarify known knowledge, we develop a hierarchical feature purification module that disentangles fine-grained class-specific features from global features, where high-level semantic abstraction is reinforced with low-level visual features. Additionally, an uncertainty-aware recall enhancement strategy suppresses unreliable predictions based on distribution priors, improving the quality of historical recall. For probing the unknown, KBK leverages semantic correlations to synthesize informative unknown features under co-occurring, preserving embedding space for future learning. Furthermore, to mitigate heterogeneous forgetting, we design a category-balanced gradient compensation loss that dynamically reweights gradient backpropagation according to forgetting speeds. Experiments on multiple benchmarks validate the effectiveness and robustness of KBK, which surpasses prior best methods by 2.7% in Avg. Acc on MS-COCO B0-C10 setting even without any replay buffers.
1. Introduction
MLCIL must preserve historical knowledge, learn current classes, and remain prepared for future classes despite partial labels and co-occurring objects. KBK addresses these conflicts by explicitly specifying known and unknown knowledge through complementary representation, recall, unknown-probing, and gradient-compensation mechanisms.
- Problem: MLCIL creates a three-way tension among retaining historical knowledge, learning current classes, and preparing for future classes under partial labeling.Unlabeled historical and prospective objects may co-occur with annotated current classes, making standard single-label remedies ineffective.
- Clarifying Known Knowledge: Hierarchical feature purification extracts fine-grained class-aware representations by reinforcing high-level semantic features with intermediate visual cues.Class embeddings attend to relevant regions, supporting discrimination among co-occurring objects and extension to new classes.
- Clarifying Known Knowledge: Uncertainty-aware recall enhancement filters low-confidence and high-uncertainty historical predictions using distribution priors to improve recalled supervision.The strategy addresses unreliable historical recall caused by partial labeling.
- Framework: KBK unifies historical, current, and prospective learning by explicitly specifying reliable known knowledge and uncertain unknown content.The framework is designed to coordinate these knowledge types across incremental sessions.
- Probing Unknown Knowledge: Semantic-guided probing synthesizes unknown features from absent-class responses, while category-balanced gradient compensation reweights class-wise optimization signals according to forgetting tendencies.Together, these mechanisms reserve embedding space for prospective concepts and address heterogeneous forgetting.
- Results: KBK attains superior performance across diverse protocols while diminishing catastrophic forgetting.The manuscript reports broader evaluation and technical extensions over prior conference work.
2. Related Work
Related work covers continual classification, multi-label representation learning, MLCIL, and related incremental vision tasks. These approaches address forgetting, label correlations, or task-specific incremental supervision, whereas MLCIL requires image-level learning with partial labels and co-occurring historical and prospective classes.
- Single-Label Class-Incremental Learning: Single-label class-incremental methods mitigate forgetting through regularization, knowledge distillation, and replay when former data are unavailable.Examples include Fisher-information parameter regularization and preservation of previous model behaviors.
- Multi-Label Classification: Static multi-label classification methods model inter-label correlations and class-aware representations but generally assume fixed label spaces and jointly observed training data.Approaches include sequence, graph, transformer, latent-context, semantic-guidance, and causal-correlation methods.
- Related Vision Tasks: MLCIL therefore requires tailored solutions because historical and prospective classes may appear without annotations in current-session images.Directly extending static multi-label methods can suffer from catastrophic forgetting, feature aliasing, and interference from unlabeled co-occurring objects.
- Multi-Label Class-Incremental Learning: MLCIL methods address replay imbalance, exemplar maintenance, label relations, and label absence, but the setting still involves conflict among retention, current-class discrimination, and future-class preparedness.These conflicts arise because only a subset of labels is annotated while multiple objects may co-occur.
- Related Vision Tasks: Incremental object detection and semantic segmentation use task-specific architectures and supervision such as bounding boxes or pixel-level annotations.Their supervision requirements prevent direct application to image-level MLCIL.
3. Proposed Method
KBK formulates MLCIL around a unified distinction between historical/current known knowledge and prospective unknown content, then coordinates representation purification, uncertainty-aware recall, unknown-feature synthesis, and gradient compensation. Its modules jointly support reliable retention, current-class discrimination, and future-class preparedness.
- Problem Formulation: MLCIL partitions classes into historical, current, and prospective knowledge, while session training observes only the current label space.Images may also contain past and future classes whose labels are unavailable during the current session.
- Framework Overview: KBK uses a unified known/unknown specification principle to coordinate historical retention, current learning, and prospective preparedness.Known knowledge comprises historical and current classes; prospective content remains outside the observed label space.
- Clarifying Known Knowledge: Hierarchical feature purification fuses intermediate spatial and deep semantic features, then uses class embeddings to extract fine-grained class-specific representations.Class embeddings query fused feature tokens, producing purified class features and enhanced patch features for parallel old- and new-class prediction.
- Clarifying Known Knowledge: Uncertainty-aware recall combines class-specific confidence thresholds with entropy filtering to suppress unreliable historical pseudo-labels.Thresholds are derived from historical confidence and entropy distributions, and updated after each session to account for distribution drift.
- Probing Unknown Knowledge: Semantic-guided unknown-feature synthesis uses co-occurring absent-class information to enrich the representation space and reserve capacity for future categories.The synthesized unknown branch acts as a coarse known/unknown boundary regularizer rather than constructing exact prototypes for every future class.
4. Experiments
KBK is evaluated across standard and split MLCIL protocols on MS-COCO and PASCAL VOC, using accuracy, mAP, and F1-based metrics. Results show strong buffer-free performance, robustness across datasets and session constructions, and complementary gains from its components.
- Experimental setup: KBK is evaluated on MS-COCO and PASCAL VOC under overlapping-image and non-overlapping split protocols, retaining partial supervision.Protocol A includes standard VOC/COCO settings, while Protocol B uses Split-VOC/Split-COCO with disjoint image splits.
- Evaluation metrics: Evaluation reports average and final-session mAP alongside per-class and overall F1 scores to measure continual multi-label performance.Average mAP summarizes performance across sessions, while last mAP measures the final session.
- Main results: 75.8% Last Acc: KBK substantially outperforms fine-tuning and PODNet on MS-COCO B0-C10 with zero buffer.Fine-tuning reaches 16.9% and PODNet 25.6% in the same comparison.
- Main results: 94.1% Avg. Acc: KBK exceeds CSC and HCP on VOC B0-C4, while achieving 90.9% under the longer B10-C2 sequence.Under VOC B4-C2, KBK improves over HCP by 2.6%; with two samples per class, it reaches 92.8% and 92.5% Avg. Acc in B10-C2 and B5-C3.
- Main results: 4.4% and 6.7%: KBK improves Avg. Acc over APPLE on Split-COCO B10-C5 and Split-VOC B0-C5, respectively.With no replay buffer, KBK matches or exceeds many replay-based methods and substantially outperforms simple fine-tuning.
- Ablation and analysis: Ablations show progressive gains from HFP, URE, SPU, and CGC, while SPU removal reduces later-session performance and KBK adds only moderate computational overhead.HFP and URE improve accuracy and recall, SPU separates prospective information from known-class regions, and CGC balances retention with current-class optimization.
- Ablation and analysis: KBK remains effective under strict storage constraints: increasing the buffer from 0 to 50 produces only a modest average-mAP improvement.Attention analyses also indicate that HFP captures class-specific local features despite co-occurring similar classes.
5. Limitations
KBK improves standard MLCIL benchmarks, but its evaluation and unknown modeling remain bounded by several practical limitations.
- Experiments mainly use PASCAL VOC and MS-COCO, limiting representation of long-tailed distributions, open-world category evolution, and domain shifts.
- SPU uses one aggregated unknown class, so it cannot capture fine-grained taxonomy among unseen categories.
- Synthesized unknown features may be less informative when absent-class features weakly correlate with the scene or are dominated by background responses.
- URE may still admit over-confident incorrect predictions under severe ambiguity or distribution shift.
- Extending KBK to incremental object detection and semantic segmentation remains future work.
6. Conclusion
KBK specifies known and unknown knowledge to coordinate historical, current, and prospective information in multi-label class-incremental learning. Its experiments and ablations support effectiveness and robustness, while broader visual-task extensions remain future work.
- KBK explicitly specifies known and unknown knowledge across sessions to accommodate historical, current, and prospective information.
- Hierarchical feature purification disentangles fine-grained class-aware features from fused semantic and visual information, reducing feature aliasing within and across sessions.
- Uncertainty-aware recall enhancement suppresses low-confidence, high-uncertainty predictions to strengthen historical supervision under partial labeling.
- Semantic relations generate informative unknown features as prospective classes, enhancing inter-class discrimination and reserving representational room for future additions.
- Category-balanced gradient compensation loss adaptively rescales gradient contributions according to differing forgetting velocities among classes.
- Experiments and ablation studies confirm KBK's effectiveness and robustness.
- Extending knowledge specification to incremental object detection and semantic segmentation remains future work.