Source-linked AI summary
Adaptive Aggregation Networks for Class-Incremental Learning
Yaoyao Liu, Bernt Schiele, Qianru Sun
TL;DR
CIL must learn new classes without losing old-class knowledge, but limited exemplar memory creates a stability–plasticity dilemma. AANets address it with stable and plastic residual blocks and adaptive aggregation weights optimized through a bilevel procedure. Across three benchmarks, the architecture consistently improves several existing CIL methods and remains effective under strict memory control.
Problem
CIL faces a stability–plasticity dilemma because higher plasticity forgets old classes while higher stability weakens learning of new classes under limited exemplar memory.
Method
AANets use separate stable and plastic residual blocks at each residual level and adapt their aggregation weights end-to-end with bilevel optimization using different data splits.
Results
AANets consistently improve four existing CIL methods across CIFAR-100, ImageNet-Subset, and ImageNet, while strict-memory experiments show only a 0.3% drop for 5-Phase CIL models.
Takeaways & Limitations
The generic AANets architecture can be incorporated into existing CIL methods to boost performance while balancing stable and plastic feature learning.
Takeaways & Limitations
Aggregation weights vary by residual level because knowledge transferability differs across network levels, and the initial model is trained on half of the total classes before being frozen.
Abstract
from arXiv · showhide
Class-Incremental Learning (CIL) aims to learn a classification model with the number of classes increasing phase-by-phase. An inherent problem in CIL is the stability-plasticity dilemma between the learning of old and new classes, i.e., high-plasticity models easily forget old classes, but high-stability models are weak to learn new classes. We alleviate this issue by proposing a novel network architecture called Adaptive Aggregation Networks (AANets), in which we explicitly build two types of residual blocks at each residual level (taking ResNet as the baseline architecture): a stable block and a plastic block. We aggregate the output feature maps from these two blocks and then feed the results to the next-level blocks. We adapt the aggregation weights in order to balance these two types of blocks, i.e., to balance stability and plasticity, dynamically. We conduct extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Subset, and ImageNet, and show that many existing CIL methods can be straightforwardly incorporated into the architecture of AANets to boost their performances.
1. Introduction
CIL must learn new classes while retaining old-class knowledge under limited replay, creating a stability–plasticity dilemma. AANets address this with separate stable and plastic residual blocks whose aggregation weights are adapted through bilevel optimization.
- Motivation: CIL presents new classes phase-by-phase and evaluates recognition across both old and new classes under a strict exemplar memory budget.The budget creates imbalance between old and new class data.
- Motivation: Higher plasticity can increase forgetting of old classes, whereas higher stability can weaken learning of new classes.Existing methods primarily balance this dilemma through data strategies such as imbalanced training and balanced fine-tuning.
- AANets: AANets use separate stable and plastic residual blocks at each residual level, with different learnability, and aggregate their output feature maps.The aggregated maps are passed to the next residual level.
- AANets: AANets adapt aggregation weights end-to-end to dynamically balance the usage of stable and plastic blocks.The weights are treated as hyperparameters and optimized alongside the network through a bilevel procedure.
- Optimization: The bilevel formulation learns block parameters from all phase data and adapts aggregation weights using a balanced subset formed by downsampling new-class data.The two optimization problems alternate while holding the other parameter group fixed.
- Contributions: AANets provide a generic architecture, training formulation, and integration strategy evaluated by incorporating four baseline CIL methods.The contribution is positioned as a modification of ResNet-like architectures for the stability–plasticity dilemma.
2. Related Work
Related work spans task-based and class-based incremental learning, with CIL focusing on new classes arriving from the same dataset. Existing approaches use regularization, data strategies, and bilevel optimization to retain prior knowledge.
- Incremental learning: Incremental learning approaches are commonly task-based, with new datasets per phase, or class-based, with new classes from the same dataset.The class-based setting is typically called class-incremental learning.
- Regularization-based methods: Regularization-based methods add loss terms such as knowledge distillation, less-forgetting constraints, and inter-class separation to mitigate imbalance effects.These methods aim to consolidate previous knowledge while learning new data.
- Bilevel optimization: Bilevel optimization updates network parameters at one level and selected hyperparameters at another.Prior incremental-learning work applied this framework to bias correction, generic-model retention, or alignment of updates with previous phases.
- Adaptive aggregation: AANets use a different data strategy from conventional CIL methods: all available data update stable and plastic blocks, while balanced exemplars adapt their aggregation weights.The architecture is designed to balance stability and plasticity through the learned aggregation.
3. Adaptive Aggregation Networks (AANets)
AANets address CIL’s forgetting problem with parallel stable and plastic residual blocks whose outputs are adaptively aggregated at each level. Their bilevel training separately learns block parameters and aggregation weights using data splits suited to feature learning and balance control.
- Architecture Details: AANets use parallel stable and plastic residual blocks at every residual level to preserve old-class knowledge while adapting to new classes.Stable blocks partially fix parameters, while plastic blocks fully adapt to new-class data.
- Architecture Details: At each level, feature maps from the two blocks are weighted and aggregated before being passed to the next level.The final-level outputs are used to train classifiers.
- Architecture Details: Stable blocks retain a frozen base model while learning neuron-level scaling weights, reducing their learnable parameters relative to fully adaptable plastic blocks.The base model is learned in the initial phase and frozen thereafter; scaling weights modify stable-block neurons in later phases.
- Architecture Details: Aggregation weights are adapted separately at different residual levels because transferred old-class knowledge differs across network levels.The architecture automatically learns these weights across phases rather than requiring manual selection.
- Optimization Steps: AANets formulate training as bilevel optimization: network parameters use available training data, while aggregation weights use balanced exemplar data.The lower level trains stable-block scaling and plastic-block convolutional parameters; the upper level adapts aggregation weights.
- Optimization Steps: The training procedure alternates updates of aggregation weights and network parameters across epochs, then updates and stores new-class exemplars.The algorithm initializes phase parameters from the preceding phase and alternates upper- and lower-level optimization.
4. Experiments
Experiments evaluate AANets on CIFAR-100, ImageNet-Subset, and ImageNet, using ResNet-based architectures, ablations, plug-in comparisons, memory controls, and feature visualizations. AANets consistently improve incorporated CIL methods, while adapted aggregation weights favor stability at lower levels and plasticity near classifiers.
- Datasets and Architectures: AANets are evaluated on CIFAR-100, ImageNet-Subset, and ImageNet with ResNet-based architectures and unchanged classifier layers.CIFAR-100 uses a 32-layer ResNet, while ImageNet benchmarks use an 18-layer ResNet.
- Ablation Settings: Table 1 compares eight ablations spanning LUCIR, double-branch baselines, block combinations, adaptive versus fixed aggregation, and strict memory budgets.The block notation A+B assigns the first type to plastic blocks and the second to stable blocks.
- Memory Control: 26% and 14.5% memory overheads are reported on CIFAR-100 and ImageNet-Subset, respectively, while strict-budget AANets lose only 0.3% on 5-phase models.The strict setting reduces exemplar counts to compensate for additional model parameters.
- Comparisons to State of the Art: 4.9% and 3.3% average improvements are reported for LUCIR and Mnemonics with AANets on CIFAR-100, with consistent gains across four baseline methods.The paper reports top performances across all settings and stronger boosts for simpler baselines such as iCaRL.
- Aggregation Weights and Visualization: Lower residual levels tend to receive larger αφ values, whereas Level 3 tends to receive larger αη values across incremental phases.The authors associate lower-level features with more stable, shareable representations and higher-level features with faster adaptation to new classes.
5. Conclusions
AANets address the stability-plasticity dilemma in CIL by separating stable and plastic learning within each residual level and adaptively aggregating their features. The approach is presented as a generic ResNet modification that can be incorporated into existing CIL methods.
- Conclusion: AANets use stable and plastic residual blocks at each level to learn old-class stability and new-class plasticity separately.Their outputs are aggregated into the final representation using level-specific and phase-specific weights adapted end-to-end.
- Conclusion: The architecture is generic and can be incorporated into existing CIL methods to boost their performance.
A. Results for Different CIL Settings.
Under equal-class-per-phase CIL settings, AANets consistently perform best across the reported configurations. The supplementary table reports last-phase accuracy averaged over three runs.
- “w/ AANets” performs best in all reported equal-class-per-phase settings and brings consistent improvements.These results are reported under strict memory-budget and “all”+“scaling” settings.
- The supplementary table reports last-phase accuracies (%) for different CIL settings.Each entry is averaged over three runs.
- Updating θbase helps CIFAR-100 but harms ImageNet-Subset in these settings.
B. Strict Memory Budget Experiments
Strict memory-budget experiments account for AANets’ additional parameters by reducing exemplar counts. Despite this trade-off, AANets remain competitive across the evaluated plug-in settings.
- AANets are evaluated as a plug-in architecture for four state-of-the-art methods under a strict shared memory budget.The budget covers both exemplars and model parameters.
- Adding AANets requires fewer exemplars because the architecture uses around 20% more parameters than the plain ResNets used in LUCIR.
- The added parameter memory corresponds to reducing exemplars by 7 images per class for CIFAR-100, 4 for ImageNet-Subset, and 1 for ImageNet.
- Even after these exemplar reductions, the AANets approach achieves top performance in the reported strict-memory experiments.
C. More Ablation Results
Additional ablations compare heterogeneous stability–plasticity blocks with repeated same-type blocks. The comparisons assess the efficiency of using distinct block types.
- Using different types of residual blocks is more efficient than using four same-type blocks at each residual level.“4×” denotes four same-type blocks.
- The ablation comparison is made between Row 7 and Row 2 or Row 5 across additional settings.
D. Additional Plots
Supplementary phase-wise plots evaluate AANets on CIFAR-100, ImageNet-Subset, and ImageNet. AANets achieve the highest accuracies in almost every phase across the reported settings.
- AANets achieve the highest phase-wise accuracies in almost every phase across CIFAR-100, ImageNet-Subset, and ImageNet.
- The plots include an Upper Bound based on joint training with all previous data accessible in every phase.
- Additional plots show learned values of αη and αφ on CIFAR-100 and ImageNet-Subset for N=5 and N=25.The curves are smoothed for visualization.
E. More Visualization Results
Figure S1 visualizes activation maps for a goldfinch sample seen in Phase 0 across different-phase models on ImageNet-Subset with N=5.
- Figure S1 shows activation maps for a “goldfinch” sample.
- The goldfinch sample was seen in Phase 0.
- The visualization compares models from different phases on ImageNet-Subset with N=5.
F. Source Code in PyTorch
The supplementary materials identify comparative figures and tables for incremental accuracies, aggregation weights, and ablations, while the paper provides PyTorch code with specified environment versions.
- PyTorch code is provided through the project repository, with Python 3.6 and PyTorch 1.2.0 advised for running it.
- Supplementary Table S2 reports average incremental accuracies for four methods with and without AANets under a strict memory budget.
- Supplementary Table S3 extends the ablation study and uses “4×” to denote four same-type blocks at each residual level.
- Supplementary Figure S2 presents phase-wise accuracies with 95% confidence intervals and compares Upper Bound with six CIL methods.
- Supplementary Figures S3 and S4 show changes in αη and αφ on CIFAR-100 and ImageNet-Subset, respectively.