PaperlayerView exact revision on arXiv

Source-linked AI summary

Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning

Deboleena Roy, Priyadarshini Panda, Kaushik Roy

arXiv:1802.05800v3cs.CVcs.AIeess.IVstat.ML

TL;DR

Incrementally arriving data makes static DCNNs difficult to update because retraining on new data can destroy earlier features. Tree-CNN uses a feature-driven, hierarchically growing network to add classes while reusing prior structure. It reduces training effort relative to fine-tuning while maintaining competitive accuracy on CIFAR-10 and CIFAR-100.

  • Problem

    Real-world data arrives over time, while static DCNNs can suffer catastrophic forgetting when retrained on new data without previous data.

  • Method

    Tree-CNN grows a hierarchy of CNNs by adding new classes as leaves, grouping them into feature-driven super-classes, and localizing changes to parts of the network.

  • Results

    Tree-CNN offers the most optimal trade-off between training effort and testing accuracy, with comparable accuracy to selected baselines and lower training effort than fine-tuning.

  • Takeaways & Limitations

    Tree-CNN supports incremental class learning by reusing previously learned features while preserving much of the accuracy with lower training effort.

  • Takeaways & Limitations

    Tree-CNN continues to grow in size over time, so the implications for memory requirements require further investigation.

Abstract

from arXiv · show

Over the past decade, Deep Convolutional Neural Networks (DCNNs) have shown remarkable performance in most computer vision tasks. These tasks traditionally use a fixed dataset, and the model, once trained, is deployed as is. Adding new information to such a model presents a challenge due to complex training issues, such as "catastrophic forgetting", and sensitivity to hyper-parameter tuning. However, in this modern world, data is constantly evolving, and our deep learning models are required to adapt to these changes. In this paper, we propose an adaptive hierarchical network structure composed of DCNNs that can grow and learn as new data becomes available. The network grows in a tree-like fashion to accommodate new classes of data, while preserving the ability to distinguish the previously trained classes. The network organizes the incrementally available data into feature-driven super-classes and improves upon existing hierarchical CNN models by adding the capability of self-growth. The proposed hierarchical model, when compared against fine-tuning a deep network, achieves significant reduction of training effort, while maintaining competitive accuracy on CIFAR-10 and CIFAR-100.

1. Introduction

DCNNs are effective for image classification but are difficult to update as data arrives incrementally, particularly because retraining on new data can destroy previously learned features. Tree-CNN addresses this by hierarchically growing a CNN-based network as new classes appear.

  • Motivation: Data is gathered incrementally in real-world settings, creating a need for models that learn new information as it becomes available.Traditional DCNN training presents all labeled data during one training process.
  • Motivation: Retraining a DCNN exclusively on new data causes catastrophic forgetting by destroying features learned from earlier data.Using previous data during retraining is therefore required.
  • Proposed approach: Tree-CNN grows hierarchically by adding new classes as leaves, with branching based on feature similarity between new and old classes.Its initial nodes assign inputs to coarse super-classes.
  • Proposed approach: Tree-CNN performs coarse classification near the root and finer classification near the leaves.Previously learned convolutional layers can be reused in the expanded network.
  • Evaluation: The paper evaluates Tree-CNN on CIFAR-10 and CIFAR-100 and compares its performance with transfer learning and fine-tuning.The experiments and comparisons are described in Sections 4 and 5.

2. Related Work

Prior incremental-learning methods address new classes through transfer, preservation, gating, or sequentially added experts, while hierarchical CNNs exploit shared features for coarse-to-fine classification. Tree-CNN extends hierarchical CNNs with self-growth across multiple learning stages rather than requiring all classes and their properties initially.

  • Incremental learning: Incremental-learning methods include one-shot transfer learning, loss balancing with a frozen original network, Learning without Forgetting, expert gating, progressive networks, and iCaRL.These approaches differ in how they preserve prior capabilities or incorporate new tasks.
  • Hierarchical features: CNN layers learn generic features that support transfer learning and hierarchical classifiers.Hierarchical systems can group images using semantic or feature-driven similarities.
  • Hierarchical features: Existing hierarchical CNNs classify coarse super-classes with basic features and perform finer discrimination at deeper nodes.These models have been reported to perform at or better than standard DCNNs.
  • Limitation of prior work: Earlier hierarchical CNNs assume a fixed dataset and use prior knowledge of all classes and their properties to construct the hierarchy.This limits their direct fit to settings where classes arrive incrementally.
  • Tree-CNN: Tree-CNN starts from a single root and generates new hierarchies for incoming classes across multiple learning stages.The method localizes changes to a small network section while requiring older data during retraining.

3. Incremental Learning Model

Tree-CNN is a hierarchical network of CNN nodes that routes inputs from a root through branch nodes to uniquely labeled leaves. Its incremental procedure grows this tree using feature-based likelihoods, retraining only affected portions while retaining root-level training on all available data.

  • Network Architecture: Each non-leaf node uses a DCNN to route inputs among its children until a uniquely class-associated leaf node is reached.The root performs the first classification, and each selected child continues the process.
  • The Learning Algorithm: The algorithm averages output responses across sample images, computes a likelihood matrix, and orders new classes by their highest softmax likelihood before placement.For each new class, the ordered list records its top three average softmax values and corresponding output nodes.
  • The Learning Algorithm: Tree-CNN begins as a single root and expands with new classes by adding, merging, or attaching classes to child nodes according to softmax-likelihood thresholds.The growth process can also impose maximum children and depth constraints to prevent an unbalanced tree.
  • The Learning Algorithm: The procedure can recurse to deeper levels by applying the same growth process to child nodes containing new classes, under user-defined structural constraints.This makes the growth decision semi-supervised: the algorithm determines placement within specified limits.
  • The Learning Algorithm: After class locations are assigned, supervised gradient-descent training updates modified or new nodes rather than the whole network.The root is trained on all available data at each incremental stage, while branch nodes are trained only on their assigned classes.

Handling input labels inside the Tree-CNN

Tree-CNN maintains node-specific label-translation tables so classes remain consistently mapped as nodes are added or merged. Every class ultimately maps to a unique leaf, with merged labels transferred to the retained node.

  • Label Management: Each Tree-CNN node maintains a LabelsTransform lookup table to preserve class-label consistency during structural changes.The table records assignments when classes join existing outputs or become new child nodes.
  • Label Management: Every class is associated with a unique leaf node, ensuring that no two leaf nodes share the same class.
  • Label Management: When two nodes merge, the node with the higher average softmax value for the new class is retained and receives the lower-valued node’s labels.If the values are equal, the paper states that the choice is random.

4. The Experimental Setup

The experiments evaluate Tree-CNN on incremental CIFAR-10 and CIFAR-100 tasks, using hierarchical CNNs that assign new classes to branches and retrain affected nodes. CIFAR-100 further tests sequential class additions under different branch-size limits.

  • 4.1. The CIFAR-10 experiment: CIFAR-10 begins with six classes in a two-level Tree-CNN, grouping them into “Vehicles” and “Animals” before adding the remaining four classes.A root CNN performs coarse super-class classification, while branch CNNs classify the assigned leaf classes.
  • 4.1. The CIFAR-10 experiment: New CIFAR-10 classes are assigned to child branches using root-generated likelihoods; the branch with the higher likelihood adds each class.The experiment sets α to 0, allowing only this assignment action.
  • 4.1. The CIFAR-10 experiment: After assignment, the CIFAR-10 root is retrained on all 10 classes, while each affected branch is retrained on its assigned old and new classes.The animal branch uses six classes, whereas the vehicle branch uses four.
  • 4.2. The CIFAR-100 experiment: CIFAR-100 contains 100 classes divided into 10 groups of 10, which are introduced incrementally to a root node initially containing 10 leaf outputs.Later learning stages add branch nodes as classes become grouped under existing outputs.
  • 4.2. The CIFAR-100 experiment: CIFAR-100 tests maximum branch-child limits of 5, 10, and 20, producing Tree-CNN-5, Tree-CNN-10, and Tree-CNN-20.The tree depth is capped at 2, with α = 0.1 and β = 0.1; updated root and affected branch CNNs are retrained on combined old and new data.

4.3. Benchmarking

The benchmarking setup compares Tree-CNN with a similarly complex VGG-like baseline trained by several fine-tuning depths. Evaluation reports testing accuracy and a training-effort measure based on weight updates and samples processed.

  • Benchmark protocol: Because incremental learning lacks a standardized benchmark protocol, the experiments use a protocol similar to iCaRL.Classes are introduced in a fixed random order, and accuracy is evaluated on all classes learned at each stage.
  • Baseline: Network B is an 11-layer VGG-like baseline with four convolutional blocks and complexity comparable to a two-stage Tree-CNN.It serves as the comparison network for incremental fine-tuning.
  • Baseline: Network B adds new classes as final-layer output nodes and uses five fine-tuning strategies that back-propagate through progressively deeper layers.The deepest strategy retrains the fully connected layer and all four convolutional blocks.
  • Evaluation metrics: The comparison uses testing accuracy and Training Effort, which estimates computation from model weights and training samples per learning stage.Tree-CNN sums effort across its nodes, while Network B computes it for the single baseline network.
  • Training configuration: All networks use image augmentation, whitening, contrast normalization, ReLU activations, mini-batch stochastic gradient descent, dropout, batch normalization, and weight decay.The weight-decay coefficient is λ = 0.001.

5. Results

Tree-CNN achieves a favorable accuracy–training-effort trade-off across incremental CIFAR-10 and CIFAR-100 experiments, while remaining competitive with stronger retraining baselines. Its hierarchical structure also shows feature- and sometimes semantically related class groupings.

  • CIFAR-10: 78.37% accuracy is achieved by B:I, which requires the least training effort but degrades substantially as classes are added.B:I retrains only the final fully connected layers.
  • CIFAR-10: ∼40% less normalized training effort than B:V and ∼30% less than B:II are required by Tree-CNN, with accuracy within 3.76% of B:V.Tree-CNN has comparable accuracy to B:II and B:III.
  • Tree growth: 1e-2 is the margin within which the three Tree-CNN variants have nearly identical training effort across nine incremental stages.Tree-CNN-5, Tree-CNN-10, and Tree-CNN-20 reach 3.4×, 2.2×, and 1.8× Network B size, respectively.
  • CIFAR-100: Tree-CNNs perform almost at par with B:IV and outperform the other Network B variants across sequential CIFAR-100 learning stages.The paper characterizes this as the most optimal trade-off between training effort and testing accuracy.
  • CIFAR-100: 10% higher accuracy than iCaRL and over 50% higher accuracy than LwF are reported for Tree-CNNs on the compared CIFAR-100 incremental-learning evaluation.iCaRL uses 2000 old-data exemplars, while LwF retrains exclusively on new data.
  • Hierarchical organization: Similar-looking and often semantically similar classes are grouped under common branches, although some groupings are unrelated.The grouping examples are shown for Tree-CNN-10 after nine incremental stages.

6. Discussion

The discussion presents Tree-CNN as a way to reduce retraining costs while preserving accuracy, but identifies model growth and class-grouping behavior as open concerns.

  • Merits: Tree-CNN confines changes to a few nodes, reducing retraining computation while using previous data.The discussion links this design to lower training effort than fine-tuning while preserving much of the accuracy.
  • Limitations: Tree-CNN continues to grow in size over time, so the implications for memory requirements remain to be investigated.Per-node inference memory is described as much lower than the entire model size.
  • Future work: The relationship between semantic label similarity and feature similarity within branches is identified as an area for further exploration.The model can generate hierarchical groupings of initially unrelated classes.

Appendix A. Incremental CIFAR-100 Dataset

The CIFAR-100 incremental dataset is formed by randomly arranging the 100 classes into ten sequential batches of ten classes each.

  • Dataset construction: 100 CIFAR-100 classes are randomly shuffled and divided into 10 groups, each containing 10 classes.The groups are used in the order listed for incremental learning.
  • Batch contents: Batch 0 contains chair, bridge, girl, kangaroo, lawn mower, possum, otter, poppy, sweet pepper, and bicycle.
  • Batch contents: Batches 1–5 list distinct ten-class groups that are sequentially introduced during the incremental task.The supplied passage enumerates classes for batches 1 through 5.

Appendix B. Final Tree-CNN for max children 5, 10, 20 (CIFAR-100)

Appendix B shows the final two-level Tree-CNN structures after nine incremental learning stages for maxChildren values of 5, 10, and 20.

  • Diagram legend: Yellow nodes indicate completely filled branches, blue nodes partially filled branches, and green nodes leaves in the final diagrams.
  • Final structures: The appendix figures depict Tree-CNN-5, Tree-CNN-10, and Tree-CNN-20 after nine incremental learning stages.

Appendix C. Full Simulation Results

Appendix C reports full simulation results for incremental class addition on CIFAR-100. The results cover both normalized training effort and test accuracy as classes are added in batches of 10.

  • Table C.8 reports normalized training effort as CIFAR-100 classes are added incrementally in batches of 10.
  • Together, the tables characterize training effort and test accuracy across incremental additions of 10 classes.
  • Table C.9 reports test accuracy under the same incremental CIFAR-100 class-addition schedule.
Loading 1802.05800v3…