PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing

Syed Shakib Sarwar, Aayush Ankit, Kaushik Roy

arXiv:1712.02719v4cs.CV

TL;DR

Retraining large DCNNs for new classes can demand substantial time and energy, and old training samples may no longer be available. The paper uses partial sharing and clone-and-branch training to update networks while retaining old-task performance. It reports comparable accuracy alongside reductions in training and resource costs.

  • Problem

    Retraining large DCNNs for new classes is costly, while previously seen training samples may be unavailable for the update.

    Source: §1 · pp. 1–2

  • Method

    The method grows a DCNN by sharing learned convolutional layers with new branches, using clone-and-branch training for successive tasks.

    Source: §1 · p. 3, §6 · p. 16

  • Results

    The method achieves comparable task-specific accuracy and combined-classification accuracy close to the upper bound without old training samples, while reducing training and resource costs.

    Source: §1 · p. 4, §6 · p. 16

  • Takeaways & Limitations

    Partial sharing lets new networks inherit base-network features and reduces training effort without retraining the whole network on old samples.

    Source: §6 · p. 16

    The approach assumes that new classes have features similar to the old classes.

    Source: §3 · p. 5

Abstract

from arXiv · show

Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification. However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing DCNN to learn new tasks while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned tasks. An updated network for learning new set of classes is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We employed a `clone-and-branch' technique which allows the network to learn new tasks one after another without any performance loss in old tasks. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing), while achieving energy efficiency, reduction in storage requirements, memory access and training time.

1 Introduction

The paper addresses incremental DCNN learning when old training data may be unavailable and full-network retraining is costly. It proposes partial network sharing and clone-and-branch training to learn new classes while preserving old-task performance and reducing update costs.

1 Introduction

  • Incremental DCNN updates are motivated by costly full-network retraining and cases where old samples are proprietary, cumbersome to reuse, or discarded for privacy.The paper frames the challenge as reusing prior learning while increasing network capacity efficiently when needed.Source: §1 · pp. 1–2
  • Prior approaches include few-shot or shallow models, ensembles, and methods that retrain whole networks or reuse old samples, leaving update cost a concern.The related-work discussion identifies limits involving repeated access to old data, inability to handle new classes, and computational overhead.Source: §1 · pp. 2–3
  • Clone-and-branch shares learned layers to initialize new branches, aiming to learn successive tasks without performance loss on old tasks.The authors say cloning gives a better starting point than random initialization and supports faster convergence and fine-tuning.Source: §1 · p. 3
  • The paper identifies hardware and energy requirements for model updates as an under-addressed concern alongside transferring knowledge between tasks.It notes that the approach requires additional memory for supplementary parameters and describes FPGAs as especially suited to its flexible architecture.Source: §1 · p. 3
  • Partial convolutional-layer sharing is proposed to reduce training complexity while accommodating new tasks without forgetting old ones.The contributions also include a method for selecting sharing levels and a training-cost model based on MAC operations and memory accesses.Source: §1 · p. 3
  • The method targets energy efficiency, storage, memory access, and training-time reductions while maintaining classification accuracy without old-task training samples.The authors report scalability across different network structures and benchmark datasets.Source: §1 · p. 4

2 Incremental Learning

Incremental learning adds newly available labeled classes over time while retaining previously acquired knowledge. The paper presents advantages that include edge-device updates, faster model updates, privacy, and reduced storage needs.

2 Incremental Learning

  • Incremental learning adds batches of labeled data for new classes over time and should grow capacity, limit update overhead, avoid old samples, and preserve prior knowledge.The paper relates forgetting in neural networks to the stability-plasticity dilemma.Source: §2 · p. 4
  • The worked example updates a four-class base network with two new classes using only the new task's data, while retaining the ability to classify both tasks.The paper distinguishes task-specific classification from combined classification and primarily focuses on the former.Source: §2 · p. 4

2.1.1 Enable training in low power devices

  • Low-power devices can update an existing network within power and hardware limits by reusing learned knowledge without old-task samples.The passage contrasts this with off-chip deep-network training and devices that cannot retrain the whole network.Source: §2.1.1 · p. 4

2.1.2 Speed up model update

  • Reusing an existing network for new-task samples without forgetting old tasks is described as making updates very fast.Source: §2.1.2 · p. 4

2.1.3 Ensure data privacy

  • Because incremental learning does not require old training data, samples can be discarded after each session to disallow misuse of private data.Source: §2.1.3 · p. 4

2.1.4 Reduce storage requirements

  • Not storing training samples reduces the storage required to update a network.Source: §2.1.4 · p. 5

3 Design Approach

The approach incrementally adds task-specific branches while freezing and reusing learned convolutional layers, under the assumption that new classes have features similar to old classes. It develops three sharing strategies, from a tuned fixed configuration to task-specific and branch-to-branch sharing.

3 Design Approach

  • The method reuses frozen convolutional layers as fixed feature extractors, assuming new classes have features similar to previously learned classes.Only newly available data is used when learning new classes.Source: §3 · p. 5

3.1 Replacing Part of the Base Network with New Convolutional Layers

  • 8-12% accuracy degradation versus same-depth networks without sharing, and about 10% degradation in all-class updated accuracy, exposed limits of training only one convolutional layer and classifier layers.The approach cloned later layers, trained them on new-class examples, and added each retrained branch without changing shared or old task-specific parameters.Source: §3.1 · p. 6

3.2 Training Methodology 1

  • A core-set base network and demo-set branch trials produce an accuracy-versus-sharing curve that guides a quality-threshold configuration for later tasks.Selecting the configuration incurs overhead, but heuristics can narrow the search to a few retraining iterations; shared layers are not backpropagated during branch training.Source: §3.2 · p. 7

3.3 Training Methodology 2

  • Task-specific sharing estimates similarity by repeatedly forwarding new-task examples through the base network and counting repeated predicted classes.The authors describe this as a simple, potentially non-ideal measure; tasks more similar to the base can share more features, subject to the base network's estimated sharing capacity.Source: §3.3 · pp. 8–9

3.4 Training Methodology 3

  • Fine-grained sharing lets a new task branch from the most similar trained branch, but the resulting network can depend on task order.The example assigns tasks 2 and 3 to task 1; changing the order in which tasks arrive produces a different updated network.Source: §3.4 · p. 10

3.5 Comparison of Different Training Methodologies

  • Method 2 reduced average sharing while increasing task-specific and combined accuracy; Method 3 slightly improved combined accuracy with more sharing while maintaining task-specific performance.Table 2 compares the different training methodologies.Source: Table 2 · p. 10, §3.5 · p. 10

4 Evaluation Methodology

The evaluation isolates algorithmic benefits from hardware-specific features by measuring compute and storage energy on a modeled baseline. The authors also assess accuracy with software tools and GPU-based training on independent validation data.

Evaluation Methodology

  • Reducing modified kernels cuts backward-pass layer-gradient and weight-gradient computations, producing energy benefits attributed to algorithmic optimization rather than a particular microarchitecture.The analysis focuses on compute and storage energy within cores, excluding off-chip and inter-core data movement.Source: §4 · p. 11
  • The modeled baseline uses many-core execution with MVM units, RTL-implemented MACs, and CACTI-modeled memory, while excluding off-chip and inter-core data movement.The MAC implementation and memory model use IBM 32nm technology.Source: §4 · p. 11
  • Accuracy is evaluated with neural-network software tools and NVIDIA GPUs, using independent validation data and no previously seen samples in later learning stages.The tools include MatConvNet and PyTorch.Source: §4 · p. 11
  • Figure 10 compares energy-accuracy trade-offs and training-time requirements for incremental training with and without convolutional-layer sharing across configurations.Source: Figure 10 · p. 11

5 Results and Discussions

Across the reported comparisons, partial network sharing performs nearly as well as incremental training without sharing and approaches cumulative training, while reducing training energy, time, storage, and memory access. Results also show that sharing benefits and accuracy depend on network architecture and the chosen sharing configuration.

5.1 Energy-Accuracy Trade-off

  • 2.45x computation energy saving is achieved by sharing 80% of ResNet101 base-network parameters on CIFAR100 while learning new classes.The reported energy is computation energy; memory-access energy is treated separately.Source: §5.1 · p. 12

5.2 Training Time Reduction

  • 1.55-6× reduction in training time per iteration is reported for CIFAR100 with ResNet101 across sharing configurations.The authors note that saved time cannot be converted into more training epochs to improve accuracy; retraining with old and new samples is time-consuming and conflicts with incremental learning.Source: §5.2 · p. 12

5.3 Storage Requirement and Memory Access Reduction

  • 67-99% reduction in storage requirement and 43-55% savings in memory access energy are reported for CIFAR100 with ResNet101.The comparisons are shown in Figure 11 for storage and memory-access requirements.Source: Figure 11 · p. 12, §5.3 · pp. 12–13

5.4 Results on ImageNet

  • ImageNet experiments divide 1,000 classes into mutually exclusive sets of 500, 300, and 200; optimal sharing varies by architecture, from about 1.5% for ResNet18 to 57% for DenseNet121.The reported combined classification accuracies for DenseNet121 and MobileNet are about 64% and 62%, respectively.Source: §5.4 · p. 13, Table 4 · p. 13

5.5 Comparison between Different Network Architectures

  • With 1 ± 0.5% accuracy degradation as the tolerance, deeper networks generally provide more energy benefits, but sharing ratios do not scale linearly with energy or time savings.For similar sharing, training time per iteration falls by 51% for ResNet34 and 62% for MobileNet on ImageNet.Source: §5.5 · p. 13, Figure 12 · p. 14
  • The authors report that deeper networks permit more sharing, while a base network trained on more classes and examples provides features that improve combined performance.For CIFAR-100, ResNet101 shares 80% versus ResNet18's 24% at equivalent accuracy degradation; ImageNet combined accuracy is closer to the cumulatively trained upper bound than CIFAR-100.Source: §5.5 · pp. 13–14

5.6 Comparison with Other Methods

  • Old-task branch and shared parameters remain unchanged during new-task learning; task-specific inference cost stays similar to baseline, while combined-classification cost grows sublinearly.The authors also describe use cases including on-chip learning in phones and drones.Source: §5.6 · p. 15
  • The proposed partial-sharing approach performs almost the same as incremental training without sharing and is not far from cumulative training, while naïve fine-tuning forgets old classes.The comparison reports performance across incremental-learning approaches.Source: §5.6 · p. 15, Figure 13 · p. 15

6 Conclusion

The paper presents partial network sharing as a way to update DCNNs for new data without retraining on previously seen samples. It reports comparable task-specific accuracy while reducing incremental-training complexity and energy use.

6 Conclusion

  • The paper identifies small, time-separated data batches as a practical setting for incremental classifiers.Source: §6 · p. 15
  • The method accommodates new, previously unseen data without retraining the whole network on previously seen data.New networks start from an existing base network and share its convolutional layers.Source: §6 · p. 16
  • Sharing convolutional layers lets updated networks inherit base-network features, improving computational effort and energy consumption during training and speeding learning.Source: §6 · p. 16
  • The approach focuses on reducing incremental-training complexity while maintaining comparable task-specific accuracy and combined accuracy close to the upper bound without old training samples.The authors report applying it to different DCNNs on real-world recognition applications.Source: §6 · p. 16
  • The authors describe the approach as useful for re-configurable hardware such as FPGAs, where network capacity can be increased for new classes without significant effort.Source: §6 · p. 16

Related papers

Loading 1712.02719v4…