Source-linked AI summary
Growing a Brain: Fine-Tuning by Increasing Model Capacity
Yu-Xiong Wang, Deva Ramanan, Martial Hebert
TL;DR
The paper asks how to adapt pre-trained CNNs to new tasks more effectively than fixed-capacity fine-tuning. It grows networks by adding units through depth or width augmentation and finds that appropriately normalized growth improves transfer and benchmark performance.
Problem
Pre-trained CNNs are commonly adapted with fixed-capacity fine-tuning, but the paper investigates how model components and parameters should change during adaptation.
Method
The paper grows pre-trained CNNs during fine-tuning by adding layers or channels and normalizes new units so their learning pace matches existing units.
Results
Increasing model capacity significantly improves transfer, with depth and width augmentation yielding benchmark gains and a slight but consistent benefit for widening.
Takeaways & Limitations
Growing and appropriately scaling a pre-trained network provides a developmental alternative to fixed-capacity fine-tuning for transfer across tasks.
Abstract
from arXiv · showhide
CNNs have made an undeniable impact on computer vision through the ability to learn high-capacity models with large annotated training sets. One of their remarkable properties is the ability to transfer knowledge from a large source dataset to a (typically smaller) target dataset. This is usually accomplished through fine-tuning a fixed-size network on new target data. Indeed, virtually every contemporary visual recognition system makes use of fine-tuning to transfer knowledge from ImageNet. In this work, we analyze what components and parameters change during fine-tuning, and discover that increasing model capacity allows for more natural model adaptation through fine-tuning. By making an analogy to developmental learning, we demonstrate that "growing" a CNN with additional units, either by widening existing layers or deepening the overall network, significantly outperforms classic fine-tuning approaches. But in order to properly grow a network, we show that newly-added units must be appropriately normalized to allow for a pace of learning that is consistent with existing units. We empirically validate our approach on several benchmark datasets, producing state-of-the-art results.
1. Motivation
CNNs transfer knowledge from large annotated source datasets, but the best way to adapt pre-trained networks to novel tasks remains unclear. This work proposes growing model capacity during transfer and reports improved adaptation, continual transfer, and benchmark performance.
- Producing large annotated datasets for every new category or task is typically unrealistic, motivating transfer from diverse base datasets such as ImageNet.
- Fine-tuning a fixed-capacity network is the dominant transfer strategy, but its parameter changes and adaptation behavior remain relatively poorly understood.
- Developmental networks grow during transfer by adding units through either deeper networks with more layers or wider networks with more channels per layer.
- Additional units guide adaptation of pre-existing units: deeper units form new compositions, while wider units discover complementary target-task cues.
- Developmental networks remain accurate on the source task, support continual transfer across multiple tasks, and produce state-of-the-art results across scene, fine-grained, and action recognition benchmarks.
2. Related Work
Prior transfer-learning work largely assumes fixed-capacity models, while related approaches address adaptation, source-task preservation, or lifelong learning in different ways. This paper systematically studies growing networks while fine-tuning the entire model.
- Much prior transfer-learning research assumes that model capacity remains fixed across tasks.
- Related methods include adaptation layers, source-task preservation strategies, and progressive networks originally proposed for reinforcement learning.
- Unlike progressive networks that widen a target network substantially and fine-tune only new units, this work adds a small fraction of units and fine-tunes the entire network.
- Other transfer and few-shot approaches can predict models rather than classes, but typically do not dynamically grow parameters across new tasks.
- The developmental network differs from non-parametric models by cumulatively growing capacity from novel tasks, connecting the approach to developmental and lifelong learning.
3. Approach Overview
Classic fine-tuning copies a source representation, replaces the classifier, and updates the fixed-capacity network on target data. The proposed developmental networks instead add representation units, using depth or width augmentation while retaining the target classifier.
- Classic fine-tuning copies the source representation module into the target CNN and introduces a randomly initialized classifier for the target task.
- The copied representation parameters are fine-tuned with continued backpropagation, typically using a smaller learning rate, while network structure and capacity remain fixed.
- The developmental approach adds new units to the target representation module, significantly improving transfer particularly when fewer target examples are available.
- New units are placed at later layers, either by constructing a new top layer in DA-CNN or widening an existing top layer in WA-CNN.
- Figure 2 contrasts classic fine-tuning with developmental architectures that augment model capacity.
- The approach also considers jointly depth-and-width augmented and recursively width-augmented configurations, while retaining the target classifier.
4. Developmental Networks
Developmental networks increase a pre-trained CNN’s capacity by adding units through depth or width augmentation, while width augmentation requires normalization and scaling to reconcile learning rates between new and existing units.
- Depth augmentation: A depth-augmented network adds a new top layer that composes pre-existing units before the target classifier.The added layer serves as an adaptation layer, limiting modifications to pre-trained layers.
- Width augmentation: A width-augmented network adds units to an existing top layer, connects them laterally, and concatenates their activations before classification.The original top-layer connections remain while additional lateral weights connect the new units.
- Learning pace: New and pre-existing units learn at different paces because the former are randomly initialized while the latter are already adapted to the source domain.Naively concatenating their differently scaled activations can let larger activations dominate, degrading performance or collapsing the network.
- Learning pace: Normalization and scaling are not crucial for depth-augmented networks but remain beneficial with negligible parameter overhead.The added parameters equal the total number of channels.
5. Experimental Evaluation
Across SUN-397 and other benchmark evaluations, developmental networks that add capacity consistently outperform fixed-capacity fine-tuning, while normalization helps reconcile new and existing units. The approach also supports clearer feature organization, source-task retention, and continual transfer across tasks.
- SUN-397 evaluation: On SUN-397, DA-CNN and WA-CNN significantly outperform vanilla fine-tuned CNNs across all four fine-tuning scenarios, with WA-CNN slightly ahead of DA-CNN.The scenarios range from fine-tuning only new layers to fine-tuning the entire network.
- SUN-397 evaluation: The recursively width-augmented WWA-CNN generally performs best, while jointly deepening and widening the network trails purely width augmentation slightly.The result indicates that adding capacity at complementary levels can matter, but combining depth and width is non-trivial.
- SUN-397 evaluation: Increasing augmented model capacity improves performance, although gains diminish as more new units are added.This pattern holds across diagnostic variations of the developmental architectures.
- SUN-397 evaluation: Normalization and scaling substantially improve width-augmented networks by rebalancing the learning pace of new and pre-trained units.Without scaling, WA-CNN can be only marginally better or worse than classic fine-tuning; with scaling, it exceeds the baseline by a large margin.
- Transfer and retention: Developmental networks retain source-task accuracy after transfer, supporting learning without forgetting, and their capacity can grow cumulatively for continual transfer across multiple tasks.The paper evaluates source-task retention on ImageNet and continual transfer through progressively increased capacity.
- Feature-space analysis: DA-CNN and WA-CNN produce clearer semantic clustering than classic fine-tuning, while WA-CNN units specialize more discriminatively toward the target task.Feature visualizations show clearer t-SNE separation and re-centralization of unit activations for the novel task.
6. Conclusions
Increasing model capacity helps pre-existing CNN units adapt and specialize during fine-tuning, with depth and width augmentation both useful and widening slightly more consistent. Normalizing and scaling added units balances their learning pace, supporting strong benchmark performance and progressive lifelong learning.
- Increasing model capacity helps existing CNN units better adapt and specialize to target tasks during fine-tuning.
- Both deeper and wider networks improve fine-tuning, with a slight but consistent benefit for widening.
- Newly added units require normalization and scaling so their learning pace remains comparable to pre-existing units.
- Benchmark results show benefits from increasing model capacity, supporting progressive growth during lifelong learning with evolving data and tasks.