Source-linked AI summary
DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, Jun Huan
TL;DR
Limited target datasets can make fine-tuning pretrained networks vulnerable to accuracy bottlenecks and overfitting, motivating alternatives to weight-based regularization. DELTA aligns attention-selected feature-map outputs between source and target networks instead of directly constraining weights. Across experiments, it significantly outperformed state-of-the-art transfer-learning methods, including L2 and L2-SP.
Problem
Limited target-task data can bottleneck fine-tuning accuracy, while directly regularizing weights may not optimally balance preserving source knowledge with adapting to target data.
Method
DELTA regularizes the difference between source and target convolutional feature maps, using supervised attention to select discriminative features rather than directly constraining network weights.
Results
DELTA significantly outperformed state-of-the-art transfer-learning methods, including L2 and L2-SP, with higher accuracy across a wide group of image-classification datasets.
Takeaways & Limitations
Feature-map behavior and attention provide DELTA with a transfer-learning regularization strategy that preserves transferable channels while reusing untransferable ones.
Abstract
from arXiv · showhide
Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP. The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.
1 INTRODUCTION
Fine-tuning transfers knowledge from a large source dataset to a small target dataset, but weight-based regularization can either erase useful knowledge or constrain adaptation. DELTA instead regularizes selected feature-map outputs using supervised attention and outperforms L2 and L2-SP on image-classification tasks.
- Motivation: Fine-tuning a network pretrained on a large source dataset can improve initialization for a small target task, but target weights may drift and overfit.Lower convolution layers are often fixed while upper layers are retrained, potentially driving parameters far from their initial values.
- Problem: Weight regularization faces a trade-off: weak constraints can cause catastrophic memory loss, whereas strong constraints may limit adaptation to the target data.These opposing risks motivate alternatives to directly constraining model parameters.
- DELTA: DELTA regularizes outer-layer behavior rather than weights by aligning source and target feature-map outputs.For convolutional networks, outer-layer outputs are feature maps, and constraining them is intended to transfer source-network generalization capacity.
- DELTA: A supervised attention mechanism selects discriminative feature-map components for the output-based regularization term.This addresses the difficulty of distinguishing task-shared features from source-specific features when comparing feature maps.
- DELTA: DELTA identifies transferable channels to preserve and untransferable channels to reuse through attention-based feature-map regularization.The paper calls this mechanism “unactivated channel re-usage.”
- Results: DELTA significantly outperformed state-of-the-art regularization algorithms, including L2 and L2-SP, across a wide group of image-classification datasets.The evaluation used a wide range of source/target dataset pairs.
2 RELATED WORK AND BACKGROUNDS
The paper situates DELTA within transfer learning and regularization methods that address overfitting by controlling divergence between source and target networks. It contrasts parameter-based SPAR approaches with behavior-based regularization that considers network outputs.
- Transfer learning: Transfer learning reuses knowledge from a source task to improve learning on a target task, often by initializing the target model with source-network weights.Fine-tuning can accelerate training, but target-task accuracy may remain bottlenecked by limited data.
- DELTA’s positioning: DELTA differs from prior approaches by constraining differences between source and target feature-map outputs and reweighting those differences according to performance gain or loss.The comparison targets outer-layer behavior instead of only parameter similarity.
- Regularization for transfer learning: SPAR methods regularize the divergence between target and source parameters, with L2-SP applying Euclidean-distance penalties to corresponding convolution-filter weights.L2-SP was reported to outperform standard weight decay in deep transfer learning.
- Limitations of parameter regularization: Parameter regularization creates a trade-off: weak constraints risk catastrophic memory loss, whereas strong constraints can confine the target model to a suboptimal neighborhood.This motivates regularizing network behavior rather than directly constraining all model parameters.
3 LEARNING FRAMEWORK AND ALGORITHMS
DELTA regularizes transfer learning by aligning source and target network behaviors, represented by outer-layer feature maps, rather than directly matching their weights. A supervised attention mechanism assigns stronger constraints to more discriminative feature maps, while a proximal term regularizes target-private parameters.
- Overall framework: DELTA defines a layer’s behavior as its output and seeks to make selected target-network layers behave similarly to the corresponding source-network layers.These outputs are treated as semantically rich and discriminative information.
- Combined regularizer: DELTA combines behavioral regularization with a parameter-based proximal term that constrains target-private parameters absent from the source network.The two terms are balanced by non-negative tuning parameters α and β.
- Feature-map representation: For each input, DELTA represents an outer-layer output with feature maps generated by the network’s convolutional filters.The feature maps are denoted FM_j(z, ω, x_i) for filters j = 1,...,N.
- Behavioral regularization: The behavioral regularizer measures Euclidean distances between corresponding source and target feature maps and aggregates them with non-negative attention weights.The weights are indexed by both the filter and the labeled input image.
- Feature-map extraction: DELTA extracts each filter’s rectified output and concatenates the resulting matrices into feature-map vectors used by the behavioral regularizer.The feature extractor computes FM_j(z, ω, x_i) for each filter and target input.
- Supervised attention: The supervised attention weight for a filter is based on the performance reduction caused by disabling that filter in the pretrained network.Softmax normalization makes the weights non-negative, assigning more attention to filters whose removal produces greater loss.
4 EXPERIMENTS AND RESULTS
Experiments across image-recognition and scene-classification benchmarks compare DELTA with established regularization methods. DELTA converges faster, remains smoother under learning-rate changes, and achieves stronger reported classification performance while attention supports its feature-map-based behavior.
- Datasets: Experiments cover Caltech 256, Stanford Dogs 120, MIT Indoors 67, CUB-200-2011, and Food-101 across object, fine-grained, scene, and noisy-image classification settings.ImageNet supplies the source domain for Caltech 256 and Stanford Dogs 120, while Places 365 supplies it for MIT Indoors 67.
- Learning curves: DELTA converges faster than L2-SP under both StepLR and ExponentialLR schedules, with a smoother curve and reduced sensitivity to the StepLR decay at 6000 iterations.StepLR decays the initial learning rate by 0.1 after 6000 iterations, whereas ExponentialLR decays it by 0.93 each epoch.
- Accuracy comparisons: DELTA with attention gains greater benefits than its no-attention variant and is compared against L2-FE, L2, and L2-SP using top-1 accuracy.The experiments report that L2-SP outperforms naive methods without starting-point regularization, while DELTA’s attention mechanism provides additional gains.
- Accuracy comparisons: Data augmentation improves classification accuracy for L2, L2-SP, and DELTA, while DELTA remains the best-performing method in the reported comparison.The procedure preserves aspect ratio during resizing and applies 10-crop testing.
- Case study: About 90% of DELTA parameter vectors move farther from the starting point than L2-SP, alongside a small number of filters driven very far away.The authors call this effect “unactivated channel re-usage” and associate it with attention allowing unactivated convolution filters to be reused.
- Case study: Attention changes activation maps toward concentrated responses around important animal regions, whereas DELTA without attention resembles other regularization methods more closely.The reported qualitative analysis highlights high activation around animals’ heads, and CUB-200-2011 part-location statistics provide additional supporting evidence.
5 CONCLUSION
The paper presents DELTA, a regularization technique that transfers source-network behaviors and semantics by constraining attention-weighted feature-map differences. Experiments on real-world datasets and common CNNs show that DELTA significantly outperforms state-of-the-art transfer-learning methods.
- 5 CONCLUSION: DELTA transfers source-network behaviors and semantics by constraining attention-weighted differences between source and target convolutional feature maps.Its attention models are obtained through supervised learning.
- 5 CONCLUSION: DELTA models feature-map differences with attentions and accelerates optimization for regularization using the starting point as reference.
- 5 CONCLUSION: DELTA significantly outperforms state-of-the-art transfer-learning methods across experiments using several real-world datasets and commonly used convolutional neural networks.
A APPENDIX
The appendix presents examples from Stanford Dogs and CUB-200-2011 and illustrates how attention affects fine-tuning in DELTA.
- A APPENDIX: Examples with different regularization methods are shown for Stanford Dogs and CUB-200-2011.
- A APPENDIX: Figure 4 illustrates the effect of the attention mechanism for fine-tuning, including DELTA without attention.