Source-linked AI summary
SpotTune: Transfer Learning through Adaptive Fine-tuning
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, Rogerio Feris
TL;DR
Limited target data makes globally choosing which pre-trained layers to freeze or fine-tune difficult, especially when target examples differ in source-task similarity. SpotTune learns input-dependent routing between pre-trained and fine-tuned layers, outperforming standard fine-tuning on 12 of 14 datasets and achieving the highest Visual Decathlon score. The paper also introduces a compact global variant that learns a shared set of k layers, though choosing k remains future work.
Problem
Fine-tuning requires choosing which layers to freeze or update, but global or manually selected choices may be suboptimal for small and heterogeneous target datasets.
Method
SpotTune trains a policy network with Gumbel Softmax to route each target image through pre-trained or fine-tuned residual blocks.
Results
12 out of 14 datasets show SpotTune outperforming standard fine-tuning, and it achieves the highest overall score on the 10-dataset Visual Decathlon benchmark.
Takeaways & Limitations
Input-dependent fine-tuning specializes the strategy for each target example and achieves state-of-the-art overall performance on Visual Decathlon.
Takeaways & Limitations
The optimal number k of globally fine-tuned blocks is left for future work.
Abstract
from arXiv · showhide
Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision. The typical way of conducting transfer learning with deep neural networks is to fine-tune a model pre-trained on the source task using data from the target task. In this paper, we propose an adaptive fine-tuning approach, called SpotTune, which finds the optimal fine-tuning strategy per instance for the target data. In SpotTune, given an image from the target task, a policy network is used to make routing decisions on whether to pass the image through the fine-tuned layers or the pre-trained layers. We conduct extensive experiments to demonstrate the effectiveness of the proposed approach. Our method outperforms the traditional fine-tuning approach on 12 out of 14 standard datasets.We also compare SpotTune with other state-of-the-art fine-tuning strategies, showing superior performance. On the Visual Decathlon datasets, our method achieves the highest score across the board without bells and whistles.
1. Introduction
SpotTune addresses the difficulty of choosing which layers to fine-tune by making layer-freezing decisions separately for each target example. It reports stronger results than standard fine-tuning across most evaluated datasets and the Visual Decathlon benchmark.
- Fine-tuning transfers a source-task model to target data, improving performance while reducing labeled-data requirements compared with training from scratch.
- Whole-network fine-tuning can overfit small target datasets, while freezing initial layers leaves the number of frozen layers as a manual choice.
- Global freeze-versus-fine-tune decisions may be suboptimal because target examples or classes can differ in similarity to the source task.
- SpotTune uses a lightweight policy network to decide per instance which layers remain frozen or are fine-tuned, trained with Gumbel Softmax sampling.
- The global variant learns a shared set of k layers that can be distributed anywhere in the network, allowing corresponding pre-trained layers to be discarded.
- 12 out of 14 datasets show SpotTune outperforming standard fine-tuning, while the method achieves the highest score among state-of-the-art methods on Visual Decathlon.
2. Related Work
Prior transfer-learning work includes feature alignment, parameter sharing, and several deep-network fine-tuning choices. SpotTune is related to dynamic routing but uses routing to select frozen versus fine-tuned parameters rather than to drop layers for computational efficiency.
- Transfer Learning: Transfer-learning research spans instance re-weighting, model adaptation, feature-space alignment, and decisions about which tasks or parameters to share.
- Transfer Learning: Common deep-learning transfer strategies fine-tune all parameters, fine-tune only the last few layers, or use a pre-trained network as a fixed feature extractor.
- Dynamic Routing: Dynamic-routing methods selectively execute units or layers per example, often targeting computational efficiency through sparse activation or layer dropping.
- Dynamic Routing: SpotTune also uses Gumbel-based dynamic routing, but selects parameters to freeze or fine-tune during learning instead of dropping layers.
3. Proposed Approach
SpotTune learns input-dependent routing that selects, for each target image and residual block, either source-task parameters or adapted parameters. It uses Gumbel Softmax to train these discrete policies and also offers a compact global variant that learns a shared set of fine-tuned blocks.
- 3.1. SpotTune Overview: SpotTune assigns each target image a policy that determines which residual blocks are fine-tuned and which remain frozen from pre-training.A lightweight policy network produces per-block freeze or fine-tune decisions conditioned on the input image.
- 3.1. SpotTune Overview: ResNet is used because its residual blocks can be dynamically swapped between pre-trained and fine-tuned versions.The paper focuses on ResNets and cites their resilience to residual block swapping as supporting this design.
- 3.2. Training with the Gumbel Softmax Policy: At test time, each image is routed through either the fine-tuned or pre-trained residual block at every decision point, with no additional residual-block executions.The policy network adds only a small computational cost relative to the original pre-trained model.
- 3.2. Training with the Gumbel Softmax Policy: Gumbel Softmax makes discrete freeze or fine-tune routing trainable with backpropagation through a smooth relaxation.The policy samples two-category decisions, while the continuous relaxation supplies gradients to the policy network.
- 3.3. Compact Global Policy Variant: The global-k variant learns a shared set of k blocks to fine-tune, reducing memory and computation while eliminating the test-time policy network.Additional losses encourage a deterministic global policy, and the selected blocks may occur anywhere in the ResNet.
4. Experiments
SpotTune is evaluated against fine-tuning and regularization baselines on five public datasets and the ten-domain Visual Decathlon Challenge. Across these experiments, it generally outperforms standard and competing strategies while learning dataset- and image-specific fine-tuning behavior.
- 4.1. Experimental Setup: The experiments compare SpotTune with fine-tuning and regularization techniques on five public datasets, measuring classification accuracy on the evaluation set.The datasets include CUBS, Stanford Cars, Flowers, Sketches, and WikiArt.
- 4.2.1. SpotTune vs. Fine-tuning Baselines: SpotTune performs better than competing fine-tuning variants across the evaluated datasets, including L2-SP, while L2-SP can be combined with SpotTune for further improvement.The fine-tuned ResNet-101 is more computationally expensive yet performs worse than SpotTune except on WikiArt.
- 4.2.3. Visualization of Block Usage: Different images within a dataset use different numbers of fine-tuned blocks, and block-usage distributions differ substantially across target datasets.These observations support image-specific policies rather than one global policy for all target images.
- 4.2.3. Visualization of Block Usage: Images with cleaner backgrounds tend to use fewer fine-tuned blocks, whereas images with more complex backgrounds often use more.This pattern is reported for example images from CUBS and Flowers and is described as possibly related to similarity with ImageNet data.
- 4.2.4. Visual Decathlon Challenge: 3612 vs 3096: SpotTune substantially improves the Visual Decathlon score over standard fine-tuning with almost the same number of parameters.The Visual Decathlon score accounts for the difficulty of different domains rather than using average accuracy alone.
- 4.2.4. Visual Decathlon Challenge: SpotTune outperforms standard fine-tuning on 12 of 14 datasets and achieves the highest score among state-of-the-art methods on Visual Decathlon.The 14 datasets combine the Visual Decathlon domains with five additional experiments, with Flowers counted in both sets.
5. Conclusion
SpotTune adapts fine-tuning to each target example and outperforms widely used fine-tuning protocols across public benchmarks, including Visual Decathlon.
- SpotTune specializes the fine-tuning strategy for each training example of the target dataset.
- SpotTune outperforms key popular fine-tuning protocols on a variety of public benchmarks.
- SpotTune achieves the new state of the art on Visual Decathlon, measured by the overall score across 10 datasets.