PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What and Where to Transfer

Yunhun Jang, Hankook Lee, Sung Ju Hwang, Jinwoo Shin

arXiv:1905.05901v1cs.LGstat.ML

TL;DR

Limited labeled data motivate transfer learning, but heterogeneous source and target tasks or architectures make transfer configuration difficult to select manually. The paper uses meta-networks and an efficient meta-learning scheme to learn which knowledge and layer connections to transfer, outperforming baseline methods across reported settings.

  • Problem

    Heterogeneous tasks and architectures make it difficult to determine which source information and layer pairs should be used for transfer without exhaustive configuration tuning.

  • Method

    Meta-networks jointly learn feature-transfer weights and source-target layer-transfer weights while training the target model with an efficient one-step adaptation scheme.

  • Results

    The method significantly outperforms baseline transfer methods across reported task, limited-data, and multi-source settings, including 65.05% versus 58.90% accuracy on CUB200.

  • Takeaways & Limitations

    Selective automated transfer can identify useful knowledge and configurations across heterogeneous or multiple network architectures and tasks.

  • Takeaways & Limitations

    Fine-tuning remains unsuitable when source and target tasks are semantically distant or their network architectures differ substantially.

Abstract

from arXiv · show

As the application of deep learning has expanded to real-world problems with insufficient volume of training data, transfer learning recently has gained much attention as means of improving the performance in such small-data regime. However, when existing methods are applied between heterogeneous architectures and tasks, it becomes more important to manage their detailed configurations and often requires exhaustive tuning on them for the desired performance. To address the issue, we propose a novel transfer learning approach based on meta-learning that can automatically learn what knowledge to transfer from the source network to where in the target network. Given source and target networks, we propose an efficient training scheme to learn meta-networks that decide (a) which pairs of layers between the source and target networks should be matched for knowledge transfer and (b) which features and how much knowledge from each feature should be transferred. We validate our meta-transfer approach against recent transfer learning methods on various datasets and network architectures, on which our automated scheme significantly outperforms the prior baselines that find "what and where to transfer" in a hand-crafted manner.

1. Introduction

Transfer learning is useful when target data are scarce, but heterogeneous tasks and architectures make manual choices about transferred knowledge and layer matching difficult. The paper proposes meta-networks that learn what to transfer and where, reporting improvements over baseline methods.

  • Transfer learning addresses the high cost of collecting enough labeled data for each target task.
  • Fine-tuning may fail for semantically distant tasks and offers no straightforward procedure when source and target architectures differ substantially.
  • Existing heterogeneous-network methods transfer features, attention maps, Jacobians, or distilled outputs but do not identify which source information and layer pair are useful.
  • The proposed meta-networks jointly learn feature weights and source-target layer-transfer weights, avoiding hand-crafted transfer configurations.
  • The efficient training scheme evaluates one-step target adaptation after transfer-objective updates, accelerating the inner loop compared with the standard scheme.
  • 65.05% accuracy on CUB200 versus 58.90% for the second-best baseline demonstrates the reported improvement in an ImageNet experiment.

2. Learning What and Where to Transfer

The method uses meta-networks to learn both which source-target layer pairs to match and which source channels to transfer. A bilevel training scheme jointly updates the target network and meta-networks so transfer choices are evaluated by target-task performance.

  • Overview: Meta-networks learn which source features to transfer to which target layers, avoiding manual layer association and feature selection.The method focuses primarily on convolutional networks but is described as applicable to other deep neural network types.
  • Weighted Feature Matching: Feature matching minimizes an ℓ2 distance between transformed target features and source feature maps.A learned linear transformation, such as a pointwise convolution, aligns target and source features and is unnecessary at test time.
  • Weighted Feature Matching: Channel weights emphasize source features according to their utility for the target task and can vary with each input image.The weighted loss assigns non-negative channel weights normalized to sum to one.
  • Where to Transfer: Layer-pair weights λ_m,n are generated by meta-networks gφ to automatically determine how much transfer occurs between candidate source-target layer pairs.The combined transfer loss sums weighted feature-matching losses over the candidate pair set C.
  • Training Meta-Networks and Target Model: The alternative three-stage bilevel scheme first trains on transfer loss, performs one target-label adaptation step, and updates meta-parameters using the resulting original loss.This increases the transfer loss’s influence during the inner loop and enables faster meta-network updates than the standard two-stage scheme.

3. Experiments

Experiments evaluate meta-transfer across heterogeneous architectures, tasks, limited-data settings, and multiple sources. Learning both what and where to transfer consistently improves performance, especially for fine-grained and data-scarce targets.

  • Setups: Experiments span 32 × 32 and 224 × 224 image-classification tasks with heterogeneous source and target architectures.TinyImageNet uses ResNet32 as source and VGG9 as target; ImageNet uses ResNet34 and ResNet18.
  • Various Target Tasks: L2T-w improves over unweighted feature matching, particularly when general ImageNet knowledge transfers to fine-grained target tasks.The reported pattern supports learning which source information to transfer while retaining hand-crafted layer matching.
  • Various Target Tasks: L2T-ww achieves the best accuracy in all reported Table 1 cases and attains a 10.4% relative improvement over the second-best baseline on CUB200.It learns layer matching from all candidate pairs rather than relying on hand-crafted configurations.
  • Various Target Tasks: After transfer from TinyImageNet to STL-10, learned transfer amounts concentrate on higher target layers, with λ1,5 = 2.62 and λ3,4 = 2.88 among the reported values.Most other pairwise transfer amounts are below 0.1 except λ1,2 = 0.21, indicating nontrivial learned layer selection.
  • Limited-Data Regimes: With 50 CIFAR-10 samples per class, L2T-ww reaches 64.91% accuracy versus 53.76% for LwF+AT and 39.99% from scratch.The method achieves similar accuracy with 50 samples per class to LwF with 250 samples per class.
  • Multi-Source Transfer: L2T-ww is the only compared method reported to effectively aggregate heterogeneous multi-source knowledge from TinyImageNet and CIFAR-10.Performance improves when adding a larger source model or a source trained on a different dataset.
  • Visualization: L2T-w increases saliency on task-specific bird and dog pixels while reducing activation on background pixels relative to unweighted feature matching.The learned weights make transferred source knowledge more task-specific.

4. Conclusion

The method selectively transfers knowledge according to task and architecture differences, using meta-networks to learn what and where to transfer. An efficient meta-learning scheme jointly trains the target model and meta-networks with few inner-loop steps.

  • Meta-learning enables selective knowledge transfer that depends on the source and target tasks and architectures.
  • Meta-networks identify what information to transfer and where to transfer it in heterogeneous networks.
  • The training scheme jointly optimizes the target model and meta-networks using only a few inner-loop steps.

A. Network Architectures and Tasks

Experiments cover heterogeneous source and target networks across small- and large-image classification tasks. Small-image targets use a TinyImageNet source, while large-image targets use an ImageNet source and four specialized datasets.

  • Small image experiments: Small-image experiments transfer from a pre-trained 32-layer ResNet on TinyImageNet to a 9-layer VGG on CIFAR-10, CIFAR-100, and STL-10.Images are resized to 32 × 32 where necessary.
  • Large image experiments: Large-image experiments use a pre-trained 34-layer ResNet on ImageNet for CUB200, MIT67, Stanford 40 Actions, and Stanford Dogs.CUB200 contains 5k training images across 200 bird species.

B. Optimization

Target networks are trained with SGD and momentum, cosine-annealed learning rates, and fixed 200-epoch schedules. Meta-networks use ADAM, and the authors report that two inner-loop steps suffice for meta-training.

  • Target-network optimization: Target networks use SGD with momentum 0.9, weight decay 10^-4, initial learning rate 0.1, and cosine annealing over 200 epochs.Mini-batches contain 128 examples for small-image experiments and 64 for large-image experiments.
  • Meta-network optimization: Meta-networks are optimized with ADAM using learning rates of 10^-3 or 10^-4 and weight decay of 0 or 10^-4.
  • Meta-training: The meta-training scheme requires only T = 2 inner-loop steps to learn what and where to transfer.Experiments are repeated three times, with average performance and standard deviation reported.

C.1. Comparison between the meta-networks and meta-weights

The comparison evaluates sample-dependent meta-networks against constant meta-weights for determining transfer amounts. Meta-networks produce different transfer importance for each sample, whereas meta-weights apply the same importance across samples.

  • Table 3 reports classification accuracy for transfer learning using meta-networks or meta-weights.
  • Meta-networks generate channel and connection importance values that determine transfer amounts for each sample.
  • Directly learned meta-weights provide the same transfer importance across all samples, unlike sample-dependent meta-networks.The comparison uses the same hyperparameters except for the meta-parameters.

C.2. Comparison between the proposed bilevel scheme and original one

The proposed bilevel scheme is more effective than the original scheme for learning meta-networks for selective transfer, especially as T increases. The original scheme achieves significantly lower accuracies and becomes impractical at larger T.

  • C.2. Comparison between the proposed bilevel scheme and original one: The proposed bilevel scheme significantly outperforms the original scheme in classification accuracy under the same T = 2 setting.The comparison uses identical hyperparameters, architectures, and source task conditions.
  • C.2. Comparison between the proposed bilevel scheme and original one: With T = 5∼100, the original scheme fails to obtain performance comparable to the proposed bilevel scheme.This result holds across substantially larger values of T than the controlled T = 2 comparison.
  • C.2. Comparison between the proposed bilevel scheme and original one: Meta-training time for the original scheme increases linearly as T increases, making it inapplicable to practical scenarios.The authors conclude that the proposed bilevel scheme is more effective for learning meta-networks for selective transfer.
Loading 1905.05901v1…