PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learning

Kshitij Dwivedi, Gemma Roig

arXiv:1904.11740v1cs.CVcs.AIcs.LG

TL;DR

The paper addresses how to choose among pre-trained models for transfer learning when labeled data are limited. It uses RSA to compare task-specific representations from a few images without further training, and finds that RSA similarity tracks transfer performance across Taskonomy and Pascal VOC, including domain-shift cases.

  • Problem

    Choosing a suitable pre-trained model for transfer learning is difficult, while evaluating many candidates through finetuning is computationally expensive.

  • Method

    RSA compares representation dissimilarity matrices from pre-trained task-specific models using a few images, without further finetuning.

  • Results

    RSA similarity closely relates to transfer-learning performance on Taskonomy and Pascal VOC semantic segmentation, and the RSA-selected model ranked in the top five for 16 of 17 Taskonomy tasks.

  • Takeaways & Limitations

    RSA can identify high-performing transfer-learning models, including under domain shift where the model trained on the same task may not be the best fit.

  • Takeaways & Limitations

    Small-model comparisons could not exactly replicate Taskonomy training because its training code was unavailable and the small models used only a subset of the dataset.

Abstract

from arXiv · show

Transfer learning is widely used in deep neural network models when there are few labeled examples available. The common approach is to take a pre-trained network in a similar task and finetune the model parameters. This is usually done blindly without a pre-selection from a set of pre-trained models, or by finetuning a set of models trained on different tasks and selecting the best performing one by cross-validation. We address this problem by proposing an approach to assess the relationship between visual tasks and their task-specific models. Our method uses Representation Similarity Analysis (RSA), which is commonly used to find a correlation between neuronal responses from brain data and models. With RSA we obtain a similarity score among tasks by computing correlations between models trained on different tasks. Our method is efficient as it requires only pre-trained models, and a few images with no further training. We demonstrate the effectiveness and efficiency of our method for generating task taxonomy on Taskonomy dataset. We next evaluate the relationship of RSA with the transfer learning performance on Taskonomy tasks and a new task: Pascal VOC semantic segmentation. Our results reveal that models trained on tasks with higher similarity score show higher transfer learning performance. Surprisingly, the best transfer learning result for Pascal VOC semantic segmentation is not obtained from the pre-trained model on semantic segmentation, probably due to the domain differences, and our method successfully selects the high performing models.

1. Introduction

The paper addresses pre-trained model selection for transfer learning by relating visual tasks through RSA, avoiding exhaustive finetuning. It evaluates this strategy on Taskonomy and Pascal VOC, finding strong alignment between RSA similarity and transfer performance.

  • Transfer learning reuses representations from previously learned tasks when labeled data for a new task is scarce or expensive.
  • Selecting a suitable pre-trained model is difficult because many models are trained on varied vision tasks, while existing relationship estimates can require expensive transfer-learning evaluations.
  • The paper proposes RSA to measure relationships between task-specific models using their learned representations.
  • RSA requires pre-trained models and representations from only a few images, without finetuning previous task models.
  • On Taskonomy, the RSA-selected model ranked in the top five for transfer performance on 16 of 17 tasks.
  • RSA similarity closely related to transfer performance on Taskonomy and Pascal VOC, including domain-shift cases where the same-task model was not best.

2. Related Works

Related work covers transfer-learning initialization, Taskonomy’s computational task relationships, and neuroscience applications of RSA. The paper positions RSA as an alternative for task taxonomy and transfer-model selection.

  • Transfer Learning: Transfer learning commonly initializes a DNN with a pre-trained model and finetunes parameters because limited annotations make training from scratch prone to overfitting.
  • Transfer Learning: Because initialization significantly affects transfer performance, a strategy is needed to select suitable pre-trained models.
  • Taskonomy: Taskonomy models task relationships through transfer performance across 26 supervised visual tasks, but its evaluation of new-task relationships is computationally expensive.
  • Representation Similarity Analysis: RSA is widely used in computational neuroscience to compare brain activity with computational and behavioral model representations.
  • Representation Similarity Analysis: This paper applies RSA to task taxonomy and transfer learning, with possible extension to choosing branching locations in multi-task networks.

3. Representation Similarity Analysis (RSA)

RSA compares task-specific DNN representations by constructing representation dissimilarity matrices and correlating their corresponding entries. The resulting Spearman score quantifies similarity between the tasks optimized by the models.

  • RSA relates models by comparing dissimilarity matrices built from pairwise relationships among their representations.
  • The paper introduces RSA as a tool for DNN relationship analysis and transfer-learning model selection.
  • For each DNN, selected images are forwarded through the model, and each image pair receives dissimilarity score 1−ρ, where ρ is Pearson correlation.
  • The method computes Spearman correlation between corresponding upper or lower triangular RDM entries to obtain task similarity.
  • The resulting correlation quantitatively measures similarity between the tasks for which the two DNNs were optimized.

4. RSA for Task Taxonomy and Transfer Learning

The experiments use RSA to build task taxonomies, test whether rankings are robust to smaller models and datasets, and compare RSA rankings with transfer performance. They evaluate both Taskonomy tasks and Pascal VOC semantic segmentation.

  • Task Taxonomy: The taxonomy experiment clusters 20 selected Taskonomy tasks using a pairwise RSA similarity matrix computed from representations of 500 sampled images.
  • Task Taxonomy: Hierarchical clustering tests whether RSA groups tasks according to types such as 2D, 3D, and semantic vision.
  • Task Taxonomy: RSA is symmetric, but the paper states that symmetry does not affect relative task-similarity rankings.
  • Model and Data Size: The study tests whether RSA rankings depend on model size or training-data amount by comparing smaller task-specific models with Taskonomy models.
  • Transfer Learning: On Pascal VOC semantic segmentation, RSA similarity showed a strong relationship with transfer performance, including cases where a different task model outperformed the same-task model.

5. Experimental set-up

The experiments use Taskonomy data, task-specific encoder–decoder models, and RSA representations to compare tasks. They also examine smaller models, earlier encoder layers, task labels, and Pascal VOC transfer learning.

  • Datasets: Taskonomy provides over 4 million indoor images with annotations for 26 image tasks; this study selects 20 single-image tasks.
  • Datasets: RSA uses 500 randomly selected Taskonomy training images as conditions for generating task-specific representations and RDMs.
  • Models: The Taskonomy models use a shared ResNet-50 encoder with channel compression to 8 dimensions and task-specific decoders.
  • Models: The compressed encoder output is used as the RSA representation so architecture and input images remain consistent across tasks.
  • Models: The study additionally evaluates earlier encoder layers, task labels, smaller Taskonomy models, and Pascal VOC models for similarity and transfer-learning analyses.

6. Results

RSA produces task similarities that recover meaningful visual-task groupings, remain stable across model and dataset sizes, and predict transfer-learning performance. Across Taskonomy and Pascal VOC, similarity rankings identify strong transfer-learning initializations without requiring transfer training for model selection.

  • Task similarity using RSA: RSA clustering groups Taskonomy tasks according to visual categories such as 2D, 3D, and semantic tasks.The method applies hierarchical clustering to a 20 × 20 RSA similarity matrix.
  • Task similarity using RSA: Similarity patterns become more task-specific at deeper encoder layers, whereas initial-layer representations are highly similar across tasks.The study compares representations from ResNet-50 blocks 1–4 and task labels.
  • Task similarity using RSA: RSA and Taskonomy produce similar task clusters, with matrix correlations of ρ = 0.62 and r_s = 0.65.The approaches differ for some tasks, including room layout and denoising.
  • Does model size impact similarity score?: Small-model task rankings closely match Taskonomy rankings, with mean correlations of ρ = 0.84 and r_s = 0.85.A second comparison using small models reports correlations of ρ = 0.85 and r_s = 0.88.
  • Model selection for transfer learning: For 16 of 17 Taskonomy tasks, RSA selects a transfer-learning model ranked in the top five by transfer performance.Across 17 tasks, RSA and transfer-learning rankings have mean correlations of ρ = 0.70 and r_s = 0.76.
  • Model selection for transfer learning: On Pascal VOC, higher RSA similarity generally corresponds to higher transfer performance, while semantic-segmentation and several 2D source models receive lower similarity scores.The highest-performing initializations include object class, scene class, occlusion edges, and semantic segmentation; autoencoding and vanishing point perform below training from scratch.

7. Conclusion

The paper presents RSA as an efficient alternative for measuring task similarity and selecting models for transfer learning. Across Taskonomy and Pascal VOC semantic segmentation, RSA similarity closely relates to transfer performance and can identify strong models under domain shift.

  • Method: RSA obtains task similarity from learned representations without further training, unlike the earlier Taskonomy approach.The method uses pre-trained models to compute similarity between computer vision tasks.
  • Efficiency: Model and training-dataset size do not significantly affect relative task-similarity rankings, enabling computational and memory savings.Small models and few data samples can produce rankings comparable to state-of-the-art models.
  • Transfer learning: RSA similarity is closely related to transfer-learning performance on both Taskonomy and Pascal VOC semantic segmentation.The relationship supports using RSA to inform transfer-learning model selection.
  • Transfer learning: Under domain shift, the model trained on the same task may not be the best transfer-learning source, while RSA can help select high-performing models.The paper identifies model selection as an application of the observed relationship between RSA similarity and transfer performance.
  • Applications: The method may also apply to multi-task models and architecture selection.These are identified as potential problem settings beyond the demonstrated applications.

Supplementary materials: Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning

The supplementary material adds details on small models, comparisons between RSA and Taskonomy, and the consistency of RSA rankings with transfer-learning performance.

  • Supplementary scope: The supplementary material accompanies the paper Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning.The supplied heading identifies the supplementary material and its paper.
  • Supplementary scope: The supplementary material reports small-model details, comparisons with Taskonomy, and consistency between RSA similarity rankings and transfer-learning performance.These topics are organized across the supplementary sections.
  • Supplementary scope: The reported supplementary analyses cover task-similarity matrices and clustering in addition to ranking consistency.The material compares RSA-based results with the Taskonomy approach.

S1. Small models for task taxonomy

Small models can reproduce task-similarity rankings well, especially for 2D tasks, although correlations are lower for 3D tasks and training differences constrain exact comparisons.

  • Small-model setup: 14 selected tasks use spatial outputs optimized with L1, L2, or triple-metric loss, allowing a shared decoder except for the final layer.The small-model architecture is reported in Table S1.
  • Similarity results: Pearson’s ρ exceeds 0.97 for most 2D tasks when comparing small-model and Taskonomy similarity rankings, except segment2d.The rankings also look visually similar for most 2D tasks.
  • Similarity results: Pearson’s ρ exceeds 0.77 for all 3D tasks, but remains lower than for 2D tasks.The supplementary comparison reports weaker, though still high, agreement for 3D tasks.
  • Limitations: Training differences and use of only a dataset subset prevent exact replication of the Taskonomy training setup.The Taskonomy training code was not publicly available, and the small models use a subset of the full dataset.
  • Similarity results: The overall Pearson correlation between small-model and Taskonomy similarity rankings for the evaluated tasks is 0.8510.The similarity matrices preserve higher similarity among corresponding 3D tasks, although within-3D scores are lower.

S2. Taskonomy[34] vs RSA(Our approach)

RSA produces clustering broadly consistent with Taskonomy while offering directly interpretable similarity scores between -1 and 1, though clustering differences may reflect unspecified methodological choices.

  • Clustering comparison: Almost all of the 20 selected single-image tasks share the same cluster under RSA and Taskonomy, except room layout and denoise.The comparison reports broad agreement between the two clustering results.
  • Similarity comparison: Figure S1 compares similarity rankings from Taskonomy and small models across 14 tasks using Pearson’s ρ.The figure reports the correlation between the two rankings below each plot.
  • Clustering comparison: The clustering differences may arise from a clustering method in the Taskonomy work that was not specified.This is presented as a possible explanation rather than a demonstrated cause.
  • Similarity representation: RSA similarity scores lie between -1 and 1, making the similarity matrix easier to visualize and evaluate.The Taskonomy approach requires exponential scaling to bring its scores into a suitable visualization range.

S3. Transfer learning in Pascal VOC

RSA remains consistent across training stages, model sizes, and image-set sizes, while higher task-similarity rankings generally correspond to stronger Pascal VOC transfer performance.

  • Consistency of RSA: At 1/10 of final training, Pearson’s correlation with the final stage is 0.88, and after 1/2 it remains above 0.99.These results support using early-training models for RSA-based task similarity.
  • Consistency of RSA: Task similarity from a small encoder closely matches the Taskonomy encoder architecture, with ρ = 0.95 and rs = 0.96.The correlation suggests small models can support RSA for selecting an initialization model for a new task.
  • Consistency of RSA: After 400 images, Pearson’s correlation with the 2000-image task-similarity ranking is always above 0.99, suggesting around 500 images suffice for RDM computation.The analysis varied the image count from 100 to 2000.
  • Transfer learning performance: Across 17 single-image tasks, transfer performance generally decreases as task-similarity ranking decreases, although the curve is not perfectly monotonic.Denoising, autoencoding, and colorization were excluded because they require modified input.
  • Transfer learning performance: Higher-ranked tasks such as object class, surface normals, and segment25d showed high transfer performance, whereas lower-ranked tasks such as autoencoding and vanishing point showed lower performance.Transfer performance was measured using mIoU.
Loading 1904.11740v1…