PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross Modal Distillation for Supervision Transfer

Saurabh Gupta, Judy Hoffman, Jitendra Malik

arXiv:1507.00448v2cs.CV

TL;DR

Learning generic representations is difficult for modalities with limited labeled data. The paper transfers mid-level representations from a labeled modality to an unlabeled paired modality, improving recognition for depth and optical flow. Its experiments use established CNN architectures rather than validating alternatives for these modalities.

  • Problem

    Image modalities beyond RGB have far fewer labeled images, raising whether generic representations require similarly large annotation efforts for each modality.

  • Method

    Supervision transfer uses paired images to train an unlabeled-modality CNN to reproduce mid-level representations learned from a labeled modality.

  • Results

    Supervision transfer improves depth-only and RGB-plus-depth object detection on NYUD2 and optical-flow-only mean average precision on JHMDB.

  • Takeaways & Limitations

    The method provides useful feature hierarchies for unlabeled modalities, supports finetuning, and can produce representations complementary to the source modality.

  • Takeaways & Limitations

    The experiments use an architecture similar to prior work and do not validate different CNN architectures for depth and flow images.

Abstract

from arXiv · show

In this work we propose a technique that transfers supervision between images from different modalities. We use learned representations from a large labeled modality as a supervisory signal for training representations for a new unlabeled paired modality. Our method enables learning of rich representations for unlabeled modalities and can be used as a pre-training procedure for new modalities with limited labeled data. We show experimental results where we transfer supervision from labeled RGB images to unlabeled depth and optical flow images and demonstrate large improvements for both these cross modal supervision transfers. Code, data and pre-trained models are available at https://github.com/s-gupta/fast-rcnn/tree/distillation

1. Introduction

The paper addresses how to learn generic representations for image modalities with far less labeled data than RGB. It transfers mid-level supervision through paired images and reports improvements for depth and optical flow recognition tasks.

  • Motivation: Image modalities such as depth, infrared, aerial, LIDAR, optical flow, and stereo have substantially less labeled data than RGB.The paper frames this disparity as a question of whether each modality requires similarly large annotation efforts.
  • Method: Supervision transfer trains a CNN for an unlabeled modality to reproduce mid-level representations learned from a labeled paired modality.The architecture uses paired images, such as RGB and depth, to transfer learned supervision.
  • Results: On NYUD2, depth-only object detection improved from 34.2% to 41.7%, while RGB-plus-depth detection improved from 46.2% to 49.1%.These results transfer ImageNet supervision from RGB images to depth images.
  • Results: On JHMDB action detection, optical-flow-only mean average precision improved from 31.7% to 35.7% without supervised pre-training.This additional experiment demonstrates transfer beyond RGB-to-depth supervision.
  • Relation to distillation: The method generalizes distillation by transferring supervision across modalities and at arbitrary semantic levels using paired images.Unlike standard distillation described here, the approach is not restricted to the same modality or the final prediction layer.

2. Related Work

Prior work largely adapts representations within the same visual modality, whereas this paper focuses on transferring supervision across different modalities.

  • Same-modality adaptation: Earlier visual-domain adaptation work studied transformations between domains sharing the same modality, initially using shallow representations.Later approaches introduced deep architectures with domain-specific feature extraction and intermediate representations.
  • Deep representations: Supervised CNN features became a generic basis for specialization to tasks and domains, with traditional and unsupervised adaptation methods built around them.These approaches generally adapt deep representations by reducing discrepancies between domains.
  • Cross-modal transfer: The paper distinguishes its approach from intramodality adaptation because cross-modal transfer lacks an obvious initial target-domain feature representation.It identifies cross-modal transfer as a less explored problem than same-modality domain adaptation.

3. Supervision Transfer

Supervision transfer learns a rich representation for an unlabeled modality by matching features from a labeled source modality on paired images, with transfer possible at selected representation layers.

  • Setup: The setup assumes a labeled source modality, an unlabeled target modality, and paired images showing the same scenes in both modalities.The source representation is layered, although the method is stated to also work with non-layered representations.
  • Source representation: The source representation maps source-modality images to feature vectors at each selected layer.These learned source features provide the supervision signal for training the target representation.
  • Training objective: The target network is trained so its output at layer L matches the source representation at a chosen layer i* for each paired image.Similarity is measured with an appropriate loss, such as Euclidean loss.
  • Feature alignment: When source and target features have different dimensions, a simple transformation such as a linear or affine map embeds the source features.The transformation aligns representations before similarity is measured.
  • Transfer definition: The method defines supervision transfer as moving supervision from layer i* of the source representation to layer L of the target representation.This formulation permits transfer at arbitrary semantic levels rather than only at the final prediction layer.
  • Supported outcomes: Experiments report good target feature hierarchies, further improvement through finetuning, and complementarity with source representations when the modalities permit.The method is intended to learn target representations without a large labeled target dataset.

4. Experiments

Experiments evaluate supervision transfer from labeled RGB to paired depth and optical flow modalities, using downstream object detection and representation analyses. The results compare transfer with standard initialization, assess transferred features, and examine modality-specific learned filters.

  • Experimental framework: Experiments use RGB as the labeled source modality and depth or optical flow as target modalities, leveraging paired images where labeled target data are scarce.The framework evaluates NYUD2 RGB-depth pairs and JHMDB RGB-optical-flow pairs, with ImageNet providing labeled RGB supervision.
  • Experimental framework: Object detection on NYUD2 uses 19 categories and mean average precision, with Fast R-CNN models evaluated on standard validation and test splits.The dataset contains paired RGB and depth images from multiple scenes, and supervision transfer uses only frames from the corresponding training split.
  • Depth transfer: 44.39% mAP improves the prior reported NYUD2 result of 38.80% using the same CNN network and supervision, establishing the Fast R-CNN baseline used thereafter.The experiments use RGB-D box proposals and finetune all layers under the stated Fast R-CNN training setup.
  • Depth transfer: 29.7% mAP from supervision transfer exceeds 25.1% from the compared baseline, while adapting the RGB supervisory network raises performance further to 30.5%.The transferred depth layers are trained from random initialization on 5K paired RGB-depth images, and the adapted-supervisor scheme is used subsequently.
  • Representation analysis: Learned depth filters differ from RGB filters, and high-scoring RGB and depth patches for paired neurons correspond to parts of the same object class.In contrast, depth CNNs initialized by copying RGB weights show little first-layer filter change; the paper also reports transfer to optical flow with distinct learned filters.

1. Quality of transferred pool5 representation (2A,

The transferred pool5 representation remains useful when detector training is restricted to later layers. Compared with copied ImageNet weights, it suffers only moderate degradation when frozen.

  • Quality of transferred pool5 representation: 30.0% mAP with frozen transferred conv1–pool5 layers declines only moderately from 30.5% with full finetuning, indicating discriminative depth features at pool5.Only fc6, fc7, and classifier layers are learned during Fast R-CNN training in this evaluation.
  • Quality of transferred pool5 representation: 19.8% mAP with frozen copied ImageNet weights is substantially below the 30.0% achieved by frozen transferred features.The comparison tests whether the transferred pool5 representation is useful independently of later detector-layer adaptation.

2. Improved transfer using better supervising net-

Supervision transfer benefits from stronger supervising networks and intermediate transfer points, while producing useful depth representations for detection and zero-shot modality adaptation.

  • Improved transfer using better supervising net-: 33.6% versus 30.5%: VGG supervision improves depth detection after finetuning all layers compared with AlexNet supervision.With parameters frozen through pool5, performance also rises from 30.0% to 32.2%.
  • Improved transfer using better supervising net-: 37.0%: combining RGB and transferred depth representations outperforms RGB alone and HHA initialization on object detection.RGB alone achieves 22.3%, while HHA initialization reaches 33.8% and AlexNet-to-AlexNet transfer reaches 35.6%.
  • Transfer-point analysis: Intermediate-layer transfer performs best at conv3, conv4, and pool5, while performance deteriorates at higher layers such as fc6 and fc7.The authors attribute higher-layer degradation to the size of the paired-image supervision dataset and expect richer data to help.
  • Transfer-point analysis: Transferring simultaneously at conv3 and fc7 improves performance over either transfer point alone, suggesting supervision closer to learned parameters helps.The paper defers broader exploration of multiple transfer-point choices.
  • Input representation: 29.2% versus 30.5%: disparity images underperform HHA images, but the gap is smaller than prior depth-representation differences.The comparison uses the same experimental setting except for the input representation.
  • Zero-shot detection: 30.4% mean AP: a zero-shot depth detector transferred from MS COCO performs comparably to the RGB detector's 33.0% without NYUD2 annotations.Combining RGB and depth predictions reaches 37.6%.

A. Supplementary Material

The supplementary material reports per-category average precision and presents sampled detections and segmentation outputs for the studied NYUD2 categories.

  • Per-category average precision is reported for summary tables on the test sets.
  • Sample detections and SDS outputs are shown for bed, chair, sofa, toilet, and table.Eighteen detections are sampled uniformly from the top k detections for each category, with k set to 0.75× the number of instances.

B. Document Changelog

The document changelog records added multimodal discussion, visualizations, and experiments, while the supplementary tables provide detection results for NYUD2 and JHMDB.

  • Document Changelog: The revised document adds discussion of multimodal literature and visualizations of neural activations in Figure 2(g-i).
  • Document Changelog: Additional experiments examine intermediate-layer quality, transfer point, HHA versus disparity embeddings, and zero-shot depth detection.
  • Supplementary Figures: Figures 3–7 show sample detections and segmentation masks for bed, chair, sofa, toilet, and table on the NYUD2 test set.
  • Supplementary Tables: Table 8 reports per-class region detection AP r (%) for NYUD2 SDS experiments.
  • Supplementary Tables: Table 9 reports object detection AP (%) on NYUD2 and compares depth-only and RGB-plus-depth configurations.The table reports improvements from 34.2% to 41.7% for depth-only detection, and from 44.4% to 47.1% or 46.2% to 49.1% when adding learned depth features to RGB features.
  • Supplementary Tables: Table 10 reports action detection AP (%) on JHMDB, comparing supervision transfer with random initialization and fully supervised pre-training.The caption states that supervision transfer reaches more than half the way toward fully supervised pre-training.
Loading 1507.00448v2…