Source-linked AI summary
Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
Zhongzheng Ren, Yong Jae Lee
TL;DR
The paper targets the cost of annotated data and the limited transferability of single-task features. It trains a synthetic-image multi-task network to predict physical cues while adversarially adapting features to real images, and reports better transfer than single-task baselines with highly competitive benchmark performance.
Problem
Large-scale annotations are costly, while single-task feature learning may not produce general-purpose representations and synthetic imagery differs from real images.
Method
A self-supervised multi-task network predicts depth, surface normals, and instance contours from synthetic images while adversarially minimizing feature-space differences between synthetic and real images.
Results
The multi-task model outperforms all single-task models on VOC classification and detection, while experiments report highly competitive transfer-learning performance.
Takeaways & Limitations
Learning complementary physical cues from synthetic imagery can produce transferable high-level visual representations for real-world tasks.
Takeaways & Limitations
The adversarial optimization can become unnecessarily difficult because the discriminator is trained to fool in both ways; stabilizing methods are left for future study.
Abstract
from arXiv · showhide
In human learning, it is common to use multiple sources of information jointly. However, most existing feature learning approaches learn from only a single task. In this paper, we propose a novel multi-task deep network to learn generalizable high-level visual representations. Since multi-task learning requires annotations for multiple properties of the same training instance, we look to synthetic images to train our network. To overcome the domain difference between real and synthetic data, we employ an unsupervised feature space domain adaptation method based on adversarial learning. Given an input synthetic RGB image, our network simultaneously predicts its surface normal, depth, and instance contour, while also minimizing the feature space domain differences between real and synthetic data. Through extensive experiments, we demonstrate that our network learns more transferable representations compared to single-task baselines. Our learned representation produces state-of-the-art transfer learning results on PASCAL VOC 2007 classification and 2012 detection.
1. Introduction
The paper addresses the cost and limited scalability of manually annotated visual-learning datasets by combining synthetic imagery, multiple physical cues, and feature adaptation to real images. Its network learns transferable representations through self-supervised multi-task prediction and adversarial domain adaptation.
- Large annotated datasets are costly to build and require continual expansion as models overfit existing benchmarks.
- Multi-task learning combines complementary supervision to encourage general features, but natural images would require expensive annotations for depth, surface normals, and segmentations.
- Synthetic images provide scalable, inexpensive access to controllable physical-property annotations for representation learning.The paper highlights control over objects, scenes, lighting, physics, illumination, and weather conditions.
- The network predicts depth, surface normal, and instance contour maps while using adversarial feature-level domain adaptation to address the synthetic-to-real gap.
- Experiments on ImageNet and PASCAL VOC show that the multi-task approach outperforms alternative single-task baselines and is highly competitive with state-of-the-art methods.
2. Related work
Prior work applies synthetic data, self-supervised representation learning, multi-task learning, and domain adaptation to visual problems, but this paper combines them for general-purpose feature learning. Its distinguishing approach adapts synthetic-trained features to real images in feature space while learning multiple tasks simultaneously.
- Synthetic data has supported diverse vision tasks, whereas this work uses it to learn general-purpose visual representations.
- Self-supervised representation methods use image cues, motion, reconstruction, spatial relations, inpainting, or adversarial objectives to learn features without standard labels.
- Earlier multi-task approaches faced sequential training, heterogeneous data requirements, or limited transferability, motivating a simultaneous multi-task formulation.
- Domain adaptation methods align representations across domains, while some GAN-based approaches operate directly at the pixel level.
- This paper performs adversarial domain adaptation in feature space so synthetic-trained representations can transfer to real images.
- The multi-task approach is presented as learning better representations than single-task learning, building on prior multi-task vision research.
3. Approach
The approach jointly learns visual features from synthetic images through contour, depth, and surface-normal prediction, while adversarially reducing feature-space differences between synthetic and real images. The adapted base features are then transferred to real-world tasks.
- Multi-task feature learning: The network jointly predicts instance contours, depth, and surface normals from each synthetic image to learn general-purpose visual features.These tasks provide complementary supervision from synthetic imagery.
- Multi-task feature learning: Synthetic instance masks are converted into semantic contour maps, avoiding noisy instance names while emphasizing object boundaries over texture and shadows.Canny edges extracted from instance masks represent object contours rather than low-level internal edges.
- Multi-task feature learning: Depth prediction uses clean synthetic depth maps and a scale-invariant loss comparing predicted and ground-truth log-depth maps.The loss combines pixelwise differences with a term encouraging errors to share a consistent sign.
- Multi-task feature learning: Surface-normal prediction uses the inverse dot product between predicted and ground-truth normal maps as its loss.The paper relates surface-normal estimation to depth because the two tasks are physically connected.
- Unsupervised feature-space domain adaptation: Adversarial adaptation trains a discriminator on synthetic and real features while updating the base network and task heads to fool it and preserve task performance.Real images directly update the discriminator; the base network is updated through the adversarial objective and synthetic task losses.
- Unsupervised feature-space domain adaptation: Using mid-level ‘conv5’ features for domain alignment produces the best transfer-learning results among the evaluated feature spaces.After training, the base network is transferred and fine-tuned on real-world tasks.
4. Results
Experiments show that synthetic-data pretraining can yield transferable visual features for real-world recognition, especially when complementary tasks and feature-level domain adaptation are combined. Ablations identify multi-task learning, adaptation placement, and dataset scale as important factors, while also exposing limits from domain and rendering differences.
- Transfer learning: The learned features transfer effectively to PASCAL VOC: they outperform prior methods in classification, rank second on VOC 2007 detection, and achieve the best result on VOC 2012 detection.The model never used real images for pretraining; real images were used only for domain adaptation.
- Transfer learning: On ImageNet classification, synthetic indoor-image pretraining improves over Gaussian and Krähenbühl et al. initialization but remains below the state of the art.The authors attribute the gap partly to differing high-level semantics and unseen categories, while noting usefulness through conv4.
- Ablations: The multi-task model outperforms Edge, Depth, and Surf. single-task models on both VOC classification and detection, indicating complementary task information.The comparison supports multi-task learning as beneficial for feature learning.
- Ablations: 67.4% VOC classification accuracy with conv5 adaptation exceeds 65.6% without adaptation, while adaptation at conv1 hurts and conv5 generally performs best.Very low layers expose rendering artifacts, whereas very high layers retain difficult semantic differences between synthetic and real datasets.
- Ablations: Increasing the synthetic dataset from 0.5 million to 1.5 million images improves both classification and detection by 0.5-0.6% points.This result supports the scalability of the self-supervised setup with additional synthetic data.
- Ablations: Fooling the discriminator in both directions slightly reduces performance relative to adapting synthetic features toward real ones, although it remains better than no adaptation.The authors suggest the two-way objective makes optimization harder and does not directly improve the synthetic pretext tasks.
5. Conclusion
The paper presents cross-domain multi-task feature learning from synthetic imagery as a first step toward general-purpose visual representations.
- The proposed network learns visual representations from synthetic imagery while adapting its representation to real images via adversarial learning.
6. Appendix
The appendix details AlexNet- and VGG16-based architectures composed of shared base networks, task-specific heads, and domain discriminators, with design choices for spatial prediction and adversarial training.
- AlexNet-based architecture: The AlexNet-based network comprises a base network, bottleneck network, task heads, and domain discriminator.The base and bottleneck can be converted into a standard AlexNet for transfer learning experiments.
- AlexNet-based architecture: The AlexNet base accepts 227×227×3 images, changes pool5 stride from 2 to 1, and uses dilated convolution in fc6 to enlarge the receptive field.These choices preserve spatial information and increase receptive-field size.
- Task outputs: Three deconvolutional layers recover full-sized outputs, with three channels for surface normals and one channel each for depth and instance contours.The output layer is also deconvolutional.
- Domain discriminator: The domain discriminator takes conv5 features as input and uses a 6 × 6 patch-discriminator output, Leaky ReLU, and batch normalization for adversarial training.The discriminator has three convolutional layers, D1–D3, with Leaky ReLU slope 0.2.
- VGG16-based architecture: The VGG16-based network omits the bottleneck to save memory and uses skip connections from intermediate convolutional layers to task-head deconvolutions for accurate pixel-level predictions.Its base accepts 224×224×3 images; listed skips include conv2_2 → Deconv4 and conv3_3 → Deconv3.