Source-linked AI summary
Cross-stitch Networks for Multi-task Learning
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, Martial Hebert
TL;DR
Existing multi-task ConvNet methods require task-specific architecture searches because the useful amount of sharing varies across tasks. This paper introduces end-to-end trainable cross-stitch units that combine task activations, achieving better performance than searched architectures and baseline methods, especially for data-starved categories. The approach is scoped to tasks sharing a single input.
Problem
Existing multi-task approaches require enumerating architectures specific to each task set, while the appropriate amount of sharing depends on the tasks.
Method
Cross-stitch units learn linear combinations of activations between task networks and integrate them into an end-to-end trainable ConvNet.
Results
Cross-stitched networks outperform baseline methods and networks found by brute-force architecture search, with considerable gains for data-starved categories.
Takeaways & Limitations
A single cross-stitched network can learn an optimal combination of shared and task-specific representations across different task types.
Takeaways & Limitations
The study considers only tasks that use the same single input, such as an image rather than an image and a depth map.
Abstract
from arXiv · showhide
Multi-task learning in Convolutional Networks has displayed remarkable success in the field of recognition. This success can be largely attributed to learning shared representations from multiple supervisory tasks. However, existing multi-task approaches rely on enumerating multiple network architectures specific to the tasks at hand, that do not generalize. In this paper, we propose a principled approach to learn shared representations in ConvNets using multi-task learning. Specifically, we propose a new sharing unit: "cross-stitch" unit. These units combine the activations from multiple networks and can be trained end-to-end. A network with cross-stitch units can learn an optimal combination of shared and task-specific representations. Our proposed method generalizes across multiple tasks and shows dramatically improved performance over baseline methods for categories with few training examples.
1. Introduction
Multi-task learning can improve recognition by sharing representations, but the effective amount and location of sharing depend on the tasks. Cross-stitch units are proposed to learn this combination without enumerating task-specific architectures.
- Multi-task learning leverages multiple supervisory tasks and shared representations to improve performance on recognition problems.
- Different tasks favor different amounts of shared and task-specific representation, spanning fully shared networks to separate task-specific networks.
- Split conv4 performs best for semantic segmentation and surface normal prediction, while detection favors Split conv2 and attribute prediction favors Split fc6.
- The best multi-task architecture depends on the task pair, so selecting it requires searching across possible split locations.
- Cross-stitch units provide a single network that captures Split architectures and more while learning an optimal shared/task-specific combination.
2. Related Work
Prior multi-task ConvNet methods jointly model related tasks by sharing some layers and retaining task-specific features. Their split architectures are usually selected experimentally for each task set, motivating a more general learned approach.
- Multi-task learning in computer vision commonly models related tasks jointly through shared and task-specific ConvNet features.
- The paper uses linear combinations of activation maps to model shared representations at each network layer.
- Existing methods typically choose a split architecture by experimenting with multiple layer splits and selecting the best one for the tasks.
- Because the best split can vary with the tasks, new task combinations require exploring new architectures.
3. Cross-stitch Networks
Cross-stitch Networks learn task sharing in ConvNets by combining activations from two task-specific streams. Their learned weights can interpolate between task-specific and shared representations during end-to-end training.
- Cross-stitch units learn shared representations for multi-task ConvNets as linear combinations of task activations.
- Cross-stitch units are integrated into a ConvNet and trained end-to-end through backpropagation of the multi-task loss.
- The architecture is restricted to tasks using the same single input, such as an image rather than an image and a depth map.
- Split architectures range from separate task-specific networks to highly shared networks, depending on where the network is split.
- Cross-stitch units learn how much sharing to use instead of enumerating possible split locations for each new task pair.
- Setting cross-task weights to zero makes layers task-specific, while larger cross-task weights produce more shared representations.
4. Design decisions for cross-stitching
Cross-stitch units combine task activations to learn shared and task-specific representations in a single multi-task ConvNet. Their design includes initialization choices intended to support stable learning and regularize tasks, including data-starved ones.
- Cross-stitch units combine activation maps from two task networks to learn representations useful for both tasks.The sub-network directly supervised by task A is network A, while network B receives direct supervision from task B.
- The units provide indirect cross-task supervision by learning shared representations, which can regularize tasks with fewer labels.
- The design uses α values initialized in [0, 1] so output activations remain on the same order of magnitude as input activations.The initialization is described as important for stable learning.
- Cross-stitch units can model shared representations as linear combinations of input activation maps.
5. Ablative analysis
The ablations examine initialization, learning-rate scaling, and learned sharing patterns for cross-stitched networks on semantic segmentation and surface normal prediction. Results favor higher cross-stitch learning rates, task-specific initialization, and layer-dependent balances between shared and task-specific representations.
- Experimental setup: The ablation studies use semantic segmentation and surface normal prediction on NYU-v2 with standard train/test splits.
- Initialization: The cross-stitched network remains robust across different convex-combination α initializations.Higher αS values bias the units toward task-specific representations.
- Learning rates: 10^2–10^3 learning-rate scaling for cross-stitch units gives the best range after 10k iterations, while higher scaling makes the loss diverge.Cross-stitch units receive higher learning rates because their gradient updates are too small relative to base-network parameter updates.
- Network initialization: Task-specific initialization outperforms ImageNet initialization for both tasks.The comparison trains the task-specific initialized network for 10k iterations and the ImageNet initialized network for 30k iterations.
- Visualization of learned combinations: Across pool1, pool5, and fc7, learned αS and αD values reveal layer- and task-dependent preferences for sharing.Both tasks prefer more task-specific representations at pool5, while surface normal prediction shows a preference for shared representations.
6. Experiments
Experiments evaluate cross-stitch networks across semantic segmentation, surface normal prediction, object detection, and attribute prediction, comparing them with one-task, ensemble, and searched split-architecture baselines. The method improves performance, particularly for categories with few training examples, while task pairs can prefer different balances of shared and task-specific representations.
- Experimental setup: Cross-stitch networks are evaluated on semantic segmentation and surface normal prediction using NYU-v2, and object detection and attribute prediction using PASCAL VOC 2008.The experiments use standard metrics, including Average Precision for detection and attribute prediction.
- Experimental setup: The method uses two AlexNet architectures combined by cross-stitch units after pooling layers, with jointly trained task losses.Attribute loss is scaled by 1/64 to balance it against the detection loss.
- Semantic segmentation and surface normals: Cross-stitch networks outperform one-task networks, ensembles, and the best searched split architecture for semantic segmentation and surface normal prediction.The cross-stitched network performs better than an ensemble with twice its parameters, and its learned sharing searches beyond brute-force split architectures.
- Data-starved categories for segmentation: Cross-stitch units considerably improve performance for data-starved segmentation categories such as bag and whiteboard.Segmentation categories differ substantially in their numbers of pixel labels, with some classes having many more instances than others.
- Object detection and attribute prediction: Detection prefers a shared representation, whereas attribute prediction prefers a task-specific representation in the best split architectures.The best detection result uses Split fc7, while the best attribute result uses Split conv2.
- Data-starved categories for attribute prediction: Attribute prediction gains are 4.6% and 4.3% mAP for the top 10 and 20 attributes with the least training data, respectively.The gain is measured relative to the baseline one-task attribute-prediction network.
7. Conclusion
The paper introduces cross-stitch units as a generalized, end-to-end approach for learning shared representations in multi-task ConvNets. They eliminate per-task architecture search and improve performance over baseline methods, especially for data-starved categories.
- Conclusion: Cross-stitch units model shared representations as learnable linear combinations and generalize across different task types.They are trained end-to-end within a ConvNet.
- Conclusion: A cross-stitched network learns an optimal combination of shared and task-specific representations without searching architectures separately for each task.The approach is presented as a replacement for enumerating multiple split architectures.
- Conclusion: Cross-stitch networks show considerable gains over baseline methods for categories with few training examples.The conclusion identifies data-starved categories as a particularly strong setting for the method.
- Conclusion: The paper identifies cross-stitch placement and weight constraints as directions for future study.These properties are not resolved by the presented work.