Source-linked AI summary
UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory
Iasonas Kokkinos
TL;DR
UberNet addresses how one CNN can jointly learn diverse vision tasks despite incomplete annotations and limited memory. It combines selective losses with a shared, lightweight multi-task architecture and reports competitive results across seven task groups at 0.6-0.7 seconds per frame on a GPU.
Problem
A single network must handle diverse vision tasks even though datasets provide incomplete, disjoint annotations and GPU memory limits grow with task-specific layers.
Method
UberNet combines a shared CNN trunk and task-specific linear operations with a loss that uses only ground truth available for each training sample.
Results
The network jointly addresses boundary, normal, saliency, segmentation, proposal, and detection tasks with competitive performance at 0.6-0.7 seconds per frame on a GPU.
Takeaways & Limitations
The techniques support combining incoherently annotated datasets and scaling training to many diverse tasks in one unified CNN.
Takeaways & Limitations
The authors report a bounded learning capacity in the common CNN trunk, with task performance acting like communicating vessels under their settings.
Abstract
from arXiv · showhide
In this work we introduce a convolutional neural network (CNN) that jointly handles low-, mid-, and high-level vision tasks in a unified architecture that is trained end-to-end. Such a universal network can act like a `swiss knife' for vision tasks; we call this architecture an UberNet to indicate its overarching nature. We address two main technical challenges that emerge when broadening up the range of tasks handled by a single CNN: (i) training a deep architecture while relying on diverse training sets and (ii) training many (potentially unlimited) tasks with a limited memory budget. Properly addressing these two problems allows us to train accurate predictors for a host of tasks, without compromising accuracy. Through these advances we train in an end-to-end manner a CNN that simultaneously addresses (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection. We obtain competitive performance while jointly addressing all of these tasks in 0.7 seconds per frame on a single GPU. A demonstration of this system can be found at http://cvn.ecp.fr/ubernet/.
1. Introduction
UberNet targets the challenge of unifying many vision tasks despite diverse datasets and limited GPU memory. It introduces training techniques for an end-to-end network that covers a broad task set with competitive performance.
- 15 vision tasks often use separate task-specific CNNs, motivating a unified architecture for simpler and faster multi-task systems.
- No single dataset provides annotations for all desired tasks, so training must combine disjoint datasets with task-specific ground truth.
- The proposed loss uses only available ground truth for each sample, enabling asynchronous end-to-end updates across incompletely annotated datasets.
- Memory complexity independent of the number of tasks enables end-to-end training for a practically unlimited task set under limited GPU memory.
- 0.6-0.7 seconds per frame on a GPU accompanies competitive results across boundary, normal, saliency, segmentation, proposal, and detection tasks.
2. UberNet architecture
UberNet combines a shared VGG-based CNN trunk with skip-layer, multi-resolution, and lightweight task-specific operations. These components produce task-specific responses while keeping the architecture suitable for diverse dense and detection tasks.
- Shared architecture: A minimal set of task-specific layers is placed on top of a common VGG-based CNN trunk.
- Skip layers: Skip layers pool intermediate and top-layer activations, preserving spatial detail that benefits boundary detection and can improve semantic segmentation.
- Skip-layer normalization: Batch normalization precedes intermediate-layer inner products, reducing the need for very low learning rates.
- Task-specific operations: Task-specific layers apply linear operations to skip-pooled activations, limiting task-specific memory and computation as tasks scale.
- Task-specific deviations: Atrous convolution produces output stride 8 instead of 16, moderately improving boundary detection and semantic segmentation.
- Multi-resolution processing: A shared-weight image pyramid handles scale variability, with concatenated multi-scale scores followed by a learned linear operation.
- Task-specific deviations: Detection uses a convolutional region proposal network followed by a fully connected classifier for 21 labels.
3. Multi-Task Training using Diverse Datasets
UberNet trains a shared CNN end-to-end across diverse task datasets by masking unavailable supervision and accumulating task-specific gradients asynchronously. This addresses missing labels and unstable small-batch updates while retaining task-specific losses and shared parameters.
- Diverse datasets: Missing ground truth is handled by setting unavailable task losses to zero rather than imputing labels.For δ_t,i = 0, the corresponding ground-truth value can be arbitrary without affecting the loss.
- Objective: The training objective sums regularized per-task losses, weighting each task with γ_t across shared trunk weights w_0 and task-specific weights w_t.The regularizer is an ℓ2 penalty, while L_t is the task-specific loss.
- Optimization challenge: Standard SGD can produce noisy or absent task-specific updates when few relevant examples occur in a minibatch, making large batches inefficient and poorly scalable.The issue arises because task-specific parameters receive gradients only from samples annotated for their task.
- Asynchronous SGD: The asynchronous backpropagation variant updates task parameters only after sufficiently many relevant images have been observed, using counters and cumulative gradients.Shared CNN parameters are updated for every training image, whereas task-specific parameters are updated according to task-relevant samples.
- Memory management: Low-memory training stores selected activations and uses them as anchors for smaller backpropagation subnetworks, reducing simultaneously stored activations and gradients.The supplied figure text describes the low-memory forward-pass setup but does not provide the complete complexity expression.
4. Memory-Bound Multi-Task Training
Training many tasks with a shared CNN is memory-intensive because naive backpropagation stores intermediate activations and task-specific states. The paper adapts low-memory backpropagation to exploit the shared trunk and decoupled task branches, reducing memory growth with the number of tasks.
- Memory challenge: Naive backpropagation stores all intermediate activations, requiring 2NL memory for a network with L layers.Stored activations are reused during the backward pass, but this makes the method memory-demanding.
- Memory-efficient backpropagation: Memory-efficient backpropagation stores activations at selected layers and recomputes subnetworks during backward passes.Stored activations act as anchor points, allowing gradients to begin at deeper layers while discarded activations are regenerated as needed.
- Memory challenge: For a shared trunk and T task branches, naive multi-task training requires 2N(LC + TLT) memory.The task-specific term grows with the number and depth of task branches.
- Multi-task adaptation: The multi-task adaptation reduces memory complexity to 2N√LC + LT, independent of the number of tasks T.Task branches clean up their allocated memory, while their gradient signals are accumulated at the branching point.
- Empirical capacity: With a 12GB GPU, the method supported seven tasks and 921x621 images, compared with 321x321 without memory-efficient training.The reported setup used a three-layer pyramid and skip-layer connections across network layers, pyramid levels, and tasks.
- Computation savings: Lazy gradient evaluation avoids task-specific branch computation when a training sample lacks that task’s ground truth, reducing computation time.Such branches contribute no gradient to the shared trunk for that sample.
5. Experiments
Experiments evaluate UberNet across diverse vision tasks and examine how multitask training affects individual-task performance. The system is broadly competitive, but performance can decline as tasks accumulate, especially under data or capacity constraints.
- UberNet is evaluated across boundary detection, normal estimation, saliency estimation, semantic segmentation, detection, and related tasks using shared experimental settings.The evaluation compares single- and multi-task networks, with common initialization and unioned task datasets.
- After semantic-segmentation and detection pretraining, VOC2007++ finetuning gives detection a small performance boost, suggesting useful additional supervision.
- Performance generally drops as tasks increase, although semantic segmentation and detection remain comparable to strong baselines even without CRF post-processing.
- Multi-scale processing improves performance over the single-scale architecture, while the two-task segmentation network achieves effectively the same performance as its starting configuration.
- Human-part performance drops by more than 10% in the multi-task case, with annotations available for only 3432 of 59552 training images.
- Boundary detection remains competitive despite reduced augmentation and learning-rate settings, while multi-task training causes an additional performance decline.
- Saliency estimation reaches a new state-of-the-art on PASCAL-S, and remains above the previous state-of-the-art in the multi-task setting.
- The experiments identify a trade-off among tasks: the shared CNN trunk appears capacity-limited, though optimization choices may also affect the observed results.
6. Conclusions and Future Work
The work presents techniques for scaling a unified CNN to many diverse tasks, while identifying task expansion, deeper architectures, and structured prediction as future directions. It also reports a trade-off in which improving normal estimation can reduce performance on other tasks.
- Conclusions: The proposed techniques enable a unified CNN to scale to many diverse tasks with memory complexity independent of task count.They also allow incoherently annotated datasets to be combined during training.
- Future Work: The authors identify adding tasks, using deeper architectures such as ResNets, and combining dense labeling with structured prediction as future work.Examples of additional tasks include symmetry, human landmarks, and texture segmentation.
- Conclusions: Improving normal estimation in the seven-task setting decreases performance on the remaining tasks.The table states that higher values are better for all tasks.
- Conclusions: The approach is positioned as an initial step toward jointly tackling multiple tasks by exploiting synergy between them.The paper connects this direction to prior work integrating segmentation and recognition.