PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Alex Kendall, Yarin Gal, Roberto Cipolla

arXiv:1705.07115v3cs.CV

TL;DR

Multi-task learning is highly sensitive to manually chosen loss weights, which are expensive to tune across regression and classification tasks. The paper learns task weightings from homoscedastic uncertainty in a unified scene-understanding model, improving classification from 59.4% to 63.4% over individual task models.

  • Problem

    Multi-task performance is highly sensitive to loss-weight choices, whose manual tuning is expensive for regression and classification objectives with different scales.

  • Method

    The paper learns task-dependent loss weights from homoscedastic uncertainty in a shared encoder-decoder model for semantic segmentation, instance segmentation, and pixel-wise depth regression.

  • Results

    63.4% versus 59.4% classification results, while uncertainty-weighted multi-task learning outperforms separately trained task models across scene-understanding tasks.

  • Takeaways & Limitations

    A unified multi-task model can jointly produce scene semantics and geometry while reducing computation and improving performance across multiple metrics.

  • Takeaways & Limitations

    The paper leaves unanswered whether multi-task learning has a single optimal weighting across all tasks.

Abstract

from arXiv · show

Numerous deep learning applications benefit from multi-task learning with multiple regression and classification objectives. In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. This allows us to simultaneously learn various quantities with different units or scales in both classification and regression settings. We demonstrate our model learning per-pixel depth regression, semantic and instance segmentation from a monocular input image. Perhaps surprisingly, we show our model can learn multi-task weightings and outperform separate models trained individually on each task.

1. Introduction

The paper addresses visual scene understanding as a multi-task problem spanning regression and classification tasks with different units and scales. It proposes homoscedastic uncertainty for principled loss weighting and jointly learns semantic segmentation, instance segmentation, and depth regression in one architecture.

  • Motivation: Visual scene understanding jointly requires learning geometry and semantics through regression and classification tasks with different units and scales.Combining tasks into one model reduces computation and allows real-time operation.
  • Problem: Performance depends strongly on appropriate task-loss weighting, while manual tuning is difficult, expensive, and sensitive to measurement scale.Prior approaches commonly used uniform or manually tuned weights.
  • Method: The proposed multi-task loss uses homoscedastic uncertainty as task-dependent weighting to balance regression and classification objectives.The method is designed to learn the weighting rather than require manual tuning.
  • Contributions: The paper reports superior performance compared with equivalent separately trained models while demonstrating the importance of loss weighting in multi-task deep learning.Its stated contributions include a principled loss, a unified architecture, and evidence for improved performance over separately trained models.
  • System: The unified architecture takes a single monocular RGB image and jointly produces semantic segmentation, instance segmentation, and per-pixel metric depth.Instance segmentation separates precise masks for individual objects, making it harder than semantic segmentation.

2. Related Work

Related work frames multi-task learning as shared-representation knowledge transfer that can improve task efficiency, accuracy, and generalization. Prior methods span diverse learning settings and vision tasks, but typically combine objectives with uniform or manually tuned loss weights.

  • Multi-task learning: Multi-task learning shares representations across complementary tasks so knowledge from one task can improve learning and generalization on others.It aims to improve learning efficiency and prediction accuracy compared with training separate models.
  • Multi-task learning: Existing approaches include fine-tuning, alternating task optimization, data-streaming methods, and reinforcement-learning methods that prevent forgetting.Multi-task learning has also been used to learn unsupervised features from multiple data sources.
  • Computer vision: Computer-vision research has combined classification with semantic segmentation or detection, while other systems jointly address detection, classification, semantic segmentation, or instance segmentation.Cross-stitch networks investigate combining multi-task neural activations, and Uhrig et al. learn semantic and instance segmentations under a classification setting.
  • Loss weighting: Previous simultaneous multi-task methods generally use uniform or manually tuned weights in a weighted sum of losses.The paper instead proposes combining losses through homoscedastic task uncertainty to balance multiple objectives.

3. Multi Task Learning with Homoscedastic Uncertainty

The section replaces expensive manual loss-weight tuning with homoscedastic uncertainty, which captures task-dependent relative confidence and yields principled adaptive weighting across regression and classification objectives. A likelihood-based loss uses uncertainty-dependent weights plus regularization, extending to arbitrary mixtures of task types without weights collapsing to zero.

  • Motivation: Prior work commonly combines task losses with a weighted linear sum, but performance is highly sensitive to manually selected weights that can take days to tune.The difficulty increases for larger models with more tasks.
  • Motivation: At an optimal weighting, a joint depth-and-semantic network can outperform separate single-task models, while nearby weights may worsen one task.The separate-task performances correspond to the plot boundaries at w = 0 and w = 1.
  • Uncertainty formulation: Homoscedastic uncertainty is task-dependent aleatoric uncertainty that remains constant across inputs but varies between tasks.It reflects relative confidence and depends on the task’s representation or unit of measure.
  • Likelihood-based loss: The likelihood-derived objective weights regression losses by 1/(2σ_i^2) and adds log σ_i terms, allowing uncertainty parameters to be learned jointly with model weights.Increasing σ_i lowers the corresponding loss contribution, while the logarithmic term regularizes excessive uncertainty.
  • Likelihood-based loss: For classification, scaling logits by a learned positive uncertainty parameter produces a joint Gaussian–softmax objective whose scale controls the classification loss contribution.Large σ2 values decrease the contribution of L2(W), whereas small σ2 values increase it, with the final term penalizing excessive scale.
  • Generalization and implementation: The construction extends to arbitrary combinations of discrete and continuous losses, remains smoothly differentiable, and prevents task weights from converging to zero.In practice, the model predicts s := log σ2 for numerical stability and valid positive variances.

4. Scene Understanding Model

The scene understanding model uses a shared convolutional encoder with task-specific decoders to produce pixel-level regression and classification outputs. It combines semantic segmentation, centroid-vector instance segmentation with OPTICS clustering, and inverse-depth regression, including handling of occluded instances.

  • Architecture: A deep convolutional encoder–decoder architecture learns pixel-level regression and classification outputs from a shared representation with separate task-specific decoders.The architecture uses multiple convolutional encoders followed by corresponding task-specific convolutional decoders.
  • Architecture: The encoder is based on DeepLabV3 with ResNet101 features, ASPP, and dilated convolutions producing feature maps subsampled by 8.The encoder provides rich contextual features before the network splits into separate decoders.
  • Semantic Segmentation: Semantic segmentation uses cross-entropy to learn pixel-wise class probabilities, averaging loss over labeled pixels in each mini-batch.
  • Instance Segmentation: Instance segmentation regresses per-pixel vectors toward instance centroids, then clusters these votes with OPTICS to identify instance centres.OPTICS is an efficient density-based algorithm that identifies an unknown number of multi-scale clusters with varying density.
  • Instance Segmentation: The centroid-voting method handles occlusion-split masks by using geometry, whereas watershed and instance-edge approaches fail in these scenarios.
  • Depth Regression: Depth regression estimates inverse depth with a pixel-wise L1 loss, using RGBD or stereo labels and ignoring pixels without inverse-depth annotations.Inverse depth can represent points at infinite distance, such as sky.

5. Experiments

Experiments on CityScapes evaluate uncertainty-weighted multi-task learning against single-task and naïvely weighted models. The method improves multi-task performance, supports dynamic and robust task weighting, and completes semantic, instance, and depth tasks with one model.

  • Dataset: CityScapes provides stereo imagery, semantic and instance segmentations across 20 classes, and SGM-labelled depth treated as pseudo ground truth.The dataset contains 2,975 training, 500 validation, and 1,525 withheld test images at 2048 × 1024 resolution.
  • Tiny CityScapes evaluation: At 128 × 256 resolution, uncertainty-weighted multi-task learning outperforms single-task models and naïve weighted-loss alternatives.Models are trained for 50,000 iterations, with disparity labels scaled after fourfold downsampling.
  • Weighting comparisons: Uniform task weighting can fail to improve single-task performance, while even approximately optimal weights perform worse than the proposed method.Finding approximately optimal weights requires an expensive grid search as the number of tasks increases.
  • Learned weighting: Homoscedastic uncertainty makes task weights dynamic during training, and the uncertainty term generally decreases to improve optimization.The resulting task weightings also vary throughout training.
  • Learned weighting: 43 : 1 : 0.16 weights semantic segmentation, depth regression, and instance segmentation, respectively, at the end of final-model training.The task-uncertainty loss is robust to parameter initialization, with parameters converging to similar optima within a few hundred iterations.
  • Full-size CityScapes benchmark: On full-size CityScapes, the method is the first single model completing semantic segmentation, instance segmentation, and depth, and it outperforms many comparable approaches.The benchmark compares against other state-of-the-art methods using comparable training data and inference tools.

6. Conclusions

The paper concludes that homoscedastic task uncertainty provides an effective, principled way to learn loss weightings automatically, improving unified multi-task scene understanding. It also identifies unresolved questions about optimal task weighting, network sharing, and relationships among tasks.

  • Conclusions: Homoscedastic task uncertainty effectively weights losses automatically and robustly to initialization.The derived loss function learns relative weighting from the data.
  • Conclusions: The unified architecture addresses semantic segmentation, instance segmentation, and per-pixel depth regression.Modelling task-dependent homoscedastic uncertainty improves representation and each task’s performance versus separate single-task models.
  • Open questions: The results suggest no single optimal weighting exists for all tasks, leaving the optimal weighting and higher-level objective unresolved.The paper raises whether multi-task learning is ill-posed without a single higher-level goal.
  • Open questions: The paper leaves open where to split the shared encoder into task-specific decoders and what depth best supports shared multi-task representations.These architectural choices are identified as questions for future work.
  • Open questions: The differing results for semantics-and-depth versus semantics-and-instance tasks motivate quantifying relationships among tasks for multi-task representation learning.The three tasks are described as complementary and useful for learning a rich scene representation.

A. Model Architecture Details

The model uses a DeepLabV3-based shared encoder with dilated convolutions and an ASPP module, then task-specific two-layer decoders produce outputs for each objective.

  • Shared feature extraction: A ResNet101 encoder with dilated convolutions produces features downsampled by a factor of 8, followed by a four-branch 256-output-channel ASPP module.The ASPP module improves contextual reasoning while extracting a shared representation.
  • Task decoders: Each task decoder applies a 1×1 convolution with 256 features, batch normalization, nonlinear activation, and a final convolution to the required output dimensions.Classification outputs match the number of semantic classes, while depth and instance segmentation use 1 or 2 channels.
  • Shared computation: Most model parameters and depth reside in the shared encoder, while task decoders remain relatively inflexible to maximize compute sharing across tasks.This architecture is intended to learn a better shared representation through multitask learning.

A.1. Optimisation

The experiments use a shared stochastic-gradient-descent optimisation setup with polynomial learning-rate decay. Training schedules and augmentations differ between the Tiny CityScapes validation and full-scale CityScapes experiments.

  • Optimisation: Training uses stochastic gradient descent with Nesterov updates, momentum 0.9, weight decay 104, and PyTorch.These settings are used throughout the paper’s experiments.
  • Optimisation: Tiny CityScapes uses 50, 000 iterations, 256 × 256 crops, batch size 8, a single NVIDIA 1080Ti GPU, and random horizontal flipping.Images are evaluated at down-sampled resolution 128 × 256.
  • Optimisation: Full-scale CityScapes uses 100, 000 iterations and batch size 16 with random flipping, scaling from 0.7 - 2.0, 512 × 512 crops, and uniform shuffled sampling.Training takes five days on one computer with four NVIDIA 1080Ti GPUs.

B. Further Analysis

The uncertainty-based weighting is robust to initialization and dynamically adjusts task weights during training. On CityScapes, learned task variances produce strongly different task weighting, while failures resemble single-task models and are often correlated across modalities.

  • Initialization robustness: Initial log σ2 values from −2.0 to 5.0 converge to the same homoscedastic-noise and task-loss minima, with uncertainty converging after only 100 iterations.The training loss may take 10,000+ iterations to converge.
  • Learned task weighting: At 500 iterations, estimated task variances are 0.60 for semantic segmentation, 62.5 for instance segmentation, and 13.5 for depth regression, yielding a weighting ratio of approximately 23 : 0.22 : 1.Weights are determined by the inverse of the uncertainty estimates.
  • Dynamic weighting: As training progresses, decreasing homoscedastic noise increases each task’s loss weight and effectively raises the overall learning rate.The decrease reflects improving task performance and lower error uncertainty; the experiments compensate for this effect.
  • Failure modes: The model shares single-task failure modes, including out-of-distribution objects, occlusion, and visually challenging situations, while errors are often correlated across all three modalities.An erroneous pixel prediction in one task will often coincide with an error in another modality.

C. Further Qualitative Results

The section presents additional qualitative results on test images from the CityScapes dataset.

  • Figure 8 shows more qualitative results on CityScapes test images.

D. Failure Examples

The model fails on CityScapes test images under challenging visual effects, contextual ambiguities, and rare-class classification. Reflections confuse predictions, road and footpath are commonly conflated, and bus, fence, and motorbike are misclassified.

  • D. Failure Examples: Reflections and other challenging visual effects confuse the model on CityScapes test examples.These examples appear in the first two rows of Figure 9.
  • D. Failure Examples: The model commonly confuses road and footpath, which the authors attribute to insufficient contextual reasoning.This error is shown in rows three and four.
  • D. Failure Examples: Rare classes—including bus, fence, and motorbike—are incorrectly classified.These failures are demonstrated in rows five through seven.
Loading 1705.07115v3…