Source-linked AI summary
Auxiliary Tasks in Multi-task Learning
Lukas Liebel, Marco Körner
TL;DR
The paper addresses the limited availability of specialized multi-label data and the challenge of improving multi-task CNN representations. It introduces auxiliary tasks and a synthetic RSU dataset, then reports gains in main-task performance and convergence speed, with task selection and coefficient behavior remaining important boundaries.
Problem
Multi-task learning for road scene understanding requires specialized datasets with many annotations, which are costly to create and may not cover all relevant phenomena.
Method
The paper adds seemingly unrelated, easy-to-label auxiliary tasks as regularization and evaluates them with a multi-task CNN on the synthetic GTA V-based synMT dataset.
Results
Auxiliary tasks improved main-task performance in some configurations and facilitated faster convergence, especially for single-image depth estimation.
Takeaways & Limitations
Auxiliary tasks can boost multi-task CNN performance and training speed, but their selection should be tailored to the desired application and main tasks.
Takeaways & Limitations
Task-loss coefficients did not necessarily converge to similar values across reinitializations, and one coefficient diverged early into an unfavorable stable state.
Abstract
from arXiv · showhide
Multi-task convolutional neural networks (CNNs) have shown impressive results for certain combinations of tasks, such as single-image depth estimation (SIDE) and semantic segmentation. This is achieved by pushing the network towards learning a robust representation that generalizes well to different atomic tasks. We extend this concept by adding auxiliary tasks, which are of minor relevance for the application, to the set of learned tasks. As a kind of additional regularization, they are expected to boost the performance of the ultimately desired main tasks. To study the proposed approach, we picked vision-based road scene understanding (RSU) as an exemplary application. Since multi-task learning requires specialized datasets, particularly when using extensive sets of tasks, we provide a multi-modal dataset for multi-task RSU, called synMT. More than 2.5 $\cdot$ 10^5 synthetic images, annotated with 21 different labels, were acquired from the video game Grand Theft Auto V (GTA V). Our proposed deep multi-task CNN architecture was trained on various combination of tasks using synMT. The experiments confirmed that auxiliary tasks can indeed boost network performance, both in terms of final results and training time.
1 Introduction
Multi-task learning jointly solves related vision tasks to learn shared representations, while this paper adds auxiliary tasks and studies them in road scene understanding using a synthetic dataset and CNN system.
- Multi-task learning jointly predicts several outputs from one image, exploiting task connections to improve individual-task performance and reduce training and inference time.
- Auxiliary tasks are minor or irrelevant application outputs used to learn a richer, more robust representation that supports the main tasks.
- Unlike principal-versus-auxiliary task formulations, these auxiliary tasks are seemingly unrelated but share features with main tasks without explicit grouping or orthogonality penalties.
- Multi-task road scene understanding supports safety-relevant perception for advanced driver assistance systems and autonomous vehicles.
- Large, multi-label datasets are difficult and expensive to annotate manually, and real data may not cover rare phenomena such as traffic accidents.
- The paper contributes a concept for auxiliary-task regularization, its RSU implementation, the synMT synthetic dataset, and a multi-task CNN architecture.
2 Introducing Auxiliary Tasks to Multi-task Learning
The paper extends shared-encoder multi-task learning with simple auxiliary tasks that regularize representation learning, and formulates a weighted combined loss for optimization.
- Multi-task networks learn a common representation in early layers and solve individual tasks in specialized later-stage branches.
- Auxiliary tasks are easy, low-effort-label tasks that restrict the optimization parameter space and act as regularizers while supporting the desired main tasks.
- The final multi-task loss combines single-task losses computed from ground-truth labels and predictions.
- Each single-task loss receives a coefficient because task losses can have different behaviors, variances, and offsets.
- The loss coefficients can be learned jointly with network parameters, while an added regularization term prevents trivial solutions.
- The synMT dataset and its acquisition source code were made publicly available.
3 Experiments and Results
The experiments use the synthetic synMT dataset and evaluate auxiliary-task multi-task CNNs for road scene understanding. Auxiliary tasks improved depth-estimation results and accelerated convergence, while semantic segmentation did not benefit consistently and task selection remained important.
- Dataset: GTA V was selected for its state-of-the-art graphics, prior annotation-extraction work, and substantially larger accessible environment than SYNTHIA.The simulated environment includes urban areas, countryside, and highways.
- Dataset: synMT contains 2.5 · 10^5 training samples and 10^3 test samples acquired from GTA V for multi-task road scene understanding.Samples include automatically extracted depth maps and semantic masks plus metadata-based labels, captured at 1028 × 720 px and 15 fps.
- Task configurations: The main tasks were single-image depth estimation and semantic segmentation, with time-of-day and weather classification used as auxiliary tasks.Time-of-day regression was motivated by the sensitivity of depth-estimation methods to lighting-related image gradients.
- Final performance: Semantic segmentation did not profit from multi-task configurations, whereas the best depth prediction used T = {τ1, τ2, τ4}, adding weather classification.The four-task configuration indicated that a representation can generalize across four tasks, but its moderate performance underscored the need for careful auxiliary-task selection.
- Training dynamics: Auxiliary tasks accelerated convergence for SIDE, especially with T = {τ1, τ2, τ3} and T = {τ1, τ2, τ4}, compared with T = {τ1, τ2}.For semantic segmentation, the single-task setup achieved the best results but converged slowly.
4 Conclusion
The paper proposes seemingly unrelated auxiliary tasks as regularization for multi-task CNNs and applies the concept to vision-based RSU with the synthetic synMT dataset. Results indicate that auxiliary tasks boost main-task performance, while studied task combinations can inform application-specific systems and real-world data acquisition.
- Auxiliary tasks can serve as regularization that improves the performance, robustness, and training speed of multi-task networks.
- The approach was evaluated for vision-based RSU using the synthetic multi-task dataset synMT and an extended state-of-the-art CNN architecture.
- Several studied task combinations provide a starting point for evaluating application-specific multi-task systems.
- The authors state that conclusions about correlations between main and auxiliary tasks can inform real-world data acquisition campaigns and network design.