Source-linked AI summary
MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving
Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, Raquel Urtasun
TL;DR
Real-time autonomous driving requires efficient semantic reasoning in addition to high performance. MultiNet addresses this with a shared-encoder architecture for classification, detection, and segmentation, achieving strong KITTI results, state-of-the-art road segmentation, and sub-100-ms joint inference.
Problem
Real-world applications such as autonomous driving require semantic reasoning methods to consider computational time alongside performance.
Method
MultiNet uses a unified, end-to-end trainable architecture with one shared encoder and task-specific decoders for classification, detection, and semantic segmentation.
Results
42.48 ms is reported for all tasks, alongside state-of-the-art road-segmentation performance on KITTI.
Takeaways & Limitations
Sharing computations enables joint inference across the three tasks while retaining strong KITTI performance.
Abstract
from arXiv · showhide
While most approaches to semantic reasoning have focused on improving performance, in this paper we argue that computational times are very important in order to enable real time applications such as autonomous driving. Towards this goal, we present an approach to joint classification, detection and semantic segmentation via a unified architecture where the encoder is shared amongst the three tasks. Our approach is very simple, can be trained end-to-end and performs extremely well in the challenging KITTI dataset, outperforming the state-of-the-art in the road segmentation task. Our approach is also very efficient, taking less than 100 ms to perform all tasks.
1. Introduction
MultiNet targets real-time autonomous-driving perception by jointly performing classification, detection, and semantic segmentation in one efficient architecture. It shares an encoder across task-specific decoders and reports strong KITTI performance with fast inference.
- Motivation: Real-world applications make network running times important alongside already high visual-perception performance.The paper notes hardware accelerators, compression, reduced precision, and distillation as existing ways to speed networks.
- Approach: MultiNet jointly performs classification, detection, and semantic segmentation through a unified encoder-decoder architecture.Its deep CNN encoder produces features shared across the tasks, while task-specific decoders produce outputs in real time.
- Approach: The detection decoder combines Yolo-style fast regression with size-adjusting ROI-align from Faster-RCNN and Mask-RCNN.The combination is intended to improve the speed-accuracy ratio.
- Results: State-of-the-art road-segmentation performance is demonstrated on the KITTI benchmark.The paper also reports that its ROI-align implementation improves detection without an explicit proposal-generation network.
2. Related Work
Prior work addresses classification, detection, segmentation, and multi-task learning through deep networks, but existing systems generally separate task parameters or do not fully share learned features. MultiNet is positioned as an end-to-end joint system that addresses this gap.
- Classification: Deep learning dominates modern classification, while residual networks represent the state of the art for training very deep models.Road classification also uses deep neural networks and sensor fusion.
- Detection: Traditional object detection uses region proposals followed by convolutional scoring, whereas end-to-end detectors are faster but have lagged in performance.MultiNet targets this speed-performance gap with an end-to-end detector using size-adjustable features through ROI pooling.
- Segmentation: Semantic-segmentation research developed from implicit sliding-window CNNs to end-to-end FCNs, deeper variants, and CRF-enhanced methods.Transposed convolutions are used to upsample low-resolution features, and the cited techniques with residual networks are described as state of the art.
- Multi-Task Learning: Multi-task learning exploits multiple tasks to learn better representations, but many prior systems fine-tune separate parameters and cannot perform joint inference.Some systems use semantic segmentation as an intermediate result for instance-level tasks.
- Multi-Task Learning: The closest cited system relies on existing object detectors and does not fully leverage segmentation features for both detection and classification.The paper presents its system as the first proposed approach able to perform this joint reasoning in the described way.
3. MultiNet for Joint Semantic Reasoning
MultiNet jointly performs classification, detection, and semantic segmentation through a shared encoder with task-specific decoders. Its decoders combine high-resolution features, proposal-free detection, and differentiable RoI-align-based rescaling for efficient inference.
- Architecture: MultiNet shares a common encoder across classification, detection, and semantic segmentation, with one task-specific decoder for each output.The architecture is feedforward, end-to-end trainable, and supports joint inference.
- Encoder: The encoder uses convolutional and pooling layers from a classification network to extract features reused by all three tasks.Experiments use VGG16 and ResNet encoder variants.
- Classification Decoder: The high-resolution classification decoder processes a 1248 × 348 input into a 39 × 12 feature grid, then applies a 1 × 1 convolutional bottleneck.Each grid feature corresponds to a 32 × 32 image region, reducing dimensionality before classification.
- Detection Decoder: The detection decoder is proposal-free and uses a coarse grid prediction followed by differentiable RoI-align rescaling to combine speed with scale-adjustable features.The initial prediction contains object confidence values and bounding-box coordinates; pooled features are concatenated with it for a refined prediction.
- Segmentation Decoder: The segmentation decoder produces a 39 × 12 low-resolution map, upsamples it with three transposed convolutions, and incorporates lower-layer features through skip connections.The skip features are processed with 1 × 1 convolutions before being added to partially upsampled results.
4. Training Details
MultiNet is fine-tuned from pretrained classification encoders and trained end-to-end on KITTI with task-specific losses and independently computed task forward passes. Detection uses grid-based labels and losses, while initialization and augmentation support training.
- Training Strategy: MultiNet training follows a fine-tuning pipeline that initializes the encoder from published ImageNet-trained weights and then trains the decoders end-to-end on KITTI.The original fully connected layers are replaced by task-specific decoders before joint training.
- Loss Functions: Classification and segmentation use softmax cross-entropy, while detection combines confidence classification with coordinate regression for grid cells.The detection representation assigns confidence and box labels to a 12 × 39 cell grid.
- Detection Labels: Detection labels a cell positive when it intersects a bounding box and encodes box coordinates relative to that cell.Cells use fixed dimensions of 32, and the regression loss is gated by the positive-cell indicator.
- Joint Optimization: Joint training sums the segmentation, detection, and classification losses while computing task forward passes independently and combining gradients during back-propagation.Independent passes allow different training parameters for each decoder.
- Initialization and Regularization: The encoder uses ImageNet initialization, segmentation upsampling layers use bilinear initialization, and training applies Adam, weight decay, dropout, and color and spatial augmentation.The reported Adam learning rate is 10^-5, with weight decay of 5 · 10^-4 and dropout probability 0.5.
5. Experimental Results
Experiments on KITTI evaluate MultiNet’s segmentation, detection, classification, and joint inference, emphasizing performance, speed, and decoder trade-offs.
- Experimental setup: KITTI evaluation covers road segmentation, object detection, classification, and comparisons between individual and joint inference.Detection uses average precision, segmentation uses MaxF1, and classification uses mean accuracy, precision, and recall.
- Segmentation: First place was achieved on the KITTI road benchmark at submission time, although newer submissions later overtook the approach.The comparison used a VGG-fc7 segmentation network trained on the whole training set.
- Detection: A rescaling layer substantially improves detection performance while making training only marginally slower, and the decoder is almost twice as fast as Faster-RCNN.The rescaling layer combines speed-oriented regression with size-adjustable features.
- Classification: The customized classification decoder outperforms vanilla ResNet and VGG decoders, demonstrating improved classification performance over those baselines.Classification data are constructed from KITTI-related labels with scenes kept entirely within either training or validation splits.
- Encoder comparison: ResNet-based encoders slightly outperform VGG, while VGG is faster; overall, VGG offers a strong performance–speed trade-off.The speed gap is larger under joint inference because ResNet computes 2048-channel features versus 512 for VGG.
6. Conclusion
MultiNet is a unified, end-to-end architecture for jointly reasoning about classification, detection, and semantic segmentation. It performs strongly on KITTI while completing all tasks in 42.48 ms.
- Conclusion: MultiNet uses a unified deep architecture to jointly reason about classification, detection, and semantic segmentation.The architecture’s output is visualized as a combined MultiNet result.
- Conclusion: 42.48 ms is the reported time for performing all tasks, underscoring MultiNet’s computational efficiency.The paper identifies future compression methods as a way to further reduce computational bottlenecks and energy consumption.
- Conclusion: MultiNet is simple, trainable end-to-end, and outperforms state of the art on the road-segmentation task in KITTI.The conclusion also reports joint-training and joint-inference results in Tables 8 and 9.