Source-linked AI summary
BlitzNet: A Real-Time Deep Network for Scene Understanding
Nikita Dvornik, Konstantin Shmelkov, Julien Mairal, Cordelia Schmid
TL;DR
Scene understanding requires both object detection and semantic segmentation, but relatively few approaches tackle them jointly while meeting real-time requirements. BlitzNet uses a single fully convolutional network to perform both tasks in one forward pass, and the paper reports competitive accuracy with mutual benefits between tasks.
Problem
Object detection and semantic segmentation are fundamental scene-understanding tasks, but only few works have tackled them jointly.
Method
BlitzNet jointly performs object detection and semantic segmentation with a single fully convolutional network and shared weights, using deconvolutional layers and ResSkip blocks.
Results
BlitzNet provides accurate segmentation and object bounding boxes in real time, with the two tasks benefiting each other in accuracy; on VOC, BlitzNet512 (s8) achieves 81.5% mAP.
Takeaways & Limitations
A single network can solve object detection and semantic segmentation together while reducing computational cost and retaining competitive accuracy.
Takeaways & Limitations
Training BlitzNet512 with stride 4 was impossible because of memory limitations on a single GPU.
Abstract
from arXiv · showhide
Real-time scene understanding has become crucial in many applications such as autonomous driving. In this paper, we propose a deep architecture, called BlitzNet, that jointly performs object detection and semantic segmentation in one forward pass, allowing real-time computations. Besides the computational gain of having a single network to perform several tasks, we show that object detection and semantic segmentation benefit from each other in terms of accuracy. Experimental results for VOC and COCO datasets show state-of-the-art performance for object detection and segmentation among real time systems.
1. Introduction
BlitzNet targets joint real-time object detection and semantic segmentation, motivated by evidence that the tasks can provide useful information to each other. It uses a single network to exploit image annotations at bounding-box, pixel, or both levels.
- Object detection identifies and localizes predefined object categories with bounding boxes, whereas semantic segmentation assigns a class label to every pixel.
- Segmentation can support detection through connected components or ground-truth masks, while detections can support segmentation through weakly supervised learning from bounding boxes.
- BlitzNet is designed to learn from images annotated at the object level, pixel level, or both levels.
- The approach uses SSD because its fully convolutional model performs detection in one forward pass and supports real-time computation.
- BlitzNet jointly performs object detection and semantic segmentation in one pipeline, using a single network to reduce computational cost and improve both tasks' accuracy.
2. Related Work
Prior work addressed detection, segmentation, or their joint training, but BlitzNet builds on real-time SSD detection and integrates both tasks more tightly with a fully convolutional shared-weight design.
- Object detection: R-CNN-based detectors provide high accuracy but have relatively low inference speed because they compute substantially for each proposal.
- Object detection: SSD classifies boxes directly with a sliding-window approach and detects objects at multiple scales in one forward pass, motivating its use as BlitzNet's foundation.
- Semantic segmentation and deconvolutional layers: Deconvolutional architectures increase output feature-map resolution, making them natural for segmentation maps expected to match input-image resolution.
- Joint semantic segmentation and object detection: Earlier joint approaches showed that simultaneous training of segmentation and detection could outperform training the tasks independently.
- Joint semantic segmentation and object detection: UberNet integrates multiple tasks but uses Faster R-CNN and is neither fully convolutional nor real-time.
- Joint semantic segmentation and object detection: BlitzNet shares network weights between tasks until the last layer, improving speed, feature sharing, and training simplicity.
3. Scene Understanding with BlitzNet
BlitzNet uses one fully convolutional network to jointly predict object detections and semantic segmentations. Its shared multiscale architecture combines downscaled and upscaled features, with task-specific convolutional prediction layers and post-processing for real-time inference.
- Global architecture: BlitzNet processes an input image with a ResNet-50 encoder, followed by downscale and upscale streams for joint scene understanding.The upscale stream uses deconvolutional layers and ResSkip blocks to combine information across resolutions.
- Task predictions: Bounding-box detection and pixelwise segmentation are predicted in one forward pass by separate single convolutional layers operating on multiscale feature maps.The architecture performs localization and classification for boxes while producing segmentation maps from the upscale stream.
- ResSkip blocks: ResSkip combines downscale and upscale feature maps by bilinear upsampling, concatenation, convolutional processing, and a residual connection.The block uses 1 × 1, 3 × 3, and 1 × 1 convolutions before summing with the upsampled input.
- Multitask design: The shared network is motivated by overlapping task requirements, allowing detection and segmentation to share weights and benefit from each other.Both tasks use per-region classification, surrounding context, rich features, and localization information.
- Inference efficiency: Top-400 preselection per class, followed by NMS retaining 50 boxes and a final top-200 selection per image, controls proposal-processing cost.The authors report marginal impact on accuracy from this strategy.
- Training and loss: Training uses the sum of detection and segmentation losses, while segmentation specifically uses cross-entropy between predicted and target pixel-class distributions.Reweighting the two losses did not produce noticeable accuracy improvements.
4. Experiments
Experiments on VOC and COCO evaluate BlitzNet’s joint detection-segmentation training, accuracy, data usage, and real-time inference. Joint learning improves both tasks, while BlitzNet remains highly accurate among real-time detectors.
- PASCAL VOC 2007: BlitzNet300 achieves 78.5 mAP on the VOC07 test set while outperforming SSD300 and YOLO as a real-time detector.The models are trained on the union of VOC07 trainval and VOC12 trainval.
- PASCAL VOC 2007: BlitzNet512 (s8) scores 81.2% mAP on VOC07, performing 0.8% better than R-FCN.R-FCN is identified as the most accurate competitive model in this comparison.
- PASCAL VOC 2007: Joint detection-segmentation training raises VOC07 detection performance to 79.1% mAP for BlitzNet300 (s4) and 81.5% mAP for BlitzNet512 (s8).These results are reported for jointly trained models.
- PASCAL VOC 2012: On VOC12, joint training improves detection by more than 1% and segmentation mIoU by 0.4% compared with single-task learning.The models are trained on VOC12 train-seg-aug and tested on VOC12 val.
- PASCAL VOC 2012: Adding VOC07 detection images to VOC12 segmentation training improves segmentation by 3.3% and detection mAP by 0.6%.The additional detection-only data also improves classification results and mitigates confusion between similar categories.
- Inference Speed Comparison: BlitzNet is the most accurate detector among real-time systems at 24 FPS and also provides semantic segmentation masks.At 19 FPS, it provides the most accurate detections among the compared counterparts; all methods use the same Titan X GPU.
5. Conclusion
BlitzNet uses a single network for joint object detection and semantic segmentation, enabling real-time inference while showing that the tasks benefit from each other.
- A single fully-convolutional network jointly solves object detection and semantic segmentation through weight sharing and real-time inference.
- The pipeline is competitive in accuracy, with detection and segmentation benefiting from their joint formulation.
- Figure 5 compares segmentation predictions from BlitzNet300 trained with and without VOC07 augmentation.
Supplementary Material
Supplementary figures provide qualitative detection results and compare BlitzNet300 with SSD300, highlighting improved detections alongside shared failure cases.
- Figure 6 presents qualitative object-detection results from BlitzNet512 trained with augmented segmentation masks.
- Figure 7 compares SSD300 and BlitzNet300 detections, showing improved cases above shared failure cases.