Source-linked AI summary
YOLOPv2: Better, Faster, Stronger for Panoptic Driving Perception
Cheng Han, Qichao Zhao, Shuyi Zhang, Yinzi Chen, Zhenlin Zhang, Jinwei Yuan
TL;DR
Low-cost autonomous driving needs object detection, drivable-area segmentation, and lane detection, while separate models for each task can be impractical in real-time systems. YOLOPv2 uses a multi-task network for these tasks. On BDD100K, it reports state-of-the-art performance in speed and accuracy, including 91 FPS versus YOLOP's 49 FPS in the same settings.
Problem
Object detection, segmentation, and lane detection remain challenging in low-cost autonomous driving, where running a separate model for each task is often impractical for real-time systems.
Source: §1 · p. 1Method
YOLOPv2 is an end-to-end multi-task network with separate task heads for object detection, drivable-area segmentation, and lane detection.
Source: §3.2.2 · p. 3, §5 · p. 5Results
Takeaways & Limitations
The authors report reduced computational cost and real-time predictions while leaving room for further experimental research.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Over the last decade, multi-tasking learning approaches have achieved promising results in solving panoptic driving perception problems, providing both high-precision and high-efficiency performance. It has become a popular paradigm when designing networks for real-time practical autonomous driving system, where computation resources are limited. This paper proposed an effective and efficient multi-task learning network to simultaneously perform the task of traffic object detection, drivable road area segmentation and lane detection. Our model achieved the new state-of-the-art (SOTA) performance in terms of accuracy and speed on the challenging BDD100K dataset. Especially, the inference time is reduced by half compared to the previous SOTA model. Code will be released in the near future.
1. Introduction
Low-cost autonomous driving needs object detection, drivable-area segmentation, and lane detection, but separate models for each task are often impractical in real time. YOLOPv2 combines these tasks in a multi-task network and reports state-of-the-art BDD100K performance, with contributions to model structure, efficiency, and generalization.
1. Introduction
- Separate models for object detection, segmentation, and lane detection are often impractical in real-time autonomous driving systems.The introduction frames multi-task learning as a potential way to save computational cost through a shared encoder.Source: §1 · p. 1
- YOLOPv2 reports the best performance on BDD100K across object detection, drivable-area segmentation, and lane detection.The reported results are 0.83 MAP, 0.93 MIOU, and 87.3 accuracy, respectively.Source: §1 · p. 1
- 91 FPS versus YOLOP's 49 FPS in the same settings; the authors say this reduces computational cost while supporting real-time predictions.The tests ran on an NVIDIA TESLA V100.Source: §1 · p. 2
- The authors describe their contributions as a more effective structure with Mosaic, Mixup, and hybrid loss; more efficient structure and memory allocation; and an architecture intended to generalize across scenarios while maintaining speed.The introduction also identifies detection as providing obstacle position and size information, and drivable-area and lane segmentation as useful for route planning and driving safety.Source: §1 · pp. 1–2
2. Related Work
The related-work section reviews real-time object detection, drivable-area and lane segmentation, multi-task networks, and training-time techniques. It describes multi-task learning as a potential response to the impracticality of running separate models for real-time driving perception.
2.1. Real-time traffic object detectors
- One-stage detectors are often preferred for real-time use because they operate faster, and YOLO-series designs inform the paper's object-detection work.The section also describes using a simple network structure with Bag of Freebies methods to improve object-detection performance.Source: §2.1 · p. 2
2.2. Drivable area and lane segmentation
- Lane segmentation must address slender lane shapes and fragmented pixels; SCNN passes information across channels, while ENet-SAD uses self-attention to learn low-level features from high-level features.The section also notes encoder-decoder and pyramid-pooling approaches for drivable-area segmentation.Source: §2.2 · p. 2
2.3. Multi-task Approaches
- Multi-task networks learn shared representations from multiple supervisory signals; cited examples combine shared feature extraction with task-specific prediction branches.The section describes YOLOP's encoder and three heads, and HybridNet's addition of BiFPN to improve accuracy.Source: §2.3 · p. 2
2.4. Bag of freebies
- Training-time augmentation can increase input diversity without increasing inference cost; Mosaic combines four images to increase batch size and data variety.The section contrasts these methods with pixel-level adjustments that preserve original pixel information in the adjusted region.Source: §2.4 · p. 2
3. Methodology
The methodology combines a shared feature encoder with three task-specific decoder heads for traffic object detection, drivable-area segmentation, and lane detection. It also adapts features and losses to the differing task requirements and introduces Mosaic and Mixup augmentation.
3.1. Overall
- Separate decoder heads for drivable-area segmentation and lane detection reflect the tasks’ different feature-level requirements and improve overall segmentation performance with negligible computational-speed overhead.The authors attribute this design choice to the differing task difficulty and feature requirements of the two tasks.Source: §3.1 · p. 3
3.2. Network Architecture
- The network uses one shared encoder for feature extraction and three decoder heads, one for each task.The tasks are traffic object detection, drivable-area segmentation, and road-lane detection.Source: §3.2 · p. 3
3.2.1 Shared Encoder
- E-ELAN replaces YOLOP’s CSPdarknet backbone, using group convolution to enable layers to learn more diverse features.Figure 2 shows the group-convolution configuration.Source: §3.2.1 · p. 3
3.2.2 Task Heads
- The detection head combines PAN and FPN features and predicts object locations, dimensions, classes, and confidence at multiple scales using three anchors per grid.The separate heads for drivable-area and lane segmentation use distinct structures: the former branches before FPN with four nearest-interpolation upsampling operations, while the latter connects after FPN and uses deconvolution.Source: §3.2.2 · p. 3
3.2.3 Design of BOF
- Detection retains YOLOP’s weighted classification, objectness, and box losses, while drivable-area segmentation uses cross-entropy and lane segmentation uses focal loss.The paper also implements a hybrid Dice-and-focal loss for segmentation; its trade-off parameter is γ.Source: §3.2.3 · p. 4
- Mosaic and Mixup augmentation are introduced into multi-task learning, with reported significant performance improvements across object detection, drivable-area segmentation, and lane detection.The authors describe this as the first such result to their best of knowledge.Source: §3.2.3 · p. 4
4. Experiments
The experiments used a TESLA V100 graphics card and torch 1.10.
4. Experiments
- Experiments were carried out using a TESLA V100 graphics card and torch 1.10.Source: §4 · p. 4
4.1. Dataset
The study uses BDD100K, a public driving-scene benchmark with 100,000 driver-perspective frames. The dataset is divided into training, validation, and test sets.
4.1. Dataset
- BDD100K contains 100,000 driver-perspective frames and supports 10 vision tasks.The paper describes it as a challenging public benchmark for driving-scene computer-vision research.Source: §4.1 · p. 4
- The dataset split contains 70,000 training images, 10,000 validation images, and 20,000 test images.Source: §4.1 · p. 4
- Compared with Cityscapes and Camvid, BDD100K has more diverse weather conditions, scene locations, and illumination.Source: §4.1 · p. 4
4.2. Training protocol
Training uses cosine annealing with a warm restart in the first three epochs and runs for 300 epochs. Input images are resized differently for training and testing.
4.2. Training protocol
- The learning rate follows cosine annealing, with an initial rate of 0.01 and a warm restart in the first 3 epochs.Source: §4.2 · p. 4
- Training runs for 300 epochs with momentum 0.937 and weight decay 0.005.Source: §4.2 · p. 4
- Images are resized from 1280×720×3 to 640×640×3 during training and to 640×384×3 during testing.Source: §4.2 · p. 4
4.3. Results
The section compares YOLOPv2 with prior multi-task models and reports task-specific results, visual comparisons, and selected experiments on BDD100K.
4.3.1 Model parameter and inference speed
- YOLOPv2 has a stronger network structure and more parameters than two SOTA multi-task models, yet runs faster under the same experimental settings.Table 1 compares model parameters and inference speed.Source: §4.3.1 · p. 4, Table 1 · p. 5
4.3.2 Traffic object detection results
- YOLOPv2 achieves higher mAP50 and competitive Recall for traffic object detection.The evaluation uses mAP50 and Recall.Source: §4.3.2 · p. 4, Table 2 · p. 5
4.3.3 Drivable area segment results
- 0.93 mIOU is YOLOPv2’s reported best result for drivable area segmentation.Table 3 evaluates segmentation performance using MIOU.Source: §4.3.3 · p. 5, Table 3 · p. 5
4.3.4 Lane detection
- YOLOPv2 achieves the highest accuracy among the compared lane-detection models.The BDD100K lane masks are drawn at 8 pixels wide for training and 2 pixels wide for testing; pixel accuracy and IoU are used as metrics.Source: §4.3.4 · p. 5, Table 4 · p. 5
4.3.5 Discussion of visualizations
- Daytime and nighttime examples compare YOLOPv2 with YOLOP and Hybridnet, showing fewer described detection, drivable-area, and lane-prediction problems for YOLOPv2.The daytime discussion also describes discontinuous lane predictions and redundant small-vehicle detections in baseline examples.Source: §4.3.5 · p. 5
4.3.6 Ablation studies
- The ablation studies evaluate selected changes and their corresponding improvements to the network.Table 5 presents a selected list of experimental changes.Source: §4.3.6 · p. 5, Table 5 · p. 5
5. Conclusion
YOLOPv2 is an end-to-end multi-task network for three driving-perception tasks, and the paper reports state-of-the-art performance on BDD100K in both speed and accuracy.
5. Conclusion
- The end-to-end network simultaneously performs traffic object detection, drivable-area segmentation, and lane detection..Source: §5 · pp. 5, 7
- The conclusion includes figures for daytime and nighttime results..Source: Figure 3 · p. 6, Figure 4 · p. 6
- YOLOPv2 achieves state-of-the-art performance on BDD100K and exceeds existing models in both speed and accuracy.The conclusion identifies the dataset as challenging.Source: §5 · p. 7