Source-linked AI summary
YOLOP: You Only Look Once for Panoptic Driving Perception
Dong Wu, Manwen Liao, Weitian Zhang, Xinggang Wang, Xiang Bai, Wenqing Cheng, Wenyu Liu
TL;DR
YOLOP addresses the need for accurate, real-time panoptic driving perception by jointly detecting traffic objects, segmenting drivable areas, and detecting lanes. It uses one encoder with three decoders and achieves strong BDD100K results, including real-time inference on Jetson TX2. The paper also finds that end-to-end joint training is effective, while future work should extend the system beyond its three current tasks.
Problem
Panoptic driving perception requires accurate, timely detection, drivable-area segmentation, and lane information despite limited embedded-device computation.
Method
YOLOP uses a lightweight shared encoder and three task-specific decoders in an end-to-end multi-task network.
Results
State-of-the-art performance across all three BDD100K tasks is achieved with 23 FPS inference on Jetson TX2.
Takeaways & Limitations
The three perception tasks can be jointly trained end-to-end without tedious alternating optimization, supporting real-time embedded deployment.
Takeaways & Limitations
The current network is limited to three tasks, with depth estimation and other driving-perception tasks left for future work.
Abstract
from arXiv · showhide
A panoptic driving perception system is an essential part of autonomous driving. A high-precision and real-time perception system can assist the vehicle in making the reasonable decision while driving. We present a panoptic driving perception network (YOLOP) to perform traffic object detection, drivable area segmentation and lane detection simultaneously. It is composed of one encoder for feature extraction and three decoders to handle the specific tasks. Our model performs extremely well on the challenging BDD100K dataset, achieving state-of-the-art on all three tasks in terms of accuracy and speed. Besides, we verify the effectiveness of our multi-task learning model for joint training via ablative studies. To our best knowledge, this is the first work that can process these three visual perception tasks simultaneously in real-time on an embedded device Jetson TX2(23 FPS) and maintain excellent accuracy. To facilitate further research, the source codes and pre-trained models are released at https://github.com/hustvl/YOLOP.
1. Introduction
YOLOP targets panoptic driving perception by jointly processing object detection, drivable-area segmentation, and lane detection under demanding accuracy, latency, and resource constraints. Its shared multi-task design achieves real-time embedded inference and state-of-the-art BDD100K performance.
- Autonomous driving perception must provide obstacle locations, drivable-road judgments, and lane positions to support vehicle decision-making.
- High precision and real-time operation are critical, but embedded systems often provide limited computational resources.
- Separate processing of detection, segmentation, and lane tasks increases latency, while their related visual information supports joint handling.
- YOLOP uses a lightweight CNN encoder followed by three task-specific decoders for simultaneous perception.
- 41 FPS on NVIDIA TITAN XP and 23 FPS on Jetson TX2 accompany state-of-the-art performance across the three BDD100K tasks.
- Ablations show the three tasks can be learned jointly without tedious alternating optimization.
2. Related Work
Prior work develops separate solutions for detection, semantic segmentation, lane detection, and multi-task learning. These studies motivate shared representations while highlighting speed, task relatedness, and training-strategy considerations.
- Deep-learning object detectors are commonly divided into two-stage methods using proposals and one-stage methods performing prediction more directly.
- YOLO-series methods accelerate detection through grid-based prediction, with later variants adding anchors and multi-scale feature processing.
- Semantic segmentation methods trade off output quality and inference speed, with pyramid pooling enhancing multi-scale features and ENet reducing feature-map size.
- Lane-detection research includes instance-segmentation, slice-by-slice convolution, and self-attention distillation approaches, with SCNN noted as time-consuming.
- Multi-task learning shares information across tasks, and related methods combine detection with masks, pose, or other location-sensitive recognition outputs.
- Joint training is considered appropriate when tasks are related; otherwise, alternating optimization may be used despite its cumbersome training process.
3. Methodology
YOLOP is a simple feed-forward network that jointly performs three driving-perception tasks using one shared encoder and task-specific decoders. Its backbone and neck extract, reuse, and fuse features to support efficient end-to-end processing.
- YOLOP combines traffic-object detection, drivable-area segmentation, and lane detection in one feed-forward network.
- A shared encoder and three subsequent decoders avoid complex redundant blocks and reduce computational consumption.
- The encoder contains a backbone network and a neck network.
- CSP-Darknet serves as the backbone because feature propagation and reuse reduce parameters and calculations, supporting real-time performance.
- The neck uses SPP and FPN modules to fuse features across scales and semantic levels through concatenation.
3.2. Decoders
YOLOP uses three task-specific decoder heads after shared feature extraction. Detection uses multi-scale grid-based prediction, while segmentation branches upsample FPN features to pixel-level outputs efficiently.
- The network has three decoder heads dedicated to traffic detection, drivable-area segmentation, and lane-line segmentation.
- The detection decoder uses a multi-scale anchor-based scheme with PAN feature fusion and three prior anchors assigned to each grid.
- PAN combines top-down semantic and bottom-up positioning information before supplying multi-scale feature maps to detection.
- The drivable-area and lane-line heads share a structure that upsamples FPN features from (W/8, H/8, 256) to (W, H, 2).
- Nearest-neighbor interpolation replaces deconvolution in upsampling to reduce computation cost while retaining fast, high-precision segmentation outputs.
3.3. Loss Function
YOLOP combines detection, drivable-area segmentation, and lane-line segmentation losses into a weighted multitask objective. The detection loss uses classification, objectness, and bounding-box terms, while segmentation uses cross-entropy and adds IoU loss for sparse lane lines.
- The detection loss is a weighted sum of classification, objectness, and bounding-box losses.Classification and objectness use focal loss, while the box term uses LCIoU.
- Both segmentation losses contain cross-entropy loss with logits to minimize pixelwise classification errors.The losses compare network pixel outputs with target labels.
- IoU loss is added to lane-line segmentation because it is especially efficient for predicting sparse lane-line categories.
- The multitask loss contains three decoder-specific parts combined into one final weighted sum.The total loss balances detection, drivable-area segmentation, and lane-line segmentation.
- The coefficients α1, α2, α3, γ1, γ2, and γ3 can be tuned to balance the components of the total loss.
3.4. Training Paradigm
YOLOP supports both end-to-end joint training and alternating optimization across tasks. Although alternating training can separate unrelated tasks or let earlier tasks guide later ones, experiments found end-to-end training sufficient for this model.
- End-to-end training learns the three tasks jointly and is useful when the tasks are related.
- Alternating optimization trains the model step by step so each step can focus on one or more related tasks.This approach can also accommodate tasks that are not all related.
- Experiments found that alternating optimization was unnecessary because end-to-end training performed well enough for YOLOP.Alternating optimization was described as potentially effective but cumbersome.
4.1. Setting
The experiments train and evaluate YOLOP on BDD100K, using augmentation and implementation procedures intended to improve robustness and performance. The setting also compares multitask and single-task networks, detection architectures, and training paradigms.
- Dataset Setting: BDD100K provides 100k annotated frames across diverse geography, environments, and weather conditions.The dataset contains 70K training, 10K validation, and 20K test images; evaluation uses the validation set because test labels are private.
- Implementation Setting: The detector uses k-means clustering to obtain prior anchors from traffic-scene detection frames.
- Training Setting: The step-by-step training procedure successively trains detection, segmentation, and then all network components jointly.The procedure freezes and reactivates parameter groups between stages.
- Implementation Setting: Data augmentation applies photometric and geometric transformations to increase image variability and robustness across environments.Transformations include hue, saturation, value, rotation, scaling, translation, shearing, and left-right flipping.
- Experimental Comparisons: Comparisons include multitask networks, single-task networks, detection architectures, lane-detection networks, and joint versus alternating training.Images are resized from 1280×720×3 to 640×384×3 for the BDD100K experiments.
4.2. Result
YOLOP is evaluated end to end across traffic-object detection, drivable-area segmentation, and lane detection against representative models. The results indicate strong accuracy and real-time performance, with qualitative benefits from sharing information across tasks.
- YOLOP is trained end to end and compared with representative models across all three perception tasks.
- Traffic Object Detection Result: YOLOP exceeds Faster R-CNN, MultiNet, and DLT-Net in detection accuracy and is comparable to YOLOv5s.YOLOv5s is faster because it lacks the lane-line and drivable-area segmentation heads.
- Traffic Object Detection Result: Qualitatively, YOLOP produces fewer false negatives and more accurate bounding boxes than Faster R-CNN.The paper attributes more reasonable predictions to information sharing across tasks.
- Drivable Area Segmentation Result: YOLOP outperforms MultiNet, DLT-Net, and PSPNet in drivable-area segmentation by 19.9%, 20.2%, and 1.9%, respectively.Its inference speed is reported as 4 to 5 times faster than theirs.
- Lane Detection Result: YOLOP dramatically exceeds the other three models on lane detection using pixel accuracy and lane IoU.The training lane-line width is 8 pixels, while the test-set lane-line width is 2 pixels.
4.3. Ablation Studies
The ablations examine training paradigms, task sharing, and prediction mechanisms for YOLOP. End-to-end multi-task training preserves task performance while reducing computation, and grid-based prediction better supports joint detection and segmentation.
- End-to-end v.s. Step-by-step: Three-step alternating training slightly outperforms two-step training, with detection-first training appearing strongest among the tested paradigms.The authors attribute the observation mainly to the detection task being harder for the model to converge on.
- End-to-end v.s. Step-by-step: End-to-end training performs well enough that alternating optimization is unnecessary for YOLOP.The authors report that alternating training can produce negligible improvements despite additional training steps.
- Ablation Studies: YOLOP’s ablation studies compare joint training with alternating paradigms and evaluate whether grid-based prediction suits the combined perception tasks.The experiments include end-to-end versus step-by-step training, multi-task versus single-task training, and region-based versus grid-based prediction.
- Multi-task v.s. Single-task: Multi-task training achieves performance close to single-task training while saving substantial time compared with executing each task individually.The comparison covers traffic object detection, drivable area segmentation, and lane line segmentation.
- Region-based v.s. Grid-based: Unlike R-CNNP, YOLOP does not show degraded multi-task performance relative to separate training of detection and semantic segmentation.The authors associate this difference with YOLOP’s grid-based heads operating on the whole encoder feature map, whereas R-CNNP first selects region proposals.
5. Conclusion
YOLOP is an end-to-end network for simultaneous object detection, drivable area segmentation, and lane detection. It achieves strong performance across the three tasks on BDD100K, runs in real time on Jetson TX2, and remains limited to three tasks while motivating improved multi-task training.
- Conclusion: YOLOP simultaneously handles object detection, drivable area segmentation, and lane detection through end-to-end training.The conclusion describes the network as simple and efficient for these three driving perception tasks.
- Conclusion: YOLOP achieves or greatly exceeds state-of-the-art performance on all three tasks on the BDD100K dataset.The conclusion characterizes BDD100K as challenging.
- Conclusion: YOLOP performs real-time inference on the embedded Jetson TX2 device.The conclusion presents this as supporting use in real-world scenarios.
- Limitations and Future Work: The current model is limited to three perception tasks, with depth estimation identified as a possible future addition.The authors also seek more appropriate multi-task training paradigms to improve task performance.