Source-linked AI summary
HybridNets: End-to-End Perception Network
Dat Vu, Bao Ngo, Hung Phan
TL;DR
Running an image through three separate networks creates latency, motivating an end-to-end multi-task perception network. HybridNets is presented as achieving state-of-the-art accuracy with fewer FLOPS than previous multi-network models.
Problem
Passing an image through three separate networks creates unreasonable latency.
Source: §1.1 · p. 2Method
HybridNets is an end-to-end multi-network trained with a strategy that balances detection and segmentation losses.
Source: §2.4 · p. 8, §4 · p. 21Results
HybridNets is reported to achieve state-of-the-art accuracy with fewer FLOPS than previous multi-network models.
Source: §4 · p. 21Takeaways & Limitations
The proposed end-to-end network responds to the latency of passing images through three separate networks.
Source: §1.1 · p. 2, §4 · p. 21The stated limitation concerns FPN: its feature information flows in one direction.
Source: §2.2 · p. 6
Abstract
from arXiv · showhide
End-to-end Network has become increasingly important in multi-tasking. One prominent example of this is the growing significance of a driving perception system in autonomous driving. This paper systematically studies an end-to-end perception network for multi-tasking and proposes several key optimizations to improve accuracy. First, the paper proposes efficient segmentation head and box/class prediction networks based on weighted bidirectional feature network. Second, the paper proposes automatically customized anchor for each level in the weighted bidirectional feature network. Third, the paper proposes an efficient training loss function and training strategy to balance and optimize network. Based on these optimizations, we have developed an end-to-end perception network to perform multi-tasking, including traffic object detection, drivable area segmentation and lane detection simultaneously, called HybridNets, which achieves better accuracy than prior art. In particular, HybridNets achieves 77.3 mean Average Precision on Berkeley DeepDrive Dataset, outperforms lane detection with 31.6 mean Intersection Over Union with 12.83 million parameters and 15.6 billion floating-point operations. In addition, it can perform visual perception tasks in real-time and thus is a practical and accurate solution to the multi-tasking problem. Code is available at https://github.com/datvuthanh/HybridNets.
1. INTRODUCTION
The introduction motivates a unified driving-perception network as an alternative to the latency of running separate task-specific networks. HybridNets combines perception tasks and feature fusion with customized anchors and a staged training strategy.
1.1. Background
- Running an image through three separate networks creates unreasonable latency, motivating work on unified encoder-decoder architectures.Prior approaches combine networks, while the introduction identifies further opportunities in feature extraction and combining lane-line and drivable-area segmentation.Source: §1.1 · p. 2
- The introduction describes BiFPN as a multi-scale fusion approach and says the paper improves on it while using an EfficientNet backbone.The network is described as using existing multi-scale features in a newly designed segmentation head.Source: §1.1 · p. 2
- 15.6 BFLOPS on 12.83M parameters is the reported cost of HybridNets at 640x384 input resolution.The introduction compares this with YOLOP at 18.6 BFLOPS on 7.9M parameters.Source: §1.1 · p. 2
- The paper proposes HybridNets for traffic-object detection, drivable-area segmentation, and lane-line detection in one end-to-end network.Its listed contributions also include customized anchors at each weighted bidirectional feature-network level and a training loss and strategy for multi-task optimization.Source: §1.1 · p. 3
1.2. Related Works
- Prior multi-task models include YOLOP, whose two similar segmentation heads leave room to reduce them to one better-performing head while accounting for hardware constraints.The introduction also notes that model scaling must be considered for real-time decision-making networks.Source: §1.2 · p. 5
2. METHODOLOGY
HybridNets shares an encoder across perception tasks and uses separate detection and segmentation decoders. Its method combines bidirectional multi-scale feature fusion, customized detection anchors, a fused segmentation branch, and staged multi-task training.
2.1. Network Architecture
- HybridNets uses one shared encoder and two separated decoders for distinct tasks.The encoder comprises backbone and neck networks; the detection head and segmentation head are the two decoders.Source: §2.1 · p. 5
2.2. Encoder
- BiFPN fuses features across resolutions through bidirectional paths and learns weights for the importance of each feature level.The paper uses this neck because FPN passes feature information in only one direction; EfficientNet-B3 is selected as the backbone.Source: §2.2 · p. 6
2.3. Decoder
- The detection head uses nine k-means-derived anchor clusters and three scales per grid cell to cover small-to-large boxes across feature-map levels.It predicts bounding-box offsets, class probabilities, and box confidence.Source: §2.3 · pp. 6–7
- The segmentation branch combines five neck feature levels, P3–P7, into a three-class output for background, drivable area, and lane line.Each level is upsampled and channel-aligned before fusion; the output is restored to the P2 size, with a P2 backbone feature added to the final fusion.Source: §2.3 · p. 7
2.4. Loss Function and Training
- For class-imbalanced segmentation, the method combines Tversky and Focal losses, while tuning parameters balance detection and segmentation losses.Training proceeds from detection, to segmentation with the encoder and detection head frozen, and finally joint training across tasks.Source: §2.4 · pp. 8, 9, 11
3. EXPERIMENTATION AND EVALUATION
The experiments evaluate HybridNets on BDD100K across traffic-object detection, drivable-area segmentation, and lane detection, alongside computation and inference comparisons. Results show strong detection and lane metrics, while drivable-area mIoU trails YOLOP and crossroads remain a challenge.
3.1. Experiment Settings
- For comparison with existing multi-task networks trained on BDD100K, the evaluation prepares three task datasets, merges car, truck, bus, and train into vehicle, merges direct and alternative into drivable, and centralizes two lane-line annotations.Training lane annotations are dilated to 8 pixels, while validation annotations are kept intact.Source: §3.1 · p. 12
3.2. Evaluation Metrics
- Traffic-object detection is evaluated with mAP50, the average of Average Precision calculated for all classes at a single IoU threshold of 0.5; segmentation uses mIoU, and lane detection uses accuracy and IoU.Source: §3.2 · p. 13, §3.4 · p. 18
3.3. Cost Computation Performance
- 12.83M parameters accompany 15.6 BFLOPs, and HybridNets uses fewer computations than the compared networks despite having more parameters than YOLOP.Depth-wise separable convolutions are cited as reducing computation; GPU inference is reported as up to 1.4x faster than previous multi-networks.Source: §3.3 · pp. 13–14
3.4. Multi-task Performance
- HybridNets detects objects as small as 3–10 pixels at a 640×384 input size, which the paper attributes to automatically customized anchor scales and aspect ratios.The paper also reports improved detection over YOLOP, including small and large objects and complex weather conditions.Source: §3.4 · pp. 14–15
- 90.5% mIoU on drivable-area segmentation is below YOLOP's 91.5%, although the authors report more accurate HybridNets outputs in their visual comparison.HybridNets uses one multi-class decoder, while YOLOP uses two task-specific decoders; the authors describe HybridNets as more flexible.Source: §3.4 · p. 16, Table 3 · p. 16, Figure 7 · p. 17
- 85.4% accuracy and 31.6% IoU make HybridNets outperform previous models on lane detection, though preprocessing leaves lane-line IoU suboptimal.The authors report that lane-line preprocessing makes annotations easier to learn, with the drawback of suboptimal results.Source: §3.4 · p. 18, Table 4 · p. 18
- The model performs well in many complex scenarios, but the authors report crossroads cases where lane lines break and drivable area is assigned to the road's other side.The paper also describes severe reflective scenes and extreme weather among the scenarios it handles well.Source: §3.4 · p. 19, Figure 10 · p. 20
4. CONCLUSION AND PERSPECTIVE
HybridNets combines end-to-end multi-task perception with customized anchors and a training loss and strategy designed to improve accuracy and performance. The paper reports state-of-the-art accuracy with fewer FLOPs than previous multi-network models and identifies broader, more robust task coverage as future work.
4. CONCLUSION AND PERSPECTIVE
- The proposed design combines an end-to-end multi-task perception network with automatically customized anchors at each BiFPN level and an efficient loss and training strategy.The paper names the resulting network HybridNets and describes the loss and strategy as intended to improve accuracy and performance.Source: §4 · p. 21
- HybridNets achieves state-of-the-art accuracy with fewer FLOPs than previous multi-network models, with better accuracy and efficiency across a broad spectrum of resource constraints.The conclusion presents the network as a new end-to-end multi-network.Source: §4 · p. 21
- Future work targets a robust network for more tasks, 3-D object detection from one input, and improved lane-line performance and drivable-area context.The conclusion also identifies improving network parameters and FLOPs as a focus.Source: §4 · p. 21