Source-linked AI summary
Multi-Branch Auxiliary Fusion YOLO with Re-parameterization Heterogeneous Convolutional for accurate object detection
Zhiqiang Yang, Qiu Guan, Keer Zhao, Jianmin Yang, Xinli Xu, Haixia Long, Ying Tang
TL;DR
PAFPN does not adequately integrate multi-scale semantic and spatial information, particularly shallow features important for small-target detection. MAF-YOLO addresses this with MAFPN, RepHELAN, and GHSK, and improves AP over comparable real-time detectors while reducing model size or computation in several comparisons.
Problem
PAFPN overlooks shallow low-level spatial information and direct fusion of layers containing crucial small-target features.
Method
MAF-YOLO combines the MAFPN neck, re-parameterized heterogeneous convolutions in RepHELAN, and GHSK for richer multi-scale feature fusion and receptive fields.
Results
MAF-YOLOn improves AP by 5.1% over YOLOv8n, by 2.5% over Gold-YOLOn with about 36% fewer parameters and 13% less computation, and by 0.6 AP over YOLOv9s.
Takeaways & Limitations
The proposed design delivers strong overall real-time detection performance while maintaining a compact architecture and enhanced multi-scale representation.
Abstract
from arXiv · showhide
Due to the effective performance of multi-scale feature fusion, Path Aggregation FPN (PAFPN) is widely employed in YOLO detectors. However, it cannot efficiently and adaptively integrate high-level semantic information with low-level spatial information simultaneously. We propose a new model named MAF-YOLO in this paper, which is a novel object detection framework with a versatile neck named Multi-Branch Auxiliary FPN (MAFPN). Within MAFPN, the Superficial Assisted Fusion (SAF) module is designed to combine the output of the backbone with the neck, preserving an optimal level of shallow information to facilitate subsequent learning. Meanwhile, the Advanced Assisted Fusion (AAF) module deeply embedded within the neck conveys a more diverse range of gradient information to the output layer. Furthermore, our proposed Re-parameterized Heterogeneous Efficient Layer Aggregation Network (RepHELAN) module ensures that both the overall model architecture and convolutional design embrace the utilization of heterogeneous large convolution kernels. Therefore, this guarantees the preservation of information related to small targets while simultaneously achieving the multi-scale receptive field. Finally, taking the nano version of MAF-YOLO for example, it can achieve 42.4% AP on COCO with only 3.76M learnable parameters and 10.51G FLOPs, and approximately outperforms YOLOv8n by about 5.1%. The source code of this work is available at: https://github.com/yang-0201/MAF-YOLO.
1 Introduction
The paper identifies limitations in PAFPN’s multi-scale fusion for small-target representation and proposes MAF-YOLO with a multi-branch auxiliary neck and heterogeneous convolutional design.
- Problem: PAFPN overlooks shallow low-level spatial information and lacks direct fusion of layers containing crucial small-target information.Its small-target detection pathway also uses a singular down-top route and two blocks, limiting minute-object feature learning and representation.
- Proposed framework: MAFPN is introduced as a plug-and-play neck that enables richer feature interaction and fusion across scales.The design includes SAF for retaining shallow backbone information and AAF for enriching output-layer gradient information through multi-directional connections.
- Proposed framework: SAF preserves shallow backbone information through bidirectional connectivity, while AAF enriches output-layer gradient information through multi-directional connections.
- Proposed framework: RepHELAN combines re-parameterized heterogeneous large convolutions with parallel small convolutions to expand perception while preserving small-object information without added inference cost.
- Proposed framework: MAF-YOLO combines MAFPN, RepHELAN, and GHSK, with the paper reporting superior object-detection performance across various aspects on MS COCO.GHSK adaptively adjusts kernel sizes across resolution layers to enlarge the effective receptive field.
2 Related works
Related work develops FPN-based feature fusion and real-time YOLO detectors through reparameterization, bidirectional connections, and mechanisms combining local and global information.
- Real-time object detection: Real-time detectors favor single-stage methods such as YOLO because complex two-stage and transformer-based structures can impose substantial parameter and computational overhead.
- YOLO detectors: Reparameterization and Task Alignment Learning have been adopted in PPYOLOE and YOLOv6 to enhance detection performance.
- Feature pyramid networks: FPN research improves multi-scale detection through cross-scale connections and information exchange, motivating continued optimization of feature-fusion structures.
- Feature pyramid networks: YOLOv6-v3, DAMO-YOLO, and Gold-YOLO extend feature fusion using backbone-shallow information, RepGFPN, or local-global integration mechanisms.The cited work states that these approaches alleviate the feature-fusion issue while leaving further optimization opportunities.
3 Methodology
MAF-YOLO replaces conventional feature fusion with MAFPN, whose SAF and AAF branches combine shallow, deep, and multi-scale information. RepHELAN complements this neck with heterogeneous large and small convolutions that broaden receptive fields while preserving small-target detail.
- Multi-Branch Auxiliary FPN: MAFPN uses SAF for shallow backbone fusion and AAF for denser multi-scale interaction in deeper neck layers.SAF integrates same-level and high-resolution backbone features, while AAF collects information through denser connections.
- Superficial Assisted Fusion: SAF combines deep features with same-level and high-resolution shallow features, using 1 × 1 convolutions to limit shallow-channel influence.The design aims to preserve localization details while preventing shallow information from dominating concatenation.
- Advanced Assisted Fusion: AAF aggregates information across four feature layers and uses 1 × 1 control channels to regulate each layer’s contribution.The cited formulation combines shallow high-resolution, shallow low-resolution, previous, and current-layer information.
- RepHELAN: RepHELAN splits features into streams and aggregates multi-scale representations using heterogeneous large-kernel and small-kernel convolutions.Its RepHDWConv runs varying-depthwise-kernel branches during training and merges them into one convolution for inference.
- RepHELAN: RepHELAN’s parallel convolutions expand effective receptive fields while preserving information relevant to small targets without reducing inference speed.The design uses large kernels for broader context and small kernels for complementary detail, then reparameterizes them into one inference convolution.
4 Experiments
The experiments evaluate MAF-YOLO on COCO 2017 using standard AP measurements across IoU thresholds and target scales. Training uses the 115k-image training split and reports results on 5000 validation images.
- Experimental setup: MAF-YOLO is evaluated on COCO 2017 with standard mean average precision across IoU thresholds and target scales.The ablation studies use 115k training images and 5000 validation images.
- Experimental setup: All MAF-YOLO scales are trained from scratch for 300 epochs on eight RTX 2080Ti GPUs without ImageNet data or pretrained weights.The implementation is based on YOLOv6-2.0 and uses dynamic cache-based mixup and mosaic mechanisms.
Implimentation details.
The RepHELAN analysis uses MAF-YOLO nano as the default model and compares its computational block against alternatives from advanced YOLO models.
- RepHELAN ablation: RepHELAN ablations use MAF-YOLO nano as the default configuration.
- RepHELAN ablation: RepHELAN achieves higher accuracy and a higher parameter utilization rate than computational blocks from other advanced YOLO models.The comparison is reported in Table 1 using MAF-YOLO nano by default.
Different computational blocks.
The RepHELAN ablation compares computational blocks and convolution-kernel strategies, culminating in the strongest reported configuration at 42.4% AP.
- 42.4% AP was achieved when large DWConv was replaced with RepHConv, improving small, medium, and large object categories.The final configuration used the RepHConv replacement within the RepHELAN ablation.
- Using only large convolutional kernels with ELAN produced a +1% AP gain but reduced small-target performance by 0.3% AP.
- RepHConv improved performance by 0.4% without added overhead from a large convolutional kernel or changes in model size.
- Adding ELAN alone provided a 0.2% AP boost with only a small increase in computational cost.
4.3 Analysis of MAFPN
The MAFPN analysis evaluates SAF and AAF separately and together, finding that their combination provides the highest model performance.
- Combining SAF and AAF produced the model’s maximum performance in the MAFPN ablation.
- Adding SAF improved performance by 0.3% and small-target performance by 1%, while increasing parameters by 0.3M.
- Adding AAF alone improved performance for objects across all scales.
Ablation study on MAFPN.
MAFPN is evaluated as a plug-and-play neck across detector architectures and tasks, improving YOLOv8n performance while also being tested with Cascade MaskRCNN.
- Replacing PAFPN with MAFPN in YOLOv8n produced a 2% AP improvement with fewer parameters and 200 fewer training epochs.
- MAFPN was also evaluated with the two-stage Cascade MaskRCNN detector for instance segmentation.
4.4 Ablation study on MAF-YOLO
Sequential ablations attribute MAF-YOLO’s gains to MAFPN, RepHELAN, and GHSK, while the comparison is framed against state-of-the-art real-time detectors.
- Adding MAFPN improved performance by 2.1% AP with 0.5M additional parameters, while RepHELAN reduced parameters by 1.2M and added 1.1% AP.
- Adding GHSK improved model accuracy by 1.2% AP with marginal parameter costs.
- Table 6 compares MAF-YOLO with other state-of-the-art real-time object detectors, including distinctions for self-distillation and pretrained-model training.
4.5 Comparison with State-of-the-Arts
MAF-YOLO outperforms several state-of-the-art real-time detectors while reducing parameters or computation in key comparisons.
- MAF-YOLOn improves AP by 5.1% over YOLOv8n despite having slightly more parameters.Against Gold-YOLOn, it reduces parameters by about 36% and computation by 13% while improving AP by 2.5%.
- MAF-YOLOs uses 22% fewer parameters than anchor-free YOLOv7s while improving AP by 2.3%.
- MAF-YOLOs achieves 0.6 higher AP than YOLOv9s with comparable parameters and calculations.
- The comparison includes two-stage and transformer-based detectors, for which MAF-YOLO demonstrates superior performance and lower weight.
5 Conclusions
The paper concludes that MAFPN addresses PAFPN limitations through assisted fusion, while GHSK and RepHELAN expand receptive fields and multi-scale representation with comparable parameter counts.
- MAFPN combines SAF for retaining shallow backbone information with AAF for preserving diverse multi-scale information through enhanced fusion.
- GHSK dynamically scales convolutional kernels throughout MAF-YOLO to expand the network’s sensory field.
- RepHELAN uses re-parameterized heterogeneous convolutions to enhance multi-scale characterization capability.
- MAF-YOLO demonstrates outstanding overall performance while maintaining a comparable number of parameters.