PaperlayerView exact revision on arXiv

Source-linked AI summary

You Only Look One-level Feature

Qiang Chen, Yingming Wang, Tong Yang, Xiangyu Zhang, Jian Cheng, Jian Sun

arXiv:2103.09460v1cs.CV

TL;DR

FPN’s effectiveness is attributed to divide-and-conquer optimization rather than multi-scale feature fusion, but its multi-level structure adds memory and speed costs. YOLOF addresses detection with one C5 feature, a Dilated Encoder, and Uniform Matching, achieving comparable RetinaNet results at 2.5× higher speed and matching DETR with faster convergence. Its anchor-based design has higher missing error than DETR, motivating future anchor-free work.

  • Problem

    The paper investigates whether FPN’s success comes from multi-scale feature fusion or divide-and-conquer, amid the complexity, memory burden, and latency of multi-level detectors.

  • Method

    YOLOF uses one C5 feature for detection, with a Dilated Encoder for multi-scale context and Uniform Matching for sparse-anchor imbalance.

  • Results

    YOLOF achieves comparable results to RetinaNet at 2.5× faster speed and matches DETR with 7× faster convergence; at 608 × 608, it reaches 44.3 mAP at 60 fps, 13% faster than YOLOv4.

  • Takeaways & Limitations

    The paper presents YOLOF as a simple, efficient single-level baseline and identifies divide-and-conquer as FPN’s most significant benefit.

  • Takeaways & Limitations

    YOLOF’s predefined anchors are sparse and inflexible during inference, causing higher missing error than DETR; an anchor-free mechanism is left for future work.

Abstract

from arXiv · show

This paper revisits feature pyramids networks (FPN) for one-stage detectors and points out that the success of FPN is due to its divide-and-conquer solution to the optimization problem in object detection rather than multi-scale feature fusion. From the perspective of optimization, we introduce an alternative way to address the problem instead of adopting the complex feature pyramids - {\em utilizing only one-level feature for detection}. Based on the simple and efficient solution, we present You Only Look One-level Feature (YOLOF). In our method, two key components, Dilated Encoder and Uniform Matching, are proposed and bring considerable improvements. Extensive experiments on the COCO benchmark prove the effectiveness of the proposed model. Our YOLOF achieves comparable results with its feature pyramids counterpart RetinaNet while being $2.5\times$ faster. Without transformer layers, YOLOF can match the performance of DETR in a single-level feature manner with $7\times$ less training epochs. With an image size of $608\times608$, YOLOF achieves 44.3 mAP running at 60 fps on 2080Ti, which is $13\%$ faster than YOLOv4. Code is available at \url{https://github.com/megvii-model/YOLOF}.

1. Introduction

The paper argues that FPN succeeds mainly through divide-and-conquer optimization rather than multi-scale feature fusion, and introduces YOLOF as a faster single-level alternative.

  • FPN provides multi-scale feature fusion and divide-and-conquer detection across object scales.
  • Controlled RetinaNet comparisons show SiMo, using only C5 without feature fusion, comes within 1 mAP of MiMo, whereas MiSo and SiSo drop by at least 12 mAP.
  • The results suggest C5 contains sufficient context and that divide-and-conquer is more important than multi-scale feature fusion for FPN success.
  • YOLOF detects from one C5 feature and combines a suitably designed encoder with uniform matching to address scale coverage and positive-anchor imbalance.
  • 2.5× faster than RetinaNet, YOLOF achieves comparable results; it also matches DETR with 7× faster convergence and reaches 44.3 mAP at 60 fps, 13% faster than YOLOv4.
  • The paper identifies divide-and-conquer as FPN’s key benefit and presents YOLOF as a simple, efficient baseline for single-level detection.

2. Related Works

Prior detectors commonly use multiple feature levels, while single-level approaches trade efficiency for reduced detection performance or require alternative designs to remain competitive.

  • Image pyramids and feature pyramids are the main approaches for obtaining multiple features in object detection.
  • FPN constructs semantic-rich pyramids by combining shallow and deep CNN features and became an essential component of modern detectors.
  • Single-level detectors such as early YOLO models can be very fast but historically suffer detection-performance declines relative to multiple-feature counterparts.
  • The detection pipeline is organized into a backbone, encoder, and decoder, with the decoder performing classification and regression to generate prediction boxes.

3. Cost Analysis of MiMo Encoders

The cost analysis shows that MiMo encoders impose substantial memory and speed penalties compared with SiSo encoders, motivating a simpler alternative.

  • 134G vs. 6G memory distinguishes MiMo from SiSo encoders in the encoder and decoder.
  • 13 FPS vs. 34 FPS shows that detectors with MiMo encoders run much slower than those with SiSo encoders.
  • The speed penalty is attributed to detecting objects on high-resolution feature maps such as C3, which has an 8× downsample rate.
  • The analysis motivates finding an alternative optimization solution that keeps detection simple, accurate, and fast.

4. Method

YOLOF replaces the complex multi-level feature pyramid with a single C5 feature, addressing limited receptive-field coverage and sparse-anchor imbalance through a Dilated Encoder and Uniform Matching.

  • Limited Scale Range: Single-level feature detectors face limited scale coverage because one C5 receptive field cannot match objects across all scales.The paper identifies receptive-field range as the first obstacle to matching SiSo performance with MiMo encoders.
  • Limited Scale Range: Dilated Encoder stacks residual blocks with different dilation rates to generate output features with multiple receptive fields covering objects at various scales.Its Projector reduces channels and refines semantic contexts before the dilated residual blocks.
  • Imbalance Problem on Positive Anchors: Sparse anchors in SiSo encoders create a positive-anchor matching imbalance compared with MiMo encoders.The number of anchors drops from 100k to 5k when multiple levels collapse to one C5 level.
  • Imbalance Problem on Positive Anchors: Uniform Matching selects the k nearest anchors for each ground-truth box so every object size receives the same number of positive anchors.IoU thresholds additionally ignore anchors with IoU above 0.7 or below 0.1.
  • YOLOF: YOLOF combines a single-level C5 detector with the Dilated Encoder and Uniform Matching to provide a fast, straightforward framework.The design targets the performance gap between SiSo and MiMo encoders without using multiple feature levels.

5. Experiments

The experiments evaluate YOLOF against RetinaNet, DETR, and YOLOv4, then analyze its components and error patterns on COCO. Results show competitive accuracy with reduced computation, while ablations identify the roles of Dilated Encoder and Uniform Matching.

  • Comparisons: YOLOF matches DETR’s performance and converges approximately 7× faster.
  • Comparisons: 13% faster than YOLOv4, YOLOF-DC5 improves overall mAP by 0.8 but trails on small objects and leads on large objects by 7.1 mAP.
  • Ablation Experiments: Dilated Encoder substantially improves large-object performance, from 43.8 to 53.2, while also slightly improving small and medium objects.
  • Ablation Experiments: Removing Uniform Matching reduces small- and medium-object performance by approximately 10 AP, while large-object performance is only lightly affected.
  • Ablation Experiments: More residual blocks improve large-object detection by expanding the feature scale range, but YOLOF uses four blocks to remain simple.
  • Error Analysis: YOLOF’s anchor-based design produces higher missing error than DETR because its anchors are sparse and insufficiently flexible at inference.

6. Conclusion

The paper attributes FPN’s success to divide-and-conquer optimization rather than feature fusion and proposes YOLOF as a simpler alternative. Fair comparisons support its efficacy and its use as a baseline for single-level detectors.

  • YOLOF addresses dense detection optimization with a single-level feature instead of FPN’s complex multi-level structure.
  • Fair comparisons with RetinaNet and DETR support YOLOF’s efficacy as a simple, efficient single-level detector.
  • The authors propose YOLOF as a baseline and source of insight for future single-level detector design.

Appendix A: More Details

The appendix details YOLOF’s encoder variants, architecture, training resources, implementation settings, and comparison protocols. It also documents the memory and schedule conditions used for model comparisons.

  • Encoder Structures: Figure 8 distinguishes MiMo, SiMo, MiSo, and SiSo encoders by their numbers of input and output feature levels.
  • Network Architecture: YOLOF consists of a backbone, encoder, and decoder operating on a single-level feature.
  • Training Time & Memory: YOLOF converges in 4.5 hours, compared with 9.8 hours for RetinaNet+ and 112.5 hours for DETR on eight 2080Ti GPUs.
  • Implementation Details: The default YOLOF training setup uses 64 images per mini-batch and an initial learning rate of 0.12.
  • Implementation Details: YOLOF’s multi-scale training samples image sizes between 400 and 1400 pixels, with the largest edge capped at 1600 pixels.
  • YOLOv4 Comparison: The YOLOv4 comparison increases dilated residual blocks from four to eight and adopts YOLOv4 augmentations and backbone settings.

Appendix B: Additional Experimental Results

Additional experiments specify the anchor configuration, ATSS matching hyper-parameter, and higher-resolution DC5 variant. DC5 improves performance over the original YOLOF but runs more slowly because its feature resolution is larger.

  • Anchor Configuration: YOLOF places five anchor sizes, from 32^2 to 512^2, at each position on its single feature map.
  • Matching Hyper-parameter: ATSS uses k = 15 in YOLOF because the original k = 9 is not the best choice for this detector.
  • Dilated C5 Results: YOLOF-DC5 uses a downsample rate of 16 and achieves higher performance than the original YOLOF at lower speed.
  • Dilated C5 Results: The DC5 configuration adds a smaller anchor, increases top-k from 4 to 8, and lowers the positive-anchor ignore threshold from 0.15 to 0.1.
Loading 2103.09460v1…