PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Classification and Localization for Object Detection

Yue Wu, Yinpeng Chen, Lu Yuan, Zicheng Liu, Lijuan Wang, Hongzhi Li, Yun Fu

arXiv:1904.06493v4cs.CV

TL;DR

The paper addresses limited understanding of how fully connected and convolution heads serve classification and localization in two-stage detection. It analyzes their complementary behavior and proposes Double-Head, gaining 3.5 AP with ResNet-50 and 2.8 AP with ResNet-101 over FPN baselines on MS COCO.

  • Problem

    Two widely used detection head structures lacked comparative understanding across classification and localization tasks.

  • Method

    The paper analyzes fc-head and conv-head, examines their spatial sensitivity, and assigns classification to fc-head and bounding box regression to conv-head in Double-Head.

  • Results

    The heads show opposite task preferences: fc-head is more suitable for classification, while conv-head is more suitable for localization.

  • Takeaways & Limitations

    Spatial sensitivity helps fc-head distinguish complete objects from parts, whereas conv-head is more robust for regressing whole objects.

Abstract

from arXiv · show

Two head structures (i.e. fully connected head and convolution head) have been widely used in R-CNN based detectors for classification and localization tasks. However, there is a lack of understanding of how does these two head structures work for these two tasks. To address this issue, we perform a thorough analysis and find an interesting fact that the two head structures have opposite preferences towards the two tasks. Specifically, the fully connected head (fc-head) is more suitable for the classification task, while the convolution head (conv-head) is more suitable for the localization task. Furthermore, we examine the output feature maps of both heads and find that fc-head has more spatial sensitivity than conv-head. Thus, fc-head has more capability to distinguish a complete object from part of an object, but is not robust to regress the whole object. Based upon these findings, we propose a Double-Head method, which has a fully connected head focusing on classification and a convolution head for bounding box regression. Without bells and whistles, our method gains +3.5 and +2.8 AP on MS COCO dataset from Feature Pyramid Network (FPN) baselines with ResNet-50 and ResNet-101 backbones, respectively.

1. Introduction

The paper identifies complementary task preferences in fully connected and convolution heads, then proposes Double-Head to assign classification and localization to their respective strengths.

  • Two-stage detectors commonly share one head for classification and bounding box regression, but the relative suitability of fully connected and convolution heads was unclear.
  • fc-head is more suitable for classification because its scores correlate more strongly with proposal IoU, while conv-head provides more accurate regression.
  • fc-head is more spatially sensitive, helping distinguish complete objects from object parts but making whole-object regression less robust.
  • Double-Head assigns classification to fc-head and bounding box regression to conv-head to leverage both heads’ strengths.
  • On MS COCO 2017, the method gains 3.5 AP with ResNet-50 and 2.8 AP with ResNet-101 over FPN baselines.

2. Related Work

The related work situates the paper among one-stage and two-stage object detectors, including methods that alter prediction heads, stages, or auxiliary branches.

  • One-stage Object Detectors: One-stage detectors predict classes and locations directly, with representative approaches including sliding-window, SSD, YOLO, focal-loss, and point-based methods.
  • Two-stage Object Detectors: Two-stage detectors extract proposal features and include methods that cascade stages, share features, add segmentation heads, or predict box IoUs.
  • Two-stage Object Detectors: Prior work also couples bounding box regression with instance segmentation in a convolution head, motivating comparisons among alternative head designs.

3. Analysis: Comparison between fc-head and conv-head

Controlled analyses show that fc-head better aligns classification scores with proposal quality, whereas conv-head produces more accurate localization; spatial-correlation analysis supports this complementarity.

  • 3.1. Data Processing for Analysis: The comparison trains fc-head and conv-head with an FPN backbone and ResNet-50 on MS COCO 2017, evaluating 5,000 validation images.fc-head uses two fully connected layers, while conv-head uses five residual blocks.
  • 3.1. Data Processing for Analysis: The analysis uses predefined proposals whose IoUs with ground truth span background-like to matched boxes, enabling a controlled comparison independent of differing RPN proposals.
  • 3.2. Comparison on Classification Task: fc-head produces higher classification scores for higher-IoU proposals, and its classification scores have higher Pearson correlation with proposal IoUs, especially for small objects.
  • 3.2. Comparison on Classification Task: The higher PCC of fc-head also appears for RPN proposals and post-NMS detections, so higher-IoU boxes receive higher classification rankings when computing AP.
  • 3.3. Comparison on Localization Task: For proposal IoUs above 0.4, conv-head regresses boxes more accurately than fc-head, demonstrating stronger localization ability.
  • 3.4. Discussion: Spatial-correlation analysis finds substantially less correlation in fc-head output features than in conv-head outputs, supporting greater spatial sensitivity in fc-head.The fc-head output map is reconstructed by splitting its fully connected weights across spatial locations.
  • 3.4. Discussion: The spatial-correlation pattern of fc-head output features resembles that of its weight parameters, linking the observed sensitivity to spatially distinct transformations.

4. Our Approach: Double-Head

Double-Head assigns classification to an fc-head and bounding box regression to a conv-head, then extends this design with unfocused-task supervision and classifier fusion.

  • Network Structure: Double-Head uses an fc-head for classification and a conv-head for bounding box regression, leveraging their respective advantages.The method is extended to Double-Head-Ext by adding bounding box regression in the fc-head and classification in the conv-head.
  • Network Structure: The conv-head stacks residual blocks, beginning with a channel-expanding block from 256 to 1024 and optionally inserting non-local blocks.Average pooling produces a 1024-dimensional feature vector at the end of the convolution head.
  • Unfocused Task Supervision: Unfocused-task supervision adds bounding box regression to the fc-head and classification to the conv-head during training.Regression provides auxiliary supervision for the fc-head, while the two classifiers capture complementary information.
  • Unfocused Task Supervision: The fc-head loss combines classification and regression losses, with λfc controlling their balance; the conv-head loss is defined analogously with λconv weighting regression.Regression is the focused task in the conv-head, so λconv multiplies its regression loss.
  • Complementary Fusion of Classifiers: Classifier fusion combines fc-head and conv-head scores as s = sfc + sconv(1 −sfc), equivalently s = sconv + sfc(1 −sconv).The fusion is applicable when λfc ≠ 0 and λconv ≠ 1 and differs from averaging classifiers.

5. Experimental Results

Experiments compare single- and double-head detectors across COCO and VOC, showing that assigning classification to fc-head and regression to conv-head is consistently advantageous. The resulting Double-Head variants outperform alternative head arrangements and FPN baselines.

  • Double-Head Variations: Double-FC and Double-Conv separate classification and bounding box regression into two identical fully connected or convolution heads, respectively.
  • Double-Head Variations: Double-Head assigns classification to fc-head and bounding box regression to conv-head, while Double-Head-Reverse switches these task assignments.
  • Ablation Study: 2.0+ AP: Double-Head outperforms single-head detectors, while outperforming Double-FC and Double-Conv by at least 1.4 AP.Double-Head-Reverse drops 6.2+ AP compared to Double-Head.
  • Ablation Study: Single-Conv has higher classification scores than Double-Conv, while their regression results are comparable.Sharing regression and classification encourages correlation between classification scores and proposal IoUs.
  • Ablation Study: Double-FC has slightly higher classification scores and higher IoUs between regressed boxes and ground truth than Single-FC.
  • Ablation Study: 40.3 AP: Double-Head-Ext outperforms Double-Head at 39.8 AP by 0.5 AP when using unfocused tasks; it uses λfc = 0.7 and λconv = 0.8.Using two heads improves performance by at least 0.9 AP across all tested loss-weight pairs, and classifier fusion adds at least 0.4 AP.
  • Main Results: +3.5 and +2.8 AP: Double-Head-Ext improves over FPN with ResNet-50 and ResNet-101, respectively, on COCO val2017.On VOC07, the method gains 1.8 AP over FPN; on COCO, it gains 3.5+ AP at IoU 0.75 and 1.4+ AP at IoU 0.5 for both backbones.
  • Main Results: Double-Head-Ext gains 1.7, 2.1, and 2.5 AP on small, medium, and large objects over the best FPN or Faster R-CNN performance by size.It achieves 42.3 AP among the compared single-stage two-stage detector methods on COCO test-dev.

6. Conclusions

The paper concludes that fully connected and convolution heads have opposite task preferences. Double-Head exploits this complementarity by pairing each head with the task it suits better.

  • fc-head is more suitable for classification, whereas conv-head is more suitable for localization.
  • fc-head is more spatially sensitive, helping distinguish complete objects from object parts but making whole-object regression less robust.
  • Double-Head uses fc-head for classification and conv-head for bounding box regression.
  • +3.5 and +2.8 AP: the method improves over FPN baselines with ResNet-50 and ResNet-101 on MS COCO.

A. APPENDIX

The appendix extends the analysis beyond object size to class difficulty and studies joint training plus qualitative head comparisons.

  • The appendix analyzes fc-head and conv-head across easy, medium, and hard classes, then examines joint training and qualitative comparisons.

A.1. Sliding Window Analysis on Different Difficulty Levels of Classes

The appendix groups object categories by FPN baseline AP and finds that fc-head classification scores remain more correlated with proposal IoUs across difficulty levels.

  • Object categories are ranked by FPN AP and divided into equal easy, medium, and hard groups for sliding-window analysis.
  • Across class-difficulty groups, fc-head classification scores are more correlated with proposal IoUs than those of conv-head.

A.2. Effect of Joint Training in Double-Head-Ext

Joint training improves both classification and regression performance for the convolution head in Double-Head-Ext compared with single-head models.

  • Double-Head-Ext significantly improves conv-head classification scores over using conv-head alone.The comparison uses sliding-window analysis on all objects with a ResNet-50 backbone.
  • Conv-head regression results also improve in Double-Head-Ext compared with the corresponding single-head model.
  • The analysis compares Double-Head-Ext with two single-head models using results shown in Figure A.3.

A.3. Qualitative Analysis

The qualitative analysis contrasts the heads’ strengths: fc-head improves classification of small objects, whereas conv-head produces more accurate localization that helps remove duplicate detections. Double-Head-Ext combines these advantages.

  • Classification: fc-head successfully detects small objects that conv-head misses because of low classification scores.Examples include a signal light, cows, and persons; Double-Head-Ext also detects them by leveraging fc-head classification.
  • Localization: conv-head produces single accurate detections where fc-head generates duplicates from inaccurate proposals.The examples are a baseball bat and a surfing board.
  • Localization: More accurate conv-head regression gives duplicate boxes higher overlap with neighboring boxes, enabling NMS to remove them.Double-Head-Ext avoids the duplication by leveraging this localization behavior.
  • Classification: Figure A.4 compares ground truth and detections from conv-head, fc-head, and Double-Head-Ext across three cases.
  • Localization: Figure A.5 compares ground truth and detections from conv-head, fc-head, and Double-Head-Ext across two cases.
Loading 1904.06493v4…