Source-linked AI summary
Learning Region Features for Object Detection
Jiayuan Gu, Han Hu, Liwei Wang, Yichen Wei, Jifeng Dai
TL;DR
Modern object detectors have largely learnable components, but region feature extraction remains dependent on hand-crafted RoI pooling heuristics. This paper unifies existing methods as weighted feature aggregation and introduces a module that learns weights from RoI geometry and image features. The method outperforms RoI pooling counterparts and achieves 39.9 mAP with FPN+ResNet-101.
Problem
Method
The paper represents region parts as weighted sums of image features and learns the weights from RoI–image geometry and image features.
Source: §1 · p. 2Results
The proposed method outperforms RoI pooling counterparts and reaches 39.9 mAP using FPN+ResNet-101.
Source: §1 · p. 2, Comparison on stronger detection… · p. 11Takeaways & Limitations
The method removes most heuristic choices and moves object detection toward fully learnable region feature extraction.
Source: §1 · p. 2Regular pooling can produce identical features for RoIs closer than the feature-map sampling resolution, and the naive implementation is computationally expensive.
Source: §2 · p. 3
Abstract
from arXiv · showhide
While most steps in the modern object detection methods are learnable, the region feature extraction step remains largely hand-crafted, featured by RoI pooling methods. This work proposes a general viewpoint that unifies existing region feature extraction methods and a novel method that is end-to-end learnable. The proposed method removes most heuristic choices and outperforms its RoI pooling counterparts. It moves further towards fully learnable object detection.
1 Introduction
Region feature extraction remains a largely hand-crafted part of object detection, motivating a general formulation and a fully learnable module that reduces heuristic choices.
1 Introduction
- Region feature extraction still relies on heuristic RoI pooling rules, unlike other major detection steps that have become learnable.RoI pooling divides regions into regular bins, aggregates nearby image features using rules such as average, maximum, or bilinear interpolation, and concatenates the results.Source: §1 · p. 1
- Deformable RoI pooling demonstrates the promise of learning region extraction, but remains constrained by regular-grid pooling and learns only bin offsets.Source: §1 · p. 1
- The work formulates each region part as a weighted sum of image features, showing that most prior extraction methods are special cases defined by hand-crafted weights.Source: §1 · p. 2
- The proposed module learns extraction weights from both RoI–image geometric relations and image features.An attention model captures geometry, while a convolution layer incorporates image-feature information.Source: §1 · p. 2
- Extensive experiments show that the method outperforms RoI pooling counterparts while moving object detection toward a fully learnable pipeline.A sparse sampling implementation substantially reduces the computational cost of the naive method with little accuracy degradation.Source: §1 · p. 2
2 A General Viewpoint on Region Feature Extraction
The paper formulates region feature extraction as weighted aggregation over image positions, unifying existing RoI pooling methods and motivating learnable weights. It then relates these methods to their spatial supports, sampling rules, and limitations.
2 A General Viewpoint on Region Feature Extraction
- A general formulation represents each region part as a weighted sum of image features over a support region, with normalized weights.The support may be the RoI, include context, or span the entire image.Source: §2 · pp. 2–3
- Regular RoI pooling divides the RoI into grid bins and applies max or average aggregation, but spatial downsampling makes nearby RoIs indistinguishable.When RoIs differ by less than 16 pixels, their sampled bin positions and features can be identical.Source: §2 · p. 3
- Aligned RoI pooling uses bilinear interpolation around each bin center, making features sensitive to subtle RoI-position changes and outperforming regular pooling.Its weights are nonzero only at the four positions surrounding the sampling point.Source: §2 · p. 3
- Existing methods remain largely hand-crafted because their supports and weights are manually specified, and image features are not used in the regular or aligned pooling weights.The general viewpoint exposes these choices as alternative definitions of the support region and weighting function.Source: §2 · p. 3
- Deformable RoI pooling learns a bin offset from image features, adaptively modeling object-shape deformation and degenerating to aligned pooling when the offset learning rate is zero.The offsets are regressed from an initial pooled region feature through an additional learnable fully connected layer.Source: §2 · p. 4
2.1 More Related Works
- The framework also covers non-grid bins, mask-restricted pooling, position-sensitive channel subsets, and one-stage center-point features as specializations of the general formulation.One-stage detection uses K = 1 and the region center, trading accuracy for speed, while other methods alter spatial bins, normalization, or channels.Source: §2.1 · pp. 4–5
3 Learning Region Features
The method learns region-feature weights from both RoI–position geometry and image features, replacing hand-crafted pooling choices with an end-to-end module. An efficient sparse sampler reduces computation while retaining accuracy close to dense sampling.
3 Learning Region Features
- The module models each region-feature weight using geometric relations between RoIs and image positions plus adaptive image-feature information.Geometry is represented with attention, while image features are incorporated through a learnable 1 × 1 convolution.Source: §3 · pp. 5–6
- The geometric component embeds box coordinates and image positions with sine-cosine features before learnable linear transformations produce the relation score.The embeddings have dimensions 4·CE for RoI boxes and 2·CE for image positions; transformed vectors share dimension Cg.Source: §3 · pp. 5–6
- Attention captures dependencies between RoI boxes and image positions, including variable locations, sizes, and aspect ratios.The box and image positions are embedded, linearly transformed into a shared dimension, and compared through an inner product.Source: §3 · pp. 5–6
- The image-adaptive component applies a learnable 1 × 1 convolution to image features, and module parameters are updated jointly during training.The resulting term uses image features adaptively in the region-feature weighting function.Source: §3 · p. 6
3.1 Complexity Analysis and an Efficient Implementation
- Sparse sampling separates positions inside the RoI from contextual positions outside it, allowing dense sampling near the RoI and sparser sampling farther away.The supporting region may be just the RoI or extend across the entire image; default maximum sampling positions are 196 for each set.Source: §3.1 · pp. 6–7
- Accuracy with sparse sampling is very close to naive dense sampling, while the method’s default maximum sample numbers yield an average actual sample number of about 200.Source: §3.1 · p. 7, Table 1 · p. 7
4 Experiments
Experiments on COCO evaluate the proposed region feature extraction module across support regions, sparse sampling, geometric and appearance terms, and detection backbones. The method benefits from broader support, preserves accuracy with sparse sampling, and matches deformable RoI pooling while achieving 39.9 mAP with FPN+ResNet-101.
4 Experiments
- Experiments use COCO train, minival, and test-dev splits with Faster R-CNN or FPN detectors and ResNet-50 or ResNet-101 backbones.The default ablation setup uses Faster R-CNN with ResNet-50, while stronger-backbone comparisons use FPN and ResNet-101.Source: §4 · pp. 7–8, Comparison on stronger detection… · p. 11
- The implementation replaces RoI pooling in FPN and uses 49 bins by default for regional feature extraction.The Faster R-CNN setup applies the proposed module to conv5 feature maps after reducing them to 256 dimensions.Source: §4 · p. 8
Effect of supporting region
- The proposed method improves steadily with larger support regions, showing that contextual information helps, whereas regular and aligned pooling gain little or none.Regular and aligned RoI pooling do not provide a clear way to exploit the whole image, so their corresponding whole-image accuracies are omitted.Source: Effect of supporting region · p. 9, Table 2 · p. 9
Effect of sparse sampling.
- Sparse sampling achieves accuracy very close to dense enumeration while significantly reducing computational overhead.The reported sparse-sampling configuration is used in subsequent experiments, and one comparison reports a 0.2 lower mAP than dense enumeration.Source: Effect of sparse sampling · pp. 9–10, Table 3 · p. 10
Effect of geometric relation and appearance feature terms.
- Adding appearance features raises mAP by 0.9 to 34.1, bringing the proposed module on par with deformable RoI pooling.Using geometric relation alone is slightly better than aligned RoI pooling and noticeably better than regular RoI pooling.Source: Effect of geometric relation and… · p. 10, Table 4 · p. 10
Comparison on stronger detection backbones.
- 39.9 mAP is achieved on COCO test-dev with FPN+ResNet-101, matching deformable RoI pooling and surpassing aligned and regular RoI pooling.The comparison uses stronger detection backbones and reports final performance for the proposed fully learnable module.Source: Comparison on stronger detection… · p. 11, Table 5 · p. 11
5 What is learnt?
The analysis shows that learned weights evolve from random distributions into distinct, foreground-focused spatial patterns, with geometric and appearance terms contributing differently. Quantitative measures confirm increasing differentiation among parts and increasing similarity between aggregated weights and object masks, suggesting an implicit connection to instance segmentation.
Qualitative Analysis
- After training, different parts focus on distinct areas of the RoI, mostly concentrating on the instance foreground.Initially, the weights are largely random across the whole image.Source: §5 · p. 11
- Geometric weights mainly attend to the RoI, whereas appearance weights focus on the entire instance foreground.This comparison is obtained by ignoring either term in turn.Source: §5 · p. 11
Quantitative Analysis
- Mean KL between parts grows dramatically during training, indicating that different parts learn to focus on different spatial positions.The divergence is larger for large objects.Source: §5 · p. 13, Figure 3 · p. 13
- KL of mask quickly becomes small, indicating that the aggregated part weights become similar to the ground-truth object mask.The max-pooled weights are normalized and compared with the normalized instance foreground mask.Source: §5 · p. 13, Figure 3 · p. 13
- The mask similarity suggests that learning the weights may be related to instance segmentation in an implicit manner.The paper identifies this connection as a direction for future investigation.Source: §5 · p. 13