Source-linked AI summary
AutoAssign: Differentiable Label Assignment for Dense Object Detection
Benjin Zhu, Jianfeng Wang, Zhengkai Jiang, Fuhang Zong, Songtao Liu, Zeming Li, Jian Sun
TL;DR
Dense object detectors need to assign positive and negative samples across spatial locations and feature scales, yet fixed human-designed rules do not adapt fully to category distributions and object appearances. AutoAssign introduces a fully differentiable, appearance-aware weighting strategy using Center Weighting and Confidence Weighting. It consistently improves existing sampling strategies by about 1% AP across backbones on MS COCO and transfers competitively across other datasets and tasks.
Problem
Dense detection requires positive/negative label assignment across spatial and scale dimensions, while fixed center-based strategies cannot capture diverse category and object-appearance distributions.
Method
AutoAssign uses a unified, fully differentiable weighting mechanism with category-wise Center Weighting and instance-specific Confidence Weighting.
Results
AutoAssign consistently improves existing sampling strategies by ∼1% AP with various backbones on MS COCO and transfers to other datasets and tasks without additional modification.
Takeaways & Limitations
The method provides a data-driven, appearance-aware label assignment strategy with broad applicability across datasets and tasks.
Takeaways & Limitations
The current weighting mechanism is not simple enough and may be simplified in future work.
Abstract
from arXiv · showhide
Determining positive/negative samples for object detection is known as label assignment. Here we present an anchor-free detector named AutoAssign. It requires little human knowledge and achieves appearance-aware through a fully differentiable weighting mechanism. During training, to both satisfy the prior distribution of data and adapt to category characteristics, we present Center Weighting to adjust the category-specific prior distributions. To adapt to object appearances, Confidence Weighting is proposed to adjust the specific assign strategy of each instance. The two weighting modules are then combined to generate positive and negative weights to adjust each location's confidence. Extensive experiments on the MS COCO show that our method steadily surpasses other best sampling strategies by large margins with various backbones. Moreover, our best model achieves 52.1% AP, outperforming all existing one-stage detectors. Besides, experiments on other datasets, e.g., PASCAL VOC, Objects365, and WiderFace, demonstrate the broad applicability of AutoAssign.
1. Introduction
Dense detectors must assign positive and negative labels across spatial locations and feature scales, but fixed human-designed rules cannot fully adapt to object appearance and category distributions. AutoAssign addresses this with a unified, fully differentiable weighting mechanism and reports state-of-the-art results across datasets.
- Motivation: Dense prediction makes label assignment essential for selecting positive and negative locations and appropriate feature levels across spatial and scale dimensions.Multi-scale features require both spatial assignment and scale assignment.
- Motivation: Fixed strategies such as anchor matching and center sampling rely on human priors that may select background locations when object appearances vary.Examples include bananas and umbrellas, for which fixed center sampling may choose locations outside objects.
- AutoAssign: AutoAssign removes anchors, IoU thresholds, and top-k rules, replacing them with a fully data-dependent, appearance-aware weighting mechanism across spatial and scale dimensions.The method treats locations inside each object’s bounding box as candidate positives and negatives for further optimization.
- AutoAssign: Center Weighting and Confidence Weighting adjust category-specific priors and instance-specific sampling strategies in both spatial and scale dimensions.Center Weighting adapts to category distributions, while Confidence Weighting adapts to each instance’s appearance and scale.
- Results: AutoAssign achieves state-of-the-art performance on MS COCO and competitive results on PASCAL VOC, Objects365, and WiderFace.The experiments are presented as evidence of effectiveness and broad applicability.
2. Related Work
Label assignment methods have progressed from fixed anchor- or center-based rules toward adaptive strategies. Existing dynamic methods modify only parts of the assignment process, leaving other decisions constrained by human design.
- Fixed Label assignment: Anchor-based detectors preset anchors and use IoU matching to assign positives and negatives across spatial and scale dimensions.This strategy underlies detectors including Faster R-CNN, YOLO, SSD, and RetinaNet.
- Dynamic Label assignment: Recent adaptive methods dynamically change anchor shapes or modify spatial or scale sampling for individual objects.GuidedAnchoring and MetaAnchor adapt anchor priors, while other methods directly change sampling.
- Dynamic Label assignment: These partially dynamic strategies free only part of label assignment from human design, leaving the remaining decisions constrained.The limitation motivates more fully data-driven assignment strategies.
3. Methodology
AutoAssign replaces hand-crafted label-assignment rules with a unified, appearance-aware, fully differentiable weighting mechanism that jointly selects spatial locations and FPN scales. It combines category-level center priors with instance-level confidence to generate positive and negative weights for training.
- Overview: AutoAssign makes label assignment fully data-dependent by replacing anchors, IoU thresholds, and top-k rules with unified weighting across spatial locations and scales.The method jointly optimizes assignment with the detector through differentiable weight maps.
- Prior-level: Center Weighting: Category-specific Center Weighting uses a learnable Gaussian-shaped function to adapt the center prior to different object-category distributions.Its learnable parameters control category-dependent center offsets and spatial importance, optimized by back-propagation.
- Loss function: Positive weights combine Center and Confidence Weighting, whereas negative weights use Confidence Weighting and IoU-based proposal information.The resulting weight maps dynamically select appropriate spatial locations and FPN stages, and contribute directly to the training loss.
- Instance-level: Confidence Weighting: Given each object, locations inside its bounding box across FPN stages begin as both positive and negative candidates for further optimization.This differs from strategies that preselect only a subset of positive locations before loss calculation.
- Instance-level: Confidence Weighting: Confidence Weighting combines classification and localization confidence to adapt positive and negative sampling to each instance’s appearance and scale.Positive confidence multiplies classification and localization confidence, while negative candidates use classification confidence alone.
4. Experiments
Experiments show that AutoAssign’s Center and Confidence Weighting modules improve label assignment by adapting spatial, scale, category, and instance-specific weights. The method achieves strong MS COCO performance and generalizes across datasets with different distributions.
- Baseline: 17.7 mAP is the clean AutoAssign baseline without positive and negative weighting modules.The baseline relies only on the center prior and omits anchors, IoU thresholds, and top-k selection.
- Overall weighting mechanism: Center Weighting and Confidence Weighting jointly perform spatial and scale assignment for each instance.Center Weighting provides coarse category-specific center priors, while Confidence Weighting adapts weights to object appearances.
- Center Weighting: Category-wise priors increase MS COCO AP by 0.2% over fixed priors, whereas shared priors decrease AP by 0.1%.The largest improvements occur for categories with unique distributions, including surfboard and hotdog.
- Center Weighting: Learned priors shift centers and change spread by category, including elliptical distributions for motorcycle and surfboard.Parking meter and hotdog priors shift downward, while motorcycle and surfboard priors adapt to shape characteristics.
- Confidence Weighting: Combining classification and localization confidence achieves higher performance than using either confidence alone.The result indicates that a joint confidence indicator is preferable for evaluating location quality.
- Comparison with State-of-the-art: 52.1% AP is achieved by the best AutoAssign model, outperforming all existing one-stage detectors on MS COCO test-dev.With a ResNet-101 backbone, AutoAssign achieves 44.5% AP under the same training setting.
5. Conclusions
AutoAssign is presented as a differentiable, data-driven label assignment strategy that improves detection across backbones and transfers across datasets and tasks. The weighting mechanism remains more complex than desired, motivating future simplification.
- AutoAssign consistently improves existing sampling strategies by approximately 1% AP across various backbones on MS COCO.
- Extensive experiments indicate that AutoAssign transfers to other datasets and tasks without additional modification.
- The current weighting mechanism is not simple enough and is identified as a target for future work.
A.1. Baseline
The baseline analysis explains how competing assignment strategies can be reconstructed by adding their characteristic components to the AutoAssign baseline. Official FCOS imprv is built progressively through center sampling, scale rules, and a center-ness branch.
- RetinaNet is implemented by adding nine anchors to each FPN stage and corresponding positive and negative IoU thresholds.The passage describes 9×5 anchors in total across five FPN stages.
- FCOS imprv is implemented by adding center sampling, artificial FPN scale assignment rules, and a center-ness branch.
- Table 7 reports the step-by-step construction of FCOS imprv on top of the baseline, with its last row corresponding to official FCOS imprv.
A.2. Initialization of Center Weighting
The initialization analysis tests whether Center Weighting’s learnable parameters are useful. Performance is sensitive to the initial parameters, while releasing the parameters allows performance to recover gradually.
- Fixing Center Weighting parameters µ and σ enables analysis of whether the learnable mechanism works and is useful.
- Standard Gaussian initialization performs well, whereas substantially different fixed parameters reduce performance.
- Releasing µ or σ after altered initialization produces gradual performance recovery, indicating that the learnable mechanism is functioning.
- Table 8 reports AutoAssign results under different settings of µ and σ, highlighting the best performance.
A.3. τ in Confidence Weighting
The Confidence Weighting analysis compares different function slopes and finds that intermediate behavior performs best. The passage relates this slope hyperparameter to γ in Focal Loss.
- Neither overly flat nor overly steep Confidence Weighting slopes achieve the best performance.
- The Confidence Weighting slope hyperparameter is conjectured to be similar to γ in Focal Loss.
- Table 9 compares different τ values in the Confidence Weighting functions and highlights the best performance.
A.4. Implicit-Objectness.
Implicit-Objectness improves detector performance across methods and benefits AutoAssign most, suggesting stronger interaction with its dynamic weighting mechanism.
- ∼1.0 AP gain from ImpObj in AutoAssign exceeds its gains on other detectors.The authors attribute this stronger benefit to better interaction between dynamic weighting and implicit objectness.
B. Visualization
Visualization indicates that AutoAssign’s improvements arise from better foreground–background separation and progressively more appropriate confidence weighting across FPN stages.
- Confidence weighting is initially weak, then converges toward appropriate FPN stages for objects of different sizes.As training progresses, location confidences become more salient and the learned strategy adapts across object scales.
- ImpObj improves both recall and precision by filtering noise and separating foreground from background.Figure 5 compares confidence visualizations with and without ImpObj; Table 10 reports its application across methods.