Source-linked AI summary
OTA: Optimal Transport Assignment for Object Detection
Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, Jian Sun
TL;DR
Object detection label assignment often uses static, per-ground-truth positive/negative boundaries that may not fit objects with varying sizes, shapes, or occlusion. OTA formulates assignment globally as an Optimal Transport problem and achieves state-of-the-art performance on COCO while outperforming other one-stage detectors on CrowdHuman.
Problem
Static positive/negative assignment strategies may use boundaries that vary in suitability across objects with different sizes, shapes, or occlusion conditions.
Method
OTA models ground truths and background as label suppliers and anchors as demanders, minimizing transportation costs defined from classification and regression losses while estimating each ground truth’s positive-label supply dynamically.
Results
OTA achieves new state-of-the-art performance on MS COCO and outperforms all other one-stage detectors on CrowdHuman; its FCOS configuration reaches 40.7% AP with dynamic k.
Takeaways & Limitations
OTA’s global assignment resolves conflicts among ambiguous anchors through minimum global costs and remains effective across standard and crowded detection benchmarks.
Takeaways & Limitations
PAA performs poorly on CrowdHuman because crowded scenes may lack clear positive/negative boundaries, while per-ground-truth clustering increases training time.
Abstract
from arXiv · showhide
Recent advances in label assignment in object detection mainly seek to independently define positive/negative training samples for each ground-truth (gt) object. In this paper, we innovatively revisit the label assignment from a global perspective and propose to formulate the assigning procedure as an Optimal Transport (OT) problem -- a well-studied topic in Optimization Theory. Concretely, we define the unit transportation cost between each demander (anchor) and supplier (gt) pair as the weighted summation of their classification and regression losses. After formulation, finding the best assignment solution is converted to solve the optimal transport plan at minimal transportation costs, which can be solved via Sinkhorn-Knopp Iteration. On COCO, a single FCOS-ResNet-50 detector equipped with Optimal Transport Assignment (OTA) can reach 40.7% mAP under 1X scheduler, outperforming all other existing assigning methods. Extensive experiments conducted on COCO and CrowdHuman further validate the effectiveness of our proposed OTA, especially its superiority in crowd scenarios. The code is available at https://github.com/Megvii-BaseDetection/OTA.
1. Introduction
Object detectors require assigning classification and regression targets to anchors, but fixed and independently defined rules struggle with object-dependent ambiguity. OTA reframes one-to-many label assignment globally as an Optimal Transport problem and reports strong benchmark performance.
- Object detectors use dense predictions over predefined anchors, making label assignment of classification and regression targets necessary for training.
- Fixed criteria such as IoU thresholds or center regions may be sub-optimal across objects with different sizes, shapes, occlusion conditions, and categories.
- Independent per-ground-truth assignment lacks context for ambiguous anchors that qualify as positives for multiple objects and may introduce harmful gradients.
- OTA formulates one-to-many label assignment as Optimal Transport, treating ground truths as label suppliers, anchors as demanders, and background as a negative-label supplier.Pairwise ground-truth costs combine classification and regression losses, while background costs use classification loss.
2. Related Work
Object detection methods traditionally use fixed assignment criteria, while adaptive approaches often optimize each object independently. Related work had not established a global optimal assignment strategy for CNN detectors in one-to-many settings.
- Anchor-based detectors commonly assign ground truths or background using fixed IoU thresholds, while anchor-free detectors assign points around object centers as positives.
- Using a single fixed criterion across objects with varied sizes, shapes, and categories can produce sub-optimal label assignments.
- Adaptive methods such as GuidedAnchoring, MetaAnchor, and NoisyAnchors modify anchors or weighting mechanisms to make assignment more data-driven.
- These adaptive approaches focus on individual objects, whereas DeTR considers global matching but is limited to one-to-one assignment through the Hungarian algorithm.
- Before OTA, global optimal assignment for CNN-based detectors operating in one-to-many scenarios remained uncharted.
3. Method
OTA formulates object-detection label assignment as a global Optimal Transport problem, incorporating positive and negative labels across ground truths and anchors. It solves the resulting plan with Sinkhorn-Knopp Iteration and decodes each anchor according to its largest transported label amount.
- OT for Label Assignment: OTA models each ground truth as a supplier of positive labels and each anchor as a demander requiring one label.Each ground truth supplies k positive-label units, while each anchor requires one unit.
- OT for Label Assignment: The transportation cost from a ground truth to an anchor is the weighted sum of classification and regression losses.The losses use cross entropy and IoU loss, with α as the balancing coefficient; alternatives include Focal Loss, GIoU, and SmoothL1 Loss.
- OT for Label Assignment: Background supplies n − m × k negative labels so total supply equals total anchor demand.The background cost is appended as an additional row to the ground-truth cost matrix.
- OT for Label Assignment: The optimal transportation plan is obtained with Sinkhorn-Knopp Iteration and decoded by assigning each anchor to the supplier sending it the largest label amount.This iterative solution addresses the large linear program arising from anchors across all feature-pyramid levels.
- Advanced Designs: OTA combines global assignment with Center Prior and Dynamic k Estimation in its complete procedure.Center Prior restricts candidate positives toward object centers, while Dynamic k Estimation estimates each ground truth’s positive-anchor count from predicted-box IoUs.
4. Experiments
Experiments on COCO and CrowdHuman evaluate OTA’s components, candidate-radius sensitivity, ambiguity handling, and performance across detector backbones and crowded scenes. The results show strong accuracy, stability under ambiguity and radius changes, and applicability to anchor-based and anchor-free detectors.
- Component Ablations: COCO ablations show OTA outperforming FCOS by 0.9% AP without auxiliary branches, while dynamic k reaches 40.7% AP.With IoU branches, the corresponding comparisons are 39.5% v.s. 40.3% and 38.8% v.s. 39.6%, depending on center prior.
- Benchmark Results: OTA-FCOS and OTA-RetinaNet obtain exactly the same AP values, demonstrating applicability to both anchor-free and anchor-based detectors.With ResNet-101-FPN, OTA reaches 45.3% AP; ResNeXt-64x4d-101-FPN reaches 47.0% AP, 49.2% AP with DCN, and 51.5% AP with multi-scale testing.
- Candidate Radius: 40.7% AP is achieved at radius r=5, while r=3 yields 40.6% AP and r=7 causes only a 0.3% drop.The results indicate that OTA is insensitive to this hyper-parameter across the tested values.
- Ambiguous Anchors: OTA maintains low ambiguity and stable performance as r increases from 3 to 7, whereas ATSS drops from 39.4% to 37.2%.PAA’s performance drops 0.8% over the same radius range, and its ambiguity count is less sensitive than ATSS’s.
- Ambiguous Anchors: Manual ambiguity resolution before OTA decreases AP by 0.7% and 0.4%, respectively, compared with OTA’s learned global handling.Visualizations also show fewer OTA positives in overlapping or junction regions than ATSS and PAA.
- Dynamic k: Dynamic k surpasses the best fixed-k result by 0.4% AP, while increasing fixed k from 10 to 20 causes no obvious performance drop beyond 0.2%.At k=1, OTA becomes one-to-one assignment and performs poorly under the 1× scheduler.
5. Conclusion
OTA formulates object-detection label assignment as an Optimal Transport problem, transporting labels from ground-truth objects and backgrounds to anchors at minimal cost. It achieves state-of-the-art performance on MS COCO and outperforms other one-stage detectors on CrowdHuman.
- OTA formulates label assignment as an Optimal Transport problem that transports labels from ground-truth objects and backgrounds to anchors at minimal transporting costs.A simple IoU-based strategy estimates how many positive labels each ground-truth object supplies.
- OTA achieves the new SOTA performance on MS COCO.
- OTA outperforms all other one-stage detectors on CrowdHuman by a large margin, demonstrating strong generalization ability.The paper attributes this advantage to OTA’s handling of ambiguous anchors.
A.1. Optimal Transport and Sinkhorn Iteration
The Optimal Transport formulation is a large linear program for dense detectors, so the paper adds entropic regularization and solves the resulting convex optimization with Sinkhorn-Knopp Iteration. The iteration repeatedly updates variables to obtain an approximate optimal transport plan.
- The Optimal Transport problem is a linear program solvable in polynomial time, but dense detectors produce large programs involving anchors across all scales.The program’s size is associated with the square of feature dimensions.
- Entropic regularization converts the optimization target into a nonlinear but convex form suitable for a fast iterative solution.The regularization term is E(πij) = πij(log πij −1), with γ controlling its intensity.
- The constrained optimization is converted into an unconstrained target using Lagrange multipliers.
- The variables vi and uj are repeatedly updated so that the required equations are satisfied simultaneously.
- Sinkhorn-Knopp Iteration produces an approximate optimal plan after repeating the updating rule T times.The paper empirically sets γ to 0.1 and T to 50.