Source-linked AI summary
Learning Open-World Object Proposals without Learning to Classify
Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, Weicheng Kuo
TL;DR
Learning-based object proposals often overfit training categories and miss novel objects, limiting open-world generalization. OLN addresses this by learning classification-free localization-based objectness, and it outperforms existing approaches across category, dataset, and long-tail evaluations.
Problem
Learning-based proposals often overfit annotated categories and treat unlabeled or unknown objects as background, hindering generalization to novel objects.
Method
OLN replaces foreground-background classifiers with localization-quality predictions based on cues including centerness, IoU, and regression.
Results
OLN outperforms existing methods on COCO cross-category and RoboNet, Object365, and EpicKitchens cross-dataset evaluations, while improving LVIS long-tail detection.
Takeaways & Limitations
Pure localization-based objectness supports proposal generalization beyond labeled categories and data sources, including large-vocabulary and egocentric-video detection.
Takeaways & Limitations
The paper does not explore advanced convolutions for feature alignment or box-regression statistics that might further improve performance.
Abstract
from arXiv · showhide
Object proposals have become an integral preprocessing steps of many vision pipelines including object detection, weakly supervised detection, object discovery, tracking, etc. Compared to the learning-free methods, learning-based proposals have become popular recently due to the growing interest in object detection. The common paradigm is to learn object proposals from data labeled with a set of object regions and their corresponding categories. However, this approach often struggles with novel objects in the open world that are absent in the training set. In this paper, we identify that the problem is that the binary classifiers in existing proposal methods tend to overfit to the training categories. Therefore, we propose a classification-free Object Localization Network (OLN) which estimates the objectness of each region purely by how well the location and shape of a region overlap with any ground-truth object (e.g., centerness and IoU). This simple strategy learns generalizable objectness and outperforms existing proposals on cross-category generalization on COCO, as well as cross-dataset evaluation on RoboNet, Object365, and EpicKitchens. Finally, we demonstrate the merit of OLN for long-tail object detection on large vocabulary dataset, LVIS, where we notice clear improvement in rare and common categories.
1. Introduction
Learning-based object proposals often overfit labeled categories, limiting novel-object generalization. OLN replaces foreground-background classification with localization-based objectness and improves cross-category, cross-dataset, and long-tail detection results.
- Learning-based proposals can overfit annotated categories, treating unlabeled or unknown objects as background and struggling with novel objects.
- OLN learns objectness from localization cues such as centerness, IoU, and regression instead of foreground-background classification.
- +3.3 AUC (+5.0 AR@10, +5.1 AR@100) is achieved by OLN over state-of-the-art methods on novel categories in COCO cross-category evaluation.
- +13∼16 AP is achieved by OLN over the standard approach on RoboNet, while Objects365 gains are +4 AR@10 and +8 AR@100.
- +1.4 AP is observed when OLN replaces RPN for LVIS long-tail detection, with gains of +3.4 APr in rare categories.
- The study also reports improvements across EpicKitchens, open-world class-agnostic detection, and proposal evaluation using Average Precision on exhaustively annotated data.
2. Related Work
Related work spans classical category-independent proposals, learning-based proposal networks, scalable detectors, cross-dataset evaluation, and localization-quality estimation. OLN differs by targeting category-independent detection without classification and by studying transfer across categories and datasets.
- Classical object proposals use hand-crafted cues such as color contrast and edges to identify objects irrespective of category.
- Learning-based proposals improve precision and recall and support two-stage detectors, with RPN identifying candidate regions for downstream localization and classification.
- Prior work studies unseen-category and cross-dataset generalization, whereas OLN focuses on modeling choices for proposal generalization.
- Multi-class detectors expand category coverage but still fail to generalize to unseen or unknown object categories.
- Cross-dataset work motivates training on COCO and testing on RoboNet, Objects365, EpicKitchens, and LVIS because objectness cues may transfer better than class-specific information.
- Localization-quality methods learn centerness, proposal refinement, or box/mask IoU, but their transfer to novel categories remains an open question and they commonly accompany classifiers.
- OLN replaces classifier heads with localization-quality estimators for both whole-image anchors and regions of interest.
3. Proposed Method
OLN replaces foreground-versus-background classifiers in proposal networks with localization-quality predictions, aiming to generalize objectness beyond labeled categories. Its two-stage design uses centerness and IoU cues, with an optional mask-IoU extension and geometric-mean scoring at inference.
- Pure localization-based objectness: Learning-based proposal objectness treats membership in predefined categories as binary classification, which can label unknown objects as background.OLN addresses this limitation with a non-discriminative, classification-free notion of objectness.
- Pure localization-based objectness: OLN estimates objectness from how well regions overlap ground-truth objects, using geometric cues such as centerness and IoU.Centerness is used in OLN-RPN, while IoU is used in the second-stage OLN-Box branch.
- Object Localization Network (OLN): OLN replaces proposal-stage classifiers with localization-quality predictions to avoid overfitting to annotated foreground categories.The approach targets generalization to unseen objects by learning localization rather than foreground-versus-background discrimination.
- Object Localization Network (OLN): The two-stage proposer follows Faster R-CNN, with fully convolutional and RoI stages whose classification heads are replaced by localization-quality prediction branches.OLN-RPN predicts box regression and localization quality from feature-pyramid maps; the RoI stage refines proposals using region features.
- Object Localization Network (OLN): OLN-Mask extends OLN-Box with a class-agnostic mask head and mask-IoU prediction to test whether additional localization learning improves generalization.Training assumes mask annotations for the training categories and uses smooth-L1 loss for mask-IoU regression.
- Object Localization Network (OLN): At inference, OLN-Box and OLN-Mask combine centerness, box-IoU, and optionally mask-IoU through geometric-mean objectness scores.The box score is p(c · b), while the mask score is 3p(c · b · m).
4. Experiments
Experiments evaluate OLN across cross-category, cross-dataset, and long-tail detection settings. Localization-based objectness consistently improves generalization, while adding classification harms proposal recall.
- Ablations: Localization cues outperform classification in ablations; centerness gives the best AR, while the second stage further improves proposal performance.
- Ablations: Adding a binary classifier consistently decreases AR across single-stage and two-stage OLN configurations, with the largest drops when classifiers appear in both stages.
- Qualitative analysis: OLN generalizes to novel objects missed by Mask R-CNN, including salient out-of-VOC objects and objects in egocentric videos.
- Cross-dataset generalization: On RoboNet, OLN exceeds RPN and Faster R-CNN by +12.7 and +15.7 AP, respectively, while Object365 gains reach +8.2 and +7.6 AR100 points over baselines.
- Long-tail detection: Replacing RPN with OLN-RPN on LVIS improves overall AR by +1.5 and AP by +1.4, with the largest gains in rare categories: +5.3 AR and +3.4 AP.
5. Conclusion
The paper addresses novel object proposal learning by replacing category-sensitive classification with localization cues. OLN improves cross-category and cross-dataset generalization and benefits long-tail detection.
- OLN learns novel object proposals through centerness, IoU, and regression cues instead of binary classification.
- OLN outperforms existing methods on COCO cross-category generalization and cross-dataset evaluations on RoboNet, Object365, and EpicKitchens.
- Replacing RPN with OLN improves large-vocabulary and egocentric-video object detection, including long-tail detection on LVIS.