Source-linked AI summary
Background-Free Objectness Learning for Class-Agnostic Detection
Dania Batool, Liliana Lo Presti, Marco La Cascia, Filippo Vella
TL;DR
Incomplete-annotation supervision treats unlabeled objects as background, biasing objectness toward the annotated taxonomy and limiting class-agnostic localization. B-FOR learns dense multi-scale center and scale fields using supervision confined to reliable annotated regions, then decodes local maxima into boxes. Across unseen-category and cross-dataset benchmarks, it improves recall by up to +10 AR@1000 and reports strong generalization, while decoding remains a limitation for low-budget recall.
Problem
Under incomplete annotations, unlabeled object-like regions become background negatives, causing objectness to depend on the annotated taxonomy rather than category-independent localization.
Method
B-FOR predicts dense multi-scale object-center and center-conditioned scale fields, supervising only reliable annotated regions while treating unlabeled locations as unobserved.
Results
+10 AR@1000 is reported over UniDetector on VOC20→COCO60, alongside strong generalization to unseen categories and across dataset shifts.
Takeaways & Limitations
B-FOR provides evidence that objectness can be learned as a spatially distributed property without explicit background supervision on unlabeled regions.
Takeaways & Limitations
The decoding stage is not jointly optimized with the learned fields, limiting low-budget recall such as AR@1.
Abstract
from arXiv · showhide
Object detectors are typically trained under closed-set supervision, where unlabeled regions are implicitly treated as background. Under incomplete annotations, this assumption introduces objectness bias: visually valid but unlabeled objects are used as negatives, tying objectness to the annotated taxonomy rather than generic object structure. This limitation is particularly problematic for class-agnostic and open-world detection. This paper proposes Background-Free Objectness Learning (B-FOR), a dense class-agnostic detection framework that learns objectness without explicit background supervision on unlabeled regions. B-FOR formulates detection as the prediction of dense multi-scale object-center and scale fields, from which object hypotheses emerge as local spatial structures. Supervision is confined to reliable annotated regions through spatially structured soft targets, avoiding foreground-background discrimination. To support decoding from emergent local maxima, the paper further introduces displacement-aware scale fields that model object extent as a spatially varying property of the learned objectness field. Experiments on PASCAL VOC, MS-COCO, and Open Images demonstrate strong generalization to unseen categories and cross-dataset object distributions. B-FOR improves recall by more than +10 AR points over prior class-agnostic baselines. Ablation studies show that both localized objectness supervision and displacement-aware scale fields are critical for class-agnostic localization under incomplete annotations. Code available at: https://github.com/Daniaawan/B-FOR.
1 Introduction
B-FOR addresses objectness bias from incomplete annotations by learning class-independent objectness without treating unlabeled regions as background. It predicts spatially structured center and scale fields, yielding improved unseen-category and cross-dataset recall.
- Motivation: Incomplete annotations cause detectors to treat unlabeled object-like regions as background, making objectness depend on the annotated taxonomy.This creates a central limitation for class-agnostic localization.
- Approach: B-FOR predicts multi-scale object-center score maps with width and height fields, allowing object instances to emerge as local spatial structures decoded into boxes.The scale fields represent object extent associated with candidate centers.
- Results: +10 AR@1000 and 2× AR@100 at IoU ≤0.5 are reported gains on unseen-category and cross-dataset class-agnostic benchmarks.The reported improvements are particularly strong for unseen objects and include cross-dataset generalization beyond annotated categories.
- Approach: B-FOR confines spatially structured supervision to reliable annotated regions and treats unlabeled locations as unobserved rather than explicit background negatives.The formulation shifts learning away from foreground-background discrimination.
- Relation to prior work: Prior approaches mitigate missing annotations with heuristics but retain foreground-background objectives, whereas B-FOR removes background labels from unlabeled regions.This distinguishes B-FOR's supervision formulation from pseudo-labeling, ignore-region, and related methods.
3 Method
B-FOR represents class-agnostic detection as dense, multi-scale prediction of object-center evidence and center-conditioned extent. Inference extracts local maxima, decodes their scale fields into boxes, and applies ranking and NMS.
- Dense representation: The model predicts object-center responses independently across spatial locations and three object scales: small, medium, and large.The score field encodes object-center evidence without global normalization.
- Dense representation: Each candidate center is paired with a scale field encoding normalized width and height, representing the object's spatial extent.This couples localized center evidence with an extent representation.
- Architecture: The FPN encoder-decoder uses separate lightweight heads at each pyramid level to estimate object-center and scale fields.Multi-scale feature maps are decoded into dense predictions on the image grid.
- Detection decoding: Inference retains locations that are local maxima above a level-specific threshold, ranks the top-K peaks, recovers boxes, and applies NMS.The final detections combine candidates from all pyramid levels before NMS and top-Dmax selection.
- Detection decoding: Local-maxima extraction, thresholding, top-K selection, and NMS are applied only during inference, not training.These decoding operations convert dense fields into a finite ranked detection set.
4 Learning Objectness
B-FOR learns class-agnostic objectness from reliable annotated regions without treating unmatched locations as background, combining localized center supervision with dense, displacement-aware scale fields. This formulation lets object hypotheses emerge as spatially coherent local structures while modeling geometry away from exact annotated centers.
- Region-Supervised Objectness Learning: B-FOR excludes locations outside annotated boxes from objectness supervision, avoiding explicit negative labels where missing annotations may hide objects.The loss is evaluated only over annotated regions, while each annotated box supervises a decoder branch selected by box scale.
- Region-Supervised Objectness Learning: Continuous spatial proximity targets replace foreground-background labels within annotated regions, producing localized supervision around annotated object centers.Gaussian soft targets are optimized with binary cross-entropy, and normalization by region size equalizes contributions across object extents.
- Region-Supervised Objectness Learning: A center-consistency regularizer uses differentiable soft-argmax predictions to stabilize local maxima near annotated object centers without adding objectness supervision.The predicted center is estimated from the regional score map, and λctr controls the regularization strength in the final objectness objective.
- Learning Dense Scale Fields: Dense scale fields predict the width and height of boxes decoded from candidate locations, making object extent spatially varying with displacement and object scale.A candidate box can enclose the annotated object by expanding symmetrically when its decoding location is offset from the annotated center.
- Learning Dense Scale Fields: Scale-field supervision is restricted to compact neighborhoods around annotated and predicted centers to limit off-center emphasis and interference between nearby objects.The predicted center is obtained with differentiable soft-argmax over the object-center score map within each annotated region.
- Learning Dense Scale Fields: Localized center supervision and dense scale fields produce spatially coherent object hypotheses from local maxima rather than isolated center activations.Unlike center-only size regression or foreground-assigned regression, the formulation learns geometry associated with candidate centers emerging throughout the objectness field.
5 Experimental Setup and Results
Experiments evaluate B-FOR on unseen-category and cross-dataset class-agnostic detection using average recall, showing strong generalization and identifying localization, decoding, and evaluation caveats.
- Evaluation Protocols: Evaluation covers unseen categories and cross-dataset generalization on VOC, COCO, and Open Images using class-agnostic average recall.Protocols include VOC seen/unseen splits, VOC-to-COCO and COCO-to-Open Images unseen-category tests, and cross-dataset evaluation with overlapping and novel categories.
- Unseen-Category Results: B-FOR achieves AROvr = 80.3 on the VOC 17/3 split, outperforming the Faster R-CNN class-agnostic baseline at 60.1.The FPN-based model also outperforms its single-scale variant across all metrics, supporting multi-scale object-center representations.
- Cross-Dataset Results: B-FOR reaches AROvr = 48.7 on VOC20→COCO60, exceeding UniDetector’s 38.1 by +10.6 points, and reaches 60.2 on COCO80→Open Images490.Recall is particularly strong for medium and large objects in the VOC-to-COCO evaluation.
- Evaluation Interpretation: Manual inspection found 48% of VOC20-trained top-1 predictions were valid unlabeled objects, while recall reached 38.0 AR@100 as the proposal budget increased.This illustrates why top-ranked detections can be counted as false positives under incomplete annotations.
- Cross-Dataset Results: B-FOR maintains AROvr = 53.9 for VOC→COCO and 88.3 for COCO→VOC, indicating generalization across dataset and annotation shifts.VOC→COCO recall is especially high for medium and large objects, at 75.8 and 93.5, respectively.
- Ablation Studies: The full ablation model achieves AROvr = 48.7; removing closed-set background supervision adds +21.5 points over LCE + LSF, while the scale-field objective adds +15.3 over local width/height regression.Restricting supervision to local neighborhoods improves AROvr from 18.5 to 33.4, and removing center loss reduces it to 23.6.
- Decoding Robustness: Decoding is stable through approximately the 80th percentile threshold, but aggressive thresholding beyond the 90th percentile suppresses informative responses.Local-maximum filtering, percentile thresholding, top-K selection, and NMS can nevertheless suppress, merge, or discard valid detections.
6 Conclusion
B-FOR reduces objectness bias by learning from reliable annotated regions without explicit background supervision, using spatially structured object-center and scale fields. It generalizes across unseen categories and dataset shifts, while decoding remains a limitation for low-budget recall.
- B-FOR avoids explicit background supervision outside ground-truth boxes and learns objectness from spatially structured soft targets in reliable annotated regions.Object hypotheses are local maxima of multi-scale object-center fields, decoded with displacement-aware scale fields.
- B-FOR achieves AROvr = 80.3 on VOC seen/unseen, +10.6 AR@1000 over UniDetector on VOC20→COCO60, and AROvr = 60.2 on COCO80→OpenImages490.These results are reported without ImageNet pretraining.
- A current limitation is that local-maximum extraction and NMS are not jointly optimized with learned fields, limiting low-budget recall such as AR@1.Learning structured decoding directly from dense objectness fields is identified as future work.
A Objectness Bias Visualization
The visualization contrasts objectness scores for labeled and unlabeled objects, showing that B-FOR produces more comparable responses than foreground-background-trained baselines.
- CenterNet and FCOS assign low objectness to unlabeled objects, whereas B-FOR produces comparable responses for labeled and unlabeled objects.The figure uses green boxes for labeled objects, red boxes for unlabeled objects, and filled circles for per-object scores.
- Removing explicit background supervision over unlabeled regions substantially alleviates objectness bias and supports a more category-agnostic objectness representation.
B Qualitative Results
B-FOR produces dense multi-scale object-center responses that extend beyond final detections and can localize visually coherent unlabeled objects. It also sometimes responds to salient object parts, which standard box evaluation may count as false positives.
- B-FOR object-center fields activate on coherent object regions across multiple scales, while box decoding and NMS retain only a subset as final detections.The fields are dense, whereas the final detection set D(I) is sparse.
- B-FOR responds to visually coherent yet unlabeled objects in COCO images using a model trained on COCO80.The qualitative results show the top-30 detections per image alongside ground-truth boxes.
- B-FOR sometimes assigns high object-center scores to salient object parts because training uses box-level evidence without semantic class or part-level annotations.Examples include person torsos, feet, animal body regions, and vehicle components detected as separate candidates.
- Objectness bias visualization compares labeled and unlabeled objects using green and red boxes, respectively, with filled circles indicating objectness scores.
- Part detections can be counted as false positives when they do not sufficiently overlap a complete annotated object box.The paper suggests object-wholeness cues or grouping constraints as possible remedies.
C Joint vs Staged Training
The training comparison separates object-center localization from scale prediction in staged training and optimizes them together in joint training. Joint training consistently performs better across protocols.
- Training strategies: In staged training, the object-center branch is optimized first and the scale-field branch afterward, separating center localization from box-extent prediction.This decoupled optimization often produces sharper intermediate score fields.
- Training strategies: Joint training optimizes object-center and scale-field branches simultaneously under the full objective.The two branches therefore learn together rather than through a sequential schedule.
- Comparison: Joint training consistently outperforms staged training across all protocols, indicating that improvements arise from combined objectness and scale-field supervision.
D Top-K Analysis
B-FOR’s recall gains are not primarily driven by retaining an arbitrarily large candidate pool: recall saturates well before the standard K=1200 setting.
- The analysis retains top-K score peaks independently at each of three pyramid levels before non-maximum suppression.
- Table 7 compares staged and joint training using the same architecture and inference pipeline, reporting AR@1000 overall and by object size.
- AR@1 and AR@10 remain effectively constant as the retained peak count K varies across the evaluated range.
- AR@100 plateaus by K≈200–300, while AR@1000 plateaus by K≈500–600.
- Results at K=1200 are unchanged from K=600 across AR@1, AR@10, AR@100, and AR@1000.
E Objectness Score in Pseudo-Background Regions
On conservative pseudo-background regions, B-FOR produces higher objectness scores than FCOS and CenterNet across labeled, unlabeled, and pseudo-background measurements.
- B-FOR achieves 0.42±0.18/0.38±0.17/0.29±0.12 for labeled/unlabeled/pseudo-background regions.
- FCOS achieves 0.24±0.19/0.09±0.15/0.04±0.07 for labeled/unlabeled/pseudo-background regions.
- CenterNet achieves 0.23±0.23/0.08±0.13/0.01±0.01 for labeled/unlabeled/pseudo-background regions.
- Pseudo-background regions are sampled from COCO-Stuff locations outside a safety margin around any COCO or LVIS object.
F Loss Definitions
The loss definitions contrast dense foreground-background supervision with direct size regression and B-FOR’s alternatives for incomplete annotations.
- Table 6 compares LCE and LL1 as standard baseline losses for isolating B-FOR’s two central design choices.
- LCE applies dense foreground-background supervision over the full image grid using Gaussian soft targets.
- LCE treats every nonannotated location as an explicit negative, whereas LROS excludes such locations from supervision.
- LL1 directly regresses width and height to the annotated box size at its true center.
- LL1 uses fixed targets at the annotated center instead of the offset-compensated targets used by the displacement-aware LSF.