Source-linked AI summary
Precise Detection in Densely Packed Scenes
Eran Goldman, Roei Herzig, Aviv Eisenschtat, Oria Ratzon, Itsik Levi, Jacob Goldberger, Tal Hassner
TL;DR
Precise detection remains difficult in densely packed scenes with many nearby, often identical objects. The paper adds Soft-IoU and EM-Merger to a detector and releases SKU-110K, outperforming existing methods across detection and counting benchmarks. However, SKU-110K results remain far from perfect.
Problem
Densely packed scenes challenge precise detection because nearby objects can produce overlapping or incorrectly separated bounding boxes.
Method
The method estimates box quality with Soft-IoU and uses EM-Merger to resolve ambiguities among overlapping detections.
Results
The method surpasses existing detection and counting methods on SKU-110K, CARPK, and PUCPR+.
Takeaways & Limitations
The approach and SKU-110K provide a benchmark and technical innovations for densely packed detection and related counting tasks.
Takeaways & Limitations
SKU-110K remains highly challenging because the best reported results are far from perfect or saturated.
Abstract
from arXiv · showhide
Man-made scenes can be densely packed, containing numerous objects, often identical, positioned in close proximity. We show that precise object detection in such scenes remains a challenging frontier even for state-of-the-art object detectors. We propose a novel, deep-learning based method for precise object detection, designed for such challenging settings. Our contributions include: (1) A layer for estimating the Jaccard index as a detection quality score; (2) a novel EM merging unit, which uses our quality scores to resolve detection overlap ambiguities; finally, (3) an extensive, annotated data set, SKU-110K, representing packed retail environments, released for training and testing under such extreme settings. Detection tests on SKU-110K and counting tests on the CARPK and PUCPR+ show our method to outperform existing state-of-the-art with substantial margins. The code and data will be made available on \url{www.github.com/eg4000/SKU110K_CVPR19}.
1. Introduction
Densely packed, visually similar objects remain difficult for leading detectors because adjacent instances create overlapping and imprecise boxes. The paper introduces Soft-IoU, EM-Merger, and SKU-110K, reporting improved detection and counting performance.
- Motivation: Densely packed scenes contain many similar or identical objects in close proximity, challenging precise separation.Examples include retail shelves, traffic, and urban landscapes.
- Motivation: Leading RetinaNet detections can overlap multiple objects or split adjacent object regions into separate detections.The problem is especially visible for difficult dark objects in packed retail images.
- Contributions: Soft-IoU estimates the Jaccard index between a detected box and the unknown ground-truth box as a detection-quality score.The layer is added to the object detector.
- Contributions: EM-Merger converts detections and Soft-IoU scores into a Mixture of Gaussians and resolves overlapping detections.The unit clusters overlapping detections to produce a single detection per object.
- Contributions: SKU-110K provides a new benchmark for item detection in store-shelf images from around the world.Its table describes benchmark properties including image and object counts, classes, density, identical items, and bounding-box labels.
- Results: The method outperforms state-of-the-art detectors on SKU-110K and improves object counting on SKU-110K, CARPK, and PUCPR+ despite not being designed for counting.The abstract describes these improvements as substantial.
2. Related work
Existing detection and duplicate-merging methods were not designed for densely packed scenes, where overlapping detections create ambiguities. The paper addresses this gap with Soft-IoU-based probabilistic clustering and introduces SKU-110K as a dense detection benchmark.
- Method: The proposed method predicts Soft-IoU confidence to represent bounding-box accuracy in addition to objectness confidence.Soft-IoU supplies information distinct from hard-labeled detection confidence.
- Merging duplicate detections: Standard NMS greedily keeps high-scoring detections and removes overlapping low-confidence neighbors as test-time post-processing.The paper contrasts this hand-crafted duplicate-merging procedure with its probabilistic approach.
- Merging duplicate detections: Few NMS alternatives improve over greedy NMS, and some are computationally demanding.Dense scenes exacerbate ambiguity because they generate many overlapping detections.
- Method: Soft-IoU scores feed an EM-Merger that probabilistically clusters overlapping Gaussian detections into one detection per object.The system extends a standard detector with Soft-IoU estimation and EM-based overlap resolution.
- Crowded scene benchmarks: Existing detection benchmarks generally lack densely packed item scenes, while several crowded-scene benchmarks target counting rather than detection.The paper identifies no prior benchmark intended specifically for densely packed detection.
- Crowded scene benchmarks: SKU-110K contains one to three orders of magnitude more items per image than nearly all related benchmarks.It also includes more than 110k classes and closely positioned identical or near-identical items.
3. Deep IoU detection network
The detector augments a standard network with Soft-IoU scores and an EM-Merger that converts overlapping detections into precise, nonoverlapping detections in densely packed scenes.
- 3.3. EM-Merger unit for inference: The EM-Merger represents detections as a Mixture of Gaussians weighted by Soft-IoU, then clusters overlapping boxes to resolve one detection per object.The resulting density provides spatial confidence maps, while EM filters, merges, or splits detection clusters.
- 3.1. Base detection network: The base detector uses an FPN with three upscaling layers and ResNet-50, adding a novel third output head to each RPN.The standard heads produce bounding-box regressions and objectness scores; the third head is the Soft-IoU component.
- 3.2. Soft-IoU layer: Soft-IoU predicts the IoU, or Jaccard index, between each regressed detection box and its ground-truth object location.The score is estimated by a fully convolutional layer added as a third head to each RPN.
- 3.2. Soft-IoU layer: Objectness and Soft-IoU encode complementary box properties: objectness indicates object presence, whereas Soft-IoU estimates localization overlap.A partially overlapping box can retain high objectness while receiving a lower Soft-IoU score because its localization is less accurate.
- 3.3. EM-Merger unit for inference: EM assigns boxes to the nearest Gaussian cluster using KL divergence and re-estimates cluster parameters in the M-step.The process is initialized with agglomerative hierarchical clustering and typically converges within ten iterations under the stated stopping rule.
- 3.3. EM-Merger unit for inference: After clustering, less-confident overlapping Gaussians are suppressed, producing an actual detection count K′ no greater than the upper bound K.Final detection windows use median dimensions from original detections whose centers fall within each Gaussian’s two-standard-deviation ellipse.
4. The SKU-110K benchmark
SKU-110K is a densely populated retail-shelf benchmark designed to test detection amid tightly packed, visually varied products, with labeled images and COCO-like detection and counting evaluation.
- SKU-110K captures supermarket shelves where efficient product arrangements create extreme object density and tightly packed scenes.
- Retail products span many brands and sub-brands, introducing fine-grained appearance differences and nuisances such as occlusion and image-quality variation.
- The benchmark differs from existing alternatives in object number and density, item-class variability, and scene type.
- Images were collected worldwide using cellphone cameras, compressed to one megapixel, and annotated with bounding boxes by skilled annotators followed by visual inspection.
- The dataset uses random train, validation, and test splits that prevent the same shelf display from appearing across subsets.Training uses 70% of images, validation 5%, and testing the remaining 2,941 images.
- Evaluation reports COCO-like detection metrics together with counting errors based on predicted and ground-truth object counts.Detection includes AP, AP.75, AR300, and PR=.5; counting uses MAE and RMSE.
5. Experiments
Experiments evaluate runtime, detection, and counting on densely packed benchmarks. The full system outperforms baselines, while Soft-IoU and EM-Merger address overlap-related errors, although SKU-110K remains difficult.
- Runtime: Runtime comparisons measure FPS and DPS for baselines and approach variants on the same CPU and GPU.
- Runtime: Adding Soft-IoU does not affect runtime, whereas EM-Merger is slower because of GPU–CPU/RAM memory swapping.The authors suggest a GPU-optimized version could approach base-detector speed.
- Detection: The SKU-110K experiments compare the proposed detector with state-of-the-art baselines, including RetinaNet, Faster-RCNN, YOLO9000opt, and a random Monkey detector.
- Detection: The full system outperforms all baselines with wide margins, with much of the advantage appearing to come from EM-Merger and Soft-IoU scores.
- Detection and counting: Compared with RetinaNet, the method produces fewer overlapping or mis-localized detections and a larger counting-accuracy gap than detection-accuracy gap.
- Limitations: The best results remain far from perfect because SKU-110K's densely packed settings are highly challenging.
- Counting: On CARPK and PUCPR+, the method exceeds recent counting methods despite not being designed for counting, with lower degradation attributed to better overlap filtering.
6. Conclusions
The paper introduces a benchmark and two technical innovations for densely packed detection, then reports superiority over existing detection and counting methods while identifying substantial room for improvement.
- SKU-110K targets retail shelf scenes that challenge state-of-the-art detectors and provides item detection boxes for this setting.
- The Soft-IoU layer estimates overlap between predicted and unknown ground-truth boxes.
- The EM-based unit resolves bounding-box overlap ambiguities in tightly packed scenes.
- Tests on SKU-110K, CARPK, and PUCPR+ show the approach surpasses existing detection and counting methods.
- The best SKU-110K results are far from saturated, leaving densely packed scenes as a challenging frontier for future work.