Source-linked AI summary
Salient Object Subitizing
Jianming Zhang, Shugao Ma, Mehrnoosh Sameki, Stan Sclaroff, Margrit Betke, Zhe Lin, Xiaohui Shen, Brian Price, Radomir Mech
TL;DR
SOS asks whether an everyday image contains salient objects and how many, without first localizing them. The paper builds an annotated dataset and trains a CNN, including with synthetic images; experiments report gains for multi-object counts and applications in detection and retrieval.
Problem
The paper studies how to predict the existence and number of salient objects without localization, a cue that can support early processing choices and applications such as retrieval.
Source: §1 · p. 2Method
The paper formulates SOS as a classification task, collects an approximately 14K-image dataset, and trains a CNN with synthetic images to improve learning.
Source: §1 · pp. 2–3Results
Synthetic pretraining improves AP by more than 2% for images with 2, 3, and 4+ salient objects, while the SOS-based detection application gains about 4% F-measure over the state of the art.
Source: §1 · p. 2Takeaways & Limitations
SOS is demonstrated for suppressing false detections and estimating detection counts, as well as handling image-retrieval queries constrained by object number.
Source: §1 · p. 2The retrieval evaluation retrieves 20 images per method and assumes at least 20 relevant images for each query.
Source: §6.2 · p. 15
Abstract
from arXiv · showhide
We study the problem of Salient Object Subitizing, i.e. predicting the existence and the number of salient objects in an image using holistic cues. This task is inspired by the ability of people to quickly and accurately identify the number of items within the subitizing range (1-4). To this end, we present a salient object subitizing image dataset of about 14K everyday images which are annotated using an online crowdsourcing marketplace. We show that using an end-to-end trained Convolutional Neural Network (CNN) model, we achieve prediction accuracy comparable to human performance in identifying images with zero or one salient object. For images with multiple salient objects, our model also provides significantly better than chance performance without requiring any localization process. Moreover, we propose a method to improve the training of the CNN subitizing model by leveraging synthetic images. In experiments, we demonstrate the accuracy and generalizability of our CNN subitizing model and its applications in salient object detection and image retrieval.
1 Introduction
The paper introduces Salient Object Subitizing (SOS): predicting whether salient objects are present and how many, without localization. It presents an expanded image dataset, a CNN approach, synthetic-image training, and applications in salient object detection and retrieval.
1 Introduction
- SOS targets predicting salient-object existence and count without localization, motivated by human subitizing of small counts from one to four.The proposed dataset contains about 14K everyday images annotated through AMT and spans varied content and object categories.Source: §1 · pp. 1–2, Figure 2 · p. 2
- SOS can provide early cues for selecting processing pipelines, delaying object recognition, and setting the number of detection windows; object-count distinctions also support image retrieval and other applications.The paper names thumbnailing, robot vision, egocentric video summarization, and snap-point prediction among possible uses.Source: §1 · p. 2
- The CNN SOS model is trained end to end, and the paper examines its features and generalization to unseen object categories.The paper also presents the SOS task as image classification.Source: §1 · pp. 2–3
- Synthetic pretraining on 20K images increases AP by more than 2% for images with 2, 3, and 4+ salient objects, including about 6% for three-object images.The synthetic images combine segmented objects with background images.Source: §1 · p. 2
- An SOS-guided salient object detection system improves F-measure by about 4% over the state of the art and suppresses background detections while estimating an appropriate detection count.SOS also supports image retrieval with object-number constraints.Source: §1 · p. 2
- Compared with the preliminary work, this paper doubles the dataset to about 14K images and adds a more advanced CNN, additional real data, synthetic training data, broader analyses, and image retrieval.The earlier dataset contained about 7K images; the expanded dataset follows the same collection procedure.Source: §1 · pp. 2–3
2 Related Work
The related work distinguishes SOS from salient-object detection and presence detection because those approaches often assume one salient object or do not predict the count. It also separates SOS from conventional counting, which typically targets crowded scenes or a specific object category.
2 Related Work
- Salient-object detection can infer counts from localized objects, but many methods assume salient objects exist and are optimized for single-object images, leading to undesirable background results and difficulty with multiple objects or complex backgrounds.The cited unconstrained method handles complex images to some extent, but the paper says counting by detection is less effective than SOS for predicting salient-object counts.Source: §2 · p. 3
- Prior salient-object presence methods either assume an image contains one object or none, or detect interesting objects in simplified test images; neither predicts the number of salient objects.The methods discussed are Wang et al. (2012) and Scharfenberger et al. (2013).Source: §2 · p. 3
- Conventional object counting often targets crowded scenes and specific categories with similar-looking objects, whereas SOS distinguishes 0, 1, 2, 3, and 4+ category-independent salient objects.SOS addresses salient objects whose appearance and size can vary across categories and images.Source: §2 · p. 3
- Visual-numerosity studies use abstract binary patterns to examine number sense in neural networks, while this work targets salient-object counts in natural images.Source: §2 · p. 3
3 The SOS Dataset
The SOS dataset contains about 14K everyday images labeled by crowdsourced workers, with ambiguous cases removed. A 500-ms human test largely matched the AMT labels, while agreement varied by category.
3.1 Image Source
- Images were gathered from COCO, ImageNet, VOC07, and SUN to combine varied object, scene, and background characteristics.The authors describe these source datasets as having different content and believe their combination can mitigate individual-dataset bias.Source: §3.1 · p. 4
3.2 Annotation Collection
- AMT workers labeled images as 0, 1, 2, 3, or 4+ salient objects, with five annotations collected per image from distinct workers.Workers received example images but no specific rules for ambiguous counting cases; each task displayed five to ten images with a two-minute limit.Source: §3.2 · p. 4
3 The SOS Dataset
- The final SOS dataset contains about 14K images, after images with fewer than four consensus labels were excluded.SUN contributed mostly category 0 images, while ImageNet had significantly more images with two and three salient objects than the other source datasets.Source: §3 · p. 3, §3.2 · p. 5
3.3 Annotation Consistency Analysis
- Per-class accuracy exceeded 95% for categories 1–3, while agreement was 90% for category 0 and 78% for 4+; 13% of 4+ images were labeled 0 offline.Despite these category-specific discrepancies, the authors interpret the test as showing that limiting viewing time did not significantly change answers on most images.Source: §3.3 · pp. 5–6, Figure 5 · p. 5
- Some consistent AMT/offline disagreements were associated with central objects seeming more salient under brief viewing, while images with many foreground objects could be labeled 0 offline.The latter examples contained far more objects than the subitizing limit of four.Source: §3.3 · p. 6
4 Salient Object Subitizing by Convolutional Neural Network
The paper frames salient-object subitizing as image classification, predicting object presence and number without localization. It also proposes synthetic-image pre-training followed by real-image fine-tuning.
4 Salient Object Subitizing by Convolutional Neural Network
- The SOS task is formulated as image classification that predicts the number of generic salient objects without localization or counting.The proposed model is an end-to-end CNN trained by fine-tuning a pre-trained model.Source: §4 · p. 6
4.1 Leveraging Synthetic Images for CNN Training
- Synthetic images are made by pasting 1–4 transformed copies of a cutout object onto a SUN background, with objects and backgrounds filtered using 0.95 confidence thresholds.The cutout objects come from THUS10000; images are rejected if a pasted object is occluded by more than 50% of its area.Source: §4.1 · pp. 6–7
- Although the synthetic images look unrealistic, the authors say they are visually consistent with the specified object counts and may diversify object, scene, position, and occlusion patterns.The proposed use is as an auxiliary source to improve CNN generalizability, rather than to train with synthetic images alone.Source: §4.1 · pp. 6–7, Figure 7 · p. 7
- Two-stage fine-tuning on synthetic data and then real data works significantly better than combining both image sources into one training set.The authors describe the approach as transferring learned features from the synthetic domain to the real domain.Source: §4.1 · p. 7
5 Experiments
Experiments compare CNN variants and feature baselines, test synthetic-image training, and analyze performance across object counts, architectures, and unseen categories. Synthetic pre-training improves results, while performance varies by task and the model’s generalization remains stronger with category coverage.
5.1 Experimental Setting
- The dataset was split into 10,966 training and 2,741 test images; AP was evaluated across five CNN runs, with feature-based SVM baselines also tested.CNN training used image resizing, augmentation, and fine-tuning; synthetic pre-training used 5,000 images per object-count class from 1 to 4.Source: §5.1 · pp. 7–8
5.2 Results
- Pre-fine-tuning with synthetic images raised CNN FT mAP by about 2 absolute points; category 3 AP rose about 6 points, while CNN FT reached 78.6% mAP after a 17-point gain.Synthetic-only training scored 54.0% mAP; directly augmenting real training data with synthetic images did not improve and slightly worsened performance, which the authors attribute to domain shift.Source: Table 3 · p. 8, §5.2 · p. 9
- CNN Syn FT achieved about 93% recall for categories 0 and 1, close to human accuracy, while a considerable human–machine gap remained for the other categories.The model tended to misclassify images into a nearby category.Source: §5.2 · p. 9, Figure 8 · p. 9
5.3 Analysis
- Counting by detection performed worse than SOS in every category except 2; for categories 0 and 4+, it lagged by about 30 absolute percentage points.The authors identify cross-image score calibration and suppression of overlapping windows under occlusion as difficulties for detection-based counting.Source: §5.3 · pp. 9–10, Figure 8 · p. 9
- VGG16 and GoogleNet performed similarly, AlexNet performed significantly worse, and synthetic pre-training helped all three architectures; the current best method exceeded the prior model by 10 points, though dataset splits differed.The authors caution that the prior-work scores are not comparable because its training and test sets were subsets of the expanded dataset.Source: §5.3 · p. 10, Table 4 · p. 10
- With only 25% of real training data, synthetic images provided about a 5% absolute mAP gain to 76%; using 50% real data without synthetic images yielded 75% mAP.The authors suggest synthetic images can reduce data-collection costs while retaining competitive performance.Source: §5.3 · pp. 10–11, Table 5 · p. 10
- Novel CNN features were relatively category-independent and captured SOS-related patterns, including close-up faces and pairs of side-by-side objects.The visualized channels were selected for low similarity to original ImageNet features and shown through their highest-activation image patches.Source: §5.3 · p. 11, Figure 10 · p. 12
- Leave-one-category-out models outperformed chance for each held-out object category, but full supervision improved performance substantially, underscoring the value of category-diverse training data.The test included the held-out category and original zero-object images; synthetic images were excluded because they lacked category labels.Source: §5.3 · pp. 11–12, Table 6 · p. 12
6 Applications
SOS guides unconstrained salient-object detection by predicting detection-window counts and supports image retrieval for number-constrained object queries. It improves detection F-measure and gives higher average retrieval nDCG across number groups, with weaker retrieval results for three-object queries.
6.1 Salient Object Detection
- SOS tunes USOD’s precision-recall parameter so the detector outputs no more windows than the predicted salient-object count.The method uses grid search to select the parameter value.Source: §6.1 · pp. 12–13
- USOD+SOS raises F-measure by about 4% over USOD, while USOD+GT adds a further 3% as the upper bound for improvement.Table 7 reports salient-object detection performance on the MSO dataset.Source: §6.1 · p. 13, Table 7 · p. 13
- On images containing salient objects, SOS improves baseline USOD by about 1 absolute percentage point in F-measure.The passage attributes this to helping determine the number of detection windows.Source: §6.1 · p. 13
- On the web-thumbnail test set, SOS outperforms Wang et al.’s model despite being trained on a different dataset, and predicts without saliency-map computation.The compared model requires several saliency maps, reported to take over 4 seconds per image.Source: §6.1 · p. 13, Table 8 · p. 14
6.2 Image Retrieval
- The SOS-based retrieval method replaces number-tag confidence with the corresponding SOS count confidence when ranking number-object queries.The experiment applies it to queries such as “three animals.”Source: §6.2 · p. 14
- SOS retrieval has consistently higher average nDCG across object-count queries, especially above one object, but three-object scores are lower.The paper links the weaker three-object performance to lower SOS accuracy and categories with few three-object images.Source: Figure 12 · p. 15, §6.2 · p. 15
- The object-tag-only baseline performs well for single-object retrieval but worst for other count groups, consistent with single tags favoring one dominant object.When no number is specified, that single-object bias may be desirable.Source: §6.2 · p. 15
- Text-based retrieval is worse than SOS across number groups; SOS’s category-agnostic formulation supports varied number-object queries despite number-tag bias toward people.The text-based method combines object-tag and number-tag confidence scores.Source: §6.2 · p. 15
7 Conclusion
The paper formulates Salient Object Subitizing as global-feature prediction of salient-object existence and count without localization, and presents a CNN, dataset, and synthetic-image training approach. Experiments report effectiveness, generalizability, and applications in object detection and image retrieval.
7 Conclusion
- SOS predicts salient-object existence and count from global image features without localization.The paper presents a CNN model and an SOS image dataset for the task.Source: §7 · p. 16
- Synthetic images are used to improve CNN model training, and experiments examine its effectiveness and generalizability.The paper also visualizes learned generic patterns and reports generalization to unseen object categories.Source: §7 · p. 16
- The paper demonstrates SOS applications in unconstrained salient-object detection and content-based image retrieval, including number-object queries.It reports improved state-of-the-art salient-object detection performance.Source: §7 · p. 16