Source-linked AI summary
Counting Everyday Objects in Everyday Scenes
Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R. Selvaraju, Dhruv Batra, Devi Parikh
TL;DR
The paper addresses category-specific object counting in natural scenes, where counts, appearances, scales, and sparsity vary substantially. It develops subitizing-inspired divide-and-conquer models with cross-cell context, finds improved counting performance on PASCAL VOC 2007 and COCO, and studies applications to detection and visual question answering.
Problem
Counting object instances in natural scenes remains challenging because objects vary in count, appearance, and scale, unlike prior work focused on restricted domains.
Method
The paper divides scenes into regions, subitizes their contents, combines counts additively, and incorporates context across cells through sequential subitizing and ensembles.
Results
The proposed approaches improve counting performance on PASCAL VOC 2007 and COCO, with an ensemble performing best, while counting also improves object detection and supports proof-of-concept VQA experiments.
Takeaways & Limitations
Counting provides a dedicated scene-understanding capability and can serve as a signal for object detection and how-many visual question answering.
Takeaways & Limitations
The sequential approach addresses a limitation of independently processed cells: similar visual signals may correspond to partial or whole object presence across cells.
Abstract
from arXiv · showhide
We are interested in counting the number of instances of object classes in natural, everyday images. Previous counting approaches tackle the problem in restricted domains such as counting pedestrians in surveillance videos. Counts can also be estimated from outputs of other vision tasks like object detection. In this work, we build dedicated models for counting designed to tackle the large variance in counts, appearances, and scales of objects found in natural scenes. Our approach is inspired by the phenomenon of subitizing - the ability of humans to make quick assessments of counts given a perceptual signal, for small count values. Given a natural scene, we employ a divide and conquer strategy while incorporating context across the scene to adapt the subitizing idea to counting. Our approach offers consistent improvements over numerous baseline approaches for counting on the PASCAL VOC 2007 and COCO datasets. Subsequently, we study how counting can be used to improve object detection. We then show a proof of concept application of our counting methods to the task of Visual Question Answering, by studying the `how many?' questions in the VQA and COCO-QA datasets.
1. Introduction
The paper studies counting category-specific object instances in natural scenes, where appearance, scale, and sparse occurrences make counting challenging. It proposes subitizing-inspired divide-and-conquer models that use context across image cells, and applies counting to detection and visual question answering.
- Counting everyday objects requires estimating category-specific instance numbers in natural scenes, such as chairs, ovens, tables, plants, and spoons.
- Natural-scene counting must handle scale and appearance variations, while most images contain zero instances for most object classes.
- The proposed associative subitizing method divides images into non-overlapping cells, subitizes each cell, and adds the regional counts.
- Sequential subitizing incorporates context across cells while sequentially subitizing them, and uses real-valued cell ground truth to address scale variations.
- The paper evaluates counting approaches, studies counting-based improvements to object detection, and examines how-many questions in VQA and COCO-QA.
2. Related Work
Prior counting work largely targets constrained settings such as crowds and surveillance, whereas everyday scenes involve different challenges. The paper therefore motivates tools suited to high intra-class variation and sparse category occurrences, while distinguishing category counting from salient-object subitizing.
- Earlier counting studies address niche or constrained settings, including crowds and surveillance video, often using regression or density-based approaches.
- Everyday-scene counting differs through high intra-class object variation and high sparsity, because most images contain zero instances for most categories.
- Salient object subitizing counts salient objects independently of category, whereas this paper counts object instances separately for each category.
3. Approach
The approach counts objects by dividing images into grid cells, assigning real-valued partial counts, and aggregating them. Sequential subitizing improves on independent cell predictions by incorporating context across cells.
- Counting framework: Counting models use CNN features to predict category counts, with glance operating on whole-image features and subitizing methods operating on grid-cell features.The models use fc7 features; glance regresses image-level integer counts, whereas aso-sub and seq-sub use cell-level real-valued counts.
- Counting framework: Real-valued cell targets represent the fraction of each object overlapping a cell, and image-level counts sum the nonnegative cell predictions.Partial counts are computed from bounding-box intersections normalized by object area; negative predictions are filtered before aggregation.
- Subitizing models: Independent aso-sub predictions can confuse visually similar partial and whole objects because they ignore object context across cells.The same local visual signal may correspond to different partial counts depending on the object’s presence in neighboring cells.
- Training choices: The experiments vary grid resolutions of 3 × 3, 5 × 5, and 7 × 7 and train seq-sub with a Huber loss optimized by Adam.The seq-sub training uses learning rate 10^-4, weight decay 0.95, and minibatches of 64.
- Subitizing models: Seq-sub addresses this ambiguity with two stacked bidirectional LSTM sequences that aggregate context across the grid before predicting cell counts.The two sequence traversals produce context vectors for each cell, which a subsequent hidden layer maps to cell-count predictions.
4. Experimental Setup
The experiments evaluate counting on PASCAL VOC 2007 and COCO using dataset splits, error metrics, baselines, and several counting-model variants. Evaluation includes both all instances and images with non-zero ground-truth counts.
- Datasets: Experiments use PASCAL VOC 2007 and COCO, with separate Count-val and Count-test splits and 20 versus 80 object categories.PASCAL uses its validation and test sets, while COCO splits its validation set into two halves.
- Datasets: The count distributions vary but are strongly biased toward low values, with zero the most frequent count per object category.This everyday-scene distribution differs from crowd-counting datasets, which contain much larger counts and less scale and appearance variation.
- Evaluation: Performance is measured with RMSE and relative RMSE, averaged across categories as mRMSE and m-relRMSE.Relative RMSE scales deviations according to ground-truth count, reflecting a logarithmic perception of counting errors.
- Evaluation: The evaluation also reports metric variants restricted to non-zero ground-truth counts to assess accuracy beyond predicting object absence.Predictions are thresholded at zero and rounded to the nearest integer before metric computation.
- Compared methods: Baselines include always-0, mean, always-1, category-mean, and gt-class predictors, alongside detection and learned counting variants.The learned variants include detect and glance configurations using different feature sources and training choices.
5. Results
The models are evaluated on PASCAL VOC, COCO, count ranges, detection, and counting-based VQA. Sequential subitizing generally performs strongly, while ensemble and threshold choices vary by setting.
- seq-sub outperforms the other evaluated methods on PASCAL VOC 2007 counting metrics.
- seq-sub achieves the best COCO performance across mRMSE, m-relRMSE, and their non-zero variants.
- Count versus Count Error: For COCO counts above 5, glance and detect degrade, while seq-sub benefits from contextual reasoning and ens performs best at lower counts.
- Detection: 0.02 mRMSE is gained by tuning Fast R-CNN hyperparameters for counting relative to the untuned detector.
- Counting to Improve Detection: 20.17% mF is achieved with ground-truth counts for threshold selection, compared with 15.26% for fixed per-category thresholds.
- Counting to Improve Detection: 1.64% mF and 1.74% mF gains over the base detector are obtained using ens and seq-sub-ft predicted counts, respectively.
- Visual Question Answering: seq-sub significantly outperforms other approaches on the evaluated counting questions, and even glance surpasses a state-of-the-art VQA model.
6. Conclusion
The paper presents sequential subitizing for category-level counting in everyday scenes and reports improved counting, detection, and proof-of-concept VQA performance.
- The proposed models improve counting performance on PASCAL VOC 2007 and COCO datasets.
- An ensemble of the proposed approaches performs best among the evaluated counting approaches.
- Counting estimates improve object detection and support proof-of-concept answers to “how many?” questions in VQA tasks.
Appendix
The appendix documents supplementary analyses, implementation details, VQA procedures, occlusion studies, and qualitative prediction examples.
- The appendix reports ablation studies for glance, aso-sub, and seq-sub on the Count-val split.
- It analyzes count predictions by comparing object sizes with count performance and overcounting versus undercounting statistics.
- It provides details of the VQA experiment used in the main paper.
- It presents occlusion studies identifying regions of interest used while estimating counts.
- It includes qualitative examples of predictions generated by the counting models.
1. Ablation Studies
The ablations show that feature choice, grid size, and context depth affect counting differently across glance, aso-sub, and seq-sub. Detection-finetuned features favor local aso-sub counting, while classification features can favor global glance counting.
- Glance: Glance performs best with a two-hidden-layer noft model, whereas noft outperforms ft with one hidden layer.The authors relate this to classification features being better suited to global image statistics.
- aso-sub-3 × 3: Detection-finetuned ft features outperform classification noft features for aso-sub-3 × 3.The authors attribute this to ft features being better adapted to local image-region statistics.
- aso-sub-3 × 3: Increasing aso-sub from one to two hidden layers does not improve performance over a single hidden layer.This contrasts with the improvement observed for the glance model.
- aso-sub: The 3 × 3 aso-sub grid performs slightly better than 5 × 5 and substantially better than 7 × 7.The comparison uses the best-performing ft features and one hidden layer.
- seq-sub: For seq-sub, increasing the grid from 3 × 3 to 5 × 5 slightly improves PASCAL performance but reduces COCO performance.Across these comparisons, ft features generally outperform noft features.
- seq-sub: Reducing Bi-LSTM units from 2 to 1 leaves ft performance unchanged but lowers noft performance by 0.01 mRMSE on both PASCAL and COCO.The comparison uses seq-sub architectures with a 3 × 3 grid.
2. Count Analysis
Count analysis compares model errors across object sizes, error directions, and ensembles on PASCAL and COCO. Sequential, asynchronous, and ensemble approaches are broadly strong, while ensemble gains and undercounting patterns differ by dataset.
- Size versus Count Error: Seq-sub, aso-sub, and ens perform consistently well across a wide range of object-category sizes on PASCAL and COCO.As object size increases, all methods become more competitive; larger-category counts may be partial.
- Undercounting versus Overcounting: On PASCAL, moving from detect to ens increases the frequency of exact counts, with ens correct more often than undercounting.Overcount frequency remains approximately unchanged across this progression.
- Undercounting versus Overcounting: On COCO, exact counts increase from detect to ens but remain much less frequent than undercounting for every model.The analysis excludes examples whose ground-truth count is zero.
- Ensemble: An ensemble of seq-sub and aso-sub achieves mRMSE 0.427 on PASCAL and 0.351 on COCO.These ensembles outperform the corresponding seq-sub-plus-glance ensembles, which obtain 0.438 and 0.363 mRMSE.
- Ensemble: The detect, glance, aso-sub, and seq-sub ensemble has mRMSE 0.38 before ensembling and 0.36 after ensembling on COCO.Including detect hurts counting performance in this comparison.
3. Occlusion Studies
Occlusion studies use masked COCO images to compare where glance, aso-sub, and seq-sub obtain counting evidence. The patterns suggest that seq-sub combines global and local cues differently as counts vary.
- Occlusion-map setup: Occlusion maps compare glance, aso-sub, and seq-sub on COCO Count-test images using 4 × 4 masks.The selected images span counts from 10 to 1, and the models agree on the predicted count for each analyzed image.
- Occlusion patterns: For lower counts, glance and seq-sub attend to similar regions, while for higher counts, seq-sub and aso-sub become more similar.The models use 3 × 3 discretization for the aso-sub and seq-sub analyses.
- Interpretation: The authors interpret the high-count similarity between seq-sub and aso-sub as evidence that distributed global cues can support counting.They give the number of visible object parts, such as elephant tails, as an example of such cues.
- Interpretation: Seq-sub is presented as interpolating between glance and aso-sub according to the count.This interpretation connects its changing occlusion patterns with its use of contextual information across cells.
4. VQA Experiment
The VQA experiment isolates numerical “how many?” questions and maps their nouns to COCO categories before applying the counting models. It handles both numeric answers and written number words.
- Data preparation: The experiment selects “how many?” questions with numerical answers from VQA and COCO-QA.VQA provides integers, while COCO-QA also includes numbers written as text that are parsed into numerals.
- Category matching: Question nouns are singularized and matched to COCO categories or super-categories using word2vec cosine similarity.The matching procedure uses embeddings trained on Wikipedia.
- Category matching: When a noun matches a COCO super-category, the answer sums the counts of its constituent COCO sub-categories.For direct category matches, the model count for that COCO category is used as the answer.
5. Qualitative Results
Qualitative COCO examples show that the ensemble performs best, while sub-counting models estimate salient-object counts better than whole-image regression. Under substantial occlusion, detection performs poorly relative to seq-sub and the ensemble.
- The qualitative COCO examples identify ens as the best-performing model in the illustrated cases.
- For sufficiently salient objects, seq-sub and aso-sub estimate counts better than glance.The passage attributes this to estimating partial counts at cell levels rather than regressing the count for the entire image.
- With significant object occlusion, detect performs much worse than seq-sub or ens.