Source-linked AI summary
TallyQA: Answering Complex Counting Questions
Manoj Acharya, Kushal Kafle, Christopher Kanan
TL;DR
Existing VQA counting datasets emphasize simple questions, while complex counting requires relationships, attributes, and reasoning beyond object detection. The paper introduces TallyQA and the relational counting network, which uses relation networks with region proposals. RCN surpasses state-of-the-art methods on both TallyQA and HowMany-QA.
Problem
Existing counting datasets mostly contain simple questions, while complex questions requiring relationships, attributes, and reasoning need separate evaluation.
Method
The paper creates TallyQA and proposes a relational counting network that combines relation networks with region proposals and background information.
Results
RCN surpasses state-of-the-art methods for open-ended counting on both TallyQA and HowMany-QA.
Takeaways & Limitations
TallyQA enables evaluation of both simple and complex open-ended counting, while RCN effectively addresses both question types.
Takeaways & Limitations
The rule-based classifier may misclassify some simple questions as complex, so Test-Complex uses human-vetted questions.
Abstract
from arXiv · showhide
Most counting questions in visual question answering (VQA) datasets are simple and require no more than object detection. Here, we study algorithms for complex counting questions that involve relationships between objects, attribute identification, reasoning, and more. To do this, we created TallyQA, the world's largest dataset for open-ended counting. We propose a new algorithm for counting that uses relation networks with region proposals. Our method lets relation networks be efficiently used with high-resolution imagery. It yields state-of-the-art results compared to baseline and recent systems on both TallyQA and the HowMany-QA benchmark.
Introduction
Existing counting systems often struggle because counting questions may require reasoning beyond object detection. This paper addresses that gap with TallyQA and a relational counting network.
- Motivation: Open-ended counting systems must predict a whole-number answer from an image and counting question.Performance can fail because systems detect the wrong objects or cannot reason about them.
- Motivation: Simple questions require only object detection, whereas complex questions require deeper analysis of attributes or relationships.“How many dogs?” is simple; “How many dogs are eating?” is complex.
- Motivation: Existing VQA datasets mostly contain simple counting questions, so complex questions must be analyzed separately to test reasoning ability.Recent region-proposal methods improve counting, but dataset composition limits evaluation of complex questions.
- Contributions: TallyQA is the world’s largest open-ended counting dataset and is over 2.5 times bigger than VQA2.It includes both simple questions and complex questions demanding more than object detection.
- Contributions: The relational counting network infers relationships between objects and background regions using relation-network-inspired modifications for dynamic image regions.The design explicitly incorporates background information and supports high-resolution imagery.
- Results: RCN surpasses state-of-the-art methods for open-ended counting on both TallyQA and HowMany-QA.
Related Work
Prior counting datasets and VQA systems emphasize simple counting and often perform poorly on number questions. Region-based methods improve object selection, but complex counting also requires comparisons among proposals and background context.
- Datasets: Complex counting questions are scarce in existing datasets, while many counting questions can be solved using object detection alone.This scarcity motivates separate evaluation of models on complex questions.
- Counting Algorithms: Open-ended counting approaches either infer counts directly from CNN features or aggregate question-relevant object bounding boxes or region proposals.
- Direct Methods: 47% accuracy on number questions was reported for the best VQA2 method, compared with 69% overall accuracy.Most number questions in VQA2 are counting questions.
- Direct Methods: CNN mean pooling and attention may destroy information needed to determine how many objects of a type are present.
- Region-Based Methods: Region-based methods select question-relevant proposals, but complex counting may require comparing proposals for duplicates, positional reasoning, and background context.
The TallyQA Dataset
TallyQA combines imported counting questions with newly collected complex questions and separates evaluation into simple and complex test sets. Its construction uses targeted annotation, quality checks, and a rule-based classifier for dataset characterization.
- Collecting Complex Questions: Targeted AMT tasks yielded 19,500 complex questions for 17,545 unique images.The questions were designed to counter earlier datasets’ predominance of simple counting questions.
- Question Design: Annotators were instructed to create non-zero complex questions with counterexamples and separate zero-answer questions about absent attributes.Examples include contrasting men wearing versus not wearing glasses and dogs without spots.
- Imported Questions: TallyQA imports whole-number counting questions from VQA2, Visual Genome, and TDIUC, including questions with answers from 0–15.For VQA2, at least 5 of 10 annotators had to agree on the answer.
- Question Classification: The simple-complex classifier uses linguistic rules after removing image-context phrases and applying SpaCy part-of-speech tagging.A question is classified as simple when it has one noun, no adverbs, and no adjectives.
- Question Classification: The classifier has zero false positives for simple questions but can misclassify some genuinely simple questions as complex.Therefore, Test-Complex contains only human-vetted questions.
- Dataset Structure: TallyQA has Train, Test-Simple, and Test-Complex splits, with test images drawn exclusively from Visual Genome.No test-split images are used for training.
A New Framework for Complex Counting
RCN extends relation networks for complex counting by reasoning over foreground proposals, their spatial relationships, and background regions. Region proposals make this relational processing practical for high-resolution real-world imagery, while explicit background modeling supports questions involving contextual attributes and relations.
- RCN architecture: RCN combines two relation-network sub-networks to model foreground–foreground and foreground–background relationships before predicting the final count.The question guides processing of foreground proposals and background regions; the two relation outputs are concatenated and passed to a count-prediction network.
- Efficient relational processing: RCN uses region proposals instead of raw convolutional feature-map indices, reducing average comparisons from 38,416 to 1,466 per counting query.The proposal generator averages 31.12 foreground regions and 16 background patches per image, yielding a 26-fold reduction in comparisons.
- Efficient relational processing: 26-fold fewer comparisons let RCN apply relation-network reasoning to high-resolution real-world imagery rather than only lower-resolution synthetic scenes.The original relation network required d^4 pairwise relationships from a d × d feature map and was evaluated on CLEVR.
- Background reasoning: Explicit background modeling helps answer complex questions involving background attributes or relationships between foreground objects and background entities.RCN computes relationships between each foreground region and each of m background patches using relatively few comparisons.
- Spatial reasoning: Spatial relationship vectors encode proposal geometry, area, visual similarity, and intersection-over-union to support relational counting.These features help count each object once and enable learned non-maximal suppression for overlapping proposals.
Experiments
Experiments evaluate RCN against counting baselines and recent systems on HowMany-QA and TallyQA’s simple and complex test sets. RCN achieves the strongest reported accuracy across these evaluations, while ablations and comparisons support the value of relational and background reasoning.
- HowMany-QA: RCN achieves the highest accuracy on HowMany-QA, outperforming IRLC, the previous best-known result.
- TallyQA: RCN achieves the best accuracy on both TallyQA Test-Simple and Test-Complex, with a larger gap over Zhang et al. on complex questions.
- TallyQA: DETECT is the worst performer on Test-Complex, although it performs better on Test-Simple, supporting the distinction between detection-only and complex counting.
- Ablations: Removing RCN’s object-background component hurts performance on both simple and complex questions, showing the value of modeling background relationships.
- Ablations: RCN outperforms Zhang et al. by 6.38% absolute on positional reasoning questions in TallyQA Test-Complex.
- Ablations: Removing proposal location features decreases accuracy by 5.4% on HowMany-QA, 2.8% on Test-Simple, and 2.4% on Test-Complex.
- Architecture comparison: The original relation network reaches 3.46 RMSE and about 20% less accuracy than RCN on HowMany-QA.
Visualizing RCN
The paper visualizes RCN’s inference process by adapting Grad-CAM to score and display the importance of image region proposals. Proposal scores are derived from object-background interactions and visualized on the original image.
- Modified Grad-CAM assigns each proposal the maximum object-background score across pairwise interactions, then scales scores from 0 to 1 for visualization.
Discussion
RCN achieves state-of-the-art counting performance across datasets, with especially strong results on complex questions. The discussion highlights both the model’s visual focus and unresolved challenges in region proposals and data requirements.
- Results: RCN achieved state-of-the-art results across all datasets, outperforming Zhang et al. on VQA2 counting and IRLC on HowMany-QA.The compared systems used the same regions and visual features, reducing a frequent confound in performance comparisons.
- Model Analysis: Modified Grad-CAM visualizations show where RCN looks when making predictions, with color intensity proportional to object-proposal importance.
- Results: The performance gap between simple and complex counting questions was especially large for RCN and Zhang et al.The authors suggest that complex questions require more data to cover their broader range of attributes and relations.
- Limitations: Region-based methods outperform direct methods, but their proposals can include non-object regions and overlapping boxes.Each image may contain 10–100 proposals; simple non-maximal suppression did not improve RCN performance.
Conclusions
The paper separates simple from complex open-ended counting, introduces TallyQA and RCN, and evaluates RCN against baseline and state-of-the-art approaches. RCN combines region proposals with relationship networks and performs especially well on complex questions.
- Conclusions: The paper distinguishes simple counting questions, answerable using object detection alone, from complex questions requiring additional reasoning.
- Conclusions: TallyQA is introduced as the world’s largest dataset for open-ended counting and will be made publicly available.
- Conclusions: RCN combines region proposals with relationship networks, enabling efficient use of relation networks with high-resolution imagery.
- Conclusions: RCN effectively answers both simple and complex counting questions and works especially well on complex questions compared with baseline and state-of-the-art approaches.