Source-linked AI summary
Point and Ask: Incorporating Pointing into Visual Question Answering
Arjun Mani, Nobline Yoo, Will Hinthorn, Olga Russakovsky
TL;DR
VQA benchmarks often use complex verbal descriptions, while real-world questions may rely on a person pointing to the target. This paper introduces three point-input question settings, datasets, and model extensions; it reports effective modified models in the general setting and presents benchmarks for all three types.
Problem
Real-world VQA may involve a spatial point of reference, but existing grounded question settings do not necessarily require that reference for accurate answers.
Source: §1 · p. 1Method
The paper defines three point-input question settings, builds a benchmark dataset for each, and modifies VQA models to incorporate the point.
Source: §1 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Visual Question Answering (VQA) has become one of the key benchmarks of visual recognition progress. Multiple VQA extensions have been explored to better simulate real-world settings: different question formulations, changing training and test distributions, conversational consistency in dialogues, and explanation-based answering. In this work, we further expand this space by considering visual questions that include a spatial point of reference. Pointing is a nearly universal gesture among humans, and real-world VQA is likely to involve a gesture towards the target region. Concretely, we (1) introduce and motivate point-input questions as an extension of VQA, (2) define three novel classes of questions within this space, and (3) for each class, introduce both a benchmark dataset and a series of baseline models to handle its unique challenges. There are two key distinctions from prior work. First, we explicitly design the benchmarks to require the point input, i.e., we ensure that the visual question cannot be answered accurately without the spatial reference. Second, we explicitly explore the more realistic point spatial input rather than the standard but unnatural bounding box input. Through our exploration we uncover and address several visual recognition challenges, including the ability to infer human intent, reason both locally and globally about the image, and effectively combine visual, language and spatial inputs. Code is available at: https://github.com/princetonvisualai/pointingqa .
1. Introduction
The introduction proposes extending VQA with questions that require a spatial point of reference. It presents three point-based question settings, benchmark datasets, and model extensions designed for their distinct challenges.
1. Introduction
- The proposed benchmarks require a point input to answer accurately and use a more realistic point rather than the bounding-box input used in prior work.Source: §1 · pp. 1–2
- The tasks take an image and a single pixel as spatial grounding, and ask for a multiple-choice answer.Source: §1 · p. 2
- PointQA-Local targets questions answerable from the point's local region, while PointQA-LookTwice requires global image understanding; their datasets contain 57,628 and 57,405 questions, respectively.Both datasets are constructed from Visual Genome annotations.Source: §1 · p. 2
- PointQA-General adapts human-written Visual7W questions into 319,300 questions over 25,420 images, and the authors modify Pythia, MCAN, and LXMERT to use point input.Source: §1 · p. 2
- The paper frames its contribution as a new VQA dimension, combining point-input questions, benchmark datasets, and model extensions for the resulting challenges.Source: §1 · p. 2
2. Related Work
Prior VQA work uses visual grounding in datasets and models, but the related-work section distinguishes that from accepting a spatial point as question input. It also notes that pointing has rarely been studied as a communicative gesture in vision.
2. Related Work
- Prior VQA datasets use object boxes or grounded regions, but in a sample of Visual Genome questions only 17% required the associated region to answer correctly.The authors state that their benchmark explicitly requires spatial grounding.Source: §2 · p. 2
- Although VQA models commonly use image attention, prior work had not explored accepting spatial grounding as model input.Source: §2 · p. 2
- Pointing had been studied in robotics and human-computer interaction mainly in constrained settings, and in vision primarily as cost-effective supervision.The authors cite human pointing as a communicative gesture motivating deeper computer-vision study.Source: §2 · p. 2
3. PointQA-Local: reasoning about a region
PointQA-Local tests whether a point can disambiguate a region for questions about object attributes. Its benchmark and model focus on local reasoning, with evaluation showing benefits from using the point and retaining attention over multiple nearby regions.
3.1. PointQA-Local dataset
- For each same-class object pair with different attributes in the same category, the dataset creates two questions with points at the objects and their respective attributes as answers.The benchmark is designed so the image contains multiple possible answers and the point disambiguates the relevant region.Source: §3 · p. 2, §3.1 · p. 3
- The dataset contains 57,628 questions across 18,380 images, with 20 unique answers; 97.2% of questions ask about color.Questions cover color, shape, and action; size is excluded because it is relative rather than an absolute object property.Source: §3.1 · p. 3
3.2. PointQA-Local models
- The model incorporates the point by removing region proposals that do not contain it, then learns attention over the remaining proposals.The rest of the Pythia pipeline is unchanged, with zero-padding when needed to reach 100 regions.Source: §3.2 · pp. 3–4
3.3. PointQA-Local evaluation
- The proposed strategy improves test-dev accuracy by at least 5.8% over simpler point strategies, with a 12.6% gap on action questions.The alternatives retain only the smallest or highest-scoring region proposal containing the point.Source: §3.3 · p. 4, Table 1 · p. 4
- Changing the question from color to action increases the median max-attention region size from 2,997 to 5,451 pixels.On average, 27 point-containing proposals are input and maximum attention is 0.548; the model’s attention varies with the question.Source: §3.3 · p. 4, Figure 3 · p. 4
- 66.5% accuracy on spatially disambiguated questions exceeded 26.5% for verbal disambiguation; the verbal-only model often selected the wrong object instance.The point-input model was evaluated on 1,855 spatial questions, while the Pythia verbal-only baseline was evaluated on 1,855 verbal questions.Source: Figure 4 · p. 4, §3.3 · p. 5
4. PointQA-LookTwice: reasoning about a local region in the broader image context
PointQA-LookTwice tests questions that require identifying a local referent and reasoning across the image. The dataset and model combine point-guided local evidence with global attention for counting.
4.1. PointQA-LookTwice dataset
- Test answers are balanced across “1” (35.4%), “2” (33.7%), and “> 2” (30.9%); object names alone support at most 42.1% accuracy.The benchmark enforces evaluation examples with different objects and different answers in the same image.Source: §4.1 · p. 5
4.2. PointQA-LookTwice model
- The model first attends to point-containing regions, then uses that local representation with question and whole-image features to compute global attention.The local representation helps select relevant regions across the image before the model combines question, local, and global representations for prediction.Source: §4.2 · p. 6
4.3. PointQA-LookTwice evaluation
- 56.5% test accuracy on “How many of these are there?” exceeded 46.1% for the image-only model and 35.4% for the modal-answer baseline.Bounding-box input reached 60.2%; adding a supercategory raised point-input accuracy to 59.1%, while naming the object class yielded 62.8%.Source: §4.3 · p. 6, Table 2 · p. 6
- For the supercategory-question model, average maximum attention was 0.42 around the point and 0.07 for global image attention.The point-local attention is relatively peaked, while global attention is diffuse.Source: §4.3 · p. 6, Figure 6 · p. 6
- On supercategory questions, global attention reached 59.1% accuracy versus 55.9% for local-only attention.The paper attributes the difference primarily to under-counting when the local-only model does not attend to other object instances; the gap is larger for answers ≥2.Source: §4.3 · p. 6
5. PointQA-General: generalized reasoning from a point input
PointQA-General extends point-based question answering to unconstrained questions, requiring models to combine local point information with broader image context.
5.2. PointQA-General models
- MCAN and LXMERT are adapted to include a local point stream alongside question and global-image streams, with cross-attention modified to incorporate point information.MCAN’s image attention is text-influenced, whereas LXMERT uses bidirectional cross-modal attention; the three-stream design treats the point separately from the full-image stream.Source: §5.2 · pp. 7–8
5.3. PointQA-General evaluation
- Three-stream MCAN achieves the highest PointQA-General accuracy, and three-stream models outperform the Q-only, Image+Q, and Point+Q ablations.The reported comparisons attribute the benefit to treating the point as a separate stream and modifying cross-attention to support contextual interactions.Source: Table 3 · p. 8, §5.3 · p. 8
- For MCAN and Pythia, two-stream models use contextual information less effectively than three-stream models, while LXMERT achieves higher accuracy with two streams.The authors suggest LXMERT’s language-only pooling strategy may contribute to the latter result.Source: §5.3 · p. 8
6. Comparison Across Three Settings
Across the three settings, the comparisons highlight the need to ground question text in the image and to reason beyond the pointed location.
Comparison Across Three Settings
- The paper identifies visual grounding and joint reasoning around the point and across the image as features that make PointQA challenging for existing VQA systems..Source: §6 · p. 9
Need for Visual Grounding
- Removing the object word from questions lowers Pythia accuracy in all three settings: 83.1% to 75.1% on PointQA-General, 62.8% to 56.5% on PointQA-LookTwice, and 75.0% to 73.5% on PointQA-Local.In a 100-question PointQA-General sample, 55 questions referred to objects beyond the pointed one.Source: §6 · p. 9
Reasoning Beyond the Point Location
- On standardized Pythia three-stream comparisons, PointQA-Local accuracy is 90.7% versus 68.1% for PointQA-LookTwice, while contextual-comparison questions in PointQA-General reach 82.2% versus 83.1% overall.The passage presents these comparisons as evidence that reasoning jointly about the point and surrounding image context is challenging.Source: §6 · p. 9
7. Conclusion
The conclusion summarizes the paper’s point-input question benchmarks and models, and expresses hope that the work will prompt further research.
7. Conclusion
- The paper introduces three novel types of visual questions that require a spatial point input for disambiguation.Source: §7 · p. 9
- For each question type, the authors create a benchmark dataset and model design.Source: §7 · p. 9
- The authors perform extensive analysis and hope the work inspires further research in this question space.Source: §7 · p. 9
A. Human Evaluations
The human evaluation samples questions from all three datasets and collects multiple volunteer answers per dataset.
A. Human Evaluations
- The study randomly samples 100 test questions from each of PointQA-Local, PointQA-LookTwice, and PointQA-General.Source: A. Human Evaluations · p. 11
- Three volunteers answer each dataset’s 100 sampled questions, for nine participants total, using notebooks showing the image, red point, and question.Source: A. Human Evaluations · p. 11
- The authors record overall human accuracy from 300 responses per dataset and note potential reasons for incorrect answers.Source: A. Human Evaluations · p. 11
A.1. PointQA-General
On the 100-question PointQA-General subset, human accuracy exceeded the three-stream Pythia model’s accuracy. Disagreements included reasonable alternative answers and occasional point-placement or attention errors.
A.1. PointQA-General
- Humans scored 90.7% and the three-stream Pythia model scored 82.0% on the 100-question subset.The human-machine gap indicates the benchmark remains challenging, with scope for further model development.Source: §A.1 · p. 11
- The most common listed disagreements were “Other reasonable answer” (4.0%) and “Point not on object” (2.3%).The authors interpret the small point-placement share as supporting point placement at the bounding-box center overall.Source: §A.1 · p. 11
- “Wrong attention” accounted for 1.3% of disagreements; “question doesn’t make sense” accounted for 1.0%, no obvious reason for 0.3%, and obstruction for 0.3%.Source: §A.1 · p. 11
A.2. PointQA-Local
On PointQA-Local’s 100-question subset, the model scored higher than human annotators. Human disagreements included reasonable alternatives, ambiguous points, and obstruction, with some reflecting incomplete source annotations.
A.2. PointQA-Local
- The PointQA-Local model scored 82.0%, compared with human accuracy of 75.7% on the 100-question subset.The authors attribute the difference partly to annotators choosing reasonable but less-correct answers from 20 choices when responding quickly.Source: §A.2 · p. 11
- The most common listed human-error categories were “other reasonable answer” (12.3%), “ambiguous point” (4.3%), and “obstruction” (3.3%).Wrong attention accounted for 2.3%; answer misannotation and no obvious answer each accounted for 1.0%.Source: §A.2 · p. 11
- The prevalence of reasonable alternative answers indicates Visual Genome annotations can be accurate but incomplete.For example, a coat may be most accurately labeled gray even though brown could also reasonably be included.Source: §A.2 · p. 11
A.3. PointQA-LookTwice
On the PointQA-LookTwice 100-question subset, human accuracy was slightly higher than the global model’s. Reported disagreements included answer misannotation and variable local-to-global reasoning, while accidental training-set overlap may have inflated model accuracy.
A.3. PointQA-LookTwice
- Human accuracy was 79.3%, while the global model achieved 77.0% on the 100-question subset.The evaluation offered annotators choices from 1 through 7, then narrowed evaluated responses to 1, 2, and >2.Source: §A.3 · pp. 11–12
- The most common listed failure modes were answer misannotation (7.7%) and variable local-to-global reasoning (6.7%).Other listed modes were other instances hard to see (3.3%), wrong attention (1.3%), ambiguous point (1.0%), and no obvious reason (0.7%).Source: §A.3 · pp. 11–12
- Questions were unintentionally drawn from the PointQA-LookTwice training set, and the 77.0% model accuracy could still be inflated despite excluding them from model training.The authors also note that dataset-misannotation disagreements would be substantially lower without those questions.Source: §A.3 · p. 12