Source-linked AI summary
Learning to Count Objects in Natural Images for Visual Question Answering
Yan Zhang, Jonathon Hare, Adam Prügel-Bennett
TL;DR
Natural-image VQA models struggle with counting, partly because soft attention can erase count information and overlapping proposals can cause double-counting. The paper introduces a differentiable component that counts from object proposals and attention information. It achieves state-of-the-art number-question accuracy on VQA v2 without harming other categories, and improves a difficult balanced-pair counting metric by 6.6%.
Problem
Natural-image VQA models struggle with counting because soft attention can lose count information and overlapping proposals can cause double-counting.
Method
The paper proposes a differentiable neural component that performs robust counting from object proposals while handling overlapping proposals.
Results
The component achieves state-of-the-art accuracy on VQA v2 number questions without degrading other categories and improves a difficult balanced-pair metric by 6.6%.
Takeaways & Limitations
The component can be integrated with soft-attention VQA models and may support object-proposal counting without true-object supervision.
Takeaways & Limitations
The approach remains tied to soft-attention models and depends on proposal scoring and a notion of pairwise proposal dissimilarity.
Abstract
from arXiv · showhide
Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.
1 INTRODUCTION
Counting in natural-image VQA is difficult because models struggle beyond dataset biases, while soft attention and overlapping proposals obscure object counts. The paper introduces a differentiable counting component that improves counting without harming other VQA categories.
- VQA models struggle to answer counting questions in natural images outside dataset biases.
- Soft attention creates a fundamental counting problem, while missing object-location labels, object diversity, and preserving non-counting performance further complicate VQA counting.
- Object proposals provide bounding boxes and features, but overlapping proposals can cause double-counting and inaccurate counts in natural images.
- A differentiable neural component enables robust counting and outperforms prior models on VQA v2 number questions without degrading other categories.The paper reports that a simple baseline with the component even surpasses large state-of-the-art ensembles.
2 RELATED WORK
Prior work spans non-differentiable and differentiable duplicate-box suppression, object-proposal and reinforcement-learning counting, synthetic VQA, and density-map methods. The paper distinguishes its component by producing counting features that integrate with standard soft-attention VQA without true-box supervision.
- Greedy NMS removes duplicate boxes, but its piecewise-constant gradient makes it problematic as part of a trainable model.
- The proposed component outputs counting features rather than selecting boxes, integrates with soft-attention VQA, and requires no true bounding-box supervision.
- Object-proposal VQA methods introduced primitive counting information through singular-versus-plural detector classes, whereas this paper takes a different approach.
- A reinforcement-learning counting mechanism reported a small accuracy increase but had unclear integration with traditional VQA and results that were difficult to compare.
- Synthetic CLEVR counting methods use simpler objects, synthetic question structures, and approximately 250,000 counting questions versus approximately 50,000 in VQA v2.
- Traditional counting approaches learn density maps or divide images into chunks before combining local counts.
3 PROBLEMS WITH SOFT ATTENTION
Soft attention can erase count information because normalized weighted sums make images with different numbers of similar objects produce identical features. The paper therefore uses attention maps and proposal relationships to support counting while addressing duplicate proposals.
- Attention-derived feature vectors alone are insufficient for counting; the attention maps themselves must also be used.
- Soft attention assigns normalized weights to spatial or proposal features and combines them into a single weighted-sum vector.
- Figure 1 represents four proposals covering three objects, including one duplicate proposal, illustrating the target of one proposal per true object.
- With one cat versus two identical cats, softmax assigns weights 1 versus 0.5 each, making the weighted-sum feature vectors identical and erasing count information.
- Multiple glimpses or attention steps do not solve the issue because each attention weight is independent of the other features being attended.
- Removing sum-to-one normalization makes feature scale depend on detected-object count, which is unsuitable because neural networks are scale-sensitive.
4 COUNTING COMPONENT
The counting component converts attention-weighted object proposals into a graph, removes duplicate edges caused by overlapping proposals, and derives a count from the resulting edge structure. It uses differentiable operations and learned functions to handle idealized and intermediate attention and overlap cases while integrating with soft attention.
- Output: The component uses differentiable, monotonic piecewise-linear functions to interpolate between fully distinct and fully overlapping proposals and to produce counting features for soft-attention models.Under the stated extreme-case assumptions, the predicted count is always an integer equal to the correct count, regardless of duplicate-proposal quantity.
- Inputs: The component counts from the n largest attention weights and their corresponding bounding boxes rather than learning directly from image pixels.Attention weights are assumed to lie in [0, 1], and the count is bounded above by n.
- Graph representation: Attention weights are converted into a weighted directed graph whose vertices are object proposals and whose edge weights are products of proposal attention weights.When attention values are near 0 or 1, these products behave like logical AND operations.
- Deduplication: The graph is transformed so that underlying objects, rather than duplicate proposals, determine the complete subgraph used for counting.A complete digraph with self-loops satisfies |E| = |V|^2, allowing the vertex count to be recovered from the summed edge weights.
- Intra-object edges: Intra-object edges are removed by masking the attention matrix with a distance matrix based on bounding-box overlap, after which self-loops are restored.The distance matrix uses IoU and assigns edges except where connected boxes overlap; activation functions regulate partial overlaps and intermediate attention values.
- Inter-object edges: Inter-object duplicates are handled by scaling edges according to how many proposals share the same outgoing-neighbor set, averaging duplicate proposals under the final sum.Two proposals for one object receive a scaling factor of 0.5, and the similarity computation was more robust to inaccurate bounding boxes than using only pairwise overlap.
5 EXPERIMENTS
Experiments evaluate the counting component on a controlled toy task and VQA v2, showing improved robustness to overlapping proposals and stronger number-question performance without harming other categories.
- Toy task: The toy task predicts counts from bounding boxes and attention weights while varying overlap through side length l and signal quality through noise q.Counts are uniformly drawn from 0 to 10, with ten boxes placed in a unit square; q=0 has no noise and q=1 has no signal.
- Toy-task results: The counting component outperforms summing attention weights in most toy-task settings, especially with low noise and high overlap.When both overlap and noise are high, performance is closely matched by the baseline because little information remains to extract.
- Activation analysis: Increasing overlap reduces the learned f1 step height and changes f2 to treat smaller distances as sufficient evidence that proposals are distinct.At the highest side lengths, f2 returns toward its default linear initialization because overlap provides little remaining signal.
- Activation analysis: Increasing noise moves f1’s threshold away from 1 and makes f2 more sigmoidal, reflecting greater uncertainty in proposal weights and distinctions.Without noise, f1 behaves approximately as a step near 1 because true proposals have weight 1.
- VQA evaluation: On VQA v2, adding the counting component significantly improves number-question accuracy without compromising other categories and surpasses an 8-model ensemble on number questions.The comparison uses an improved VQA baseline and results reported on the official leaderboard.
- VQA evaluation: On validation counting questions, the module’s benefit exceeds its benefit across number questions generally, while replacing it with NMS does not improve the baseline.The NMS comparison suggests differentiability through the counting module is useful for learning to count.
- VQA evaluation: The counting module’s absolute improvement remains on the more difficult balanced-pair metric, providing evidence of counting beyond dataset-bias fitting.Balanced-pair accuracy requires distinguishing subtle differences between paired images.
- VQA evaluation: Activation functions in the trained VQA model share characteristics with high-noise toy-task settings, suggesting current attention and proposal quality remain inaccurate.This observation is presented as an explanation for the relatively small apparent increase in counting performance.
6 CONCLUSION
The counting component addresses VQA counting difficulty through differentiable bounding-box deduplication and can extend object-proposal counting beyond VQA.
- The component alleviates counting difficulties through differentiable bounding-box deduplication.
- It can be used with future VQA models that retain soft attention and with object-proposal counting tasks lacking ground-truth object labels.Such uses require per-proposal scoring and a notion of pairwise dissimilarity.
- The interpretable activation-function weights encode inductive biases for counting arbitrary objects with relatively little supervisory information.
- VQA v2 requires a versatile skill set that current models do not have.
A PIECEWISE LINEAR ACTIVATION FUNCTION
The paper represents activation functions as normalized, monotonic piecewise-linear curves whose learnable weights preserve endpoint behavior and support efficient computation.
- The activation-function domain [0, 1] is divided into d equal intervals connected by line segments.
- Each function fk uses d weights, with each weight defining the gradient on one interval.
- d is fixed to 16, while preliminary experiments found no significant difference for d = 8 or d = 32.
- Absolute-valued nonnegative weights enforce monotonicity, and normalization ensures f(1) = 1.
- The max term selects neighboring interval boundary values and linearly interpolates between them; unit initialization makes the functions initially linear.
- Caching normalized cumulative sums gives linear time complexity in d and efficient GPU implementation.
- Deep Lattice Network extensions could combine the functions more sophisticatedly while preserving monotonicity and the [0, 1] range.
B BASELINE ARCHITECTURE
The baseline adapts a prior VQA architecture to VQA v2 with object-proposal features and several architectural and training changes, including regularization and the counting-component integration.
- The model adapts Kazemi and Elqursh’s baseline architecture to VQA v2 with tweaks that slightly improve validation accuracy.
- Object proposal features are the most significant architectural change, while other tweaks were optimized for baseline validation accuracy rather than counting.
- Vision and question features are fused with an additional squared-difference term beyond concatenation, linear projection, and ReLU.
- The architecture replaces the question LSTM with a same-sized GRU, uses dynamic unrolling, applies batch normalization, and trains with learning rate 0.0015 and batch size 256.The model is trained for 100 epochs.
- Dropout regularization helps explain why the single-model baseline outperforms most single-model state-of-the-art results.Ensembling the regularized baseline provided a much smaller preliminary benefit than ensembling unregularized networks.
- Figure 6 marks counting-component modifications in red and distinguishes trainable blue blocks from non-trainable gray blocks.It also indicates that one of the two attention glimpses is shared with the counting component.
C FULL PLOTS OF ACTIVATION FUNCTIONS
The toy-dataset plots show how the learned activation-function shapes change as the parameters l and q vary, and compare those shapes with a VQA v2-trained model.
- Figure 7 plots activation-function shapes as l varies with q = 0.5 on the toy dataset.
- Figure 8 plots activation-function shapes as q varies with l = 0.5 on the toy dataset.
- Figure 9 compares a VQA v2-trained model’s activation functions with toy-dataset shapes for q around 0.4, 0.7, and 1.0 at fixed l = 0.2.
D EXAMPLE TOY DATASET DATA
The toy dataset varies bounding-box side lengths and noise while visualizing true, irrelevant, and weighted input objects.
- Toy data varies bounding-box side lengths l and noise q for randomly placed true and irrelevant objects.
- The ground-truth column distinguishes true objects in blue from irrelevant objects in red.
- Input samples encode bounding-box weights through color intensity, with dark blues near 1, dark reds near 0, and lighter colors near 0.5.
- Each bounding-box weight is defined as a_i = (1 − q) score + qz.
E QUALITATIVE EXAMPLES OF INTERMEDIATE ACTIVATIONS
The qualitative examples overlay bounding boxes and visualize attention, distances, and the resulting count matrix to inspect intermediate activations.
- Validation images are shown with overlaid bounding boxes alongside attention matrix A, distance matrix D, and count matrix C.
- White matrix entries indicate values near 1, whereas black entries indicate values near 0.
- The count c is computed as the usual square root of the sum over the elements of C.
- In the third example, attention activations span more rows and columns than there are actual objects, indicating overlapping bounding boxes.