Source-linked AI summary
Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
Arun Mallya, Svetlana Lazebnik
TL;DR
Generic ImageNet features are not well tailored to VQA's wide range of visual concepts, and VQA datasets are too small to train adequately for all of them. This paper trains activity-prediction networks on HICO and MPII and transfers their features to MadLibs questions about activity and person-object relationships, reporting state-of-the-art activity prediction and improved VQA performance.
Problem
VQA spans many visual concepts, but its datasets are too small to provide adequate training data for all of them.
Source: §1 · p. 1Method
The paper trains CNNs on HICO and MPII for activity prediction and transfers their specialized features to MadLibs activity and person-object questions.
Source: §1 · p. 2Results
The models achieve state-of-the-art performance on two activity datasets and improve performance on MadLibs activity and person-object questions.
Source: §1 · p. 2, §6 · p. 13Takeaways & Limitations
Specialized activity-dataset knowledge can improve VQA performance, though the demonstrated questions are fairly narrow in scope.
Source: §6 · p. 13HICO and MPII do not cover every question target: an absent target object can leave correct activity labels unhelpful for answering.
Source: §5 · p. 13
Abstract
from arXiv · showhide
This paper proposes deep convolutional network models that utilize local and global context to make human activity label predictions in still images, achieving state-of-the-art performance on two recent datasets with hundreds of labels each. We use multiple instance learning to handle the lack of supervision on the level of individual person instances, and weighted loss to handle unbalanced training data. Further, we show how specialized features trained on these datasets can be used to improve accuracy on the Visual Question Answering (VQA) task, in the form of multiple choice fill-in-the-blank questions (Visual Madlibs). Specifically, we tackle two types of questions on person activity and person-object relationship and show improvements over generic features trained on the ImageNet classification task.
1 Introduction
VQA systems often use generic ImageNet features, although VQA datasets are too small to adequately train for the wide range of visual concepts they cover. This paper trains activity-prediction networks on specialized datasets and transfers them to person-activity and person-object questions, reporting improved MadLibs accuracy.
Motivation
- VQA covers varied visual concepts, while its datasets are too small to provide adequate training data for all of them.Many systems instead rely on generic ImageNet-pretrained CNN features, with or without VQA fine-tuning.Source: §1 · p. 1
Approach
- The paper argues that question-specific expert networks trained on specialized datasets can improve question-answering performance and help interpret model decisions.The proposed networks predict human activity labels using HICO and MPII.Source: §1 · pp. 1–2
Contributions
- The paper transfers activity-prediction networks trained on HICO and MPII to improve MadLibs questions about person activity and person-object relationships.The networks fuse full-image and person-box features; HICO boxes are detected, while MPII boxes are provided.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
The related work situates activity prediction within a growing set of still-image datasets, including HICO and MPII. It motivates combining person-box evidence with global image context and describes weakly supervised learning for image-level labels.
Activity datasets
- HICO contains nearly 50,000 images across 600 human-object interaction categories, while MPII covers 393 activities that need not involve objects.HICO categories are verb-object pairs drawn from 80 MS COCO object categories.Source: §2 · p. 3
Weak supervision
- For HICO's image-level labels, the paper treats each person box as an instance and the image as a bag, using Multiple Instance Learning to address missing person-specific labels.A bag is positive when it contains at least one positive instance and negative when all its instances are negative.Source: §2 · p. 3
Context for activity recognition
- The paper reports that simpler fusion of whole-image and person-box features outperformed R∗CNN and Scene-RCNN on datasets with hundreds of labels.It says latent-box and multiresolution approaches performed poorly, possibly because of overfitting and, for latent boxes, infeasible sampling.Source: §2 · p. 3
3 Action Recognition Method
The method combines person-box and full-image features before predicting activity labels, uses a maximum over people to connect instance scores to image labels, and weights the loss for class imbalance.
3.1 Network Architecture
- The network pools 512 × 7 × 7 ROI features from the person box and full image, then fuses them before the VGG-16 fc6 layer.Fusion-1 stacks the features before 1 × 1 dimensionality reduction; Fusion-2 reduces each separately before stacking.Source: §3.1 · p. 4, Figure 2 · p. 4
- Unlike R∗CNN and Scene-RCNN, the model combines features before prediction rather than averaging separate predictions, and the paper reports better performance for this early fusion strategy.It is also faster than R∗CNN because it does not sample boxes during training or testing.Source: §3.1 · pp. 4–5
3.2 Multiple Instance Learning for Label Prediction
- For each image and action, the model uses the highest score across person boxes, so a positive image label requires at least one person to perform the action.The image-level prediction is compared with its ground-truth label to compute the loss and backpropagation gradients.Source: §3.2 · p. 5
3.3 Weighted Loss Function
- The weighted cross-entropy addresses HICO's severe class imbalance, where negative examples average 6,000:1 relative to positives and reach 38,116:1 in the worst case.The stated weights are 10 for positive examples and 1 for negative examples.Source: §3.3 · p. 5
4 Activity Prediction Experiments
Experiments on HICO and MPII compare person-and-context fusion networks with prior models and supervision variants. Fusion models perform better in the reported comparisons, while qualitative results show remaining errors in detection and crowded scenes.
Datasets and setup
- HICO has 600 image-level interaction categories and imperfect detected person boxes; MPII has 393 actions and annotated training instances, with all people in an image assumed to share its label.Both training sets are unbalanced; HICO has 51 categories with only one positive example.Source: §4 · p. 6
HICO Results
- On HICO, VGG-16 improves on AlexNet by 10 mAP, while the person-box-only network reaches 14.6 mAP and performs poorly.The authors note that interacting objects often lie outside the person's box; R∗CNN performs slightly worse than the full-image network, while Scene-RCNN is marginally better than R∗CNN.Source: Table 1 · p. 6, §4 · p. 7
- On HICO, fusion improves over the full-image network; weighted loss adds about 2.5 mAP for both fusion models, and Fusion-2 has a slight edge over Fusion-1.Fusion-2's dimensionality-reduction parameter count is half of Fusion-1's, a proposed explanation for its edge.Source: Table 1 · p. 6, §4 · p. 7
MPII Results
- On MPII, fusion networks outperform previous methods, and full supervision improves over MIL by around 0.4 mAP when image labels are assumed to apply to every person.Training with detected rather than ground-truth person boxes gives very similar performance when label assignment is unambiguous.Source: §4 · p. 8, Table 2 · p. 8
Qualitative Results
- Qualitative HICO predictions reasonably assign labels to people despite lacking explicit person-label supervision, but unusual object use and person-detection errors cause mistakes.In crowded scenes, especially sports settings, the weakly supervised MIL framework can fail to distinguish which person receives a label.Source: §4 · pp. 8–9
5 Visual Question Answering Results
The section evaluates activity-trained visual features on person-activity and person-object MadLibs questions, finding that specialized features outperform vanilla ImageNet VGG features. Qualitative examples show interpretable person-specific predictions, while failures include unfamiliar objects, incorrect predictions, and confusing answer choices.
Dataset and Tasks
- The evaluation covers person-activity and person-object relationship multiple-choice questions, with Easy and Hard distractors and a filtered Hard subset requiring human agreement.The Madlibs test sets contain 6,501 activity and 7,595 relationship questions; filtered Hard questions require annotators to agree with the answer at least 50% of the time.Source: §5 · pp. 9–10
Models and Baselines
- Image features include vanilla VGG-16, activity-network full-image features, and Fusion-2 features, paired with answer-choice representations in a joint nCCA embedding.The person-region inputs use the question’s target people; for relationship questions, the object box is ignored.Source: §5 · pp. 10–11
Question Answering Performance
- Against vanilla VGG, concatenated HICO+MPII class-label predictions improve Easy Activity by 6.8% and Easy Relationship by 7.5%.These are the best reported results for the two Easy tasks.Source: Table 3 · p. 11, §5 · p. 11
- For Hard Activity, gains are 6% unfiltered and 6.7% filtered; for Hard Relationship, gains are 4.7% unfiltered and 5.8% filtered.All gains compare concatenated HICO+MPII class-label predictions with the vanilla VGG baseline.Source: Table 3 · p. 11, §5 · p. 11
Qualitative Results
- Predicted activity labels and scores can clarify answer choices, with top labels often aligning closely to the correct choice and distinguishing the target person from background people.For questions targeting multiple people, class-label scores are max pooled across those people.Source: §5 · pp. 11–12, Figure 5 · p. 12
- The predicted “hold-book” score is higher for the boy on the right, showing person-dependent predictions despite training with weak supervision and MIL.Source: §5 · p. 13
- Common failures include questions about objects absent from HICO and MPII, wrong activity predictions, and confusing choices when multiple activities fit the image.The illustrated target object “picture” is absent from the activity datasets, so correct activity labels provide no useful answer information.Source: §5 · p. 13, Figure 6 · p. 13
6 Conclusion
The conclusion presents person-centric activity models using local and global context, trained with weak supervision through Multiple Instance Learning, and reports state-of-the-art performance on two datasets. It also reports that specialized activity-dataset knowledge improves VQA performance, while noting that the demonstrated question types are narrow.
6 Conclusion
- The models combine local and global context for person-centric activity prediction and use Multiple Instance Learning to train with weak supervision.The paper reports state-of-the-art performance on two datasets, outperforming more complex models such as R∗CNN.Source: §6 · p. 13
- Specialized activity-dataset knowledge improves VQA performance, although the demonstration covers fairly narrow question types.The authors envision combining more features, such as person attributes, detected objects, and scene information, according to the question and image.Source: §6 · p. 13