Source-linked AI summary
What value do explicit high level concepts have in vision to language problems?
Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton van den Hengel
TL;DR
V2L models typically map CNN image features directly to text without explicit high-level semantics, leaving the value of such concepts uncertain. The paper inserts a learned attribute representation between CNN analysis and LSTM generation, then tests image- and knowledge-sourced attributes across captioning and VQA. It reports improved performance across tasks, including 57.62% VQA accuracy after WordNet-based attribute expansion, while noting a remaining gap from human performance and difficulty with “why” questions.
Problem
Direct CNN-RNN V2L models omit explicit high-level semantic representations, so the paper asks what value such concepts have for image captioning and visual question answering.
Method
The method predicts human-meaningful image attributes with a CNN and feeds the resulting likelihood vector into task-specific LSTM language models, with optional WordNet-based attribute expansion.
Results
Explicit attributes improve V2L performance across evaluated tasks; VQA reaches 57.62% accuracy with knowledge-sourced expansion, versus 55.84% without it and 54.06% for the baseline.
Takeaways & Limitations
Attribute representations improve V2L performance and provide a route to external commonsense knowledge for commonsense-related questions.
Takeaways & Limitations
A substantial gap remains between the proposed models and human performance, with especially low accuracy on “why” questions requiring commonsense knowledge and reasoning.
Abstract
from arXiv · showhide
Much of the recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. We propose here a method of incorporating high-level concepts into the very successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art performance in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. In doing so we provide an analysis of the value of high level semantic information in V2L problems.
1. Introduction
V2L systems commonly translate CNN image features directly into language, despite the representational gap between images and human language. This paper investigates whether explicitly modeling human-meaningful semantic attributes improves CNN-RNN performance across captioning and question answering.
- 1. Introduction: V2L translates between fundamentally different information forms, yet direct CNN-RNN mappings currently achieve state-of-the-art results in captioning and visual question answering.Images arise from complex physical processes, while language is designed for human communication.
- 1. Introduction: The paper asks whether explicit high-level concepts still provide value after direct image-feature-to-text models became successful.It specifically examines adding high-level information to the CNN-RNN framework.
- 1. Introduction: The proposed model predicts human-meaningful attributes with a CNN, then uses their likelihood vector as input to an RNN for captions or answers.Each attribute corresponds to a word mined from training image descriptions.
- 1. Introduction: 0.73 BLEU-1 is achieved in the Microsoft COCO Captioning Challenge, reported as state of the art on the leaderboard at writing time.The paper also reports state-of-the-art results on several VQA datasets.
- 1. Introduction: 55.84% accuracy is achieved on VQA test-standard versus a 54.06% baseline, rising to 57.62% with WordNet-based knowledge-sourced attribute expansion.On Toronto COCO-QA, the reported WUPS@0.9 scores are 71.15 versus 66.78 for the current state of the art.
2. Related Work
Earlier V2L systems used retrieval, templates, or intermediate visual concepts, while the dominant CNN-RNN trend learned image-to-language mappings directly. The paper revisits the omitted high-level representation and evaluates its value across V2L tasks.
- 2. Related Work: Earlier image-captioning methods retrieved or ranked captions, co-embedded images and text, or generated descriptions from structured visual detections.Template-based systems commonly used detected objects, modifiers, locations, or other attributes as inputs.
- 2. Related Work: Fang et al. detected 1000 visual words before generation, whereas this paper feeds visual attributes into an LSTM with a much larger word vocabulary.The paper emphasizes that its focus is measuring how much high-level concepts help, which Fang et al. did not discuss.
- 2. Related Work: The dominant CNN-RNN architecture learns image-to-sentence mappings directly and offers end-to-end training, wider caption variety, and stronger benchmark performance than earlier approaches.Its image-to-word intermediate mapping is bypassed.
- 2. Related Work: The impact of bypassing intermediate high-level representations remains unclear, including whether the RNN language model compensates for their absence.The paper frames this uncertainty as motivation for directly investigating high-level information in V2L.
- 2. Related Work: VQA requires answering previously unseen, free-form questions about image content, making it a distinct and challenging V2L task.Existing VQA approaches also commonly pass CNN features to an RNN language model.
3. An Attribute-based V2L Model
The model predicts human-meaningful visual attributes from images, then uses the resulting semantic representation as input to task-specific language models for captioning and question answering.
- 3.1. The Attribute Predictor: The image-analysis stage predicts caption-derived attributes as a multi-label classification problem using a supervised deep CNN.The attribute vocabulary can include nouns, verbs, and adjectives, and the CNN is trained with element-wise logistic loss.
- 3.2. Language Generator: For language generation, each image is represented by a fixed-length attribute-probability vector whose dimensions correspond to attributes.This vector replaces direct CNN image features as the image input to the LSTM-based generators.
- 3.2. Language Generator: The framework uses task-specific generators for captioning, single-word VQA, and sentence-based VQA, with question encoding for open-ended answers.Single-word VQA predicts an answer from the final question-conditioned LSTM output, while sentence-based VQA decodes an answer sequence.
- 3.1. The Attribute Predictor: A region-based attribute predictor processes proposal regions with a shared CNN and max-pools their outputs into the high-level image representation Vatt(I).The network uses an ImageNet-pre-trained VggNet, fine-tuned on the target image-attribute data.
- 3.2. Language Generator: The captioning LSTM generates word sequences from the attribute vector and preceding words, using start and end tokens during training.The model predicts each next word through the LSTM feed-forward process and learns word and attribute embeddings jointly with the LSTM parameters.
4. Image Captioning
The image-captioning evaluation compares attribute-based models with CNN-feature baselines and prior methods across standard metrics and COCO server tests. Attribute-based models achieve the strongest reported results while using compact visual and recurrent representations.
- Evaluation results: Attribute-based models generate the best COCO captioning results across all reported evaluation metrics, improving nearly 15% on B-1 and 30% on CIDEr over baselines on average.The comparison includes BLEU, METEOR, CIDEr, and PPL; ground-truth-attribute results are shown but excluded from rankings.
- Evaluation results: Att-CNN+LSTM outperforms Att-SVM+LSTM, indicating stronger performance from the region-based attributes prediction network than the SVM classifier.Both variants use predicted attribute representations, but Att-CNN+LSTM performs better in the reported comparison.
- Ablation comparison: Combining CNN features with the attributes vector yields B-1=0.71, below using the attributes vector alone in the same setting.This comparison supports using the predicted attribute representation as the LSTM input without directly combining it with CNN features.
- COCO evaluation server: 0.73 B-1 is achieved on the 40,775-image COCO test set, with the method surpassing human performance on 13 of 14 reported metrics.The system ranks first on B-1, B-2, and B-3 and places in the top five on the other evaluation metrics.
- Representation efficiency: A 256-d attributes score vector supports state-of-the-art performance with a relatively small recurrent layer compared with methods using higher-dimensional visual features.Lower visual and recurrent dimensions are associated with fewer recurrent-training parameters and lower computation cost in the authors’ comparison.
5. Visual Question Answering
The evaluation tests attribute-based CNN-RNN models on single-word and open-ended VQA, then examines whether WordNet-based attribute expansion helps commonsense questions. Explicit attributes outperform baselines, while knowledge-sourced attributes further improve commonsense-related performance.
- Evaluation: The experiments compare attribute-based models with CNN-feature baselines and other state-of-the-art methods across Toronto COCO-QA and VQA.Toronto COCO-QA uses single-word answers, whereas VQA uses sentence-based, open-ended questions.
- Evaluation: 61.38% accuracy is achieved on the Toronto COCO-QA dataset, where the attribute-based model outperforms the compared baselines and state-of-the-art methods.The dataset evaluates object, number, color, and location questions using accuracy and WUPS metrics.
- Evaluation: The Att-CNN+LSTM model outperforms the LSTM Q+I baseline in nearly all VQA answer types, especially questions categorized as ‘others’.The authors hypothesize that the separately trained attribute layer discards irrelevant image information before the LSTM processes it.
- Evaluation: Commonsense-related question types remain difficult, with low accuracy on ‘why’ questions and a substantial gap between the proposed models and human performance.The paper identifies commonsense knowledge and reasoning as normally required for these questions.
- Attribute Expansion using WordNet: WordNet expansion selects linked, sufficiently frequent candidate words and combines them into knowledge-sourced attributes that are fed to the LSTM.The candidates are linked to the image-attribute vocabulary and appear in at least five training question examples; the vector uses max-pooling.
- Attribute Expansion using WordNet: 9.88% is achieved on ‘why’ questions with knowledge-sourced attributes, while overall VQA accuracy reaches 57.62%, exceeding the image-sourced attribute model.The authors associate the improvement with reduced dependence on exact textual expression and added related information, such as shared relations between beds and hammocks.
6. Conclusion
The conclusion reports that explicit image-content representations improve V2L performance across captioning and question answering. Attribute representations also provide access to commonsense knowledge needed for commonsense reasoning questions.
- 6. Conclusion: The paper investigates an intermediate attribute-prediction layer within the CNN-LSTM framework for image captioning and two question-answering tasks.The three models address image captioning, single-word question answering, and sentence question answering.
- 6. Conclusion: Explicit image-content representations improve V2L performance in all evaluated cases, including image captioning and question answering.The image-captioning model also outperforms the state of the art on several captioning datasets at the time of writing.
- 6. Conclusion: 61.38% accuracy is reported on Toronto COCO-QA, while 57.62% accuracy achieves state-of-the-art performance on VQA.The conclusion describes the VQA result as a substantial improvement over the baseline.
- 6. Conclusion: Attribute representations enable access to high-level commonsense knowledge needed for commonsense reasoning-related questions.The conclusion presents this access as a further role of the attribute representation beyond its performance gains.