PaperlayerView exact revision on arXiv

Source-linked AI summary

VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions

Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, Jiebo Luo

arXiv:1803.07464v2cs.CV

TL;DR

VQA research has largely emphasized answer accuracy, while textual explanations can make model decisions more understandable and traceable. VQA-E pairs predicted answers with explanations, using a caption-derived dataset and a multi-task model. The authors report improved answer prediction and better explanation quality for visually specific questions, with limitations for subjective questions whose evidence is indirect.

  • Problem

    Most VQA work focuses on answer accuracy, while explanations are often absent despite their role in making answers more understandable and traceable.

    Source: §1 · pp. 1–2

  • Method

    VQA-E derives textual explanations for VQA v2 question-answer pairs from captions and trains a multi-task model to predict answers and generate explanations.

    Source: §1 · pp. 2–3

  • Results

    The authors report that explanation supervision improves answer prediction and that synthesized explanations are generally good in human evaluation, though weaker for subjective or common-sense questions.

    Source: §1 · p. 3, Evaluation results summary · p. 8

  • Takeaways & Limitations

    The dataset and model support textual justification of VQA answers, while the reported explanation quality is strongest for visually specific questions.

    Source: §6 · p. 14

    Explanations are inadequate for subjective questions with indirect visual evidence, for which the authors say extra knowledge bases are needed.

    Source: Subjective questions: Do you/Can… · p. 7, §6 · p. 14

Abstract

from arXiv · show

Most existing works in visual question answering (VQA) are dedicated to improving the accuracy of predicted answers, while disregarding the explanations. We argue that the explanation for an answer is of the same or even more importance compared with the answer itself, since it makes the question and answering process more understandable and traceable. To this end, we propose a new task of VQA-E (VQA with Explanation), where the computational models are required to generate an explanation with the predicted answer. We first construct a new dataset, and then frame the VQA-E problem in a multi-task learning architecture. Our VQA-E dataset is automatically derived from the VQA v2 dataset by intelligently exploiting the available captions. We have conducted a user study to validate the quality of explanations synthesized by our method. We quantitatively show that the additional supervision from explanations can not only produce insightful textual sentences to justify the answers, but also improve the performance of answer prediction. Our model outperforms the state-of-the-art methods by a clear margin on the VQA v2 dataset.

1 Introduction

VQA systems often provide answers without explanations, leaving decisions difficult to understand and trace; attention maps show where a model looks but not what it captures, and are inaccessible to visually impaired users. VQA-E introduces textual explanations, a caption-derived dataset, and a multi-task model that generates explanations alongside answers.

1 Introduction

  • Attention maps indicate where a model looks but do not fully reveal what it captures, may accompany wrong answers, and are inaccessible to visually impaired users.These limitations motivate exploring textual explanations as a complement to visual attention.Source: §1 · p. 1
  • Textual explanations can justify answers, elaborate question-and-answer context, or specify abstract concepts, supporting interpretation and communication.The passage notes that public datasets including VQA, COCO-QA, and Visual7W do not provide explanations for their annotated answers.Source: §1 · p. 2
  • VQA-E requires models to provide textual explanations and supplies a dataset automatically derived from VQA v2 image-question-answer triples using available captions.Each image in VQA v2 is coupled with five MSCOCO descriptions, which may include question-answer-related information.Source: §1 · p. 2
  • A user study found synthesized explanations good for most question-answer pairs, but somewhat inadequate for subjective or common-sense questions.The study evaluated a randomly selected subset.Source: §1 · p. 3
  • The VQA-E model jointly predicts answers and generates explanations, and explanation supervision is reported to improve answer accuracy and outperform state-of-the-art methods on VQA v2.The paper also reports better localization of important image regions.Source: §1 · p. 3

2 Related Work

Prior VQA work uses attention to locate question-relevant image regions, while textual explanations address limitations of visual-only justification. VQA-E provides a large, free-to-create textual explanation dataset in contrast to a smaller human-annotated multimodal dataset.

Attention in Visual Question Answering.

  • Visual attention uses the question to search for relevant image regions, but it is inaccessible to visually impaired users and limited in real-world applications.Prior work also found that attention-based models may not attend to the same regions as humans when answering questions.Source: Attention in Visual Question Answering · p. 3

Model with Explanations.

  • VQA-E builds a reference dataset to train and evaluate VQA explanation models because existing classification explanation metrics are inapplicable without predefined question-and-answer categories.The related work identifies bird-classification explanations as an example of the preceding approach.Source: Model with Explanations · p. 3
  • The pipeline converts each question-answer pair into a statement, aligns its constituency tree with the relevant caption tree, and merges a shared-node subtree into the caption.A grammar check and correction tool refines the combined sentence.Source: Figure 2 · p. 4
  • VQA-E focuses on textual explanations and is free to create, with 269,786 examples versus 41,817 in the human-annotated Multimodal Explanations dataset.The related work describes Multimodal Explanations as a high-quality multimodal explanation dataset for VQA.Source: Model with Explanations · p. 4

3 VQA-E Dataset

The section constructs VQA-E explanations by matching image captions to question-answer pairs, merging their information, and filtering low-similarity outputs. It analyzes coverage across question types and assesses explanation quality through a user study.

Approach.

  • The pipeline selects the most relevant caption, merges the question and answer into a declarative statement, fuses it with the caption, and grammar-corrects the result.The caption is selected using a mean-max similarity formula, which the authors say works better than their tested TF-IDF alternative.Source: Approach · pp. 4–5

Similarity distribution.

  • Explanations below a similarity threshold of 0.6 are removed to reduce noise, and examples improve gradually as similarity scores rise.The authors report a clear histogram trough at 0.6.Source: Similarity distribution · p. 5, Figure 3 · p. 5

3.2 Dataset Analysis

  • 269,786 of 658,111 VQA v2 question-answer pairs (41%) receive relevant explanations with high similarity scores.The section points to Table 1 for additional dataset statistics.Source: §3.2 · p. 6
  • For ‘what sport is’ and ‘what room is’ questions, the approach successfully generates explanations for 90% and 87% of question and answer pairs, respectively, versus 40% for general ‘what’ questions.Relevant explanations are generally more common for ‘is/are’ and ‘what’ questions than for ‘how’, ‘why’ and ‘do’ questions.Source: §3.2 · p. 6

Subjective questions: Do you/Can you/Do/Could?

  • The percentages of good explanations for ‘do you’, ‘can you’, ‘do’ and ‘could’ questions were 4%, 5%, 13% and 6%, respectively, far below the 41% average.These question types may involve subjective feeling, logical thinking or behavioral reasoning; image clues may be opaque and indirect, making good explanations hard to synthesize.Source: Subjective questions: Do you/Can… · p. 7

3.3 Dataset Assessment – User Study

  • The user study assesses explanation fluency, correctness, relevance to the question-answer pair, and complementarity to the answer.Evaluators assess explanations on a 1–5 scale; Table 2 reports averages for 2,000 questions.Source: §3.3 · p. 7, Table 2 · p. 8

Evaluation results summary.

  • Synthesized explanations score close to 5 for fluency and correctness and above 4 for relevance and complementarity, indicating good overall quality in the user study.The latter two measures distinguish a general image caption from an explanation tied to a visual question-answer pair.Source: Evaluation results summary · p. 8, Table 2 · p. 8

4 Multi-task VQA-E Model

The multi-task VQA-E model uses image and question representations with question-guided visual attention to predict an answer and generate an explanation simultaneously. It combines image features, a GRU question embedding, answer prediction, and explanation generation in one architecture.

4.1 Image Features

  • The study compares global, grid, and bottom-up image features: global features use ResNet-152, P=1, and no visual attention, whereas bottom-up features comprise P=36 salient regions proposed by Faster R-CNN.Source: §4.1 · p. 9

4.2 Question Embedding

  • The question is tokenized into word embeddings, encoded by a GRU, and represented by its final state.Source: §4.2 · p. 9

4.3 Visual Attention

  • Question-guided attention scores image patches, normalizes their weights, and sums the weighted features into an attended image representation.The model uses simple one-glimpse, one-way attention; question and attended-image representations are then projected and fused by element-wise multiplication into joint representation h.Source: §4.3 · pp. 9–10

4.4 Answer Prediction

  • Answer prediction regresses scores for 3,129 candidate answers using soft targets derived from the ten human answers per question.The soft targets represent annotator disagreement, and the final regression layer predicts each candidate answer's correctness.Source: §4.4 · p. 10

4.5 Explanation Generation

  • Explanation generation uses an LSTM language model conditioned on joint representation h, and the multi-task loss sums the VQA and VQE losses.The language model is trained on the ground-truth explanation.Source: §4.5 · pp. 10–11

5 Experiments and Results

The experiments compare explanation-only, answer-only, and joint answer/explanation variants, then assess explanation generation, answer prediction, and qualitative attention. Joint supervision performs best on explanation generation and improves answer accuracy; the test-standard comparison reports gains in overall VQA v2 accuracy.

Model setting.

  • The variants include explanation-only and answer-only models, joint answer-and-explanation training, and joint models trained with relevant or randomly selected captions.The relevant-caption variant uses the caption selected during explanation synthesis; the random variant uses another caption for the same image.Source: Model setting · p. 11

5.2 Evaluation of Explanation Generation

  • QI-AE with bottom-up features performs best across explanation metrics; using both image and question outperforms either alone, and bottom-up features outperform grid features.Image-only explanation generation outperforms question-only generation, reversing the language-bias pattern described for VQA answer prediction.Source: §5.2 · pp. 11–12

5.3 Evaluation of Answer Prediction

  • Across question types, QI-AE consistently outperforms QI-A, with larger explanation-supervision gains for attention-based models than for Global models.The authors argue explanation supervision helps the model exploit image content and attend to important regions.Source: §5.3 · p. 12
  • Random non-relevant captions make QI-AE(random)-Bottom-up less accurate than QI-AE-Bottom-up and even QI-A-Bottom-up.The authors suggest low-quality or irrelevant explanations might confuse the model.Source: §5.3 · pp. 12–13
  • On VQA v2 test-standard, the method beats state-of-the-art on Yes/No, Other, and overall accuracy, but trails BUTD on Number.The comparison is reported for the test-standard split.Source: Table 5 · p. 13, §5.3 · p. 13

5.4 Qualitative Analysis

  • Qualitative examples show QI-AE adding relevant detail and, unlike QI-A, attending to a person’s hand to answer “Feeding giraffe” correctly.The figure visualizes attention with a red box over the region receiving the greatest attention weight.Source: §5.4 · pp. 13–14, Figure 7 · p. 14

6 Conclusions and Future Work

The paper introduces VQA-E and a multi-task architecture that generates explanations alongside predicted answers. The authors report benefits for answer accuracy and image-region understanding, while noting that explanations remain inadequate for subjective questions with indirect evidence.

6 Conclusions and Future Work

  • Explanations in the dataset are high quality for visually specific questions but inadequate for subjective questions whose evidence is indirect.The authors say extra knowledge bases will be needed to find good explanations for subjective questions.Source: §6 · p. 14
  • The multi-task architecture generates reasons for predicted answers and is reported to improve answer accuracy and localization and understanding of important image regions.The authors describe explanation supervision as providing both textual justification and improved answer prediction.Source: §6 · p. 14
  • The authors identify reinforcement learning approaches used in image captioning as a direction for future model training.Source: §6 · p. 14

Related papers

Loading 1803.07464v2…