Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions

1. Introduction

• VQA answers average 1.1 words, while complex questions may need sentences and multi-word answers can have many valid paraphrases, limiting classification-based systems.VCR provides multi-word answers averaging 7.55 words, but still selects from four options.Source: §1 · p. 1
• ViQAR generates a complete multi-word answer and accompanying rationale to justify it, rather than selecting an answer from a fixed set.The authors say the rationale can justify the answer to end-users and help generate a better answer.Source: §1 · pp. 1–2
• The proposed model uses four sequential recurrent networks: it generates an answer and rationale, then uses them to refine both.The generation module produces the initial pair; the refinement module generates a refined answer and rationale.Source: §1 · p. 2
• The model is trained on VCR and evaluated against ground-truth answers and rationales using generative language metrics and human Turing tests.The paper also reports ablation studies of model choices and components.Source: §1 · p. 2

2. Related Work

• Earlier VQA and VCR work, including approaches with explanations, remains classification-based; ViQAR instead targets complete generation of multi-word answers and rationales.Image captioning describes an image globally, whereas ViQAR answers a question about understanding a local image region.Source: §2 · p. 3

3. ViQAR: Task description

• The task requires generated answers to be correct, complete, and natural, with a well-formed rationale that justifies the answer.The decoder generates an answer and rationale for an image question.Source: §2 · p. 3, Figure 3 · p. 4
• ViQAR maps image, question, and optional caption features to generated answer and rationale sequences, unlike VQA and VCR finite-choice classification.The task separates answer generation from rationale generation, which takes the answer as an input.Source: §3 · p. 3
• The proposed methodology uses iterative refinement because answers and rationales can help each other; experiments also study settings without captions.The authors report that caption availability helps performance.Source: §3 · p. 3

4. Proposed methodology

• The shared input combines mean spatial image features with projected question and caption features; question and caption embeddings are obtained using BERT.The image features comprise a fixed number of regions, each represented by a feature vector.Source: §4 · p. 4
• The architecture pairs answer and rationale generation with refinement, motivated by their mutual influence: each can help produce a more refined version of the other.The generation module contains an answer generator and rationale generator; the refinement module contains an answer refiner and rationale refiner.Source: §4 · p. 4
• The answer generator feeds its representation to the rationale generator; the rationale representation then informs answer refinement, which in turn informs rationale refinement.A refined answer is generated after la + lr steps, a refined rationale after la further steps, and the full architecture runs in 2la + 2lr steps.Source: §4 · p. 4
• Each LSTM uses attention over visual features and a language module to produce conditional word probabilities for sequence generation.The attended image feature is a weighted sum of visual features; attention and embedding parameters are shared across the four LSTMs.Source: §4 · p. 5
• Training maximizes the joint likelihood of generated and refined answer-rationale sequences by minimizing the sum of four cross-entropy losses.The authors report that tested weighted averages of the cross-entropy terms did not perform better than a simple sum.Source: §4 · p. 5

5.1. Experimental setup

• VCR [44] supplies ground-truth answers and rationales and, unlike VQA-E [25] and VQA-X [32], includes longer open-ended answers suited to rich answer generation.CLEVR [17] lacks training rationales and uses synthetic shapes, unlike the real-world images used to train the model.Source: §5.1 · p. 6, Table 4 · p. 6
• The evaluation combines word-overlap metrics with embedding-based cosine-similarity metrics to assess structural closeness and semantic similarity to ground-truth sentences.The metrics include METEOR, CIDEr [37], ROUGE, SkipThought [21], Vector Extrema, Universal Sentence Encoder, InferSent [7], and BERTScore [45].Source: §5.1 · pp. 6–7

5.2. Performance evaluation of ViQAR

• The Q+I+C Generation-Refinement model outperformed both baselines on every evaluation metric, and performed most consistently across the proposed variants.The comparison uses a two-stage LSTM baseline and a VQA-based baseline; Table 1 reports quantitative evaluation on VCR.Source: Table 1 · p. 6, §5.2 · p. 7
• In a qualitative example, the generated rationale that person2 wears a school uniform supports the answer that person2 is a student.Another example has a semantically incorrect answer despite grammatically correct, complete answer-rationale output; improving semantic correctness is identified as future work.Source: §5.2 · p. 7, Figure 5 · p. 7
• Human evaluators judged generated VCR answers and rationales fairly grammatical, relevant to the question, and acceptably relevant to the answer.Thirty evaluators rated 50 image-question samples on five criteria; 16 samples used ground-truth answers and rationales, and the remainder were generated.Source: Table 3 · p. 6, §5.2 · p. 7

5.3. Further analysis of ViQAR

• One refinement module performed best: adding a second or removing it lowered performance, while the authors hypothesize extra parameters hinder learning.Table 2 compares variants with different numbers of refinement modules.Source: Table 2 · p. 6, §5.3 · pp. 7–8
• Without additional training, the VCR-trained model generalized reasonably well to Visual7W, producing answers and rationales relevant to image-question pairs.Visual7W does not provide ground-truth rationales; the authors also assessed generated outputs in a human Turing test.Source: §5.3 · p. 8, Figure 7 · p. 8
• Objective metrics can score qualitatively successful generations poorly when their wording differs from a single ground-truth sentence, leaving evaluation of this generative task challenging.The authors use objective metrics and a human Turing test and hope this work motivates better metrics.Source: §5.3 · p. 8

• The model's human Turing test results were comparable to ground truth.Source: §6 · p. 8
• ViQAR generates a multi-word answer and rationale from an image and question, moving beyond classical VQA's classification paradigm.Source: §6 · p. 8
• The end-to-end generation-refinement architecture reflects the dependence between answers and rationales.Source: §6 · p. 8
• The model also transferred to tasks without ground-truth rationales.Source: §6 · p. 8

• The supplement adds results for the VCR classification task.Source: Supplementary Material: Beyond VQA:… · p. 11
• It provides additional qualitative ViQAR results, including a study of the Refinement module and its addition to VQA-E.Source: Supplementary Material: Beyond VQA:… · p. 11
• It reports qualitative results on transferring the model to VQA and discusses objective evaluation metrics and the need to go beyond them.Source: Supplementary Material: Beyond VQA:… · p. 11

• For each question, the evaluation selects among four answer and four rationale choices by similarity to the generated outputs.Source: S1. VCR classification task · p. 11
• Overall accuracy counts only samples where both the answer and rationale are correctly classified.Source: S1. VCR classification task · p. 11
• The reported results indicate that ViQAR is difficult, motivating opening the problem to the community.Source: S1. VCR classification task · p. 11

• The model generates answer-rationale pairs for explanation, activity, temporal, mental, scene, and hypothetical questions.Source: S2. Additional qualitative results · p. 11
• The supplement also presents examples where the model fails to generate a good answer-rationale pair.Source: S2. Additional qualitative results · p. 11
• The generations can be grammatically correct and complete, but improving their semantic correctness remains future work.Source: S2. Additional qualitative results · p. 11

S3. Impact of refinement module

• The Generation-Refinement model generates meaningful answers with supporting explanations for complex image-question pairs where the model without refinement fails.The authors infer that answer-rationale interplay is more useful for multi-word answers, the focus of this work.Source: S3. Impact of refinement module · p. 11
• Adding refinement to VQA-E did not improve classification accuracy, which the authors attribute potentially to the dataset’s mostly one-word answers.They infer that answer-rationale interplay is more useful in multi-word answer settings.Source: S3. Impact of refinement module · p. 11

S4. Qualitative results on transfer to VQA task

• The model trained on VCR generates relevant answers and rationales for Visual7W without explicit training on Visual7W.The authors describe this as reasonable generalization to another VQA dataset.Source: S4. Qualitative results on transfer to… · p. 11
• The Visual7W transfer evaluation concerns standard VQA data without ground-truth rationales.Figure S4 presents additional qualitative results for the ViQAR task on Visual7W.Source: S4. Qualitative results on transfer to… · p. 11
• The authors characterize the Visual7W transfer results as adding a promising dimension to the work.Source: S4. Qualitative results on transfer to… · p. 11

S5. On objective evaluation metrics for generative tasks: A Discussion

• The evaluation combines several objective metrics with human Turing tests to assess answer-rationale generation.The authors use Turing tests to better estimate model performance because textual similarity may not be the only sign of quality.Source: S5. On objective evaluation metrics… · pp. 11, 13
• Textual similarity to a single ground truth may not reflect quality when multiple valid answers and rationales can express the same content differently.The authors note generated content can be relevant despite differing from the ground-truth pair and receiving low metric scores.Source: S5. On objective evaluation metrics… · p. 13
• Figure S5 pairs questions with ground-truth and generated outputs and their evaluation scores to illustrate cases of low scores alongside qualitatively good results.Source: S5. On objective evaluation metrics… · p. 13, Figure S5 · p. 16