VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation

1. Introduction

• COCO’s dense annotations and VQA questions concern overlapping images, motivating links between segmentation masks and question-answer pairs.Source: §1 · pp. 1–2

1.1. Applications of the segmentation-QA links

• VQS links COCO segmentations to VQA question-answer pairs by asking annotators to choose masks that visually answer each question.Annotators select only the target entities that answer the question, simplifying the task and increasing agreement.Source: §1 · p. 2, §1.1 · p. 2

1.1.1 Supervised attention for VQA

• VQS provides explicit attention supervision for VQA, addressing latent attention learned indirectly from text answers and supporting compelling MLP performance.The paper describes machine-generated attention maps as poorly correlated with human attention maps.Source: §1.1.1 · p. 2, §1.1 · p. 2

1.1.2 Question-focused semantic segmentation (QFSS)

• QFSS produces pixel-wise segmentations as visual answers, combining VQA’s versatile questions with visual grounding’s segmentation output.The paper proposes mask aggregation and compares it with baselines and an oracle method assuming instance segmentations are available at test time.Source: §1.1.2 · pp. 2–3

2. Linking image segmentations to text QAs

• The collection links COCO instance masks with VQA question-answer pairs and keeps only images containing at least three instance segmentations.Source: §2 · p. 3

2.1. Annotation instructions

• Annotators saw the image, its masks, the question, and the answer; they selected target entities, added tight boxes for uncovered regions, and handled count, full-image, and ambiguity cases.For “How many” questions, the selected segments plus boxes had to match the answer; full-image and ambiguous cases were later removed.Source: §2.1 · pp. 3–4, §2.3 · p. 5

2.3. Data statistics

• Over 70% of questions use one segmentation; each question-image pair averages 6.7 candidate segmentations and 1.6 selected visual answers.Figure 3 shows the distribution of segmentation counts per question-image pair.Source: Figure 3 · p. 4, §2.3 · p. 5
• “What” questions are most common at 46%, while “is/are” and “does/do” total 32.1%; VQS elicits masks even for yes/no answers.Figure 4 visualizes the distribution of question types.Source: Figure 4 · p. 4, §2.3 · p. 5

2.2. Tasks addressed by the participants

• The task required object identification, segmentation of interactions, spatial-relation reasoning, fine-grained activity labeling, and commonsense-guided search.Examples include identifying a cup, representing rider-bike interactions, and selecting all entities involved in an activity.Source: §2.2 · p. 4

3.1.1 Augmenting MLP by supervised attention

• The attention model learns question-conditioned region weights by matching them to normalized masks linked to each question-image pair.The model appends these attention features to the original MLP input; image features include ResNet pool5 and attribute-detection scores.Source: §3.1.1 · pp. 5–6

3.1.2 Experimental results

• Learned attention features improve the plain MLP + ResNet by 1.5% absolute on VQA real multiple-choice.An ensemble of five MLP + ResNet + Atten. models and five MLP + Attri. + Atten. models ranked second on Test Standard as of submission.Source: §3.1.2 · p. 6, Table 1 · p. 6

3.1.3 What is good supervision for attention in VQA?

• VQS segmentations give slightly better VQA results than bounding boxes, which in turn outperform human attention maps (HAT).The authors attribute HAT’s weaker supervision to its small object parts and substantial background areas.Source: §3.1.3 · p. 6, Table 2 · p. 6

3.2. Question-focused semantic segmentation

• QFSS asks a model to produce a visual answer by segmenting image entities relevant to a question; the study compares aggregation, a baseline, and an oracle.The task is presented as a new problem enabled by VQS.Source: §3.2 · pp. 6–7

3.2.1 Mask aggregation for QFSS

• Mask aggregation combines 25 SharpMask proposals with question-dependent softmax weights and learns against user-selected masks using an L2 loss.The paper describes the current model as shallow and notes that it could be made deeper.Source: §3.2.1 · p. 7
• The QFSS oracle replaces proposals with MS COCO instance masks available at test time and classifies whether each mask belongs in the answer; DeconvNet is a competing baseline.The oracle is treated as an upper bound because its masks are more accurate and its classification task is considered easier than mask aggregation.Source: §3.2.1 · p. 7

3.2.2 Experiments on QFSS

• Mask aggregation outperforms DeconvNet on QFSS but remains significantly worse than the oracle upper-bound method.The upper bound assumes instance segmentations are available at test time.Source: §3.2.2 · p. 8, Table 3 · p. 8
• Bag-of-words and word-embedding question representations produce distinguishable QFSS results, and qualitative examples include two masks answering a “How many” question.The test split contains 5,873 images and 14,875 questions.Source: Figure 8 · p. 7, §3.2.2 · p. 8

• VQS links COCO instance segmentations to VQA question-answer pairs, connecting annotations from the same visual scenes.Source: §4 · p. 8
• The links transfer supervision between semantic segmentation and VQA, supporting supervised attention and the new QFSS task.For VQA real multiple-choice, the authors report state-of-the-art results by augmenting MLPs with attention features; for QFSS, they propose mask aggregation and compare it with baseline and oracle methods.Source: §4 · p. 8
• The authors describe the work as a first step and plan to explore more annotation types and richer models.Source: §4 · p. 8

• Participants are shown an image and question, then asked to tick the colors of segmentations that visually answer it.Source: A. Annotation Interface · p. 11
• The interface allows participants to draw bounding boxes in addition to selecting segmentations.Source: A. Annotation Interface · p. 11
• Annotators received training using slides prepared for the task.Source: A. Annotation Interface · p. 11

• For “what color is the ball?”, VQS supplies a ball mask without background, while HAT is described as rough.Source: B. VQS vs. VQA-HAT · p. 11
• VQS uses segmentation as the desired output, whereas HAT maps mainly support analysis and potential improvement of VQA models that produce text answers.Source: B. VQS vs. VQA-HAT · p. 11
• The comparison concerns participant-linked image segmentations and human attention maps in VQA-HAT.Source: B. VQS vs. VQA-HAT · p. 11

C. The influence of VQS segmentation mask resolution on the supervised attention in VQA

• Attention features weight image-region representations according to the question, and the number of regions determines attention resolution.The paper cautions that very small regions can make visual cues too subtle and produce trivial solutions.Source: C. The influence of VQS segmentation… · p. 11
• Higher segmentation-mask resolution leads to better VQA results on Test-Dev.The paper interprets this as indicating a need for accurate segmentation annotations for supervised attention.Source: C. The influence of VQS segmentation… · p. 11, Table 4 · p. 11

D. Some implementation details in the VQA and VQS experiments

• The VQA ensemble combines ten models: five use image attribute features and five use ResNet features.Validation data selects the ten models and their convex combination of decision values before evaluation on Test-Dev and Test-Standard.Source: D. Some implementation details in the… · p. 11
• The VQS experiments train the whole network with ADAM at learning rate 0.001 and batch size 16.Convergence takes about one week on one Titan X GPU after 15 epochs.Source: D. Some implementation details in the… · pp. 11–12
• Figure 10 shows the GUI used to collect links between image segmentations and questions and answers.These links are called VQS.Source: Figure 10 · p. 12
• About 0.02 improvement on the challenging VQS task is reported for LSTM language embeddings over bag-of-words and word2vec.The comparison is reported among additional DeconvNet approach results.Source: D. Some implementation details in the… · p. 12