PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Ning Xie, Farley Lai, Derek Doran, Asim Kadav

arXiv:1901.06706v1cs.CV

TL;DR

Existing visual reasoning benchmarks have bias or synthetic, repetitive structures, motivating a real-world task for fine-grained image–text inference. The paper introduces Visual Entailment, constructs SNLI-VE, and develops EVE; EVE reaches up to 71.56% validation accuracy and 71.16% testing accuracy while supporting attention-based visualizations.

  • Problem

    Existing visual reasoning datasets exhibit distributional biases or, as with CLEVR, synthetic and repetitive visual and sentence structures that limit real-world coverage.

  • Method

    The paper defines Visual Entailment, builds SNLI-VE from SNLI and Flickr30k, and develops EVE with image–text attention and interpretable visualizations.

  • Results

    EVE-Image achieves 71.56% validation accuracy and 71.16% testing accuracy on SNLI-VE, and EVE is evaluated against VQA and image-captioning baselines.

  • Takeaways & Limitations

    Visual Entailment provides a real-world benchmark for fine-grained reasoning over images and text, with EVE offering attention visualizations as interpretability evidence.

  • Takeaways & Limitations

    SNLI-VE inevitably retains hypothesis-conditioned bias inherited from SNLI, and external knowledge may be needed for relationships difficult to learn from the dataset.

Abstract

from arXiv · show

Existing visual reasoning datasets such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning but the dataset is synthetic and consists of similar objects and sentence structures across the dataset. In this paper, we introduce a new inference task, Visual Entailment (VE) - consisting of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence as in traditional Textual Entailment tasks. The goal of a trained VE model is to predict whether the image semantically entails the text. To realize this task, we build a dataset SNLI-VE based on the Stanford Natural Language Inference corpus and Flickr30k dataset. We evaluate various existing VQA baselines and build a model called Explainable Visual Entailment (EVE) system to address the VE task. EVE achieves up to 71% accuracy and outperforms several other state-of-the-art VQA based models. Finally, we demonstrate the explainability of EVE through cross-modal attention visualizations. The SNLI-VE dataset is publicly available at https://github.com/ necla-ml/SNLI-VE.

1. Introduction

The paper introduces Visual Entailment, a real-world fine-grained reasoning task that determines whether an image entails a text hypothesis. It builds SNLI-VE and develops EVE, an attention-based model evaluated against VQA and image-captioning baselines.

  • The task addresses limitations in existing datasets, including question-conditioned bias in VQA and synthetic, repetitive visual and sentence structures in CLEVR.
  • Unlike Textual Entailment, Visual Entailment uses an image premise and assigns entailment, neutral, or contradiction labels to image–text pairs.
  • SNLI-VE combines Stanford Natural Language Inference with Flickr30k to create real-world image–sentence pairs for Visual Entailment.
  • EVE models image–text interactions with attention and provides interpretable attention visualizations.
  • Visual Entailment requires systematic cross-modal understanding between real-world images and natural-language hypotheses.

2. Related Work

Related work spans natural language inference, visual question answering, image captioning, visual relationship detection, and interpretability. EVE extends textual entailment into the visual domain and uses self-attention to model scene-element and word relationships.

  • Natural Language Inference research includes textual entailment methods that encode premises and hypotheses separately or align their words with attention.
  • Visual Question Answering answers natural-language questions from visual information, while VQA-v2.0 and CLEVR target bias and reasoning limitations.
  • Image captioning generates sentences describing image content, whereas EVE discerns fine-grained image information conditioned on a hypothesis.
  • Visual relationship detection models relationships among image constituents, but modeling all object pairs can be computationally expensive in scenes with many objects.
  • EVE uses self-attention to learn relationships between scene elements and words instead of bi-gram or tri-gram modeling.
  • The paper demonstrates interpretability through attention visualizations.

3. Visual Entailment Task

The paper defines Visual Entailment as classifying whether a natural-language hypothesis is entailed, neutral, or contradicted by a real-world image, and constructs SNLI-VE from SNLI and Flickr30k. The dataset targets fine-grained reasoning while addressing partition overlap and measuring known hypothesis-conditioned bias.

  • 3.1. Formal Definition: Visual Entailment pairs a real-world image premise with a natural-language hypothesis and assigns entailment, neutral, or contradiction labels.The task replaces textual premises in traditional entailment with image premises and requires determining whether the hypothesis follows from the image.
  • 3.2.1 Dataset criteria: The dataset uses real-world images, supports multiple hypotheses per image, and requires fine-grained reasoning about subtle hypothesis changes.Its design criteria also require disjoint image partitions and attention to dataset bias.
  • 3.2.2 SNLI-VE Construction: SNLI-VE is built by replacing each agreed-label SNLI text premise with its corresponding Flickr30k image.This construction preserves SNLI hypotheses and labels while converting each pair into an image-hypothesis instance.
  • 3.2.2 SNLI-VE Construction: SNLI-VE partitions are disjoint by image, preventing validation and test images from also appearing in training.The authors repartition by images and balance label counts across training, validation, and testing partitions.
  • 3.2.2 SNLI-VE Construction: SNLI-VE inherits hypothesis-conditioned bias from SNLI despite evenly distributed labels, so the paper provides a hypothesis-only lower-bound baseline.The authors explicitly identify this inherited bias as an unavoidable limitation of the construction.
  • 3.3. SNLI-VE and VQA Datasets: SNLI-VE has the largest vocabulary and maximum question length among the compared datasets, with a heavy-tailed length distribution.Its maximum length is 56, and both mean and median lengths exceed those of VQA-v2.0.

4. EVE: Explainable Visual Entailment System

EVE combines text and image branches with self-attention and text-image attention to model fine-grained cross-modal interactions for Visual Entailment. Its variants use image or region-of-interest features, and attention visualizations demonstrate interpretability.

  • 4. EVE: Explainable Visual Entailment System: EVE processes the text hypothesis and image premise in separate branches, then fuses their features through fully connected layers to predict the conclusion.The text branch uses an RNN, while the image branch extracts image features.
  • 4. EVE: Explainable Visual Entailment System: EVE-Image uses ResNet101 feature-map objects, whereas EVE-ROI uses region proposals extracted with Mask R-CNN.The two variants differ in whether image features are organized from spatial feature-map positions or detected regions of interest.
  • 4.1. Self-Attention: Self-attention identifies salient and related features within the text and image branches before cross-modal fusion.The text branch applies self-attention to projected word embeddings, whose MLP transformation can be trained adaptively.
  • 4.2. Text-Image Attention: EVE models fine-grained cross-modal information by combining image and region-of-interest information with attention.The architecture is designed to identify salient inputs and learn image-text interactions.
  • 4.2. Text-Image Attention: Text-image attention uses image features as queries and text features as references, producing weighted image features that are fused with text features for decision making.The interaction weights are computed with the scaled dot-product attention mechanism.

5. Experiments

Experiments compare hypothesis-only, captioning, relational, attention, and EVE models on SNLI-VE, showing that cross-modal attention and richer feature interactions improve performance while revealing dataset and representation limitations.

  • 5.1. Hypothesis Only: 67% accuracy for the hypothesis-only baseline verifies bias in SNLI-VE hypotheses without image-premise information.The authors use image features to seek performance beyond this baseline.
  • 5.2. Image Captioning: 67.83% validation and 67.67% testing accuracy from generated captions only slightly improve on the hypothesis-only baseline.The authors suspect captions omit image details needed by some hypotheses, especially in complex scenes.
  • 5.3. Relational Network: 67.56% validation and 67.55% testing accuracy show that the Relational Network yields only marginal improvement on SNLI-VE.Its limited cross-modal feature fusion may hinder performance on natural images and free-form hypotheses.
  • 5.4. Attention Top-Down and Bottom-Up: 70.53% validation and 70.30% testing accuracy make Attention Top-Down stronger than the ROI-based Attention Bottom-Up model.Attention Bottom-Up reaches 69.34% validation and 68.90% testing accuracy; the authors associate the difference with background information and ROI quality.
  • 5.5. EVE-Image and EVE-ROI: 71.56% validation and 71.16% testing accuracy make EVE-Image the best-performing model, while EVE-ROI reaches 70.81% and 70.47%.The improvement is likely related to self-attention and text-image attention through SDP, which may capture hidden relations and cross-modal interactions.
  • 5.5. EVE-Image and EVE-ROI: Attention visualizations show EVE-Image focusing on the guitar region for the hypothesis “A human playing guitar,” supporting an entailment conclusion.The paper presents these visualizations as evidence of EVE's explainability.
  • 5.6. Discussion: Existing VQA and CLEVR models remain limited on SNLI-VE because the task requires fine-grained information from real images and nonsynthetic sentences.The discussion also identifies external knowledge, localized hypothesis labels, and stronger attention-based alignment as possible directions.

6. Conclusion

The paper introduces visual entailment, the SNLI-VE dataset, and the EVE architecture for fine-grained reasoning over real-world images and text.

  • Visual entailment requires fine-grained reasoning over image premises and text hypotheses.
  • SNLI-VE uses real-world Flickr30k images as premises and corresponding SNLI text hypotheses.
  • The authors propose EVE and evaluate it against multiple baselines, including state-of-the-art VQA-based models.
  • The paper anticipates further VE annotations for large image datasets and improved fine-grained visual-reasoning models.

Supplementary: Additional Examples

Additional examples show SNLI-VE image premises paired with multiple hypotheses and EVE-Image predictions.

  • Each example pairs an image premise with three selected hypotheses carrying different labels.The total number of hypotheses for an image premise is not limited to three.
  • Figure 7 presents random SNLI-VE examples with prediction results from EVE-Image.
Loading 1901.06706v1…