PaperlayerView exact revision on arXiv

Source-linked AI summary

e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks

Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, Thomas Lukasiewicz

arXiv:2105.03761v2cs.CVcs.CLcs.LG

TL;DR

Vision-language explanation research lacks reusable evaluation frameworks and has few datasets, while automatic language-generation metrics struggle with valid explanations that differ in meaning. The paper introduces a benchmark, a large explanation dataset, and e-UG, which achieves state-of-the-art performance across datasets.

  • Problem

    VL-NLE research has few datasets and no unified evaluation framework, while valid explanations can differ semantically and existing automatic metrics poorly capture them.

    Source: §1 · p. 1

  • Method

    The paper introduces e-ViL, a unified evaluation benchmark, e-SNLI-VE, and e-UG, which conditions GPT-2 on contextualized image and question embeddings from UNITER.

    Source: §1 · p. 1, §3.2 · p. 3, §5.1 · p. 6

  • Results

    e-UG achieves state-of-the-art performance by a large margin across all datasets, and automatic NLG metrics correlate weakly with human judgment.

    Source: §6 · p. 8

  • Takeaways & Limitations

    e-ViL offers a reusable benchmark for future comparisons of VL-NLE models.

    Source: §6 · p. 8

    Faithfulness of explanations to model reasoning and stronger correlation between automatic NLG metrics and human judgment remain future-work questions.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recently, there has been an increasing number of efforts to introduce models capable of generating natural language explanations (NLEs) for their predictions on vision-language (VL) tasks. Such models are appealing, because they can provide human-friendly and comprehensive explanations. However, there is a lack of comparison between existing methods, which is due to a lack of re-usable evaluation frameworks and a scarcity of datasets. In this work, we introduce e-ViL and e-SNLI-VE. e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks. It spans four models and three datasets and both automatic metrics and human evaluation are used to assess model-generated explanations. e-SNLI-VE is currently the largest existing VL dataset with NLEs (over 430k instances). We also propose a new model that combines UNITER, which learns joint embeddings of images and text, and GPT-2, a pre-trained language model that is well-suited for text generation. It surpasses the previous state of the art by a large margin across all datasets. Code and data are available here: https://github.com/maximek3/e-ViL.

1. Introduction

The paper addresses scarce datasets and inconsistent evaluation for vision-language natural-language explanations, where existing metrics struggle to capture semantically varied but valid explanations. It introduces a reusable benchmark, a large explanation dataset, and a new model that outperforms prior work across three datasets.

1. Introduction

  • Natural-language explanations can express fine-grained reasoning beyond image-region highlighting and may help explain model decisions to lay users.The paper motivates explainability partly by trust, accountability, and understanding or correcting model biases.Source: §1 · p. 1
  • VL-NLE evaluation lacks consensus because existing datasets are few and limited, and valid explanations can differ in both syntax and meaning.Prior human evaluations used different approaches, leaving no objective comparison.Source: §1 · p. 1
  • e-ViL provides the first benchmark spanning three datasets and a reusable unified evaluation framework, enabling comparison of four VL-NLE models.The paper also studies correlations between automatic metrics and human evaluation.Source: §1 · pp. 1–2
  • e-SNLI-VE contributes over 430k instances, making it the largest VL-NLE dataset at the time of the paper.Source: §1 · p. 1
  • e-UG combines UNITER with GPT-2 and surpasses the state of the art by a large, significant margin across all three datasets.Source: §1 · p. 2

2. Related Work

Prior vision explanations often highlight image regions or representations, while natural-language explanations can convey higher-order reasoning. Existing VL-NLE datasets and evaluation practices have limitations, motivating e-SNLI-VE and a unified evaluation framework.

2. Related Work

  • Saliency and attention maps indicate important image regions, while activation vectors map internal representations to human-known concepts; these approaches may cover only part of model reasoning.Natural-language explanations can instead describe higher-order reasoning and complex concepts.Source: §2 · p. 2
  • VQA-X contains 33k QA pairs but includes explanations that can be guessed from the answer, while VQA-E explanations were found low quality and are excluded from the benchmark.VCR targets questions requiring substantial commonsense and presents answers and explanations as multiple-choice options.Source: §2 · p. 2
  • e-SNLI-VE extends VL-NLE datasets with over 430k instances whose explanations rely on image content.Source: §2 · p. 2
  • Existing VL-NLE human evaluations use different datasets, annotator questions, assessment formats, and scoring formulas, preventing direct model comparison.Automatic NLG metrics such as BLEU also perform poorly for evaluating NLEs.Source: §2 · p. 2

3. The e-SNLI-VE Dataset

e-SNLI-VE combines human-written explanations with visual entailment data and applies label correction and explanation-quality filters. Its construction addresses errors in neutral labels and explanations that do not fit image-hypothesis pairs.

3. The e-SNLI-VE Dataset

  • e-SNLI-VE merges e-SNLI explanations with SNLI-VE image-sentence pairs, then uses filtering and manual relabeling to address challenges in combining them.Source: §3 · pp. 2–3

3.1. Correcting SNLI-VE

  • On a manually evaluated subset, 38.6% of SNLI-VE neutral labels were erroneous, so validation and test neutral examples were relabeled by MTurk.Overall, 39% of neutral labels changed to entailment or contradiction; an automatic training filter reduced neutral-label error from 39% to 24% on the subset and from 39% to 30% against validation relabeling.Source: §3.1 · p. 3

3.2. Adding Explanations to SNLI-VE

  • The dataset sources explanations from e-SNLI, but those explanations may not fit image-hypothesis pairs, prompting re-annotation and quality filters.Keyword filtering removed 4.6% of instances, while similarity and uncertainty filters reduced low-quality explanation shares for entailment and contradiction, respectively.Source: §3.2 · pp. 3–4
  • External evaluation found e-SNLI-VE ground-truth explanation quality not far off that of the human-annotated VQA-X and VCR datasets.Table 1 reports e-SNLI-VE summary statistics and identifies contradiction, neutral, and entailment classes.Source: §3.2 · p. 4, Table 1 · p. 4

4. The e-ViL Benchmark

The e-ViL benchmark frames vision-language prediction and explanation as paired tasks, then evaluates explanations alongside task performance across three datasets.

4.1. Task Formulation

  • The task formulation pairs a module predicting answer a from visual and textual inputs with one generating an explanation e for that answer.The final model combines the task module M_T and explanation module M_E.Source: §4.1 · p. 4

4.2. Datasets

  • The benchmark covers e-SNLI-VE, VQA-X, and VCR, which vary in size and domain.VQA-X uses 29.5k, 1.5k, and 2k train/dev/test instances; VCR is reformulated as explanation generation and uses its original validation set for test.Source: §4.2 · p. 4
  • 82.8% of e-SNLI-VE explanations received yes or weak yes in human review, versus 87.9% for VCR and 91.4% for VQA-X.Among 100 re-annotated neutral examples, 17% had flaws; annotators fully agreed on labels for only 54% of 100 image-hypothesis pairs.Source: §4.2 · p. 4, Table 2 · p. 4

4.3. Evaluation

  • The overall score S_O multiplies task performance S_T by explanation score S_E, assigning zero to an explanation when its answer is incorrect.S_E averages explanation scores for correctly answered examples; the framework uses human evaluation because current automated NLG metrics have limitations for NLEs.Source: §4.3 · p. 5
  • Human annotators judge whether each explanation justifies its answer and can flag insufficient justification, image inaccuracies, or nonsensical text.The evaluation question is asked in relation to the image and question or hypothesis.Source: §4.3 · p. 5
  • For each model-dataset pair, evaluation samples 300 correctly answered unique-image examples, while three annotators score each instance independently.Models are evaluated separately with ground-truth explanations included as an unlabeled anchor; the authors report employing over 200 annotators and using statistical tests.Source: §4.3 · p. 5

5. Experimental Evaluation

The experiments compare four VL-NLE models across datasets using human evaluation and automatic metrics. e-UG leads on human explanation scores, while automatic metrics correlate only weakly with human judgments.

5.1. Models

  • e-UG combines UNITER’s contextualized image-region and question-word embeddings with GPT-2 to condition explanation generation on the image and question.The embeddings are prepended to the textual question and predicted answer; inference uses greedy decoding.Source: §5.1 · p. 6

5.2. Training

  • Training uses separate models per dataset, with VQA-X MT models pretrained on VQA v2; hyperparameters are selected using a harmonic-mean score of BERTScore F1 and n-gram metrics.The n-gram component is the harmonic mean of ROUGE-L, SPICE, CIDEr, and METEOR, emphasizing weaker scores.Source: §5.2 · p. 6

5.3.1 Human Evaluation

  • e-UG outperforms the other methods on all datasets, averaging 5.7 points above RVT on explanation score SE.The overall score SO gap widens further, alongside UNITER’s higher VL-task performance than the other models’ MT modules.Source: §5.3.1 · pp. 6–7, Table 3 · p. 7
  • Human judges found that insufficient answer justification was the most frequent shortcoming, while e-UG was significantly better at accurately describing image content.e-UG and PJ-X were better at producing coherent sentences; VQA-X was easiest for models on grammaticality and answer justification.Source: §5.3.1 · p. 7, Table 5 · p. 8

5.3.2 Correlation of NLG Metrics with Human Evaluation

  • BERTScore and METEOR correlate most strongly with human judgments across datasets, but their maximum correlation is only 0.293.Metric reliability is highest on VQA-X and lowest on VCR; BLEU, ROUGE, and CIDEr show no statistically significant correlation on VCR.Source: §5.3.2 · p. 7, Table 6 · p. 8

5.3.3 Explanations as Learning Instructions

  • The study compares task-only training with joint training on task T and explanation task E to test explanations as learning instructions.The comparison covers e-UG on all three datasets and FME and PJ-X on VCR and e-SNLI-VE; the supplied passage does not state the outcome.Source: §5.3.3 · p. 8, Table 4 · p. 8

6. Summary and Outlook

The paper concludes that e-ViL enables reusable comparisons of VL-NLE models and that e-UG achieves state-of-the-art performance across datasets. It also identifies explanation faithfulness and stronger automatic-metric alignment with human judgment as open questions.

6. Summary and Outlook

  • e-UG achieves state-of-the-art performance by a large margin on all datasets using contextualized image-question embeddings.The authors also describe e-ViL as a reusable benchmark for comparing VL-NLE models and report weak correlation between automatic NLG metrics and human judgment.Source: §6 · p. 8
  • Future work should address whether explanations faithfully reflect model reasoning and develop automatic NLG metrics that correlate more strongly with human judgment.Source: §6 · p. 8

Appendices

The appendix uses examples to contrast model explanations and human-evaluation scores, and to illustrate how the datasets differ in their reasoning demands. It also describes e-SNLI-VE as requiring image-specific evidence for meaningful explanations.

A. Qualitative Examples

  • Figure 5 pairs test-set examples from each dataset with ground-truth and model-generated explanations and their predicted human-evaluation scores.The authors point to examples in which e-UG gives better, more image-grounded explanations.Source: A. Qualitative Examples · p. 12
  • e-SNLI-VE examples require image-specific details, while VQA-X examples can often be answered with generic explanations and VCR examples call for commonsense or general knowledge.The appendix notes that one VCR example depends on recognizing a Guns N’ Roses T-shirt, while an e-SNLI-VE contradiction explanation must identify a concrete image element.Source: A. Qualitative Examples · p. 12

B. e-SNLI-VE

  • The appendix includes a datasheet and directs readers to further details on preprocessing filters and MTurk evaluation.It identifies Sections B.4 and D as the locations for those details.Source: B. e-SNLI-VE · p. 12

B.1. e-SNLI-VE Datasheet

The datasheet describes e-SNLI-VE’s composition, construction, and quality controls. It also documents constraints from filtering, residual noise, and ambiguity in image-sentence labels.

B.1.2 Composition

  • 430,796 instances form a reduced sample of the original 570k SNLI sentence pairs, with filtering disproportionately affecting the neutral class.The dataset was created to extend VL-NLE datasets with large-scale fine-grained reasoning examples.Source: §B.1.2 · p. 12, §B.1.1 · p. 12
  • Each instance contains an image, hypothesis, entailment/contradiction/neutral label, and explanation; the dataset relies on publicly available Flickr30k images.Train, development, and test splits are provided with the release.Source: §B.1.2 · p. 12

B.1.3 Collection Process

  • e-SNLI-VE combines e-SNLI explanations with SNLI-VE image-sentence pairs, then uses filters and manual relabeling to address mismatches.Neutral examples in the validation and test sets were reannotated; preprocessing included false-neutral, keyword, similarity, and uncertainty filters.Source: §B.1.3 · p. 15, §B.1.4 · p. 15

B.1.5 Distribution

B.2. Relabeling e-SNLI-VE via MTurk

  • MTurk workers relabeled neutral validation and test examples by choosing a label, highlighting supporting words, and writing an explanation using at least half of them.Each input received three annotations, with trusted examples and a 90% prior-approval threshold used for quality control.Source: §B.2 · p. 15
  • 83.5% of explanations in a manually assessed random subset of 100 examples were judged relevant.Relevance was scored by the fraction of required attributes included in each explanation.Source: §B.2 · p. 16

B.4. Details on Filters

  • On a hand-annotated subset of 535 examples, label error rates were 9.7% for entailment and 8.6% for contradiction, compared with less than 1% reported in the original paper.The authors suggest task ambiguity as one possible explanation for the higher rates; a wrong label also makes its explanation count as bad.Source: §B.4 · p. 16

C.1. Model Architectures

The appendix details how benchmark models were implemented, trained, and adapted across datasets. It also describes the reproduction choices and task-specific settings used for comparison.

C.1. Model Architectures

  • PJ-X uses a simplified MCB module, FME emphasizes visual-feature faithfulness, and RVT feeds image object tags, the answer, and question to GPT-2 for explanation generation.For RVT, the benchmark adds a BERT answer-prediction module because the original model omits question answering.Source: §C.1 · p. 17

C.2. Joint or Separate Training.

  • Joint training backpropagates explanation loss into the task model but may harm optimization; the benchmark therefore trains models both jointly and separately.The text notes possible overfitting risk for VQA-X task models pretrained on larger VQA v2 data.Source: §C.2 · pp. 17–18

C.3. Reproducing Previous Results

  • Reimplemented PJ-X and FME NLG metrics came close to the original reports, despite implementation deviations and use of FME’s base model.PJ-X received LayerNorm and gradient clipping; FME extensions were omitted when they did not improve performance or were difficult to reproduce.Source: §C.3 · p. 18, Table 8 · p. 19

C.4. Hyperparameters

  • RVT and e-UG hyperparameters were selected by grid search on VQA-X, then reused for e-SNLI-VE and VCR; dynamic loss weighting did not warrant added complexity.BERT on VCR required batch size 128 to converge.Source: §C.4 · p. 18, Table 9 · p. 19

C.5. Adaptations for VCR

  • For VCR, UNITER treats each image-question-answer tuple as binary true/false and selects among four answers using a softmax over their true scores.PJ-X and FME score responses with a dot product, while RVT uses BERT-FORMULTIPLECHOICE.Source: §C.5 · p. 18, Table 10 · p. 19

D. Human Evaluation Framework

The appendices describe the human-evaluation materials and cost, then outline additional benchmark results and analyses of the e-ViL score.

D. Human Evaluation Framework

  • Appendix D includes the MTurk evaluation instructions and interface, and reports a cost of 108–117 dollars to evaluate one model on one dataset.Source: D. Human Evaluation Framework · p. 19

E. Results

  • The results appendix covers automatic NLG metrics and extended e-SNLI-VE performance results.Source: E. Results · p. 19
  • The appendix also compares different ways to compute the e-ViL SE score.Source: E. Results · p. 19

E.1. Automatic NLG Metrics

Automatic NLG metrics do not always match human ratings, although e-UG is best overall and consistently exceeds RVT by a small margin. On e-SNLI-VE, every model has lower accuracy and explanation scores for neutral examples.

E.1. Automatic NLG Metrics

  • e-UG is best overall and consistently outperforms RVT by a small margin on automatic NLG metrics, but FME has the highest VCR explanation score.The automatic scores are computed only for test-set explanations whose predicted answers were correct.Source: §E.1 · pp. 19–20
  • A question-and-answer-only GPT-2 model closely matches RVT on most metrics, while RVT is slightly better in most cases.The authors interpret this as evidence that RVT uses visual information ineffectively and relies mostly on spurious correlations.Source: §E.1 · p. 20

E.2. Detailed Results for e-SNLI-VE

  • On e-SNLI-VE, all models have lower accuracy and explanation scores for neutral examples than for the other classes.The authors suggest neutral examples require more reasoning and remain noisier and less represented in training despite cleaning.Source: §E.2 · p. 20

E.3. Statistical Analysis of the SE Score

The statistical analyses support e-UG's advantage in human explanation ratings over the other models. This pattern remains relatively similar under alternative scoring methods, including median pooling.

E.3. Statistical Analysis of the SE Score

  • Figure 16 adds standard errors to the numerical SE scores given in Table 3, and confirms that e-UG uniformly outperforms the other models on those scores.Source: §E.3 · p. 20
  • e-UG significantly outperforms the remaining models in the mixed-model analysis of human SE ratings (p < 0.001).The model uses model as a fixed factor and dataset as a random effect; RVT also significantly outperforms PJ-X (p = 0.007).Source: §E.3 · pp. 20–21

E.4. Alternative SE Scores

  • Alternative SE scoring methods produce different values, but the performance differences between models remain relatively similar.The main-paper score averages numerical mappings of up to three evaluations per explanation, then averages across the sample.Source: §E.4 · pp. 21–22

E.4.1 Median Pooling

  • With median pooling, e-UG performs best across datasets, and RVT ranks second on VCR and VQA-X; differences among PJ-X, FME, and RVT are relatively small.The cumulative link mixed model also finds e-UG significantly outperforms all other models (p < 0.001).Source: §E.4.1 · p. 22

E.4.2 Comparative SE Score

The comparative score judges generated explanations against ground-truth explanations for the same image-question pair. e-UG scores strongest across datasets, while model-generated explanations remain worse than ground truth overall.

E.4.2 Comparative SE Score

  • For each image-question pair, annotators compare ground-truth and generated explanations without knowing which is which, marking generated explanations as equally good, better, or worse.The score is 1 when the generated explanation is as good as or better than ground truth, and 0 otherwise; it does not provide an absolute quality measure.Source: §E.4.2 · p. 22
  • e-UG scores strongest across all datasets; the other models perform similarly except that PJ-X performs worse on VCR.The comparative score is based on whether an explanation is judged as good as or better than ground truth.Source: §E.4.2 · p. 22, Figure 17 · p. 23
  • All model-generated explanations were rated significantly worse than ground-truth explanations in the extended mixed-model analysis.The analysis predicts positive ratings using dataset and annotator as random effects and model as a fixed effect.Source: §E.4.2 · pp. 22–23
Loading 2105.03761v2…