PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law

Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, Anton van den Hengel

arXiv:2005.09241v1cs.CVcs.LG

TL;DR

VQA-CP’s use as an out-of-distribution benchmark is undermined by exploiting split construction, selecting models on the test set, and retraining for in-domain evaluation. The paper evaluates simple methods and proposes benchmark-use and dataset-design guidelines, showing that such methods surpass published results on some or all question types.

  • Problem

    VQA-CP evaluation uses known split construction and the OOD test set for model selection, limiting its assessment of generalization beyond dataset-specific biases.

  • Method

    The paper identifies three VQA-CP evaluation flaws, evaluates embarrassingly simple methods, and develops recommendations for benchmark use and future dataset design.

  • Results

    Embarrassingly simple methods surpass published results on VQA-CP for some or all question types.

  • Takeaways & Limitations

    VQA-CP should compare the same model on in-domain and OOD data while future benchmarks vary distribution shifts and report multiple dimensions.

  • Takeaways & Limitations

    The paper notes that subtle data leakage and biased human-attention annotations may affect methods using them, but their benefit has not been investigated.

Abstract

from arXiv · show

Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set. OOD benchmarks are designed to present a different joint distribution of data and labels between training and test time. VQA-CP has become the standard OOD benchmark for visual question answering, but we discovered three troubling practices in its current use. First, most published methods rely on explicit knowledge of the construction of the OOD splits. They often rely on ``inverting'' the distribution of labels, e.g. answering mostly 'yes' when the common training answer is 'no'. Second, the OOD test set is used for model selection. Third, a model's in-domain performance is assessed after retraining it on in-domain splits (VQA v2) that exhibit a more balanced distribution of labels. These three practices defeat the objective of evaluating generalization, and put into question the value of methods specifically designed for this dataset. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types. We provide short- and long-term solutions to avoid these pitfalls and realize the benefits of OOD evaluation.

1 Introduction

The paper argues that VQA-CP’s OOD evaluation is undermined by methods and practices that exploit known split construction and test-set selection, exemplifying Goodhart’s law. It exposes three flaws, demonstrates that simple methods can surpass published results, and proposes guidelines for better OOD evaluation.

  • Motivation: OOD testing evaluates generalization by changing the joint input-label distribution between training and testing, particularly through differences in task-irrelevant factors [3] [4] [23] [25] [29].The paper distinguishes OOD evaluation from the IID assumptions underlying most datasets.
  • Three critical issues: VQA-CP methods exploit the known, approximately inverse answer distributions between training and test splits, while the OOD test set is also used for model selection.Because VQA-CP lacks an official validation set, this practice is widespread; in-domain evaluation after retraining on VQA v2 hides the problem.
  • Three critical issues: The paper identifies three flaws: exploiting OOD split construction, selecting models on the OOD test set, and evaluating in-domain performance only after retraining on standard VQA v2 splits.Together, these practices defeat the purpose of evaluating generalization and amount to subtly cheating the OOD evaluation.
  • Goodhart’s law: These practices turn VQA-CP performance into a standalone objective rather than evidence of advances in vision-and-language generalization, putting many published methods into question.The paper frames this as a striking example of Goodhart’s law: when benchmark metrics become targets, they cease to measure the intended capability.
  • Contributions: Embarrassingly simple methods, including random answer generation, surpass published VQA-CP results on some or all question types.This demonstration supports the paper’s critique of the benchmark’s current use.
  • Contributions: The paper contributes guidelines for continued VQA-CP use to better capture the benefits of OOD evaluation.It presents these guidelines alongside its analysis of the benchmark’s experimental and methodological flaws.

2 Background

VQA formulates visual question answering as predicting an answer from a question and image, typically as classification over predefined answers [44]. VQA datasets contain substantial biases, while VQA-CP deliberately makes split membership depend on question type and answers, enabling OOD evaluation but also exposing dataset-specific dependencies.

  • VQA formulation: VQA models predict scores over K predefined answers from question tokens and visual features, treating visual question answering as a classification task [44].The model is represented as f(q, v) = apred, with ground-truth answers scored over the same candidate-answer space.
  • Dataset biases: VQA v2 exhibits strong static biases in its question, image, and answer distributions, including frequent question types and overrepresented visual content.Examples include recurring questions about what a person is doing and images featuring activities such as surfing and playing Wii.
  • Statistical dependencies: Although VQA-CP is intended to prevent reliance on language biases, conditioning models on question type or split membership learns dataset-specific dependencies rather than only P(A | Q, V).Such models approximate P(A | Q, V, T, S), exploiting dependencies idiosyncratic to VQA-CP.
  • VQA-CP construction: Unlike randomly split VQA v2, VQA-CP assigns samples using question type and ground-truth answer, creating a train–test shift in the joint distribution of questions and answers.For VQA v2, split membership is approximately independent of the data, whereas VQA-CP makes split membership dependent on T and A.

3 Existing methods for VQA-CP and their issues

Existing VQA-CP methods often target dataset-specific question–answer distribution patterns rather than generalization. Three practices undermine the benchmark: exploiting known split artefacts, selecting models on the OOD test set, and evaluating in-domain performance after retraining on VQA v2.

  • Issue 1: Exploiting split construction: Methods exploit known prefix-dependent test distributions, including direct use of question prefixes or their ground-truth annotations, and the approximately inverse train–test answer distributions [10] [36].This artificial inverse relationship can also be exploited implicitly through model-selection practices.
  • Issue 2: Test-set model selection: Using the OOD test set for validation, hyperparameter tuning, or early stopping causes adaptive overfitting and artificially inflates test accuracy, defeating the benchmark’s purpose [22].The problem is especially consequential because tuning and early stopping massively influence yes/no and number-question accuracy.
  • Issue 3: Retrained in-domain evaluation: Retraining models on VQA v2 for in-domain evaluation measures a different model on a more balanced answer distribution, and methods relying on VQA-CP’s answer inversion significantly lose performance in-domain.This drop indicates limited benefits in overall generalization.

4 Proposed methods

The paper proposes simple, intentionally non-useful VQA-CP baselines designed to test whether high performance can exploit the benchmark’s construction issues. These include random answer sampling, inverted sampling, learned attention models, answer masking, and random-image regularization.

  • Proposed methods: The proposed baselines are intentionally impractical and evaluate how much VQA-CP performance can arise from exploiting its construction issues.They are presented as strong baselines rather than methods with real-world utility.
  • Proposed methods: Random predictions sample answers from the empirical training distribution conditioned on question type, identified by matching each question’s prefix to 65 predefined prefixes [20].The method estimates P(A | T) from the training set and samples accordingly.
  • Proposed methods: Random predictions, inverted, samples from the reciprocal of the training answer distribution, exploiting the approximate inverse relationship between training and test distributions.Empty answer-question bins remain empty when constructing the inverted distribution.
  • Proposed methods: The learned baseline is a standard bottom-up-top-down attention model trained with binary cross-entropy loss.The network produces answer logits before sigmoid or softmax activation.
  • Proposed methods: The top-answer-masked variant assigns the lowest possible score to the answer with the highest predicted score, targeting train-test answer shifts.At test time, it sets the top-scoring answer logit to −∞.
  • Proposed methods: The random-image regularizer pairs questions with unrelated minibatch visual features and minimizes an auxiliary softmax loss that suppresses the correct answer while elevating alternatives.It combines the main binary cross-entropy loss with the auxiliary loss using a scalar λ, with equal numbers of original and randomized instances per minibatch; the design avoids gradient reversal and splitting.

5 Experiments

Experiments show that simple distribution-aware predictors outperform published methods on VQA-CP’s OOD test, while exposing a trade-off between in-domain and OOD accuracy. Existing improvements are concentrated on yes/no/number questions, limiting conclusions about broader generalization.

  • Setup: Models are trained on VQA-CP and VQA v2, with 8,000 VQA-CP training instances held out to measure in-domain validation performance.This held-out validation set follows the evaluation setup proposed in [22] [14] [41].
  • Results: Top answer masked improves VQA-CP test accuracy from 43% to 82% on yes/no and from 12% to 27% on number questions.Despite strong OOD performance, its VQA-CP validation accuracy is extremely low, indicating an in-domain/OOD trade-off rather than overall improvement.
  • Results: Random predictions inverted surpass all other methods on VQA-CP test, exceeding 83% on yes/no and 49% on number questions.Sampling from the training distribution performs well on the in-domain validation set, whereas inverted sampling performs well on the OOD test.
  • Results: The random-image regularizer tunes the in-domain/OOD trade-off through λ, with higher weight improving OOD accuracy while reducing in-domain accuracy.The effect is clearest with the proposed held-out validation set and is less obvious after retraining on VQA v2.
  • Results: Most claimed OOD improvements arise from yes/no/number questions because methods are tuned for overall accuracy, leaving performance on other questions underrepresented.A simple classifier identifies answer type with over 99% accuracy, enabling this concentration of tuning; exceptions include methods tuned on other questions [13] [14] [41].

6 Recommendations and discussion

The authors retain VQA-CP as a useful benchmark but recommend changes to preserve its purpose of measuring generalization beyond language biases. They also propose broader OOD dataset designs and stronger scientific standards for statistical reporting and analysis.

  • Recommendations for using VQA-CP: VQA-CP remains useful for measuring VQA progress if evaluation preserves its original focus on generalization and resistance to language biases.
  • Recommendations for using VQA-CP: The same model, training, and hyperparameters should be evaluated in-domain and OOD, with an 8k-instance holdout from VQA-CP training data replacing retraining on VQA v2 for in-domain evaluation.The authors also recommend focusing analysis exclusively on other questions, although the supplied passage truncates the rationale.
  • Recommendations for future datasets: Future OOD benchmarks should test multiple distribution-shift levels and types, spanning IID to extreme OOD splits and varying confounders rather than one controlled question-prefix/answer correlation.
  • Recommendations for future datasets: A single aggregate metric may be insufficient for evaluating OOD generalization.
  • Discussion: OOD evaluation also requires attention to subtle data leakage, including biased human-attention annotations that may expose training labels closer to the OOD test distribution.The authors call for higher standards of scientific investigation, statistical reporting, and analysis, including more rigorous empirical assessment than relying on top-benchmark scores alone.

Supplementary material · A Implementation details · A.1 Implementation of the proposed methods

The proposed methods use thresholded answer sets and standard BUTD implementations, with the random-image regularizer applied to a BUTD model pretrained using standard BCE loss.

  • A.1 Implementation of the proposed methods: Answer-discarding methods retain yes, no, 1, 2, 3, and 1,100 other answers after thresholding to prevent extreme probabilities for rare training answers.The threshold avoids effectively inverting the distribution for answers observed below a fixed training-frequency threshold.
  • A.1 Implementation of the proposed methods: The random-image regularizer uses a BUTD model pretrained with standard BCE loss for 25 epochs, while other BUTD-based methods follow standard implementations and hyperparameters.

A.2 Experimental setup

The experiments follow common but flawed evaluation practices: training on VQA-CP and then VQA v2, selecting epochs by test or validation accuracy, and comparing existing methods at their reported best test performance.

  • A.2 Experimental setup: Models are trained first on VQA-CP v2 and then on VQA v2, following the study’s stated common practice.
  • A.2 Experimental setup: Epochs are selected using the highest VQA-CP test accuracy or VQA v2 validation accuracy, respectively.
  • A.2 Experimental setup: For in-domain evaluation on VQA-CP, 8,000 training instances are held out, while existing methods use the highest overall test accuracy reported by their authors.The in-domain protocol follows [41] [14].

B Additional results · C Distribution of answers per question type in VQA v2 and VQA-CP

Additional results report VQA-CP “Other” question accuracy for baseline and proposed methods, examine random-image regularization across weights and question types, and compare answer distributions by question prefix in VQA v2 and VQA-CP.

  • B Additional results: Table 3 reports accuracy on VQA-CP “Other” questions for methods trained and evaluated on that question type.The table includes both each method’s baseline and proposed versions because existing methods use different underlying baseline models.
  • B Additional results: The random-image regularizer is evaluated across regularizer weights for all questions and for yes/no, number, and other answers.Figure 5 organizes these results from all questions to the three answer categories.
  • B Additional results: The additional-results section distinguishes overall performance from performance on specific answer categories.Figure 5 separates all questions from yes/no, number, and other answers, while Table 3 isolates “Other” questions.
  • B Additional results; C Distribution of answers per question type in VQA v2 and VQA-CP: Together, the figures characterize regularization sensitivity and answer-frequency structure across the two VQA datasets.Figure 5 varies the regularizer weight, whereas Figure 6 compares answer histograms across question prefixes.
  • C Distribution of answers per question type in VQA v2 and VQA-CP: Figure 6 compares histograms of the ten most frequent answers for every question prefix in VQA v2 and VQA-CP.The final empty prefix serves as the catch-all default, and stop words are omitted from the visualization.
  • C Distribution of answers per question type in VQA v2 and VQA-CP: The answer-distribution comparison is organized by question prefixes rather than by individual questions.Because stop words are omitted, distinct prefixes can appear identical in the figure, such as “What is” and “What is the.”
Loading 2005.09241v1…