PaperlayerView exact revision on arXiv

Source-linked AI summary

Shifting the Baseline: Single Modality Performance on Visual Navigation & QA

Jesse Thomason, Daniel Gordon, Yonatan Bisk

arXiv:1811.00613v3cs.CL

TL;DR

Random or majority-class comparisons alone may miss informative dataset biases in multimodal navigation and QA. The paper evaluates input ablations of released models across three benchmarks and finds unimodal models outperform benchmark baselines across all of them, sometimes exceeding full models. It recommends including unimodal ablations when evaluating multimodal models.

  • Problem

    Existing work often compares multimodal models with random or majority-class baselines, although dataset biases can occur in any subset of modalities.

    Source: §1 · p. 1

  • Method

    The authors evaluate released models and input-removal ablations on three visual navigation and QA datasets, replacing selected inputs with zero vectors while preserving architecture and parameter count.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Across all benchmarks, unimodal models outperform the original-work baselines, and some also outperform their full-model counterparts.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The authors recommend including unimodal ablations of proposed multimodal models to vet and highlight their learned representations.

    Source: §1 · p. 1, §6 · p. 5

    The QA results are not directly comparable with Gordon et al. (2018), who did not quantify QA effectiveness independently of scene exploration.

    Source: §3.2 · p. 2

Abstract

from arXiv · show

We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research. Where existing work often compares against random or majority class baselines, we argue that unimodal approaches better capture and reflect dataset biases and therefore provide an important comparison when assessing the performance of multimodal techniques. We present unimodal ablations on three recent datasets in visual navigation and QA, seeing an up to 29% absolute gain in performance over published baselines.

1 Introduction

Because dataset biases can occur in any modality, random or majority-class baselines may not reveal whether a multimodal model is solving the task. The paper evaluates unimodal ablations in visual navigation and question answering and recommends broader modality ablations.

1 Introduction

  • Because biases can occur in any subset of modalities, the authors argue that random or majority-class baselines alone are insufficient; unimodal models better expose dataset regularities.The paper notes that unimodal ablations can outperform corresponding full models.Source: §1 · p. 1
  • Across three recent papers, unimodal ablations often outperform accompanying baselines for real-home navigation with crowdsourced descriptions and simulated navigation and egocentric question answering with synthetic questions.Source: §1 · p. 1
  • Future work should ablate both language and vision, since action history and other modalities can also be informative in these tasks.Such baselines can expose gains from unimodal biases regardless of training or architecture details.Source: §1 · p. 1

2 Ablation Evaluation Framework

The evaluation framework compares each benchmark’s released model with versions that remove selected inputs, while keeping the original architecture and parameter count unchanged.

2 Ablation Evaluation Framework

  • At each timestep, agents receive an observation and produce an action that can change their location or heading.Source: §2 · p. 1
  • Matterport training has peaked action-transition and marginal-action distributions, which can support simple sequence rules such as avoiding a left turn immediately after a right turn.Agents may also learn an average number of forward steps.Source: Figure 2 · p. 2
  • Each benchmark’s released code and training paradigm define the full model, and input-removal ablations replace selected inputs with zero vectors.The framework defines the full model and three ablations.Source: §2 · p. 2
  • The ablations isolate access to action, vision, or language inputs while preserving the full model’s architecture and number of parameters.Only the model inputs change.Source: §2 · p. 2

3 Benchmark Tasks

The benchmark suite covers navigation and question answering in real-home and simulated settings. Its task definitions specify the models, baselines, and evaluation conditions used for each benchmark.

3.1 Matterport Room-to-Room

  • Matterport Room-to-Room evaluates navigation with real-home images and crowdsourced language descriptions, comparing a released full model with a random-direction baseline.The baseline takes up to five forward actions and turns right when forward movement is unavailable.Source: §3.1 · p. 2

3.2 Interactive Question Answering

  • IQUAD V1 includes existence, counting, and spatial-relation questions; its construction uses random configurations to weaken majority-class baselines.The introduced THOR-Nav task places an agent randomly and asks it to approach one of three objects in response to a natural-language question.Source: §3.2 · p. 2
  • THOR-Nav navigation uses a CNN model with image and ground-truth semantic-mask inputs, a tiled question, and the previous action; the baseline takes 100 random actions.Ground-truth views replace YOLO during exploration because of speed constraints; the QA baseline is the majority class.Source: §3.2 · p. 3

3.3 Embodied Question Answering

  • EQA questions identify a single unambiguous object in a specific environment, exclude easy questions, and place agents a fixed number of actions from the object at evaluation.The navigation model uses a planner and lower-level controller, while the navigation baseline omits hierarchical control; the QA task also includes an oracle-navigation model and a majority-class baseline.Source: §3.3 · p. 3

4 Experiments

Across Matterport, THOR-Nav, and EQA, unimodal ablations often outperform the baselines used in or derived from the original works. The experiments also show that results vary by training regime, navigation setting, and QA dataset.

4 Experiments

  • On unseen environments, unimodal ablations outperform corresponding full models on Matterport by 2.5% absolute success rate and on EQA by 0.06m in distance to target.Across the benchmarks, unimodal baselines outperform models used in or derived from the original works.Source: §4 · p. 3, §4.1 · p. 3

4.1 Navigation

  • A+L outperforms Full on unseen Matterport, while A+V outperforms Full on EQA and uses scene-structure biases to navigate better than chance on THOR-Nav and EQA.The authors suggest Matterport vision may be noisier than helpful in unseen scenes and that EQA question information may be too distant to infer.Source: §4.1 · p. 4
  • Action-history-only agents outperform baseline agents on Matterport and THOR-Nav, suggesting they learn navigation correlations beyond random actions or programmatic walks.The paper links this near-zero-input performance to minimal sensing and topological trajectory biases.Source: §4.1 · p. 4
  • With student forcing, the Matterport Full Model may align vision and language better through more exploration and training, enabling unseen-environment generalization that fails with teacher forcing.The authors present this explanation as possible, not certain.Source: §4.1 · p. 4
  • At 10 actions from the target, EQA navigation ablations tend to outperform Full on both distance metrics, and A+V performs best among the ablations.A+V ties with or outperforms Full in all but one setting.Source: §4.1 · p. 4

5 Related Work

Related work frames navigation and egocentric QA as language-to-world tasks and notes that models can exploit dataset artifacts; the EQA example illustrates distinct language and vision shortcuts.

5 Related Work

  • Navigation approaches progressed from semantic parsing to neural language-to-world models, while egocentric QA requires agents to navigate and interact to gather answer-relevant information.Source: §5 · p. 5
  • For IQUAD V1, majority-class and chance baselines reach 50%, 50%, and 25% when conditioned on question type; the baseline averages these values.Source: §5 · p. 5
  • In EQA, language-only models identify likely answers, while vision-only models use salient room and color features without access to the question.Source: Figure 4 · p. 5
  • Models can exploit unintended artifacts in language annotations and multimodal data, and the paper’s unimodal ablations uncover similar biases in navigation and QA benchmarks.Source: §5 · p. 5

6 Conclusions

The paper adds unimodal analysis to recent navigation and QA datasets, finding that these models outperform benchmark baselines across the examined benchmarks and can surpass multimodal counterparts.

6 Conclusions

  • The paper introduces an evaluation framework and performs missing unimodal analysis on several new datasets.Source: §6 · p. 5
  • Unimodal models outperform baselines across all examined benchmarks and can even outperform their multimodal counterparts.Source: §6 · p. 5
  • The authors recommend including unimodal ablations of proposed multimodal models to vet and highlight their learned representations.Source: §6 · p. 5

Related papers

Loading 1811.00613v3…