PaperlayerView exact revision on arXiv

Source-linked AI summary

Does computer vision matter for action?

Brady Zhou, Philipp Krähenbühl, Vladlen Koltun

arXiv:1905.12887v2cs.CVcs.AIcs.LGcs.RO

TL;DR

The paper asks whether explicit computer-vision representations help action, given that some sensorimotor systems map pixels directly to actions. It tests this with controlled simulated tasks and finds that representations improve training, performance, and generalization.

  • Problem

    The paper asks whether computer-vision representations of scene content are useful for action, amid work training systems directly from pixels to actions.

    Source: CONCLUSION · p. 2

  • Method

    The authors compare agents using raw images alone with agents given visual representations across simulated driving, trail traversal, and battle tasks.

    Source: RESULTS · p. 2

  • Results

    Across three challenging tasks, explicit intermediate representations improve sensorimotor performance, and models with them train faster, perform better, and generalize better to unseen environments.

    Source: ANALYSIS · p. 2, CONCLUSION · p. 2

  • Takeaways & Limitations

    The findings support the conclusion that explicit computer-vision representations matter for action in the tested settings.

    Source: CONCLUSION · p. 2

Abstract

from arXiv · show

Computer vision produces representations of scene content. Much computer vision research is predicated on the assumption that these intermediate representations are useful for action. Recent work at the intersection of machine learning and robotics calls this assumption into question by training sensorimotor systems directly for the task at hand, from pixels to actions, with no explicit intermediate representations. Thus the central question of our work: Does computer vision matter for action? We probe this question and its offshoots via immersive simulation, which allows us to conduct controlled reproducible experiments at scale. We instrument immersive three-dimensional environments to simulate challenges such as urban driving, off-road trail traversal, and battle. Our main finding is that computer vision does matter. Models equipped with intermediate representations train faster, achieve higher task performance, and generalize better to previously unseen environments. A video that summarizes the work and illustrates the results can be found at https://youtu.be/4MfWa2yZ0Jc

RESULTS

The experiments compare raw-image agents with agents given explicit visual representations across three simulated tasks. Figure 1C summarizes the main results, where intermediate representations help.

  • The experiments use urban driving, off-road trail traversal, and battle in Grand Theft Auto V and VizDoom simulations.Source: RESULTS · p. 2
  • Agents receive raw visual input alone or additional segmentation, depth and normals, optical flow, or material-property representations.Source: RESULTS · p. 2
  • Intermediate representations clearly help across the main results summarized in Figure 1C.Source: RESULTS · p. 2

ANALYSIS

Explicit intermediate representations improve sensorimotor coordination across three tasks, including when a lightweight network predicts them in situ. Their benefits are particularly apparent in generalization to unseen scenarios.

  • Across three challenging tasks, agents using explicit scene representations learn significantly better sensorimotor coordination, even with imperfect in-situ predictions.The representations are predicted by a small, lightweight deep network.Source: ANALYSIS · p. 2
  • In urban driving, image-only and image-plus-vision agents perform nearly equally on training data, but the latter outperforms on new areas with an order of magnitude less training experience.Source: ANALYSIS · p. 2
  • Agents that predict and use intermediate representations in situ generalize better to unseen test scenarios than pixels-to-actions agents, without privileged test-time information.Source: ANALYSIS · p. 2
  • Related studies of static indoor visual navigation also report faster training and better generalization with intermediate representations, broadly aligning with these findings despite different setups.Source: ANALYSIS · p. 2

CONCLUSION

The paper asks whether computer vision matters for action and concludes that it does. Models with explicit intermediate representations train faster, perform better, and generalize better to unseen environments.

  • The results indicate that computer vision matters for action.Source: CONCLUSION · p. 2
  • Models equipped with explicit intermediate representations train faster and achieve higher task performance.Source: CONCLUSION · p. 2
  • Those models also generalize better to previously unseen environments.Source: CONCLUSION · p. 2

SUPPLEMENTARY MATERIALS

The supplementary materials contain a synopsis of findings, materials and methods, and experiments and analysis.

S1. SYNOPSIS OF FINDINGS

Controlled simulations compare pixels-to-actions agents with agents given explicit vision representations. The representations improve task performance, with especially strong benefits for depth and segmentation, generalization, and training efficiency.

S1. SYNOPSIS OF FINDINGS

  • The experiments use instrumented immersive 3D game simulations to compare end-to-end agents with agents receiving computer-vision representations.The tasks include driving, rough-terrain traversal, and battling opponents.Source: S1. SYNOPSIS OF FINDINGS · p. 3
  • Explicit vision representations significantly improve sensorimotor performance across three challenging tasks and simulation platforms.The comparison is between pixels-to-actions agents and agents receiving representations such as semantic label maps and depth maps.Source: S1. SYNOPSIS OF FINDINGS · p. 3
  • Depth estimation and semantic scene segmentation provide the highest individual capability boosts, while using all capabilities together is more effective still.Source: S1. SYNOPSIS OF FINDINGS · p. 3
  • Vision representations particularly help generalization: image-plus-vision agents outperform image-only agents on new urban areas despite an order of magnitude less training experience.The paper also reports that the performance advantage persists when baselines train indefinitely, tracing it to reduced overfitting of training environments.Source: S1. SYNOPSIS OF FINDINGS · p. 3
  • The findings also hold when agents predict intermediate representations in situ, without privileged information.Source: S1. SYNOPSIS OF FINDINGS · p. 3

S2. MATERIALS AND METHODS

The methods use two game simulators and three sensorimotor tasks to compare agents with different visual inputs. Agents are trained with task-specific control methods, while a U-net predicts vision modalities.

Environments

  • GTAV is an instrumented, realistic city simulation with varied weather, a day-night cycle, and a large game world; VizDoom supports parallel simulations at up to 200× real time.GTAV rendering commands provide vision modalities, and its scripting interface exposes vehicle controls.Source: Environments · p. 4

Sensorimotor tasks

  • The tasks are urban driving, off-road dirt-bike traversal, and VizDoom battle, evaluated by route success metrics or enemies killed.Urban driving and trail traversal use success rate and track-length-weighted success rate; battle aggregates frags over 200,000 time-steps.Source: Sensorimotor tasks · p. 4

Modalities

  • Agents receive raw images and may also receive depth and normals, semantic and instance segmentation, optical flow, or albedo.Albedo is available only in GTAV; training and test regions are geographically separate for GTAV, while VizDoom splits across labyrinth layouts.Source: Modalities · pp. 4–5, Sensorimotor tasks · p. 4

Agents

  • Reactive networks predict actions; GTAV agents learn from autopilot demonstrations, while VizDoom agents use Direct Future Prediction.GTAV training injects noisy actions to support recovery from mistakes; the DFP agent predicts future measurements and action-conditioned differences.Source: Agents · p. 5

Vision networks

  • A U-net infers the vision modalities, using two images when optical flow is predicted and separate loss functions for each modality.The network uses one encoder and a separate decoder for each predicted modality.Source: Vision networks · p. 5, Figure S3 · p. 5

S3. EXPERIMENTS AND ANALYSIS

Controlled experiments across driving, trail traversal, and battle find that explicit vision modalities significantly improve sensorimotor performance. The experiments also compare predicted and unsupervised representations, training-set performance, and the benefits of individual vision capabilities.

S3. EXPERIMENTS AND ANALYSIS

  • Urban-driving and off-road success rates roughly double with all ground-truth vision modalities; VizDoom frags rise by more than 20%, with all results statistically significant.The reported GTAV metrics are success rate and weighed success rate; VizDoom performance is average frags.Source: S3. EXPERIMENTS AND ANALYSIS · p. 6
  • Even a simple in-situ U-net that predicts vision modalities significantly improves on image-only agents across all tasks.The predicted condition has no access to ground-truth data at test time.Source: S3. EXPERIMENTS AND ANALYSIS · p. 6
  • The unsupervised agent requires an order of magnitude more training data than the predicted-vision agent to reach its performance.Unsupervised and predicted agents use the same vision-network architecture, while the unsupervised agent receives no additional supervision.Source: S3. EXPERIMENTS AND ANALYSIS · p. 6
  • All agents fit the training tracks nearly perfectly, indicating that their differences in other experiments reflect different generalization abilities.Table S1 reports performance on a subset of test tracks included in training for this comparison.Source: S3. EXPERIMENTS AND ANALYSIS · p. 6, Table S1 · p. 7
  • On battle, the image-plus-vision agent reaches higher performance with less training data than the image-only agent.Figure S4 plots performance as training-set size grows; shaded areas show standard deviation across evaluation runs.Source: S3. EXPERIMENTS AND ANALYSIS · p. 6, Figure S4 · p. 6

Discussion

  • Semantic and instance segmentation and depth and normal maps boost performance, while optical flow and intrinsic material properties appear less helpful.Segmentation highlights traversable areas and obstacles; depth provides direct obstacle-distance estimates that can guide prioritization.Source: Discussion · p. 6

Related papers

Loading 1905.12887v2…