PaperlayerView exact revision on arXiv

Source-linked AI summary

Auxiliary Tasks and Exploration Enable ObjectNav

Joel Ye, Dhruv Batra, Abhishek Das, Erik Wijmans

arXiv:2104.04112v2cs.CVcs.RO

TL;DR

ObjectNav requires agents to find object instances in unseen environments, where generic CNN+RNN agents have performed poorly and map-based methods dominate. This work equips a generic learned agent with auxiliary tasks and exploration objectives, achieving state-of-the-art benchmark results while analyzing recurrent dynamics and memory.

  • Problem

    Generic CNN+RNN agents perform poorly on ObjectNav, motivating methods that combine analytic and learned components with explicit spatial maps.

  • Method

    The paper trains a simple CNN+RNN agent with auxiliary learning tasks, an exploration reward, and a tethered policy using separate reward signals.

  • Results

    24.5% success and 8.1% SPL improve over prior state-of-the-art by 37% and 8% relatively, respectively, on ObjectNav.

  • Takeaways & Limitations

    End-to-end learning can achieve state-of-the-art ObjectNav results with generic representation-learning and exploration objectives, without requiring explicit maps.

  • Takeaways & Limitations

    The benchmark has substantial split variability, including a large VAL-to-TEST-STD performance drop and 5% success gaps between test splits.

Abstract

from arXiv · show

ObjectGoal Navigation (ObjectNav) is an embodied task wherein agents are to navigate to an object instance in an unseen environment. Prior works have shown that end-to-end ObjectNav agents that use vanilla visual and recurrent modules, e.g. a CNN+RNN, perform poorly due to overfitting and sample inefficiency. This has motivated current state-of-the-art methods to mix analytic and learned components and operate on explicit spatial maps of the environment. We instead re-enable a generic learned agent by adding auxiliary learning tasks and an exploration reward. Our agents achieve 24.5% success and 8.1% SPL, a 37% and 8% relative improvement over prior state-of-the-art, respectively, on the Habitat ObjectNav Challenge. From our analysis, we propose that agents will act to simplify their visual inputs so as to smooth their RNN dynamics, and that auxiliary tasks reduce overfitting by minimizing effective RNN dimensionality; i.e. a performant ObjectNav agent that must maintain coherent plans over long horizons does so by learning smooth, low-dimensional recurrent dynamics. Site: https://joel99.github.io/objectnav/

1. Introduction

ObjectNav requires searching unseen environments for target objects, making it harder than PointNav and challenging vanilla CNN+RNN agents. This work equips a generic learned agent with auxiliary objectives, exploration, and tethered policies, while analyzing recurrent dynamics and memory.

  • ObjectNav requires exploration because the target object may be anywhere, whereas PointNav provides GPS+Compass progress toward specified coordinates.
  • The approach adds semantic inputs, inverse-dynamics and coverage-prediction tasks, and an exploration reward to improve generic-agent learning.
  • Tethered policies separate exploration-guided acting from sparse-reward ObjectNav adaptation, improving efficiency over simple fine-tuning.
  • The analysis links failures to memory, associates simpler visual inputs with smoother recurrent dynamics, and hypothesizes that auxiliary tasks constrain effective RNN dimensionality.
  • 24.5% success on ObjectNav was achieved by training a simple CNN+RNN with auxiliary objectives, a 37% relative improvement over prior state of the art.

2. Approach

The agent operates in unseen Matterport environments using RGBD, GPS+Compass, semantic features, recurrent beliefs, auxiliary prediction tasks, and exploration-based learning signals. Tethered policies address the mismatch between exploration rewards and efficient ObjectNav termination.

  • ObjectNav Definition and Dataset: ObjectNav requires reaching and viewing an instance of the specified category within 1m, using discrete movement, turning, tilt, and stop actions under a 500-step limit.
  • Additional Features: The agent uses RGBD, GPS+Compass, semantic segmentation, and a Semantic Goal Exists feature derived from the fraction of pixels occupied by the goal category.
  • Agent Architecture: Independent GRU belief modules support separate auxiliary tasks, then fuse their states through attention for the actor-critic policy.
  • Learning Signals: Learning combines success, slack, and voxel-visitation exploration rewards, with exploration decayed by visit count and timestep.
  • Tethering to an Exploration Policy: Tethering attaches separate policy heads so exploration and sparse ObjectNav rewards can train distinct policies through off-policy updates.
  • Auxiliary Tasks: Six auxiliary tasks include contrastive, predictive, inverse-action, and coverage-prediction objectives with horizons k = 4, 8, 16, 6, 4, and 16 as specified.
  • Results: 24.5% success was achieved by the auxiliary-task and exploration-enabled agent after RedNet fine-tuning, establishing a new state of the art on TEST-STD.

3. Results

The proposed agents achieve strong ObjectNav performance, while experiments show sensitivity to segmentation quality, action space, exploration rewards, and sparse-reward adaptation.

  • 3. Results: 24.5% success on TEST-STD is a 37% relative improvement over prior state-of-the-art, although the tuned agents are less efficient by SPL.The 6-action agent reaches 24.5% success after RedNet finetuning; both 4- and 6-action agents are less efficient than prior methods.
  • 3. Results: 19.9% success from zero-shot RedNet segmentation is 11% above the prior best and three times the E2E POINTNAV-trained baseline.
  • 3. Results: Exploration-reward agents continue exploring after finding goals, motivating tethered sparse-reward adaptation to improve efficiency.Direct sparse-reward finetuning does not cure exploration-related pathologies and degrades Success; tethered training preserves performance better.
  • 3. Results: All agents degrade with RedNet segmentation, with 6-action agents affected more strongly, while ground-truth segmentation shows minimal overfitting across TRAIN and VAL.Under ground-truth segmentation, SPL declines slightly from TRAIN to VAL, and 6-action agents outperform 4-action agents.
  • 3. Results: Ablations show modest gains from auxiliary tasks, a large performance drop without the SGE sensor, and better Success preservation from tethered than direct sparse-reward training.The auxiliary-task gains are below 1% for ADP, 1% for GID, and 2% for CP; removing SGE causes a 14% drop.
  • 3.1. Stable segmentation is critical to untuned 6-action agents.: The stability hypothesis attributes 6-action sensitivity to agents exploiting simpler floor views, which reduce the burden of maintaining memory under noisy visual inputs.Unstable RedNet segmentation particularly harms floor views, producing erratic repeated turning actions.

4. Agent Analysis

The analysis finds diverse behavioral failures and examines how representations and recurrent dynamics relate to observations, memory, auxiliary tasks, and exploration. It suggests auxiliary tasks organize fixed-point memory while stable low-dimensional dynamics remain a hypothesis constrained by the need to track paths.

  • 4.1. Behavioral Analysis: Agent failures are diverse: exploration problems coexist with getting stuck, exploration-reward pathologies, and scene-quality issues.The analysis covers 300 validation episodes and manually labels failures for base and tethered agents.
  • 4.1. Behavioral Analysis: Exploration-related pathologies persist under the sparse OBJECTNAV reward, while tethering changes the failure profile and introduces early quitting.The sparse reward degrades success, and tethered agents explore worse according to behavioral coding.
  • 4.2. Representational Analysis: Probes find time and maximum SGE encoded across beliefs, but performance varies substantially and fusion can lose abstraction relative to individual beliefs.The fused belief performs worse than individual beliefs, indicating semantic differences among beliefs and the fused representation.
  • 4.3. Dynamical Analysis: Observation embeddings and beliefs generally have correlated curvature, supporting analysis of recurrent dynamics as related to trajectory dynamics, except for CP.The analysis tests whether beliefs are grounded in observations and actions before studying recurrent dynamics proper.
  • 4.3. Dynamical Analysis: Auxiliary tasks consistently specify fixed-point memory spans across seeds, action suites, and objectives, with CP inducing larger spans than the base belief.A memory span counts eigenvalue time constants above a threshold; smaller memories imply more dimensions decay toward fixed points.
  • 4.3. Dynamical Analysis: The proposed ideal of stable low-dimensional dynamics must be reconciled with tracking the agent’s path while continuing to explore new areas.The passage presents this as an unresolved constraint rather than an established property of the agent.

5. Related Work

The related work frames representation learning and recurrent-computation analysis as central to reinforcement learning and embodied agents. It emphasizes that auxiliary tasks may have an underappreciated role in preventing overfitting in complex environments.

  • 5. Related Work: Auxiliary tasks are widely used to improve agent performance, but their role in preventing overfitting to complex environments remains underappreciated.The discussion situates semantic and learned representations within reinforcement-learning research.
  • 5. Related Work: Prior analyses show that recurrent representations can reflect environmental and compositional knowledge beyond static visual representations.This motivates examining recurrent computation from perspectives beyond simple probing tasks.

6. Discussion

The discussion argues that a recurrent agent with an implicit environment representation can achieve state-of-the-art OBJECTNAV performance. It presents auxiliary tasks and exploration policy as generic ingredients for making simple learned agents effective in complex environments.

  • 6. Discussion: A recurrent agent with an implicit environment representation achieves state-of-the-art performance for OBJECTNAV.The conclusion identifies auxiliary tasks and an exploration policy as central generic ingredients.

A. Appendix

The appendix provides additional detail on the agent’s behavior and methodology.

  • A. Appendix: The appendix elaborates on agent behavior and methodology.

A.1. Agent Performance against goal distance, category, and scene

Performance varies across categories, scenes, and goal distances, with rarer categories and more distant goals reducing success; tethered agents favor shorter paths.

  • A.1. Agent Performance against goal distance, category, and scene: Performance variance across OBJECTNAV splits is large enough to blur the definition of state of the art.
  • A.1. Agent Performance against goal distance, category, and scene: Scene success varies from 0.3 to 0.85, while rarer categories and more distant goals reduce the base agent’s success.The authors attribute the TEST-STD drop mainly to increased difficulty rather than a particularly heavy-tailed scene distribution.
  • A.1. Agent Performance against goal distance, category, and scene: Tethered agents preferentially succeed on shorter paths, whereas base agents can succeed on much more distant goals.

A.2. Sparse Reward Reduces Exploration and Causes Quitting

The tethered agent underperforms the base agent because sparse reward reduces exploration and can trigger premature quitting from negative value estimates.

  • A.2. Sparse Reward Reduces Exploration and Causes Quitting: Reduced exploration and early quitting due to estimation error are the primary reasons the tethered agent performs worse than the base agent.The early-quitting behavior is described as a likely general symptom of sparse reward.

A.3. Train-Val Gap

The agent shows little apparent overfitting overall, but moderate overfitting begins after success exceeds 0.5.

  • A.3. Train-Val Gap: Moderate overfitting begins past 0.5 success, despite Table 3 suggesting virtually no overfitting.

A.4. 2D vs 3D GPS Comparison

Evaluation with 2D GPS produces little performance change relative to 3D GPS, while the training and validation curves provide additional learning-dynamics context.

  • A.4. 2D vs 3D GPS Comparison: 2D GPS causes no large performance change compared with 3D GPS, suggesting the agents do not substantially use the vertical dimension.
  • A.4. 2D vs 3D GPS Comparison: Figure A4 compares exploration rates and value-function traces for tethered and base agents under success and failure conditions.
  • A.4. 2D vs 3D GPS Comparison: Figure A5 reports base 6-action agent training and validation curves, with training statistics computed directly from rollout logs.

A.5. Additional Auxiliary Task Ablations

The ablations show that auxiliary tasks contribute to performance, while SGE is more effective as an input feature than as an auxiliary objective. Additional analyses examine failure modes, behavioral instability, probing results, recurrent dynamics, and tethered-policy updates.

  • A.5. Additional Auxiliary Task Ablations: SGE performs better as a feature than as an auxiliary task, while removing all auxiliary tasks causes a moderate performance drop.The auxiliary SGE objective is quickly learned but counterintuitively decreases performance.
  • Failure Modes: Tethered-agent failures often involve quitting despite reasonable exploration and no visible goal, rather than exploration-reward-specific commitment failures.The paper provides a failure-mode breakdown and associated quantitative heuristics.
  • Behavioral Analysis: Higher action entropy correlates with failures, especially for the tethered agent, while predicted segmentation destabilizes some successful episodes and stabilizes some failures.The comparison uses rolling-window action entropy over 300 validation episodes.
  • Behavioral Probes: Visit count and room category are better predicted by timestep than by belief, while distance-to-goal prediction reaches at most about 0.1 R2.These probes assess visit count, distance to goal, and room category.
  • Recurrent Dynamics: Fixed-point eigenvalue spectra show little variation across sampled fixed points, but ObjectNav recurrent fixed-point subspaces remain high-dimensional and difficult to interpret.The analysis identifies slow fixed points by optimizing hidden states under an average input before examining spectra and subspaces.
  • Additional Analyses: Several additional interventions fail to improve performance, including curiosity rewards, action-recall supervision, and projecting beliefs out of a probed time dimension.The time-control intervention could not solve commitment failures without degrading performance.
Loading 2104.04112v2…