PaperlayerView exact revision on arXiv

Source-linked AI summary

VTNet: Visual Transformer Network for Object Goal Navigation

Heming Du, Xin Yu, Liang Zheng

arXiv:2105.09447v1cs.CV

TL;DR

Object-goal navigation needs visual representations that relate observations to directional actions. VTNet combines spatially aware object and region features and pre-trains them with navigation-action supervision; it reports a 14.0% relative SPL improvement in unseen environments. Its evaluation setup uses monocular RGB observations without maps, 3D meshes, or depth cameras.

  • Problem

    Object-goal navigation requires informative visual representations that connect an agent's observations with directional navigation signals.

    Source: §1 · p. 1

  • Method

    VTNet combines spatial-enhanced local object descriptors and positional global descriptors through attention, then pre-trains the representation to associate visual features with navigation signals.

    Source: §1 · pp. 1–2

  • Results

    14.0% relative improvement on Success Weighted by Path Length (SPL) is reported for learned visual representations in unseen environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The learned representations encode object relationships and correlate with navigation signals, and the paper reports improved visual navigation efficiency in unseen environments.

    Source: §1 · p. 2

    The task setting provides only egocentric monocular RGB images, without prior maps, 3D meshes, or additional sensors such as depth cameras.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Object goal navigation aims to steer an agent towards a target object based on observations of the agent. It is of pivotal importance to design effective visual representations of the observed scene in determining navigation actions. In this paper, we introduce a Visual Transformer Network (VTNet) for learning informative visual representation in navigation. VTNet is a highly effective structure that embodies two key properties for visual representations: First, the relationships among all the object instances in a scene are exploited; Second, the spatial locations of objects and image regions are emphasized so that directional navigation signals can be learned. Furthermore, we also develop a pre-training scheme to associate the visual representations with navigation signals, and thus facilitate navigation policy learning. In a nutshell, VTNet embeds object and region features with their location cues as spatial-aware descriptors and then incorporates all the encoded descriptors through attention operations to achieve informative representation for navigation. Given such visual representations, agents are able to explore the correlations between visual observations and navigation actions. For example, an agent would prioritize "turning right" over "turning left" when the visual representation emphasizes on the right side of activation map. Experiments in the artificial environment AI2-Thor demonstrate that VTNet significantly outperforms state-of-the-art methods in unseen testing environments.

1 INTRODUCTION

VTNet learns visual representations for object-goal navigation by combining object and region features with spatial cues and associating those representations with directional navigation signals. The paper reports improved navigation efficiency in unseen environments.

1 INTRODUCTION

  • VTNet combines spatial-enhanced local object descriptors and positional global descriptors through attention, then associates the resulting representation with directional navigation signals.The descriptors serve as key and query, respectively; pre-training aligns the decoded representations with navigation actions.Source: §1 · pp. 1–2
  • The local descriptor uses all detected instances to model object relationships, while the global descriptor encodes region positions to connect image regions with navigation actions.The local descriptors include object locations, and the global descriptor adds positional embeddings to region features.Source: §1 · pp. 1–2
  • Directly training VTNet with a navigation policy network fails to converge, motivating pre-training that teaches the visual transformer to imitate expert directional actions.The introduction describes human instructions as the warm-up for this instructional representation.Source: §1 · p. 2
  • After pre-training, an LSTM maps the current visual representation and previous states to actions, and A3C learns the navigation policy.The paper reports that the resulting method significantly outperforms state-of-the-art methods in AI2-Thor.Source: §1 · p. 2
  • 14.0% relative improvement on Success Weighted by Path Length (SPL) in unseen environments is reported for the learned visual representation.The comparison is described as an improvement to the efficiency of state-of-the-art visual navigation systems.Source: §1 · p. 2

2 RELATED WORKS

Prior visual-navigation work ranges from map-based approaches to reinforcement-learning systems using visual features, memory, or scene guidance. VTNet is positioned as a visual representation method that emphasizes object-instance relations and directional signals.

2 RELATED WORKS

  • Map-based navigation approaches use mapping, localization, and path planning, but maps are usually unavailable in unseen environments.Prior approaches include mapping around obstructions and inferring robot position with SLAM.Source: §2 · p. 3
  • Visual-navigation methods use reinforcement learning with varied inputs and auxiliary guidance, while prior transformers have modeled language or scene memory rather than local and global visual features tied to directional signals.The paper distinguishes VT from Prevalenet and from a transformer used to select embedded scene-memory slots.Source: §2 · p. 3
  • Target-oriented methods include object-relation graphs, but ORG selects one high-confidence instance per category and may be affected by false positives.The paper also notes that ORG's object features come from the second Faster R-CNN backbone layer.Source: §2 · p. 3

3 VISUAL NAVIGATION REVISIT

The paper revisits object-goal navigation and its general pipeline. Its setup specifies the observations and actions available to an agent, while the pipeline separates visual representation learning from policy learning.

3 VISUAL NAVIGATION REVISIT

  • The revisit section frames visual navigation as a pipeline with visual representation learning followed by navigation-policy learning.The standard pipeline extracts image features into a vector representation, then learns a policy to choose actions.Source: §3 · p. 3, §3.2 · p. 4

3.1 TASK DEFINITION AND SETUP

  • The task uses egocentric monocular RGB observations without maps, 3D meshes, or additional sensors such as depth cameras.The agent moves among grid points using six actions: MoveAhead, RotateLeft, RotateRight, LookUp, LookDown, and Done.Source: §3.1 · p. 3

3.2 PIPELINE

  • VTNet's system uses a visual transformer to decode spatial-aware descriptors, pre-trains it with optimal navigation actions, and then trains an A3C policy with navigation rewards.The overview describes spatial-enhanced local and positional global descriptors as inputs to the visual transformer.Source: Figure 2 · p. 4

3.1 TASK DEFINITION AND SETUP

  • An episode succeeds only when the agent chooses Done within the allowed steps, sees a target, and is within 1.5 meters of it.All three conditions must hold simultaneously.Source: §3.1 · p. 4

4 PROPOSED VISUAL TRANSFORMER NETWORK

VTNet represents a scene using descriptors that encode object relationships and spatial locations, then uses attention and navigation-signal pre-training to support policy learning.

4 PROPOSED VISUAL TRANSFORMER NETWORK

  • VTNet combines spatial-aware object and region descriptors through attention, while pre-training associates its visual representations with navigation signals.The method is intended to ease the training difficulty of the visual transformer and facilitate navigation policy learning.Source: §4 · p. 5

4.1 SPATIAL-ENHANCED LOCAL DESCRIPTOR

  • Unlike ORG, which selects one proposal per category, VTNet uses all detected instances to preserve information and model relationships among objects.DETR features include bounding boxes and class labels and are described as scale-robust; local descriptors are enhanced with object locations.Source: §4.1 · p. 5

4.2 POSITIONAL GLOBAL DESCRIPTOR

  • VTNet adds positional embeddings to region features and uses the resulting positional global descriptors as queries in its decoder.Each global feature represents a particular observation region.Source: §4.2 · p. 5

4.3 VISUAL TRANSFORMER

  • The encoder uses local descriptors as keys and values, while region-specific positional global descriptors query encoded object features to explore region–action correspondences.The paper illustrates the intended correspondence with prioritizing RotateRight when a target lies on the right side of the field of view.Source: §4.3 · p. 6

4.4 PRE-TRAINING VISUAL TRANSFORMER

  • Because direct policy training can produce uninformative features, VTNet pre-trains its visual transformer to imitate optimal navigation instructions.Dijkstra’s algorithm generates the instructions; pre-training uses an MLP to predict actions from current visual representations without the LSTM or previous actions and states.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments evaluate VTNet on AI2-Thor against other navigation methods and examine the effects of its descriptors, transformer, and pre-training. VTNet outperforms ORG and the baseline, while ablations identify components associated with navigation performance.

5.1 PROTOCOLS AND EXPERIMENTAL DETAILS

  • AI2-Thor evaluation uses 22 target categories across four scene types; 80 of 120 rooms train the models, with 40 split between validation and test.Performance is evaluated using success rate and Success Weighted by Path Length (SPL).Source: §5.1 · pp. 6–7

5.3 EVALUATION RESULTS

5.2 COMPETING METHODS

  • Comparisons include Random, SP, WE, SAVN, ORG, and a vanilla VTNet baseline.The baseline feeds concatenated DETR local-instance and global features to A3C, without the spatial-enhanced descriptors or visual transformer.Source: §5.2 · p. 7

5.3 EVALUATION RESULTS

  • VTNet surpasses the baseline by +9.6% in success rate and +0.085 in SPL; the baseline uses only detection features and a global feature for navigation.The baseline does not exploit relations among local instances or associations between visual observations and actions.Source: §5.3 · p. 7
  • In the illustrated test episodes, VTNet and ORG reach the targets, while SAVN and the baseline miss them; VTNet uses fewer steps than ORG.SAVN and the baseline stop after 7 and 19 steps, respectively, without reaching the target.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 VARIANT AND ABLATION STUDY

  • Without spatial enhancement, the network fails to converge; removing global features, positional embeddings, or the VT decoder degrades navigation performance.The authors associate positional embeddings with exploiting spatial information in image regions and the decoder with attending global descriptors to local ones.Source: §5.4 · pp. 8–9
  • DETR instance features outperform both Faster R-CNN feature variants in navigation performance.The authors attribute DETR features’ greater suitability for feature fusion to embedded global-context information.Source: §5.4 · pp. 8–9, Table 2 · p. 9
  • Without VT pre-training, the agent fails to learn an effective navigation policy, so the authors do not report its performance.The pre-training procedure supplies transformer initialization and prior knowledge associating visual observations with navigation actions.Source: §5.4 · p. 9

6 CONCLUSION

VTNet combines spatial-aware object and region descriptors through attention, and its pre-training associates visual representations with navigation actions. The paper reports that it outperforms state-of-the-art methods in effectiveness and efficiency.

6 CONCLUSION

  • VTNet fuses spatial-enhanced local object and positional global descriptors with multi-head attention, exploiting all detected instances; pre-training associates representations with navigation actions.The paper reports that this approach outperforms state-of-the-art methods in effectiveness and efficiency.Source: §6 · p. 9

A.1 FEATURE DETAILS IN VTNET

The feature flowchart details how VTNet builds local and global descriptors and combines them to produce its visual representation.

A.1 FEATURE DETAILS IN VTNET

  • VTNet concatenates 100 instance and location features locally, adds positional embeddings to a 7×7 global feature, then fuses the descriptors through its encoder and decoder.The positional global descriptor and output visual representation each have shape R49×256; the local descriptor has shape R100×256.Source: §A.1 · p. 13, Figure 4 · p. 13

A.2 FAILURE CASE STUDY

VTNet can detect the target yet terminate before reaching the required distance, with the reported failures linked to missing depth information and variation in target size.

A.2 FAILURE CASE STUDY

  • 1.530m and 1.572m are both above the 1.5m threshold in the two failed episodes.The targets were visible, but the agents ended the episodes before moving within the required distance.Source: Figure 5 · p. 13, §A.2 · p. 13
  • Figure 5 uses blue boxes for targets, red lines for agent-to-target distance, and gray curves for agent trajectories.Source: Figure 5 · p. 13
  • The agent may mistakenly estimate that it is within 1.5 meters because it has no depth information and target objects vary in size.In the described cases, the agent finds the target but chooses the termination action too early.Source: §A.2 · p. 13

A.3 CASE STUDY

Figure 6 compares VTNet with SAVN, Baseline, and ORG in testing environments; the caption reports that VTNet reaches targets using the fewest steps.

A.3 CASE STUDY

  • VTNet successfully reaches the targets and uses the shortest steps among the four compared models.Source: Figure 6 · p. 14
  • The figure compares VTNet with SAVN (Wortsman et al., 2019), Baseline, and ORG (Du et al., 2020).Source: Figure 6 · p. 14
  • Green and red curves indicate success and failure cases, respectively, while blue boxes mark target objects.Source: Figure 6 · p. 14

A.4 DIFFERENT VISUAL TRANSFORMER ARCHITECTURES.

The appendix examines transformer depth, pre-training, and attention visualizations, reporting an architecture with the highest success rate and illustrating target-focused attention.

A.4 DIFFERENT VISUAL TRANSFORMER ARCHITECTURES.

  • The highest success rate is achieved with four multi-head self-attention modules and two encoder and decoder layers.The authors report that overly deep transformers may fail to converge to an optimal policy, while a single encoder and decoder layer has insufficient capacity.Source: Table 3 · p. 15, §A.4 · p. 15

A.5 NECESSITY OF PRE-TRAINING SCHEME

  • With pre-training, VTNet spends nearly 10 steps per training episode, while the un-pretrained navigator often stops around 5 steps after tens of thousands of episodes.The authors attribute difficulty training from scratch to uninformative visual representations that can trap the policy in a local minimum.Source: §A.5 · p. 15, Figure 7 · p. 15

A.6 ADDITIONAL VISUALIZATION RESULTS

  • Attention visualizations show high scores on target areas, with the agent attending to detected instances and global image regions.The examples highlight StoveBurner, GarbageCan, and Kettle targets.Source: Figure 8 · p. 16
Loading 2105.09447v1…