PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas H. Li, Mingkui Tan, Chuang Gan

arXiv:2210.07506v1cs.CV

TL;DR

Current approaches make mapping and planning difficult to learn implicitly, while semantic maps can omit object classes and attributes needed to identify instruction-relevant objects. The paper combines semantic and fine-grained map features with weakly supervised object localization, improving VLN-CE success rates by 4.0% in seen and 4.6% in unseen environments.

  • Problem

    End-to-end VLN methods implicitly learn mapping, instruction-vision correspondence, and planning, while semantic maps may omit object classes and attributes needed to distinguish instruction-relevant objects.

    Source: §1 · pp. 1–2

  • Method

    WS-MGMap combines segmentation-based semantics and projected fine-grained map features with an auxiliary task that localizes instruction-relevant objects using path proximity as coarse supervision.

    Source: §1 · p. 2

  • Results

    Success rate improves by 4.0% in seen and 4.6% in unseen VLN-CE environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Qualitative results show the multi-granularity map helps localize instruction objects incorrectly classified by current maps, without extra localization annotation.

    Source: §5 · p. 9

    The semantic loss requires ground-truth semantic annotations, and the 2D top-down map is hard to use when agents go to another floor.

    Source: §5 · p. 9

Abstract

from arXiv · show

We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (e.g., color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. Code is available at https://github.com/PeihaoChen/WS-MGMap.

1 Introduction

The section motivates richer mapping for continuous vision-and-language navigation, where current approaches either learn mapping and planning implicitly or lack object attributes needed to distinguish instruction-relevant objects. It proposes a multi-granularity map and weakly supervised object localization, with higher reported success rates in seen and unseen VLN-CE environments.

1 Introduction

  • End-to-end methods learn mapping, instruction-vision correspondence, and path planning implicitly, increasing learning difficulty; semantic maps also omit classes and attributes needed to distinguish objects.The paper gives the example of distinguishing a “long wooden dining table” from two table instances.Source: §1 · pp. 1–2
  • The multi-granularity map combines segmentation-model object semantics with projected latent features that capture fine-grained details such as color, texture, and shape.The latent features are projected to a top-down map.Source: §1 · p. 2
  • A weakly supervised auxiliary task predicts locations of instruction-relevant objects, using regions near the instruction path as coarse ground truth instead of manual map annotations.The task encourages the map encoder to represent the precise semantics of map regions.Source: §1 · p. 2
  • The learned map and instruction feed a waypoint predictor for the next goal, while an off-the-shelf local policy selects low-level actions.The local policy selects actions such as moving forward or turning.Source: §1 · p. 2
  • Success rate improves by 4.0% in seen and 4.6% in unseen VLN-CE environments with WS-MGMap.The method also robustly localizes objects incorrectly recognized by segmentation models.Source: §1 · p. 2

2 Related Works

Prior navigation maps encode occupancy, topology, semantics, or deep visual features, but each represents only selected environmental information. The paper positions multi-granularity mapping and instruction-relevant localization as a way to represent diverse object semantics and attributes for VLN.

Vision-and-language navigation

  • For VLN, structured semantic top-down maps provide object classes and locations but are limited in representing all classes or object attributes needed to ground instructions.The paper contrasts its approach with these map-based methods by targeting both semantics and attributes.Source: §2 · p. 3

Map representation for navigation

  • Occupancy, topological, semantic, and deep-feature maps encode different environmental information, while VLN instructions require richer object information than these map types individually provide.The proposed multi-granularity map combines complementary granularity information.Source: §2 · p. 3

Weakly-supervised learning for object localization

  • Unlike prior weakly supervised localization work using category information and CAM heat maps, this paper uses instruction-relevant object localization to learn discriminative maps for VLN.Source: §2 · p. 3

3 Vision-and-Language Navigation using Multi-Granularity Map

The section formulates navigation in continuous VLN-CE and presents a multi-granularity map that combines fine-grained object details with semantic information. Weakly supervised object localization trains map representations that support waypoint-based navigation.

3.1 Problem formulation

  • VLN-CE requires following language-described paths in continuous space using low-level actions, without a predefined navigable graph.Source: §3.1 · p. 3

3.2 Perceiving environment via multi-granularity map

  • U-Net latent features form a 12-cm-per-pixel fine-grained map, while a hallucination module predicts 27-category global semantics for fusion.The semantic map predicts objects beyond the camera’s limited field of view.Source: §3.2 · pp. 4–5

3.3 Weakly-supervised map representation learning via object localization task

  • The localization task predicts instruction-object regions and uses path proximity to create a coarse target distribution, encouraging discriminative map representations.The task uses the instruction-relevant path as guidance because exact object locations are unknown.Source: §3.3 · p. 5

3.4 Waypoint navigator and overall learning objective

  • Instruction-relevant localization weights map features for waypoint and progress prediction; fixed DD-PPO converts predicted waypoints into low-level actions, with STOP triggered above a progress threshold.Source: §3.4 · pp. 5–6

4 Experiments

On VLN-CE, WS-MGMap is compared with prior methods and evaluated through ablations of map granularity, localization supervision, and feature layers. The results favor the full multi-granularity map and soft localization supervision, with visualizations showing more precise object localization.

4.1 Experimental setups

  • Experiments use VLN-CE, with 16,844 path-instruction pairs from 90 Matterport3D scenes and about 150k augmented examples; navigation success means stopping within 3m of the goal.Training uses teacher forcing on augmented data followed by DAgger fine-tuning; the segmentation U-Net is pretrained and frozen.Source: §4.1 · p. 6

4.2 Comparisons with state-of-the-art methods

  • WS-MGMap raises success rate against CM2 by 4.0% on val-seen and 4.6% on val-unseen, and outperforms same-setting methods on NE, OS, SR, and SPL.It also outperforms some panoramic-image baselines on val-unseen; the authors argue trajectory length is less informative than SPL, which accounts for both episode length and success.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Ablation studies

  • Success rate rises from 31.6% with fine-grained maps alone and 33.3% with semantic maps alone to 38.9% with both; raw RGB-D falls to 26.5%.The authors attribute the benefit to complementary information: fine-grained features help recognize attributes, while semantic maps reduce the burden of recognizing every object from fine-grained details.Source: §4.3 · p. 8, Table 3 · p. 8
  • Removing object-localization loss, and thus auxiliary supervision for map–instruction correspondence, lowers success from 38.9% to 33.1%; hard binary localization targets outperform no auxiliary task but trail soft targets.Soft ground truth assigns each map region a probability from 0 to 1.Source: §4.3 · p. 8
  • Last-layer features outperform other feature-layer variants on almost all metrics, while the middle-layer variant does not converge.The authors attribute the last-layer result to U-Net skip connections combining lower- and higher-layer information; middle features have 9 × 9 spatial resolution.Source: §4.3 · pp. 8–9, Table 5 · p. 9

4.4 Visualization results

  • In the red-carpet example, WS-MGMap localizes the instruction object more precisely than variants without fine-grained maps or localization supervision, despite segmentation labeling it as floor.The semantic-only variant lacks the red attribute and carpet category, while the variant without localization supervision struggles to align map representations with diverse instruction objects.Source: §4.4 · p. 9, Figure 3 · p. 9

5 Discussion

The paper concludes that combining fine-grained details with semantic information and weakly supervised object localization improves map representation for VLN. It also identifies annotation, floor changes, and real-world evaluation as limitations or future-work concerns.

5 Discussion

  • The semantic loss requires ground-truth semantic annotations, the 2D map is difficult to use across floors, and real-world performance has not been thoroughly evaluated.The authors suggest adapting 3D mapping and note that direct real-world adaptation may risk accidents such as property damage or pedestrian collisions.Source: §5 · p. 9
  • The conclusion credits the multi-granularity map with helping localize instruction objects that current maps classify incorrectly.The method combines fine-grained details and semantic information with a weakly supervised localization task that needs no extra localization annotation.Source: §5 · p. 9

Checklist

The checklist reports that training details and compute resources are documented, but error bars are omitted because of computational cost. The appendix outlines additional architecture, experimental, and analysis material.

Checklist

  • The checklist marks theoretical assumptions and proofs as not applicable and indicates that limitations and potential negative impacts are discussed in Section 5.It also marks consent and personal-information discussions as absent, while human-subject procedures are marked not applicable.Source: Checklist · pp. 12–13
  • Error bars are not reported because running experiments with them would be too computationally expensive; training details and compute resources are documented in Section 4.1.The checklist also indicates that existing assets are cited and their licenses are mentioned.Source: Checklist · p. 13

Appendix for “Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation”

  • The appendix outlines further architecture and training details, ablations, RxR-Habitat results, waypoint analysis, and ambiguity-case visualizations.It also lists additional instruction-relevant localization analysis.Source: Appendix for “Weakly-Supervised… · p. 14

A More architecture details

The appendix provides architecture and implementation details for the map and localization modules, followed by experimental settings and metric definitions.

A More architecture details

  • The appendix describes the semantic hallucination module, map encoder, and object-localization module architectures in Figures A and B.Layer notation follows PyTorch conventions, with tensor shapes represented as (C, H, W).Source: A More architecture details · pp. 14–15
  • The fine-grained, global semantic, and multi-granularity maps use different sizes to save GPU memory, and their sizes can be aligned by changing convolution strides.The RGB encoder is ImageNet-pretrained ResNet18, the depth encoder is point-goal-navigation-pretrained ResNet50, and L denotes instruction word count.Source: A More architecture details · p. 14

B More experimental details

  • Training uses Adam at 2.5e-4; teacher forcing runs for 30 epochs, while dagger training collects 5,000 trajectories per iteration over 10 iterations.Each dagger iteration trains on all collected trajectories for four epochs; the oracle-action probability during collection is 0.5n at iteration n.Source: B More experimental details · p. 15
  • The appendix defines SR and OS using a 3-meter goal threshold, with SR additionally requiring STOP within a 500-step budget.SPL weights success by path length, TL is average trajectory length, and NE is average geodesic distance from the final position to the goal.Source: B More experimental details · p. 15

C More analysis on instruction-relevant object localization

The appendix reports quantitative localization results and visualizes examples involving objects outside the segmentation category list and objects specified by attributes.

C More analysis on instruction-relevant object localization

  • 36.7% IoU for WS-MGMap exceeds 32.3% without the fine-grained map and 6.9% without the localization auxiliary task.IoU compares predicted and ground-truth locations defined as the top 10% of probability area in the respective distributions.Source: C More analysis on… · pp. 15–16
  • Figure C shows localization of out-of-category objects such as refrigerators, shelves, and railings, as well as attribute-described objects such as wooden slatted floors and white doors.The semantic map in the third column is not used by the method; its multi-granularity map is built progressively from fine-grained and semantic information.Source: C More analysis on… · p. 15, Figure C · p. 16

D More ablation study on semantic hallucination module

Under VLN-CE’s 90-degree horizontal field-of-view setting, the semantic hallucination ablation performs worse when map prediction is restricted to the visible area.

D More ablation study on semantic hallucination module

E More ablation results on DAgger training

In the second training stage, replacing DAgger with imitation learning and using ground-truth trajectories at all times significantly reduces performance.

E More ablation results on DAgger training

F More experimental results on RxR-Habitat dataset.

WS-MGMap transfers from R2R-Habitat to the RxR-Habitat English split without finetuning and outperforms LAW variants on both validation splits.

F More experimental results on RxR-Habitat dataset

G More analysis on navigation episodes.

The navigation analysis compares waypoint accuracy across map and localization-task variants, while the ambiguity analysis reports how often attribute-specific instructions encounter ambiguous objects.

G More analysis on navigation episodes

  • The agent builds a multi-granularity map progressively and predicts a waypoint every three time steps from the map and instruction.Source: G More analysis on navigation episodes · p. 17
  • 61.2% of predicted waypoints fall within the ground-truth path area, versus 57.0% without the fine-grained map and 55.1% without the localization auxiliary task.The ground-truth path area is the 10% area with the highest probability in 2D distribution P.Source: G More analysis on navigation episodes · p. 17

H More visualization on instruction-object ambiguity

  • AMT annotations found that 51% of instructions contain specifically attributed object words and 33% of those trajectories have instruction-object ambiguity.Source: H More visualization on… · pp. 17–18
Loading 2210.07506v1…