PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Topological Graphs for Macro Action based Closed-loop Reinforcement Learning of Vision Language Navigation in Continuous Environment

Shuhao Ye, Sitong Mao, Yuxiang Cui, Yufei Wei, Xuan Yu, Shichao Zhai, Wen Chen, Shunbo Zhou, Rong Xiong, Yue Wang

arXiv:2609.03906v1cs.RO

TL;DR

VLN-CE requires long-horizon closed-loop navigation, where distribution shift and ambiguous expert actions challenge imitation learning, while micro-action RL is sample-inefficient. The paper uses a hierarchical MDP with topological frontier actions and graph-based PPO for closed-loop fine-tuning. It reports state-of-the-art performance on R2R-CE and RxR-CE.

  • Problem

    In VLN-CE, distribution shift and trajectory deviations challenge imitation learning, while long-horizon micro-action RL faces reward sparsity and credit-assignment difficulties.

    Source: §I · p. 1

  • Method

    The paper reformulates VLN-CE as a hierarchical MDP with macro frontier actions and uses an action-aware value head in graph-based PPO for closed-loop fine-tuning.

    Source: §I · p. 2

  • Results

    The model achieves state-of-the-art performance on R2R-CE and RxR-CE.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    The authors conclude that topological frontier selection reduces the decision horizon and makes closed-loop RL optimization tractable for VLN-CE.

    Source: §V · p. 7

Abstract

from arXiv · show

Vision-Language Navigation in Continuous Environments (VLN-CE) requires an agent to follow natural language instructions through unseen environments. Existing imitation learning (IL) pipelines struggle in this closed-loop setting: behavior cloning suffers from distribution shift, and DAgger's expert actions become ambiguous upon trajectory deviation. While Reinforcement Learning (RL) offers a natural paradigm to address this, directly applying RL to micro action spaces is sample-inefficient due to reward sparsity. To overcome this bottleneck, we reformulate VLN-CE as a Hierarchical Markov Decision Process (MDP), explicitly decoupling high-level planning from low-level control. By abstracting the environment into a topological graph, our high-level policy operates on a macro action space of frontier nodes, with a training-free low-level controller acting as its state transition, which significantly compresses the decision horizon and makes closed-loop RL tractable. To support RL optimization on the macro MDP, we propose an action-aware value head to effectively evaluate state values under the dynamic frontier action space, powering a graph-based PPO. Extensive experiments demonstrate the effectiveness of our architecture. Finally, our model achieves state-of-the-art performance on the R2R-CE and RxR-CE benchmarks.

I. INTRODUCTION

VLN-CE combines long-horizon closed-loop navigation with distribution shift, while micro-action RL faces sparse rewards and inefficient credit assignment. The paper reformulates navigation as a hierarchical MDP and applies closed-loop RL to macro frontier actions, reporting state-of-the-art results on R2R-CE and RxR-CE.

I. INTRODUCTION

  • VLN-CE's hundreds of closed-loop micro actions make navigation vulnerable to distribution shift, error accumulation, and eventual task failure.DAgger's expert actions can become ambiguous after trajectory deviation, while direct micro-action RL faces credit-assignment challenges.Source: §I · p. 1
  • Earlier approaches use topological frontiers to compress decisions or rely on large vision-language models whose semantic priors aid generalization.The cited graph-based methods reduce the decision horizon to 5–20 choices.Source: §I · p. 1
  • A hierarchical MDP separates high-level frontier planning from low-level routing, reducing the decision horizon and making closed-loop RL tractable.The low-level controller handles short-horizon routing as the macro MDP's state transition.Source: §I · p. 2
  • The approach uses a Graph Transformer planner, a rotate-then-forward controller, and an action-aware value head for graph-based PPO.The value head evaluates state values under the frontier graph's dynamic action space; the policy receives closed-loop Reinforcement Fine-Tuning after pretraining and DAgger initialization.Source: §I · p. 2
  • The model achieves state-of-the-art performance on both R2R-CE and RxR-CE.The reported experiments evaluate the proposed architecture on both benchmarks.Source: §I · p. 2

II. RELATED WORK

Prior VLN-CE work spans micro-action policies, waypoint selection, topological-graph planning, and LVLM agents, while RL faces long-horizon costs in continuous environments. The paper frames hierarchical planning with macro frontier choices and low-level routing as an alternative setup for closed-loop RL.

A. VLN in Continuous Environments (VLN-CE)

  • Topological graphs formalize waypoint selection for high-level planning, but gains from increasingly powerful visual feature extractors have begun to plateau.Earlier approaches mapped visual and textual inputs directly to micro actions; waypoint prediction shifted the task toward high-level selection.Source: A. VLN in Continuous Environments… · p. 2
  • LVLM navigation agents benefit from pretrained language understanding but often rely on monocular views and image-frame histories rather than panoramic inputs and topological structure.Source: A. VLN in Continuous Environments… · p. 2

B. Reinforcement learning (RL) in VLN-CE

  • Direct RL in VLN-CE is costly because of micro action spaces and long trajectories; waypoint RL also requires jointly generating navigable, instruction-compliant waypoints.Graph-based VLN separates waypoint generation from instruction following, which the passage describes as an ideal framework for RL.Source: B. Reinforcement learning (RL) in VLN-CE · p. 2
  • Prior RFT approaches are either offline and open-loop or use hundreds of micro-action decisions with dense distance-to-goal rewards; this work proposes macro-action closed-loop RFT without dense reward shaping.Source: B. Reinforcement learning (RL) in VLN-CE · p. 2

A. Problem Formulation: A Hierarchical MDP Perspective

  • The hierarchical formulation assigns RL to a topological-graph macro planner selecting among unvisited frontiers, while a heuristic low-level controller routes toward each selected subgoal.The macro sequence compresses hundreds of micro decisions to 5–20 steps; the micro controller executes short-horizon routing using the discrete motor-action space.Source: A. Problem Formulation: A Hierarchical… · pp. 2–3

B. Closed-loop RFT via Graph-based PPO

The paper chooses on-policy PPO to fine-tune the IL-initialized high-level planner on the macro MDP, using step-wise state-value estimation for credit assignment.

1) Action-Aware Value Estimation:

The action-aware value design addresses partial observability in a macro MDP whose frontier actions change over time. It incorporates available future paths into the state evaluated by the critic.

  • A critic observing only the historical graph trajectory cannot distinguish a well-connected hub from a dead end before the next step.At a bottleneck, all currently available frontier actions may lead to dead ends, making their action values low.Source: 1) Action-Aware Value Estimation · p. 3
  • The proposed action-aware value estimation incorporates the dynamic frontier action space into the state as a forward-looking feature.The graph representation expands from explored history to a holistic graph containing unvisited frontiers.Source: 1) Action-Aware Value Estimation · p. 3
  • A lightweight scalar value head runs parallel to the policy head and shares its cross-modal backbone, adding negligible computational and memory overhead.The critic conditions its value estimate on currently available future paths.Source: 1) Action-Aware Value Estimation · pp. 3–4

2) Outcome-Driven Reward and GAE:

The macro policy is trained with a terminal trajectory reward combining task success, path efficiency, and path fidelity. The resulting temporal-difference errors feed generalized advantage estimation and PPO updates.

C. Hierarchical Policy Architecture

The hierarchical policy combines an online topological graph, a multimodal high-level planner, and a heuristic low-level controller. Its graph represents explored locations and frontier actions, which are encoded with language and visual features for planning.

C. Hierarchical Policy Architecture

  • The system comprises an online graph constructor, a multimodal high-level planner, and a heuristic low-level controller.The planner evaluates the graph state and outputs macro-action logits and a scalar value.Source: C. Hierarchical Policy Architecture · p. 4

1) Topological Graph Construction:

  • The topological graph stores explored nodes and unvisited frontiers, which a waypoint predictor extracts from navigable surroundings and prunes when too close to existing nodes.Explored nodes store 12 panoramic RGB-D images; each frontier is tied to an angular grid in its parent node's observation.Source: 1) Topological Graph Construction · p. 4

2) Modality Encoding:

  • The model encodes instructions with a BERT-style encoder and compresses each explored node's 12 visual views into four directional tokens using attention pooling.Each frontier inherits its parent's angular-grid feature, creating a one-to-one mapping between frontier tokens and macro actions; step and pose embeddings are also added.Source: 2) Modality Encoding · p. 4

3) Graph-Aware Cross-Modal Fusion:

The fusion network incorporates topological relationships and language-derived navigational cues into graph representations. A dual-head planner scores frontier actions and estimates state value, while a heuristic controller executes selected frontiers.

3) Graph-Aware Cross-Modal Fusion:

  • Graph-aware self-attention adds a learned spatial bias derived from topological path distance and relative orientation to the graph attention logits.Relative orientation is represented by its sine and cosine to encode angular continuity.Source: 3) Graph-Aware Cross-Modal Fusion · p. 4
  • After cross-modal fusion, the model distills navigational cues from text to refine the graph representation used by the decision heads.The fusion network uses cross-attention between modalities and self-attention within each modality.Source: 3) Graph-Aware Cross-Modal Fusion · p. 4

4) Dual-Head Decision and Low-Level Control:

  • The actor scores frontier tokens and STOP, while attention pooling over the explored history and frontiers compresses the graph into a scalar state value.The value head uses a learnable query to pool the graph representation.Source: 4) Dual-Head Decision and Low-Level… · p. 4
  • The low-level controller routes to a selected frontier with a rotate-then-forward strategy, traversing graph nodes for distant targets and updating explored nodes and frontiers after execution.It can stop when collisions cannot be resolved by its tryout strategy; graph updates occur regardless of execution success.Source: 4) Dual-Head Decision and Low-Level… · p. 4

D. Training Paradigm

Training progresses from pretraining and DAgger with value-head warmup to closed-loop PPO-based reinforcement fine-tuning. The RFT update is regularized against the DAgger-trained reference policy.

D. Training Paradigm

  • The planner is trained through a progressive three-stage curriculum.The supplied training description specifies pretraining and DAgger as the first two stages, followed by closed-loop RFT.Source: D. Training Paradigm · p. 5
  • During DAgger, the model warms up its Action-Aware Value Head alongside the imitation objective.The joint loss combines cross-entropy imitation loss with a clipped MSE value loss weighted by the value coefficient.Source: D. Training Paradigm · p. 5
  • In the third stage, closed-loop PPO fine-tunes the planner with a KL-divergence regularizer against the DAgger-trained reference policy.The shared-backbone architecture is updated by minimizing the joint objective.Source: D. Training Paradigm · p. 5

P + cvLCLIP

The evaluation covers R2R-CE and RxR-CE, and reports that closed-loop RFT improves on the DAgger baseline on both benchmarks. Ablations examine the value head, RL algorithms, reward formulation, and update frequency.

1) Datasets and Metrics:

2) Implementation Details:

  • Visual encoders are frozen while other modules remain trainable, and the RFT stage takes about 16 hours; the high-level planner makes a macro decision in 0.05 seconds at inference.Pretraining and DAgger each take approximately 72 hours.Source: 2) Implementation Details · p. 5

C. Ablation Studies on the RL Framework

  • A history-only value critic shrinks the SR gain to +1.1 ΔSR versus +3.0 ΔSR and increases training time from 16 to 25 hours relative to the action-aware design.The history-only critic cannot share the policy's cross-modal backbone because its inputs omit generated frontiers.Source: C. Ablation Studies on the RL Framework · p. 5
  • GRPO yields +1.7 ΔSR and SAC +2.1 ΔSR, while SAC's delayed multimodal backbone adds a computational penalty of approximately 21 hours.The comparison holds total trajectory samples consistent; the passage attributes GRPO's moderate gains to lacking a critic.Source: C. Ablation Studies on the RL Framework · pp. 5–6

B. Comparison with State-of-the-Art Methods

  • Compared with the Ours-DAgger baseline, RFT on the macro MDP raises R2R-CE SR from 65.1% to 68.1% and SPL from 53.5% to 57.3%; on RxR-CE, SR reaches 62.3% and nDTW reaches 66.8%.Source: B. Comparison with State-of-the-Art… · p. 5

C. Ablation Studies on the RL Framework

  • Dense soft rewards underperform outcome-driven sparse rewards, and strict on-policy updates outperform two-epoch updates on the same sampled batch for PPO and GRPO.The authors report that dense rewards can lead to geometric shortcuts that do not enforce instruction fidelity, while repeated batch updates overfit local environmental noise.Source: C. Ablation Studies on the RL Framework · p. 6

D. Comparison with Micro Action Space Paradigms

Macro waypoint RFT compresses the decision horizon and achieves substantial gains, while micro-action RFT yields no open-loop gain and only marginal closed-loop improvement.

D. Comparison with Micro Action Space Paradigms

  • The micro-action comparison shares the visual-linguistic backbone and DAgger training, then diverges into open-loop and closed-loop RFT variants.The micro-action policy autoregressively predicts four motor actions, with a new explored node added after moving 3 meters from existing nodes.Source: D. Comparison with Micro Action Space… · p. 6
  • Open-loop RFT produces zero gain over DAgger: reward rises while validation performance declines, indicating overfitting rather than generalization.The authors attribute prior gains from similar methods to weaknesses in their imitation-learning stage, which this DAgger baseline addresses.Source: D. Comparison with Micro Action Space… · p. 6, Table III · p. 6, Figure 3 · p. 6
  • Closed-loop micro-action RFT yields only +0.6 SR and severe reward variance, as each motor command becomes an RL step.The authors link the longer decision horizon to harder credit assignment and an uninformative sparse terminal reward for early decisions.Source: Figure 3 · p. 6, D. Comparison with Micro Action Space… · p. 7
  • Macro waypoint selection compresses hundreds of motor steps to 5–20 choices, enabling closed-loop RFT to converge and achieve substantial gains.The section presents this horizon reduction as addressing the credit-assignment bottleneck.Source: Figure 3 · p. 6, D. Comparison with Micro Action Space… · p. 7

E. Orthogonality of RFT and Dataset Scaling for Performance Gains

RFT gains persist across pretraining dataset scales, and a case study shows the RFT agent recovering from an initial wrong turn. The conclusion reports state-of-the-art benchmark performance while noting a waypoint-predictor reachability boundary.

E. Orthogonality of RFT and Dataset Scaling for Performance Gains

F. Case Study

  • At step 5, RFT uses topological memory to backtrack to the starting frontier and complete the task, while DAgger wanders; RFT favors the correct node over DAgger's flat action distribution.Both models initially move in the wrong direction because of heading ambiguity.Source: F. Case Study · p. 7

V. CONCLUSION

  • Closed-loop RFT via PPO achieves state-of-the-art performance on the R2R-CE and RxR-CE datasets.The conclusion frames hierarchical frontier selection as making closed-loop RL optimization tractable.Source: §V · p. 7
  • The waypoint predictor restricts macro actions to its output distribution, so theoretical global reachability is not guaranteed.The conclusion identifies globally reachable macro action spaces as future work.Source: §V · p. 7

Related papers

Loading 2609.03906v1…