PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied-Navigator: Point, Think, Memorize, and Align for Efficient Navigation

Hongyan Feng, Sunlai Chen, Xuanyu Liu, Miao Pan, Yangfan Xie, Yuxiang Cui, Zhongxiang Zhou, Rong Xiong, Wenqi Zhang, Jianwei Yin, Yueting Zhuang, Xuhong Zhang

arXiv:2608.17512v1cs.RO

TL;DR

VLM-based embodied navigation remains limited by mismatched 3D action reasoning, rigid reasoning schedules, and inefficient memory. TAMP-Nav addresses these issues with pixel-based actions, selective reasoning and trajectory memory, and two-level GRPO, achieving 66.2% SR on R2R-CE with 90k training trajectories.

  • Problem

    VLM navigation remains limited by mismatched 3D geometric reasoning and memory mechanisms that either overflow context or lose spatiotemporal information.

  • Method

    TAMP-Nav combines Pixel-to-3D action selection, selective reasoning with anchor-trajectory memory, and two-level GRPO alignment, supported by 90k selective-CoT training trajectories.

  • Results

    66.2% SR on R2R-CE is achieved with state-of-the-art performance and high sample efficiency from 90k training trajectories.

  • Takeaways & Limitations

    TAMP-Nav provides an efficient navigation framework that aligns cognitive planning with physical environmental feedback and supports adaptive reasoning.

  • Takeaways & Limitations

    Physical deployment is limited to zero-shot transfer because GRPO training relies on privileged simulator signals and the system lacks online reinforcement learning on robots.

Abstract

from arXiv · show

Although Large Vision-Language Models (VLMs) have significantly advanced embodied navigation, their direct deployment remains challenging, as existing methods often force VLMs into unnatural action spaces that misalign with their 2D pre-training priors, compounded by rigid reasoning schedules and inefficient memory management. To overcome these limitations, we propose TAMP-Nav, a unified framework for efficient embodied navigation. First, we introduce a Pixel-to-3D Action Formulation (Point) that reformulates navigation into 2D visual prompting. Specifically, the VLM merely selects 2D pixels, which are then projected into 3D coordinates for a low-level SLAM controller. This design naturally aligns embodied execution with the VLM's inherent 2D visual capabilities. Second, we propose an integrated Selective Reasoning and Anchor-Trajectory Memory mechanism (Think and Memorize), which dynamically triggers Chain-of-Thought and retains high-fidelity memory only at critical nodes, compressing redundant trajectories into lightweight Space-Time Indicators, thereby preserving critical historical information and enhancing spatio-temporal perception. Finally, we design an efficient Two-Level Alignment Paradigm (Align) via Group Relative Policy Optimization (GRPO). By superimposing global outcome rewards with fine-grained process rewards, this dense supervision tightly aligns the agent's cognitive planning with physical environmental feedback, endowing the model with adaptive reasoning capabilities. Experiments demonstrate that TAMP-Nav achieves state-of-the-art performance (e.g., 66.2% SR on R2R-CE) with high runtime and sample efficiency (requiring only 90k training trajectories).

1 Introduction

TAMP-Nav addresses three bottlenecks in embodied navigation: the geometric mismatch between VLMs’ 2D pre-training and 3D action spaces, inefficient rigid reasoning, and inadequate long-horizon memory. It introduces Pixel-to-3D action formulation, selective reasoning with Anchor-Trajectory Memory, and two-level GRPO alignment, achieving 66.2% SR on R2R-CE with 90k training trajectories.

  • Challenges: VLMs’ 2D pre-training makes implicit 3D geometric reasoning prone to spatial hallucinations and low sample efficiency.Existing methods map observations to atomic actions or regress 3D coordinates, creating a geometric gap with VLM priors.
  • Challenges: Rigidly triggering Chain-of-Thought at every step or fixed intervals improves decision quality but causes severe inference latency.These designs fail to balance runtime efficiency and navigation performance.
  • Challenges: Long-horizon memory faces attention dilution and overflow when retaining full histories, while indiscriminate forgetting can discard critical information.The resulting loss of key information harms spatio-temporal perception.
  • Point: TAMP-Nav’s Pixel-to-3D action space lets the VLM output 2D pixel coordinates that are projected into 3D and executed by a local SLAM controller.This decouples visual prompting from low-level execution and leverages 2D VLM pre-training priors for 3D navigation.
  • Think and Memorize; Align: 66.2% SR on R2R-CE and 90k training trajectories demonstrate TAMP-Nav’s state-of-the-art performance and sample efficiency.Its Think and Memorize mechanism triggers deep CoT at critical topological nodes, stores explicit anchors, compresses routine intervals into Space-Time Indicators, and Align applies global outcome plus fine-grained process rewards through two-level GRPO.

2 MultiNav-CoT Dataset

MultiNav-CoT is a curated 90k-trajectory dataset derived from VLN-CE, filtered for quality and equipped with selective Chain-of-Thought annotations. It supports sparse reasoning through spatiotemporal key-node mining and structured multi-stage reasoning generation.

  • Dataset Construction: 90k trajectories comprise MultiNav-CoT, derived from VLN-CE and filtered to remove rendering artifacts and Gemini-assessed low-quality instructions.Examples of rendering artifacts include black images.
  • Dataset Construction: Selective Chain-of-Thought annotations supervise sparse reasoning and address the reinforcement-learning cold-start problem.
  • Spatiotemporal Key Node Mining: Spatiotemporal key-node mining combines semantic relevance and scene-transition signals, then greedily filters score-ranked frames by minimum spatial distance.Ssem measures CLIP-based relevance to the instruction, while Svis measures visual scene transitions.
  • Structured CoT Generation: Structured CoT generation decomposes annotation into task phase localization, current observation analysis, and future action reasoning.The multi-stage prompt pipeline is based on Gemini 2.5 flash and reduces the cognitive load of a single query.

3 Method

TAMP-Nav unifies 2D visual pointing, selective reasoning with structured trajectory memory, and two-level GRPO to connect VLM planning with efficient low-level navigation. Its design grounds actions in projected 3D points, preserves critical spatiotemporal history, and combines local operational supervision with global trajectory objectives.

  • Framework Overview: TAMP-Nav continuously integrates visual observations with Anchor-Trajectory Memory, selectively triggers reasoning at complex scenarios, and executes grounded navigation actions.The framework is organized as an interaction loop linking perception, memory, reasoning, and execution.
  • Pixel-to-3D Action Formulation: The VLM selects one of four 360° egocentric views and a 2D target pixel, which is projected into 3D and dispatched to a low-level SLAM controller.This relieves the VLM from learning geometric transformations and focuses it on visual-semantic grounding.
  • Anchor-Trajectory Memory: The memory combines Explicit Anchors at reasoning nodes with Space-Time Indicators encoding position, time, and orientation into a fixed-length token.Orientation uses continuous sin(yaw), cos(yaw) features to avoid the 0°/360° discontinuity.
  • Anchor-Trajectory Memory: Between anchors, redundant visual features are discarded while spatiotemporal trajectory information is retained, concentrating visual overhead on critical nodes.Explicit anchors also preserve high-fidelity visuals and Chain-of-Thought as planning beacons.

4 Experiments

Experiments evaluate TAMP-Nav on VLN-CE benchmarks, long-horizon trajectories, emergent reasoning, efficiency, ablations, and real-robot transfer. The method achieves strong navigation performance while using sparse reasoning, compact interactions, and limited training data.

  • VLN-CE Benchmarks: TAMP-Nav achieves SOTA performance on validation-unseen splits, reaching 66.2% on R2R-CE and 65.7% on RxR-CE.Evaluation uses SR, OSR, SPL, nDTW, and related standard VLN-CE metrics.
  • Efficiency: 90k training trajectories and 700k interactions yield competitive results, versus 763k trajectories for DualVLN and 3.37M interactions for NavFoM.Dense-reward RL improves R2R-CE SR from 55.7% with SFT-only to 66.2%.
  • Efficiency: 9 interaction steps per trajectory reduce the horizon from approximately 30 steps required by DualVLN and StreamVLN.The reduction is attributed to Pixel-to-3D actions, sparse reasoning, and lightweight memory.
  • Long-Horizon Tasks: 49.8% SR on long-horizon trajectories surpasses StreamVLN at 30.9% and DualVLN at 41.9%.The evaluated subset contains 5927 trajectories with expert paths exceeding 12.5 meters, or more than 50 atomic forward actions.
  • Selective Reasoning: RL alignment reduces reasoning allocated to straight corridors from 38% in SFT to 11%, concentrating CoT at critical topological nodes.TAMP-Nav reaches 66.2% SR with a 26.3% CoT ratio, nearly matching the Dense CoT upper bound of 66.8%.
  • Real-World Transfer: 60.0% real-world SR without real-robot fine-tuning outperforms StreamVLN at 49.0% and DualVLN at 53.0%.The representative execution demonstrates instruction following and reasoning-on-demand behavior.

5 Limitations

TAMP-Nav remains sensitive to depth, SLAM reliability, and simulator-dependent training signals, limiting robust physical deployment. Its heuristic reasoning triggers and planar memory representation may also miss critical states or explicit multi-floor localization.

  • System dependencies: Depth errors, odometry or SLAM drift, localization failures, and dynamic environments can degrade Pixel-to-3D projection, STI encoding, and low-level execution.The strongest tested depth perturbation caused a 2.8-point SR decrease, but more severe failures may cause larger degradation.
  • System dependencies: GRPO training requires privileged simulator signals, and physical deployment is limited to zero-shot transfer because online reinforcement learning is unsupported.Real-world rollouts are slower than simulation, while complete online rewards would require endpoint annotations, human feedback, or a reliable learned verifier.
  • Reasoning and memory: Semantic-similarity and visual-change heuristics may under-select critical decision states when they are visually non-salient or instructions differ from R2R-CE and RxR-CE.These settings may require additional training examples or a learned triggering criterion.
  • Reasoning and memory: STI encodes planar position (x, y) and orientation but omits height, preventing explicit floor distinction and leaving task-progress descriptions as only partial compensation.A full 3D pose representation may improve robustness in multi-floor buildings and uneven terrain.

6 Discussion and Conclusion … C Algorithm for Key Node Selection

TAMP-Nav unifies pixel-based action selection, selective reasoning with anchor-trajectory memory, and two-level GRPO to align visual-language planning with physical navigation while preserving efficiency. The paper concludes that the framework currently targets VLN-CE and identifies broader embodied tasks and dynamic environments as future directions.

  • 6 Discussion and Conclusion: TAMP-Nav combines Pixel-to-3D Action Formulation, Selective Reasoning and Anchor-Trajectory Memory, and Two-Level GRPO to align high-level reasoning with low-level execution.The framework maintains high sample and inference efficiency.
  • A Detailed Reward Formulations: Local rewards evaluate target approach, collision avoidance, stopping, reasoning value, and output-format adherence during navigation.Reasoning receives reward only when it produces a decision outperforming the mean approach reward across sampled candidates.
  • A Detailed Reward Formulations: Global rewards combine task success, SPL-based trajectory efficiency, and reasoning density to assess completed trajectories and discourage excessive thinking.Reasoning density assigns zero reward when the reasoning-step ratio exceeds 0.6.
  • B.1 Action Formulation in VLM-based Navigation: Because VLMs have 2D-native priors and weak explicit 3D reasoning, TAMP-Nav selects image pixels, projects them into 3D, and delegates execution to SLAM.This design bypasses much of the 2D-to-3D representation gap.
  • B.2 Reasoning and Memory in VLM-based Navigation: TAMP-Nav triggers deep CoT only at critical decision nodes and compresses routine paths into spatiotemporal tokens, bounding computation while preserving long-horizon awareness.The mechanism addresses rigid dense reasoning and inefficient memory use.
  • B.3 Reinforcement Learning for VLM-based Navigation: Two-Level GRPO superimposes global terminal advantages with local step advantages, providing dense supervision without sparse-reward credit-assignment problems or stepwise expert forcing.This balances trajectory-level outcomes with fine-grained navigation feedback.
  • C Algorithm for Key Node Selection: Key-node mining uses visual-semantic importance scoring, distance-based greedy filtering, and temporal infilling to trigger reasoning sparsely while preserving topological connectivity.The combined score S(t) = Ssem(t) + Svis(t) uses trajectory-level Min-Max normalization to balance semantic and visual contributions.
  • C Algorithm for Key Node Selection: Semantic importance measures instruction-observation similarity, while visual transition importance measures frame-to-frame feature change, with the first frame assigned zero raw transition score.Both scores are independently standardized within each trajectory before combination.

D Real-World Experiment Details

TAMP-Nav deploys on a Unitree Go2 using decoupled VLM perception and classical low-level control, and demonstrates zero-shot transfer in unseen physical environments without real-world fine-tuning.

  • Hardware and System Setup: The platform uses a Unitree Go2 quadruped with four annular RGB-D cameras providing approximately 360° coverage and an underneath-mounted Hesai XT16 LiDAR.Each camera provides an approximately 100° field of view.
  • Hardware and System Setup: TAMP-Nav runs high-level perception remotely, converting real-time multi-view camera images into 2D pixel coordinates as navigation targets.Aligned depth from the selected RGB-D camera projects these pixels into 3D world coordinates.
  • Hardware and System Setup: FastLIO estimates ego-motion, while FAR Planner performs local path planning and dynamic obstacle avoidance for low-level execution.This delegates odometry, path execution, and motion safety to a classical SLAM stack.
  • Qualitative Analysis in Unseen Environments: The agent accurately localizes objects and follows sequential spatial instructions across unmapped physical environments containing diverse objects, corridors, and obstacles.Five representative execution trajectories are presented in Figure 7.
  • Qualitative Analysis in Unseen Environments: TAMP-Nav processes complex multi-stage instructions over longer physical distances and supports zero-shot sim-to-real transfer after simulation-only training.The framework receives no real-world fine-tuning.

E Analysis of STI token

The STI positional encoder converts pose coordinates into a unified token representation for the VLM and preserves their underlying geometric structure. Visualization confirms consistent recovery of yaw, x, and y across corresponding spatial dimensions.

  • Pose encoding: The positional encoder maps the initial pose p = (x, y, yaw) to an STI token embedding used as the unified pose representation for the subsequent VLM.The embedding dimension is denoted by d.
  • Pose reconstruction: High-dimensional STI embeddings are reduced into a 4-dimensional latent variable aligned with normalized ground-truth pose components.The reduced representation is used to reconstruct predicted pose variables (ˆx, ˆy, ˆ yaw).
  • Visualization: Figure 8 maps ground-truth poses into a 3D HSV-like coordinate system, with yaw encoded circumferentially, x radially, and y vertically.Each block’s spatial position is determined by its original ground-truth pose.
  • Visualization: The reconstructed color field changes consistently with spatial layout, indicating preserved pose information in the positional encoder.Hue tracks reconstructed ˆ yaw, saturation tracks ˆx, and value tracks ˆy; smooth changes are expected along circumference, radius, and vertical height, respectively.

F Simulation Execution Example · G Prompts for Structured CoT Generation

The simulation example shows TAMP-Nav grounding a multi-step instruction through selective reasoning at critical nodes and pixel-level point-and-click actions. Its structured-CoT pipeline uses multi-stage prompts for instruction filtering, task-phase localization, and current-observation analysis.

  • F Simulation Execution Example: The trajectory highlights pixel-level point selection as the mechanism for selecting navigation actions.Figure 9 presents the representative simulated trajectory and its point-selection behavior.
  • F Simulation Execution Example: TAMP-Nav demonstrates robust visual-semantic grounding and logical planning on a multi-step stairway, turning, and doorway instruction.The example uses the instruction to go upstairs, turn left, and wait at the bedroom doorway.
  • F Simulation Execution Example: The agent triggers Chain-of-Thought at critical topological nodes, including locating stairs, choosing the turn, and executing the final stop.These decisions illustrate reasoning-on-demand during navigation.
  • G Prompts for Structured CoT Generation: TAMP-Nav uses a multi-stage prompt-engineering pipeline to improve reasoning quality and mitigate hallucinations during MultiNav-CoT dataset construction.The documented prompts are textually refined to expose cognitive guidance and reasoning rules while omitting complex JSON validation and defensive directives.
  • G Prompts for Structured CoT Generation: Instruction quality filtering is applied during pre-processing before structured Chain-of-Thought generation.This procedure is identified as Table 7 in the prompt-construction pipeline.
  • G Prompts for Structured CoT Generation: Stage 1 localizes the task phase by summarizing navigation history.This stage is identified as Table 8.
  • G Prompts for Structured CoT Generation: Stage 2 analyzes the current observation as part of the structured reasoning pipeline.This stage is identified as Table 9.

H.1 Supervised Fine-Tuning (SFT) Phase

The SFT cold-start phase fine-tunes Qwen2.5-VL-7B on MultiNav-CoT augmented with approximately 10% gqa_interleaved_cot to internalize navigation logic while mitigating catastrophic forgetting. Training runs for one epoch with memory-efficient optimization and specified learning-rate, sequence-length, and gradient-accumulation settings.

  • SFT data and objective: The cold-start SFT fine-tunes Qwen2.5-VL-7B on MultiNav-CoT augmented with approximately 10% gqa_interleaved_cot to preserve general visual reasoning.gqa_interleaved_cot is derived from GQA and augmented with interleaved Chain-of-Thought annotations.
  • Optimization setup: SFT runs for 1 epoch in bf16 using Gradient Checkpointing, Flash Attention 2, and AdamW with a cosine learning-rate scheduler.The setup is designed to optimize memory efficiency.
  • Optimization settings: The peak learning rate is 5 × 10−6, with 0.03 warmup, 0.01 weight decay, and a maximum gradient norm of 5.0.The maximum sequence length is 4096 tokens.
  • Batching and sequence length: Per-device batch size is 1, with gradients accumulated over 32 steps to achieve a larger effective batch size.The maximum sequence length is configured to 4096 tokens.

H.2 Two-Level GRPO Phase

The Two-Level GRPO phase aligns agent reasoning with physical execution using hierarchical trajectory and step-level rollouts. It reuses SFT data and applies bounded exploration, generation, and optimization settings.

  • Two-Level GRPO configuration: G = 8 trajectory-level rollouts, M = 4 step-level candidates, and a 24-step maximum rollout length define the Two-Level GRPO setup.The environmental setups and navigation instructions are directly reused from the SFT phase.
  • Exploration and generation: 0.7 sampling temperature and 0.9 top-p encourage diverse trajectory sampling, with prompts capped at 4096 tokens and completions at 512 tokens.The KL divergence penalty coefficient β is set to 0.0.
  • Optimization: 800 training steps use bf16 precision, AdamW with a cosine scheduler, a 2 × 10^-6 peak learning rate, batch size 2, and gradient accumulation of 32.The peak learning rate is reduced for the reinforcement learning phase.
Loading 2608.17512v1…