PaperlayerView exact revision on arXiv

Source-linked AI summary

ABot-N1: Toward a General Visual Language Navigation Foundation Model

Ruiyan Gong, Yingnan Guo, Junjun Hu, Jintao Kong, Xiaoxu Leng, Tianlun Li, Weize Li, Fei Liu, Zhicheng Liu, Jia Lu, Minghua Luo, Chenlin Ming, Yanfen Shen, Jiyue Tao, Zhengbo Wang, Mingyang Yin, Minqi Gu, Zihao Guan, Wei Guo, Guoqing Liu, Huachong Pang, Menglin Yang, Zeqian Ye, Xiaoxiao Geng, Zhining Gu, Honglin Han, Di Jing, Hongyu Pan, Mingchao Sun, Kuan Yang, Jianfang Zhang, Yanghong Chen, Ye He, Wei Mei, Jiahao Shi, Xiangpo Yang, Yanqing Zhu, Yang Cai, Jingjing Ma, Shihui Su, Zixiao Tang, Linbo Zheng, Zedong Chu, Xiaolong Wu, Wenbin Tang, Mu Xu

arXiv:2607.10383v3cs.CVcs.AIcs.RO

TL;DR

Visual Language Navigation models must support diverse embodied tasks while combining grounded reasoning, control, and interpretability. ABot-N1 decouples cognition from control with a slow vision-language reasoner that produces linguistic traces and pixel goals, plus a fast action expert, achieving state-of-the-art results across five tasks, including 77.3% POI-goal success.

  • Problem

    Existing navigation abilities are often pursued in isolation with task-specific architectures, bespoke goal interfaces, and narrow training data, limiting versatile open-world navigation.

  • Method

    ABot-N1 uses a slow–fast dual-system architecture in which a vision-language reasoner produces Chain-of-Thought traces and pixel goals that guide a fast action expert across five navigation tasks.

  • Results

    ABot-N1 achieves state-of-the-art performance across five tasks, improving every primary metric over ABot-N0 and reaching 77.3% POI-goal success.

  • Takeaways & Limitations

    The unified pixel-goal interface supports positive cross-task transfer and robust navigation from a single multi-task checkpoint without task-specific modifications.

  • Takeaways & Limitations

    The reported benchmark protocols use task-specific collision budgets, allowing fewer than three outdoor collisions but no indoor collisions for success.

Abstract

from arXiv · show

Visual Language Navigation foundation models aim to unify deep reasoning for grounded spatial decisions with broad versatility for diverse embodied tasks. Current approaches typically achieve this integration via monolithic policies that map observations directly to actions, yet they often suffer from coordinate drift and poor handling of long-tail semantics. Furthermore, these black-box mappings lack interpretability, hindering the simultaneous achievement of generality, robustness, and transparency. We present ABot-N1, a step toward a general Visual Language Navigation foundation model, that addresses these challenges by decoupling cognition from control via a slow-fast architecture guided by dual visual-language signals. More specifically, a slow vision-language reasoner performs explicit Chain-of-Thought reasoning while producing a pixel goal. This compact set of image-space anchor points serves as a universal interface for diverse tasks, including point-goal, object-goal, poi-goal, instruction-following, and person-following. Subsequently, a fast action expert leverages both the textual cues and the pixel guidance to generate continuous waypoints at the native control frequency. By bridging high-level intents and low-level control through pixel-grounded anchors paired with explicit linguistic traces, our approach ensures robust, generalizable, and interpretable navigation across simulation and real-world benchmarks. ABot-N1 establishes new state-of-the-art records, delivering massive gains specifically in urban-scale navigation: boosting POI arrival by 35.0% (to 77.3%) and achieving 95.4%/92.9% SR in complex indoor and outdoor scenes. It also maintains superior robustness across object-reaching, person-following, and instruction-following tasks. New Point-Goal/POI-Goal benchmarks are released as open source to advance the field of urban-scale navigation.

1 Introduction

ABot-N1 addresses the limitations of monolithic visual-language navigation policies with a hierarchical slow–fast architecture that separates deliberative reasoning from fast control. It unifies five navigation tasks through Chain-of-Thought reasoning and pixel-goal anchors while introducing cityscale benchmarks for evaluation.

  • Motivation: General-purpose navigation foundation models improve cross-task transfer but still face coordinate misalignment, semantic brittleness, and poor interpretability and safety.These challenges affect point navigation, target navigation, and black-box decision tracing during real-world deployment.
  • Architecture: ABot-N1 uses a hierarchical slow–fast architecture in which a 4B-parameter multimodal VLM performs deliberative reasoning and a fast controller executes actions.The slow subsystem processes instructions, visual memory, and observations to produce a Chain-of-Thought rationale and spatial guidance.
  • Architecture: A shared language-plus-vision interface decomposes metric coordinates, instructions, object categories, POI names, and target persons into CoT-explained pixels for consistent control.The CoT supplies semantic reasoning, while the pixel goal provides actionable egocentric visual anchors across tasks.
  • Design Benefits: ABot-N1 addresses coordinate errors through image-space re-grounding and improves long-tail target handling by separating semantic recognition from reactive control.Human-readable CoT and pixel anchors also expose whether failures arise from semantic reasoning or spatial targeting.
  • Contributions: The authors release ABotN-PointBench and ABotN-POIBench to evaluate cityscale point-goal and POI-goal navigation in complex indoor, outdoor, and commercial environments.The benchmarks address the scarcity of open-source cityscale evaluation frameworks and support comprehensive assessment of navigation methods.
  • Contributions: ABot-N1 integrates pixel-grounded supervision, CoT trajectories, and GRPO-style post-training across point-goal, object-goal, POI-goal, instruction-following, and person-following tasks.The training recipe aligns the slow system’s reasoning with downstream navigation rewards.

2 Related Works

Related work progresses from task-specific navigation specialists toward unified foundation models, brain–body architectures, and embodied reasoning that combines linguistic and visual grounding. ABot-N1 builds on these trends by coupling explicit Chain-of-Thought with structured pixel goals for a general navigation interface.

  • General-purpose navigation: Prior VLN specialists target individual instruction-following, object-goal, point-goal, or person-following benchmarks and use incompatible goal interfaces that resist transfer and joint training.These task-specific architectures specialize either the goal interface or policy backbone for a single benchmark.
  • General-purpose navigation: Recent generalists consolidate multiple navigation tasks under video-conditioned VLM policies and larger foundation models, including NavFoM, ABot-N0, and training-free OpenFrontier.ABot-N1 inherits the commitment to a single, multi-task architecture while departing from prior generalists.
  • Brain–body decoupling: Brain–body navigation architectures separate slow multimodal deliberation from lightweight, high-frequency motor control to accommodate their different time constants and uncertainty profiles.This division follows a dual-process framing in which a slow deliberative System 2 is paired with fast execution.
  • Brain–body decoupling: ABot-N1 replaces opaque latent or free-form textual interfaces with human-readable Chain-of-Thought plus structured pixel goals, enabling imitation-friendly reasoning-level learning.Its slow system is post-trained with GRPO, while the pixel-grounded bottleneck connects deliberation to control.
  • Linguistic and visual reasoning: Embodied reasoning combines explicit linguistic CoT with image-space grounding, and ABot-N1 unifies both by producing CoT that resolves into pixel goals for a fast controller.The approach extends visual prompting beyond user-drawn marks and text-only sub-goals toward a shared navigator that can support many physical tasks.

3 Preliminaries

The preliminaries formulate versatile embodied navigation as a unified, map-free, goal-conditioned sequential decision problem spanning five task types. They also motivate factorizing a pretrained vision-language reasoner from a lightweight action policy to transfer broad visual and linguistic priors into navigation.

  • Unified Problem Formulation: Embodied navigation is cast as a unified conditional sequential decision problem covering five heterogeneous user goals from visual observations to executable motion commands.The formulation is intended to support precise metric and abstract semantic goals within one framework.
  • Unified Problem Formulation: At each step, the navigator receives multi-camera ego-centric RGB observations, optional proprioception or odometry, and a task specification encoding user intent.The visual input is represented as It ∈ R^(N×H×W×3), with optional qt and goal condition g.
  • Unified Problem Formulation: The policy predicts H future low-level commands as continuous SE(2) waypoints with completion flags, replanning in closed loop until arrival or a step-budget limit.An episode succeeds when the final pose lies within a task-specific tolerance of the true target.
  • Goal Specification: A single natural-language instruction interface represents point-goal, instruction-following, object-goal, POI-goal, and person-following tasks without per-task heads or bespoke encoders.The instruction can encode metric offsets, route descriptions, object categories, points of interest, or person attributes and identities.
  • Policy Factorization: The navigator is factorized into a pretrained VLM reasoner and lightweight action policy, transferring web-scale linguistic and visual priors for recognition, spatial understanding, and instruction parsing.The intermediate guidance signal may comprise latent features, decoded language, visual anchors, or combinations thereof.

4 Methods · 4.1 Model Architecture

ABot-N1 separates deliberative reasoning from reactive control through asynchronously coupled slow and fast systems sharing a visual front-end. The slow system emits linguistic reasoning and pixel goals, while the fast system converts these signals into continuous waypoint actions at high frequency.

  • 4.1 Model Architecture: ABot-N1 decomposes navigation into asynchronously coupled slow and fast sub-networks that share visual inputs but differ in capacity, frequency, and responsibility.The slow system reasons deliberatively, whereas the fast system executes reactive control.
  • 4.1 Model Architecture: The slow system uses Qwen-3.5-4B to process trajectory memory, tri-view reference observations, task specifications, and previous decisions.Previous-decision feedback supports temporal consistency in instruction-following and person-following by reducing contradictory or oscillating behavior.
  • 4.1 Model Architecture: The slow system outputs an explicit natural-language Chain-of-Thought trace and pixel goals projected onto current tri-view images.Chain-of-Thought is used for instruction-following and object-goal, while point-goal, POI-goal, and person-following rely on pixel goals alone.
  • 4.1 Model Architecture: Affordance pixels mark safe traversable waypoints, while target pixels mark visible final goals such as objects, POI entrances, or tracked people.Affordance pixels are typically about ∼3 m ahead indoors or ∼5 m outdoors, and target pixels appear near final approach.
  • 4.1 Model Architecture: Task-specific pixel usage assigns point-goal only an affordance pixel, whereas instruction-following adds a target pixel near the route endpoint.For point-goal, the metric coordinate goal is consumed internally by the slow system rather than emitted as an image-space anchor.
  • 4.1 Model Architecture: The fast system uses Qwen-3.5-2B to fuse current memory and observations with the slow system’s CoT, pixel goals, reference observation, and task specification.A QFormer-style cross-attention module distills navigation features before an MLP decodes the action output.
  • 4.1 Model Architecture: The action expert predicts H=5 continuous SE(2) actions containing position, heading encoded as (sin θ_i, cos θ_i), and completion flags.Adaptive step sizes increase outdoors and decrease indoors to balance efficiency and obstacle clearance.
  • 4.1 Model Architecture: Asynchronous inference lets the fast system track cached pixel goals between slow updates, preserving responsive control without stalling for deliberative reasoning.Visual closed-loop tracking compensates for reasoning latency while the slow system operates at a lower frequency.

4.2 Pretraining

ABot-N1 pretrains its slow reasoner and fast controller through supervised imitation on a heterogeneous, five-task corpus spanning simulation, outdoor trajectories, and real-robot teleoperation. The data engine provides 30 M samples with distinct high- and low-level supervision, while training combines language/pixel prediction with waypoint, heading, and arrival objectives.

  • Corpus and composition: Both systems are pretrained by supervised imitation on heterogeneous data spanning photorealistic indoor simulation, outdoor driving and street-view trajectories, and real-robot teleoperation.The corpus is designed to cover the diversity of real deployment across all five navigation tasks.
  • Training objectives: The slow system is trained with cross-entropy over CoT tokens and pixel coordinates, while the fast system uses smooth-L1 losses for position and heading angle plus a binary arriving loss.The passage notes that smooth-L1 can induce mode collapse for multimodal navigation targets by favoring conditional means.
  • Corpus and composition: 30 M pre-training samples cover point-goal, object-goal, POI-goal, instruction-following, and person-following, split into 13.3 M high-level samples for the slow system and 16.4 M low-level samples for the fast system.The fast-system inputs use task-specific combinations of CoT, affordance pixels, and target pixels.
  • Data engine: The data engine uses 800+ indoor and 22 outdoor high-fidelity 3D Gaussian Splatting scenes to generate complementary expert and synthesized rollouts for pretraining.The scene bank supports trajectory generation across the training pipeline.
  • Training objectives: Slow-system CoT supervision tracks completed and current instruction sub-instructions, while affordance and target pixels separate long-horizon path tracking from terminal grounding.This supervision follows the ABot-N0 schema and is combined with unified pixel annotations.

4.3 Post-Training

Post-training uses GRPO to optimize episode-level navigation outcomes with group-relative rewards over valid textual and pixel outputs. A composite reward and safety-aware sampling strategy jointly preserve parseability, target alignment, and stable policy improvement.

  • GRPO Optimization: GRPO samples candidate CoT-plus-pixel outputs, scores them with a composite reward, and optimizes group-relative advantages against the previous policy.The composite reward combines format, target-alignment, and safety-clearance terms.
  • GRPO Optimization: The KL anchor constrains exploration because unparseable outputs receive zero downstream reward and can collapse the gradient.This protects the parseable output manifold during joint reasoning-and-pixel exploration.
  • Composite Reward: The reward gates target alignment and safety on valid JSON formatting, penalizes missing or distant target points, and back-projects predictions to assess worst-view clearance.Target rewards use cross-view matching and exponential L2-distance decay, while safety clearance accounts for obstacles, illegal zones, and traffic-rule violations.
  • Safety-Aware Sampling: Training samples are stratified into Safe, Critical, and Danger zones to balance gradient stability, variance control, and safety enforcement.Safe samples provide high-GSNR stabilization, whereas Danger samples provide strong hard-safety gradient directions and therefore receive the smallest share.
  • Safety-Aware Sampling: 5 : 3 : 2 is the adopted sampling ratio for Safe, Critical, and Danger zones, respectively.The proportions are Psafe = 0.5, Pcritical = 0.3, and Pdanger = 0.2.

5 Benchmark

The benchmark suite targets closed-loop, social-rule-aware navigation across heterogeneous environments and entrance-level arrival at named points of interest. It introduces ABotN-PointBench and ABotN-POIBench, built on a shared high-fidelity 3DGS reconstruction pipeline.

  • Benchmark Suite: ABotN-PointBench and ABotN-POIBench address missing evaluations for coordinate-conditioned and name-conditioned navigation, respectively.Both benchmarks share the same high-fidelity 3DGS reconstruction stack and unified scene construction pipeline.
  • ABotN-PointBench: 31 real-world scenes—16 indoor and 15 outdoor—cover shopping malls, parks, road intersections, and daily commute corridors with fine-grained walkability annotations.Each scene is reconstructed as a high-fidelity 3DGS environment, distinguishing legally traversable from violating regions.
  • ABotN-PointBench: 465 distinct reference trajectories are generated with A∗ on annotated occupancy maps, while targets span indoor 3–20 m and outdoor 5–20, 20–35, and 35–50 m ranges.These trajectories support efficiency metrics such as SPL and stratify short-horizon precision against long-horizon endurance.
  • ABotN-PointBench: Outdoor episodes use SR<3col, counting success only when arrival occurs with fewer than three collision events to tolerate minor occupancy-boundary deviations.Outdoor tiers are Low (5–20 m), Medium (20–35 m), and High (35–50 m), with 75 episodes per tier.
  • ABotN-POIBench: ABotN-POIBench is presented as the first closed-loop, high-fidelity benchmark for real-world POI-goal navigation, evaluating arrival at a named POI’s physical entrance.It addresses protocols that measure only single-step waypoints or coarse block- or graph-node-level arrival.
  • ABotN-POIBench: 163 distinct POIs across 11 commercial regions are annotated with names and physical entrance frames, and success is measured at a 2 m entrance threshold as SR<2m.The benchmark covers ∼126,398 m2, >38 M Gaussian points, ∼2.81 M mesh vertices, and ∼4.40 M mesh faces.

6 Experiments · 6.1 Simulation Evaluation

ABot-N1 is evaluated as a jointly trained five-task model under established or newly introduced closed-loop simulation protocols. It achieves state-of-the-art or leading performance across instruction-following, object-goal, point-goal, POI-goal, and person-following tasks, while demonstrating positive cross-task transfer.

  • 6.1 Simulation Evaluation: Simulation evaluates ABot-N1 across five core tasks using standard literature protocols or closed-loop benchmarks introduced for point-goal, POI-goal, and object-goal navigation.Unless specified, ABot-N1 denotes the joint multi-task checkpoint co-trained on all five tasks.
  • 6.1.1 Instruction-Following: VLN-CE R2R/RxR: ABot-N1 achieves R2R-CE Val-Unseen NE of 3.32 m, SR of 70.89%, and SPL of 67.5%, while using only front-left-right tri-view RGB.It sets a new state of the art on these metrics and achieves the best NE on RxR-CE, without depth or odometry.
  • 6.1.2 Object-Goal: Short-Horizon OVON: ABot-N1 raises object-goal SR by +11.7 points and SPL by +16.4 points over ABot-N0, while reducing DTG from 1.44 m to 0.82 m.The single-task ABot-N1† reaches SR 85.5 and DTG 0.59 m, whereas joint ABot-N1 exceeds it on SPL, 51.8 vs. 50.8.
  • 6.1.3 Point-Goal: ABotN-PointBench: 92.9% overall SR is achieved by ABot-N1 on the outdoor PointBench split, improving over ABot-N0 by +16.0 SR with strong social compliance.On the hardest tier, SR is 88.0 vs. 65.3, a +22.7-point jump, under the redundant collision criterion SR<3col.
  • 6.1.3 Point-Goal: ABotN-PointBench: 95.4% SR is achieved by ABot-N1 on the indoor PointBench split under the strict zero-collision criterion SR<1col, with the highest SPL across difficulty levels.The outdoor evaluation uses SR<3col, while indoor evaluation requires zero collisions.
  • 6.1.4 POI-Goal: ABotN-POIBench: 77.3% SR<2m is achieved by ABot-N1 on ABotN-POIBench, improving from 42.3% for POINav by +35.0 points, while SPL rises from 40.3% to 72.6%.Success requires termination within a 2 m radius of the annotated entrance frame.
  • 6.1.5 Person-Following: EVT-Bench: 90.1 SR / 89.8 TR is achieved on easy STT person-following, while hardest AT reaches 70.0 SR / 87.8 TR.These results improve over ABot-N0 by +3.2 SR / +2.2 TR on STT and +2.7 SR / +8.3 TR on AT; ABot-N1 leads on all SR metrics and nearly all TR metrics.
  • 6.1.6 Cross-Task Summary: +3.9 SR on R2R-CE, +16.0 SR on outdoor PointBench, +0.558 SR<2m on POIBench, and +3.2 STT-SR / +2.7 AT-SR on EVT-Bench summarize ABot-N1’s gains over ABot-N0.Across tasks, ABot-N1 matches or exceeds ABot-N1† except for a marginal object-goal gap, supporting positive transfer from the unified pixel-goal interface.

6.2 Real-World Deployment

ABot-N1 was deployed fully onboard an AMap TuTu quadrupedal robot, using integrated perception, upgraded compute, and asynchronous slow-fast inference for real-time control. Qualitative episodes across five tasks demonstrate pixel-grounded reasoning, safe navigation, recognition, instruction execution, and person-following.

  • Platform and deployment: ABot-N1 was deployed on the proprietary AMap TuTu quadrupedal robot for extensive experiments across diverse indoor and outdoor environments.The platform uses 12-DOF actuation, omnidirectional LiDAR, RTK-GNSS, and a Tri-Camera array with 270◦ horizontal FOV.
  • Platform and deployment: The unified onboard framework integrates BEV mapping, legged odometry, obstacle filtering, VLN inference, and locomotion control without offloading.Upgrading from Jetson Orin NX to Jetson AGX Orin 64 GB provides 4× memory expansion and increased GPU throughput for concurrent inference and perception.
  • Runtime architecture: 10 Hz closed-loop control is maintained by asynchronous slow-fast execution, with the fast system continuously using cached slow-system plans and current observations.The slow system uses Qwen-3.5-2B, while the fast system uses a 306 M-parameter DiT with DINOv2-Base visual encoding.
  • Point-Goal Navigation: Point-goal episodes demonstrate traversability-aware obstacle avoidance, construction detours, fork selection, and traffic-light-compliant crossing through pixel guidance and temporal action gating.The system waits for a green signal before crossing, enabling safe, rule-compliant navigation beyond metric-only point-goal policies.
  • Object-Goal and POI-Goal Navigation: Object-goal and POI-goal episodes show long-range recognition, spatial-relation reasoning, obstacle avoidance, accessibility reasoning, and decoupled recognition-and-approach using target and affordance pixels.Examples include shaded outdoor benches, chairs distinguished by water bottles, partially occluded fire extinguishers, storefront signage, slopes, and impassable stairs.
  • Instruction-Following and Person-Following: Instruction-following and person-following episodes demonstrate instruction-pointer tracking, landmark grounding, identity persistence, reactive control, elevation handling, and collision-aware tracking.The system follows commands through stairs, gym transitions, and bar-counter approach while tracking designated people through distractors, stairs, corners, and temporary occlusion.

7 Conclusion

ABot-N1 is a general visual-language navigation foundation model that separates high-level cognition from low-level control through a fast–slow architecture. Its dual visual-language guidance provides a universal action interface for five navigation tasks from one multi-task checkpoint.

  • Architecture: ABot-N1 factorizes cognition from control with a slow vision-language reasoner and a fast action expert.The reasoner produces Chain-of-Thought traces and pixel-level grounding, while the action expert converts them into reactive waypoints.
  • Universal interface: Dual visual-language guidance through affordance and target pixels serves as a universal action interface.This interface bridges the reasoner’s high-level outputs and the action expert’s low-level waypoint generation.
  • Task coverage: Five navigation tasks are accommodated from a single multi-task checkpoint: point-goal, instruction-following, object-goal, POI-goal, and person-following.The conclusion identifies these tasks as the scope enabled by the universal action interface.

8 Contributions

The contributions section lists the paper’s contributors, comprising two groups of authors.

  • Contributors: The first listed contributor group includes Ruiyan Gong, Yingnan Guo, Junjun Hu, Jintao Kong, Xiaoxu Leng, Tianlun Li, Weize Li, Fei Liu, and Zhicheng Liu.The group also includes Jia Lu, Minghua Luo, Chenlin Ming, Yanfen Shen, Jiyue Tao, Zhengbo Wang, and Mingyang Yin.
  • Contributors: The second listed contributor group includes Minqi Gu, Zihao Guan, Wei Guo, Guoqing Liu, Huachong Pang, Menglin Yang, and Zeqian Ye.

Data

The Data section lists contributors and acknowledges broader team support, including Wenbin Tang, Zhining Gu, Shihui Su, Zixiao Tang, Yang Cai, Linbo Zheng, and Jingjing Ma.

  • The listed contributors include Xiaoxiao Geng, Zhining Gu, Honglin Han, Di Jing, Hongyu Pan, Mingchao Sun, Kuan Yang, and Jianfang Zhang.
  • Additional listed contributors are Yanghong Chen, Ye He, Wei Mei, Jiahao Shi, Xiangpo Yang, and Yanqing Zhu.
  • The broader team is thanked for its support, particularly Wenbin Tang, Zhining Gu, Shihui Su, Zixiao Tang, Yang Cai, Linbo Zheng, and Jingjing Ma.
Loading 2607.10383v3…