PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System

Jiazhao Zhang, Gengze Zhou, Hale Yin, Yiyang Huang, Zixing Lei, Qihang Peng, Haoqi Yuan, Jie Zhang, Xudong Guo, Xiaoyue Chen, An Yang, Fei Huang, Zhibo Yang, Junyang Lin, Dayiheng Liu, Jingren Zhou, Zhuoyuan Yu, Jingyang Fan, Zhixuan Liang, Pei Lin, Ye Wang, Haoyang Li, Anzhe Chen, Kun Yan, Xiao Xu, Jiahao Li, Lulu Hu, Minying Zhang, Shurui Li, Wenhu Xiao, Shuai Bai, Xuancheng Ren, Chenxu Lv, Chenfei Wu, Xiong-Hui Chen

arXiv:2606.18112v3cs.ROcs.CV

TL;DR

Existing navigation models lack an externally reconfigurable observation strategy for diverse tasks. Qwen-RobotNav introduces task modes and controllable observation encoding, achieving state-of-the-art results across benchmarks and zero-shot transfer beyond simulation.

  • Problem

    Existing navigation models lack a controllable interface for externally reconfiguring observation strategies across long-horizon, multi-task agentic navigation.

  • Method

    Qwen-RobotNav unifies navigation as waypoint prediction and exposes task modes plus parameterised observation encoding for dynamically selecting behaviours and visual context.

  • Results

    Qwen-RobotNav achieves state-of-the-art results across navigation benchmarks, including 72.1% SR on R2R and 90.0% TR on EVT-Bench.

  • Takeaways & Limitations

    Extensive evaluation and zero-shot transfer to real-world robots indicate that the unified, task-adaptive design generalises beyond simulation.

  • Takeaways & Limitations

    In zero-shot AlpaSim evaluation, Qwen-RobotNav lags behind models designed for that setting, despite non-trivial closed-loop performance.

Abstract

from arXiv · show

Agentic navigation systems require a base navigation model whose observation strategy can be externally reconfigured at inference time, because instruction following, object search, target tracking, and autonomous driving share the same perception-planning backbone yet demand fundamentally different strategies for consuming the visual stream. We present Qwen-RobotNav, a scalable navigation model built on Qwen-RobotNav that addresses it through a parameterised interface with two complementary dimensions: multiple task modes that select the navigation behaviour, and controllable observation parameters (e.g., token budget, per-camera weights) that govern how visual history is encoded. With training-time randomization over all parameters, Qwen-RobotNav is robust to any inference-time configuration requiring zero architectural modification to the Qwen-RobotNav backbone. We train Qwen-RobotNav on 15.6M samples; co-training with vision-language data prevents the collapse into reactive action-sequence mappers observed in trajectory-only training. The parameterised interface also makes Qwen-RobotNav a natural building block for agentic systems: for long-horizon scenarios, an upper-level planner decomposes goals into sub-tasks and dynamically switches Qwen-RobotNav's task mode and context strategy mid-episode, composing complex behaviours from repeated calls to the same model. Extensive experiments show that Qwen-RobotNav sets new state-of-the-art results across major navigation benchmarks. The model exhibits favourable scaling from 2B to 8B parameters, with joint multi-task training developing a shared spatial-planning substrate that transfers across task families, and demonstrates strong zero-shot generalisation to real-world robots across diverse environments.

1 Introduction

Qwen-RobotNav addresses agentic navigation by exposing task modes and controllable observation parameters through a unified waypoint-trajectory interface. Trained with randomized configurations and vision-language data, it supports dynamically composed long-horizon behaviours and achieves strong benchmark performance without task-specific fine-tuning.

  • Motivation: Agentic navigation requires a controllable observation interface so an outer planner can orchestrate capabilities across long-horizon, multi-step goals.This extends beyond multi-task capability to dynamically configurable observation strategies within larger agentic systems.
  • Model: Qwen-RobotNav unifies navigation tasks as waypoint trajectory prediction and exposes task modes plus observation controls for context modelling.Task modes include VLN, PointNav, and ObjNav, while observation controls govern visual context consumption.
  • Interface: Training-time randomization over token budget, temporal decay, and camera weighting enables inference-time context changes without task-specific retraining or tuning.The model is not trained at a fixed setting, reducing overfitting to one observation regime.
  • Agentic deployment: A two-tier system uses an upper-level planner to decompose complex goals and dynamically switch Qwen-RobotNav’s task mode and context strategy during an episode.This composes long-horizon behaviours from repeated calls to the same navigation model.
  • Training: 15.6M samples combine 85% navigation trajectories across five task families with 15% navigation-related vision-language reasoning data.Vision-language co-training preserves language understanding and mitigates degradation associated with trajectory-only training.

2 Navigation Model

Qwen-RobotNav is a navigation model built on Qwen3-VL with a lightweight waypoint-regression head and a unified interface for task-adaptive visual-history encoding. Its parameterized observation strategy allocates visual tokens across cameras and timesteps, enabling deployment-time context switching after randomized training across configurations.

  • Motivation: Navigation requires compressing growing multi-camera histories because visual token counts scale as O(T · N), while tasks differ in their needs for global memory, recency, and visual fidelity.Target tracking favors a short high-resolution recency window, whereas plan verification requires global episode memory.
  • Architecture: Qwen-RobotNav inherits Qwen3-VL and adds a lightweight 4-layer MLP action head that outputs 24 dimensions for K=8 waypoints with (xk, yk, θk).The pretrained backbone provides visual-language reasoning, while the minimal head maps the final hidden state to trajectory waypoints.
  • Parameterized Observation Encoding: The unified interface controls four axes: token budget B, temporal decay γ, per-camera weights wc, and frame sampling mode.The reported ranges include B=2048–4096 and γ∈[1, 3], with random or latest sampling controlling history coverage versus recency.
  • Parameterized Observation Encoding: Token allocation assigns each frame-camera cell a minimum, distributes remaining budget proportionally to temporal and camera weights, and iteratively redistributes surplus above bmax.The feasible allocation requires T′Nbmin ≤B ≤T′Nbmax; camera weights can emphasize the forward view over redundant rear context.
  • Observation Representation: Explicit viewpoint and temporal tags are necessary because otherwise the LLM cannot distinguish camera identity or observation order; descriptive names perform slightly better.These identity signals support spatial-layout and episode-history reasoning, while descriptive names provide richer semantic associations.
  • Training and Generalization: Randomized training across parameter configurations, with equal proportions of random and latest frame sampling, enables zero-shot context-strategy switching without task-specific tuning.This property supports deployment-time switching between global-history coverage and recency-window strategies in the agentic system.

3 Qwen-RobotNav for Agentic Navigation

Qwen-RobotNav provides an agent-ready navigation interface in which an upper-level planner selects task modes and observation configurations for each sub-goal. The system supports long-horizon reasoning by composing repeated calls to the same model and returning compact trajectory evidence, notebook memory, and retrievable visual context.

  • Configurable navigation interface: The planner configures each navigation call with a sub-goal instruction, task mode, and observation configuration, without architectural changes or task-specific fine-tuning.The same model weights serve different task phases; only tool-call arguments change.
  • Agentic composition: Long-horizon behavior is composed by repeatedly switching task modes and observation configurations, such as moving from ObjNav search to Tracking or local PointNav.The upper-level planner decomposes goals into sub-goals and dispatches navigation or auxiliary vision-tool calls.
  • Configurable navigation interface: Qwen-RobotNav supports VLN, PointNav, ObjNav, and Tracking modes for route following, spatial targeting, object search, and target tracking.These modes provide distinct navigation behaviors while sharing the same navigation model.
  • Configurable navigation interface: The planner adapts visual context by changing token budget, recency bias, and frame sampling to match each sub-goal’s history and locality requirements.For example, object search can use broader history, while tracking or local PointNav can emphasize recent observations.
  • Evidence and memory: The harness compresses completed rollouts into trajectory evidence and notebook entries while preserving key-frame retrieval for later visual reinspection.This keeps planner context concise without discarding access to detailed visual evidence.

4 Data

The training data comprises 15.6M samples drawn from navigation trajectories and vision-language sources, with distributions reported by dataset and task category.

  • Data composition: 15.6M training samples are aggregated across navigation trajectory and vision-language sources.The figure reports the total training-set size.
  • Data composition: The dataset distribution reports per-dataset sample counts across all contributing sources.This view covers both navigation trajectory and vision-language datasets.
  • Data composition: The training data is also organized by aggregated task-category distributions.The figure provides a second view that groups samples by task category.

4.1 Navigation Trajectory Planning

Qwen-RobotNav trains on five mutually non-redundant navigation task families spanning language and geometric grounding, exploration, dynamic tracking, and cross-embodiment planning. The 15.6M-sample corpus jointly develops a shared spatial-planning substrate across these capabilities.

  • Training corpus: The training corpus contains 15.6M samples and jointly spans five task families selected to cover complementary perception, planning, and control demands.The families are instruction following, point-goal navigation, object-goal navigation, active target tracking, and autonomous driving.
  • Guided navigation: Instruction following and point-goal navigation provide guided-navigation supervision from rich procedural language to compact directional or coordinate-based commands.Instruction following trains language-to-control grounding, while point-goal navigation isolates geometric planning, obstacle avoidance, and smooth goal approach.
  • Exploration: Object-goal navigation requires implicit spatial mapping and target-location hypotheses without step-by-step instructions.It instantiates the exploration capability dimension.
  • Dynamic interaction: Active target tracking demands real-time motion anticipation and re-identification of moving persons in crowded scenes.This task family introduces dynamic interaction into trajectory planning.
  • Point-goal curriculum: Point-goal training allocates 348K samples to direct approaches, 174K to short-range goals, and 400K to long-range goals, forming a curriculum from grounding to path search.Long-range episodes require route alternatives, navigation around walls and furniture, and multi-room planning over extended horizons.
  • Cross-embodiment planning: Autonomous driving supplies cross-embodiment trajectory supervision in complex, dynamic, and safety-critical traffic environments.Compared with indoor navigation, driving requires planning under higher-speed multi-agent dynamics, traffic rules, diverse road geometries, and stringent safety constraints.

4.2 Autogenerated Navigation Data with a Video Generator

The paper introduces an automated pipeline that converts text-to-video generations into photorealistic navigation trajectories, complementing simulator data without requiring 3-D scene reconstruction. It further improves robustness through task-dependent language, visual, observation, camera, and speed augmentations.

  • Contribution: The resulting T2V-derived dataset provides photorealistic trajectory supervision across diverse real-world-like scenes while reducing the simulator-to-real visual domain gap.It complements simulator-derived data with greater visual diversity and avoids dependence on reconstructed 3-D assets.
  • Data generation: The pipeline generates paired first-person video prompts and navigation instructions across 12 scene categories and 7 interaction complexities.A large language model produces the paired prompt and instruction for each sample.
  • Quality control: A vision-language model filters generated videos for capability-specific quality, including consistency, navigation correctness, goal arrival, motion continuity, and collision avoidance.Target-tracking videos additionally require target visibility, pursuit behaviour, and identity consistency across frames.
  • Trajectory extraction: Retained videos are converted into robot-centric [x, y, yaw] trajectories using monocular depth-and-pose estimation, followed by rule-based kinematic filtering.Filtering removes implausible motion such as negligible displacement, heading jitter, teleportation, abnormal acceleration, and high-frequency noise.

4.3 Vision Language Data

The model augments trajectory supervision with approximately 1.0M general vision-language samples and 873K navigation-specific reasoning samples to strengthen perception, spatial reasoning, and systematic action prediction. It further reformulates discrete VLN trajectories as multi-round, multi-image conversations, preserving decision history for spatial reasoning.

  • General vision-language data: Approximately 1.0M general vision-language samples supplement navigation trajectories across eight complementary categories, including question answering, captioning, grounding, instruction following, and multi-image reasoning.The data include ∼669K visual question answering, ∼6K image captioning, ∼178K visual grounding, ∼30K instruction following, and ∼38K multi-image reasoning and comparison samples.
  • Navigation-specific reasoning: 873K navigation-specific reasoning samples provide free-form question-answering and structured reasoning formats derived from existing VLN trajectories.The reasoning targets navigation decisions and complements general vision-language data with domain-specific spatial understanding.
  • Navigation-specific reasoning: Each navigation sample is decomposed into four independent question–answer pairs covering history comprehension, scene understanding, instruction progress, and action prediction.This decomposition requires reconstructing trajectory context, analysing current observations, and assessing progress before deriving an action.
  • Discrete multi-round navigation data: Approximately 362K discrete multi-round navigation samples reformulate graph-based VLN trajectories as multi-image, multiple-choice action-prediction conversations.Each navigation step becomes a single-step action prediction task with four annotated visual answer options.
  • Discrete multi-round navigation data: Maintaining each discrete trajectory as a complete multi-turn dialogue preserves decision history and enables spatial reasoning without an explicit history encoding module.The formulation adds VLN datasets beyond R2R and RxR, including CVDN.

5 Experiments · 5.1 Deployment · 5.2 Evaluation

Qwen-RobotNav is evaluated across cloud and on-device deployment settings and multiple embodied-navigation benchmarks. It achieves strong results in vision-language navigation, object-goal navigation, and active visual tracking, including competitive performance under constrained visual inputs.

  • 5.1 Deployment: Remote-server deployment achieves 196 ms (5.1 Hz) average end-to-end latency versus 204 ms (4.9 Hz) on-device, but with greater variance from network conditions.On-device inference eliminates network transmission and provides more consistent latency for latency-sensitive tasks such as tracking.
  • 5.2 Evaluation: Qwen-RobotNav is evaluated on VLN-CE, OVON, and EVT-Bench against navigation foundation models NavFoM and ABot-N0 plus specialist methods.The evaluation covers vision-language navigation, open-vocabulary object-goal navigation, and active visual tracking.
  • 5.2.1 Vision-and-Language Navigation: On panoramic VLN-CE, Qwen-RobotNav-8B reaches 76.5% SR and 72.5 nDTW on RxR, surpassing NavFoM by 12.1% SR and ABot-N0 by 7.2% SR.On monocular R2R, Qwen-RobotNav-4B achieves 66.9% SR and 60.5% SPL, exceeding DualVLN by 2.6% SR and 2.0% SPL.
  • 5.2.1 Vision-and-Language Navigation: On VLN-PE with the flash controller, Qwen-RobotNav-8B achieves 65.50% SR and 61.19% SPL, surpassing InternVLA-N1 by 5.1% SR and 6.3% SPL.It also attains the lowest navigation error (3.73 m) and highest oracle success rate (72.99%) among all methods.
  • 5.2.2 Object-Goal Navigation: For closed-vocabulary ObjectNav, Qwen-RobotNav uses RGB observations alone, while most prior methods rely on depth sensors or odometry and report HM3D v1 results.Qwen-RobotNav is evaluated on the harder HM3D v2 benchmark, limiting direct comparability with published baselines.
  • 5.2.2 Object-Goal Navigation: On HM3D-OVON, Qwen-RobotNav-4B achieves 57.7% / 60.1% / 53.1% SR on Seen / Synonyms / Unseen, using a single forward-facing camera.It ranks first on two splits and second on Unseen behind ABot-N0’s 54.0%.
  • 5.2.3 Active Visual Tracking: On EVT-Bench, Qwen-RobotNav achieves 90.0% TR for 4B and 89.7% TR for 8B, exceeding ABot-N0 by 2.4%, NavFoM by 9.5%, and TrackVLA++ by 9.0%.Qwen-RobotNav-8B also records 5.70% CR, the lowest collision rate among generalist models.

5.3 Embodied Question Answering

Qwen-RobotNav-based variants outperform prior methods across HM-EQA, MT-HM3D, and EXPRESS-Bench. Qwen3.6-Plus+Qwen-RobotNav achieves the best overall performance on all three benchmarks.

  • Embodied Question Answering: Qwen-RobotNav-based variants outperform prior methods across HM-EQA, MT-HM3D, and EXPRESS-Bench.Table 7 compares these results with prior state-of-the-art methods.
  • Embodied Question Answering: 76.7 SR on HM-EQA, 54.4 SR on MT-HM3D, and 79.27 LLM Score on EXPRESS-Bench are achieved by Qwen3.6-Plus+Qwen-RobotNav.This variant delivers the best overall performance across the three benchmarks.
  • Embodied Question Answering: Qwen3.6-Plus+Qwen-RobotNav consistently surpasses recent strong baselines such as FAST-EQA and Memory-EQA.The passage specifically identifies these methods as recent strong baselines.

5.4 Autonomous Driving

Qwen-RobotNav achieves strong closed-loop autonomous-driving performance on NAVSIM and demonstrates temporally coherent multi-view trajectory planning. In zero-shot AlpaSim evaluation, it tests out-of-domain transfer and long-horizon driving robustness without AlpaSim-specific training or adaptation.

  • NAVSIM: 91.4 PDMS: Qwen-RobotNav-4B outperforms NavFoM by 7.1 points and AutoVLA by 2.3 points on NAVSIM navtest.Evaluation conditions the model on ground-truth trajectories from the previous three frames as historical priors.
  • AlpaSim: AlpaSim evaluates Qwen-RobotNav on 920 PhysicalAI-AV NuRec scenarios with at-fault closed-loop metrics, counting close encounters only when the ego vehicle is responsible.The benchmark examines closed-loop safety and long-horizon driving robustness beyond NAVSIM.
  • NAVSIM: Qwen-RobotNav produces temporally consistent curved trajectories that follow the road curvature and progressively complete a left turn under multi-view observations.The model uses ego-state and historical trajectory information, adapting its path across timesteps as scene context changes.
  • AlpaSim: Zero-shot AlpaSim rollout measures out-of-domain transfer because Qwen-RobotNav receives neither AlpaSim-specific training nor closed-loop adaptation, though it lags behind Alpamayo-R1 models.The evaluation directly rolls out the model in the simulator on PhysicalAI-AV NuRec scenarios.
  • AlpaSim: In qualitative AlpaSim cases, Qwen-RobotNav cautiously handles intersections, executes turns while avoiding road boundaries, and proceeds along outgoing lanes.A second scenario shows the vehicle responding correctly to a green traffic light by starting from a stopped state and proceeding through the intersection.

5.5 Ablation Study

The ablation study shows that Qwen-RobotNav benefits from broader navigation training data and controllable observation settings, including visual token budget and temporal decay. Larger data fractions improve representative navigation benchmarks, while token-budget increases substantially improve performance before benefits become non-monotonic.

  • Effect of training data scale: Increasing navigation training data from 12.5% to 100% produces clear overall gains across representative benchmarks.The strongest improvements occur on instruction-following and driving tasks.
  • Effect of training data scale: Broader trajectory coverage particularly improves long-horizon instruction following, including VLN-CE RxR.The passage attributes this improvement to better grounding between language and navigation trajectories.
  • Effect of token budget and temporal decay: Increasing the visual token budget substantially improves performance over low-budget settings, although gains are not strictly monotonic at higher budgets.The sweep evaluates Qwen-RobotNav-4B on 500 VLN-CE R2R Val-Unseen episodes using SR, SPL, nDTW, and OSR.

5.6 Real-World Deployment Results

Qwen-RobotNav demonstrates real-world deployment on a quadruped robot in unseen, heterogeneous environments, following pure language instructions and fine-grained motion commands. The episodes show landmark-grounded navigation, precise spatial control, reverse traversal, and support for long-horizon agentic execution.

  • Exhibition hall deployment: 21.78 m: The robot navigates from a living room to a medical room in an unseen exhibition hall using pure language instructions without goal images or coordinates.It grounds instructions through landmarks including furniture, doorways, and signage while traversing visually distinct zones.
  • Exhibition hall deployment: Reverse language command: The model switches locomotion mode and walks backward along the route to a position closely matching its initial starting pose.This demonstrates execution of diverse motion primitives and accurate reverse trajectory control.
  • Apartment deployment: Natural-language control: In an apartment, the robot stops at a specified nightstand, performs a requested detour before exiting, and follows fine-grained spatial directives.The commands govern precise stopping locations and route choices across bedroom and living-room areas.
  • Agentic execution: Agentic division of labor: An upper-level agent decomposes open-ended requests into sub-goals, while Qwen-RobotNav executes grounded navigation segments and returns trajectory evidence for subsequent planning.The same interface supports long-horizon navigation, landmark-based reasoning, memory updates, and evidence-grounded response generation outside simulation.

6 Conclusion

Qwen-RobotNav reframes multi-task navigation as externally controllable context modeling over a shared perception-planning backbone. Its reconfigurable observation interface supports compositional long-horizon behaviours and generalises across navigation tasks and real-world robots.

  • Core design: Qwen-RobotNav treats multi-task navigation as a context-modeling problem, not an architecture or task-head problem.Instruction following, target tracking, and autonomous driving share a perception-planning backbone but consume observations differently.
  • Core design: Its parameterised observation interface lets an outer agent or operator select context strategies per sub-task at inference time.Training-time randomisation makes the model robust to any configuration without retraining.
  • Implications: The same model switches between global-history exploration and recency-focused local manoeuvring, enabling complex long-horizon behaviours through mode composition.This reframes navigation from a fixed policy into a reconfigurable navigation primitive.
  • Validation: Experiments across navigation, tracking, autonomous driving, and embodied question answering validate the design, while zero-shot transfer confirms generalisation to real-world robots.The work positions task-adaptive observation encoding as a promising direction for broadly capable navigation foundation models.

7 Contributions and Acknowledgments

The paper credits its core contributors and additional contributors, and acknowledges institutional and hardware support for real-robot experiments.

  • Contributions: Core contributors include Jiazhao Zhang, Gengze Zhou, Hale Yin, Yiyang Huang, Zixing Lei, Qihang Peng, and additional collaborators.The passage lists the full core-contributor group and marks several authors with contribution symbols.
  • Contributions: Additional contributors include Zhuoyuan Yu, Jingyang Fan, Zhixuan Liang, Pei Lin, Ye Wang, and other collaborators.The passage provides the complete list of additional contributors.
  • Acknowledgments: The National Pilot Base for Embodied Intelligence Applications provided the real-robot experimental environment and equipment.The acknowledgment specifically credits the base with enabling real-robot experiments.
  • Acknowledgments: Unitree provided a Unitree Go 2 for the experiments.The acknowledgment identifies the contributed robot hardware.
Loading 2606.18112v3…