PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Generalist Navigation Policies via Action Self-Distillation

Bowen Feng, Keqin Wang, Yulong Yang, Amogh Joshi, Christine Allen-Blanchette, Dhruv Shah, Felix Heide

arXiv:2610.17593v1cs.ROcs.LG

TL;DR

Training language-conditioned navigation commonly requires costly language and reasoning annotations, despite many robot datasets containing only trajectories. ASD uses privileged trajectory feedback to generate goals and reasoning for a student trained from deployable observations, and reports the highest overall success across 13 real-world tasks, including a 30% indoor improvement over OmniVLA.

  • Problem

    Language-conditioned navigation training typically requires costly goal, scene-semantic, or reasoning annotations, while many robot navigation datasets contain only observations and trajectories.

    Source: §1 · p. 1

  • Method

    ASD uses trajectory feedback to let a self-teacher infer language goals and reasoning, then trains a student to predict trajectories and reasoning from observations alone.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across 13 real-world tasks, ASD achieves the highest overall success rate and outperforms OmniVLA by 30% in indoor settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ASD demonstrates zero-shot navigation across out-of-distribution language goals, including directional, color-based, and object-centric commands, as well as multi-stage and unseen indoor tasks.

    Source: §1 · p. 2

    ASD still depends on demonstration trajectories, and its self-generated goals and reasoning may degrade in ambiguous scenes, with small targets, or under fine-grained spatial instructions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language provides a natural interface for humans to specify and reason about goals, but training language-conditioned robotic policies and reasoning typically requires expensive manual annotations. We propose Action Self-Distillation (ASD), a method for learning language-conditioned reasoning for robotic navigation without human-annotated language or reasoning labels. ASD uses an open-source VLM, which contains broad world knowledge and reasoning capabilities, as both a teacher network and a student policy. During training, the self-teacher receives privileged trajectory feedback and infers the underlying navigation goal together with goal-oriented reasoning. The student is then trained to condition on the inferred navigation goal, predict future ego-frame trajectory, and provide a description of the scene semantics, plan, and reasoning, from onboard observations alone. Across 13 challenging indoor and outdoor real-world navigation tasks, our method improves success rate over the strongest baseline by $30\%$ in indoor settings, $30\%$ on multi-stage tasks, and achieves $95\%$ success rate on outdoor tasks, validating that privileged trajectory feedback can enable self-distillation for scalable generalist navigation policies without human annotations.

1 Introduction

Language-conditioned navigation is useful, but training it typically depends on costly language and reasoning annotations that many trajectory-only datasets lack. ASD uses privileged trajectory feedback to generate goals and reasoning for a student policy, and achieves strong results across 13 real-world tasks.

1 Introduction

  • Training language-conditioned navigation typically requires costly goal, scene-semantic, or reasoning annotations, while many navigation datasets contain only observations and trajectories.The research question is whether a single pretrained VLM can supervise itself from trajectory-only data without language labels or multi-model annotation pipelines.Source: §1 · p. 1
  • ASD trains a self-teacher on privileged trajectory feedback to infer goals and generate reasoning, then distills those outputs into a student using deployable observations alone.The student learns from raw image-trajectory pairs without human-provided language goals or reasoning annotations.Source: §1 · p. 2, Figure 1 · p. 2
  • Across 13 real-world tasks, ASD achieves the highest overall success rate, outperforming OmniVLA by 30% indoors and NaVILA by 30% on multi-stage tasks, while reaching 95% outdoors.The comparison covers zero-shot natural-language goals in indoor and outdoor settings.Source: §1 · p. 2

2 Related Work

Prior navigation work established visual-goal policies and language-based instruction following, but annotation pipelines and external-model dependencies remain relevant distinctions. ASD uses self-distillation to elicit language goals and contextual descriptions within training.

Learning-based Ground Navigation

  • Large-scale learned navigation policies such as GNM, ViNT, and NoMaD use visual-goal conditioning rather than natural-language instructions.NoMaD combines goal-directed navigation and exploration through diffusion policies and goal masking.Source: §2 · p. 3

Natural Language Goals and Reasoning in Navigation

  • Language-navigation systems range from model-composition approaches to real-robot policies, while OmniVLA and LeLaN rely on external labeling pipelines for training annotations.The passage describes LM-Nav, NavGPT, NavGPT-2, NaVILA, and VAMOS as examples of language-based navigation approaches.Source: §2 · p. 3

Reasoning Distillation and Self-Distillation

  • ASD uses the same open-source VLM as teacher and student to elicit language goals and contextual descriptions through trajectory feedback, unlike approaches relying on separate external teacher models.The related-work discussion contrasts ASD with action-pseudo-labeling approaches and embodied reasoning work using external teachers.Source: §2 · p. 3

3 Training Generalist Navigation Policies via Action Self-Distillation

ASD trains a language-conditioned navigation policy from trajectory-only robot data by generating goals and reasoning with a privileged-feedback self-teacher. At deployment, the student uses observations and operator-supplied language goals without privileged trajectory input.

3 Training Generalist Navigation Policies via Action Self-Distillation

  • ASD is presented as a method for navigation policies that follow natural-language goals while reasoning about their actions.Source: §3 · p. 3

3.1 Problem Formulation

  • The policy predicts future trajectories from ego-centric camera observations and state history, with trajectories serialized as tokens for a pretrained VLM.The training datasets contain no reasoning annotations or goal labels; language supervision is generated by ASD.Source: §3.1 · p. 3

3.2 Training: Language Following and Reasoning via Action Self-Distillation

  • The self-teacher uses ground-truth trajectory feedback to generate a semantic goal and reasoning trace, while the student learns from observations and the extracted goal; teacher weights track the student by EMA.Reasoning tokens receive KL-divergence supervision, and trajectory tokens receive cross-entropy supervision against the ground-truth trajectory; the output structure includes perception, agents, prediction, planning, and an executable trajectory.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Deployment: Navigating to Language Goals

  • At deployment the self-teacher and privileged trajectory are absent, and an operator supplies a free-form language goal in place of the teacher-extracted goal.The student is trained on teacher-generated goals from diverse training scenes, enabling zero-shot generalization to operator instructions not seen during training.Source: §3.3 · p. 5

4 Experiments

Real-world experiments evaluate ASD against navigation VLAs across indoor and outdoor language-conditioned tasks, then assess its generalization, steerability, multi-stage execution, and reasoning. The reported results include strong average success rates and behavior that varies with language goals.

4.1 Experimental Setup

  • The real-world evaluation used a FrodoBot rover, trained on RECON and FrodoBot-2K, and tested nine indoor and four outdoor tasks against OmniVLA and NaVILA.Indoor tasks varied initial robot and object positions and orientations; outdoor experiments used four campus tasks.Source: §4 · p. 5, §4.1 · pp. 5–6, §4.2 · p. 6

4.2 Indoor Language-Conditioned Navigation

  • 84% was ASD’s best average success rate indoors, with advantages over the leading baseline of 60%, 50%, and 10% on three hard tasks.The tasks were right chair, monitor, and the hallway task, respectively.Source: Table 1 · p. 5, §4.2 · p. 6

4.3 Outdoor Language-Conditioned Navigation

  • 95% was ASD’s best average success rate across four campus outdoor tasks, including navigation to the statue, tree, and car.OmniVLA achieved 60% on Statue; NaVILA achieved 0% on Tree and 10% on Statue.Source: Table 2 · p. 6, §4.3 · p. 6

4.4 Behavior Assessment

  • ASD produced distinct trajectories for different language goals in the same scene despite training pairs containing only one goal and trajectory per observation.The paper contrasts this steerability with OmniVLA’s broader training: over 9,500 hours collected from 10 platforms.Source: Figure 3 · p. 7, §4.4 · p. 7
  • Indoor testing used objects and goals absent from the primarily outdoor training data; ASD averaged 84% success, and reached 80% on the OOD statue goal.On Statue, ASD outperformed OmniVLA by 20% and NaVILA by 70%.Source: §4.4 · p. 7
  • ASD succeeded on the two-stage Doorway task 80% of the time and the four-stage Hallway task 60% of the time.OmniVLA achieved 0% on Hallway, while NaVILA achieved 30% on Doorway.Source: Table 1 · p. 5, §4.4 · pp. 7–8
  • ASD’s reasoning traces align with goals and predicted trajectories; success fell from 80% to 26% without training traces and 14% without inference-time reasoning.The reasoning examples connect planned turns to target objects and open space ahead.Source: §4.4 · p. 8, Table 4 · p. 9
  • Qwen2.5-VL-ASD achieved 80% success versus 22% for Qwen2.5-VL offline-label SFT and 36% for Opus 5 SFT; Qwen3.5-VL-ASD reached 92%.On Left chair, Claude Opus 4.7 achieved 70% success but averaged 846.1 seconds per completion, while ASD achieved 100%.Source: §4.4 · pp. 8–9, Table 3 · p. 8, Table 4 · p. 9

5 Conclusion

ASD uses privileged trajectory feedback to train a language-conditioned navigation policy without language annotations, and reports strong zero-shot results across indoor, outdoor, and multi-stage tasks. The appendix compares its generalization and reasoning supervision with supervised fine-tuning.

5 Conclusion

  • ASD outperforms OmniVLA by 30% in indoor settings and NaVILA by 30% on multi-stage tasks; outdoors, it achieves 100% success on in-distribution tasks and 80% on OOD commands.The conclusion reports results across 13 real-world zero-shot tasks on a mobile robot.Source: §5 · p. 9
  • ASD still depends on demonstration trajectories as privileged feedback, and self-generated goals and reasoning may degrade in ambiguous scenes, with small targets, or under fine-grained spatial instructions.The authors suggest interactive human feedback as a possible replacement for trajectories.Source: §5 · p. 9

A Comparing ASD vs. Supervised Finetuning

  • With about 100 hours of demonstration data versus OmniVLA’s about 9500 hours, ASD shows better generalization on indoor OOD evaluations where it was not explicitly fine-tuned.The comparison is reported despite ASD being fine-tuned on substantially less data.Source: A Comparing ASD vs. Supervised… · p. 13

A.1 Privileged Soft-label Distillation Objective

  • ASD distills reasoning from a feedback-conditioned teacher distribution, transferring its token-level distribution rather than only a hard reasoning label.The student is trained from deployable inputs, while the teacher uses privileged trajectory feedback; optimization is equivalent to minimizing teacher-weighted cross entropy.Source: §A.1 · p. 13

A.2 Comparison to Standard SFT

  • The teacher’s trajectory feedback yields reasoning that explains the demonstrated action, while distributional supervision can represent multiple valid descriptions that hard-label imitation would force into one target.The paper gives alternative left/right descriptions of the same car arrangement as an example.Source: §A.2 · p. 14

B.1 Datasets

ASD is trained and evaluated using RECON and FrodoBots-2K data, with prompts that structure scene reasoning and trajectory prediction. The supplementary material outlines the training setup, reproducibility code, and goal-conditioned inference inputs.

B.1 Datasets

  • RECON and FrodoBots-2K provide the mixed training and validation data, with a 90%:10% split and a randomly selected 25% of FrodoBots-2K scenes.RECON covers a university campus; FrodoBots-2K spans sidewalk navigation across more than 10 cities.Source: §B.1 · p. 14

B.2 Model and Training Setup

  • The self-teacher uses the Qwen2.5-VL-3B-Instruct backbone, starts from the student’s pretrained weights, and is updated as an exponential moving average of the student.The reported setup uses 8 A6000 Pro GPUs; Table 5 lists the main training and inference hyperparameters, with other hyperparameters using Transformers defaults.Source: §B.2 · pp. 14–15, Table 5 · p. 15

B.3 Full Prompt Templates for Training

  • Training prompts elicit perception, agent prediction, planning, and future trajectory outputs, while the teacher additionally infers a high-level task sentence for goal-conditioned training.The task sentence is injected into the student prompt by the trainer, and the student does not generate it.Source: §B.3 · pp. 16–17
  • Teacher and student responses use the same reasoning format, with the teacher-only inferred-goal span excluded from token-level forward-KL alignment.Source: §B.3 · p. 17

B.4 Training code for reproducibility

  • The supplementary training code includes prompt construction and batching, instruction templates, and the two-pass self-distillation loop.Source: §B.4 · p. 17

B.5 Inference Details

  • Goal-conditioned inference takes a new observation, observation history, and language goal as input and returns a future trajectory.Source: §B.5 · p. 17

C Language-conditioned Navigation Tasks

The evaluation spans indoor directional, object-selection, and multi-stage tasks, plus four outdoor goals. It also specifies shared execution criteria and the compared methods’ inference setups.

C Language-conditioned Navigation Tasks

C.1.1 Directions

  • Directional tasks test reaching a door or selecting the left or right chair, while object tasks test identifying a specified chair, bin, or monitor among scene objects.Source: §C.1.1 · pp. 18–19, §C.1.2 · p. 19

C.1.3 Multi-stage

  • The doorway task requires passing through a door; the hallway task adds a right turn and hallway following after the same initial setup.Source: §C.1.3 · p. 19

C.2 Outdoor Tasks Setting

  • Outdoor tasks require following a sidewalk or navigating to a statue, tree across grass, or a specified-color car.Source: §C.2 · p. 19

C.3 Evaluation Fairness, Execution and Stopping Criteria

  • Evaluation uses ten trials per task without corrective interventions during execution, permits three replans during planning, and counts collision or loss of the goal from view as failure.A blinded human evaluator stops an episode once the instruction is satisfied; ASD has no learned stopping criterion.Source: §C.3 · p. 19

C.4.1 OmniVLA

  • The comparisons include language-conditioned OmniVLA and NaVILA, vanilla Qwen2.5-VL, Claude Opus 4.7, and open-loop trajectory prediction against learned policy and fine-tuned VLA baselines.OmniVLA predicts eight actions; NaVILA uses 64-frame context for Doorway and Hallway and 8-frame context otherwise; the open-loop setup predicts a 6.4-second trajectory from image and history.Source: §C.4.1 · p. 20, §C.4.2 · p. 20, §C.4.3 · p. 20

D.4 Open-Loop Quantitative Evaluation.

In open-loop trajectory prediction, ASD has the lowest error in most reported comparisons across RECON and FrodoBots-2K, outperforming its direct SFT baseline and the non-VLA behavior-cloning methods.

D.4 Open-Loop Quantitative Evaluation.

  • ASD uniformly surpasses the non-VLA behavior-cloning methods, while Alpamayo-1 and OpenVLA perform worse under SFT alone.Source: §D.4 · p. 22
  • 7 of 8 comparisons show ASD with the lowest error across RECON and FrodoBots-2K, and ASD significantly improves over Qwen+SFT.The comparison reports ATE, ADE, MAE, and MSE for 2-second prediction; the authors attribute the Qwen+SFT gap to reasoning-token supervision because the models share backbone, data, and optimization.Source: Table 7 · p. 22, §D.4 · p. 22

D.5 Real World Experiments Qualitative Analysis

Qualitative comparisons report ASD successes on multi-object, multi-stage, and out-of-distribution statue tasks, alongside stable self-teacher reasoning diagnostics. The accompanying video contrasts real-world demonstrations and execution speed with baselines.

D.5 Real World Experiments Qualitative Analysis

  • ASD reaches 90% success on Chair and 70% on Monitor, versus OmniVLA’s 40% and 20%, respectively, and avoids the distractor-object errors shown for OmniVLA.Source: §D.5 · p. 22, Figure 8 · p. 23
  • ASD reaches 80% on Doorway and 60% on Hallway, while OmniVLA gets 70% on Doorway and 0% on Hallway, and NaVILA gets 30% and 50%, respectively.The examples show ASD passing through the doorway with a left turn and completing the full hallway instruction.Source: §D.5 · p. 23, Figure 9 · p. 23, Figure 10 · p. 24
  • On the out-of-distribution Statue task, ASD achieves 80% success versus OmniVLA’s 60% and NaVILA’s 10%, with uneven-terrain execution deviation identified as ASD’s main failure mode.The paper notes that ASD predicts and executes eight steps at once, during which uneven terrain can cause deviation from the planned trajectory.Source: §D.5 · p. 24, Figure 11 · p. 24
  • In a stress test, teacher-student KL rises smoothly from 0.34 to 0.68 nats per token, while reasoning entropy shifts from 1.36 to 1.67 nats rather than reaching collapse extremes.With α = 0.999 and an update every 10 steps, approximately 93% of teacher EMA weight remains on the pretrained checkpoint.Source: §D.5 · p. 25

E Video Narratives

  • The project video reports that ASD runs at normal speed, while leading baselines require 4×, 6×, or 8× video speed-ups to match its real-time execution.Source: E Video Narratives · p. 25

E.1 Multi-Stage Task

ASD generalizes from single-stage training to multi-stage navigation, including tasks requiring spatial reasoning to resolve ambiguous paths.

E.1 Multi-Stage Task

  • ASD demonstrates multi-stage navigation in prompts combining door traversal, a right turn, and hallway following.The demonstrations also include navigating right to a sidewalk and then following it.Source: §E.1 · p. 25
  • 60% success on multi-stage Hallway navigation, versus 50% for NaVILA, despite ASD training only on single-stage tasks.NaVILA also shows limited generalization on single-task prompts, with 27% success indoors and 33% outdoors.Source: §E.1 · p. 25
  • The sidewalk task requires distinguishing the sidewalk ahead from the one to the right, then navigating to and following the correct path.The passage presents this as fine-grained spatial ambiguity in an unconstrained outdoor environment.Source: §E.1 · p. 25

E.2 Extreme OOD Task

ASD generalizes to an unseen target object, reaching the statue from varied starting positions that require different steering directions.

E.2 Extreme OOD Task

  • 80% success on the Statue task, compared with 60% for OmniVLA and 10% for NaVILA.The statue was absent from the training dataset.Source: §E.2 · pp. 25–26
  • Varying initial positions required specific left or right steering, testing whether success depended on a fixed directional bias.The passage characterizes the results as evidence of semantic understanding and precise goal-directed steerability.Source: §E.2 · p. 25
  • The unseen-target evaluation asks the agent to navigate to a statue that was not in the training dataset.The main Table 2 reports the Statue task alongside the OmniVLA and NaVILA success rates.Source: §E.2 · pp. 25–26

E.3 Indoor Tasks

Indoor demonstrations compare ASD with leading baselines across directional choice, object selection, long-horizon traversal, and multi-stage navigation.

E.3 Indoor Tasks

  • The video comparisons cover medium- and hard-difficulty tasks and show successful ASD navigation alongside baseline failures.The reported examples include directional choice, object selection, door traversal, and multi-stage navigation.Source: §E.3 · p. 26
  • 80% success on “Navigate to the right chair,” versus 20% for OmniVLA and 10% for NaVILA.The task places three identical chairs ahead; OmniVLA consistently chooses the left chair instead of the rightmost one.Source: §E.3 · p. 26
  • 90% success on “Navigate to the chair,” versus 40% for OmniVLA in a two-object selection task.OmniVLA deviates toward a black net, while ASD follows a direct trajectory to the target.Source: §E.3 · p. 26
  • 70% success on “Navigate to the monitor,” versus 20% for OmniVLA in a three-object selection task.The passage reports OmniVLA deviating toward the black net while ASD takes a more direct trajectory to the monitor.Source: §E.3 · p. 26
  • ASD outperforms OmniVLA by 10% on “Navigate through the door,” while avoiding the door-frame collision shown in the baseline video.The task tests long-horizon planning and execution through a narrow opening.Source: §E.3 · p. 26
  • ASD outperforms NaVILA by 10% on the door-turn-hallway task, with smoother turns and normal-speed execution.NaVILA requires a 4.7× speed-up to execute the task efficiently.Source: §E.3 · p. 26

E.4 Outdoor Tasks

Outdoor demonstrations compare ASD with baselines on Statue, Tree, and Car tasks, highlighting both target precision and inference speed.

E.4 Outdoor Tasks

  • The outdoor video comparisons contrast baseline failure modes with ASD’s successful execution on Statue, Tree, and Car.Source: §E.4 · p. 26
  • On the Statue task, OmniVLA identifies the target direction but fails to get close enough, accounting for a 20% success-rate advantage for ASD.The video comparisons cover the Statue, Tree, and Car tasks.Source: §E.4 · p. 26
  • Both ASD and OmniVLA achieve 100% success on Tree and Car, while OmniVLA requires 8.6× and 6.8× speed-ups, respectively, to match ASD’s execution efficiency.Source: §E.4 · p. 26
Loading 2610.17593v1…