PaperlayerView exact revision on arXiv

Source-linked AI summary

RIWANav: Recursive World-Action Models with Self-Improvement for Urban Navigation

Jing Xie, Shouwei Ruan, Yubin Wang, Yuxiang Zhang, Haitao Yang, Songchang Jin, Dianxi Shi

arXiv:2610.08640v1cs.RO

TL;DR

Long-horizon urban navigation needs reliable sequential decisions, but imitation learning offers limited supervision for policy-induced failures and physical trial-and-error is costly. RIWANAV alternates imagined-feedback policy improvement with policy-guided world-model adaptation. Experiments report improvements over frozen-world-model feedback and improved prediction fidelity on policy-generated actions.

  • Problem

    Imitation learning provides limited supervision for policy-induced failures, while collecting long-horizon physical trial-and-error interaction is costly.

    Source: §I · p. 1

  • Method

    RIWANAV alternates imagined-outcome policy optimization with world-model adaptation from expert-consistent policy-generated action–video pairs prioritized by novelty and prediction error.

    Source: §I · pp. 1–2, B. World Simulation for Policy… · p. 3, C. Curriculum Learning for World Model… · p. 4

  • Results

    Controlled experiments show that imagined feedback improves navigation beyond policy-only training, policy-guided adaptation adds gains over a frozen model, and the adapted model improves prediction fidelity on improved-policy actions.

    Source: §V · p. 7

  • Takeaways & Limitations

    The results support recursive self-improvement of world–action models as a practical approach to urban navigation.

    Source: §V · p. 7

    The current task-specific implementation relies on a fixed offline dataset, restricts world-model supervision to actions close to expert motion, and increases memory and computation.

    Source: F. Discussion and Limitations · p. 7

Abstract

from arXiv · show

Long-horizon urban navigation requires sequential local decisions whose errors can compound over time. Imitation learning (IL) rarely learns from failures, while physical trial-and-error reinforcement learning (RL) is costly. Action-conditioned world models can provide imagined feedback by predicting visual consequences for candidate actions. However, a frozen world model may become less reliable as the policy evolves. In this paper, we introduce RIWANAV, a post-training framework that casts the coupled adaptation of a world model and an action model (policy) as task-specific recursive self-improvement (RSI). Each cycle alternates two updates. The world model evaluates policy actions through imagined outcomes, providing comparative feedback for group-relative policy optimization (GRPO). The improved policy then constructs a grounded self-curriculum, selecting expert-consistent action-video pairs by behavioral novelty and prediction error. The refined world model supplies feedback for the next policy update, closing the recursive self-improvement loop. Experiments show that RIWANAV outperforms training baselines and prior methods, validating the proposed recursive self-improvement loop between the policy and world model. Real-world trials further demonstrate its practical applicability.

I. INTRODUCTION

Long-horizon urban navigation is vulnerable to compounding local errors, while imitation learning offers limited failure supervision and physical trial-and-error is costly. RIWANAV couples policy improvement with world-model adaptation, and experiments report gains across seen and unseen environments and real-world trials.

I. INTRODUCTION

  • Imitation learning mainly supervises successful expert actions, while long-horizon physical trial-and-error interaction is costly.In urban navigation, an inaccurate local motion changes the next observation and can compound across decisions.Source: §I · p. 1
  • A frozen world model can become misaligned with evolving policy behavior, distorting subsequent policy updates.RIWANAV adapts the world model alongside the policy so each supplies updated feedback for the other's next improvement stage.Source: §I · p. 1
  • RIWANAV recursively improves a navigation policy and a visual world model through coupled post-training.The world model evaluates candidate actions through imagined outcomes for policy optimization; the improved policy then supplies expert-consistent action–video pairs for world-model adaptation.Source: §I · p. 2
  • The grounded self-curriculum filters policy proposals by expert consistency and prioritizes accepted action–video pairs by behavioral novelty and prediction error.The refined world model supplies imagined feedback for the next policy update, closing the training loop.Source: §I · p. 2
  • Experiments show consistent improvements across seen and unseen environments, with real-world deployment further validating practical effectiveness.Real-world trials demonstrate practical feasibility beyond offline evaluation.Source: §I · p. 2

II. RELATED WORK

Prior navigation work develops policies through data, supervision, and policy design, while world-model research uses learned future dynamics for planning and policy learning. This paper focuses instead on continued adaptation between the policy and world model as behavior evolves.

A. Real-World Navigation

  • Real-world navigation methods use diverse robot trajectories and goal-conditioned supervision, including city-scale video, landmark-grounded language, and multimodal policies.Other approaches use goal-masked diffusion [11] or generate instruction-conditioned visual paths [13].Source: A. Real-World Navigation · p. 2

B. World Models for Policy Learning

  • World models support planning and policy learning through learned future dynamics, including latent imagination and visual prediction methods.NWM ranks navigation trajectories using controllable video prediction, while Fast-WAM [6] jointly models video and actions without explicit future generation.Source: B. World Models for Policy Learning · p. 2
  • Unlike work that treats the world model as a fixed source of imagined feedback, this paper studies continued policy–model adaptation as policy behavior evolves.The cited work spans navigation and embodied-control world models, and approaches connecting visual prediction with policy optimization,.Source: B. World Models for Policy Learning · p. 2

C. Recursive Self-Improvement

RIWANAV applies recursive self-improvement to navigation by alternating policy optimization with world-model adaptation. The policy selects expert-consistent action–video pairs for the next model update, whose imagined feedback guides subsequent policy learning.

C. Recursive Self-Improvement

  • The paper connects recursive self-improvement with model-based policy learning: the world model improves the policy through imagined feedback, and the evolving policy guides the next model update.This closed loop is described as enabling the two components to improve one another recursively.Source: C. Recursive Self-Improvement · p. 2

III. METHOD

  • Each RIWANAV cycle alternates policy optimization using imagined candidate-action outcomes and world-model adaptation using policy-generated actions consistent with expert motion.A grounded self-curriculum prioritizes the resulting action–video pairs by behavioral novelty and current prediction error.Source: §III · p. 2

A. Overall

  • During recursion, the policy proposes actions, but only expert-consistent actions are paired with recorded future frames for the replay pool.The self-curriculum prioritizes those pairs by behavioral novelty and current prediction error; the refined model supplies feedback for the next policy-improvement phase.Source: A. Overall · pp. 2–3
  • The recursive updates change both the world model and policy, as well as the feedback exchanged between them.The figure marks which component is optimized in each phase.Source: Figure 2 · p. 3

B. World Simulation for Policy Improvement

The policy is initialized by supervised learning, then improved by comparing imagined outcomes for candidate actions from the same state. GRPO uses these relative evaluations while expert and reference-policy terms constrain policy drift.

B. World Simulation for Policy Improvement

  • Supervised fine-tuning on expert trajectories initializes the policy and supplies a fixed behavior reference, but does not compare alternative policy-proposed actions.The initialization combines frozen DINOv2 visual features, CLIP text features, and pose embeddings in a transformer waypoint head.Source: B. World Simulation for Policy… · p. 3
  • The current world model scores candidate actions’ imagined consequences, and GRPO compares candidates sampled from the same state to guide local policy improvement.States come from the offline dataset, while candidate actions come from the current policy.Source: B. World Simulation for Policy… · p. 3
  • Clipping, a KL penalty to the initial policy, and an expert-anchor loss constrain updates driven by imagined outcomes.The expert anchor compares current predictions with recorded expert waypoints.Source: B. World Simulation for Policy… · p. 3

C. Curriculum Learning for World Model Improvement

The improved policy supplies candidate actions for world-model training, but only valid, expert-consistent pairs with recorded futures enter replay. A curriculum prioritizes accepted pairs by novelty and prediction error, and the refined model feeds the next policy update.

C. Curriculum Learning for World Model Improvement

  • Expert consistency measures action-to-expert discrepancies using ADE, MADE, AOE, and MAOE, retaining locally consistent actions to support reliable supervision.The score aggregates distance and orientation measures; rollout validity and individual maximum-error tolerances also govern admission.Source: C. Curriculum Learning for World Model… · p. 4
  • Only valid actions meeting the expert-consistency threshold and maximum-error tolerances are paired with recorded futures for world-model training.Rejected actions are skipped for world-model training but remain available for policy optimization.Source: C. Curriculum Learning for World Model… · p. 4
  • Among accepted pairs, the curriculum prioritizes behavioral novelty relative to the initial policy and prediction error under the current world model.Novelty is based on deviation from the initial policy mean, while prediction error is measured between predicted and recorded futures.Source: C. Curriculum Learning for World Model… · p. 4
  • The sampled replay batch trains the world model on recorded futures, and the refined model then supplies imagined feedback for the next policy-improvement phase.The curriculum weights pairs by expert consistency, novelty, and prediction error.Source: C. Curriculum Learning for World Model… · p. 4

IV. EXPERIMENTS

The experiments assess whether learned visual world models improve navigation, whether policy-guided refinement beats a frozen model, and whether refinement improves accuracy on the improved policy’s behaviors. A physical-robot deployment evaluates transfer beyond offline benchmarking.

IV. EXPERIMENTS

  • The experiments ask whether visual world-model feedback improves navigation beyond policy-only training and whether policy-guided refinement improves over a frozen model.They also test whether the model becomes more accurate on behaviors from the improved policy.Source: §IV · p. 4
  • Controlled comparisons, ablations, and world-model analyses address the three research questions in sequence.Source: §IV · p. 4
  • A physical-robot deployment evaluates transfer beyond offline benchmarking.Source: §IV · p. 4

A. Experimental Setup

The setup uses shared URBANNAV data and evaluation protocols to compare policy-only, frozen-world-model, and RIWANAV post-training frameworks. It also specifies the model, curriculum, and deployment configurations.

A. Experimental Setup

  • Evaluation uses URBANNAV’s official train, test-seen, and test-unseen splits, with shared inputs, waypoint and stop predictions, episode lists, trajectory mapping, and a 3-m success radius.Metrics are SR, OSR, NE, SPL, nDTW, and SDTW; higher is better except for NE, and results are means and standard deviations over three seeds unless stated otherwise.Source: A. Experimental Setup · p. 4
  • The controlled comparison holds policy architecture, initialization, data, interface, and evaluation protocol fixed while varying post-training framework.The baselines include imitation learning, PPO and GRPO without action-conditioned visual feedback, and PPO-WM and GRPO-WM with a frozen COSMOS model; RIWANAV uses GRPO and adapts COSMOS with a grounded self-curriculum.Source: A. Experimental Setup · p. 4
  • The policy compares four candidates per state and predicts eight waypoints over eight seconds, while COSMOS predicts 17 frames at 2 fps from 16 relative-pose actions.The self-curriculum uses qmin = 0.35 and equal novelty and prediction-error weights; the world model updates every 500 policy steps using at most 500 replay examples.Source: A. Experimental Setup · p. 4
  • Real-world deployment uses a custom-built mobile robot with a forward-facing RGB camera.Source: A. Experimental Setup · p. 4

B. Comparative Experiments

RIWANAV closes the feedback loop between the policy and world model, outperforming frozen-world-model feedback on both test splits and all reported metrics.

2) Comparison with Prior Approaches:

RIWANAV leads the prior approaches across reported metrics, while ablations indicate further gains from policy-induced replay and the grounded self-curriculum.

2) Comparison with Prior Approaches:

C. Ablation Study

  • Adding the grounded self-curriculum to policy-induced replay further improves SR and SPL by 3.42% and 2.48%, respectively.Policy-induced replay with uniform sampling improves SR/SPL by 6.60%/5.39% over static expert replay; the combined approach improves SR/SPL by 7.94%/6.50% over the frozen-world-model baseline.Source: C. Ablation Study · p. 6

D. Evidence for Recursive Self-Improvement

The final RIWANAV world model improves prediction on both expert and improved-policy actions. Qualitative comparisons likewise show more coherent scene structure under policy-generated actions.

E. Real-World Deployment

Real-world trials show that the RIWANAV-trained policy can guide a physical robot toward language-specified landmarks. The paper also identifies scope and resource constraints on the current approach.

E. Real-World Deployment

  • In urban scenarios, the RIWANAV-trained policy uses language instructions and onboard visual observations to guide a mobile robot toward specified landmarks.The reported trials cover different route layouts and lighting conditions.Source: E. Real-World Deployment · p. 7, Figure 5 · p. 7

F. Discussion and Limitations

  • The current implementation is task-specific and relies on a fixed offline dataset; recorded futures supervise only policy actions close to expert motion, limiting counterfactual exploration without new interaction data.Source: F. Discussion and Limitations · p. 7

V. CONCLUSION

  • RIWANAV couples imagined world-model feedback for policy improvement with a grounded self-curriculum for the next world-model update.Controlled experiments report gains over frozen-world-model feedback and improved prediction fidelity on improved-policy actions.Source: §V · p. 7

Related papers

Loading 2610.08640v1…