PaperlayerView exact revision on arXiv

Source-linked AI summary

NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewards

Chia-Yu Hung, Navonil Majumder, Haoyuan Deng, Liu Renhang, Yankang Ang, Amir Zadeh, Chuan Li, Dorien Herremans, Ziwei Wang, Soujanya Poria

arXiv:2511.14659v1cs.ROcs.AI

TL;DR

VLA models remain limited in reliability and generalization across embodiments and real-world environments. This paper introduces NORA-1.5 with a flow-matching action expert and uses world-model and action-based rewards to construct DPO preferences; the resulting model and post-training improve performance across simulated and real-robot settings.

  • Problem

    Existing VLA approaches rely heavily on imitation learning and embodiment-specific SFT, whose dependence on limited demonstrations restricts generalization beyond expert data.

  • Method

    NORA-1.5 couples NORA with a flow-matching action expert, while reward models use action-conditioned world-model rollouts and deviation-from-ground-truth signals to form DPO preference datasets.

  • Results

    Reward-driven post-training consistently improves performance in simulation and real-robot settings, while NORA-1.5 achieves state-of-the-art performance on simulated benchmarks and transfers to real-world experiments.

  • Takeaways & Limitations

    World-model-driven goal and action rewards provide a potent proxy-reward strategy for DPO-based adaptation of VLA models across embodiments.

  • Takeaways & Limitations

    Flow-matching action generation underperforms autoregressive decoding on Galaxea A1, likely because the real-robot dataset is smaller and the expert lacks extensive flow-matching pre-training.

Abstract

from arXiv · show

Vision--language--action (VLA) models have recently shown promising performance on a variety of embodied tasks, yet they still fall short in reliability and generalization, especially when deployed across different embodiments or real-world environments. In this work, we introduce NORA-1.5, a VLA model built from the pre-trained NORA backbone by adding to it a flow-matching-based action expert. This architectural enhancement alone yields substantial performance gains, enabling NORA-1.5 to outperform NORA and several state-of-the-art VLA models across both simulated and real-world benchmarks. To further improve robustness and task success, we develop a set of reward models for post-training VLA policies. Our rewards combine (i) an action-conditioned world model (WM) that evaluates whether generated actions lead toward the desired goal, and (ii) a deviation-from-ground-truth heuristic that distinguishes good actions from poor ones. Using these reward signals, we construct preference datasets and adapt NORA-1.5 to target embodiments through direct preference optimization (DPO). Extensive evaluations show that reward-driven post-training consistently improves performance in both simulation and real-robot settings, demonstrating significant VLA model-reliability gains through simple yet effective reward models. Our findings highlight NORA-1.5 and reward-guided post-training as a viable path toward more dependable embodied agents suitable for real-world deployment.

1. Introduction

NORA-1.5 addresses limited VLA reliability and generalization by combining a flow-matching action expert with reward-guided DPO post-training. The approach uses world-model and action-based rewards to improve adaptation across simulated and real-robot embodiments.

  • Motivation: Existing VLA adaptation relies heavily on imitation learning and embodiment-specific SFT, which can inherit bias from limited manually curated demonstrations.This limits generalization beyond the quality of expert data.
  • NORA-1.5: NORA-1.5 couples a pre-trained autoregressive VLA with a flow-matching action expert and achieves state-of-the-art results across simulated and real-world benchmarks.The model is jointly trained on Open X-Embodiment data.
  • Reward-guided post-training: World-model rollouts replace expensive physical action-rollout pipelines with lightweight learned evaluators that can score many synthetic trajectories using available compute.The evaluators combine predicted outcomes with geometric or heuristic checks.
  • Reward-guided post-training: The post-training framework ranks VLA-generated actions with an action-conditioned world model, distance-based rewards, and subgoal scoring before DPO optimization.These reward signals provide scalable criteria for constructing preference datasets.
  • Architectural analysis: Flow-matching experts and autoregressive VLAs provide mutual benefits: shared representations support action generation, while expert feedback improves trajectory-level planning.The work also reports consistent gains from reward-based DPO post-training in simulation and on real robots.

2. Preliminaries

The preliminaries describe NORA as a 3B-parameter autoregressive VLA built from Qwen-2.5-VL-3B and Open X-Embodiment data. V-JEPA2-AC supplies an action-conditioned world model for predicting future visual embeddings from actions.

  • 2.1. NORA: NORA is a 3B-parameter autoregressive VLA fine-tuned from Qwen-2.5-VL-3B on Open X-Embodiment data to predict action tokens.Its VLM backbone supports multimodal reasoning, representation learning, and instruction following.
  • 2.2. V-JEPA2-AC: V-JEPA2-AC predicts future frame embeddings from current visual frames and an action sequence, serving as the action-conditioned world model for VLA post-training.It uses a V-JEPA2 vision encoder with an added predictor network.

3. NORA-1.5

NORA-1.5 extends the NORA backbone with a dedicated flow-matching action expert that directly generates action sequences. The expert accepts representations from NORA to produce actions through flow-matching-based regression.

  • 3. NORA-1.5: NORA-1.5 adds a dedicated flow-matching action expert to NORA, using the backbone’s representations to generate action sequences directly.The design is motivated by the performance and efficiency of flow-matching action heads.

2. Post-Training 1. Pre-Training

NORA-1.5 first trains a VLA with imitation learning, then uses learned reward signals to rank generated actions and optimize preferences with DPO.

  • 1. Pre-Training: A VLA is first pretrained through imitation learning, after which generated actions are evaluated to construct preference data for optimization.
  • 1. Pre-Training: The action expert directly regresses action sequences over a fixed horizon using flow matching, conditioned on representations from NORA.The horizon is N = 5, and the expert uses NORA transformer keys and values.
  • 2. Post-Training: Reward modeling addresses the cost of executing many candidate trajectories in simulation or on physical robots by using learned evaluators instead.World and video models can estimate action consequences without fully engineered simulators.
  • 2. Post-Training: The resulting preference pairs are used with DPO to align action-expert outputs during reward-guided post-training.
  • 2. Post-Training: The reward framework combines world-model goal alignment with an action-based distance heuristic to balance diverse feasible trajectories against noisy or overly constrained references.The world-model reward compares predicted outcomes with final goals or intermediate subgoals, while the action reward measures proximity to ground-truth actions.
  • 2. Post-Training: Dense per-step rewards rank candidate actions at each timestep, enabling localized preference information and deeper step-level exploration during DPO.Preference pairs are formed by ranking sampled actions under the reward definitions.

4. Experiments

Across simulated and real-robot benchmarks, NORA-1.5 generally outperforms prior VLA baselines, while DPO post-training improves robustness and task success. The experiments also show that reward choice and data regime affect gains, with hybrid and subgoal-based rewards often strongest.

  • Performance of NORA-1.5: NORA-1.5 outperforms strong baselines across SimplerEnv, LIBERO, and Galaxea A1 evaluations.The reported advantage spans simulated and real-world settings, though task-specific exceptions remain.
  • Performance of NORA-1.5: Flow-matching action generation consistently improves NORA-1.5 over NORA across benchmarks, but performs worse than autoregressive decoding with limited Galaxea A1 data.The real-robot fine-tuning set contains 50K frames, versus 4M frames for SimplerEnv.
  • Impact of DPO-based Post-training: DPO gains are generally consistent in LIBERO but smaller on LIBERO-Object and overall LIBERO evaluations because the SFT baseline is already strong.LIBERO-Long shows consistent gains of 1% to 1.7% across reward-modeling techniques.
  • Impact of DPO-based Post-training: DPO improves NORA-1.5 by 13% across thirteen real-robot tasks, raising correct-object grasping by 11% and reducing distractor grasps by 4%.The largest gains, 15%-16%, occur on unseen tasks and objects; DPO also reduces average action chunks from 9.7 to 7.0.
  • Ablations: On Galaxea A1, WM subgoal + GTA consistently yields stronger gains than WM endgoal + GTA, while GTA alone can underperform SFT on challenging unseen-object tasks.The authors attribute this pattern to localized subgoal guidance being less noisy than endgoal guidance and to multiple valid real-world trajectories.

5. Conclusion

The work shows that preference-optimization post-training improves NORA-1.5 adaptation in both simulation and real-world settings when paired with suitable reward modeling.

  • Preference-optimization post-training improves NORA-1.5 adaptation in both real-world and simulated settings when appropriate reward models are used.The authors describe world-model-driven goal and action rewards as effective proxy rewards for DPO.

A. Evaluation Settings and Metrics

The evaluation covers simulated LIBERO and SimplerEnv benchmarks plus real-world transfer to a Galaxea A1 arm, using success-oriented metrics across varied tasks and generalization settings.

  • The evaluation spans simulated LIBERO and SimplerEnv benchmarks and real-world experiments on a Galaxea A1 robotic arm.The benchmarks cover diverse simulated settings, while Galaxea A1 tests cross-embodiment transfer.
  • LIBERO: LIBERO evaluates generalization across spatial layouts, objects, task goals, and long-horizon tasks using four corresponding subsets.Each task subset is evaluated across 500 episodes after combining the subsets for single-model fine-tuning.
  • SimplerEnv: SimplerEnv evaluates pick-and-place, object-relocation, drawer-opening, and drawer-closing tasks while reducing the simulation-to-reality gap.Its evaluation includes visual-matching and other task settings described in the benchmark setup.
  • Cross-Embodiment Evaluation: The Galaxea A1 transfer study uses 1,000 teleoperated episodes with randomized object locations and nine evaluation tasks across seen and unseen settings.The evaluation includes unseen objects with seen distractors and unseen instructions with seen distractors.
  • Metrics: Simulation trials assign success 1 when the prompted task is completed and 0 otherwise, while real-world evaluation also reports partial success.Partial success distinguishes complete failures from trials showing substantial progress.

B. Related Works

Prior VLA work emphasizes imitation learning, flow-matching action generation, world models, and preference optimization; this work combines complementary reward signals with DPO for VLA post-training.

  • Vision–Language–Action Models: Existing VLA models primarily use supervised imitation learning on large cross-embodiment datasets, sometimes followed by embodiment-specific fine-tuning.The related-work discussion notes that these approaches do not study reward-based post-training of VLA policies.
  • Flow-Matching and Action Representation: Flow-matching action models generate smooth continuous trajectories, while FAST-based methods compress continuous actions into discrete tokens for efficient autoregressive decoding.The discussion positions these as complementary approaches to action representation and generation.
  • World Models for Visual Robot Control: World models predict future observations conditioned on current observations and actions and can support planning or model-based visual robot control.V-JEPA2 provides a latent video-prediction model extendable to action-conditioned dynamics.
  • Preference-based Post-Training and Reward Design: This work brings preference-based post-training to VLAs by constructing synthetic preferences from world-model goal rewards and distance-to-expert-action heuristics.Combining these rewards with DPO improves performance over purely supervised training on LIBERO and SimplerEnv.

C. Model Architecture and Training Details

NORA-1.5 adds a newly initialized flow-matching action expert to the NORA backbone and jointly trains token prediction with flow matching.

  • NORA-1.5 adds an approximately 400-million-parameter action expert and jointly trains FAST-token prediction with flow matching.The training objective is L_Loss = L_CE + αL_FM with α = 10.
  • The action expert interacts with Qwen 2.5 VL embeddings through self-attention while attending only to language and image embeddings.This prevents FAST action representations from leaking into the action expert.
  • After initializing the action expert, the model is jointly optimized for 150,000 gradient steps with a global batch size of 512.Training uses the same Open-X-Embodiment subset as NORA-Long and a maximum learning rate of 5e-5.

D. Galaxea Data Collection

The Galaxea A1 data collection gathered 1,000 teleportation-based episodes across nine simple pick-and-place tasks with randomly placed objects and no fixed ordering.

  • 1,000 episodes covered nine Galaxea A1 pick-and-place tasks, with roughly 100 or more episodes per task.Objects were randomly placed on the table, and collection did not follow a fixed order.

E. Baselines

The evaluation compares NORA-1.5-related policies with a broad set of vision-language-action baselines spanning generalist, spatial, reasoning, diffusion, and flow-matching designs.

  • Spatial and temporal reasoning baselines include SpatialVLA, TraceVLA, CoT-VLA, and ThinkAct.Their designs use spatial information, visual traces, subgoal images, or structured embodied plans to guide manipulation.
  • The baseline suite includes generalist policies such as OpenVLA, HPT, Octo-Base, RoboVLM, π0, and π0-FAST.These models cover pretrained or cross-embodiment approaches using transformer, diffusion, and flow-matching components.
  • The comparison also includes action-reasoning and multimodal-agent baselines MolmoAct, Emma-X, Magma, and GR00T N1.5.These models represent depth-aware action reasoning, hierarchical guidance, visual planning traces, and humanoid-oriented dual-system policies.
  • NORA and NORA-Long provide closely related reference policies, with NORA using a 3B autoregressive architecture and NORA-Long extending its action horizon.Both are trained on Open X-Embodiment data, while NORA uses the FAST tokenizer for action representation.
Loading 2511.14659v1…