PaperlayerView exact revision on arXiv

Source-linked AI summary

MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

Ruicheng Zhang, Mingyang Zhang, Jun Zhou, Xiaofan Liu, Zunnan Xu, Zhizhou Zhong, Puxin Yan, Haocheng Luo, Xiu Li

arXiv:2512.06628v4cs.ROcs.CV

TL;DR

Scarce diverse, long-horizon manipulation data and limitations in existing video world models motivate MIND-V. It combines hierarchical planning, intermediate representations, and video generation with staged rollouts and physics-guided post-training; experiments report state-of-the-art synthesis and utility for policy learning.

  • Problem

    Scalable embodied intelligence is constrained by scarce diverse, long-horizon robotic manipulation data, and existing methods incompletely address long-horizon coherence, semantic-to-pixel generation, and physical plausibility.

    Source: §1 · p. 1

  • Method

    MIND-V combines an SRH for task planning, a BSB for domain-invariant representations, and an MVG for video synthesis, with staged future rollouts and GRPO post-training guided by a physical-coherence reward.

    Source: §1 · p. 2, §3.1 · p. 3, Behavioral Semantic Bridge (BSB) · p. 3

  • Results

    MIND-V achieves state-of-the-art video synthesis and provides an effective simulation environment for training embodied AI policies.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors present MIND-V as a scalable, autonomous framework for synthesizing robotic manipulation data and supporting embodied policy training.

    Source: §1 · p. 2, §5 · p. 9

    Staged Visual Future Rollouts add inference overhead by generating multiple candidate videos per subtask transition, and upstream SRH inaccuracies can propagate downstream.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scalable embodied intelligence is constrained by the scarcity of diverse, long-horizon robotic manipulation data. Existing video world models in this domain are limited to synthesizing short clips of simple actions and often rely on manually defined trajectories. To this end, we introduce MIND-V, a cognitive hierarchical world model designed to synthesize physically plausible and logically coherent videos of long-horizon robotic manipulation. Inspired by cognitive science, MIND-V bridges high-level reasoning with pixel-level synthesis through three core components: a Semantic Reasoning Hub (SRH) that leverages a pre-trained vision-language model for task planning; a Behavioral Semantic Bridge (BSB) that translates abstract instructions into domain-invariant representations; and a Motor Video Generator (MVG) for conditional video rendering. MIND-V employs Staged Visual Future Rollouts, a test-time optimization strategy to enhance long-horizon robustness. To enforce adherence to physical laws, we introduce a GRPO reinforcement learning post-training phase guided by a novel Physical Foresight Coherence (PFC) reward. PFC leverages the V-JEPA2 world model as a physics referee to penalize implausible dynamics in the latent feature space. Experiments confirm MIND-V's SOTA performance in long-horizon simulation and its significant value for policy learning, introducing a scalable and fully autonomous framework for embodied data synthesis.

1 Introduction

MIND-V targets the scarcity of diverse, long-horizon robotic manipulation data and the coherence, semantic translation, and physical-plausibility challenges that limit existing video models. It combines hierarchical planning and video generation with staged future rollouts and physics-guided post-training; experiments report state-of-the-art synthesis and value for policy learning.

1 Introduction

  • Scalable robot learning is constrained by scarce high-quality, diverse, long-horizon manipulation data, while existing approaches incompletely address coherence, semantic-to-pixel translation, and physical plausibility.Directly fine-tuned video models can follow instructions imprecisely and degrade visually; trajectory-controlled models trade autonomy and scalability for controllability.Source: §1 · p. 1
  • Staged Visual Future Rollouts use a propose-verify-refine loop at subtask transitions to select coherent continuations and mitigate error accumulation.The method simulates multiple candidate futures and supports replanning.Source: §1 · p. 2
  • GRPO post-training uses a Physical Foresight Coherence reward and a pretrained world model as a physics referee to score generated dynamics in latent space.The stated aim is to steer the MVG toward physically plausible outputs.Source: §1 · p. 2
  • MIND-V is presented as the first hierarchical video world model for long-horizon manipulation, with an SRH for planning, a BSB for domain-invariant representations, and an MVG for video synthesis.The architecture separates high-level reasoning from pixel-level synthesis.Source: §1 · p. 2
  • Experiments report state-of-the-art long-horizon manipulation video synthesis and significant utility for downstream policy learning.The paper presents MIND-V as a scalable, autonomous paradigm for robotic world modeling.Source: §1 · p. 2

2 Related Work

Prior embodied video world models offer either semantic understanding with limited fine-grained control or more precise trajectory guidance that requires dense manual annotations. Controllable video generation spans high-level semantic and low-level structural conditioning.

2.1 Video World Models for Embodied AI

  • Embodied video world models learn physical dynamics or use text-to-video planning, but fine-grained execution control can remain limited, while explicit trajectory guidance requires dense manual annotations.MIND-V is described as autonomously decomposing commands into generator instructions without additional manual supervision.Source: §2.1 · p. 2
  • The cited embodied approaches face a trade-off between fine-grained control and scalable, autonomous data generation.Trajectory-guided systems improve action modeling precision but rely on dense annotations; other methods can lack fine-grained manipulation control.Source: §2.1 · p. 2

2.2 Controllable Video Generation

  • Existing controllable video generation ranges from high-level semantic signals such as text prompts to low-level structural inputs such as masks.The passage introduces this conditioning spectrum without specifying comparative outcomes.Source: §2.2 · p. 2

3 Method

MIND-V connects high-level task planning to pixel-level video generation through the SRH, BSB, and MVG. Its method adds staged future rollouts for iterative planning and trains the MVG first with supervised fine-tuning, then with GRPO-based alignment.

Semantic Reasoning Hub (SRH)

  • The SRH decomposes instructions into ordered subtasks, grounds them with affordance localization, and plans frame-aligned trajectories from initial observations.Each subtask specifies an action, object, and destination; the localizer identifies object masks and interaction points.Source: Semantic Reasoning Hub (SRH) · p. 3, §3.1 · p. 3

Behavioral Semantic Bridge (BSB)

  • The BSB translates SRH outputs into domain-invariant conditioning through object and robot masks, phase-decomposed trajectories, and phase-transition frame indices.Its three trajectory phases are pre-interaction, interaction, and post-interaction; decoupling task logic from visual appearance is intended to support generalization to novel environments and tasks.Source: Behavioral Semantic Bridge (BSB) · pp. 3–4

3.1 Framework

  • The MVG injects BSB-derived spatiotemporal guidance into a Diffusion Transformer during denoising to condition video generation on planned kinematics.A motion-embedding module encodes the guidance tensor into features fused with intermediate hidden states; the caption describes the flow from semantic representation to guidance tensor, motion signals, and latent diffusion transformer.Source: Behavioral Semantic Bridge (BSB) · p. 4, Figure 3 · p. 4

3.2 Staged Visual Future Rollouts

  • Staged Visual Future Rollouts evaluates parallel short-horizon video candidates at each subtask transition and uses VLM feedback to refine plans when the top rollout misses a success threshold.The SRH proposes semantically diverse BSB variants; the VLM evaluates task success, physical plausibility, and visual quality before selection or replanning.Source: §3.2 · pp. 4–5

Stage 1: Supervised Fine-Tuning (SFT)

  • SFT trains the MVG to map BSB representations to pixel-level execution with a denoising objective, producing a reference policy for later alignment.Training on short subtask videos is stated to suffice for generalization to arbitrary long-horizon tasks within the hierarchical framework.Source: Stage 1: Supervised Fine-Tuning (SFT) · p. 5, §3.3 · p. 5

Stage 2: Physical Alignment via GRPO Post-Training

  • GRPO post-training treats denoising as a Markov decision process and uses group-relative reward normalization with clipped policy updates and KL regularization toward the SFT policy.The KL term is described as mitigating reward hacking and maintaining generation quality while preserving kinematic conditioning.Source: Stage 2: Physical Alignment via GRPO… · pp. 5–6
  • The composite GRPO reward combines physical and aesthetic scores, with weights of 0.2 and 1, respectively.The aesthetic reward is supplied by a VLM that scores clarity, artifacts, and realism.Source: Stage 2: Physical Alignment via GRPO… · p. 5
  • PFC scores local physical plausibility by cosine similarity between V-JEPA2 latent predictions and actual future states, then weights lower-scoring windows more heavily.The softmax weighting focuses optimization on larger errors; its temperature controls focus sharpness.Source: Stage 2: Physical Alignment via GRPO… · p. 5

4 Experiments

The experiments compare MIND-V with video-generation baselines, test its use for policy learning, and examine its components and reward sensitivity. Reported findings include higher long-horizon benchmark scores, improved policy success, and performance declines in several ablations.

4.1 Experiment Settings

  • The test set contains 108 Bridge V2 and unseen web-sourced samples; short tasks use V-Bench, while long tasks also use user study, PFC, and task success.Long-horizon comparisons are restricted to autonomous world-model baselines, and each task comprises 2 to 4 subtasks.Source: §4.1 · p. 6

4.2 Qualitative and Quantitative Comparison

  • MIND-V surpasses the second-best long-horizon method by 9.0% on PFC Score, 76.7% on Task Success Rate, and 172.2% on User Preference.The comparison reports consistent baseline outperformance across short- and long-horizon tasks; baseline failures include logical hallucinations, physical implausibility, and inaccurate semantic grounding.Source: §4.2 · p. 6, Figure 5 · p. 7, Table 2 · p. 8

4.3 World Model for Policy Learning

  • MIND-V guidance raises mean MimicGen success from 27.7% (Base) to 43.5%, exceeding the 33.4% achieved by IL continuous training.The experiment evaluates Coffee, StackThree, and Square using 128 demonstration trajectories per task.Source: §4.3 · p. 7, Figure 6 · p. 7, Table 3 · p. 8

4.4 Ablation Study

  • Removing GRPO lowers PFC Score, replacing the affordance module lowers subtask success, and disabling staged rollouts degrades extended-horizon performance.Replacing the VLM causes only a marginal drop, indicating that MIND-V benefits from but does not depend on a specific VLM.Source: §4.4 · p. 8, Table 4 · p. 8
  • MIND-V’s reward-weight choices show robust long-horizon performance across a reasonable parameter range.The sensitivity analysis reports the rationale for the selected reward weights.Source: §4.4 · p. 8, Table 5 · p. 8

5 Conclusion

MIND-V combines hierarchical reasoning with pixel-level video synthesis for long-horizon robotic manipulation. The paper reports state-of-the-art video synthesis and an effective simulation environment for embodied-policy training.

5 Conclusion

  • MIND-V integrates a Semantic Reasoning Hub, Behavioral Semantic Bridge, and Motor Video Generator in a decoupled architecture.The architecture separates high-level semantic reasoning from low-level pixel synthesis.Source: §5 · p. 8
  • GRPO post-training with a Physical Foresight Coherence reward and staged visual rollouts target physical plausibility and error accumulation.Source: §5 · p. 9
  • Evaluations report state-of-the-art video synthesis and a highly effective simulation environment for training embodied AI policies.Source: §5 · p. 9

6 Limitations

The authors identify inference overhead from staged rollouts and sensitivity to inaccuracies in the Semantic Reasoning Hub as limitations. A VLM-based fallback is introduced to mitigate affordance-localization failures.

6 Limitations

  • Staged rollouts add inference overhead because each subtask transition generates multiple candidate videos.Source: §6 · p. 9
  • Upstream affordance-localization failures can propagate downstream; a VLM-based fallback infers target coordinates and destinations to mitigate them.Source: §6 · p. 9
  • The fallback refines inferred coordinates into point prompts for a segmentation model to extract precise masks.Source: §6 · p. 9

A Analysis of Computational Cost and Hyperparameters

The analysis examines computational cost as task length increases and studies the effect of the number of staged visual future rollouts.

A Analysis of Computational Cost and Hyperparameters

A.1 Scalability with Task Length

Across tasks with one to three subtasks, simulation time scales linearly while peak VRAM remains constant. The MVG accounts for most resource use, whereas SRH planning adds a minor, stable overhead.

A.1 Scalability with Task Length

  • The evaluation measures simulation time and peak VRAM across tasks ranging from one to three subtasks.The results are summarized in Table 6 and visualized in Figure 7.Source: §A.1 · p. 12
  • Peak VRAM remains approximately 70 GB as task length grows, while average simulation time stays approximately 60 seconds per sub-task.The hierarchical autoregressive design allocates memory for one sub-task and reuses it; total simulation time scales linearly with the number of subtasks.Source: §A.1 · p. 12
  • The MVG consumes approximately 65-70% of execution time and approximately 86% of VRAM, while SRH planning adds minor, stable overhead.These are the internal resource distributions reported in Table 6.Source: §A.1 · p. 12

A.2 Analysis of the Number of Rollout Samples (K)

The rollout count K controls how many candidate future trajectories are simulated at each sub-task transition. K=3 is the reported balance between performance and computational cost.

A.2 Analysis of the Number of Rollout Samples (K)

  • K is the number of candidate future trajectories simulated at each sub-task transition.The rollout mechanism selects among these futures to favor physically plausible continuations.Source: §A.2 · p. 12
  • Reported SRH inference times are pure inference duration and exclude network transmission latency; API response time and network latency affect speed.The cited reference value uses the Gemini-2.5 API.Source: §A.2 · p. 12
  • Task Success Rate rises from 35.2% at K=1 to 61.3% at K=3; Peak VRAM rises from 70.1 GB at K=3 to 122.0 GB at K=5, supporting K=3.Performance gains diminish from K=3 to K=5, while both time and memory costs increase with larger K.Source: §A.2 · p. 12

B Dataset Construction

The dataset pipeline converts raw Bridge V2 videos into structured BSB annotations through object-mask generation and trajectory decomposition. It represents manipulation using object and robot masks, motion-based phases, and phase-transition points.

B Dataset Construction

  • The automated pipeline generates ground-truth BSB annotations from Bridge V2 videos in two stages: Object Representation Generation and Trajectory Decomposition.Figure 9 depicts the annotation pipeline; failures such as incorrect grounding or trajectory errors are flagged for human correction.Source: B Dataset Construction · pp. 12–13, Figure 9 · p. 14, §B.2 · p. 15

B.1 Object Representation Generation

  • Object Representation Generation obtains target-object and robot-arm masks, using instruction-grounded object identification and segmentation alongside a template mask for the arm.The VLM extracts the target-object noun phrase; Grounded SAM2 uses it to produce the target mask in the initial frame. A randomly sampled 10% of data is reserved for testing.Source: §B.1 · pp. 13–14

B.2 Trajectory Decomposition and Phase Segmentation

  • Trajectory decomposition tracks the object and gripper mask centroids, then divides motion into pre-interaction, interaction, and post-interaction phases.The phases span from task start until object motion begins, while it moves, and from when it stops until subtask end; phase-transition points are defined by object-motion endpoints.Source: §B.2 · p. 14

C Additional Visual Results

The qualitative examples contrast MIND-V with baselines across long-horizon, short-horizon, and out-of-distribution tasks. They illustrate physical and semantic failures in baselines and coherent, instruction-following behavior in MIND-V.

Long-horizon Tasks

  • On the long-horizon sequence, MIND-V completes both subtasks, while WoW-14B shows physical violations and interacts with the wrong object.The paper attributes MIND-V’s coherence to SRH task decomposition and staged visual rollouts that filter implausible transitions and prevent long-term semantic drift.Source: C Additional Visual Results · p. 15

Short-horizon Tasks

  • In short-horizon examples, MIND-V depicts plausible object interactions and interprets “clean the floor” as grasping a cloth and wiping, unlike baselines that clip through objects or take no action.The paper says GRPO-based physical alignment mitigates dynamic violations, while SRH infers executable action sequences for abstract commands.Source: C Additional Visual Results · p. 15, Figure 11 · p. 16

Complex OOD Scenarios and Diverse Interactive Actions

  • MIND-V’s OOD examples show accurate intent execution in cluttered or stylized scenes and interactions with articulated objects and deformable materials.The paper links stable backgrounds and visual-domain generalization to BSB’s separation of semantic intent from visual appearance.Source: C Additional Visual Results · p. 16

D Network Architecture Details

The MVG renders videos through latent-space diffusion and injects BSB guidance for kinematic control. The section also specifies GRPO settings and a user-study protocol for assessing long-horizon videos.

D Network Architecture Details

  • The MVG uses a 3D VAE and CogVideoX-5B DiT backbone, compressing [B, 3, T, H, W] videos to [B, 16, T/4, H/8, W/8] latents for diffusion before decoding them to pixels.Its 30-block DiT receives BSB guidance as a 1920-dimensional feature, additively fused in even-numbered blocks to steer denoising toward the planned trajectory.Source: D Network Architecture Details · pp. 16–18, Table 8 · p. 18
  • Figure 12 describes OOD generalization in cluttered and stylistically distinct scenes, plus interactions with articulated objects and deformable materials.Examples include grabbing bread or dumplings, picking peaches, opening a cabinet, closing a microwave, and folding a tablecloth.Source: Figure 12 · p. 17

E Key GRPO Hyperparameters

  • GRPO samples 8 videos per prompt, uses ϵ=0.2 and β=0.01, and weights aesthetic and physical rewards at 1.0 and 0.2, respectively.PFC uses τ=0.1 to emphasize the least plausible temporal windows; settings include 1,500 iterations, 50 DDIM steps, and 37-frame videos at 480 × 640 resolution.Source: E Key GRPO Hyperparameters · p. 18

F User Study Protocol

  • User Preference is the percentage of valid selections for the best overall video across instruction completion, physical plausibility, and visual coherence.Thirty volunteers judged randomized subsets of the 108-sample evaluation set using anonymized, randomly ordered videos; incomplete responses were excluded.Source: F User Study Protocol · pp. 18–19

Related papers

Loading 2512.06628v4…