FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Model

1 INTRODUCTION

• Existing interactive world models rely on extra datasets or task-specific action labels, or cannot represent subtle whole-scene movements such as dexterous pen spinning.A scalable action representation should cover movements across diverse objects, robots, and tasks while being easy to obtain or label at scale.Source: §1 · p. 1
• Image flow represents pixel-level changes across objects and robots, captures subtler movement than language, and can be obtained from video using off-the-shelf trackers.Prior work also uses flows to train low-level manipulation policies.Source: §1 · p. 2
• FLIP plans from an initial image and language goal using flow proposals, flow-conditioned future-video generation, and a vision-language value module to search long-horizon plans.The value module evaluates generated videos, and tree search synthesizes plans that maximize discounted returns.Source: §1 · p. 2
• Experiments report model-based planning across simulation and real-world manipulation, long-horizon video synthesis, low-level policy guidance, and advantages over module baselines.The reported settings include LIBERO, FMB, cloth folding and unfolding, and Bridge-V2; the world model also demonstrates interactive properties, zero-shot transfer, and scalability.Source: §1 · p. 2
• The three FLIP modules comprise a new flow generator, a new flow-conditioned video generator, and a training method for an existing vision-language representation model.The action and dynamics modules use CVAE and DiT architectures, respectively, while the value module uses LIV.Source: §1 · p. 2

2.1 WORLD MODELS FOR DECISION MAKING

• Earlier world models learned dynamics in low-dimensional states or planned in latent space; modern model-based reinforcement learning emphasizes latent-space imagination with coupled dynamics and action modules.Recent work also uses scalable video-generation architectures and large datasets to simulate interactive environments.Source: §2.1 · p. 3
• FLIP differs by using separate flow-centric action and dynamics modules alongside a vision-language value model for manipulation planning.This is the authors’ positioning relative to the described world-model approaches.Source: §2.1 · p. 3

2.2 FLOW AND VIDEO MODELS FOR MANIPULATION

• In manipulation research, flows have supported policy guidance, correspondence, tool use, and cross-embodiment representations, while videos have supported learning several representations and visual simulation.FLIP bases its action, dynamics, and value modules on video and language inputs.Source: §2.2 · p. 3

3.1 PROBLEM FORMULATION

• A manipulation task is a goal-conditioned POMDP with image-space observations and sparse goal-conditioned rewards; the learned world model plans, while a low-level policy executes.The task objective is to maximize return.Source: §3.1 · p. 3

3.2 FLOW GENERATION NETWORK AS ACTION MODULE

• The action module proposes future query-point flows for sampling-based planning, using uniformly sampled dense grids tracked over short clips and a CVAE that predicts relative displacements.The CVAE separately predicts delta scale and direction at each query point for flow reconstruction.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 FLOW-CONDITIONED VIDEO GENERATION NETWORK AS DYNAMICS MODULE

• The dynamics module generates short future video clips conditioned on image history, language, and predicted flow, using cross-attention for flow and history and AdaLN-Zero for language and timestep.The observed image history remains unnoised and is not denoised during diffusion.Source: Figure 2 · p. 4, §3.3 · p. 5

3.4 VISION-LANGUAGE REPRESENTATION LEARNING AS VALUE MODULE

• LIV supplies goal-conditioned frame values, but imperfect long videos produce jagged fine-tuned curves; treating fixed-length clips as states makes the curves smoother.The method replaces adjacent frames with adjacent video-clip states in the fine-tuning objective.Source: §3.4 · pp. 5–6, Figure 3 · p. 5

4.1 MODEL-BASED PLANNING WITH FLOWS, VIDEOS, AND VALUE FUNCTIONS

• Because autoregressive long-horizon video generation is often inaccurate from compounding errors, FLIP hill-climbs over flow proposals and corresponding short videos, using value scores to extend beams.A constant pre-goal reward of −1 favors shorter plans, and periodically replacing the lowest-value beam with the highest discourages exploitative routes.Source: §4.1 · p. 6

4.2 PLAN-CONDITIONED LOW-LEVEL POLICY

• The low-level policy conditions on observation history, language goal, predicted flow, and video plan to predict robot actions, and is trained on a few action-labeled demonstrations.The policy executes the plan in the environment.Source: §4.2 · p. 7

5.1 MODEL-BASED PLANNING FOR MANIPULATION TASKS

• FLIP outperforms UniPi and FLIP-NV across tested manipulation tasks; flow guidance improves over direct long-horizon video synthesis, and value guidance removes incorrect search routes.Figure 4 shows generated plans for LIBERO-LONG, FMB, cloth folding, and cloth unfolding.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 LONG-HORIZON VIDEO GENERATION EVALUATION

• Across evaluation datasets, FLIP consistently outperforms video-generation baselines; flow-guided concatenation of short-horizon videos supports long-horizon synthesis.FMB results are worst for all methods, partly because training videos contain discontinuous stage transitions; history observations can sometimes bridge these gaps.Source: §5.2 · p. 8

5.3 PLAN-GUIDED LOW-LEVEL POLICY

• Plan-guided policies outperform diffusion policies and ATM-DP; Ours-FV has the best average success rate, while video-only Ours-V is competitive but high-variance.Adding flow conditions to video guidance is associated with lower variance than video-only guidance.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 EXPERIMENTS ON FUNDAMENTAL MODULES OF FLIP

• For flow generation, predicting scale and direction beats ATM and absolute-coordinate prediction, while auxiliary losses help; FLIP-SC generally matches IRASim on short-horizon video generation.The dynamics results indicate that dense flow conditions still require a fine-grained mechanism to be used effectively.Source: §5.4 · pp. 9–10, Table 4 · p. 9, Table 3 · p. 9
• On Language-Table and cloth folding, FLIP fine-tuning produces smoother value curves than the original LIV method.The original fine-tuned curves show sharp oscillations.Source: Figure 6 · p. 9, §5.4 · p. 10

5.5 APPLICATIONS AND SCALING

• When humans specify image flows for a robot arm, the dynamics module generates videos showing the arm moving in the specified direction.The demonstrations use SAM2 to select the arm region and manually provide flows.Source: §5.5 · p. 10, Figure 8 · p. 10
• Without fine-tuning, FLIP generates natural robot-arm movement for unseen observations and instructions on LIBERO-LONG.The authors describe this as a degree of knowledge transfer.Source: §5.5 · p. 10, Figure 9 · p. 10
• Increasing model size consistently improves the action module’s smoothed ADE and the dynamics module’s smoothed Latent L2 loss on validation data.Figure 10 reports these two validation metrics.Source: §5.5 · p. 10, Figure 10 · p. 10

6 CONCLUSION AND LIMITATION

• FLIP uses a trained world model to synthesize long-horizon plans and guide low-level policy learning from video and language data.The authors describe its potential to scale with larger data and computation budgets.Source: §6 · p. 10
• Extensive video generation makes planning slow and restricts FLIP to quasi-static manipulation tasks.The authors identify planning speed as a major limitation.Source: §6 · p. 10
• FLIP does not use scene physical properties or 3D information; the authors propose extending physical 3D world models to 3D scenarios.Source: §6 · p. 10

A.1 LATENT DIFFUSION MODELS

• The diffusion formulation adds noise over timesteps and learns reverse denoising, while the model also learns covariance with the full KL loss.Source: §A.1 · p. 16
• Latent diffusion compresses frames to VAE latents, tokenizes patches into video tokens, and decodes denoised latents back to pixels.The video tokens have shape T × P × D.Source: §A.1 · p. 16
• Spatial-temporal attention applies spatial attention within each frame and temporal attention across timesteps at the same location.Source: §A.1 · p. 16

A.2 LANGUAGE-VISION REPRESENTATION

• The original LIV fine-tuning loss is computed on sampled sub-trajectory batches from each task, with s ≤ k < T_i.Source: §A.2 · p. 16

A.3 LOW-LEVEL POLICY

• Low-level policies use observation history, language, predicted flow, and future video to predict robot actions, executing 8 steps before replanning.The policy predicts 16-step action sequences using 500 denoising steps.Source: §A.3 · p. 16
• Ours-FV is the illustrated policy architecture; Ours-F and Ours-V use the same architecture without their corresponding input conditions.Source: Figure 11 · p. 17

• All models use an observation history of 16 and a future-flow horizon of 16.Source: §B.1 · p. 17
• Table 5 reports the CVAE hyperparameters.Source: Table 5 · p. 17
• Table 6 reports the dynamics-module hyperparameters.Source: Table 6 · p. 17

• Pretraining on other tasks may not bring significant improvement for low-level policy learning in this comparison.The passage relates this pattern to LIBERO lifelong-learning results, where pretraining did not help and sometimes hurt policy training.Source: §B.2 · p. 18
• The original OpenVLA paper fine-tuned its pretrained model on LIBERO-LONG tasks, reporting a 53.7 ± 1.3% success rate at 256×256 image resolution.Source: §B.2 · p. 18
• The authors suggest that 128×128 resolution may miss scene details and that official demonstrations may be weaker than OpenVLA's recollected demonstrations.Source: §B.2 · p. 18
• Ours-90 performs similarly to Ours-F and Ours-FV, while OpenVLA cannot handle LIBERO-LONG long-horizon tasks in zero-shot or fine-tuned settings.Source: Figure 12 · p. 18, §B.2 · p. 18

• CVAE performs better on LIBERO-LONG while diffusion performs better on Bridge-V2, with only minor performance differences.Source: §B.3 · p. 18
• The reported main action-module improvements come from predicting flow deltas as scale and direction rather than absolute coordinates, plus auxiliary losses.Source: §B.3 · p. 18
• Diffusion inference is slower than CVAE inference, so CVAE remains the default despite diffusion performing better on some tasks.Source: §B.3 · p. 18

• Real-world tests use a 6-DOF X-arm with two RealSense D435i cameras, 10 Hz control, and 20 randomly initialized episodes per task in a tabletop setting.One camera is on another robot arm as the third-view camera, and the other is on the robot arm’s wrist as the eye-in-hand camera.Source: §B.4 · pp. 18–19
• Tea scooping requires spoon use and tea transfer, while cloth unfolding requires flattening randomly thrown denim shorts within 15 grasps and drags.The tea task combines precise grasping, tool use, and rigid-body and granular-object interaction; the cloth task uses a projection-area success threshold above 85% of maximum area.Source: §B.4 · pp. 19–20
• The real-world policy significantly outperforms baselines across both tasks, despite using no more than 50 demonstrations per task.The authors attribute improved success rates to model-based planning that can correct deviations back toward a good trajectory.Source: §B.4 · p. 20, Table 8 · p. 20

B.5 DATA SCALABILITY

• LIBERO-LONG planning success rates improve as the number of demonstrations per task increases.Table 9 reports planning results across different training-data amounts.Source: §B.5 · p. 21, Table 9 · p. 21

B.6 GENERATIVE PLANNING WITH VISUAL DISTRACTION

• With an apple added to the initial image, video generation eventually distorts cups and grays the image, while flow generation performs well before that failure.The reported visual-distraction experiments describe flow as more resistant to distraction than video generation.Source: §B.6 · p. 21, Figure 16 · p. 21
• When the apple is placed away from the target object, generative planning can continue for more steps.Source: Figure 17 · p. 22