PaperlayerView exact revision on arXiv

Source-linked AI summary

Future Optical Flow Prediction Improves Robot Control & Video Generation

Kanchana Ranasinghe, Honglu Zhou, Yu Fang, Luyu Yang, Le Xue, Ran Xu, Caiming Xiong, Silvio Savarese, Michael S Ryoo, Juan Carlos Niebles

arXiv:2601.10781v1cs.CV

TL;DR

Forecasting reliable, generalizable future motion remains challenging, especially from noisy, real-world video data. FOFPred uses a language-conditioned unified VLM–Diffusion architecture to predict dense optical flow, and evaluations show motion-guided representations enhance language-conditioned robotic control and video synthesis.

  • Problem

    Reliable and generalizable forecasting of future, spatially dense motion representations from sufficiently supervised, cross-domain data remains challenging.

  • Method

    FOFPred predicts language-conditioned dense optical flow with a unified Vision-Language Model and Diffusion architecture trained on web-scale captioned human activity videos.

  • Results

    Evaluations across robotic manipulation and video generation show that motion-guided representations enhance language-conditioned control and synthesis.

  • Takeaways & Limitations

    FOFPred demonstrates the cross-domain utility of dense future optical-flow representations for language-conditioned robotics and video generation.

  • Takeaways & Limitations

    FOFPred is sensitive to text-prompt phrasing and is expensive to deploy in real time because it has approximately 7B parameters and requires at least 24GB of GPU memory for inference.

Abstract

from arXiv · show

Future motion representations, such as optical flow, offer immense value for control and generative tasks. However, forecasting generalizable spatially dense motion representations remains a key challenge, and learning such forecasting from noisy, real-world data remains relatively unexplored. We introduce FOFPred, a novel language-conditioned optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture. This unique combination enables strong multimodal reasoning with pixel-level generative fidelity for future motion prediction. Our model is trained on web-scale human activity data-a highly scalable but unstructured source. To extract meaningful signals from this noisy video-caption data, we employ crucial data preprocessing techniques and our unified architecture with strong image pretraining. The resulting trained model is then extended to tackle two distinct downstream tasks in control and generation. Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred, confirming the value of a unified VLM-Diffusion architecture and scalable learning from diverse web data for future optical flow prediction.

1. Introduction

FOFPred formulates language-conditioned future optical-flow forecasting as a unified VLM–Diffusion problem trained on web-scale human activity videos. The model supports language-driven robotic manipulation and motion-video generation through separately fine-tuned downstream heads.

  • Motivation: Future optical flow and other motion representations explicitly capture desired dynamics for robot control and video generation.Optical flow represents future pixel movements, while trajectories and direction commands provide alternative motion representations.
  • Challenge: Future motion forecasting is challenging because downstream models need future movement information rather than previously observed motion.The challenge includes defining future motion and curating sufficiently large training data with strong supervisory signals.
  • Method: FOFPred forecasts language-conditioned future optical-flow images with a unified VLM–Diffusion architecture trained on web-scale captioned human-activity videos.Optical flow is spatially dense and captures movement for every pixel, while the architecture combines VLM reasoning with pretrained diffusion image-generation ability.
  • Training: FOFPred addresses noisy web-video data by learning from diverse videos and captions, unlike prior approaches that often avoid such data or predict RGB frames.The introduction identifies noisy videos and captions as a central issue in scalable web-data training.
  • Downstream Tasks: The model is evaluated on language-driven robotic manipulation and language-guided motion-video generation using separately fine-tuned diffusion policy and video diffusion heads.This cross-domain setup is presented as the first use of a VLM–Diffusion backbone for both tasks under language-driven settings.

2. Related Work

Related work develops unified multimodal architectures for controlled visual manipulation and explores spatial conditioning for video-generation motion control. However, language-based explicit motion control remains underexplored, motivating FOFPred’s focus.

  • Unified Model for Control Generation: Unified foundation models integrate diverse modalities and support highly controlled visual manipulation across hundreds of tasks.These generalist systems commonly build on transformers and large-scale pre-training, including Gato.
  • Unified Model for Control Generation: Video-focused unified models use spatio-temporal reasoning for action grounding, planning, and motion-aware policy learning.MAGMA uses Trace-of-Mark annotations, while FlowVLA predicts optical flow before the next visual frame to disentangle motion dynamics from static appearance.
  • Motion Control in Video Generation: Video-generation methods control motion through sparse user-provided trajectories, points, or explicit motion models.Examples include DragNUWA, DragAnything, Tora, TrackGo, Controllable Video Generation, MotionCtrl, and Motion-I2V.
  • Motion Control in Video Generation: Language-based explicit motion control remains underexplored, which is the focus of this work.The stated gap concerns controlling motion in video generation through language rather than only spatial conditioning signals.

3. Method

FOFPred generates future optical-flow sequences from images and language using an end-to-end unified VLM–diffusion architecture trained on video-caption pairs without pixel-level annotations. Its RGB-space flow representation and downstream integrations support language-driven robotic control and text-to-video motion generation.

  • Framework overview: FOFPred generates future optical-flow sequences from one or more images and a language instruction, learning from video-caption pairs without pixel-level human annotation.The framework uses a unified LLM-diffusion architecture to generate sequences of future optical-flow frames.
  • Architecture: The model encodes captions and visual frame pairs with a VLM and VAE, projects both features through MLPs, and decodes DiT outputs into future optical-flow sequences.The VLM produces textual features, while the VAE encodes visual inputs and decodes the diffusion-transformer output.
  • Architecture: Temporal modeling modifies OmniGen’s 2D RoPE for input and output frame sequences and adds full spatio-temporal attention without additional learnable parameters.These modifications allow the diffusion transformer to model temporal axes in frame sequences.
  • Optical-flow representation: FOFPred represents optical flow in RGB space by mapping polar magnitude and rotation values to HSV, enabling direct use of existing VAE models.This avoids training or fine-tuning a dedicated VAE for optical flow.
  • Training: Training computes future-flow targets from adjacent future frames, while classifier-free guidance and partial masking vary textual, temporal, and viewpoint conditions.The target is defined as y[i] = F(x_i, x_i+1), and visual observations may include multiple viewpoints.
  • Downstream applications: For downstream use, FOFPred is integrated with a diffusion policy for language-driven robot control and with GWTF in a two-stage text-to-video pipeline.The text-to-video pipeline predicts future flow from an initial frame and motion description, then interpolates it into a dense motion signal.

4. Experiments

Experiments evaluate FOFPred on language-driven robotic manipulation and motion-focused text-to-video generation, followed by ablations of its design choices. FOFPred achieves strong results across CALVIN, RoboTwin 2.0, and SSv2 evaluations, while ablations assess pretraining, architecture, and motion disentangling.

  • Experimental scope: Experiments cover language-driven manipulation on CALVIN and RoboTwin 2.0, motion-controlled text-to-video generation on SSv2, and ablations of FOFPred’s design choices.The ablations examine pretraining, backbone architecture, image-editing pretraining, and motion disentangling.
  • RoboTwin 2.0: FOFPred achieves consistent improvements over the VPP baseline on RoboTwin 2.0’s five selected bimanual manipulation tasks.The evaluation follows the benchmark’s official easy-setting protocol, with results reported as success rates.
  • Text-to-video generation: FOFPred shows consistent generation-quality improvements over the CogVideoX baseline for language-driven motion control on SSv2.The evaluation uses SSv2’s validation split and an identical protocol for FOFPred and the implemented CogVideoX baseline.
  • CALVIN: 0.787 Task 5 Success Rate and 4.48 Average Length of task completion are achieved by FOFPred on CALVIN’s ABC→D benchmark.FOFPred records the highest success rate across all five chained tasks and marginally surpasses DreamVLA’s 4.44 average length.
  • Ablations: Web human-video pretraining, the VLM–diffusion backbone, image-editing pretraining, and motion disentangling are each evaluated for their impact on motion guidance prediction.The full VLM–diffusion model with image-editing pretraining achieves the highest score, while suitable motion-target disentangling is necessary to learn meaningful signals.

5. Conclusion

FOFPred bridges language and motion through dense pixel-level displacement prediction, using disentangled web-video supervision and a unified VLM–diffusion backbone. The model improves language-conditioned control and synthesis while retaining diverse motion generation, but remains sensitive to prompts and costly to deploy.

  • Contributions: FOFPred bridges language and motion through dense, pixel-level displacement prediction.Its disentangling algorithm separates object and camera motion to refine noisy web-video supervision.
  • Contributions: The unified VLM–diffusion backbone supports robust multimodal learning from diverse, noisy caption data.Experiments across robotics and video generation show motion-guided representations enhance language-conditioned control and synthesis.
  • Limitations: FOFPred is sensitive to text prompts, with small wording changes sometimes producing wrong predictions.For example, changing “moving from right to left” to “moving left” results in wrong predictions for some samples.
  • Limitations: The approximately 7B-parameter model is expensive for real-time deployment and requires at least 24GB of GPU memory for inference.The paper identifies model size and inference cost as key limitations.
  • Strengths: FOFPred captures diverse motion patterns and can generate meaningful future optical flow in as little as 1 reverse diffusion iteration.Across seeds for the same frame-caption pair, it produces a diverse distribution of mostly meaningful future optical flow, attributed to diffusion-based training.
  • Future Work: Future work will investigate automated text-label rephrasing, lightweight model distillation, motion diversity, fast convergence, and real-time inference.These directions target the model’s prompt sensitivity, deployment cost, and unresolved diversity and convergence questions.

Supplementary Material · Contributions

The contributions describe project leadership, technical development, evaluation, strategic direction, and research oversight across the team. They also highlight efforts supporting training scalability, robotic downstream tasks, and project organization.

  • Contributions: KR led the project’s preliminary language-driven optical-flow prediction idea, codebase development, and most evaluations.The passage attributes project implementation and evaluation leadership to KR.
  • Contributions: HZ contributed to model architecture, optical-flow calculation, video evaluations, and debugging technical issues.HZ discussed all project aspects and supported several key design and debugging decisions.
  • Contributions: YF supported robotic evaluation pipelines, helped implement RoboTwin evaluations, and performed several robotic b…The supplied passage truncates the final description of YF’s robotic contributions.
  • Contributions: RX streamlined early exploration, supported scaling the training pipeline, and provided critical project-design feedback.These contributions supported both early ideation and later training development.
  • Contributions: CX and SS provided strategic vision, oversaw the research environment, and shaped the high-level framing of the research problem.Their contributions focused on project-level vision and research framing.
  • Contributions: MR directed robotic downstream tasks and discussed the project’s idea, scope, and implementation.MR’s role centered on the robotic application direction and project development discussions.
  • Contributions: JN organized the overall project, set the research direction, and dis…The supplied passage truncates the final description of JN’s contributions.

Appendix Contents … A.2. DiT Modifications for Temporal Modeling

FOFPred combines a 3B-parameter VLM, 83M-parameter VAE, and 4B-parameter DiT, with projected textual, visual, and noise features aligned for diffusion conditioning. Its DiT adapts OmniGen with temporal RoPE and full spatio-temporal attention to model future optical-flow sequences.

  • A. Additional Architectural Details: FOFPred’s architecture contains a 3B-parameter VLM, 83M-parameter VAE, and 4B-parameter DiT, which is the modified and trained core module.The architecture focuses on conditional input processing and DiT modifications.
  • A.1. Conditional Input Processing: Conditional processing uses textual feature fc and visual feature fv, each passed through an MLP to match the DiT channel dimension.The two features provide distinct textual and visual conditioning signals.
  • A.1. Conditional Input Processing: The Qwen2.5-VL textual feature fc starts at 2520 channels and is projected to common dimension D from caption c paired with visual inputs xt−1 and xt.The VLM receives interleaved language and visual inputs.
  • A.1. Conditional Input Processing: The Flux.1 VAE visual feature fv starts at 16 channels, is reshaped into 64 = 16 × 2 × 2 channel vectors, and projected to D.A 2 × 2 grid supplies the reshaping operation before MLP projection.
  • A.1. Conditional Input Processing: The initial VAE noise vector uses the visual reshaping and projection MLP, while projected features ˆfc and ˆfv are appended to the DiT noise sequence.Both MLP outputs are enforced to share D = 2520, the DiT input channel dimension.
  • A.2. DiT Modifications for Temporal Modeling: The DiT adapts OmniGen to explicitly handle temporal sequence prediction for future optical flow ˆyt+1:t+4 from inputs xt−1 and xt.The adaptation targets future optical-flow prediction over the specified input and output sequences.
  • A.2. DiT Modifications for Temporal Modeling: Time-Aware 3D RoPE reinterprets OmniGen’s (L, H, W) axes so Axis 1 encodes temporal offsets or frame indices, while Axes 2 and 3 encode within-frame spatial positions.The temporal index applies to xt−1, xt, and the latent noisy output y.
  • A.2. DiT Modifications for Temporal Modeling: Full spatio-temporal attention operates over fin = [ ˆfc, ˆfv, fy], enabling the DiT to capture motion dynamics and sequence dependencies for future optical-flow frames.The position shift is incremented after each frame so tokens receive temporal timestamps, distinguishing input frames and modeling their sequence.

B. Optical Flow Representation · C. Optical Flow Calculation · D. Motion-Guided Frame Sampling

The method converts dense optical flow into a stabilized HSV-derived RGB representation, compensates camera motion to isolate object-centric flow, and filters video frames using motion-guided sampling. These steps prioritize meaningful motion while reducing outliers, residual artifacts, and computational cost.

  • B. Optical Flow Representation: The flow field F ∈ R^2×H×W is converted from Cartesian components into polar magnitude and angle coordinates.The magnitude is normalized by η = 64.0 and clamped to [0, 1], while θ is shifted into [0, 2π].
  • B. Optical Flow Representation: Polar flow coordinates are mapped to HSV and converted into an RGB image, with hue encoding direction and saturation encoding normalized magnitude.The value channel is fixed at 1 to reduce color variance across consecutive frames caused by outliers.
  • C. Optical Flow Calculation: Camera motion is estimated from dense correspondences formed by uniformly sampled grid points projected with raw flow vectors.The point pairs are used for RANSAC-based homography estimation rather than computationally expensive sparse feature extraction.
  • C. Optical Flow Calculation: Object-centric motion is obtained as F_obj = F_raw − F_cam after synthesizing camera displacement across the coordinate grid.A post-processing magnitude threshold suppresses residual noise and artifacts from imperfect alignment.
  • C. Optical Flow Calculation: The offline preprocessing of 500,000 training videos completes in roughly 30 hours using 4 A100 GPUs.This timing includes motion-guided frame sampling, whose cost is negligible relative to relative flow calculation.
  • D. Motion-Guided Frame Sampling: Motion-guided sampling uses a two-stage filtering pipeline to prioritize high-motion segments before expensive ground-truth label generation.Lucas-Kanade optical flow runs on 32 × 32 spatial grids, and the motion proxy uses the top-k magnitude percentile with k = 10 rather than the mean.

E. Additional Ablations

Additional ablations show that spatially dense future optical flow benefits robot control, while replacing or removing motion-forecast inputs tests their contribution against generic visual representations.

  • Dense vs. Sparse Motion: Dense optical flow is evaluated against sparse motion representations on CALVIN using average episode length under identical training and evaluation settings.The sparse variants include 16 × 16 sub-sampling of predicted flows and an ATM re-implementation trained on the same dataset.
  • Dense vs. Sparse Motion: 4.39 average length is achieved by the default dense model, compared with 2.92 for the ATM sparse baseline on CALVIN.The comparison demonstrates an advantage for spatially dense motion information.
  • Ablations on Motion Forecast Conditioning: Future optical-flow motion inputs improve robot-control performance relative to removing motion or using VAE visual embeddings.The ablation reports average length on the CALVIN benchmark and describes the improvement as decisive.
  • Ablations on Motion Forecast Conditioning: The motion-conditioning ablation compares the full framework with policies that remove motion input or replace it with static VAE visual embeddings.The full robot-control extension uses state, text-goal, and future optical-flow inputs; the no-motion variant retains only state and text-goal inputs.

F. Detailed Limitations

FOFPred’s base model has a seed-sensitive limitation: different diffusion noise vectors can produce unexpected camera motion, sometimes instead of explicitly capturing the prompted object motion.

  • Base-model limitations: The detailed limitation analysis focuses on FOFPred’s base model, defined as the result of large-scale pretraining.The discussion expands the limitations and future directions previously presented in Section 5.1.
  • Seed sensitivity: Four predictions for “Moving the bowl from left to right” show that starting noise vectors can conflate object motion with camera motion.The visualized predictions use four different starting noise vectors for the reverse diffusion process.
  • Seed sensitivity: Across different seeds, FOFPred sometimes generates unexpected camera motion rather than explicitly capturing the desired object motion.This issue is observed in base-model predictions and can occur even when the camera motion is relevant to the prompt.

G. Visualizations

Visualizations compare FOFPred with CogVideoX for first-frame- and text-conditioned video generation on SSv2 examples. The framework improves motion adherence but can distort object appearance despite correct movement.

  • Visual comparison: FOFPred visualizations compare generated videos against ground truth and the CogVideoX T2V baseline using SSv2 validation examples conditioned on first frames and text goals.The showcased goals include moving a glue stick away from the camera, pulling a toy car left to right, and pushing a fidget spinner right to left.
  • Visual comparison: FOFPred consistently improves motion adherence over CogVideoX in the visualized Text-to-Video examples.The comparison uses the two-stage FOFPred–GWTF pipeline, with GWTF extending CogVideoX to accept a user-provided motion prompt.
  • Failure cases: FOFPred can distort object appearance even when objects undergo the correct movement, as illustrated by the toy car example.The figure identifies this as a failure mode alongside successful motion control.
Loading 2601.10781v1…