F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions

1 INTRODUCTION

• Reactive state-to-action policies are short-sighted and brittle because they lack predictive foresight as objects, contexts, and instructions change.The introduction identifies dynamic scenes, ambiguous language, diverse objects, and long-horizon coherence as challenges for reliable performance.Source: §1 · pp. 1–2
• Prior approaches either lack semantic grounding, remain reactive despite VLM understanding, or predict video without semantic grounding, limiting robust long-horizon control.The introduction contrasts end-to-end action experts (Zhao et al., 2023; Chi et al., 2023), VLM-integrated policies (Black et al., 2024; Bjorck et al., 2025), and visual prediction policies (Hu et al., 2024; Liao et al., 2025b).Source: §1 · pp. 1–2, Figure 1 · p. 2
• Inspired by predictive inverse dynamics models (Black et al., 2023; Du et al., 2023), F1 derives actions from current observations and anticipated visual outcomes.Its Mixture-of-Transformer architecture assigns dedicated experts to understanding, foresight generation, and action execution.Source: §1 · p. 2
• Progressive attention enforces causal information flow from understanding to foresight to action, while next-scale prediction generates goal-conditioned visual planning targets.Intra-expert attention captures token interactions; the generation expert synthesizes foresight to guide subsequent control.Source: §1 · pp. 2–3
• F1’s three-stage training aligns the generation expert with a pretrained MLLM, pretrains on public robot datasets, then adapts to task-specific embodiments and skills.The authors state that this progression stabilizes optimization and supports generalizable foresight for dynamic, long-horizon tasks.Source: §1 · p. 3
• F1 is a pretrained 4.2B-parameter VLA model that the authors report improves success rates and robustness over reactive baselines in simulation and real-world tasks.The reported gains are particularly associated with dynamic environments and long-horizon tasks requiring planning and foresight.Source: §1 · p. 3

2.1 ARCHITECTURE OVERVIEW

• F1 uses understanding, generation, and action experts to turn instructions and observations into goal-conditioned visual targets and action chunks via predictive inverse dynamics.The understanding expert combines instruction and visual information; the generation expert conditions foresight on task goals and historical observations, and the action expert predicts actions to reach that visual target.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 UNIFIED UNDERSTANDING-GENERATION-ACTION TRANSFORMER

• The understanding expert aligns language goals with observations, while the generation expert uses next-scale prediction to synthesize an explicit visual target for control.The understanding expert encodes observations with SigLIP and fuses perceptual features with the language prompt. The generation expert encodes recent frames with multi-scale residual VQ, aggregates motion features temporally, and autoregressively generates foresight tokens for decoding.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
• UGA attention makes foresight an explicit causal intermediary: generation attends to understanding, and action attends to both, while information cannot flow backward.Within experts, attention is bidirectional except for the generation expert’s causal, scale-conditioned foresight tokens; the action expert uses chunked prediction and flow matching for coherent short-horizon actions.Source: §2.2 · pp. 4–5

2.3 TRAINING RECIPE

• F1 trains on more than 320k trajectories across 136 tasks and 5 embodiments in three stages: align generation, pretrain the full model, then adapt to task-specific demonstrations.Stage I trains the generation expert against the pretrained understanding expert; Stage II uses public robot datasets, and post-training adapts all experts to task-specific demonstrations.Source: §2.3 · p. 5
• Autoregressive next-scale prediction improves generation stability and long-horizon coherence, while flow matching models continuous action generation and joint optimization supports generalization across tasks and embodiments.The later stages generate each foresight token conditioned on previously predicted tokens and train a vector field transforming Gaussian noise toward expert actions.Source: §2.3 · pp. 5–6

2.4 IMPLEMENTATION DETAILS

• F1 combines PaliGemma-style understanding and Gemma-based generation and action experts, initializes understanding and action from π0, and uses a pretrained residual VQ-VAE for image quantization.Its approximately 330k-episode corpus spans 136 tasks and diverse skills and episode lengths; inference restricts foresight prediction to 4 scales for efficient real-time control.Source: §2.4 · p. 6

3.1 COMPARISON ON THE REAL-WORLD TASKS

• F1 averages 92.6% grasp and 82.2% task success, versus π0’s 78.5% grasp and 65.2% success across nine real-world tasks.On Handover (R2H), F1 reaches 93.3% success, compared with π0 at 40% and gr00t-N1 at 13.3%.Source: Table 1 · p. 7, §3.1 · p. 7

3.2 COMPARISON ON THE SIMULATION BENCHMARK

• F1 outperforms comparison methods across all LIBERO suites, with a pronounced advantage over reactive methods on LIBERO-Long, and substantially outperforms other models on SimplerEnv Bridge.The reported benefits are associated with foresight-guided planning on LIBERO and stronger generalization to precise pick-and-place positions on SimplerEnv Bridge.Source: Table 2 · p. 8, §3.2 · p. 8, Table 3 · p. 8

3.3.1 RESULTS ON THE SIMULATION BENCHMARK

• On the simulation ablations, freezing the generation expert lowers performance from 77.5% to 73.8%, while removing it lowers performance from 77.5% to 60.3%.The findings support adapting the generation expert during later training and retaining visual foresight for planning.Source: §3.3.1 · pp. 9–10
• Removing Stage II pretraining causes an approximately 3.3% overall performance drop, and four foresight scales provide the reported optimal balance against two- and six-scale variants.The ablations attribute a useful inductive prior and optimization stability to robotics pretraining; the four-step horizon is reported to avoid unnecessary noise or computational overhead.Source: §3.3.1 · pp. 9–10, Table 4 · p. 9

3.3.2 ABLATION STUDIES ON THE REAL-WORLD TASKS

• On real-world ablations, F1 outperforms No-Gen on Handover (R2H) and Mixture, achieving 93.3% and 73.3% success versus 40.0% and 60.0%, respectively.F1 also performs better than Cotrain-Scratch on many tasks; the passages attribute benefits to the generation expert and Stage II pretraining.Source: §3.3.2 · p. 10, Figure 5 · p. 10

3.4.1 DYNAMIC ENVIRONMENT

• In continuous dual-arm dynamic grasping, F1 achieves 66.7% success; on Lettuce and Bread it reaches 80.0% each, versus π0’s 53.3% and 46.7%.The test uses a moving-belt kitchen setup and an ARX LIFT II embodiment absent from pretraining, with only 47 post-training demonstrations.Source: §3.4.1 · p. 11, Figure 7 · p. 11

3.4.2 ADAPTATION LEARNING

• On Franka sweep and sort tasks, F1 reports higher success, fewer attempts, and fewer empty sweeps than π0, and higher success on the second and third consecutive grasps.The passage links fewer empty sweeps to improved spatial grounding and describes performance sustained across repeated interactions.Source: §3.4.2 · p. 11, Table 5 · p. 12

3.4.3 LONG-HORIZON TASK

• Across a ten-step, approximately two-minute task, F1 achieves near-perfect performance early and maintains meaningful success later, while π0 struggles beyond the first four stages.F1’s performance declines as the sequence progresses, which the passage attributes to error accumulation and increasing temporal difficulty.Source: §3.4.3 · pp. 11–12, Table 6 · p. 12

4.1 QUANTITATIVE ANALYSIS OF GENERATION

• F1 evaluates generated frames for actionable guidance rather than visual realism alone, using a multimodal protocol based on Qwen2.5-VL-32B-Instruct.The evaluator receives the task instruction, four historical robot frames, one predicted next-step frame, and the ground-truth frame.Source: §4.1 · p. 12
• F1’s binary generation-quality scores show scene consistency improving rapidly, while task progress following improves steadily and often exceeds object consistency.The evaluation scores scene consistency, object consistency, and task progress following; the authors attribute weaker object consistency to the lack of large-scale object-centric pretraining.Source: §4.1 · p. 13, Figure 8 · p. 13

4.2 QUALITATIVE ANALYSIS OF GENERATION

• Generated foresight frames capture plausible task progress across manipulation examples, but struggle with fine-grained details and deformable objects.Despite these fidelity limitations, the frames generally convey enough task progression to guide downstream control.Source: §4.2 · p. 13, Figure 9 · p. 14

4.3 CORRELATION BETWEEN GENERATION AND ACTIONS

• Image token accuracy (around 40–45% on average) positively correlates with action token accuracy across LIBERO at τ=0.01, 0.02, and 0.05, despite high action accuracy.The results indicate that pixel-level image prediction is not required for effective planning, while improving visual token prediction may enhance action reliability.Source: §4.3 · pp. 14–15, Figure 10 · p. 15

5.1 VISION LANGAUGE ACTION MODEL

• Most VLA models predict actions reactively, limiting robustness in dynamic, long-horizon tasks despite temporal-memory and reinforcement-learning efforts.F1 contrasts this with integrating visual foresight into decision-making.Source: §5.1 · p. 15

5.2 INVERSE DYNAMICS MODEL

• Prior methods add domain-specific intermediate representations or predict future images and videos, but rarely use visual guidance at inference; F1 conditions actions on predicted foresight.Inverse dynamics extracts actions from consecutive images, easing image-to-action mapping.Source: §5.2 · p. 15

5.3 UNIFIED VISION LANGUAGE MODEL

• Unified vision models use discrete tokens, modular MLLM-diffusion assemblies, or MoT experts, yet retain information loss, weak unification, latency, or external-encoder reliance and omit action.F1 extends the unified paradigm to understanding, generation, and action.Source: §5.3 · p. 16

6 CONCLUSION AND FUTURE WORK

• F1 combines goal-conditioned visual foresight and multimodal grounding in a Mixture-of-Transformer VLA with experts for understanding, foresight generation, and action execution, trained through progressive alignment, pretraining, and adaptation.Its next-scale prediction mechanism and progressive attention regulate information flow across modules.Source: §6 · p. 16
• Across simulation benchmarks and physical platforms, F1 consistently surpasses reactive baselines with higher success rates and improved generalization in dynamic, long-horizon tasks.Source: §6 · p. 16
• Future work includes testing diverse embodiments and task families, adding structured world models or physics-informed priors, integrating reinforcement learning or online adaptation, and incorporating human feedback or interactive correction.Source: §6 · p. 16

A DATASET DETAILS

• Stages I–II primarily use internet datasets for broad behavior and visual-dynamics coverage, while Stage III adapts skills with higher-quality in-house demonstrations.Post-training data covers handover, sweeping, sorting, kitchen activities, and long-horizon manipulation, using Genie-G1, Franka, and ARX LIFT II.Source: A DATASET DETAILS · p. 24
• 330.9K trajectories and 73.8M frames span five robot embodiments, third-person and wrist/head views, and 3–30 FPS.The corpus combines internet-scale datasets with curated in-house demonstrations.Source: Table 7 · p. 24, A DATASET DETAILS · p. 24

• Stage I learns general visual representations with batch size 1280, learning rate 3.0 × 10−4, and 512K training steps.Source: B TRAINING DETAILS · p. 24
• Stage II introduces action prediction for 100K steps, using batch size 2880, learning rate 5.0 × 10−5, and a 0.1:1 generative-to-action loss weight.Source: B TRAINING DETAILS · p. 24
• Stage III finetunes on downstream tasks with shared batch size 128 and common learning-rate and loss-weight settings, while training duration and Action Chunk Size vary by task.Source: B TRAINING DETAILS · p. 24

• The task suite covers multiple robotic platforms and ranges from basic pick-and-place operations to complex long-horizon sequences.Tasks are categorized by core manipulation requirements and complexity levels.Source: C REAL-WORLD TASK DETAILS · p. 24
• Evaluations test precision manipulation, dual-arm coordination, human-robot interaction, and dynamic adaptation.Source: C REAL-WORLD TASK DETAILS · p. 24

C.1 BASIC PICK-AND-PLACE MANIPULATION

• Tasks include grasping household objects and placing them in target containers, such as moving a pen to its holder or a bag of chips to a basket.The figure presents examples of these fundamental pick-and-place tasks across different object types and target containers.Source: §C.1 · p. 25, Figure 11 · p. 25
• Challenges include handling rigid, smooth, and deformable objects while maintaining accurate placement across receptacles with varying constraints.Examples include pens, bags of chips, and plastic bottles.Source: §C.1 · p. 25

C.2 FINE-GRAINED PRECISION MANIPULATION

• Fine-grained placement tests precise grasping of a delicate flower stem and accurate insertion into a vase’s narrow opening.The constrained placement must avoid damaging the flower.Source: §C.2 · p. 25, Figure 12 · p. 26

C.3 DUAL-ARM COORDINATION AND HUMAN-ROBOT INTERACTION

• Bimanual tasks assess spatial-temporal synchronization and inter-arm object transfer through coordinated sequences involving both robotic arms.Examples include picking up a bag of bread and handing it over before placing it in a basket.Source: §C.3 · p. 26, Figure 13 · p. 26
• Human-robot interaction tasks include handing a bottle of black tea to a person, requiring coordinated manipulation and handover.Source: §C.3 · p. 26, Figure 13 · p. 26

C.4 DYNAMIC ENVIRONMENT ADAPTATION

• Dynamic tasks test real-time tracking and interception of moving targets, including acquiring food items from a moving belt.The lettuce-and-bread task requires trajectory prediction and real-time interception.Source: §C.4 · p. 26, Figure 14 · p. 27
• Dynamic task execution also requires responding to unexpected events, such as falling objects or changing environmental conditions.Source: §C.4 · p. 26

C.5 LONG-HORIZON SEQUENTIAL MANIPULATION

• Long-horizon tasks require planning over 10-step sequences, memory management, coordinated tool use, and replanning when unexpected events arise.They also span objects with rigid, deformable, and liquid properties.Source: §C.5 · p. 26, Figure 15 · p. 27
• Long-horizon manipulation also tests sequential coordination of multiple tools and task coherence across extended operation chains.Source: §C.5 · p. 26, Figure 15 · p. 27

D DEPLOY PLATFORM AND LATENCY ANALYSIS

• Foresight generation and action decoding contribute most of the runtime, while image preprocessing and encoding remain relatively lightweight.The latency breakdown is reported for three synchronized camera views.Source: D DEPLOY PLATFORM AND LATENCY ANALYSIS · p. 28, Table 9 · p. 28
• Wired Ethernet connects the robots to the host machine, so reported latency excludes wireless transmission delays.The setup is intended to make measured latency primarily reflect model computational overhead.Source: D DEPLOY PLATFORM AND LATENCY ANALYSIS · p. 28, Table 9 · p. 28
• Approximately 235ms is F1’s total inference time on an Intel i9 CPU and RTX 4090 GPU with wired Ethernet, excluding wireless transmission delays.The measurements use three synchronized camera views and primarily reflect model computation.Source: Table 9 · p. 28, D DEPLOY PLATFORM AND LATENCY ANALYSIS · p. 28

E PROMPT TEMPLATE

• The evaluation prompt specifies the task instruction, four historical frames, the predicted next-step frame, and the ground-truth frame.It is used to evaluate foresight image quality with Qwen2.5-VL-32B-Instruct (Bai et al., 2025).Source: E PROMPT TEMPLATE · p. 28

Prompt Template

• The evaluator assigns binary scores for scene consistency, object consistency, and task progress following.These are the three aspects used to assess generated future-observation quality.Source: E PROMPT TEMPLATE · p. 28, Figure 16 · p. 29

E PROMPT TEMPLATE

• The evaluator is instructed to output three 0/1 numbers together with a brief explanation.Source: E PROMPT TEMPLATE · p. 28