Source-linked AI summary
F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions
Qi Lv, Weijie Kong, Hao Li, Jia Zeng, Zherui Qiu, Delin Qu, Haoming Song, Qizhi Chen, Xiang Deng, Jiangmiao Pang
TL;DR
F1 targets the fragility of reactive VLA policies in dynamic and long-horizon visual environments. It integrates semantic understanding, goal-conditioned visual foresight, and action execution through a Mixture-of-Transformer architecture and three-stage training recipe, and experiments report improved success rates, robustness, and transfer across real-world and simulated tasks.
Problem
Reactive VLA policies struggle with short-sighted behavior and robustness when scenes, objects, and instructions evolve over time.
Method
F1 combines understanding, foresight generation, and action execution using predictive inverse dynamics, next-scale visual prediction, progressive attention, and three-stage training.
Results
Experiments on real-world and simulation benchmarks report that F1 improves success rates, robustness, and generalization compared with reactive baselines, especially in dynamic and long-horizon tasks.
Takeaways & Limitations
The study provides evidence that coupling predictive foresight with multimodal grounding can produce policies better suited to dynamic and long-horizon scenarios.
Takeaways & Limitations
F1 has limited visual fidelity for fine-grained details and deformable objects because it was not pretrained on large-scale generative datasets.
Abstract
from arXiv · showhide
Executing language-conditioned tasks in dynamic visual environments remains a central challenge in embodied AI. Existing Vision-Language-Action (VLA) models predominantly adopt reactive state-to-action mappings, often leading to short-sighted behaviors and poor robustness in dynamic scenes. In this paper, we introduce F1, a pretrained VLA framework which integrates the visual foresight generation into decision-making pipeline. F1 adopts a Mixture-of-Transformer architecture with dedicated modules for perception, foresight generation, and control, thereby bridging understanding, generation, and actions. At its core, F1 employs a next-scale prediction mechanism to synthesize goal-conditioned visual foresight as explicit planning targets. By forecasting plausible future visual states, F1 reformulates action generation as a foresight-guided inverse dynamics problem, enabling actions that implicitly achieve visual goals. To endow F1 with robust and generalizable capabilities, we propose a three-stage training recipe on an extensive dataset comprising over 330k trajectories across 136 diverse tasks. This training scheme enhances modular reasoning and equips the model with transferable visual foresight, which is critical for complex and dynamic environments. Extensive evaluations on real-world tasks and simulation benchmarks demonstrate F1 consistently outperforms existing approaches, achieving substantial gains in both task success rate and generalization ability.
: A VISION-LANGUAGE-ACTION MODEL BRIDGING UNDERSTANDING AND GENERATION TO ACTIONS
The paper lists Qi Lv, Weijie Kong, and Hao Li among its authors.
- Qi Lv is listed as the first author.
- Weijie Kong is listed as the second author.
- Hao Li is listed as the third author.
1 INTRODUCTION
F1 addresses the limitations of reactive manipulation policies by combining semantic understanding, visual foresight, and action execution in one framework. Its predictive architecture and progressive training recipe target robustness and generalization in dynamic, long-horizon tasks.
- Related paradigms: Earlier policies either lack semantic grounding and task generalization, remain temporally reactive despite using VLMs, or predict observations without VLM-based semantic grounding.
- Motivation: Reactive state-to-action mappings can produce short-sighted and fragile behavior in dynamic, complex manipulation tasks.
- F1 framework: F1 uses predictive inverse dynamics to derive actions from current observations and anticipated visual outcomes.
- Architecture: F1 combines understanding, foresight generation, and action execution through a Mixture-of-Transformer architecture with progressive attention and next-scale prediction.Next-scale prediction synthesizes goal-conditioned visual foresight as explicit planning targets for control.
- Training strategy: The three-stage recipe aligns foresight with a pretrained multimodal language model, learns shared visuomotor knowledge, and adapts to embodiments and fine-grained skills.The stages are designed to stabilize optimization and provide transferable foresight.
- Evaluation: F1 improves success rates and robustness over reactive baselines in real-world and simulation benchmarks, particularly for dynamic and long-horizon tasks.The model is presented as a pretrained VLA model with 4.2B parameters.
2 THE F1 FRAMEWORK: BRIDGING PERCEPTION, FORESIGHT, TO ACTIONS
F1 bridges understanding, visual foresight generation, and action execution through three specialized experts in a Mixture-of-Transformer architecture. Its training progressively aligns these experts and uses autoregressive foresight plus flow-matching action prediction for goal-directed control.
- Unified Understanding-Generation-Action Transformer: F1 uses understanding, generation, and action experts to connect language and observations with foresight images and executable actions.The understanding expert builds semantically aligned representations, the generation expert predicts future visual states, and the action expert uses foresight to predict short-horizon actions.
- Unified Understanding-Generation-Action Transformer: The generation expert predicts goal-conditioned future images from current observations, language goals, and historical observations.Residual VQ encoding and temporal aggregation provide compact inputs for autoregressive visual-token generation, which is decoded into a future image.
- Unified Understanding-Generation-Action Transformer: UGA progressive attention enforces causal information flow from understanding to generation to action, preventing action information from leaking backward into foresight.Intra-expert attention supports token interactions, while inter-expert attention preserves the understanding–foresight–action hierarchy.
- Training Recipe: F1 trains through staged alignment, broad visuomotor pretraining, and task-specific post-training for embodiment adaptation.Stage I aligns generation with pretrained understanding; Stage II jointly pretrains all experts; the post-training stage adapts the model to task-specific demonstrations.
- Training Recipe: Autoregressive next-scale prediction improves training–inference consistency, while flow matching connects foresight-conditioned context to continuous robot actions.The autoregressive formulation supports generation stability and long-horizon coherence; flow matching transforms Gaussian noise toward target actions.
3 EXPERIMENTS
F1 is evaluated through simulation benchmarks, real-world platforms, and ablations targeting foresight, training, and planning-scale choices. Across these settings, it generally outperforms baselines, while component studies show that visual foresight, joint optimization, pretraining, and a four-step horizon contribute to performance.
- Real-World Tasks: 92.6% average grasp rate and 82.2% average success rate are achieved by F1 across nine Genie real-world tasks.The best baseline, π0, achieves 78.5% grasp and 65.2% success rates; each model is evaluated 15 times per task.
- Simulation Benchmarks: F1 is evaluated on LIBERO and SimplerEnv Bridge, which test spatial reasoning, long-horizon planning, fine-grained control, and precise interaction.LIBERO reports Success Rate and Ranking; SimplerEnv Bridge reports Grasp Success and Overall Success.
- Simulation Benchmarks: F1 consistently achieves superior performance across LIBERO suites and substantially outperforms other models on SimplerEnv Bridge.The SimplerEnv Bridge comparison reports a marked improvement in average success rate over the next-best baselines.
- Ablation Studies: Jointly training the generation expert improves performance from 73.8% for Frozen-Gen to 77.5% for the full model, while removing pretraining causes an approximately 3.3% drop.The studies compare end-to-end adaptation with a frozen generation expert and large-scale robotics pretraining with training from scratch.
- Ablation Studies: Removing the generation expert drops performance from 77.5% to 60.3%, showing that explicit visual foresight is important for planning and goal alignment.The No-Gen variant removes the entire visual foresight branch and produces a more reactive policy.
4 RELATIONSHIPS BETWEEN GENERATION QUALITY AND ACTIONS
F1 evaluates visual foresight by its task relevance and examines whether generation quality aligns with action reliability. The results show that imperfect visual predictions can still provide useful planning cues, while better token prediction tracks stronger action accuracy.
- Generation-quality evaluation: The multimodal evaluation scores scene consistency, object consistency, and task progress following to measure whether predicted frames support downstream control.Each dimension is scored as 0/1 and aggregated into a task-relevant generation-quality measure.
- Generation-quality evaluation: F1’s generation quality develops hierarchically: scene consistency improves early, while object consistency and task progress following are more challenging.The evaluation is designed to assess actionable guidance rather than pixel-level realism alone.
- Qualitative analysis: Qualitative examples show plausible next-step frames aligned with ground-truth trajectories across supermarket manipulation and clothing-folding tasks.The predicted frames capture task-level plausibility, supporting the generation expert’s role as a visual planner.
- Qualitative analysis: F1 has limited visual fidelity for fine-grained details and deformable objects, including shopping carts, plastic bags, and clothing.The paper attributes this weakness partly to the absence of large-scale generative pretraining, while noting that essential task progression remains informative.
- Correlation between generation and actions: Across four LIBERO suites and τ=0.01, 0.02, 0.05, image token accuracy consistently positively correlates with action token accuracy.Average image token accuracy remains around 40–45%, yet generated foresight still provides task-relevant cues associated with high action token accuracy.
- Correlation between generation and actions: Modest image token accuracy does not preclude effective action planning, while improving visual token prediction remains a promising route to greater action reliability.The analysis treats pixel-level precision as unnecessary for useful control but links better visual prediction with improved downstream action performance.
5 RELATED WORK
Related VLA research has progressed from reactive action policies toward models with multimodal understanding and auxiliary future representations. However, existing approaches still lack a unified combination of semantic grounding, temporal modeling, and foresight-driven control.
- Reactive policies: Early manipulation policies map observations directly to low-level actions but lack semantic grounding and generalization across tasks and embodiments.These approaches represent a purely reactive paradigm.
- Prediction and unified models: Visual prediction-based policies anticipate future observations as auxiliary signals, but without VLM semantics their predictions lack semantic grounding.Unified understanding-and-generation models address related integration challenges but may suffer information loss or weakened semantics from discrete visual tokenization.
6 CONCLUSION AND FUTURE WORK
The paper concludes that F1 couples goal-conditioned visual foresight with multimodal grounding through modular experts and progressive training. Experiments support improved success rates and generalization, while broader embodiments, world models, adaptation, and human feedback remain open directions.
- Conclusion: F1 uses three dedicated experts for understanding, foresight generation, and action execution within a Mixture-of-Transformer architecture.Next-scale prediction and progressive attention regulate information flow across the modules.
- Conclusion: F1 consistently surpasses reactive baselines with higher success rates and improved generalization in dynamic and long-horizon tasks.The evidence comes from experiments across simulation benchmarks and physical platforms.
- Conclusion: Coupling predictive foresight with multimodal grounding is presented as a viable direction for robust visuomotor control.The unified framework is intended to reduce dependence on purely reactive mappings and support transfer across tasks and embodiments.
- Future work: Future work targets broader embodiments and task families, structured or physics-informed world models, reinforcement learning, online adaptation, and human feedback.These directions are proposed to test generality, improve long-horizon reasoning, enable continual refinement, and better align agents with human intentions.
A DATASET DETAILS
F1 is trained on a heterogeneous corpus combining internet-scale robot data with curated in-house demonstrations. The three-stage recipe uses broad datasets for pretraining and higher-quality demonstrations for task-specific adaptation.
- Data statistics: The combined training corpus contains 330.9K trajectories and 73.8M frames across five robot embodiments, multiple viewpoints, and 3–30 FPS frame rates.Embodiments include Genie-G1, Franka, WidowX, Google Robot, and ARX LIFT II.
- Training stages: Pretrain Stages I–II use Agibot-World, OXE-Fractal, OXE-Bridge-v2, and LIBERO to provide broad manipulation and visual-dynamics coverage.These are internet-scale datasets used before task-specific adaptation.
- Training stages: Post-train Stage III adapts F1 to specific skills using a smaller, higher-quality collection of in-house demonstrations.The demonstrations span diverse tasks and settings.
B TRAINING DETAILS
F1 is trained through three stages that progress from general visual representation learning to action prediction and downstream-task finetuning.
- Training stages: Stage I learns general visual representations using a batch size of 1280, learning rate 3.0 × 10−4, and 512K training steps.Stage I uses a large-scale representation-learning setup.
- Training stages: Stage II adds action prediction with batch size 2880, learning rate 5.0 × 10−5, 100K steps, and a 0.1:1 generative-to-action loss weight.The loss weighting balances generative and action objectives.
- Training stages: Stage III finetunes F1 on downstream tasks, adjusting training steps or epochs and Action Chunk Size for data volume and task difficulty.The stage uses a smaller batch size of 128 and task-specific settings.
C REAL-WORLD TASK DETAILS
The real-world evaluation spans multiple robotic platforms and manipulation tasks organized by capability and complexity, while downstream training settings vary across tasks.
- Evaluation scope: The task suite ranges from basic pick-and-place to long-horizon sequences, covering precision manipulation, dual-arm coordination, human-robot interaction, and dynamic adaptation.These categories test progressively more complex manipulation requirements.
- Task-dependent training: Downstream Stage III settings are not uniform because task demonstrations and difficulty differ substantially.Table 8 identifies task-dependent training settings as a consequence of these differences.
C.1 BASIC PICK-AND-PLACE MANIPULATION
The real-world task suite covers manipulation from basic object placement to precision, bimanual, dynamic, and long-horizon operations, each adding distinct control or planning demands.
- Basic pick-and-place manipulation: Basic pick-and-place tasks involve grasping and placing household objects with varied physical properties.Examples include moving pens, chip bags, and bottles into designated containers.
- Fine-grained precision manipulation: Fine-grained manipulation requires delicate gripping and accurate placement, such as inserting a flower through a vase’s narrow opening.The thin stem and constrained opening create simultaneous damage-avoidance and positioning requirements.
- Dual-arm coordination and handover: Bimanual tasks evaluate spatial-temporal synchronization and inter-arm transfer during handover sequences.Representative tasks require one arm to pick an object, transfer it, and complete placement or hand it to a person.
- Dynamic environment adaptation: Dynamic-environment tasks test real-time tracking, trajectory prediction, and adaptive control when targets or events change during execution.One example requires intercepting lettuce and bread with different hands in a changing environment.
- Long-horizon sequential manipulation: Long-horizon tasks require planning over 10-step sequences, coordinated tool use, memory management, and replanning around unexpected events.They also involve objects ranging from rigid and deformable items to liquids.
D DEPLOY PLATFORM AND LATENCY ANALYSIS
Deployment measurements use a wired workstation setup designed to isolate model computation, while foresight-image quality is evaluated across scene consistency, object consistency, and task progress.
- Deployment platform: Experiments run on an Intel i9 CPU and NVIDIA RTX 4090 GPU, with wired Ethernet excluding wireless transmission delays from reported latency.The setup makes measured latency primarily reflect computational overhead.
- Foresight quality evaluation: The foresight evaluator receives a task instruction, four historical frames, a predicted next-step frame, and a ground-truth frame.These inputs are specified in the evaluation prompt template.
- Foresight quality evaluation: Foresight quality is scored using binary judgments of scene consistency, object consistency, and task progress following.The evaluator outputs one binary score for each of the three aspects.