Source-linked AI summary
Being-H0.7: A Latent World-Action Model from Egocentric Videos
Hao Luo, Wanpeng Zhang, Yicheng Feng, Sipeng Zheng, Haiweng Xu, Chaoyi Xu, Ziheng Xi, Yuhui Fu, Zongqing Lu
TL;DR
VLAs can rely on shortcut mappings because action supervision is sparse, while pixel-space future prediction is costly and may model details irrelevant to control. Being-H0.7 addresses this with latent queries and future-informed dual-branch alignment, achieving strong performance across six simulation benchmarks and diverse real-world tasks without inference-time visual rollout.
Problem
Sparse action supervision can lead VLAs to learn shortcut mappings instead of representations of dynamics, contact, and task progress, while pixel-space future prediction is costly and indirect for control.
Method
Being-H0.7 inserts learnable latent queries between perception and action, aligning a current-context prior with a future-observation posterior during training while discarding the posterior at inference.
Results
Being-H0.7 achieves state-of-the-art or comparable performance across six simulation benchmarks and leads all five real-robot ability-oriented suites.
Takeaways & Limitations
The model combines future-aware latent reasoning with direct-action-policy efficiency by avoiding future-frame generation and pixel-space rollout at inference.
Takeaways & Limitations
Latent queries do not by themselves guarantee meaningful future-relevant structure, so future-informed alignment is required to shape the latent reasoning space.
Abstract
from arXiv · showhide
Visual-Language-Action models (VLAs) have advanced generalist robot control by mapping multimodal observations and language instructions directly to actions, but sparse action supervision often encourages shortcut mappings rather than representations of dynamics, contact, and task progress. Recent world-action models introduce future prediction through video rollouts, yet pixel-space prediction is a costly and indirect substrate for control, as it may model visual details irrelevant to action generation and introduces substantial training or inference overhead. We present Being-H0.7, a latent world-action model that brings future-aware reasoning into VLA-style policies without generating future frames. Being-H0.7 inserts learnable latent queries between perception and action as a compact reasoning interface, and trains them with a future-informed dual-branch design: a deployable prior branch infers latent states from the current context, while a training-only posterior branch replaces the queries with embeddings from future observations. Jointly aligning the two branches at the latent reasoning space leads the prior branch to reason future-aware, action-useful structure from current observations alone. At inference, Being-H0.7 discards the posterior branch and performs no visual rollout. Experiments across six simulation benchmarks and diverse real-world tasks show that Being-H0.7 achieves state-of-the-art or comparable performance, combining the predictive benefits of world models with the efficiency and deployability of direct VLA policies.
1 Introduction
Being-H0.7 addresses shortcut-prone VLA control by organizing future-relevant, action-useful information in a compact latent reasoning space rather than predicting future pixels. Its dual-branch training aligns a deployable current-context prior with a future-informed posterior, while experiments report strong simulation and real-world performance.
- Motivation: Sparse action supervision can encourage VLAs to map visual cues directly to actions instead of representing dynamics, contact, and task progress.This can produce strong in-distribution behavior but weaker robustness when control requires anticipating interaction-driven changes.
- Motivation: Pixel-space future prediction may model visually irrelevant details, despite manipulation primarily requiring compact cues such as contact, object motion, affordances, and task progress.The paper therefore frames future modeling around action-relevant latent structure rather than high-fidelity video reconstruction.
- Method: Being-H0.7 inserts learnable latent queries between multimodal context and actions to form a compact intermediate state that guides action generation.The queries attend to instruction, observation history, and robot state, progressively organizing interaction-relevant information before actions are produced.
- Method: The deployable prior infers predictive factors from current context, while a training-only posterior uses subsequent observations to provide privileged future-informed supervision.At inference, the posterior is removed and the model performs no future-frame generation or pixel-space rollout.
- Results: The model combines predictive world-model benefits with the efficiency and deployability of direct VLA-style policies across simulation benchmarks and real-world tasks.The contribution passage describes strong performance across diverse settings without requiring costly inference-time pixel rollouts.
2 Related Work
Related work spans direct VLA policies, explicit video-based world-action models, latent future-aware action modeling, and human-centric learning. Being-H0.7 differs by using future information to shape a latent reasoning space while avoiding explicit future-video rollout at inference.
- Vision-Language-Action Models: VLAs adapt pretrained vision-language representations to map observations and instructions directly to actions, while structured intermediate representations still primarily use current observations.Examples include textual planning, bounding boxes, dense correspondence fields, 3D points, and trajectory traces.
- World-Action Models: World-action models use video generation or world modeling to capture temporal dynamics and plausible future evolution for robot control.Some methods use video models as predictive representation learners, while others jointly model future video and action.
- World-Action Models: Being-H0.7 differs from explicit rollout methods by jointly shaping a latent reasoning space that directly participates in action generation.Its future signal is used to organize latent structure rather than reconstruct future video frames.
- Latent Future-Aware Action Modeling: Latent future-aware approaches include future-representation supervision, action-side alignment with future-queried states, and compression of future observations into action-useful conditions.These methods form a related but distinct trend within world-action modeling.
- Human-Centric Learning: Human-centric learning transfers interaction priors from human behavior through portable physical interfaces or abundant human video corpora.The paper positions large-scale egocentric-video pretraining within this broader effort to reduce dependence on costly robot demonstrations.
3 Method
Being-H0.7 connects direct action prediction and future-aware world modeling through a compact latent reasoning space. A dual-branch design uses future observations during training to shape deployable latent states that guide action generation without requiring future-frame prediction at inference.
- 3.1 Latent Reasoning: Latent queries are inserted between multimodal context and action tokens to organize task-relevant information into an action-oriented intermediate state.They interact across Transformer layers with instructions, observations, state, and actions before shaping downstream action generation.
- 3.1 Latent Reasoning: Action supervision alone does not guarantee that latent queries will learn future-relevant structure rather than weak intermediate representations or shallow local decoding cues.The method therefore adds explicit future-informed alignment to shape latent reasoning as world modeling.
- 3.2 Joint Alignment with Future Information: The dual-branch design pairs a deployable prior branch using current context and learnable queries with a training-only posterior branch using future embeddings.Future observations are encoded by a frozen ViT and aggregated by a Perceiver resampler into embeddings matching the prior query shape.
- 3.2 Joint Alignment with Future Information: Joint alignment matches hidden states at latent reasoning positions, explicitly shaping the shared space to encode future-relevant, action-oriented structure.The alignment is applied across matched layers, while the posterior branch supplies a future-informed view of information useful for downstream action decisions.
- 3.3 Efficient Dual-Branch Implementation: A packed Mixture-of-Transformers sequence and dual-branch attention mask implement both branches efficiently while isolating branch-specific tokens and sharing current context.The branches are coupled through alignment rather than direct cross-branch attention, avoiding two fully separate backbone passes.
- 3.3 Efficient Dual-Branch Implementation: Flow matching trains both branches to generate actions, while norm preservation and spectral-diversity regularization discourage collapse of aligned latent states.The regularizers prevent vanishing magnitude and directional collapse in the latent reasoning space.
4.1 Training details
Training uses RGB-only visual observations and a future-informed latent alignment setup, followed by task-specific optimization of action generation and alignment.
- 4.1 Training details: Being-H0.7 uses RGB-only visual observations, resizing context images to 224 × 224 and future posterior frames to 256 × 256.
- 4.1 Training details: Pretraining uses H = 4 observations, T = 20 action chunks, K = 16 latent queries, and alignment across the last L = 9 Transformer layers.
- 4.1 Training details: Table 1 compares Being-H0.7 with other methods across multiple embodied manipulation benchmarks, including CALVIN and LIBERO-plus variants.
- 4.1 Training details: Downstream post-training optimizes action generation and latent alignment on task-specific demonstrations, omitting anti-collapse regularizers.
4.2 Simulation
Being-H0.7 is evaluated across six simulation benchmarks covering transfer, household manipulation, dexterity, robustness, bimanual control, and long-horizon generalization. It achieves state-of-the-art overall performance and strong benchmark-specific success rates.
- 4.2 Simulation: Being-H0.7 is evaluated on six simulation benchmarks spanning tabletop transfer, household tasks, dexterous manipulation, robustness, bimanual control, and long-horizon execution.
- 4.2 Simulation: 99.2% average success rate is achieved on LIBERO across its four task suites.Evaluation averages success over 500 trials per suite.
- 4.2 Simulation: 62.1% success rate is achieved on RoboCasa’s everyday household tasks in diverse and unseen kitchen environments.The benchmark uses 24 tasks and a few-shot setting with 50 demonstrations per task.
- 4.2 Simulation: 49.2% average success rate is achieved on GR1’s dexterous humanoid manipulation tasks.GR1 contains 24 complex tabletop tasks requiring fine-grained dexterity and coordination.
- 4.2 Simulation: 82.1% zero-shot success on LIBERO-plus improves to 84.8% after fine-tuning on LIBERO-plus.The benchmark assesses robustness and zero-shot generalization under controlled environmental perturbations.
- 4.2 Simulation: 89.6% success under severe visual domain randomization on RoboTwin 2.0 compares with 90.2% in clean scenes, a 0.6% drop.RoboTwin 2.0 varies clutter, lighting, backgrounds, tabletop height, and language instructions.
4.3 Real-world Experiments
Being-H0.7 is evaluated across three real-robot embodiments and 12 tasks organized into five ability-oriented suites, using a unified inference and evaluation protocol. It leads all five suites while showing particular strength on timing-sensitive, physically constrained, and long-horizon tasks, with an efficient deployment loop.
- Embodiments: Being-H0.7 is evaluated on PND Adam-U, Unitree G1, and Franka FR3, with Linkerbot O6 hands and a shared state/action interface.The platforms differ in body configuration and action dimensions, while the deployment stack remains unified.
- Task Suites: The 12 real-robot tasks are grouped into dynamic scene, physical reasoning, motion reasoning, long-horizon execution, and generalization suites.Tasks may carry overlap tags and combine capabilities such as moving-target response, trajectory reasoning, containment, gravity, and multistage subgoals.
- Evaluation Protocol: Each task uses randomized layouts and rollout order, hidden policy endpoints, fixed binary success criteria, and 20 blind trials per method.All compared policies run through a unified black-box inference server to keep the surrounding execution stack identical.
- Results Overview: Being-H0.7 leads all five ability-oriented suites across the three embodiments, with improvement appearing throughout the benchmark rather than in one isolated task.The suite-level results span reactive, physical, sequential, and generalization-oriented capabilities.
- Results Overview: The strongest qualitative advantages appear on dynamic and motion-centric tasks, while the model also remains ahead on physical reasoning, long-horizon execution, and generalization.The results connect reactive success with runtime responsiveness and future-aware latent state, and connect longer manipulation success with causal consistency across constrained stages.
- Deployment: UAC-enabled Being-H variants reach the 3–4 ms/step regime without changing GPU memory footprint, supporting efficient continuous control across embodiments.The deployment protocol preserves temporal continuity and reduces visible control stutter while accommodating different control frequencies and action dimensions.
5 Conclusion
Being-H0.7 bridges direct action prediction and world modeling through a compact latent reasoning space. Aligning deployable prior and future-aware posterior branches injects future-relevant reasoning without costly inference-time pixel rollouts.
- Being-H0.7 bridges direct action prediction and world modeling through a compact latent reasoning space.
- Aligning a deployable prior branch with a future-aware posterior branch injects future-relevant reasoning into action generation without inference-time pixel-level rollout.
- Large-scale human video pretraining contributes to an effective and scalable framework for embodied models.
Author List
The paper lists its authors and identifies contribution and authorship roles. Four authors share equal contribution, and one author is designated corresponding author.
- The author list names Hao Luo, Wanpeng Zhang, Yicheng Feng, and Sipeng Zheng among the contributors.
- The author list also includes Haiweng Xu, Chaoyi Xu, Ziheng Xi, Yuhui Fu, and Zongqing Lu.
- Hao Luo, Wanpeng Zhang, Yicheng Feng, and Sipeng Zheng are marked as equal contributors, while Zongqing Lu is the corresponding author.