Being-M0.7: A Latent World-Action Model for Humanoid Robots

• Scarce robot demonstrations and human datasets that often provide only video or motion complicate transfer to executable humanoid actions.Robot demonstrations require synchronized egocentric video, proprioception, and whole-body commands; human motion alone does not specify executable robot actions.Source: §1 · p. 2
• Joint prediction of future visual latents and motion encourages visual representations to encode future kinematics from more than 10,000 hours of mixed-modality human data.The corpus combines video-only, motion-only, and paired video–motion streams.Source: §1 · p. 2
• Robot mid-training adapts the visual–motion prior to robot viewpoints and dynamics, then gated action post-training grounds its frozen predictive context in current observations and executable whole-body commands.The action expert combines prior visual representations with current images and proprioception.Source: §1 · p. 2
• Being-M0.7 achieves the highest aggregate simulation success rate among compared baselines on SIMPLE and matches the strongest baseline on real-world Unitree G1 tasks.Source: §1 · pp. 2–3

• Reference-based methods guide whole-body coordination, while reference-free approaches learn from objectives and perception; both motivate retaining movement structure while adapting it to the object and task [22] [23] [24].Source: §2 · p. 3
• Egocentric motion work links first-person observations with body motion and future-state prediction, motivating visual–motion dynamics as temporal context for action [33].The paper focuses on transferring predictive knowledge to humanoid execution through joint visual-latent and motion dynamics, a shared human–humanoid representation, and an action expert.Source: §2 · p. 3
• Compared with humanoid foundation and world-action models [34] [4] [35], Being-M0.7 emphasizes paired and unpaired visual–motion learning, robot mid-training, and gated post-training.The adapted prior is frozen during action learning; robot supervision trains how its visual context informs executable commands.Source: §2 · p. 3

3 Method

• The framework combines mixed-modality pre-training, robot mid-training, and gated action post-training to transfer a human visual–motion prior into robot control.The prior is adapted to robot trajectories, then its frozen visual context is combined with current robot observations by an action expert.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Problem Formulation

• The prior predicts future visual latents and motion from observed history and task instruction, while deployment uses predicted visual futures as conditioning for fine-grained action chunks.Motion prediction is disabled during post-training and deployment; future visual representations retain learned information about kinematics.Source: §3.1 · p. 4

3.2 Data Curation

• More than 10,000 raw-data hours are organized into paired video–motion, video-only, and motion-only streams, with unpaired samples supervising their modality through shared parameters.Paired samples supervise joint prediction, extending training beyond the paired corpus while retaining cross-modal interaction where both modalities are observed.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 Model Architecture

• The separate flow-based action expert combines noisy action tokens and current image–proprioception observations with frozen-prior hidden states injected through gated cross-attention.The gate varies predictive-context contribution across tokens, while local self-attention keeps action tokens connected to current observations.Source: §3.3 · pp. 5–6

3.4 Training and Inference Recipes

• Pre-training uses flow matching for visual and motion predictions, with modality losses applied only when that modality is observed and auxiliary geometry losses when motion is available.Source: §3.4 · p. 6
• Robot mid-training adapts the prior on robot video, motion, and task text; post-training freezes it and trains the expert with prior-generated visual futures and action supervision.Training on the prior’s own predictions reduces conditioning mismatch with inference, and the frozen prior receives no action-loss gradients.Source: §3.4 · p. 6
• At inference, the prior periodically generates and caches visual-latent plans, while the expert uses fresh images and proprioception to denoise action chunks and refreshes context before it is fully consumed.Source: §3.4 · p. 6

4.1.1 Hardware Setup

• The Unitree G1 platform uses dual 7-DoF arms and Linker Hand O6 hands, with a head-mounted D435i camera and external RTX 4090 inference.Source: §4.1.1 · pp. 6–7

4.1.2 VR-Based Whole-Body Teleoperation Pipeline

• VR trackers feed SMPL pose estimation and SONIC, which sends 29-DoF commands at 50 Hz while recording synchronized RGB, proprioception, and motion signals.The operator uses a PICO headset, ankle trackers, and controllers for full-body input.Source: §4.1.2 · p. 7, Figure 4 · p. 7

4.1.3 Baselines and Multi-Task Training

• Comparisons use GR00T-N1.6, Ψ0, and π0.5; each method is jointly trained as one task-conditioned model across six simulation or three real-world tasks.Source: §4.1.3 · p. 7

4.2 Simulation Experiments

• 128/180 (71.1%) gives Being-M0.7 the highest SIMPLE aggregate success rate, ahead of Ψ0 (116/180), π0.5 (95/180), and GR00T-N1.6 (61/180).The benchmark covers six tasks at three domain-randomization levels, totaling 180 trials.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Ablation Study

• Removing pre-training lowers success to 88/180 (48.9%) and removing mid-training to 107/180 (59.4%); language-only context yields 105/180 (58.3%).These are reductions of 22.2, 11.7, and 12.8 percentage points, respectively, from the full model.Source: §4.3 · p. 8

4.4 Real-World Experiments

• Being-M0.7 achieves 13/15 real-world successes (86.7%), matching GR00T-N1.6 and exceeding π0.5 (7/15) and Ψ0 (3/15).It ties GR00T-N1.6 at 3/5 on Walk to Fish and 5/5 on Walk to Mirror and Walk to Doll, exceeding both other baselines on each task.Source: §4.4 · p. 9, Table 3 · p. 9

5 Conclusion

• Ablations support the contributions of pre-training, robot mid-training, and predictive context to downstream control on SIMPLE and real-world Unitree G1 tasks.The conclusion describes these experiments as demonstrating the framework’s effectiveness.Source: §5 · p. 9
• Figure 5 shows three real-world sequences: mirror-based grasping of a hidden doll, net scooping of a toy fish, and grasping a visible doll.The rows correspond, respectively, to Walk to Mirror, Walk to Fish, and Walk to Doll.Source: Figure 5 · p. 10
• Longer-horizon performance remains unestablished, and limited SONIC end-effector tracking accuracy makes robot demonstration collection challenging.Future work will extend evaluation to longer-horizon tasks and refine SONIC tracking precision.Source: §5 · p. 10

A.1 Data Sources and Motion Processing

• The shared 22D head-root representation stores root translation and rotation, initial head height, and local positions of both hands and feet.The 6D rotation encodes two axes of head orientation; positions and heights are measured in meters.Source: §A.1 · p. 14
• Human hand and foot positions are transformed into a floor-aligned head-root frame whose origin lies below the initial head position.Robot motion uses the same component order and coordinate convention without requiring identical human and robot joint-angle parameterizations.Source: §A.1 · p. 14
• With use_root_pos=true, frame-to-frame head displacements are accumulated before subsampling to 5 Hz, yielding relative head positions.The initial displacement is reset to zero when constructing a robot window.Source: §A.1 · p. 14
• Human motion is standardized per channel when stored statistics are available, whereas robot mid-training uses unstandardized 22D motion vectors.Human standard deviations below 10^-5 are replaced by one; absent statistics leave human values unstandardized.Source: §A.1 · p. 14

Input Embeddings

• Visual and motion inputs use separate learned projections into a common hidden space, with temporal and modality embeddings; instructions are encoded separately.Video-only and motion-only samples receive supervision only for the available modality.Source: §A.2 · p. 15

Missing Modalities

• Missing-modality placeholders cannot exchange information with valid tokens, while paired samples retain cross-modal attention.Presence flags and temporal-validity masks constrain attention; invalid queries are excluded from supervision and expert context.Source: §A.2 · p. 15
• With use_robot_motion=false, motion states receive no flow updates and only visual future states are integrated and cached for the expert.The shared forward pass still evaluates masked motion slots, but discards their predictions.Source: §A.2 · p. 15

MoT Blocks

• The Mixture-of-Transformers uses modality-specific projections and feed-forward networks with shared multimodal attention, allowing valid visual and motion tokens to exchange information.Both streams also cross-attend to instruction features, and flow timestep embeddings modulate each modality’s computation.Source: §A.2 · p. 15
• Future video-motion tokens jointly attend to history and one another, enabling the model to denoise the future chunk in parallel.Observed context tokens cannot access the future; modality-specific outputs predict velocity fields in visual latent and continuous motion spaces.Source: §A.2 · p. 15

Layer-Wise Action Interface

• During post-training and inference, only visual hidden states from the prior are supplied to the action expert alongside current robot observations.The expert uses generated future visual tokens as keys and values; only noisy action tokens query them, and the gated residual is applied to action tokens.Source: §A.2 · p. 15, Figure 6 · p. 16

B.1 Architecture Hyperparameters

• Table 4 reports architecture hyperparameters for the video-motion prior and action expert.The supplied table caption identifies the table’s scope but does not give its values.Source: Table 4 · p. 16, §B.1 · p. 16

B.2 Hyperparameters

• Pre-training mixes paired, video-only, and motion-only data without explicit source reweighting, so stream proportions depend on valid examples after filtering.Mid-training reads video, text, and 22D motion; post-training reads video, text, current proprioception, and demonstrated action chunks.Source: §B.2 · pp. 16–17
• The DINO and DistilBERT encoders remain frozen; post-training freezes the adapted prior and trains the action expert and its observation/context projections.The post-training prior starts from a mid-training checkpoint at 50,000 optimizer updates.Source: §B.2 · p. 17
• The action expert’s current image uses final-layer patch tokens, and language features come from frozen DistilBERT with instructions limited to 64 tokens.Images are resized to 224 × 224; padding is masked in language cross-attention.Source: §B.2 · p. 17

B.3 Geometry-aware Auxiliary Losses

• During pre-training, geometry-aware auxiliary losses supplement flow matching with global trajectory and local end-effector velocity regularization when motion is available.Their weights are 0.1 for trajectory and 1.0 for local velocity.Source: §B.3 · pp. 17–18

B.4 Post-training Details

• During post-training, the prior generates visual futures in 3 sampling steps, whereas inference uses a 10-step schedule; action flow matching uses 10 denoising steps.Post-training trains the expert on the prior model’s own predictions rather than corrupted ground-truth future latents.Source: §B.4 · p. 18, §B.5 · p. 19
• The expert’s context contains hidden states from the 24 future frames, excluding the observed input frame, while its current observation is supplied separately.The same future-token selection is used in post-training and inference.Source: §B.4 · pp. 18–19
• Real-world post-training applies world-context dropout with probability 0.1 to encourage use of current observations; simulation post-training does not use it.The dropout decision is shared across expert layers.Source: §B.4 · p. 18
• The real-world action is a 66D vector, and the current robot observation combines an egocentric image with 63D proprioception.World cross-attention is applied at every third expert layer, with corresponding prior and expert layers paired by depth.Source: §B.4 · pp. 18–19

B.5 Inference Details

• The expert initializes an action chunk from Gaussian noise, denoises it, and outputs 30 executable commands for the low-level controller.Each 30-command chunk runs at 50 Hz for 0.6 seconds before the next query.Source: §B.5 · pp. 19–20
• Inference caches hidden states for all 24 generated future positions and reuses them across action queries until refreshing the prior before its prediction window is exhausted.Each query still uses the latest robot observation; the future context is refreshed before the window ends.Source: §B.4 · p. 18, §B.5 · pp. 19–20

C.1 Benchmark and Evaluation Protocol

• Six Unitree G1 tasks span tabletop grasping, bimanual transfer, and mobile manipulation in the SIMPLE simulation framework.SIMPLE combines MuJoCo physics with Isaac Sim rendering and supports demonstrations from motion planning and VR teleoperation.Source: §C.1 · p. 20
• Each method is evaluated on 180 trials: ten trials for each of six tasks at each of three randomization levels.The easy, medium, and hard settings correspond to Levels 0, 1, and 2; higher levels cumulatively add variation.Source: §C.1 · p. 20, Table 6 · p. 20
• Baseline results come from the authors’ own runs using official implementations, and aggregate success rate is computed across all 180 trials.Ablations use the same task configurations, default seeds, success criteria, and trial counts as the main comparison.Source: §C.1 · p. 20

C.2 The Six Evaluated Tasks

• The tasks vary the required reaching height.Source: §C.2 · p. 20
• The suite includes variation in whether tasks require use of two hands.Source: §C.2 · p. 20
• Tasks also differ in how they integrate locomotion with object interaction, and Table 7 maps task names to simulation environments and objectives.The table caption specifies that environment identifiers use the G1Wholebody suffix with the simple/ prefix.Source: §C.2 · p. 20, Table 7 · p. 21

C.3 Task Demonstration Snapshots

• Figure 7 shows one SIMPLE demonstration trajectory per task, with four selected egocentric observations per row progressing through reaching and interaction.Some sequences also show transport or handover; frame indices are the originals from each sequence.Source: §C.3 · p. 22, Figure 7 · p. 23
• The full model underperforms on XMoveBendPick relative to language-only context, despite substantial improvements on XMovePick and BendPick.The authors suggest that similar visual observations may introduce task interference under joint multi-task training, and that greater model capacity may help.Source: §C.3 · p. 22

C.4 Visual Variation Across Evaluation Levels

• Figure 8 shows first-frame examples from six tasks across three cumulative randomization levels, viewed through the left head camera.The scenes are separate evaluations, and a shared episode index does not imply matched random seeds.Source: Figure 8 · p. 25, §C.4 · p. 25

Evaluation setup

• Each method was evaluated over five trials per task, with 30 seconds per trial and human-judged task-specific success criteria.The three tasks together yielded fifteen trials per method; targets varied across left, center, and right workspace regions.Source: §D.1 · p. 26

Walk to Fish (W2F)

• Walk to Fish requires approaching a water tank, positioning the body, and using a handheld net to scoop a toy fish.The quantitative evaluation covers the complete approach-and-scooping sequence, guided by visual feedback.Source: §D.1 · p. 26, Figure 9 · p. 26

Walk to Mirror (W2M)

• Walk to Mirror requires locating a hidden toy from its mirror reflection before reaching into a partially enclosed box to grasp it.The task combines indirect visual observations with whole-body positioning and manipulation.Source: §D.1 · p. 27, Figure 10 · p. 27

Walk to Doll (W2D)

• Walk to Doll evaluates locomotion transitioning to a grasp of a directly visible doll through coordinated approach, body positioning, and arm motion.Together with Walk to Mirror, it assesses approach-and-grasp sequences with direct rather than mirror-based target observations.Source: §D.1 · p. 27, Figure 11 · p. 27