HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents

• The report identifies fine-grained visual perception and embodied prediction, interaction, and planning as the two capabilities current VLMs need to improve for physical environments.It attributes the gaps to weak capture of granular details for physical grounding and to mainstream VLMs’ training on static web-scale datasets rather than action-oriented embodied tasks.Source: §1 · p. 3
• The family comprises a 2B-activated, 4B-total MoT model for real-time edge deployment and a 32B-activated, 407B-total MoE model for complex perception and embodied reasoning.The models are built on the VLM paradigm to bridge LLM agents and physical agents.Source: §1 · p. 3
• The model combines native-resolution visual encoding, modality-adaptive MoT computation, visual latent tokens, and over 100M pre-training samples spanning perception, reasoning, and planning.The report also describes real-robot data and high-quality reasoning data to improve performance on real-world problems and complex tasks.Source: §1 · p. 3
• An iterative self-evolving post-training process combines cold-start data, reinforcement learning, rejection-sampling SFT, and large-to-small on-policy distillation.Distillation transfers knowledge from the large model to improve the edge variant.Source: §1 · p. 3

2.1 HY-ViT 2.0: Efficient Native-Resolution Visual Encoder

• The 400M-parameter HY-ViT 2.0 supports arbitrary-resolution inputs and is distilled from a larger internal ViT for efficient, accurate edge-device representations.It adds larger-scale pretraining, tiny-LLM language supervision, and visual reconstruction supervision; a larger ViT’s 2k-code codebook compresses each 8×8 image patch into one discrete code to supervise visual-token outputs.Source: §2.1 · p. 5

2.2 Modality-Adaptive Computing with Mixture-of-Transformers

• MoT assigns duplicated FFN and QKV parameters to visual tokens, improving visual modeling while mitigating language-capability degradation from heavy visual training.The duplicated parameters double the model’s inherently limited parameter count with negligible training and inference overhead; bidirectional visual attention and next-code prediction provide additional visual supervision.Source: §2.2 · pp. 5–6

2.3 Visual Latent Tokens Connecting Vision and Language

• Learnable visual latent tokens appended to each image or video frame connect visual and textual content, with large-ViT global features supervising their outputs during pretraining.The paper reports that these tokens benefit small VLMs and further improve model performance.Source: §2.3 · p. 6

3 Pre-training

• Physical-world visual perception is introduced during large-scale pre-training, followed by mid-training that combines embodied, spatial, and general-domain data for agentic reasoning.The initial corpus covers 2D and 3D grounding, depth estimation, and segmentation; the later stage aligns capabilities with downstream embodied requirements.Source: §3 · p. 6

3.1 Pre-training Data

• The data mixture combines low-level visual perception and specialized embodied and spatial datasets with large-scale general-understanding data.The training pipeline draws on diverse, high-quality vision-language data for both pre-training and mid-training.Source: §3.1 · p. 6

3.1.1 Visual Perception Data

• Perception training covers 2D/3D detection, absolute and relative depth, semantic segmentation, and object pointing and counting.The datasets contain 62M detection, approximately 36M depth, approximately 5M segmentation, and approximately 11M pointing and counting samples. Detection annotations are unified and teacher-verified when generated; depth data uses filtered point samples and normalized focal lengths; segmentation masks are filtered and tokenized; and pointing/counting data emphasizes object-dense scenes.Source: §3.1.1 · p. 7

3.1.2 Embodied-Centric Data

• Embodied-centric data combines annotated sources and real-world egocentric robotics sequences across perception, semantic understanding, and planning and manipulation.These tiers respectively provide spatial and physical awareness, connect visual input with contextual reasoning, and supervise sequential decision-making and action execution.Source: §3.1.2 · p. 7

3.1.3 Spatial-Centric Data

• Spatial-centric data targets geometric structure, visual correspondence, and spatial relationships, organized into correspondence, geometry, configuration, measurement, and dynamics.It complements interaction-focused embodied data with foundational reasoning about 3D environments, using ScanNet, ScanNet++, ARKitScenes, and self-collected data.Source: §3.1.3 · p. 9

3.1.4 General Understanding Data

• General-understanding data spans semantics, STEM, fine-grained visual parsing, complex problem-solving, and agentic operations, supporting both training stages.Datasets are divided between pre-training and mid-training according to scale and annotation quality, and trained jointly with specialized embodied corpora.Source: §3.1.4 · p. 10

3.2 Training Recipe

• Training has two sequential stages: pre-training optimizes a multimodal corpus of more than 600B tokens for foundational visual-linguistic alignment, followed by mid-training on approximately 25M curated samples integrating embodied, spatial, and general-understanding datasets.Source: §3.2 · p. 10

3.3 Training Strategy

• Pre-training jointly optimizes language, vision, and global latent-token losses; mid-training and fine-tuning retain only the autoregressive language loss.The vision loss supervises visual tokens and the global loss aligns latent-token representations with the teacher ViT’s image-level feature.Source: §3.3 · pp. 11–12

4.1.1 Data Construction

• Approximately 100k cold-start CoT instances train the MoT-2B and MoE-A32B variants.The instances come from high-complexity, multi-step problems and additional in-house reasoning data.Source: §4.1.1 · p. 12
• A human-model collaborative pipeline constructs CoT trajectories for sampled high-complexity, multi-step problems.The problems are drawn from spatial, embodied, and general data sources, alongside in-house reasoning data.Source: §4.1.1 · p. 12
• An LLM evaluates generated CoTs for reasoning quality, logical correctness, and sequence repetition.The pipeline also verifies exact-match accuracy of final deduced answers.Source: §4.1.1 · p. 12

4.1.2 Training Recipe

• Cold-start SFT uses the standard cross-entropy loss while explicitly disabling sequence packing.Each training sample is processed individually to isolate and emphasize its independent reasoning chain.Source: §4.1.2 · p. 12
• The base learning rate remains 5e-5 throughout cold-start SFT.Unlike pre-training and mid-training, samples are not packed during this stage.Source: §4.1.2 · p. 12

4.2.1 Data Construction

• The curriculum retains partially successful examples near the model’s capability frontier and balances them across perception, prediction, interaction, and planning.Each RL stage uses a newly constructed set of 50K samples, refreshing the training distribution as the model improves.Source: §4.2.1 · p. 12

4.2.2 Reward Designs

• Task-aware rewards match each sample’s output structure, using deterministic scoring when reliable parsing is possible and an LLM judge for open-ended reasoning.Geometric and trajectory tasks receive graded similarity rewards; structured answers use exact-match or partial-credit rewards, while continuous estimates use smoothly decaying relative-error rewards.Source: §4.2.2 · p. 13

4.2.3 Training Recipe

• The training recipe uses GRPO (Shao et al., 2024), normalizing rewards within sampled response groups and applying a clipped policy-ratio update.The method shares each group’s normalized advantage across rollout tokens; it uses G = 16 and an essentially on-policy update scheme, with asymmetric clipping reported as more stable for long-chain multimodal RL.Source: §4.2.3 · p. 14

4.3 Evolving Deep Thinking with Iterative Training

• RL can improve task reward without ensuring high-quality reasoning traces, motivating training that targets reasoning depth and consistency.Correct answers may result from coherent spatial reasoning or unstable shortcuts.Source: §4.3 · p. 14
• RFT retains examples solved in some but not all rollouts, then applies a teacher-quality threshold to focus training on the model’s learnable frontier.This removes examples that are already saturated or remain out of reach.Source: §4.3 · pp. 14–15
• Approximately 1M candidate examples are filtered into around 300K high-quality reasoning traces for subsequent SFT.Source: §4.3 · p. 15
• RL explores through reward-driven search, while RFT makes its high-quality discoveries explicit positive supervision for the model to reproduce.RL supervision is indirect and relative; RFT consolidates newly discovered reasoning patterns into more stable behavior.Source: §4.3 · p. 15
• Alternating RL optimization with RFT supervised refinement turns occasional success into more reliable embodied-model capability.RL improves the model online, while RFT distills high-quality reasoning traces from the resulting behaviors.Source: §4.3 · p. 15

4.4 Large-to-Small On-Policy Distillation

• On-policy distillation aims to preserve the large model’s embodied competence and thinking style within the compact model’s much smaller capacity budget.The goal is not merely model compression.Source: §4.4 · p. 15
• The student rolls out its own response, then minimizes token-level KL divergence from the teacher’s next-token distributions on those same student-generated prefixes.The teacher is applied under teacher forcing on the student’s prefixes, and the objective averages KL divergence over the response tokens.Source: §4.4 · p. 15
• Training on states induced by the student’s own policy addresses its actual errors, reduces training–inference mismatch, and transfers more than final-answer imitation.This matters because the large model’s RL- and RFT-acquired capabilities are distributed across its reasoning process.Source: §4.4 · p. 15
• In the post-training pipeline, RL expands the large model’s capability frontier, RFT consolidates high-quality reasoning traces, and OPD transfers these refined behaviors to the compact model.OPD is the final bridge from capability discovery in the large model to deployment in the small model.Source: §4.4 · p. 15

5.1 Results of HY-Embodied-0.5 MoT-2B

• MoT-2B achieves the best results on 16 of 22 benchmarks and ranks second on 4, with particularly strong spatial understanding and competitive performance despite its compact size.The evaluation covers visual perception, embodied understanding, and spatial understanding; the paper attributes the model’s performance to embodied-centric design as well as scale.Source: §5.1 · pp. 16–17
• HY-Embodied-0.5 MoT-2B performs comparably to size-matched general VLMs on general visual tasks while retaining strong embodied and spatial understanding.The comparison includes Qwen3-VL-2B-Thinking and InternVL 3.5-2B across general visual knowledge, perception and reasoning, document parsing, and text-centric visual question answering.Source: §5.1 · pp. 17–18, Figure 7 · p. 18

5.3 Analysis

• Qualitative examples show MoT-2B producing depth estimates closer to ground truth than Qwen3 VL, Seed2.0 VL, and RoboBrain-2.5, alongside precise visual grounding and object localization.The examples also show point-level localization, region-level captioning, and coordinate-based visual Chain-of-Thought for complex counting.Source: §5.3 · pp. 18–19, Figure 8 · p. 20

5.2 Results of HY-Embodied-0.5 MoE-A32B

• 67.0 overall score puts A32B ahead of Gemini 3.0 Pro (63.6), Seed 2.0 (66.2), Qwen 3.5 A17B (66.1), and Kimi K2.5 (61.1) across 22 benchmarks.A32B placed first in 7 tasks (32%) and second in 6 (27%); Gemini 3.0 Pro and Seed 2.0 were assessed via their official APIs in thinking mode.Source: §5.2 · p. 18, Table 2 · p. 19

5.3 Analysis

• Across grounding, scene understanding, and planning examples, the model localizes objects in clutter, interprets 3D spatial relations, and infers logical next actions in multi-step tasks.The examples include identifying objects by relative position and determining subsequent manipulation steps from a high-level objective and completed-step history.Source: §5.3 · pp. 19–20, Figure 9 · p. 20
• Both model variants use step-by-step Chain-of-Thought to analyze spatial relationships and affordances, correct their reasoning, and address complex visual, embodied, and spatial problems.Examples include evaluating robot trajectories, selecting interaction points, perspective-taking from multi-view images, navigation planning from video, and 3D geometric deduction.Source: §5.3 · p. 21, Figure 10 · p. 21
• With identical data, initialization, and hyperparameters, MoT converges faster with lower final training loss than Dense-2B while adding negligible inference-time overhead.Inference comparisons fix image input at 576 tokens and generated output at 100 tokens; decoding dominates total inference time.Source: §5.3 · p. 22, Figure 11 · p. 22
• Visual latent tokens link visual full attention with language causal attention, localizing salient objects and spatial regions while language attention focuses on corresponding entities, relations, states, and action instructions.The paper presents this alignment as evidence that latent tokens bridge the modality gap and support grounding visual observations and embodied affordances in language.Source: §5.3 · pp. 22–23, Figure 12 · p. 22

6 Robot Control Results

• The VLA builds on the HY-Embodied-0.5-MoT-2B base and extends its Action Expert following the structural design of π0/π0.5.Source: §6 · p. 23
• Training first used 5K hours of UMI data without exposure to a specific robot embodiment, then real-robot SFT used 300–700 demonstrations per task and matched π0 and π0.5 training conditions.The UMI stage used 32 GPUs, batch size 32 per GPU, and 200K iterations.Source: §6 · p. 23
• 85% success on Precision Plug-in Packing matched π0.5 and exceeded π0 (80%); Tableware Stacking reached 80%, above π0 (60%) and near π0.5 (85%).Source: §6 · pp. 23–24
• 75% success on Mug Hanging exceeded π0 (45%) and π0.5 (50%), the largest improvement among the three evaluated tasks.Source: §6 · p. 24

• State-of-the-art performance across 22 benchmarks and robust execution in real-world robotic manipulation tasks demonstrate HY-Embodied-0.5’s physical-world competence.Source: §7 · p. 24
• A modality-adaptive Mixture-of-Transformers architecture with visual latent tokens supports fine-grained spatial and visual perception for physical grounding.Source: §7 · p. 24
• The embodied post-training pipeline compresses deep reasoning capabilities into an efficient 2B-parameter variant designed for edge deployment.Source: §7 · p. 24
• The authors aim to further bridge language and action models toward training a real-world brain for complex applications.Source: §7 · p. 24

Trajectory Prediction under End-Effector Control; Trajectory Prediction under Joint Control

• The model predicts end-effector and joint-control trajectories, moving a towel toward a table corner and a spatula toward the table’s front edge.The examples give normalized image-coordinate waypoints for each task.Source: B Visualizations · pp. 31–32

2D Bounding Box Grounding; Point-Based Localization

• Grounding examples identify a helmet and microwave with bounding boxes, while point-based localization selects unoccupied regions within a marked area and beside a picture frame.The examples return normalized box coordinates and point coordinates.Source: B Visualizations · pp. 33–34

Metric Reasoning: Distance Estimation; Metric Reasoning: Area Estimation

• Metric-reasoning examples estimate the TV-to-chair distance at about 1.0 meters and the room area at about 9 square meters.The distance estimate uses objects across video frames; the area estimate reasons from the kitchen space and its dimensions.Source: B Visualizations · pp. 35–36

Multi-View Spatial Reasoning; Orientation-Aware Spatial Reasoning

• Multi-view examples infer that a gray-green tufted backrest is left of a bottle and a TV is right of a sneaker from the specified viewpoints.The reasoning relates backgrounds across four views to the requested directions.Source: B Visualizations · pp. 37–38

Embodied Perception; Affordance-Aware Localization; Navigation-Oriented Localization

• Embodied examples identify the farthest object as a cabinet, place a dresser below a mirror, select the spoon as the object to grasp, and choose the point right of a speaker for navigation.These examples combine relative-position judgments with affordance-aware and navigation-oriented localization.Source: B Visualizations · pp. 39–41

Embodied Question Answering

• Embodied question-answering examples choose a green collision-free gripper path and identify the purple arrow as pointing to a motorcycle.Source: B Visualizations · p. 43

Sorting-Based Reasoning Task; Counting-Based Reasoning Task

• Reasoning-task examples order flower-basket preparation as images 4, 2, 3, 1 and count two trash or recycling bins in a room.Source: B Visualizations · pp. 44–45

B Visualizations

• The wet-floor sign example identifies slipperiness as the reason for the warning, rather than decoration, blocking entry, or buying cleaning tools.The reasoning uses the “CAUTION WET FLOOR” text and falling-person symbol.Source: B Visualizations · p. 46
• The suitcase example infers that the person is preparing to leave home for a period of time from placing clothes in an open suitcase.It distinguishes packing for travel from laundry, storage, or choosing an outfit.Source: B Visualizations · p. 46
• Figure B16 presents examples of general visual question answering tasks.Source: Figure B16 · p. 46