PaperlayerView exact revision on arXiv

Source-linked AI summary

HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents

Tencent Robotics X, HY Vision Team, :, Xumin Yu, Zuyan Liu, Ziyi Wang, He Zhang, Yongming Rao, Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, Yves Liang, Haitao Lin, Minghui Wang, Yubo Dong, Kevin Cheng, Bolin Ni, Rui Huang, Han Hu, Zhengyou Zhang, Linus, Shunyu Yao

arXiv:2604.07430v1cs.CV

TL;DR

HY-Embodied-0.5 addresses the challenge of adapting VLMs for real-world agents that must perceive, reason, and act in physical environments. It combines modality-adaptive architecture, visual latent tokens, embodied data, iterative post-training, and on-policy distillation across compact and large variants. Across 22 benchmarks, the models achieve strong results, including 16 best results for MoT-2B and a 67.0% average for MoE-A32B versus Gemini 3.0 Pro’s 63.6%.

  • Problem

    General VLMs still require stronger fine-grained spatial and temporal perception and embodied reasoning for prediction, interaction, and planning in physical environments.

  • Method

    The suite combines modality-specific MoT computation, visual latent tokens, iterative self-evolving post-training, and large-to-small on-policy distillation in 2B- and 32B-activated variants.

  • Results

    Across 22 benchmarks, MoT-2B achieves the best performance on 16, while MoE-A32B averages 67.0%, exceeding Gemini 3.0 Pro’s 63.6%.

  • Takeaways & Limitations

    The results support HY-Embodied-0.5 as a foundation for both edge-deployed embodied models and complex visual-agent reasoning.

  • Takeaways & Limitations

    The evaluation notes repetitive thinking patterns in some Qwen3.5 small models and therefore compares Qwen3-VL models instead.

Abstract

from arXiv · show

We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents. To bridge the gap between general Vision-Language Models (VLMs) and the demands of embodied agents, our models are developed to enhance the core capabilities required by embodied intelligence: spatial and temporal visual perception, alongside advanced embodied reasoning for prediction, interaction, and planning. The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning. To support the fine-grained visual perception essential for embodied tasks, we adopt a Mixture-of-Transformers (MoT) architecture to enable modality-specific computing. By incorporating latent tokens, this design effectively enhances the perceptual representation of the models. To improve reasoning capabilities, we introduce an iterative, self-evolving post-training paradigm. Furthermore, we employ on-policy distillation to transfer the advanced capabilities of the large model to the smaller variant, thereby maximizing the performance potential of the compact model. Extensive evaluations across 22 benchmarks, spanning visual perception, spatial reasoning, and embodied understanding, demonstrate the effectiveness of our approach. Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro. In downstream robot control experiments, we leverage our robust VLM foundation to train an effective Vision-Language-Action (VLA) model, achieving compelling results in real-world physical evaluations. Code and models are open-sourced at https://github.com/Tencent-Hunyuan/HY-Embodied.

1 Introduction

HY-Embodied-0.5 targets the gap between general VLMs and real-world embodied agents by improving fine-grained perception, embodied reasoning, and practical deployment. It introduces multiple architectural and training innovations and reports broad benchmark gains across visual, spatial, and embodied tasks.

  • Motivation and approach: The models address fine-grained visual perception and embodied reasoning for prediction, interaction, planning, and physical-world understanding.
  • Model family: HY-Embodied-0.5 is a foundation-model family purpose-built for real-world agents, with 2B-activated and 32B-activated variants for edge deployment and complex reasoning.The variants use MoT and MoE designs, respectively.
  • Contributions: The approach combines modality-adaptive architecture, visual latent tokens, large-scale perception and embodied data, iterative self-evolving post-training, and large-to-small distillation.
  • Evaluation: The evaluation suite contains 22 public benchmarks spanning visual perception, spatial reasoning, and embodied understanding.
  • Results: The MoT-2B model achieves the best performance on 16 of 22 benchmarks, while the MoE-A32B model averages 67.0% and exceeds Gemini 3.0 Pro’s 63.6%.

2 Model Architecture

The architecture combines an efficient native-resolution visual encoder with modality-adaptive transformer computation and visual latent tokens. These components specialize visual processing while preserving integration between visual and textual representations.

  • Visual encoder: HY-Embodied-0.5 uses a vision encoder and language model, with a 400M-parameter ViT optimized for efficient edge-device visual representation.
  • Visual encoder: HY-ViT 2.0 supports arbitrary-resolution inputs and uses language supervision, visual reconstruction supervision, and distillation from a stronger internal ViT.
  • Mixture-of-Transformers: MoT assigns non-shared parameters to language and vision tokens, improving visual modeling while mitigating degradation of language capabilities.
  • Mixture-of-Transformers: Distinct attention masks use bidirectional attention for visual tokens, reflecting the non-unidirectional structure of visual data.
  • Visual latent tokens: Learnable visual latent tokens are appended to each visual element and supervised with global ViT features to connect visual and language representations.

3 Pre-training

Pre-training combines general understanding with spatial, visual-perception, and embodied data to establish physical-world competencies. The corpus covers perception, semantic understanding, planning, manipulation, and long-horizon reasoning.

  • Data mixture: The pre-training mixture combines low-level visual perception, spatial cognition, embodied tasks, and large-scale general understanding data.
  • Perception and interaction: The corpus includes detection, depth estimation, segmentation, pointing, counting, grounding, affordance, trajectory, and embodied-understanding data.
  • Embodied-centric data: Embodied data is organized into perception, semantic understanding, and planning tiers covering physical awareness, contextual reasoning, sequential decisions, and action execution.
  • Training data scale: Pre-training uses over 200B tokens, while mid-training uses over 12M high-quality question-answer pairs for complex real-world execution.
  • Reasoning: Long-horizon reasoning data targets action sequencing, multi-image action comprehension, future-state prediction, visual puzzles, and intuitive physics.
  • Spatial-centric data: Spatial-centric data targets geometric structures, visual correspondences, and spatial relationships through correspondence, geometry, configuration, measurement, and dynamics tasks.

4.1 Supervised Fine-tuning

Supervised fine-tuning reinforces long-chain reasoning using difficult multi-step problems and collaboratively generated, evaluated Chain-of-Thought trajectories. Cold-start SFT processes samples independently with standard cross-entropy training.

  • Data selection: SFT samples high-complexity, multi-step problems from spatial, embodied, general, and in-house reasoning data.
  • Reasoning supervision: Human-model collaboration produces Chain-of-Thought trajectories that an LLM evaluates across multiple dimensions.
  • Cold-start SFT: Cold-start SFT uses standard cross-entropy loss while disabling sequence packing so each reasoning chain is processed independently.

4.2 Reinforcement Learning

The reinforcement-learning stage uses capability-adaptive data selection, task-aware rewards, and GRPO-based optimization to handle heterogeneous embodied outputs. Training includes controls for balanced coverage, stable relative learning signals, and long multimodal responses.

  • Each RL round retains samples that are neither consistently solved nor consistently failed, then balances them across perception, prediction, interaction, and planning.Each RL stage uses a newly constructed set of 50K samples, refreshing the effective training distribution as the model improves.
  • Reward Designs: Reward functions match output structure, using grounding, trajectory, exact-match, regression, or LLM-judged rewards for different embodied task types.Deterministic and structure-aware rewards are preferred whenever targets can be reliably parsed; LLM judgment is reserved for open-ended cases.
  • Reward Designs: Geometric and trajectory tasks receive graded similarity rewards, while constrained discrete tasks use exact or partial-credit scoring and continuous estimates use smoothly decaying relative-error rewards.Examples include IoU, Hungarian-matched IoU, normalized point distance, Chamfer distance, DTW-based measures, and normalized longest common subsequence.
  • Training Recipe: GRPO samples G responses, normalizes rewards within each group into shared token-level advantages, and updates the policy with a clipped policy-ratio objective.This relative normalization is suited to heterogeneous tasks whose raw reward scales are not directly comparable.
  • Training Recipe: Training uses G = 16, masks zero-variance groups, filters overlong or repetitive responses, and applies asymmetric importance-ratio clipping over [0.8, 1.35].The RL setup also uses 16,384-token prompt and response limits, batch size 128, learning rate 8 × 10^-7, and 5 epochs per stage.

4.3 Evolving Deep Thinking with Iterative Training

The paper alternates reinforcement learning with rejection-sampling fine-tuning to turn reward-driven discoveries into stable, high-quality reasoning behavior. The iterative procedure filters for partially mastered tasks and teacher-approved reasoning traces before supervised refinement.

  • RFT begins from the latest RL checkpoint, evaluates multiple rollouts on curated data, and retains examples solved in some but not all attempts.This removes saturated examples and examples that remain out of reach.
  • A stronger teacher filters the retained samples by reasoning quality, reducing approximately 1M candidate examples to around 300K high-quality traces for SFT.
  • RL explores through reward-driven search, whereas RFT converts successful high-quality traces into explicit supervised signals for reproducing better reasoning patterns.
  • Alternating RL and RFT across cycles gradually transforms occasional success into more reliable capability and supports deeper reasoning in embodied models.

4.4 Large-to-Small On-Policy Distillation

The paper uses on-policy distillation to transfer the large model’s embodied reasoning to the compact model under the student’s own decoding states. This extends beyond offline teacher-response imitation by matching teacher and student token distributions along student-generated trajectories.

  • On-policy distillation targets preserving the large model’s embodied competence and thinking style within the compact model’s smaller capacity budget.
  • The student first generates its own response, after which the teacher is evaluated with teacher forcing on the same student-generated prefixes.
  • The student minimizes the average token-level KL divergence between teacher and student next-token distributions over student rollouts.
  • Because supervision occurs on states induced by the student policy, on-policy distillation reduces training–inference mismatch and transfers more than final-answer imitation.
  • OPD follows RL and RFT as the final bridge from large-model capability discovery to compact-model deployment.

5 Evaluation

Evaluation spans 22 benchmarks for visual perception, spatial understanding, and embodied understanding, plus qualitative reasoning and real-robot experiments. The compact model performs strongly across categories, while the A32B model reaches a higher overall score than several frontier baselines.

  • Evaluation Settings: The 22-benchmark evaluation covers foundational perception, 3D and multi-view spatial reasoning, environmental grounding, affordance recognition, trajectory prediction, and task planning.
  • MoT-2B Results: The MoT-2B model ranks first on 16 of 22 benchmarks and second on 4 additional benchmarks.
  • MoT-2B Results: Its strongest comparative advantage appears in spatial understanding, while it remains competitive in visual perception, embodied understanding, planning, and trajectory-intensive tasks.
  • MoT-2B Results: Despite its compact size, MoT-2B outperforms larger baselines on most benchmarks, indicating a balance between model size and embodied capability.
  • General Understanding: In general visual tasks, MoT-2B performs comparably to size-matched general VLMs while showing strong embodied and spatial understanding.
  • MoE-A32B Results: The MoE-A32B model achieves an overall score of 67.0 across 22 benchmarks, exceeding Gemini 3.0 Pro by 3.4 points and placing first on 7 tasks and second on 6.
  • Qualitative Results: Qualitative results show fine-grained visual localization, spatially grounded counting, sequential embodied reasoning, and long-chain reasoning across both model variants.

6 Robot Control Results

The HY-Embodied-0.5 VLA extends the MoT-2B base model for real-world robot control and is evaluated after broad pretraining followed by task-specific fine-tuning. It performs competitively across three physical manipulation tasks, with its strongest advantage on Mug Hanging.

  • Model and training: The VLA extends HY-Embodied-0.5-MoT-2B with an Action Expert module for real-world robot control experiments.The Action Expert follows the structural design of π0/π0.5.
  • Model and training: 5K hours of UMI data are used for initial fine-tuning before embodiment-specific supervised fine-tuning.The initial stage uses data without exposure to any specific robot embodiment.
  • Model and training: Task-specific deployment evaluations use 300–700 real-robot demonstration episodes, with π0 and π0.5 trained under identical conditions.The baselines share the same real-robot data volume and training iterations.
  • Results: 80% success on Tableware Stacking exceeds π0’s 60% and remains below π0.5’s 85%.The comparison uses the same task-specific evaluation setting.
  • Results: 75% success on Mug Hanging exceeds π0’s 45% and π0.5’s 50%, showing the largest reported advantage.Mug Hanging is described as the most challenging task given the baseline performances.

7 Conclusion

HY-Embodied-0.5 is presented as a foundation VLM for real-world embodied tasks, combining modality-adaptive perception with embodied post-training. Its benchmark and robotic manipulation results support translating digital intelligence into physical-world competence.

  • Conclusion: HY-Embodied-0.5 is designed as a foundation vision-language model for real-world embodied tasks.The model targets the divide between general VLMs and real-world agent demands.
  • Conclusion: The model combines a modality-adaptive Mixture-of-Transformers architecture with visual latent tokens for spatial and visual perception.These components are described as supporting physical grounding.
  • Conclusion: Its embodied post-training pipeline compresses complex reasoning capabilities into an efficient 2B-parameter variant for edge deployment.The conclusion frames this compression as part of the model’s efficiency strategy.
  • Conclusion: Performance across 22 demanding benchmarks and real-world robotic manipulation tasks demonstrates physical-world competence within the reported evaluations.The report identifies these benchmark and robot results as evidence for the model’s effectiveness.

A Contributors

The report identifies project sponsors, a supervisor, a project leader, core contributors, and additional contributors.

  • Project roles: Project sponsors are Zhengyou Zhang, Linus, and Shunyu Yao.
  • Project roles: Han Hu is the project supervisor.
  • Project roles: Yongming Rao is the project leader.
  • Project roles: Core contributors are Xumin Yu, Zuyan Liu, Ziyi Wang, and He Zhang.
  • Project roles: Additional contributors are Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, Yves Liang, Haitao Lin, Minghui Wang, Yubo Dong, Kevin Cheng, Bolin Ni, and Rui Huang.

B Visualizations

The visualizations illustrate HY-Embodied-0.5’s capabilities in visual grounding, spatial reasoning, and embodied trajectory prediction across perception and robot-control tasks.

  • Trajectory Prediction: The visualizations cover both end-effector and joint-control trajectory prediction settings.Separate examples are labeled for end-effector control and joint control.
  • Trajectory Prediction: Trajectory examples show the model planning normalized waypoint sequences for end-effector and joint-control tasks.The examples include moving a towel to a table corner and moving a spatula toward the table’s front edge.
  • 2D Bounding Box Grounding: The model grounds natural-language object descriptions to normalized 2D bounding boxes, including helmets and microwaves.The outputs use box tags with coordinates normalized to [0, 1000].
  • Point-Based Localization: Point-based localization examples identify points in specified regions or vacant spaces using normalized coordinates.The illustrated answers place multiple points within the designated red-marked areas.
  • Spatial Reasoning: Spatial reasoning examples infer directional relations from viewpoints and determine relative object distance.The examples reason about what lies to the right of a sneaker and identify the cabinet as farthest among listed objects.
  • Embodied Understanding: An affordance example selects the spoon as the relevant grasp point when the task requires placing it on a towel.The reasoning distinguishes the manipulable object from points on the towel and unrelated objects.
Loading 2604.07430v1…