Source-linked AI summary
PhysBrain 1.0 Technical Report
Shijie Lian, Bin Yu, Xiaopeng Lin, Changti Wu, Hang Yuan, Xiaolin Hu, Zhaolong Shen, Yuzhuo Miao, Haishan Liu, Yuxuan Tian, Yukun Shi, Cong Huang, Kai Chen
TL;DR
Robot trajectories provide limited coverage for broad physical understanding, motivating PhysBrain 1.0 to use human egocentric video as complementary supervision. The system converts video into structured physical QA, transfers the resulting priors through capability-preserving and language-sensitive adaptation, and reports strong results across multimodal and embodied benchmarks. Its scope remains bounded by annotation and depth-estimation errors, embodiment differences, and limited benchmark coverage.
Problem
Robot trajectories alone provide limited coverage for learning broad physical understanding across changing viewpoints, scenes, object states, and task compositions.
Method
PhysBrain 1.0 converts human first-person video into structured physical QA, then transfers the learned priors to robot control through capability-preserving and language-aware adaptation.
Results
PhysBrain 1.0 performs strongly across multimodal understanding and embodied-control benchmarks, including ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa-GR1.
Takeaways & Limitations
Human-derived physical commonsense supervision can serve as a bridge from multimodal understanding to downstream robot action within the evaluated settings.
Takeaways & Limitations
The approach remains limited by annotation and depth-estimation errors, differences between human and robot embodiments, and incomplete coverage of long-horizon real-world autonomy and severe distribution shift.
Abstract
from arXiv · showhide
Vision-language-action models have advanced rapidly, but robot trajectories alone provide limited coverage for learning broad physical understanding. PhysBrain 1.0 studies a complementary route: converting large-scale human egocentric video into structured physical commonsense supervision before robot adaptation. Our data engine extracts scene elements, spatial dynamics, action execution, and depth-aware relations, then turns them into question-answer supervision for training PhysBrain VLMs. The resulting physical priors are further transferred to VLA policies through a capability-preserving and language-sensitive adaptation design. Across multimodal QA benchmarks and embodied control benchmarks, including ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa, PhysBrain 1.0 achieves SOTA results and shows especially strong out-of-domain performance on SimplerEnv. These results suggest that scaling physical commonsense from human interaction video can provide an effective bridge from multimodal understanding to robot action.
1 Introduction
PhysBrain 1.0 shifts embodied training from relying mainly on robot trajectories toward acquiring physical commonsense from human egocentric video before robot adaptation. It builds structured physical supervision for a stronger VLM, then transfers those priors to control while preserving multimodal capability and language alignment.
- Robot trajectories alone may not teach physical regularities needed for robust action across viewpoint, scene, object-state, and task changes.
- PhysBrain 1.0 first builds a multimodal model with stronger physical understanding, then adapts it to embodied control.
- Human first-person video offers broader, easier-to-obtain supervision that exposes contact, reachability, state change, tool use, spatial constraints, and multi-step structure.
- A schema-driven pipeline extracts structured scene information before generating physically grounded QA instead of using raw video directly.
- PhysBrain 1.0 transfers physical priors into robot control through an adaptation design intended to preserve multimodal capability and language alignment.
- The system performs well across multimodal understanding and embodied-control benchmarks, including ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa-GR1.
2 PhysBrain 1.0 Data Engine
The PhysBrain 1.0 data engine compiles egocentric video into explicit physical records and diverse QA supervision. Its staged pipeline represents scene elements, spatial dynamics, action execution, depth relations, and quality checks before training.
- Generic captions omit physical structure needed for action generation, including geometry, contact progression, relative distance, and reachability.
- The pipeline extracts explicit scene meta-information describing objects, physical attributes, spatial arrangement, depth relations, and action-driven scene changes.
- Raw video is parsed into a physical record, augmented, checked, and rendered into QA through constrained stage interfaces that expose errors before propagation.
- Egocentric clips from Ego4D, BuildAI, and EgoDex are filtered and converted into structured records, while different data subsets support extraction, depth augmentation, QA generation, and broad vision-language competence.
- Each source record contains scene_elements, spatial_dynamics, and action_execution, separating physical appearance and state from spatial change and actionable motion.
- Depth augmentation enables relative-depth, absolute-depth, and metric-distance QA for more physically grounded spatial reasoning.
- QA generation uses the shared physical record to create diverse questions about layout, depth, state, actions, safety, temporal order, and reasoning rather than one caption per clip.
- The training target encourages an embodied reasoning order—perceive, infer state, plan, execute—while structural and modality-specific checks reduce unsupported artifacts without eliminating semantic or depth errors.
3 PhysBrain 1.0 Architecture
PhysBrain 1.0 adapts a physically informed base VLM to robot control through dual pathways that preserve general multimodal capability while training an embodied action pathway. Language-aware action conditioning and flow-matching decoding connect physical understanding to continuous control.
- PhysBrain 1.0 Architecture: The architecture keeps a frozen general pathway while training a separate embodied pathway to specialize in robot action prediction.The embodied pathway receives semantic features from the general pathway through asymmetric fusion, while stop-gradient features prevent action learning from updating the general pathway.
- PhysBrain 1.0 Architecture: PhysBrain 1.0 first trains a base VLM on physically grounded QA derived from structured scene information in human egocentric video.The supervision targets object state, spatial layout, depth, feasible interaction, temporal dynamics, and multi-step task structure.
- PhysBrain 1.0 Architecture: Robot demonstrations map previously learned physical priors onto a particular embodiment and action space rather than teaching all physical regularities from scratch.This transfer completes the pipeline from human-video-derived physical QA to capability-preserving robot control.
- PhysBrain 1.0 Architecture: Prior action queries attend to vision without direct language access, whereas posterior action queries attend to both vision and language.Their paired representations support an action-conditioned language alignment objective that encourages instruction-relevant action representations.
- PhysBrain 1.0 Architecture: The action decoder uses a flow-matching objective to generate continuous robot actions from language-conditioned action-query states.The predicted trajectory uses an end-effector-frame action space with translational and rotational components.
4 Experiments
PhysBrain 1.0 improves both multimodal understanding and embodied control by pretraining on physically grounded supervision and adapting to embodiment-specific robot data. It achieves the best reported average performance across the four VLA evaluations, with especially strong gains on SimplerEnv and RoboCasa-GR1.
- VLM Experiment Results: PhysBrain 8B achieves the best scores on ERQA, PhysBench, MME, MMMU, OCRBench, and TextVQA, while PhysBrain 4B leads RealWorldQA.Compared with Qwen3-VL-8B, PhysBrain 8B improves ERQA from 43.0 to 45.5 and PhysBench from 48.5 to 50.2.
- VLA Experiment Results: 80.2% average success on SimplerEnv-WidowX is 1.0 percentage point above Xiaomi-Robotics-0 and 23.1 points above π0.5 and Isaac-GR00T-N1.6-Bridge.The benchmark trains on BridgeV2 data and evaluates on four held-out simulation tasks, testing out-of-domain generalization.
- VLA Experiment Results: 91.33% average success on SimplerEnv-GoogleRobot is 2.30 percentage points above Xiaomi-Robotics-0.PhysBrain 1.0 also improves Move Near from 88.8% to 94.8% and reaches 100.0% on Pick Coke Can.
- VLA Experiment Results: 64.5% average success on RoboCasa-GR1 is 10.7 percentage points above VP-VLA and 15.7 points above QwenOFT with Qwen3VL.The result covers 24 tabletop tasks with a GR1 robot using bimanual dexterous hands.
- VLA Experiment Results: 98.8% average success on LIBERO slightly exceeds Xiaomi-Robotics-0's 98.7%, while remaining strong across all four task suites.LIBERO is near saturation for several recent VLA systems, so the reported margin is smaller than on SimplerEnv and RoboCasa-GR1.
- VLA Experiment Results: PhysBrain 1.0 achieves the best average score in every reported VLA table across heterogeneous embodiments, task distributions, and evaluation protocols.The largest gains occur on RoboCasa-GR1 and the two SimplerEnv settings, while LIBERO matches or slightly exceeds prior results.
5 Real-World Experiments
The real-world experiments evaluate PhysBrain 1.0 on Franka tabletop vegetable manipulation, comparing it with π0.5 under matched post-training data and trial protocols. PhysBrain 1.0 achieves higher single-object grasping success across all evaluated categories and gains especially on challenging objects.
- Experimental Setup: The experiments use a Franka Research 3 arm with a Robotiq 2F-85 gripper for tabletop vegetable grasping requiring geometric, material, and contact understanding.The setup includes diverse vegetables and two RGB cameras: external and wrist-mounted.
- Data Collection and Post-Training: Each vegetable category contributes 50 SpaceMouse-controlled demonstration trajectories covering varied initial poses and orientations.The demonstrations are recorded in LeRobot 3.0 format and used for embodiment-specific post-training.
- Evaluation Protocol: The evaluation reports task success over 50 independent trials, requiring stable object lifting for grasping and full instruction completion for long-horizon tasks.Long-horizon instructions require multi-step selection, planning, grasping, and placement.
- Baselines: The comparison with π0.5 uses identical Franka demonstration data and the same 50-trial protocol, isolating the effect of pre-adaptation physical priors.π0.5 is a vision-language-action flow model pretrained on large-scale robot demonstrations.
- Results: 63.3%: PhysBrain 1.0 succeeds in 285 of 450 single-object grasping trials, compared with π0.5 at 47.1% from 212 of 450 trials.The average gain is 16.2 percentage points, with improvements on every evaluated single-object category and notable gains for deformable, ambiguous, and smooth objects.
6 Discussion
PhysBrain 1.0 shifts physical learning toward human first-person video while retaining robot trajectories for embodiment-specific adaptation. Its architecture aims to preserve general multimodal capabilities and language sensitivity, but the approach remains limited by annotation, depth, embodiment, and benchmark coverage.
- Training Logic: Robot trajectories remain necessary for grounding the model in a concrete embodiment, action parameterization, and benchmark distribution, but need not carry all physical learning.Human first-person video supplies priors about objects, spatial relations, distance, state changes, and dynamics before robot adaptation.
- Human-Derived Physical Priors: Human video contains scalable priors about contact, friction, reachability, stability, and feasible motion that are not inherently robot-specific.Examples include adjusting wrist orientation for awkward approaches, slowing for fragile objects, and searching around occluded handles.
- Capability Preservation: The dual-pathway design keeps a stable general pathway during robot adaptation while a trainable embodied pathway learns action prediction.The language-aware action objective addresses capability erosion when robot data are limited.
- Limitations: The data engine can retain semantic mistakes, missing objects, ambiguous contacts, and incorrect physical interpretations despite staged error detection.Depth-aware supervision also inherits local inaccuracies from depth estimation and object grounding, especially for transparent, reflective, or heavily occluded objects.
- Scope Boundaries: Human priors do not match robot morphology, workspace, force limits, or sensing, so adaptation remains required for executable policies.Benchmark results also do not cover all long-horizon autonomy, deformable-object interaction, safety-critical execution, or severe distribution shift.
7 Conclusion
PhysBrain 1.0 implements an understanding-first strategy by converting human interaction video into structured physical supervision before robot adaptation. It transfers these priors through capability-preserving, language-aware control components, repositioning robot trajectories as an embodiment-specific adaptation layer.
- Conclusion: PhysBrain 1.0 converts human first-person interaction video into physically grounded supervision that strengthens VLM understanding of objects, space, depth, dynamics, planning, and execution.The strategy is organized around understanding first and action next.
- Data Engine: The data engine separates structured scene meta-information from final model supervision.This separation supports explicit records over scene elements, spatial dynamics, action execution, and depth-aware relations before generating QA.
- VLA Adaptation: The architecture transfers physical priors into robot control through a physically informed base model, capability-preserving adaptation, a language-aware action objective, and a continuous action decoder.These components connect structured physical understanding with action generation.
- Implications: Robot trajectories serve as an adaptation layer mapping human-derived physical priors onto a specific embodiment and action space.The conclusion presents this as a pragmatic view of robot data efficiency and a direction for scaling physical-world understanding before action imitation.
8 Contributions
The paper’s contributions span project leadership, data-engine design and quality control, model architecture, multimodal and embodied evaluation, real-robot experiments, and writing. The listed roles identify contributors for each area.
- Data Engine: Data engine design is attributed to Xiaopeng Lin, Hang Yuan, Xiaolin Hu, Changti Wu, Yuzhuo Miao, and Yuxuan Tian.
- Data Operations: Data annotation is attributed to Changti Wu, Yuzhuo Miao, Xiaolin Hu, Hang Yuan, and Shijie Lian, while quality control is attributed to Hang Yuan, Xiaolin Hu, Yuzhuo Miao, Xiaopeng Lin, and Bin Yu.
- VLA Development: VLA model architecture is attributed to Shijie Lian, Bin Yu, and Xiaopeng Lin, with VLA training and evaluation attributed to Bin Yu, Shijie Lian, Xiaopeng Lin, and Zhaolong Shen.
- Evaluation: VLM training and evaluation is attributed to Xiaopeng Lin, Shijie Lian, Bin Yu, and Changti Wu, while real-robot experiments are attributed to Zhaolong Shen, Xiaopeng Lin, Shijie Lian, and Bin Yu.
- Writing: Writing is attributed to Shijie Lian, Bin Yu, Haishan Liu, Zhaolong Shen, and Xiaopeng Lin.
- Project Leadership: Project leadership is attributed to Kai Chen, Cong Huang, and Yukun Shi.