PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence

Xiaopeng Lin, Shijie Lian, Bin Yu, Ruoqi Yang, Zhaolong Shen, Changti Wu, Yuzhuo Miao, Yurun Jin, Yukun Shi, Jiyan He, Cong Huang, Bojun Cheng, Kai Chen

arXiv:2512.16793v2cs.RO

TL;DR

Egocentric VLA requires long-horizon understanding and planning, but current models struggle and large-scale robot data is costly to collect. The paper converts human egocentric videos into multi-level supervision for PhysBrain; it reports improved egocentric capabilities and strong VLA performance, including sample-efficient adaptation.

  • Problem

    Current multimodal models struggle with egocentric long-horizon understanding and planning, while robot manipulation data is costly to scale and how to use human videos as supervision remains an open question.

    Source: §1 · pp. 1–2

  • Method

    The Egocentric2Embodiment pipeline converts human egocentric videos into multi-level VQA supervision, forming E2E-3M for training PhysBrain.

    Source: §1 · p. 3, §6 · p. 12

  • Results

    Across egocentric VLM and VLA evaluations, PhysBrain improves egocentric capabilities and VLA performance; PhysBrain-8B achieves 67.4% average success rate, surpassing Isaac-GR00T at 57.1% and comparable to RoboBrain2.5 at 67.6%.

    Source: §1 · p. 2, §5.2.2 · p. 11, §6 · p. 12

  • Takeaways & Limitations

    Human egocentric supervision provides a physically grounded source for embodied-brain training and supports sample-efficient VLA adaptation without robot-data pretraining.

    Source: §1 · p. 3

Abstract

from arXiv · show

Robotic generalization relies on physical intelligence: the ability to reason about state changes, contact-rich interactions, and long-horizon planning under egocentric perception and action. Vision Language Models (VLMs) are essential to Vision-Language-Action (VLA) systems, but the reliance on third-person training data creates a viewpoint gap for humanoid robots. Collecting massive robot-centric data is an ideal but impractical solution due to cost and diversity constraints. Conversely, human egocentric videos offer a highly scalable data source with rich interaction context, yet the embodiment mismatch prevents the direct application. To bridge this gap, we propose an Egocentric2Embodiment Translation Pipeline that transforms raw human egocentric videos into multi-level, schema-driven embodiment supervision with enforced evidence grounding and temporal consistency, enabling the construction of the Egocentric2Embodiment dataset (E2E-3M) at scale. An egocentric-aware embodied brain, termed PhysBrain, is obtained by training on the E2E-3M dataset. PhysBrain exhibits substantially improved egocentric understanding, particularly for planning. It provides an egocentric-aware initialization that enables more sample-efficient VLA fine-tuning and higher success rates, demonstrating effective transfer from human egocentric supervision to downstream robot control.

1 Introduction

VLA performance depends on embodied brains that ground planning and interaction in the agent’s own perceptual stream, yet current models struggle with egocentric long-horizon understanding. This work turns human first-person videos into structured supervision and reports gains in egocentric understanding and VLA performance.

1 Introduction

  • Current multimodal models struggle with egocentric long-horizon understanding, planning, and reliability amid viewpoint changes, hand–object occlusions, and partial body visibility.The paper attributes these demands to cross-frame inference about contact and object state.Source: §1 · pp. 1–2
  • Robot manipulation data is costly and difficult to scale, while human first-person videos offer diverse everyday interaction data and an open route to egocentric supervision.Robot-data collection is constrained by hardware, labor, safety, and lengthy collection cycles.Source: §1 · p. 2
  • The Egocentric2Embodiment pipeline converts human egocentric videos into structured, multi-level first-person VQA supervision for embodied-brain learning.Its supervision includes planning decompositions, key states, interaction constraints, and temporal relations, extending beyond static visual recognition.Source: §1 · pp. 2–3
  • PhysBrain improves egocentric VLM benchmarks, especially perception and planning, and achieves strong VLA results; few-shot adaptation is sample-efficient and exceeds large-robot-data-trained VLA systems without robot-data pretraining.The figure summarizes egocentric VLM and VLA benchmark evaluations.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Human egocentric videos provide effective supervision for embodied brains in egocentric settings and improve generalization in VLA tasks.The paper presents E2E-3M as a structured, validated egocentric VQA dataset.Source: §1 · p. 3

2 Related Work

Prior work documents gaps in egocentric vision-language understanding and advances in VLA systems and human-demonstration learning. This paper positions its contribution upstream of robot-action alignment, translating human egocentric data into supervision for embodied brains.

2.1 First-Person Vision Language Model

2.2 Vision Language Action

  • VLA research unifies language understanding, visual perception, and motor control, with work expanding cross-embodiment pretraining, reasoning, world models, and reinforcement learning.The surveyed approaches also address language forgetting and action prediction.Source: §2.2 · p. 3

2.3 Learning VLAs from Human Demonstration

  • Human-demonstration VLA approaches align human data with robot action spaces, while this work targets upstream embodied-brain supervision to complement robot-data pipelines.The related methods include human–robot action spaces, hand-motion codes, hand pose, and human hands as proxy end-effectors.Source: §2.3 · p. 4

3 Egocentric Embodied Supervision

Section 3 presents a pipeline for turning human egocentric video into structured, validated embodied supervision and the E2E-3M dataset. It describes the pipeline stages, the dataset’s source domains, and its object and action diversity.

3.1 Egocentric2Embodiment Translation Pipeline

  • The schema-driven Egocentric2Embodiment engine converts raw egocentric videos into multi-level supervision for embodied planning and interaction reasoning.Source: §3.1 · p. 4

3.1.1 Data Intake and Pre-processing

  • Clips use scenario-aware fixed-interval, event-driven, or kinematic-aware segmentation; temporal spans and episode metadata condition downstream annotation.Source: §3.1.1 · pp. 4–5

3.1.2 Annotation Scheme

  • Seven VQA modes and mode-specific templates standardize annotation, while VLM engines generate visually grounded answers using egocentric hand references and manipulation-specific phrasing.Source: §3.1.2 · p. 5

3.1.3 Quality Assurance and Validation Logic

  • A validation loop checks metadata grounding, egocentric hand consistency, and mode-specific temporal logic; failed samples are regenerated, and a human audit checks retained supervision quality.Source: §3.1.3 · p. 5

3.1.4 Structured Egocentric Corpus

  • Retained entries record frames, VQA mode, QA pair, and validation outcome, supporting traceability and reproducibility of structured supervision for planning and interaction reasoning.Source: §3.1.4 · p. 5

3.2.1 Data Sources and Domain Coverage

  • E2E-3M combines Ego4D household activities, BuildAI factory workflows, and EgoDex laboratory manipulation sequences for complementary coverage of embodied experience.Source: §3.2.1 · pp. 5–6

3.2.2 Diversity Analysis

  • Diversity analysis measures per-domain noun coverage and mode-specific verb diversity, normalizing verb counts by QA-pair count to account for interaction density.Source: §3.2.2 · p. 6
  • Household data has high object diversity, Factory data lower diversity, and Lab data falls between them; action-centric modes retain high lexical diversity across domains, including Factory Mechanics and Reasoning.Source: §3.2.2 · p. 6

4 Methodology

PhysBrain is obtained by fine-tuning base VLMs on embodied supervision from egocentric videos mixed with FineVision data. PhysVLA pairs the VLM’s high-level representations with a flow-matching action expert to study transfer to downstream control.

4 Methodology

  • PhysBrain is produced by supervised fine-tuning base VLMs on translated egocentric supervision mixed with an equal-sized FineVision subset.The passage names Qwen3-VL-4B and Qwen3-VL-8B as example base models.Source: §4 · p. 7
  • The study evaluates transfer of PhysBrain’s egocentric gains under standard VLA instantiations while avoiding a new VLA architecture and keeping the action expert lightweight.The setup follows a GR00T-style paradigm and aims to minimize confounding from additional heuristics or hand-crafted priors.Source: §4 · p. 7
  • PhysVLA uses the VLM as System 2 for high-level multimodal representations and a flow-matching action expert as System 1 for continuous actions.The action expert is conditioned on the last-layer hidden states of PhysBrain.Source: §4 · p. 7, Figure 4 · p. 7
  • The action expert denoises an action trajectory by cross-attending to VLM features and predicts a time-independent velocity conditioned on those features and optional proprioceptive state.The stated regression objective trains the velocity prediction; inference uses 8 denoising steps to produce an action chunk of K=16.Source: §4 · p. 7
  • Table 1 compares performance on EgoPlan and EgoThink benchmarks, with detailed subtasks belonging to EgoThink and its overall average in the final column.The supplied caption does not provide benchmark values.Source: Table 1 · p. 8

5 Experiment

The experiments assess egocentric understanding and VLA control, then test how results vary with model architecture and egocentric-supervision scale.

5.1.1 Egocentric Understanding Evaluation

  • Evaluation covers multiple-choice EgoPlan benchmarks and open-response EgoThink, with prompts standardized across models.EgoThink generations are scored using a single GPT-4 judging protocol.Source: §5.1.1 · pp. 8–9

5.2.2 Experiment Results

  • 67.4% average success on SimplerEnv surpasses Isaac-GR00T (57.1%) and is comparable to RoboBrain2.5 (67.6%).PhysBrain-8B achieves this using E2E-3M pretraining; the table compares VLA models in the WidowX SimplerEnv simulation.Source: Table 2 · p. 9, §5.2.2 · p. 11

5.1.1 Egocentric Understanding Evaluation

  • PhysBrain-8B reaches 47.4/46.9 on EgoPlan-Benchmark1/2 versus Qwen3-VL-8B’s 44.3/40.5, with the most pronounced improvement in planning.PhysBrain-4B also improves over its Qwen3-VL-4B backbone on both benchmarks.Source: §5.1.1 · p. 9

5.2.2 Experiment Results

  • 55.25% is PhysBrain-8B’s best average success rate on RoboCasa; PhysBrain-4B reaches 49.75%, above QwenGR00T’s 47.8%.The comparison includes GR00T, OFT, and FAST action encodings.Source: Table 3 · p. 10, §5.2.2 · p. 11

5.2.1 Experiment Settings

  • The VLA evaluation uses four WidowX tasks in SimplerEnv and 24 RoboCasa tasks, reporting Avg@50 over 5 and 50 trials, respectively.PhysBrain is fine-tuned within the VLA paradigm using downstream robotics data before evaluation.Source: §5.2 · p. 9, §5.2.1 · p. 11

5.3 Ablation Study

  • Fine-tuning on the dataset improves performance over base models across 4B, 7B, and 8B configurations, including a 19.5% VLA-benchmark gain for the 7B model.Source: §5.3 · p. 11
  • Removing Ego4D lowers PhysBrain-8B performance on both VLM and VLA benchmarks, linking the full dataset’s scale to stronger results.The authors report a positive relationship between egocentric-supervision volume and model proficiency.Source: §5.3 · pp. 11–12, Table 5 · p. 11

6 Conclusion

The paper uses human egocentric videos to connect vision-language models with physical intelligence for robotic generalization. E2E-3M-trained PhysBrain improves egocentric capabilities and achieves high success rates as a VLA backbone.

6 Conclusion

  • Human egocentric videos are used to connect vision-language models with physical intelligence for robotic generalization.The paper reports nearly three million verified E2E-3M instances across household, factory, and laboratory settings.Source: §6 · p. 12

A.1 Dataset Composition

This appendix describes E2E-3M’s domains and examples, the dataset’s frequent nouns and verbs, and benchmark prompt formats.

A.1 Dataset Composition

  • E2E-3M contains approximately three million verified instances across household, factory, and laboratory domains, covering varied objects and interactions.Figure 5 gives an overview of dataset composition.Source: §A.1 · p. 17, Figure 5 · p. 17

A.2 Verb and Noun Distribution

  • The dataset’s frequent nouns include “table,” “surface,” and “box,” while common verbs include “hold,” “grasp,” and “move.”The word clouds summarize object and action diversity across the dataset.Source: §A.2 · pp. 17–18

A.3 Example Scenarios

  • Examples span Laboratory, Factory, and Household scenarios, with seven modes covering mechanics, trajectory, spatial, attribute, temporal, summary, and reasoning information.Examples include rolling a shirt and sanding wood.Source: Figure 6 · p. 17, §A.3 · p. 18

B.1 Benchmark Prompt Details

  • EgoPlan uses multiple-choice prompts, whereas EgoThink uses open-ended questions; all models receive the same prompt format.The prompts encourage shorter, focused answers to reduce scoring differences associated with response length.Source: §B.1 · p. 18

B.2 VLM Implementation Details

The VLM implementation fine-tunes Qwen3-VL models using LoRA.

B.2 VLM Implementation Details

  • Qwen3-VL-4B and Qwen3-VL-8B are fine-tuned with LoRA for one epoch on 32 NVIDIA H100 GPUs.Training uses batch size 2, learning rate 5e-4, AdamW weight decay 0.1, and a cosine scheduler with 0.05 warmup ratio.Source: §B.2 · p. 19

C.1 VLA Implementation Details

VLA fine-tuning initializes language-model weights from PhysBrain or VLM baselines and uses distributed training across eight GPUs.

C.1 VLA Implementation Details

  • VLA fine-tuning runs for up to 100K steps across 8 GPUs, with a per-device batch size of 16.Gradient accumulation is 1 and gradient clipping uses a maximum norm of 1.0.Source: §C.1 · p. 19
  • Training uses AdamW with a learning rate of 4e-5 and cosine learning-rate scheduling.The optimizer citation is Loshchilov and Hutter (2019).Source: §C.1 · p. 19
  • DeepSpeed with ZeRO2 accelerates the distributed training.The DeepSpeed citation is Rajbhandari et al. (2020).Source: §C.1 · p. 19

D.1 More Complementary Evaluation on E2E Dataset

E2E-3M fine-tuning complements VST’s static spatial priors, with particularly strong gains in egocentric movement. Simulation visualizations also show PhysBrain completing tasks in SimplerEnv and handling long-horizon household activities in RoboCasa.

D.1 More Complementary Evaluation on E2E Dataset

  • E2E-only supervised fine-tuning of VST tests whether E2E supervision adds egocentric and dynamic spatial reasoning beyond its existing spatial priors, without task-specific training samples.VST is pretrained on large-scale, high-quality spatial intelligence datasets.Source: §D.1 · p. 19
  • VST’s overall accuracy rises from 45.33 to 59.33 after E2E-3M fine-tuning, while Egocentric Movement rises from 26.09 to 91.30.Action Consequence improves from 54.05 to 64.86 and Perspective from 39.39 to 48.48; Object Movement changes from 39.13 to 34.78 and Goal Aim remains 58.82.Source: §D.1 · p. 19

D.2.1 Simulation validations on SimplerEnv and RoboCasa

  • In four SimplerEnv rollouts, PhysBrain completes precise pick-and-place and block-stacking tasks with temporal consistency and precise spatial reasoning despite limited robot-specific fine-tuning.The tasks require object-pose perception and fine-grained control.Source: §D.2.1 · p. 19, Figure 7 · p. 20
  • RoboCasa trajectories show PhysBrain performing long-horizon household activities with articulated objects, smooth interaction dynamics, and robust planning in cluttered kitchens.The visualizations concern a more visually complex and physically realistic environment.Source: §D.2.1 · p. 19, Figure 8 · p. 21
Loading 2512.16793v2…