PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models

DeepCybo Team, Yu Bin, Haipeng Cao, Zheng Chang, Kai Chen, Youning Chen, Kailin Deng, Yichao Du, Xiaotong Fu, Haoyang Ge, Yunlong Guo, Chenliu Hao, Jiyan He, Xuguo He, Yakun Hou, Kai Hu, Cong Huang, Tuopusen Huang, Yu Huang, Hong Li, Peize Li, Shijie Lian, Xiaopeng Lin, Yun Lin, Haibao Liu, Haochen Liu, Qiuzhi Liu, Shengcai Liu, Zhiqiang Liu, Tao Luo, Peng Ren, Shuo Ren, Chaoyi Ruan, Zhaolong Shen, Yukun Shi, Qiyuan Su, Yuxuan Tian, Yining Wang, Changti Wu, Hao Wu, Xueyin Xu, Ruoqi Yang, Zhaoyang Yang, Hang Yuan, Zhaoyang Zeng, Hanwen Zhang, Ruimeng Zhang, Yao Zhang, Yibo Zhang, Yuxiang Zhang, Zhirui Zhang, Ziyi Zhang, Zubin Zheng, Zishen Zhuang

arXiv:2609.14973v1cs.CVcs.RO

TL;DR

PhysBrain 1.5 addresses the challenge of learning broad embodied understanding alongside action generation and future-state prediction despite heterogeneous supervision. It uses a shared autoregressive model over discrete language, motion, and visual tokens, pretrained with human interaction experience and adapted on human, robot, and simulated data. Across 28 embodied benchmarks, its 8B model scores 72.5 on average, reaches a new open-source state of the art, and retains broad multimodal capabilities.

  • Problem

    Embodied learning needs a shared formulation that accommodates language, spatial, motion, and dense visual supervision while supporting physical generation and broad understanding.

  • Method

    PhysBrain 1.5 jointly models language responses, end-effector trajectories, and future RGB, depth, and robot-mask states as discrete tokens under a shared autoregressive objective.

  • Results

    Across 28 embodied understanding benchmarks, the 8B model achieves a 72.5 average score, a new open-source state of the art, and approaches leading proprietary systems.

  • Takeaways & Limitations

    The results support learning from interaction experience as an effective route toward capabilities needed for the physical loop, alongside demonstrated trajectory and future-scene prediction.

  • Takeaways & Limitations

    Action data preserve each source's native coordinate axes because reliable global canonicalization is difficult and some datasets lack calibration or global-frame information.

Abstract

from arXiv · show

We present PhysBrain 1.5, a unified model for understanding physical environments, generating actions, and predicting future states. Motivated by the physical loop of observation, interaction, and environmental change, we bring these capabilities into a common learning framework. Starting from a general vision--language model, we encode language responses, end-effector motion, and dense visual targets as discrete sequences and jointly optimize them with autoregressive next-token prediction. Pre-training draws its embodied supervision entirely from human interaction videos, using task-centered episodes to pair semantic and spatial context with recovered motion and subsequent observations. We then adapt the model through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Across 28 embodied understanding benchmarks, our 8B model achieves an average score of 72.5, setting a new open-source state of the art and performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on 14 benchmarks while retaining general multimodal capabilities. Beyond these understanding evaluations, qualitative examples show the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.

1 Introduction

PhysBrain 1.5 frames embodied intelligence as a physical loop and unifies understanding, action generation, and future-state prediction in one learning framework. It uses discrete multimodal tokens, human interaction supervision, and later mixed embodied data to achieve strong results across broad benchmarks.

  • Motivation: The model addresses the physical loop, in which agents observe environments, interpret goals and spatial relations, anticipate outcomes, act, and learn from changed observations.This framing motivates reusable capabilities for understanding, acting, and predicting within physical environments.
  • Related Work: The work builds on models that separately or jointly study spatial reasoning, grounding, planning, robot behavior, predictive world modeling, and multimodal physical-environment understanding.Prior efforts include general VLMs, specialized embodied systems, VLAs, predictive models, and recent unified approaches.
  • Motivation: Joint learning must accommodate heterogeneous supervision spanning language, spatial targets, temporally structured motion, dense visual outputs, embodiments, annotations, and temporal granularities.The formulation is designed to support physical generation alongside broad embodied understanding despite these differences.
  • Approach: PhysBrain 1.5 unifies embodied understanding, end-effector action generation, and future-state prediction through discrete tokens and a shared autoregressive backbone.Language, motion, RGB, depth, and robot-mask outputs are trained with a common next-token objective and task-specific loss masks.
  • Training: Embodied pre-training derives supervision entirely from human interaction experience, extending an understanding-first approach toward joint learning across the physical loop.The broader system later incorporates human, robot, and simulated interaction data through supervised fine-tuning.
  • Results: 72.5 average score across 28 embodied benchmarks establishes a new open-source state of the art, with first-place results on 14 benchmarks and close performance to GPT-6-Astra and Gemini 3.6 Flash.The 8B model ranks first on 14 benchmarks and second on 10 among open-source models.

2 Model Architecture

PhysBrain 1.5 extends a pretrained vision–language model into a shared autoregressive system for embodied understanding, action generation, and future visual-state prediction. It represents language, motion, and visual targets as discrete tokens while preserving task-specific output structures and motion context.

  • Unified representation and objective: Language, action, and visual-state outputs are represented in one expanded vocabulary and optimized with masked next-token prediction.Task formatting and loss masks select whether the model answers in language, generates an action trajectory, or produces a visual-state sequence.
  • Embodied action generation: Action generation predicts 16-step wrist trajectories using relative translation, 6D relative orientation, and absolute gripper closure.Human and robot motions share a 10D end-effector representation, while ActionPiece discretizes trajectories into action-token sequences.
  • Overview: PhysBrain 1.5 jointly trains embodied understanding, end-effector action generation, and future visual-state prediction through shared backbone, embeddings, and output head.The three objectives provide complementary supervision for a shared physical representation.
  • Output decoding: Action and visual decoders convert shared token sequences into end-effector trajectories and future scene representations, respectively.The architecture retains task-specific decoders while training all task families through a common autoregressive interface.
  • Embodied action generation: Optional preceding action chunks provide local motion context, turning prediction into conditional trajectory continuation across embodiment-specific conventions.The model uses recent motion to preserve continuity across action-chunk boundaries without requiring globally canonicalized coordinate frames.
  • Visual-state generation: Future-state generation predicts spatially aligned RGB, depth, and robot-mask outputs as interleaved discrete visual tokens.The three modalities share a VQ representation and are serialized in RGB–depth–mask order at corresponding spatial locations.

3 Data

The data pipeline organizes physical-aware pre-training and embodied SFT around perception, action, and future-state prediction, using human interaction videos plus robot and simulated experience. It converts heterogeneous observations, annotations, motions, and future states into filtered, temporally aligned training samples.

  • Data organization: PhysBrain 1.5 organizes supervision into perception, action generation, and future-state prediction across physical-aware pre-training and embodied SFT.Pre-training derives these supervision types from human interaction videos, while SFT adds real-robot and simulated interactions.
  • Physical perception data: 24.3M physical-perception samples combine image- and video-level annotations with grounded captions and physical VQA from human interaction videos.Targets cover detection, segmentation, depth, pointing, counting, 3D detection, temporal grounding, tracking, task recognition, progress, and state changes.
  • Human action data: 31.2M human-action samples pair recovered wrist trajectories with task descriptions, observations, and temporally corresponding action continuations.Samples are filtered using pose-recovery reliability, video–trajectory alignment, and motion plausibility checks.
  • Future-state data: 26.8M human-interaction future-state samples pair pre-interaction context with aligned future RGB, depth, and human-part masks after temporal and cross-modal validation.The targets describe appearance, depth structure, and interacting human regions at a common future timestamp.
  • Embodied SFT data: Embodied SFT combines human interaction data, real-robot trajectories, simulated interactions, and 1M general language and vision-language instruction samples.The embodied-action collection spans 17 real-robot and simulation sources plus approximately 500 hours of high-quality human motion data.

4 Training

PhysBrain 1.5 starts from Qwen3-VL-Instruct (8B), extends its vocabulary for physical outputs, and trains in two stages under a shared autoregressive objective. The training setup jointly updates text, action, and future visual-state generation.

  • Training setup: PhysBrain 1.5 uses Qwen3-VL-Instruct (8B) and extends its vocabulary with action and visual-state tokens.The implementation uses ms-swift with a Megatron-Core backend.
  • Training stages: Two stages—physical-aware pre-training followed by embodied SFT—jointly optimize text, action-token, and future visual-state generation.Each stage runs for one epoch, with the second initialized from the first-stage checkpoint and data mixed by reported sample counts.
  • Optimization: The model trains with distributed Adam, a 3% learning-rate warmup, cosine decay, 32,768-token sequences, sequence packing, and bfloat16 precision.The peak learning rate is 2 × 10^-5, with gradient-norm clipping at 1.0.

5 Evaluation

Evaluation covers embodied understanding, general multimodal capabilities, action continuation, and future visual-state prediction. PhysBrain 1.5 leads open-source embodied models while retaining broad multimodal performance and producing qualitatively coherent physical predictions.

  • 5.1 Embodied understanding: 72.5 average across 28 embodied benchmarks makes PhysBrain 1.5 the top evaluated open-source model, with 14 benchmark wins and top-two placement on 24.It surpasses Hy-Embodied-VLM-1.0 by 6.5 points, beats Qwen3-VL-Instruct on all 28 benchmarks, and ranks top two across all five capability categories.
  • 5.1 Embodied understanding: Qualitative embodied examples span perception, 3D and multi-view reasoning, planning, grounding, affordance identification, and waypoint-based trajectory reasoning.The model counts objects, estimates distances, decomposes tasks, localizes placement regions, and generates image-space waypoints.
  • 5.2 General multimodal understanding: General multimodal performance remains comparable to the Qwen3-VL-Instruct (8B) base across image, video, visual reasoning, text, chart, and diagram understanding.The evaluation includes 12 established image and video benchmarks covering hallucination, document understanding, fine-grained recognition, and GUI grounding.
  • 5.3 Physical generation: Offline action prediction generates next-16-step end-effector trajectories from an RGB observation, task instruction, and preceding 16-step action history.Illustrated ID and OOD predictions broadly reproduce reference motion trends, though positional deviations remain visible and the evidence is not closed-loop execution.
  • 5.3 Physical generation: One-second future-state predictions preserve scene layout and task-consistent changes across spatially coherent RGB, depth, and robot-mask outputs.The examples cover diverse robot embodiments and viewpoints.

6 Conclusion

PhysBrain 1.5 unifies embodied understanding, action generation, and future-state prediction around the physical loop through shared discrete-token autoregressive modeling. Its reported results combine strong embodied benchmark performance with broad multimodal capability and qualitative physical prediction.

  • Conclusion: PhysBrain 1.5 represents language, end-effector trajectories, and multimodal visual states as discrete tokens under one autoregressive backbone.Physical-aware pre-training uses human interaction videos, while SFT adds human, real-robot, and simulated interaction data.
  • Conclusion: 72.5 average performance across 28 embodied benchmarks establishes a new open-source state of the art while retaining broad multimodal understanding.Qualitative results also show trajectory generation and spatially coherent future RGB, depth, and robot-mask predictions.

B.1 Embodied Evaluation Protocols and Metric Standardization

The evaluation protocol standardizes embodied benchmarks through shared interfaces and a unified point-localization metric. The metric combines localization precision and target coverage while using instance masks and one-to-one matching to assess predictions more completely.

  • B.1.1 Framework and Protocol Alignment: EmbodiedEvalKit3 supplies consistent input protocols, prompting templates, and evaluation metrics across a broad benchmark collection.RoboVQA uses all 16 official frames instead of the framework’s eight-frame setting; trajectory tasks report 100 − NRMSE so higher scores consistently indicate better performance.
  • B.1.2 Unified Point-Localization Metric: Point-localization tasks use micro-averaged F1 rather than heterogeneous precision- or recall-only criteria.The unified score jointly penalizes invalid predictions and missed targets.
  • B.1.2 Unified Point-Localization Metric: Validity is determined by whether predicted points fall inside ground-truth target regions, with one-to-one point-to-region matches measuring covered targets.Instance masks are preferred over coarse bounding boxes when available, including for RoboRefit.
  • B.1.2 Unified Point-Localization Metric: PixMo-Points uses distance-based Hungarian matching before validating assigned predictions against corresponding instance masks.Counts are aggregated over the complete benchmark to compute micro-averaged precision and recall.
  • B.1.2 Unified Point-Localization Metric: The metric treats explicit no-object predictions as correct when samples contain no valid target, while PointBench retains additional point-count requirements.This extends evaluation beyond simple coordinate proximity.
  • B.1.3 Additional Benchmarks: Six additional benchmarks are integrated from their official releases while preserving their released questions, choices, visual inputs, ordering, and top-1 accuracy scoring.The complete ERQA-PLUS release contains 1,766 questions, ViewSpatial-Bench contains 5,712, and MindCube TinyBench contains 1,050.

B.2 General Multimodal Evaluation Protocol

The general multimodal suite spans image, video, spatial, document, chart, and hallucination-related capabilities. Action-token perplexity is also evaluated on in-distribution and held-out RoboDojo data during pre-training.

  • B.2 General Multimodal Evaluation Protocol: The suite evaluates visual perception, multimodal reasoning, real-world scene understanding, video reasoning, object hallucination, diagram and chart reasoning, and document-text understanding.MME, MMStar, RealWorldQA, VideoMME, MVBench, POPE, AI2D, ChartQA, DocVQA, and TextVQA cover complementary task types.
  • B.3 Action Perplexity: Figure 7 evaluates action-token perplexity on disjoint in-distribution test episodes and RoboDojo data excluded from training.The RoboDojo evaluation measures behavior on an out-of-distribution source.

B.2.1 Image Evaluation

Image evaluations use a Qwen-aligned inference profile with fixed generation, sampling, seed, and image-area settings. ScreenSpot and selected VQA tasks use specified coordinate, extraction, fallback, and split protocols.

  • B.2.1 Image Evaluation: The Qwen-aligned image profile uses non-thinking inference, 32,768 maximum generated tokens, temperature 0.7, top-p 0.8, top-k 20, and seed 3407.Repetition penalty is 1.0 and presence penalty is 1.5.
  • B.2.1 Image Evaluation: Image inputs use a maximum area of 4,014,080 pixels and minimum areas of 602,112 for RealWorldQA and 200,704 for other image tasks.These settings define the image-resolution bounds used in evaluation.
  • B.2.1 Image Evaluation: ScreenSpot uses tool-call coordinates on a fixed 0–1000 grid, while RealWorldQA uses rule-based extraction with a shared local Qwen3 8B fallback.The fallback differs from Qwen’s remote fallback judge; DocVQA and TextVQA use validation splits.

B.2.2 Video Evaluation

Video evaluation uses Qwen3-VL preprocessing with benchmark-specific frame limits, aspect-ratio-preserving resizing, and fixed pixel budgets. VideoMME and MVBench are scored from option-letter responses using accuracy-based protocols.

  • B.2.2 Video Evaluation: Video preprocessing targets 2 FPS and caps clips at 128 frames for VideoMME and 32 for MVBench.The default minimum is four frames, subject to source availability, with frame counts rounded down to multiples of two.
  • B.2.2 Video Evaluation: Frames are resized with bicubic interpolation while preserving aspect ratio and aligning dimensions to multiples of 32.A default per-frame ceiling of 786,432 pixels is further constrained by the total video budget.
  • B.2.2 Video Evaluation: VideoMME is evaluated on 2,700 test questions across short, medium, and long videos using overall multiple-choice accuracy without subtitles or audio.Answer options are scored through benchmark-specific extraction rules without an external LLM judge.
  • B.2.2 Video Evaluation: MVBench covers 20 subtasks and 4,000 examples, reporting the unweighted mean accuracy across subtasks.It uses the clips supplied by the benchmark’s video release.
  • B.3 Action Perplexity: Figure 7 shows action-token perplexity falling from 17.12 to 5.07 on ID data and from 12.39 to 6.57 on held-out RoboDojo during pre-training.Despite local fluctuations, RoboDojo perplexity follows an overall downward trend.

B.4 Qualitative Visualizations

Qualitative visualizations demonstrate PhysBrain 1.5 across pointing, trajectory generation, and spatial reasoning tasks beyond the 28 embodied benchmarks. The examples illustrate generalization across action-oriented and multimodal spatial scenarios.

  • B.4 Qualitative Visualizations: Examples drawn outside the 28 evaluation benchmarks span pointing, trajectory point generation, and spatial relationship question answering.The appendix uses these visualizations to illustrate generalization to a broader range of scenarios.
  • B.4 Qualitative Visualizations: Pointing examples cover diverse complex scenarios, while trajectory examples predict target-object movement from action instructions.
  • B.4 Qualitative Visualizations: Question-answering examples include counting people by clothing and locating objects relative to other scene elements.The examples answer both visual counting and spatial-relation questions, including cheese-grater and onion locations.
  • B.4 Qualitative Visualizations: Spatial visualizations include challenging reasoning, object scale estimation, orientation reasoning, object-motion understanding, and future-video prediction.
Loading 2609.14973v1…