PaperlayerView exact revision on arXiv

Source-linked AI summary

RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

Haotian Liang, Mingkang Chen, Yufei Huang, Yuchun Guo, Xiaomeng Zhu, Xiangli Shi, Kaixuan Wang, Yunxuan Mao, Weijie Zhou, Ling Chen, Shirong Zeng, Yueyu Long, Yuchen Si, Yajuan Zhu, Xingyu Zhou, Minghui Wang, Wanjia He, Xin Yang, Lingzhu Xiang, Zhiqing Liu, Bohan Ma, Xiran Huang, Tianshuo Yang, Zhiheng Liu, Xuantang Xiong, Zisheng Lu, Ping Luo, Yao Mu, Han Hu, Zhengyou Zhang

arXiv:2607.14187v1cs.AIcs.RO

TL;DR

Embodied agents need plans that connect abstract task reasoning with the physical states actions should produce. RxBrain unifies language reasoning and visual imagination in embodied planning, and experiments show coupled planning with promising multimodal and real-robot performance.

  • Problem

    Existing evaluations typically assess visual understanding, embodied reasoning, and video generation separately, leaving interleaved text–visual planning insufficiently evaluated.

  • Method

    RxBrain uses a unified multimodal architecture and training pipeline to represent plans as language-guided visual state predictions and joint subgoal planning.

  • Results

    RxBrain produces coupled textual and visual plans, achieves Splan = 0.68 on RxBrain-Bench-JointPlan, and shows encouraging real-robot action-generation performance without large-scale action-data pretraining.

  • Takeaways & Limitations

    The results provide an initial step toward embodied cognition models that jointly reason about task structure and the physical states required for execution.

  • Takeaways & Limitations

    The relatively small model may limit highly complex reasoning, fine-grained visual imagination, and long-horizon planning, while out-of-distribution settings require additional fine-tuning.

Abstract

from arXiv · show

Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.

1 Introduction

RxBrain is introduced as an embodied cognition foundation model that couples language reasoning with visual imagination within a single embodied planning sequence. The paper also presents an automatic video-to-supervision pipeline and RxBrain-Bench to evaluate joint textual and visual plan representation.

  • Motivation: Embodied cognition requires reasoning about both actions and the physical world states those actions should produce.Textual reasoning supports abstract task understanding, causal inference, constraint satisfaction, and long-horizon decomposition, while visual goal imagination represents desired physical states.
  • RxBrain: RxBrain integrates language reasoning and visual imagination in one planning sequence rather than treating them as separate capabilities.Language expresses task decomposition, planning primitives, constraints, temporal order, and decision logic; visual imagination grounds these elements through world-state prediction and joint subgoal planning.
  • Capabilities: RxBrain supports embodied visual question answering, multi-frame visual generation, and step-by-step joint textual reasoning with visual goal imagination.Given instructions and visual observations, it reasons about the current scene, imagines future or goal states, and generates joint language and visual planning steps.
  • Training: An automatic data-construction pipeline converts embodied videos into joint text-visual planning supervision.The pipeline addresses the scarcity and expense of manually annotating data that links actions to the physical states they should produce.
  • Evaluation: RxBrain-Bench evaluates whether embodied plans use complementary language and visual components, with language specifying actions, constraints, and decisions and images depicting goal and intermediate world states.The paper also evaluates general image generation, embodied understanding, and planning capabilities across broader embodied and multimodal benchmarks.

2 Model

RxBrain is a unified embodied foundation model that combines language and visual processing to support embodied reasoning, world-state prediction, and joint language-visual subgoal planning. Its modality-aware Mixture-of-Transformers architecture enables autoregressive text generation, future multi-frame prediction, and interleaved text-vision generation.

  • Model overview: RxBrain supports embodied understanding and reasoning, world-state prediction, and joint language-visual embodied subgoal planning through one unified model.These capabilities are enabled by a modality-aware Mixture-of-Transformers architecture.
  • Architecture: The modality-aware Mixture-of-Transformers routes tokens by modality to specialized Transformer pathways and exchanges information through global self-attention.The backbone includes language and vision attention branches plus modality-specific feed-forward experts for text, visual understanding, and visual generation.
  • Architecture: RxBrain shares the vision Transformer and attention projections between visual understanding and generation, while using a widened visual-generation FFN expert.The visual-generation expert’s intermediate dimension increases from 6,144 to 12,288.
  • Generation modes: The model supports autoregressive text generation, future multi-frame visual prediction, and interleaved text-vision generation for embodied planning.Multi-frame prediction generates four future image frames, while interleaved generation produces reasoning text and visual state predictions in a left-to-right trajectory.
  • Interleaved generation: In interleaved generation, RxBrain alternates reasoning text with imagined frames, using an <Image> token to trigger visual imagination and re-encoding each synthesized frame into the context.Reasoning uses autoregressive next-token prediction, while imagined frames are synthesized through flow matching in the VAE latent space.

3 Joint Planning Data Construction

The joint planning data pipeline converts embodied videos into verified text-visual planning samples that capture task semantics, physical state transitions, and multiple temporal levels. Its corpus spans four source categories, while annotation, verification, and structuring produce supervision for prediction, planning, and visual imagination.

  • Corpus: The verified corpus spans four source categories and contains 50,177.2 hours and 21,506,919 trainable segments from approximately 28.61 M candidates at a reported 75.18% pass rate.The categories are Real-Robot Data, UMI Data, Simulation Data, and Egocentric Human Data, across 46 processed source splits.
  • Pipeline: A three-stage pipeline—Temporal Segment Annotation, Quality Verification, and Segment Structuring—converts raw videos into continuous prediction, planning, high-level planning, and final-state imagining tasks.These tasks provide supervision from local state transitions to higher-level planning structures.
  • Temporal Segment Annotation: Temporal Segment Annotation localizes planning steps and pairs each with a short name, detailed description, start frame, end frame, and metadata.Keyframe sampling emphasizes visually informative changes, while boundary refinement selects frames where state changes start and complete.
  • Quality Verification: Quality Verification retains segments with clear visual transitions and accurate textual descriptions by checking semantic quality, visual grounding, and text-transition consistency.It rejects low-quality or near-duplicate segments, corrects inaccurate descriptions when possible, and retains 75.18% of candidate segments.
  • Segment Structuring: Segment Structuring organizes verified segments into packed joint text-visual samples with task wrappers, goal conditions, textual planning information, and visual context or target frames.The samples are constructed at temporal scales ranging from state changes within one step to the final state of a complete task, including L0–L3 levels.

L3 TASK

L3 trains RxBrain to imagine a whole task’s final outcome from its initial frame and concise goal, using a goal generated from the video’s complete subgoal sequence. Each video contributes at most one L3 sample, emphasizing long-range outcome prediction without specifying every intermediate step.

  • Hierarchical supervision: The hierarchical pipeline connects initial observations to final task states while organizing supervision from atomic state changes through step-level planning and subgoals.L0 models intra-step changes, L1 links consecutive steps, and L2 groups adjacent steps into higher-level subgoals.
  • L3 final-state imagination: L3 predicts the final frame from the initial frame and a concise task goal representing the whole task’s intended outcome.The task goal is generated from the full sequence of subgoals.
  • L3 final-state imagination: Each video contributes at most one L3 sample, training long-range task-outcome imagination without requiring every intermediate step.This final-state task complements the hierarchy’s lower-level visual state prediction and planning supervision.

4 Training

RxBrain uses a unified multimodal objective to jointly train language understanding, visual generation, and interleaved embodied generation on a shared Mixture-of-Transformers backbone. A two-stage curriculum then develops unified vision-language capabilities before adapting the model to embodied multimodal generation and planning.

  • Unified multimodal objective: RxBrain jointly optimizes language understanding, visual generation, and interleaved embodied generation, with modality-specific supervision sharing one Mixture-of-Transformers backbone.The shared backbone supports distinct generation modes while retaining specialized supervision objectives.
  • Unified multimodal objective: Language generation uses causal autoregressive prediction, with each token attending only to preceding text tokens and available multimodal context.This branch follows the standard next-token prediction formulation.
  • Unified multimodal objective: Visual generation predicts velocity fields in VAE latent space through flow matching and applies the learned field to future-frame prediction and image synthesis.Image synthesis solves the corresponding ordinary differential equation during inference.
  • Interleaved generation: Interleaved generation predicts each next text segment and visual state from preceding multimodal context, conditioning visual generation additionally on the current reasoning text.Training samples are ordered sequences of paired reasoning segments and visual states.
  • Interleaved generation: Inference-aware visual-state construction reconstructs and encodes training states through the same pipeline used at inference, reducing train-inference mismatch and interleave-rollout error accumulation.The overall objective combines text and vision losses weighted by λt and λv.
  • Two-stage curriculum: A two-stage curriculum first establishes unified vision-language understanding and generation, then fine-tunes on embodied instruction, multi-frame prediction, and interleaved planning data.Interleaved trajectories are converted into text-image generation samples pairing each visual state with its language context.

5 RxBrain-Bench

RxBrain-Bench evaluates embodied cognition across perception, reasoning, world-state prediction, and joint planning. Its three tracks test scene understanding, action-conditioned visual imagination, and plans that couple textual reasoning with visual goal states.

  • Benchmark scope: RxBrain-Bench spans action and trajectory understanding, state judgment, grounding, planning, task completion, error recovery, spatial relations, and limited multi-view and temporal reasoning.These dimensions systematically assess embodied perception, understanding, and reasoning.
  • Tracks: The benchmark comprises three tracks: EVQA, WorldPred, and JointPlan.Together, they cover embodied scene reasoning, visual goal imagination, and multimodal embodied planning.
  • EVQA and WorldPred: EVQA tests deployment-grounded reasoning from visual observations, while WorldPred requires generating a short future video showing the physical state transition caused by a natural-language action.EVQA uses industrial, retail, and service scenarios; WorldPred tests both action execution and resulting scene change.
  • JointPlan: JointPlan evaluates whether models represent embodied plans through complementary textual reasoning and visual goal imagination rather than treating understanding and visual prediction separately.It targets the benchmark’s core capability by combining text and images within the same plan.
  • Evaluation protocol: Evaluation uses exact-match accuracy for EVQA and MLLM-as-judge rubrics plus auxiliary frame-fidelity metrics for the two generative tracks.JointPlan runs autoregressively in a free-running closed loop, feeding generated text and visual states into subsequent predictions.

6 Evaluation

RxBrain is evaluated across text-to-image generation, embodied and spatial understanding, world state prediction, and joint subgoal planning. Results show strong unified performance, including competitive image generation, broad embodied understanding, and coupled multimodal planning, with visual imagination remaining a key limitation.

  • Evaluation setup: Evaluation covers four axes: text-to-image generation, embodied and spatial understanding, world state prediction, and joint subgoal planning.The first two use standard benchmarks and RxBrain-EVQA comparisons, while the generative tracks use dedicated evaluations.
  • Text-to-image generation: RxBrain scores 82.4 on GenEval, matching Bagel (82) and exceeding Cosmos-3-Nano (71.68), while preserving broad embodied understanding.This evaluates general open-domain image generation alongside embodied capabilities rather than treating them as separate model strengths.
  • Embodied and spatial understanding: RxBrain leads CV-Bench (88.59), EmbSpatial (82.3), DA-2k (83.4), 3DRSBench (54.1), MMSI-Bench (35.8), MindCube (47.5), and SITE-Bench-Image (54.9).It is second-best on SAT (74.33), ViewSpatial (47.3), and VSI-bench (43.53).
  • RxBrain-Bench-EVQA: RxBrain achieves 72.7 overall on RxBrain-Bench-EVQA, with best results on Success Detection (85.1) and Simulation Planning (51.6), and second-best State Estimation (69.8).The benchmark contains 1,123 multiple-choice questions and 258 simulation-planning samples grounded in robot-manipulation episodes.
  • World state prediction and joint subgoal planning: RxBrain achieves Sgen = 0.62 on WorldPred and Splan = 0.68 on JointPlan, with language reasoning stronger than visual generation and goal-image correctness a primary bottleneck.WorldPred’s weakest criterion is temporal and physical plausibility (0.53), while JointPlan reaches 0.83 in observation understanding and 0.78 in subtask planning versus 0.52 in goal-image correctness.

7 Extending RxBrain to Action Generation

RxBrain is extended to robotic action generation through a modality-specialized action branch that reuses pretrained multimodal and generative representations. On three real-world manipulation tasks, the method achieves strong success rates, averaging 87% across 100 trials per task.

  • Architecture: The action model adds a modality-specialized branch with dedicated attention, feed-forward, and normalization parameters while retaining shared global attention across action, observation, instruction, and state tokens.The branch is initialized by copying parameters from the vision understanding branch.
  • Gated Expert Fusion: A zero-initialized channel-wise gate fuses the pretrained generation expert with the action expert, initially preserving the standalone action model while learning to inject predictive and planning features during training.Both experts process the same post-attention action representation using branch-specific normalization.
  • Action representation and flow-matching head: Proprioceptive state is encoded into one token, future actions are represented as action chunks, and flow matching generates normalized end-effector pose commands from noise.Inference denoises action chunks with a few-step Euler ODE and then de-normalizes them into commands.
  • Experimental Setup and Tasks: The method is evaluated on three multi-stage household manipulation tasks using the DOBOT X-Trainer and ARX dual-arm platforms.The tasks are Set the Table, Fold and Store Glasses, and Pick Trash.
  • Results: 87% average success rate is achieved across three tasks, with 97% on Set the Table, 95% on Fold and Store Glasses, and 68% on Pick Trash.Each policy is evaluated with 100 real-robot trials per task, comparing the method with π0 and π0.5 using task success rate.

8 Related Work

Related work spans increasingly capable vision-language models for scene understanding and embodied reasoning, predictive world models for future-state imagination, and unified multimodal architectures that connect understanding, generation, and action. This progression motivates embodied cognition models that couple language reasoning with visual subgoal guidance for planning and verification.

  • Vision-language models: Vision-language models have progressed from contrastive image-text alignment to generative and natively multimodal systems supporting captioning, question answering, OCR, long-video understanding, and spatial grounding.These models provide increasingly reliable scene understanding for agents that must ground perception in the physical world.
  • Embodied vision-language models: Embodied VLMs incorporate sensor inputs, language-tokenized robot actions, web-scale vision-language knowledge, spatial grounding, and task-level planning.PaLM-E and RT-2 exemplify the integration of multimodal understanding with embodied reasoning and robotic control.
  • World models: World models predict environmental evolution through temporally coherent video generation, latent predictive representations, and action-conditioned visual forecasting for robot policies.Embodied world action models extend this predictive capability toward policy learning, controllable rollout, and long-horizon evaluation.
  • Unified multimodal models: Unified multimodal models combine visual understanding and generation within one architecture, with newer systems extending this integration to world simulation and action modeling.Examples include language-plus-diffusion training and Mixture-of-Transformers architectures for interpreting scenes and imagining future states.
  • Embodied cognition foundation models: Next-generation embodied cognition models are expected to add language-visual subgoal guidance, translating high-level goals into intermediate visual objectives for planning and verification.This direction complements embodied language reasoning and world-state prediction rather than treating understanding and generation as separate task pathways.
  • Embodied action generation: Embodied action generation increasingly uses large pretrained VLM- and world-model-based policies, including unified multimodal approaches that jointly model actions with visual and language modalities.The related work identifies unified multimodal action generation as an emerging direction for embodied policies.

9 Conclusion

RxBrain is presented as an embodied cognition foundation model that unifies language-visual reasoning and imagination to connect task structure with physical world states. The conclusion highlights its promise while noting limitations in scale and outlining future scaling efforts.

  • Conclusion: RxBrain represents embodied plans through joint language-visual reasoning and imagination, linking task logic with physical states to be achieved.Language specifies planning steps, constraints, and decisions, while visual imagination grounds them in goal and intermediate world states.
  • Limitations: The current model is relatively small, limiting highly complex reasoning, fine-grained visual imagination, and long-horizon planning.The passage suggests scaling both the model and training data to improve the quality and consistency of joint planning outputs.
  • Future work: Future work will scale RxBrain to larger model sizes to strengthen joint textual-visual reasoning, imagination, and embodied planning.The expected improvements include more coherent planning primitives, more accurate world state prediction, and more reliable joint subgoal planning.

A Contributors and Acknowledgements

The project acknowledges its sponsors, leaders, core contributors, broader contributors, and discussion collaborators. Zhengyou Zhang is identified as a corresponding author, while Haotian Liang and Mingkang Chen share equal contribution.

  • Project organization: Project sponsors were Zhengyou Zhang and Han Hu.
  • Project organization: Yufei Huang and Yuchun Guo served as project leaders.
  • Contributors: Core contributors included Haotian Liang, Mingkang Chen, Xiaomeng Zhu, Xiangli Shi, Kaixuan Wang, Yunxuan Mao, Weijie Zhou, and Ling Chen.Haotian Liang and Mingkang Chen are marked as equal contributors.
  • Contributors: Additional contributors included Shirong Zeng, Yueyu Long, Yuchen Si, Yajuan Zhu, Xingyu Zhou, Minghui Wang, Wanjia He, Xin Yang, Lingzhu Xiang, Zhiqing Liu, Bohan Ma, Xiran Huang, Xuantang Xiong, Zisheng Lu, Tianshuo Yang, and Zhiheng Liu.The paper also appreciates Ping Luo and Yao Mu for fruitful discussions; Zhengyou Zhang is identified as a corresponding author.

B Data Sources … C Additional Data Construction Details

The appendix details a verified multimodal corpus spanning 46 splits and 50,177 retained hours across real-robot, UMI, simulation, and egocentric human data. It also documents dataset-specific inclusion rationales, modalities, collection setups, and retained hours.

  • B Data Sources: The corpus contains 46 splits and 50,177 retained hours: 20 Real-Robot, 1 UMI, 3 Simulation, and 22 Egocentric Human splits.Each processed split is described by its collection setup, inclusion rationale, and hours retained after verification.
  • C Additional Data Construction Details: Additional construction details report per-source verification outcomes, distinct processing splits, collection modalities, inclusion purposes, and retained hours while avoiding unsupported release-scale claims.For example, AI Data curated and raw remain separate because they undergo different screening, and several sources report only verified retained hours when external references are unavailable.
  • B.1 Real-Robot Data: Real-Robot Data combines single- and dual-arm, humanoid, mobile, and multi-embodiment sources with RGB, depth, tactile, force, language, and robot-state signals.Examples include 107 thousand RoboMIND trajectories across 479 tasks and 96 object classes, plus more than 1 million AgiBot World trajectories across 217 tasks.
  • B.2 UMI Data: UMI data uses handheld universal manipulation grippers with wrist-mounted views that capture gripper-object contact, retaining 17,505.8 hours after verification.This is the single UMI split in the corpus.
  • B.3 Simulation Data: Simulation Data includes BEHAVIOR-1K, InternData-A1, and SynData, retaining 1,102.4, 197.8, and 17.2 hours respectively after verification.BEHAVIOR-1K adds scripted, precisely labeled state transitions across 1,000 activities, 50 scenes, and more than 5,000 objects.

C.1 Temporal Segment Annotation Method Details · C.2 Quality Verification Review Items · D Per-Scene Taxonomy Statistics

The appendix details a temporal-segment annotation pipeline that samples content-adaptive keyframes, parses candidate actions globally, and refines boundaries into visual anchors. It also specifies verification criteria and reports per-scene taxonomy distributions across fourteen scene categories.

  • C.1 Temporal Segment Annotation Method Details: The annotation pipeline fixes segment notation, samples keyframes adaptively, proposes candidate steps with one global parse, and refines boundaries bidirectionally into precise visual anchors.The pipeline is designed to produce supervision from weak video by combining global candidate generation with local boundary refinement.
  • C.1 Temporal Segment Annotation Method Details: Each annotation is an ordered sequence of segments, with each segment representing one atomic manipulation whose visible state change is bounded by start and end anchors.Repeated manipulations remain separate segments, and retained segments pass their textual and visual supervision to later structuring.
  • C.1 Temporal Segment Annotation Method Details: Keyframes are selected from downscaled LUV frames using smoothed visual-change scores, robust per-clip thresholds, and greedy minimum-spacing pruning.Frames are decoded at νs = min(4, ν), resized to 192 × 108, and threshold parameters use quantiles with a low-activity floor.
  • C.1 Temporal Segment Annotation Method Details: A single global MLLM parse receives chronological keyframes and returns a summary, goal, viewpoint, annotatability flag, and ordered candidate steps with temporal indices.Each candidate step contains a short name, detailed description, and start and end positions in the global keyframe set.
  • C.1 Temporal Segment Annotation Method Details: Boundary refinement corrects coarse onset bias by bracketing boundaries with dense windows, selecting the end before the start, and enforcing valid nondegenerate anchor pairs.Each window contains nc = 8 candidates; independently refined neighboring segments may overlap while retaining chronological order.
  • C.2 Quality Verification Review Items: Quality Verification checks localized semantic transitions, visual availability of agents and objects, frame and viewpoint validity, and text–visual consistency.Valid transitions with inaccurate descriptions are corrected, whereas invalid, ambiguous, corrupted, or inconsistent segments are rejected.
  • D Per-Scene Taxonomy Statistics: Per-scene taxonomy statistics cover fourteen scene categories, with each sunburst chart showing action-subtype shares through wedge angles proportional to verified trainable-segment counts.The figures present the subtype breakdown separately for each scene category.

E Training Configuration and Data Mixture

Training uses two stages: generation-heavy large-scale pretraining followed by understanding-heavy supervised fine-tuning. The stages differ in loss weighting, optimization settings, and data-mixture composition.

  • Stage 1: Stage 1 trains all HY-Embodied-0.5 modules and emphasizes flow-matching generation loss with λCE=0.25 and λFM=1.0.Stage 1 is large-scale pretraining.
  • Stage 2: Stage 2 resumes from the Stage 1 final checkpoint and balances the losses with λCE=λFM=1.0.It also uses a lower peak learning rate, longer packed sequence, and smaller per-device batch than Stage 1.
  • Data mixture: Stage 1 mixes two data buckets at integer ratio 6:4, yielding a 60.0% text-to-image generation bucket and 40.0% vision–language understanding bucket.The understanding bucket contains 113.4M examples from LLaVA-OneVision-1.5 and MAmmoTH collections.
  • Data mixture: Stage 2 mixes six buckets at integer ratio 100:12:150:70:90, with general and embodied understanding as the largest source at 34.6%.The two embodied generative buckets together account for 37.0%.

F RxBrain-Bench: Generation-Side Evaluation Protocol · G Inference Acceleration for Real-Robot Deployment

RxBrain-Bench evaluates interleaved language-image planning and continuous video generation with shared judge-based protocols, perceptual scoring, and composed baselines. For real-robot deployment, lossless inference optimizations reduce latency while preserving numerically equivalent actions.

  • F RxBrain-Bench: Generation-Side Evaluation Protocol: IL maps 1–3 observation frames and an instruction to interleaved subtask text and goal images across 4,756 held-out trajectories from 11 datasets.Planning depth n…
  • F RxBrain-Bench: Generation-Side Evaluation Protocol: Both generative tracks use GPT-5.5 VLM judging with JSON-schema criterion scores, two independent calls, and 512-pixel longest-side image resizing.The IL rubric combines textual criteria with locally computed image similarity.
  • F RxBrain-Bench: Generation-Side Evaluation Protocol: The perceptual pass computes LPIPS, CLIP cosine, and DINO cosine locally, with DINO cosine supplying IL image similarity and missing runs forfeiting the full 10% weight.Generation and perceptual processing are coordinated so judging receives the required image metrics.
  • F RxBrain-Bench: Generation-Side Evaluation Protocol: 0.522, 0.503, and 0.431 are the Splan scores for Cosmos3-Nano, BAGEL-7B-MoT, and Qwen-Agent, respectively.Cosmos3-Nano uses an image-to-video detour, BAGEL performs native image editing, and Qwen-Agent combines specialist reasoner and generator models.
  • F RxBrain-Bench: Generation-Side Evaluation Protocol: 0.575 versus 0.397 are the Sgen scores for Cosmos3-Nano and Wan2.2-TI2V-5B; their action-correctness scores are 0.60 and 0.39, respectively.Both baselines generate continuous image-to-video clips from the observation’s last frame and instruction, with four frames aligned to ground-truth keyframes.
  • G Inference Acceleration for Real-Robot Deployment: Real-robot acceleration targets observation lag and jitter in HyVLA-0.5 asynchronous serving while requiring single-trajectory numerical equivalence.The optimizations include prefix KV caching and action-only fast decoding over 16 action tokens instead of the approximately 989-token prefix.
  • G Inference Acceleration for Real-Robot Deployment: 210 ms to 143 ms is the combined latency reduction, corresponding to −32% and 1.47× speedup, while position deviation remains < 0.65 mm and orientation deviation < 0.73◦.The resulting 143 ms latency satisfies the real-time closed-loop control budget; FP8 quantization and image-token reduction provide no net gain at this model scale.
Loading 2607.14187v1…