PaperlayerView exact revision on arXiv

Source-linked AI summary

Orca: The World is in Your Mind

Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, Runze Xiao, Huaihai Lyu, Senwei Xie, Euan Liu, Klara Tian, Tianfeng Long, Yichi Zhang, Zhengliang Cai, Ruike Chen, Jifan Zhao, Ruochuan Shi, Zihan Tang, Jing Lyu, Wenxing Tan, Ningbo Zhang, Yangtao Hu, Yuming Gao, Xiansheng Chen, Junkai Zhao, Congsheng Xu, Boan Zhu, Ziqi Wang, Yupu Feng, Qiongqiong Zhang, Yingli Zhao, Yulong Ao, Shaoxuan Xie, You Liu, Guocai Yao, Leiduo Zhang, Xiaodan Liu, Yunyan Zhang, Yance Jiao, Xinyan Yang, Jiaxing Wei, Xu Liu, Tengfei Pan, Shaokai Nie, Chunlei Men, Sen Cui, Xiaojie Jin, Hongyang Li, Jianlan Luo, Yao Mu, Yunchao Wei, Jun Yan, Hang Zhao, Xiaolong Zheng, Jiaming Li, Yonghua Lin, Tiejun Huang, Zhongyuan Wang, Pengwei Wang

arXiv:2606.30534v2cs.CV

TL;DR

Existing approaches often model isolated tokens, frames, or actions rather than a unified representation of multimodal world states. Orca learns a world latent space through next-state prediction and tests it with text, image, and action readouts, which improve consistently as pre-training scales.

  • Problem

    A central research gap is how to model evolving world states from continuously learned multimodal signals rather than isolated task-specific predictions.

  • Method

    Orca learns a unified world latent space through next-state prediction, combining dense video transitions with sparse language-described event and VQA supervision for multimodal readouts.

  • Results

    Text, image, and action readouts consistently improve as Orca’s pre-training scales, supporting the feasibility and scalability of its world-learning paradigm.

  • Takeaways & Limitations

    Orca provides an early exploratory milestone toward general-purpose world foundation models by shifting modeling from isolated outputs to internal world-state representations.

  • Takeaways & Limitations

    Experiments are mainly limited to 4B and 0.8B models, and current training uses only one-tenth of the constructed inventory data.

Abstract

from arXiv · show

We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

1. Introduction

Orca is introduced as a world learner that builds a world latent space from multimodal signals, addressing intelligence beyond isolated next-token, next-frame, or next-action prediction. It combines unconscious and conscious learning to model dense natural and sparse meaningful state transitions, then exposes the latent through multimodal readouts.

  • Motivation: The introduction argues that general intelligence should continuously absorb multimodal world signals to model latent states and internalize physical laws, causal relationships, and dynamic evolution.This motivation frames Orca as an initial step toward continuously learning and self-evolving intelligence.
  • Orca overview: Orca learns a world latent space from multimodal visual and language signals as a general interface for downstream text, image, and action readouts.Visual signals include videos and images, while language provides causal explanations and task intentions.
  • Learning paradigms: Unconscious learning predicts next-frame latents from continuous video to capture natural, dense state transitions without labeled tags.The supervision is provided by the continuous video itself.
  • Learning paradigms: Conscious learning uses textual constraints to learn meaningful, sparse event-level state transitions associated with decisions and task outcomes.This paradigm models state transitions under instruction constraints.
  • Evaluation scope: Orca evaluates whether its paradigms are feasible and scalable and whether stronger world modeling improves downstream readouts, rather than targeting task-specific SOTA.The backbone is frozen during decoder post-training, with only lightweight readout modules trainable.

2. Orca

Orca models multimodal world signals as latent states and state transitions, using unconscious and conscious learning to capture dense natural and sparse meaningful dynamics. Its encoder learns a unified world latent space, which modality-specific decoders read out.

  • Macro: Orca maps multimodal world signals into latent states and models transitions driven by invisible dynamics and explicit conditions, including future prediction and past backtracking.Signals may include language, vision, audio, physical measurements, and beyond-human modalities; explicit conditions can be human instructions.
  • Encoder: Orca’s encoder uses visual and language signals with a pretrained vision-language model to learn a unified world latent space through both learning paradigms.The encoder is designed for state abstraction and state transition modeling.
  • Details: Unconscious learning derives dense, natural transitions from observation alone, while conscious learning uses language-described events, intentions, or causal premises to model sparse, meaningful transitions.Unconscious learning uses nearest-future observations; conscious learning can target adjacent future or past events.
  • Encoder: For unconscious learning, Orca predicts the latent representation of the next adjacent video frame from the current frame, with the target encoded by a frozen vision encoder.The prediction is produced through two MLP layers and trained against the next-frame latent representation.
  • Encoder: For conscious learning, Orca predicts a latent from an event-conditioned frame and instruction, and also performs VQA response generation from videos and related questions.Event segments are defined by meaningful events, with instructions referring to adjacent future or previous events.
  • Decoder: A modality-specific decoder reads multimodal information from the learned latent space; decoder details are deferred to Section 3.2.The decoder is not the focus of this section.

3. Training

Orca trains a unified world latent in pre-training, then freezes its backbone and trains lightweight modality-specific readouts for language, vision, and action. Pre-training combines observation-only transitions, event-conditioned transitions, and VQA response generation over large-scale video, event, and VQA data.

  • Training strategy: Orca trains in two stages: pre-training learns the world latent, while downstream training freezes the backbone and updates only modality-specific readout modules.The readouts support language, vision, and action information.
  • Learning Objectives: Pre-training combines observation-only state transitions, event-conditioned state transitions, and VQA response generation using learnable queries for transition objectives and the LM head for VQA.The first two objectives supervise latent states rather than reconstructing pixels; VQA uses standard next-token prediction.
  • Training Data: The training inventory contains real-world video, event, and VQA data supporting dense dynamics, language-described transitions, and world-state interpretation.It includes 125K hours of video, 160M event annotations, and 11.5M VQA examples, with only one-tenth of the video used in this version.
  • Downstream Readouts: Frozen Orca latents feed separate language, image, and action readouts, whose trainable modules express text, generate pixels, or produce robot action chunks.The vision readout uses an MLP adaptor with frozen SD3.5 components, while the action readout uses an MLP adaptor and a DiT-based Action Expert.
  • Optimization: Throughput increases from 0.66 to 2.91 Samples/Sec/GPU, achieving approximately a 4.4× acceleration compared to StarVLA.The optimization details and results are reported in Appendix D.

4. Evaluation

Orca’s next-state-prediction paradigm scales with model size and pre-training data, while stronger frozen world latents improve zero-shot text, image, and action readouts. Across these evaluations, Orca achieves the best overall result among same-size VLMs and large-size world models, with complementary objectives shaping different downstream capabilities.

  • Scaling analysis: Total loss decreases with more pre-training video, and the 4B model achieves lower objective loss than the 0.8B model, showing scalable learning.The loss continues downward rather than converging quickly, indicating continued benefits from more data and larger models.
  • Scaling analysis: As pre-training data increases, text, image, and action readouts all improve despite a frozen backbone and no action-labeled pre-training data.The downstream evaluations are zero-shot and use no benchmark-specific training data or benchmark tuning.
  • Downstream evaluation: Orca achieves the best overall result among same-size VLMs and large-size world models, demonstrating advantages of its proposed learning paradigm.
  • Image prediction: Orca improves visual state-transition prediction, achieving the best average PRICE performance while preserving morphology, scene consistency, contact relationships, and instruction following.Compared with image-generation baselines, Orca reduces irrelevant-object teleportation, hallucinated hands, poor instruction adherence, and prior-knowledge biases.
  • Action generation: Orca outperforms Qwen3.5 in all out-of-distribution action settings from scratch, reaches beyond a 0% success-rate baseline, and remains comparable to pre-trained 𝜋0.5.Its trajectories make more intermediate progress, stagnate less, and recover more effectively from execution errors.
  • Ablation analysis: Joint observation, event, and VQA objectives provide the most balanced readouts, with observation supervision especially aiding action and event supervision especially aiding vision.VQA preserves language interfaces and strengthens semantic grounding, while the three objectives constrain natural dynamics, semantic conditions, and language supervision.

5. Conclusion

Orca introduces a world-learning paradigm centered on a unified world latent space learned from multimodal signals and exposed through dedicated readouts. The paper identifies limitations in modalities, supervision, scale, evaluation, horizons, readouts, objectives, and embodiment, while proposing directions toward native world-state modeling and broader evaluation.

  • Conclusion: Orca learns internal world-state representations before exposing them through dedicated readout interfaces, rather than optimizing isolated downstream tasks.This paradigm shifts modeling toward a shared world latent space supporting multiple downstream capabilities.
  • Discussion & Limitation: Orca remains limited by primarily vision-language signals, frozen ViT-space supervision, insufficient model scale, and limited benchmark coverage.The paper calls for richer sensory and physical signals, native world-space learning from scratch, larger models, and more comprehensive real-world evaluation.
  • Discussion & Limitation: Current event annotations mainly supervise short-horizon transitions, limiting modeling of long-term evolution across hours, days, or longer periods.The paper also notes trade-offs among language, image, and action readouts as pre-training progresses, especially in the 0.8B model.
  • Discussion & Limitation: The learned latent is currently read out only through language, vision, and action, while the loss design remains insufficiently consistent with Next-State-Prediction.The authors identify hearing, quantum circuits, proteins, and simpler supervision as areas requiring future expansion.
  • Discussion & Limitation: Embodied evaluation is stringent but uses relatively short and easy tasks, limiting conclusions about broader embodied capability.The paper presents this as a current limitation of its embodiment settings rather than a resolved evaluation scope.
  • Future Works: Future work proposes a unified state-transition evaluation system covering prediction, intervention response, physical quantifiability, and counterfactual inference.Additional directions include aligning more modalities to shared states, self-evolving data-generation loops, and extending world representations toward scientific domains.

6. Author List · Appendix

The appendix lists core contributors, contributors across operational and data roles, expert consultants, and research leads. Contributions span pre-training, infrastructure, evaluation, real-robot work, product operations, and research leadership.

  • 6.1. Core Contributors: Core contributors are credited for model pre-training, data infrastructure, evaluation, real-robot work, and downstream post-training.The core-contributor list also names Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, and Runze Xiao among its primary contributors.
  • 6.1. Core Contributors: The core-contributor assignments distinguish technical development from evaluation and embodied deployment responsibilities.The listed categories include model pre-training, data infrastructure, evaluation, real-robot work, and downstream post-training.
  • 6.2. Contributors: Additional contributors support infrastructure, real-robot data, product and operations, brand management, platform management, and system management.These contributors include Yingli Zhao, Yulong Ao, Shaoxuan Xie, You Liu, Guocai Yao, Leiduo Zhang, Xiaodan Liu, Yunyan Zhang, Yance Jiao, Xinyan Yang, Jiaxing Wei, Xu Liu, Tengfei Pan, Shaokai Nie, and Chunlei Men.
  • 6.2. Contributors: The appendix recognizes organizational roles beyond research and engineering, including product, brand, platform, and system management.These categories appear alongside infrastructure and real-robot data contributions in the contributor list.
  • 6.3. Expert Consultant (Ordered by English last name alphabetically): Expert consultants are listed alphabetically by English last name: Sen Cui, Xiaojie Jin, Hongyang Li, Jianlan Luo, Yao Mu, Yunchao Wei, Jun Yan, Hang Zhao, and Xiaolong Zheng.The appendix explicitly labels this group as expert consultants and specifies the ordering convention.
  • 6.4. Research Leads: Research leads are identified as Jiaming Li, Yonghua Lin, Tiejun Huang, Zhongyuan WangB, and Pengwei WangB.The appendix separately presents these five individuals under the research-leads category.

A. Orca Conception … B.2. Next Token Prediction

Orca centers modeling on multimodal next-state prediction, learning a unified world latent through complementary observation-driven and semantically conditioned transitions. The related-work discussion distinguishes this formulation from latent-prediction, next-token, and multimodal autoregressive paradigms.

  • A. Orca Conception: Orca shifts from next-token, next-frame, and next-action prediction toward modeling a unified world latent for understanding, prediction, and intervention.Language, vision, and action serve as different observations or readouts of the same underlying world state.
  • A. Orca Conception: Unconscious learning captures dense natural dynamics from continuous visual experience, while conscious learning uses language, events, instructions, and questions to model meaningful state transitions.The two modes target physical regularities and semantic conditions associated with causal explanations and task intentions.
  • A. Orca Conception: The learned world latent supports readouts to language for explanation and reasoning, vision for prediction and imagination, and action for intervention.
  • B. Related Work: Related work is organized by each paradigm’s primary learning objective rather than by the full set of downstream capabilities its models may exhibit.Unified models spanning multiple capability domains are categorized by their dominant training formulation.
  • B.1. Self-Supervised Learning: JEPA-style models predict semantic representations for self-supervised visual learning, whereas Orca broadens latent prediction into multimodal world-state transitions under implicit dynamics and explicit semantic conditions.The comparison positions Orca’s observation-only state transition as related to, but broader than, JEPA-style representation learning.
  • B.2. Next Token Prediction: Recent multimodal systems blur token- and frame-prediction boundaries, with Emu3, Emu3.5 (Cui et al., 2025), and BAGEL (Deng et al., 2025) unifying multimodality under next-token prediction.Emu3.5 addresses the slow image-generation speed and poor performance described for this autoregressive paradigm.
  • B.2. Next Token Prediction: Unlike next-token models that organize knowledge through autoregressive language modeling, Orca uses language as a semantic condition guiding state transitions toward target states.VQA response generation preserves the language interface while strengthening commonsense and semantic grounding.

B.3. Next Frame Prediction … C.1.1. Pre-Training Objective Definitions

Orca distinguishes world modeling from frame synthesis and action-centered learning by prioritizing physically grounded, interaction-conditioned state transitions. Its pre-training combines observation-only and event-conditioned latent transitions with VQA generation under a weighted objective and mixed sampling schedule.

  • B.3. Next Frame Prediction: Image-generation models map language or multimodal conditions to visual observations, while video-generation models extend this prediction to temporally coherent visual sequences.The passage frames these models as frame-level prediction approaches spanning static images and videos.
  • B.3. Next Frame Prediction: Orca targets physically constrained state transitions rather than merely synthesizing visually coherent frames, emphasizing action execution, consistency, plausibility, contact relationships, and instruction following.These priorities apply to interaction-conditioned transitions involving scenes and objects.
  • B.4. Next Action Prediction: Unlike VLA and world-action models centered on action or joint video-action prediction, Orca learns scene changes, object motion, and local physical interactions before using the representation for action.Orca does not use action labels during pre-training, following a world-learning-first philosophy.
  • C. Training Settings: The appendix provides implementation details for the training procedure described in Section 3.This establishes the scope of the training-settings material without specifying additional procedures in the supplied passage.
  • C.1.1. Pre-Training Objective Definitions: Orca’s pre-training objective combines observation-only state-transition loss, event-conditioned state-transition loss, and VQA response-generation loss.The first two losses match predicted visual latents to frozen vision-encoder latents, while VQA uses standard next-token prediction.
  • C.1.1. Pre-Training Objective Definitions: Lpre = 0.1 Lobs + 0.5 Levt + 0.4 Lvqa, with state-transition and VQA samples mixed at an approximate 5 : 1 ratio.Event-conditioned loss averages transitions toward adjacent earlier and later event states selected by language conditions.

C.1.2. Query-Based Implementation … C.2.3. Action Readout

The appendix specifies Orca’s query-based state-transition implementation and downstream readout interfaces. These readouts expose the learned latent through language, image generation, and robot action prediction using lightweight modality-specific training components.

  • C.1.2. Query-Based Implementation: Orca implements observation-only transitions with a learned query whose hidden state enters a two-layer visual transition head, while target latents come from the frozen vision encoder.Given v_t and Query 1, the model predicts the latent of the temporally next frame; the ground-truth latent is obtained from the frozen VLM vision encoder.
  • C.1.2. Query-Based Implementation: For event-conditioned transitions, an instruction specifies the transition direction and target event, while Query 2 reads out the corresponding instruction-conditioned predictive state.The model predicts a latent from a randomly selected frame in the instruction-specified target event using the previous-event and next-event directions.
  • C.1.3. Pre-Training Hyperparameters: Table C1 organizes Orca’s pre-training hyperparameters into model-scale, optimization, and objective-specific settings.The appendix identifies Table C1 as the source of the main pre-training hyperparameters.
  • C.2.1. Language Readout: The language readout reuses the VLM backbone’s language-modeling head to autoregressively produce responses for VQA, event interpretation, and causal explanation.It introduces no additional trainable module and takes a visual observation plus instruction as input.
  • C.2.2. Vision Readout: The vision readout feeds Orca’s predicted visual latent through an MLP adaptor into a pretrained Stable Diffusion 3.5 decoder, training only the adaptor and decoder-attention LoRA while freezing VAE and MMDiT weights.The adaptor projects the latent into SD3.5’s joint conditioning space, and target images are resized to 768 × 768 during training.
  • C.2.3. Action Readout: The action readout uses a DiT-based Action Expert with flow-matching loss to predict short-horizon robot action chunks from Orca latents, noisy actions with time embeddings, and proprioception.The ground-truth action chunk is Gaussian-noise perturbed, and the expert predicts the corresponding velocity.

D. Infrastructure … E.2. Image Prediction

Orca’s infrastructure redesign addresses the higher memory and communication costs of multimodal training through distributed, memory-efficient, and communication-overlap techniques. The supplied evaluation passages define text-generation benchmarks and baselines, while no passages describe E.2 image-prediction settings or results.

  • D. Infrastructure: Orca’s infrastructure targets multimodal training bottlenecks through distributed sharding, memory-efficient execution, and communication optimization.The training integrates visual embedding, language modeling, future visual-latent prediction, and action-related branches, increasing memory pressure and communication cost relative to standard VLM training.
  • D. Infrastructure: FSDP2 enables flexible parameter, gradient, and optimizer-state sharding, while resharding reduces peak GPU memory and unsharded lightweight blocks avoid excess communication overhead.These changes replace the DeepSpeed backend and aim to improve multi-GPU efficiency while maintaining training stability.
  • D. Infrastructure: Activation recomputation trades additional computation for substantial activation-memory savings, enabling larger batch sizes and improving throughput under memory constraints.Selected activation boundaries are checkpointed and reconstructed during backpropagation instead of retaining all intermediate activations.
  • D. Infrastructure: Chunked cross-entropy loss partitions the token dimension to avoid materializing the full logits tensor, reducing peak memory under long-sequence and large-vocabulary settings.The method addresses the log-softmax intermediate tensor’s memory spike in the VLM forward stage.
  • D. Infrastructure: Forward/backward pre-fetching overlaps upcoming-layer parameter all-gathers with current-layer computation, reducing communication-induced GPU idle time and improving device utilization.The technique addresses stalls exposed by FSDP2 all-gather operations.
  • D. Infrastructure: 2.91 samples/sec/GPU on H100 GPUs is 3.0× higher than the FSDP2 baseline and 4.4× higher than the StarVLA training pipeline.This result comes from the optimized infrastructure described in Table D1.
  • E.1. Text Generation: Text generation is evaluated with MVBench, TemporalBench, and 3DSRBench, covering video understanding, fine-grained temporal dynamics, and 3D spatial reasoning.MVBench uses multiple-choice QA, TemporalBench tests temporal properties and operational prediction, and 3DSRBench evaluates height, location, orientation, and multi-object reasoning.
  • E.1. Text Generation: The text-generation baselines include V-JEPA 2.1 as a latent world-model baseline and Emu3 as a unified discrete-token multimodal baseline.V-JEPA 2.1 emphasizes spatially grounded and temporally consistent visual understanding, whereas Emu3 uses next-token prediction across images, text, and videos.

E.2.1. Benchmarks … E.3. Action Generation

The paper evaluates instruction-conditioned image-to-image state prediction for real-world interactions with PRICE-V0.1, using judge-based scoring that emphasizes executed actions, scene consistency, and physically plausible state changes. It compares image-prediction systems against established generative baselines while framing the task around embodied action outcomes.

  • E.2.1. Benchmarks: PRICE-V0.1 evaluates whether a model can generate the target state after an instructed real-world interaction from an initial image.It is an instruction-conditional image-to-image generation task designed for interactive scenarios.
  • E.2.1. Benchmarks: The benchmark combines robot and first-person interaction data from AgiBot-World, HomeInteract, PE-Video, and PSI-Ego, with instruction, initial-image, and target-image samples.HomeInteract is closed-source data collected by a dual-arm wheeled robot in home scenes.
  • E.2.2. Metrics: Generated results receive integer scores from 1 to 5 from Gemma 4-31B, judging instruction following, scene consistency, and the resulting action execution.The evaluation also uses Gemini 3.1 Pro, GPT 5.4, and Doubao Seed 2.0 Pro as closed-source systems.
  • E.3. Action Generation: For embodied actions, scoring penalizes images that appear teleported rather than executed, including unchanged agent pose, position, or contact state.In-progress or imperfect execution is allowed, and minor occlusion or detail loss is acceptable.
  • E.2.2. Metrics: The scoring philosophy uses the full 1–5 range and does not automatically fail outputs for minor blur, texture shifts, small artifacts, or partial ambiguity.This makes the evaluation lenient toward limited visual imperfections when the intended outcome remains assessable.
  • E.2.2. Metrics: The metric requires preserving the input scene while accurately depicting commanded state changes such as moving, opening, picking up, or lifting objects.The evaluator considers the original environment, viewpoint, and major-object layout when assessing consistency.
  • E.2.3. Baselines: Image prediction is evaluated against OmniGen2 (Wu et al., 2026) and FLUX.1-Kontext (Black Forest Labs et al., 2025), both presented as multimodal image-generation or editing baselines.OmniGen2 uses separate text and image readout pathways, while FLUX.1-Kontext is a flow-matching model for in-context generation and editing.

E.3.1. Real-Robot Benchmark … F. More Visualization

The real-robot evaluation tests Orca across five manipulation tasks, controlled OOD settings, complementary metrics, and matched baselines. Detailed results and visualizations emphasize task progress and recovery from failures.

  • E.3.1. Real-Robot Benchmark: Orca is evaluated on five dual-arm wheeled humanoid manipulation tasks, with 200 real-robot trajectories collected per task for action-expert post-training.The tasks are Take Book, Stacked Bowls, Pull Out Tissue, Stamp, and Scoop Sugar.
  • E.3.2. Metrics: Real-robot performance combines task-specific rule-based completion scores with PRM-asa-Judge trajectory-level diagnostics of progress and execution quality.Rule-based scores count the highest completed stage before termination.
  • E.3.3. Baselines: Orca is compared with V-JEPA 2.1, Qwen3.5, and π0.5 using frozen backbones and, for the first two comparisons, identically configured action experts trained on 200 trajectories per task.The action experts for V-JEPA 2.1, Qwen3.5, and Orca are initialized from scratch and trained for 20k steps with global batch size 128.
  • E.3.1. Real-Robot Benchmark: The benchmark tests environment OOD through unseen tablecloth and background settings, and object OOD through unseen but semantically related task objects or containers.Task objects and instructions remain unchanged for environment OOD.
  • E.3.4. Detailed Real-Robot Results: Detailed appendix results report task-level completion under manually designed rule-based criteria for real-robot OOD settings.Each task is evaluated within 60 seconds, and evaluation stops after severe collision or an unrecoverable object fall.
  • E.3.5. Additional Qualitative Visualizations: Additional trajectory visualizations show Orca maintaining higher intermediate progress in failed trials and recovering from intermediate grasp failures.Examples span Stamp, Pull Out Tissue, Stacked Bowls, and Scoop Sugar.
  • F. More Visualization: In qualitative failures, Orca advances farther through manipulation stages than Qwen3.5 or π0.5 before stopping.The examples include stamp transport, tissue grasping, and bowl stacking.
  • F. More Visualization: In Scoop Sugar, Orca retries failed spoon grasps and can eventually recover, whereas Qwen3.5 shakes without effective re-grasping and JEPA remains largely stagnant.The visualizations show both partial recovery and repeated recovery attempts.

F.1. Cross-Benchmark Capability Analysis for Text Generation

Orca’s cross-benchmark text-generation capabilities span state transition, commonsense reasoning, spatial relations, and dynamic motion. Examples show stronger causal-temporal, spatial, and motion reasoning than Qwen across varied video questions.

  • Capability dimensions: Orca’s cross-benchmark capabilities cover state transition, commonsense reasoning, spatial relations, and dynamic motion.These dimensions are defined as generalized capabilities that transcend individual benchmark boundaries.
  • State transition: Orca correctly predicts near-term outcomes in state-transition questions where Qwen selects different answers.Orca answers “A” for the copying process after pressing start and “D” for the door opening, while Qwen answers “D” and “A,” respectively.
  • Commonsense reasoning: Orca’s advantage is particularly pronounced on complex commonsense VQA requiring reasoning beyond the visible scene and inferring hypothetical outcomes.The supplied examples identify commonsense reasoning as a cross-benchmark capability, though no specific commonsense answer pair is provided.
  • Dynamic motion: Orca demonstrates stronger dynamic-motion reasoning by correctly identifying the yellow cube’s movement direction when Qwen chooses a different direction.The example attributes stronger forward simulation to temporal continuity and motion inertia acquired through unconscious learning.
  • Spatial relations: Orca handles spatial relations across distance, proximity, and orientation questions, outperforming Qwen on two of three examples.Orca answers the tank-and-flags distance and truck-facing-direction questions differently from Qwen, while both agree on the traffic-light proximity question.
Loading 2606.30534v2…