PaperlayerView exact revision on arXiv

Source-linked AI summary

Orca: The World is in Your Mind

Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, Runze Xiao, Huaihai Lyu, Senwei Xie, Euan Liu, Klara Tian, Tianfeng Long, Yichi Zhang, Zhengliang Cai, Ruike Chen, Jifan Zhao, Ruochuan Shi, Zihan Tang, Jing Lyu, Wenxing Tan, Ningbo Zhang, Yangtao Hu, Yuming Gao, Xiansheng Chen, Junkai Zhao, Congsheng Xu, Boan Zhu, Ziqi Wang, Yupu Feng, Qiongqiong Zhang, Yingli Zhao, Yulong Ao, Shaoxuan Xie, You Liu, Guocai Yao, Leiduo Zhang, Xiaodan Liu, Yunyan Zhang, Yance Jiao, Xinyan Yang, Jiaxing Wei, Xu Liu, Tengfei Pan, Shaokai Nie, Chunlei Men, Sen Cui, Xiaojie Jin, Hongyang Li, Jianlan Luo, Yao Mu, Yunchao Wei, Jun Yan, Hang Zhao, Xiaolong Zheng, Jiaming Li, Yonghua Lin, Tiejun Huang, Zhongyuan Wang, Pengwei Wang

arXiv:2606.30534v3cs.CV

TL;DR

Orca addresses the gap between isolated task prediction and modeling world states. It learns a world latent from video and language-conditioned events, and its jointly used pre-training objectives produce balanced text, image, and action readouts.

  • Problem

    Orca frames the gap as modeling isolated downstream tasks rather than learning world states and exposing them through dedicated readouts.

    Source: §5 · p. 13

  • Method

    Orca learns a world latent from multimodal signals using dense video transitions and language-conditioned sparse events, supported by 125K hours of video and 160M event annotations.

    Source: §1 · p. 2

  • Results

    Jointly using the three pre-training objectives yields the most balanced performance across text, image, and action readouts.

    Source: §4.3 · p. 13

  • Takeaways & Limitations

    Orca is an early exploratory milestone toward general-purpose world foundation models.

    Source: §5 · p. 13

    Resource constraints limited experiments mainly to 0.8B and 4B models, and training used only one-tenth of the inventory data.

    Source: §5 · p. 14

Abstract

from arXiv · show

We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

1. Introduction

Orca is presented as an initial world learner that builds a multimodal world latent through state-transition modeling and exposes it to text, image, and action readouts. Its two learning paradigms target dense natural transitions and sparse, instruction-constrained meaningful transitions.

1. Introduction

  • The introduction frames general intelligence as requiring continuous multimodal world learning and unified state-transition modeling across observed and unknown domains.The proposed model should build world states rather than be defined only by next-token, next-frame, or next-action prediction.Source: §1 · p. 2
  • Orca learns a world latent space from visual and language signals, using language for causal explanations and task intentions and providing a general interface for multimodal readouts.The readout interfaces produce text, images, and actions.Source: §1 · p. 2
  • Unconscious learning captures dense natural transitions from continuous video, while conscious learning uses textual constraints to model sparse, meaningful event-level transitions.The video paradigm uses self-supervision to predict next-frame latents; the instruction-constrained paradigm learns meaningful transitions at the event level.Source: §1 · p. 2
  • Orca’s text, real-world interactive image, and embodied-action readouts improve with pre-training scale and outperform similarly scaled specialized baselines.The evaluation probes whether the learning paradigms are feasible and scalable and whether a stronger world latent enables stronger readouts; the backbone is frozen and only lightweight readout modules are trained.Source: §1 · p. 2
  • Orca’s pre-training inventory contains 125K hours of video and 160M event annotations across interaction, manipulation, robot execution, and event-level transitions.Source: §1 · p. 2
  • The paper also provides an analysis of Orca’s current limitations to inform future work.Source: §1 · p. 2

2. Orca

Orca models the world by mapping multimodal signals into a latent state and learning how that state changes over time. It combines observation-only learning of dense natural transitions with language-conditioned learning of sparse meaningful transitions.

2.1. Modeling

  • Orca abstracts multimodal world signals into latent states that evolve forward or backward through implicit dynamics and explicit conditions.Implicit factors include unobserved drivers of change, while explicit conditions include observed instructions.Source: §2.1 · p. 3
  • Unconscious learning captures dense natural transitions from observations alone, while conscious learning uses semantic conditions to model sparse, meaningful transitions.Conditions may describe past or future events, task intentions, or causal premises.Source: §2.1 · p. 3

Orca

  • The encoder uses a pre-trained VLM aligned with language and vision spaces to learn the unified world latent from visual and language signals.Conscious learning also trains event-conditioned state prediction and generates language answers to questions about the video.Source: §2.2 · p. 4
  • A modality-specific decoder reads the learned latent space to extract multimodal information; decoder details are deferred to Section 3.2.Source: §2.2 · p. 4

3. Training

Orca is pretrained with state-transition and VQA objectives on video, event, and language data, then evaluated through frozen-backbone readouts for language, vision, and action. Training infrastructure optimizations increase throughput to 2.91 Samples/Sec/GPU.

3. Training

  • Orca is trained in two stages: pre-training learns its world latent from large-scale visual and language data, followed by frozen-backbone downstream readout training.Source: §3 · p. 4

3.1.1. Pre-Training Recipe

  • Pre-training combines observation-only and event-conditioned state transitions with VQA response generation, using learnable queries for transitions and the backbone LM head for VQA.The transition objectives use vision-encoder latent supervision rather than pixel-level reconstruction; VQA uses standard next-token loss.Source: §3.1.1 · pp. 4–5

3.1.2. Pre-Training Data

  • The pre-training inventory combines real-world video, language-annotated events, and VQA data, including 125K hours of video, 160M event annotations, and 11.5M VQA examples.Only one-tenth of the available video data is used in this version; event annotations are derived from video through coarse and fine-grained segmentation, while VQA combines language and video.Source: §3.1.2 · pp. 5–6

3.2. Downstream Post-Training

  • For downstream post-training, Orca’s backbone remains frozen and only the modality-specific readout modules are trained to expose its latent as text, images, or actions.The selected trainable modules depend on the intended task, such as image modules for vision or action modules for embodied tasks.Source: §3 · p. 4, §3.2 · p. 6

3.2.1. To Language: Text Generation

  • For text generation, Orca produces a natural-language response from a visual observation and instruction through its LM head, without an additional decoder.Source: §3.2.1 · p. 6

3.2.2. To Vision: Image Prediction

  • For image prediction, a trainable MLP adaptor conditions a pretrained, frozen SD3.5 model, with only the adaptor and LoRA parameters updated.Training pairs current and target video frames; frozen Orca produces the target-frame latent from the image and instruction, which is then passed to the image readout.Source: §3.2.2 · p. 6

3.2.3. To Action: Action Generation

  • For action generation, a from-scratch DiT-based Action Expert uses flow-matching loss and latent conditioning to produce robot action chunks through multi-step denoising.The action readout is trained on five tasks, with only 200 trajectories per task, and only its MLP adaptor and Action Expert are trainable.Source: §3.2.3 · p. 7

3.3. Infrastructures

  • Throughput rises from 0.66 to 2.91 Samples/Sec/GPU, approximately a 4.4× acceleration compared to StarVLA (StarVLA Community, 2026).The training stack uses FSDP2, chunked cross-entropy with activation recomputation, and communication pre-fetching that overlaps all-gather with computation while removing unnecessary sharding for visual blocks.Source: §3.3 · p. 7

4. Evaluation

Orca’s loss falls as pre-training data and model size increase, while stronger pre-trained latents improve downstream text, image, and action readouts.

4.1.1. Loss of Proposed Learning Paradigm

  • Orca’s total loss decreases with more pre-training data, and the larger model achieves lower objective loss; loss continues to fall rather than converging quickly.The paper presents these trends as evidence that the learning paradigm is effective and scalable.Source: §4.1.1 · pp. 7–8, Figure 5 · p. 7

4.1.2. Relationship between latent and downstream readout performance

  • Downstream text-generation readout averages TemporalBench, MVBench, SWITCH, and 3DRSBench, while image prediction averages PRICE-V0.1; both are zero-shot, whereas action averages five real-robot out-of-domain tasks.Task details are in Sections 4.2.1–4.2.3.Source: §4.1.2 · p. 8
  • With Orca’s backbone frozen, scaling pre-training data improves text, image, and action readouts; action gains emerge despite using no action labels in pre-training.The authors suggest this video-based action capability may partly alleviate low generalization caused by scarce robot data.Source: §4.1.2 · p. 8, Figure 6 · p. 8

4.2. Downstream Readout Analysis

  • The downstream evaluations use no benchmark-specific training data, and Orca is not tuned on the evaluation benchmarks.Source: §4.2 · p. 8

1. Text Generation.

Across text, image, and action readouts, Orca’s learned world latent supports strong OOD reasoning and prediction, physically grounded visual state changes, and robust real-world action generation. Ablations indicate that observation-only, event-conditioned, and VQA objectives contribute complementary capabilities.

4.2.1. Comparison on Text Generation

  • Text evaluation spans MVBench, TemporalBench, 3DSRBench, and SWITCH, probing state transitions, commonsense reasoning, spatial relations, and dynamic motion.The capability analysis reports more accurate state-transition prediction, stronger commonsense and counterfactual reasoning, stronger spatial understanding, and more consistent dynamic motion.Source: §4.2.1 · p. 9
  • Orca achieves the best overall text-generation result among same-size VLMs and large-size world models.The comparison is based on Table 1.Source: §4.2.1 · p. 9, Table 1 · p. 9

4.2.2. Comparison on Image Prediction

  • Orca achieves the best average performance on PRICE-V0.1 and remains competitive across real-world interaction sources.This result indicates that its learned world latent contains predictive information about future visual states under real-world interactions; Table 3 and Figure 7 present the comparison.Source: Table 3 · p. 10, §4.2.2 · p. 11, Figure 7 · p. 11
  • Compared with image-generation baselines, Orca better preserves robot morphology, scene and object consistency, contact relationships, and instruction following in interaction-conditioned predictions.The baselines exhibit irrelevant-object appearance or teleportation, hallucinatory hands, poor instruction adherence, and prior-driven biases.Source: §4.2.2 · p. 11, Figure 7 · p. 11

4.2.3. Comparison on Action Generation

  • With the same from-scratch Action Expert, Orca outperforms Qwen3.5 in all OOD settings and is comparable to pretrained π0.5.The real-robot evaluation uses five dual-arm tasks under environment and object OOD settings; the authors report a breakthrough from a 0% success rate and a significant effect of Orca’s learning paradigm.Source: §4.2.3 · pp. 11–12, Table 4 · p. 12
  • Orca makes more intermediate progress, stagnates less, and recovers better after execution errors, with higher failure near-success and drawdown recovery scores.Figure 8 illustrates recovery after repeated spoon-grasp failures, while π0.5 remains unstable.Source: §4.2.3 · pp. 12–13, Figure 8 · p. 12

4.3. Ablation

  • Jointly using observation-only transition, event-conditioned transition, and VQA objectives yields the most balanced text, image, and action readouts.Observation-only transition especially benefits action generation through dense natural dynamics; event-conditioned transition supports semantically guided image prediction, while VQA preserves language readout and semantic grounding.Source: §4.3 · p. 13, Table 5 · p. 13

5. Conclusion

Orca learns a world latent space from multimodal signals and exposes it through dedicated readouts, shifting the modeling target from next-token, frame, or action prediction to next-state prediction. The conclusion presents Orca as an early milestone and outlines limitations and future research directions.

Conclusion

  • Orca first learns internal representations of world states from multimodal signals, then exposes them through dedicated readout interfaces for downstream tasks.The authors describe this as an early exploratory milestone toward general-purpose world foundation models.Source: §5 · p. 13

Discussion & Limitation

  • Orca mainly learns from vision and language, while future world learners should align richer sensory and physical signals to the same underlying state.The paper identifies audio, tactile, force, light, and proprioception as useful additions; sound can reveal boiling before visual changes appear.Source: §5 · pp. 13–14
  • Supervising visual prediction within a frozen pre-trained vision encoder simplifies training but aligns the learned state space with semantics rather than a unified world space.The authors propose learning a native world latent directly from multi-source signals, rather than relying on a pre-trained modality space.Source: §5 · p. 14
  • Experiments at 4B and 0.8B scale use only one-tenth of the 125K-hour, 160M-annotation inventory, indicating that world learning requires sufficient model capacity as well as data.The 4B model exhibits a trade-off among language, image, and action readouts during pre-training, more pronounced at 0.8B.Source: §5 · p. 14
  • PRICE-V0.1 spans multiple real-world sources but remains limited in scale, diversity, and interaction richness; the paper proposes a broader state-transition evaluation framework.The proposed framework would assess state prediction, intervention response, physical quantifiability, and counterfactual inference.Source: §5 · p. 14
  • Most event annotations supervise short, minute-level transitions, supporting local learning but not long-term evolution over hours, days, or longer.Source: §5 · p. 14
  • Current readouts cover language, vision, and action; the authors also identify limits in the loss design and in the length and difficulty of embodied tasks.They call for simpler losses suited to next-state prediction, and note that current embodiment tasks are relatively short and easy despite stringent settings and lower performance.Source: §5 · p. 14

Future Works

  • Future work proposes a self-evolving model-data-evaluation loop and extending unified state-transition representations from embodied intelligence to scientific domains.The loop would generate interaction trajectories and counterfactual samples, evaluate and value-filter them, then feed them back into training; target domains include quantum mechanics, the universe, and life sciences.Source: §5 · p. 14

6. Author List · Appendix

The author list names core contributors and contributors across project roles, followed by expert consultants and research leads.

6.1. Core Contributors

  • Core contributors are listed as Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, and Runze Xiao.The source marks Yihao Wang, Mingyu Cao, Yanqing Shen, and Runze Xiao with an asterisk, and Yuheng Ji with an asterisk and dagger.Source: §6.1 · p. 15
  • Core contributors are credited for model pre-training, data infrastructure, and evaluation.Model pre-training: Huaihai Lyu, Senwei Xie, Mingyu Cao. Data infrastructure: Euan Liu, Klara Tian, Tianfeng Long, Yichi Zhang, Zhengliang Cai, Ruike Chen, Jifan Zhao, Yanqing Shen. Evaluation: Ruochuan Shi, Zihan Tang, Jing Lyu, Runze Xiao.Source: §6.1 · p. 15
  • Core contributors are also credited for real-robot work, downstream post-training, and infrastructure.Real robot: Jing Lyu, Wenxing Tan, Ningbo Zhang, Yangtao Hu, Euan Liu, Yuming Gao, Xiansheng Chen, Junkai Zhao, Runze Xiao. Downstream post-training: Senwei Xie, Huaihai Lyu, Congsheng Xu, Boan Zhu, Ziqi Wang. Infrastructure: Yupu Feng, Qiongqiong Zhang.Source: §6.1 · p. 15

6.2. Contributors

  • Contributors are listed for infrastructure and real-robot data.Infrastructure: Yingli Zhao and Yulong Ao. Real-robot data: Shaoxuan Xie, You Liu, and Guocai Yao.Source: §6.2 · p. 15
  • Other contributor roles include product and operations, brand management, platform management, and system management.The listed contributors are Leiduo Zhang, Xiaodan Liu, Yunyan Zhang, and Yance Jiao (product and operations); Xinyan Yang and Jiaxing Wei (brand management); Xu Liu and Tengfei Pan (platform management); Shaokai Nie and Chunlei Men (system management).Source: §6.2 · p. 15

6.3. Expert Consultant (Ordered by English last name alphabetically)

  • The expert consultants listed are Sen Cui, Xiaojie Jin, Hongyang Li, Jianlan Luo, Yao Mu, Yunchao Wei, Jun Yan, Hang Zhao, and Xiaolong Zheng.The subsection states that names are ordered alphabetically by English last name.Source: §6.3 · p. 15

6.4. Research Leads

  • The research leads listed are Jiaming Li, Yonghua Lin, Tiejun Huang, Zhongyuan WangB, and Pengwei WangB.Source: §6.4 · p. 15

A. Orca Conception … B.2. Next Token Prediction

Orca centers world-state transitions in a unified latent learned from multimodal signals, rather than predictions tied to individual modalities. The related-work sections situate this approach among latent world models and next-token paradigms.

A. Orca Conception

  • Orca models the world’s latent state, shifting from next-token, next-frame, and next-action prediction toward state transitions that support understanding, prediction, and intervention.Language, vision, and action are treated as observations or readouts of the same underlying state.Source: A. Orca Conception · p. 22
  • Unconscious learning captures dense dynamics from continuous visual experience, while conscious learning uses language, events, instructions, and questions to represent meaningful transitions.The learned latent can be read out through language for explanation and reasoning, vision for prediction and imagination, and action for intervention.Source: A. Orca Conception · p. 22

B. Related Work

  • Related work is organized by each paradigm’s primary learning objective, with unified models categorized by their dominant training formulation..Source: B. Related Work · p. 23

B.1. Self-Supervised Learning

  • JEPA-style work predicts semantic representations rather than pixels, with later studies extending latent prediction across images, text, video, interaction, and planning.Examples include I-JEPA, VL-JEPA, MC-JEPA, the V-JEPA series, LeJEPA, Causal-JEPA, LeWorldModel, and GeoWorld.Source: §B.1 · p. 23
  • Unlike JEPA-style visual representation learning, Orca learns latent world states from multimodal signals and models transitions under implicit dynamics and explicit semantic conditions, in both temporal directions.It predicts future states and backtracks to past states.Source: §B.1 · p. 23

B.2. Next Token Prediction

  • Multimodal models span visual understanding, reasoning, agents, and robotic manipulation, while unified autoregressive systems also generate or edit images and predict future frames.The passage describes these capabilities across visual-language models, native multimodal models, and embodied models.Source: §B.2 · pp. 23–24
  • Orca uses language as a semantic condition that guides state transitions through events, intentions, and causal premises; VQA generation reinforces commonsense and semantic grounding..Source: §B.2 · p. 24

B.3. Next Frame Prediction … C.1.1. Pre-Training Objective Definitions

The sections contrast frame-prediction and action-centered approaches with Orca’s world-learning-first philosophy, then specify its pre-training losses and sampling mix.

B.3. Next Frame Prediction

  • Image generation models map language or multimodal conditions to visual frames, while video generation extends frame prediction to temporally coherent sequences.The reviewed models advance fidelity, instruction tracking, editing, temporal coherence, controllability, and multimodal conditioning.Source: §B.3 · p. 24
  • Orca distinguishes its world modeling from image and video synthesis by targeting physically constrained state transitions involving actions, scenes, objects, contacts, and instructions.Image and video generation models can produce coherent, visually appealing frames, but Orca emphasizes interaction-conditioned transitions.Source: §B.3 · pp. 24–25

B.4. Next Action Prediction

  • VLA models center on embodied generalization, video-action models use visual dynamics to aid policies, and world-action models jointly model actions and future states.Examples include OpenVLA, VPP, and Motus.Source: §B.4 · p. 25
  • Orca pre-trains by modeling scene and object state transitions without action labels, aiming to support embodied adaptation with limited robot data and in OOD settings.Its representation emphasizes temporal changes, object motion, and local physical interactions.Source: §B.4 · p. 25

C. Training Settings

C.1.1. Pre-Training Objective Definitions

  • Orca pre-training combines observation-only and event-conditioned latent state-transition losses with VQA next-token loss, using frozen-encoder visual latents for transition supervision.The predicted visual-token latents are matched to ground-truth latents; VQA answers use standard next-token prediction.Source: §C.1.1 · p. 26
  • The objective weights observation-only, event-conditioned, and VQA losses by 0.1, 0.5, and 0.4, respectively, while sampling state-transition and VQA examples at approximately 5:1.Event-conditioned prediction targets the previous or next event state selected by language, averaging latent-matching losses across both directions.Source: §C.1.1 · p. 26

C.1.2. Query-Based Implementation … C.2.3. Action Readout

Orca’s appendix details how query-based training models observation-only and event-conditioned state transitions, then describes pre-training settings and language, vision, and action readouts. The readouts expose Orca’s learned latent through language, image generation, and robot action chunks.

C.1.2. Query-Based Implementation

  • For observation-only transitions, q1’s last-layer state passes through a two-layer visual transition head to predict the next-frame latent, supervised by a frozen vision encoder.The input is the current observation.Source: §C.1.2 · p. 26
  • For event-conditioned transitions, instruction e_(t+Δ) specifies direction and target event, while q2 reads out the predictive state for a random frame in that event.The objective includes previous-event and next-event directions.Source: §C.1.2 · p. 26

C.1.3. Pre-Training Hyperparameters

  • Table C1 groups Orca’s pre-training hyperparameters into model-scale, optimization, and objective-specific settings.The passage does not provide the individual values.Source: §C.1.3 · p. 27, Table C1 · p. 27

C.2. Downstream Readout Post-Training Settings

  • Downstream readout post-training implements language, SD3.5-based vision, and DiT-based action readouts, with the overall architecture illustrated in Figure 4.These implementation details extend the readout training described in Section 3.2.Source: §C.2 · p. 27

C.2.1. Language Readout

  • The language readout reuses the VLM backbone’s LM head to answer autoregressively from an observation and instruction, supporting VQA, event interpretation, and causal explanation.Source: §C.2.1 · p. 28

C.2.2. Vision Readout

  • The vision readout conditions a pretrained SD3.5 decoder on Orca’s predicted visual latent; its VAE and MMDiT weights stay frozen while the adaptor and decoder-attention LoRA are trained.The target image is resized to 768 × 768, and the predicted image is produced by multi-step denoising.Source: §C.2.2 · p. 28, Table C2 · p. 28

C.2.3. Action Readout

  • The action readout combines Orca’s predictive q1 latent, noisy actions with time embeddings, and robot proprioception in a DiT Action Expert that predicts short-horizon action chunks.The expert is trained with flow matching: Gaussian-noised ground-truth chunks are used to train velocity prediction.Source: §C.2.3 · pp. 28–29, Table C3 · p. 29

D. Infrastructure … E.2. Image Prediction

The section reports infrastructure changes that support scalable Orca training and describes the benchmarks and baseline categories used for text-generation evaluation.

D. Infrastructure

  • Orca migrates from DeepSpeed to FSDP2, using flexible sharding and resharding while omitting sharding for lightweight visual blocks to reduce overhead.Source: D. Infrastructure · pp. 29–30
  • Activation recomputation checkpoints selected boundaries and reconstructs states during backpropagation, trading extra computation for memory savings and larger batches.Source: D. Infrastructure · p. 30
  • Chunked cross-entropy computes loss by token blocks, avoiding the full logits tensor and reducing peak memory under long-sequence, large-vocabulary settings.Source: D. Infrastructure · p. 30
  • Forward/backward pre-fetching overlaps upcoming-layer parameter all-gathers with current-layer computation to reduce communication stalls and GPU idle time.Source: D. Infrastructure · p. 30
  • 2.91 samples/sec/GPU on H100s is 3.0× the FSDP2 baseline of 0.97 samples/sec/GPU and 4.4× the StarVLA training pipeline.Source: Table D1 · p. 30, D. Infrastructure · p. 30

E.1. Text Generation

  • Text-generation evaluation uses MVBench, TemporalBench, SWITCH, and 3DSRBench to assess video understanding, temporal dynamics, real-world interface interaction, and 3D spatial reasoning.TemporalBench uses short-video QA and Multiple Binary Accuracy (MBA).Source: §E.1 · pp. 30–31
  • The text-generation baseline set spans latent world models, native multimodal models, general-purpose VLMs, and compact multimodal models.Source: §E.1 · p. 31

E.2.1. Benchmarks … F. More Visualization

The appendix specifies PRICE image-prediction and real-robot evaluation protocols, baselines, detailed OOD results, and qualitative examples of Orca’s task progress and recovery.

E.2.1. Benchmarks

  • PRICE-V0.1 evaluates instruction-conditional image-to-image prediction of action outcomes in real scenes, emphasizing physical constraints and common sense over conventional image editing.It derives from four robot or first-person datasets and covers manipulation, scene-state changes, action outcomes, and human/object or robot/object interactions.Source: §E.2.1 · p. 31

E.2.2. Metrics

  • Gemma 4-31B scores generated target images from 1 to 5 for instruction execution, scene consistency, and physical plausibility; the average score is reported as a percentage.Gemini 3.1 Pro, GPT 5.4, and Doubao Seed 2.0 Pro are also used; minor blur, texture shifts, small artifacts, and local ambiguity are not automatic failures.Source: §E.2.2 · pp. 32–33

E.2.3. Baselines

  • The image-prediction comparison includes OmniGen2, FLUX.1-Kontext, and FLUX.2 [klein] as generative baselines for image-generation and editing capability.The passage describes their respective unified generation/editing, in-context editing, and text-conditioned generation or multi-reference editing capabilities.Source: §E.2.3 · p. 33

E.3.1. Real-Robot Benchmark

  • The real-robot benchmark evaluates five manipulation tasks, with 200 trajectories collected per task for downstream Action Expert post-training, and tests environment and object OOD generalization.Environment OOD varies unseen tablecloth/background settings while keeping objects and instructions fixed; object OOD substitutes unseen but semantically related objects or containers.Source: §E.3.1 · pp. 33–34, Figure E2 · p. 33, Table E1 · p. 34

E.3.2. Metrics

  • Real-robot performance is assessed with task-specific rule-based stage scores and PRM-asa-Judge trajectory-level diagnostics of progress and execution quality.Evaluation is limited to 60 seconds per task and stops after severe collision lock or an unrecoverable object fall; only the highest achieved score counts.Source: §E.3.2 · p. 34, Table E2 · p. 35

E.3.3. Baselines

  • Real-robot baselines are V-JEPA 2.1 and Qwen3.5 hidden states conditioned on Orca’s same action expert, plus pretrained 𝜋0.5; matched experts train from scratch on 200 trajectories per task.The V-JEPA 2.1 and Qwen3.5 backbones and the 𝜋0.5 VLM are frozen; Orca’s backbone freezing is not stated. 𝜋0.5 follows its official post-training configuration: 30k steps with global batch size 32.Source: §E.3.3 · p. 34

E.3.4. Detailed Real-Robot Results

  • Table E3 reports task-level rule-based completion scores for real-robot OOD settings; the appendix details this protocol while PRM-as-a-Judge results appear in the main-text analysis.Task scores use manually designed criteria; Table E2 defines the scoring stages and termination conditions.Source: §E.3.4 · p. 34, Table E3 · p. 35, Table E2 · p. 35

E.3.5. Additional Qualitative Visualizations

  • Additional trajectories illustrate Orca maintaining higher intermediate progress in failed tasks and recovering after grasp failures.Examples span Stamp, Pull Out Tissue, and Stacked Bowls failures, plus Scoop Sugar retries that recover or eventually achieve a spoon grasp.Source: §E.3.5 · p. 36, Figure E3 · p. 36, Figure E4 · p. 36

F.1. Cross-Benchmark Capability Analysis for Text Generation

The section organizes text-generation capability across state transition, commonsense reasoning, spatial relations, and dynamic motion, illustrating these dimensions with cross-benchmark examples.

F.1. Cross-Benchmark Capability Analysis for Text Generation

  • For state transition, Orca predicts toast reappearing (Qwen: being eaten) and copying processing first (Qwen: confirming settings), illustrating causal temporal change and irreversible-process evaluation.The figure caption describes state transition as understanding causal temporal dynamics and physical state changes.Source: Figure F1 · p. 38, §F.1 · p. 39, Figure F2 · p. 39
  • The commonsense-reasoning comparison highlights Orca’s advantage in complex VQA requiring inference beyond the visible scene and hypothetical-outcome reasoning.Source: Figure F3 · p. 39
  • On cube-motion direction, Orca answers down-left while Qwen answers up-left; the caption links unconscious learning to temporal continuity, motion inertia, and stronger forward simulation.Source: Figure F4 · p. 40
  • Spatial-relation examples cover tank–flag distance, proximity to a traffic light, and truck-facing direction; the figure caption notes robustness with complex occlusions and multi-object reasoning.Source: Figure F5 · p. 40

Related papers

Loading 2606.30534v3…