Source-linked AI summary
Cortex 2.0: Grounding World Models in Real-World Industrial Deployment
Adriana Aida, Walid Amer, Katarina Bankovic, Dhruv Behl, Fabian Busch, Annie Bhalla, Minh Duong, Florian Gienger, Rohan Godse, Denis Grachev, Ralf Gulde, Elisa Hagensieker, Junpeng Hu, Shivam Joshi, Tobias Knobloch, Likith Kumar, Damien LaRocque, Keerthana Lokesh, Omar Moured, Khiem Nguyen, Christian Preyss, Ranjith Sriganesan, Vikram Singh, Carsten Sponner, Anh Tong, Dominik Tuscher, Marc Tuscher, Pavan Upputuri
TL;DR
Industrial manipulation requires reliable long-horizon execution, but reactive VLA policies do not explicitly evaluate future outcomes. Cortex 2.0 adds visual-latent world-model planning and PRO scoring to select among candidate futures before acting, and it outperforms evaluated baselines across four industrial tasks. The results support world-model-based planning as a reliable approach for complex industrial environments, although current evaluations cover only a subset of embodiments and task families.
Problem
Reactive VLA models select actions from current observations without explicitly reasoning about future outcomes, limiting reliable long-horizon industrial manipulation.
Method
Cortex 2.0 generates candidate future trajectories in visual latent space, scores them with PRO for progress, risk, and completion likelihood, and conditions the policy on the selected trajectory.
Results
Across all four benchmarks, Cortex 2.0 achieves higher success rates and shorter execution times than evaluated baselines and is the only method completing tasks without human intervention.
Takeaways & Limitations
World-model planning filters bad branches before execution and supports reliable operation in cluttered, occluded, and contact-rich industrial environments.
Takeaways & Limitations
Current evaluations cover a subset of embodiments and task families, with planning horizon Hwm and budget k fixed per task.
Abstract
from arXiv · showhide
Industrial robotic manipulation demands reliable long-horizon execution across embodiments, tasks, and changing object distributions. While Vision-Language-Action models have demonstrated strong generalization, they remain fundamentally reactive. By optimizing the next action given the current observation without evaluating potential futures, they are brittle to the compounding failure modes of long-horizon tasks. Cortex 2.0 shifts from reactive control to plan-and-act by generating candidate future trajectories in visual latent space, scoring them for expected success and efficiency, then committing only to the highest-scoring candidate. We evaluate Cortex 2.0 on a single-arm and dual-arm manipulation platform across four tasks of increasing complexity: pick and place, item and trash sorting, screw sorting, and shoebox unpacking. Cortex 2.0 consistently outperforms state-of-the-art Vision-Language-Action baselines, achieving the best results across all tasks. The system remains reliable in unstructured environments characterized by heavy clutter, frequent occlusions, and contact-rich manipulation, where reactive policies fail. These results demonstrate that world-model-based planning can operate reliably in complex industrial environments.
1 Introduction
Cortex 2.0 addresses the brittleness of reactive VLA policies in long-horizon industrial manipulation by planning over candidate futures before acting. Its world-model and PRO-based design is evaluated across embodiments and increasingly complex tasks.
- Motivation: Reactive VLA policies select actions from current observations without explicitly evaluating future outcomes.Long-horizon failures can compound, and a single unrecovered error can disrupt an industrial workflow.
- Approach: Cortex 2.0 generates candidate future trajectories in visual latent space and commits to the trajectory with the highest PRO score.PRO evaluates task progress, risk likelihood, and completion likelihood before execution.
- Contributions: The system integrates a visual-latent world model into Cortex to enable k-step lookahead planning.Planning operates in visual space rather than being tied to a particular robot embodiment.
- Contributions: PRO scores candidate rollouts using task progress, completion likelihood, and risk likelihood to derive an advantage signal for action heads.This scoring module connects imagined futures to policy conditioning.
- Contributions: The planning loop transfers across single-arm, dual-arm, and humanoid embodiments because it operates in visual space.The contribution is framed as cross-embodiment planning rather than embodiment-specific control.
- Evaluation: Cortex 2.0 achieves the highest success rates across four real-world tasks with zero human interventions.The benchmark covers tasks of increasing complexity, including pick and place, sorting, screw sorting, and shoebox unpacking.
2 Related Works
Related work establishes the VLA, diffusion, flow-matching, world-model, multimodal-sensing, and data-scaling foundations relevant to Cortex 2.0. Cortex 2.0 combines these directions with inference-time rollout scoring and deployment data.
- VLA policies: Generalist robot policies map camera observations and language prompts to actions, with later systems improving cross-embodiment pretraining and inference efficiency.The related systems include RT-1, RT-2, Octo, OpenVLA, and OpenVLA-OFT.
- Hierarchical and generative policies: Hierarchical VLA systems predict intermediate subtask representations before fine-grained control, while flow matching and efficient tokenization target trajectory quality and speed.The cited examples include π0, π0.5, FAST, and GR00T N1.
- World models: World models support latent imagination, but training-time rollout generation can risk compounding model errors.Cortex 2.0 follows work using world models at inference time to score candidate rollouts before execution.
- Cortex 2.0 positioning: Cortex 2.0 keeps imagined futures in the same representational space as real observations so PRO scores learned from executed trajectories transfer to imagined futures.The world model is pretrained on internet-scale video and fine-tuned on deployment recordings at 30 Hz.
- Multimodal sensing: Force and tactile signals complement RGB by exposing grasp stability, compliance, and suction reliability during contact-rich manipulation.Cortex 2.0 uses force feedback optionally and adds it to multimodal observations when supported by the robot.
- Cortex 2.0 positioning: Cortex 2.0 augments a flow-matching VLA with a visual-latent world model that generates k candidate futures and scores them before execution.Training also incorporates proprietary deployment recordings collected continuously across deployments.
3.1 Overview
Cortex 2.0 uses a hierarchical pipeline in which visual and task inputs produce structured context, imagined futures, PRO scores, and policy actions. Training combines flow-matching and world-model objectives while using multimodal deployment data.
- Architecture: Cortex 2.0 combines scene encoding, candidate-future generation, PRO ranking, and flow-based action heads in a four-level hierarchical design.The action heads commit to the highest-scoring trajectory.
- Observation encoding: The system encodes RGB, robot state, force feedback, and task-instruction embeddings into a visual latent representation.The force-feedback input is included when supported by the robot.
- Planning and control: A high-level VLM creates task context, the world model generates k trajectories over Hwm, and PRO ranks them before the policy produces an action chunk over Hact.The selected trajectory and binarized advantage indicator condition the policy.
- PRO scoring: The selected rollout is compared with the average candidate score to define an advantage signal for policy conditioning.The score Sj denotes the PRO score of rollout τj.
- Policy conditioning: The advantage is binarized with a task-dependent threshold into an indicator that conditions the VLA policy.At inference, the indicator biases action generation toward realizing the selected future.
- Training: Training jointly combines the flow-matching action loss and world-model loss, while Cortex 2.0 uses open-source, deployment, and synthetic multimodal data.PRO is pretrained separately on industrial deployment data and kept frozen during this stage.
3.2 Cortex 2.0 Architecture
Cortex 2.0 generates candidate futures in visual latent space and uses PRO to score and select trajectories before execution. Its world model produces task-conditioned rollouts, while PRO evaluates progress, risk, and completion likelihood.
- High-Level VLM: The high-level VLM converts the current visual latent into a task-conditioned embedding containing subgoals and scene-grounding variables.The embedding includes objects, spatial relations, and contact priors that steer planning and execution.
- PRO Scoring: PRO processes each predicted latent sequence with a temporal model and three heads for progress, risk, and termination.The risk head penalizes trajectories involving high-speed contact, compression, edge impacts, or surface scraping.
- PRO Scoring: PRO combines its signals into a composite rollout score, and the system selects the highest-scoring candidate for execution.Figure 3 depicts scoring across candidate trajectories and selection of the best rollout.
- Execution Conditioning: The selected rollout produces a binarized advantage indicator that conditions the VLA policy toward realizing the chosen future.The world model is pretrained on internet-scale video and fine-tuned on deployment recordings at 30 Hz.
- PRO Transfer: PRO is trained on real executed trajectories and applied to imagined world-model latents in the same visual latent space.This allows deployment-derived scoring signals to evaluate future trajectories before any action is executed.
- World Model: The world model generates k candidate future latent sequences from the current latent and task context using independent noise realizations.At inference, each candidate is produced by ODE integration and passed to PRO for scoring.
3.3 VLA Policy
The VLA policy receives the current state, task context, selected rollout, and PRO advantage signal to generate actions. An embodiment-specific action mapper converts future-state outputs into robot commands.
- Policy Conditioning: The policy conditions action generation on the task context, current visual latent, selected rollout token, and PRO advantage indicator.These inputs are projected into VLM feature space and processed through the transformer.
- Action Generation: The flow-matching action head generates continuous action chunks by integrating an ODE from noise to the action representation.The action state evolves according to the learned velocity field from ν = 0 to 1.
- Action Mapping: Actions are represented as future states, improving consistency across embodiments compared with raw control commands.A lightweight Action Mapping Module adapts these outputs to robot-specific commands and physical constraints.
3.4 Training
Training separates PRO pretraining from joint world-model and policy optimization. PRO learns from deployment telemetry, is then frozen, and supplies fixed advantage supervision during subsequent training.
- Stage One: PRO is pretrained independently on real industrial trajectories with telemetry-derived progress, risk, and termination supervision.Its supervision terms are optimized only during this first training stage.
- Stage Two: After stabilization, PRO is frozen and provides advantage supervision while the world model and action heads are trained jointly.The frozen PRO receives no gradient updates during the second stage.
- Optimization Scheme: Knowledge insulation blocks gradients to the pretrained VLM backbone in stage one, whereas stage two jointly optimizes all components except PRO.This preserves the pretrained PRO while allowing the remaining system to adapt end-to-end.
- Inference Configuration: The planning budget k and horizons Hwm and Hact are inference-time hyperparameters that do not affect training.
3.5 Cross-Embodiment Design
Cortex 2.0 uses the same visual-latent planning loop across four tasks and single- or dual-arm platforms. Embodiment-specific adaptation is isolated to the action heads.
- Shared Planning Loop: Across four manipulation tasks and both single- and dual-arm configurations, the system generates, scores, and executes the best visual-latent rollout.The planning loop operates without task- or embodiment-specific modification.
- Embodiment Adaptation: The action heads handle embodiment-specific adaptation while the planning process remains shared across robot platforms.
4 Dataset Composition
Cortex 2.0 is trained on a heterogeneous corpus combining proprietary deployment data, targeted teleoperation, open-source cross-embodiment datasets, and synthetic simulation data. The world model is pretrained broadly before deployment-specific fine-tuning, with operational data feeding a continuous improvement cycle.
- Corpus composition: Cortex 2.0 combines proprietary deployment data, targeted teleoperation demonstrations, open-source cross-embodiment datasets, and synthetic simulation data.Table 1 summarizes the resulting corpus composition.
- Real production data: The warehouse fleet has accumulated over 500 million manipulation interactions, while training uses a curated subset of 10 million interactions preserving task and failure-mode diversity.Recorded signals include wrist-camera RGB images, proprioception, force–torque, vacuum pressure, and contact telemetry.
- Open and synthetic data: Public datasets including Open X-Embodiment, BridgeData V2, and DROID broaden embodiment and task diversity alongside RoboCasa synthetic simulation data.Together, these sources bridge real and simulated data for control learning and high-level reasoning.
- In-house collection: In-house collection spans warehouse tasks under single- and dual-arm configurations and uses egocentric recordings plus full-body human motion capture.These complementary channels support high-scale workflow data and coordinated bimanual tasks.
- Training strategy: The world model is pretrained on all visual data before fine-tuning on deployment recordings, while live-operation data continuously feeds back into training.The stated cycle links improved world models, more accurate planning, better execution, and cleaner data.
5 Experiments
The experiments evaluate Cortex 2.0 against open-source visuomotor policies on single- and dual-arm platforms across increasingly complex manipulation tasks. Cortex 2.0 achieves the strongest results across the benchmark suite, with the planning rollout count trading higher success for greater inference time.
- Experimental setup: Cortex 2.0 is evaluated against π0.5, Diffusion Policy, and RDT-2 on single-arm and dual-arm manipulation platforms under equivalent computational budgets.The platforms use Universal Robot arms, parallel grippers, wrist-mounted cameras, and 30 Hz action execution.
- Evaluation protocol: Unrecoverable states include safety-critical collisions, persistent control deadlocks, and scene states from which the policy cannot recover.Interventions resume execution from the last recoverable state rather than resetting the task.
- Planning trade-off: Success rate increases with rollout count k, while inference time per step increases linearly, exposing a foresight-quality versus computational-cost trade-off.Performance rises from 0.962 with 1 rollout to 0.996 with 30 rollouts, while time per step rises from 310 ms to 9200 ms.
- Single-arm pick-and-place: Cortex 2.0 achieves the highest mean score in single-arm pick-and-place after only 160 fine-tuning episodes.The baselines show distinct failures including approach variance, placement misalignment, premature drops, and grasp failures.
- Item and trash sorting: Cortex 2.0 achieves the highest per-operation success rate and shortest average task duration in item sorting, completing all rollouts without human intervention.All baseline policies require intervention in every rollout, and several reach the 15-minute execution limit without completing the task.
- Screw sorting: Cortex 2.0 substantially outperforms all baselines in screw sorting, with near-perfect per-operation success, shortest average time, and no unrecoverable states.The reported performance gap is largest for this task; RDT-2 records zero successful placements.
- Shoebox unpacking: Cortex 2.0 achieves the highest holistic success rate in shoebox unpacking, completes the task faster, and requires no human intervention across rollouts.The result indicates temporal consistency across the full four-step sequence, while π0.5 more often fails during later stages.
- Summary and discussion: Across all four benchmarks, Cortex 2.0 has higher success rates and shorter execution times than every evaluated baseline and is the only method without human intervention.The authors further report success rates above 90% with limited fine-tuning and 99% after continued in-domain operation.
6 Conclusion
Cortex 2.0 augments VLA control with world-model-based physical foresight, shifting manipulation from reactive execution toward planning before action. It consistently outperformed baselines across warehouse evaluations while supporting cross-embodiment transfer and deployment-driven improvement, although current evaluations cover only a subset of embodiments and task families.
- Conclusion: Cortex 2.0 shifts VLA manipulation from try-and-see control to plan-and-try by scoring candidate futures before execution.The system generates visual-latent futures, evaluates them with PRO, and filters bad branches before they become unrecoverable states.
- Conclusion: Cortex 2.0 consistently outperformed all baselines across warehouse pick-and-place, item sorting, and shoebox handling evaluations.The reported evaluations required zero human interventions.
- Conclusion: Visual-latent training enables the same world model and PRO scoring function to transfer across single-arm, dual-arm, and other robot platforms.The claimed transfer relies on visual representations encoding physical regularities independent of specific kinematics.
- Conclusion: Deployment expansion supplies additional operational data that feeds back into the world model and PRO, improving planning quality over time.The paper frames the research-to-production gap as being closed through expanded deployment, iteration, and scale.
- Conclusion: Current evaluations cover only a subset of embodiments and task families, with planning horizon Hwm and budget k fixed per task.Future work targets online adaptation and uncertainty-aware dynamic budget allocation in PRO.