PaperlayerView exact revision on arXiv

Source-linked AI summary

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

AgiBot Research Team, Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xiong, Linqing Zhong, Yifei Wei, Feng Han, Long Zhang, Da Huang, Nanshu Zhao, Chenghao Yin, Mo Wu, Zhaodong Yan, Kongtao Hu, Yuxiang Yan, Aogelijiang Niyazi, Yu Fang, Jia Zeng, Lizhu Meng, Daizhen Lv, Haoyu Cao, Zhiwen Hou, Lianjin Ye, Yuehan Niu, Zhikai Cai, Xuan Hu, Hui Min, Xiongfeng Cai, Yue Liao, Jing Wu, Soujanya Poria, Ye Li, Sanping Zhou, Maoqing Yao

arXiv:2609.05588v1cs.ROcs.CV

TL;DR

Whole-system pretraining and scaling remain underexplored for world-action models, which typically build on pretrained video generators. GE-Act 2.0 pretrains its generative and action components from scratch, and scaling manipulation data improves zero-shot performance across nearly all evaluated skill groups.

  • Problem

    Existing world-action research typically builds on pretrained video generators rather than studying pretraining and scaling the whole system.

    Source: §1 · p. 1

  • Method

    GE-Act 2.0 combines a control-oriented autoencoder, single-step visual planner, inverse dynamics model, and KASO alignment, with trainable components initialized from scratch.

    Source: §3.1 · p. 4, §6 · p. 24

  • Results

    100 real-robot tasks across two embodiments show broad zero-shot OOD improvements across nearly all evaluated skill groups as manipulation data increases.

    Source: §6 · p. 24

  • Takeaways & Limitations

    The findings establish GE-Act 2.0 as a scalable approach to learning transferable manipulation capabilities through world-action pretraining.

    Source: §6 · p. 24

    GE-Act 2.0 focuses on instruction-conditioned control without explicit deliberation; robust open-environment execution also requires reasoning, planning, memory, and self-correction.

    Source: §7 · p. 24

Abstract

from arXiv · show

World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.

1 Introduction

GE-Act 2.0 tackles WAM pretraining and scaling by initializing its trainable components from scratch and learning across complementary manipulation data. It combines control-oriented representation, separate visual-planning and inverse-dynamics pretraining, and action-compatible co-training, then evaluates pretrained capability directly under OOD conditions.

1 Introduction

  • WAMs predict how scenes unfold to support action generation, but prior work has largely relied on pretrained video generators rather than studying WAM pretraining and scaling as a whole.The introduction frames this as leaving unresolved how to pretrain visual generation and inverse dynamics before connecting them.Source: §1 · p. 1, Every interaction is training data · p. 2

Every interaction is training data

  • GE-Act 2.0 uses a control-oriented autoencoder to compress visual input while retaining information needed for future prediction and action recovery.CoAE is trained with reconstruction and multi-teacher feature alignment.Source: Every interaction is training data · p. 2
  • Its single-step visual planner enables visual generation on action-free video and inverse dynamics on instruction-free trajectories to be pretrained separately before end-to-end co-training.The planner produces a complete future in one differentiable flow-generator pass.Source: Every interaction is training data · p. 2
  • KASO addresses the validity gap by co-training only on predicted futures that the active inverse dynamics model judges compatible with the recorded action.The mismatch occurs when a generated future depicts a valid behavior different from the demonstration’s recorded behavior.Source: Every interaction is training data · pp. 2–3
  • Unlike evaluations that use downstream fine-tuning, GE-Act 2.0 is tested directly on OOD tasks with a fixed protocol across scaling checkpoints.This isolates capability acquired during pretraining and how it changes with training data.Source: Every interaction is training data · p. 3
  • Scaling manipulation data produces broad zero-shot OOD gains across 100 real-robot tasks and two embodiments, without task-specific fine-tuning.The scaling comparison spans 300 to 30,000 hours of co-training data.Source: Figure 1 · p. 2, Every interaction is training data · p. 3

2 Related Work

The related work situates GE-Act 2.0 across world-action architectures, robot-control representations, fast generation, action supervision, and evaluation. Its distinguishing choices include a completed-future interface to a separately pretrained action model, action-oriented latent evaluation, and compatibility-based supervision.

2.1 Generalist Policies and World-Action Models

  • GE-Act 2.0 uses a completed future visual state to connect its generator to a separately pretrained inverse dynamics model, unlike parallel-branch or feature-coupled designs.This interface permits generator and action-model pretraining on distinct data before end-to-end co-training.Source: §2.1 · p. 3

2.2 Representations for Robot Video and Control

  • CoAE combines a compact generative latent space, alignment to pretrained visual teachers, and evaluation by fixed-capacity action recovery from encoded video windows.This addresses reconstruction-oriented video latents while connecting representation quality to motion and action information.Source: §2.2 · pp. 3–4

2.3 Fast Video Generation for World Models

  • MeanFlow and improved MeanFlow motivate GE-Act 2.0’s single-step planner, extended to conditional multi-view futures as a differentiable train–deployment interface for an action model.Unlike approaches that distill a pretrained multi-step teacher, MeanFlow methods learn an average-velocity field traversing the noise interval in one forward pass (Geng et al., 2025, 2026).Source: §2.3 · p. 4

2.4 Connecting Generated Futures to Actions

  • KASO addresses outcome-mode mismatch by selecting generated futures compatible with the recorded action according to the active IDM, rather than merely using generated video or passing gradients through sampling.It ranks candidate futures against recorded futures in action space and supervises only compatible candidates; a similar best-of-K rule appeared independently in generative modeling (Gladstone et al., 2026).Source: §2.4 · p. 4

2.5 Evaluating Pretrained Robot Models

  • GE-Act 2.0 uses no-fine-tuning evaluation as its primary measure of pretrained capability, reporting excluded visual factors and absent task adaptation rather than treating fine-tuned results as isolated pretraining evidence.Simulation and world-model benchmarks serve as horizontal reference points, not direct evidence of capability acquired through manipulation-video pretraining.Source: §2.5 · p. 4

3 GE-Act 2.0

This section presents GE-Act 2.0’s components and training, including CoAE control and instruction probes and the video–action validity gap addressed by KASO. It describes how KASO selects generated futures compatible with recorded actions and reports a controlled real-robot ablation in which it improves pickup success over the compared training strategies.

3.1 System Overview / 3.2 A Control-Oriented Autoencoder for World–Action Models / 3.2.1 Beyond Reconstruction with Multi-Teacher Alignment

  • The system uses CoAE latents for the SVP’s future prediction and the IDM’s action prediction; CoAE compresses 256×384 frames to 24 tokens per frame.CoAE combines pixel, perceptual, and adversarial reconstruction losses with alignment to frozen teachers providing language-aligned semantics, spatiotemporal structure, and dense visual features. This addresses dense-token and generic-reconstruction limitations of inherited video-autoencoder spaces.Source: §3.1 · pp. 4–5, §3.2.1 · pp. 5–6, §3.2 · p. 5

3.2.2 Control and Instruction Grounding under Extreme Compression

  • CoAE achieves 97.95% instruction-grounding accuracy, ahead of all tested representations, with 24% lower error than DC-AE.The probe balances 5,000 episodes across ten benchmark tasks and instruction categories; CoAE’s 2.05% error is roughly half V-JEPA 2.1’s 4.15% error.Source: §3.2.2 · p. 7, Figure 4 · p. 7, Table 1 · p. 7
  • CoAE’s action-recovery MAE is 0.01673, only 13–31% above DINOv3 and V-JEPA 2.1 while using one-sixteenth as many tokens per frame.The probe freezes each encoder and uses the same inverse-dynamics probe and evaluation protocol; lower MAE is better.Source: §3.2.2 · pp. 6–7, Figure 4 · p. 7, Table 1 · p. 7

3.3 A Single-Step Visual Planner as a World–Action Interface / 3.3.1 One-Step Generation Unlocks Underutilized Data

  • The SVP grounds instructions in observations and generates complete future latents in one differentiable pass, enabling separate IDM pretraining and end-to-end co-training on broader interaction data.A conventional multi-step generator requires differentiating through all denoising steps to pass action gradients, whereas the single-pass future lets action loss update both SVP and IDM. The IDM can exploit instruction-free play, failed attempts, and deployment rollouts with aligned observations and actions.Source: §3.3 · pp. 7–8, §3.3.1 · p. 8

3.3.2 Multi-Scale Visual Prediction / 3.3.3 Toward Unified Understanding and Generation

  • Dense near-term and sparse far-horizon predictions connect fine interaction dynamics with longer-term task intent; a frozen VLM grounds instructions in the current scene.The SVP’s VLM contextualizes text with the head-view observation, and the IDM receives language through generated future states rather than as a direct input.Source: §3.3.2 · p. 8, §3.3.3 · pp. 8–9

3.3.4 Conditional MeanFlow for One-Step Visual Planning

  • Conditional MeanFlow maps noise to a complete future latent sequence in one forward pass while leaving conditioning frames unchanged and keeping predicted futures differentiable.The method uses a shared DiT trunk, mean- and instantaneous-velocity heads, and a Jacobian–vector product; training supervises prediction coordinates while masking conditioning-token derivatives.Source: §3.3.4 · pp. 9–10

3.4.1 Inverse Dynamics Model / 3.4.2 Knowledge Transfer with Conflicting Supervision

  • The IDM predicts dense control-rate actions and sparse longer-horizon actions, while co-training connects separately pretrained SVP and IDM checkpoints and retains their original objectives.The controller executes the dense action chunk; sparse actions serve as auxiliary targets. Retaining SVP video and recorded-video IDM losses yields E2E+PT and helps prevent co-training from overwriting pretrained capabilities.Source: §3.4.1 · p. 10, §3.4.2 · p. 10

3.4 World-Action Alignment and KASO / 3.4.3 The Validity Gap

  • KASO addresses the validity gap: a generated future can depict a different valid outcome mode from the recorded action, creating mismatched supervision even when the future is plausible.The paper describes KASO as preserving action diversity in the pretrained policy, broadening action-space coverage for subsequent methods such as reinforcement learning.Source: §3.4 · p. 10, §3.4.3 · p. 11

3.4.3 The Validity Gap

  • In the toy system, KASO recovers four video modes and two action modes, while decoupled pretraining yields diffuse actions and E2E+PT collapses actions to their mean.E2E co-training collapses both distributions; the controlled setup gives each method identical initialization and magnifies generated off-plane deviations equally.Source: §3.4.3 · pp. 11–12, Figure 6 · p. 12

3.4.4 KASO: Knowledge-Aligned Selective Optimization

  • KASO selects the top-k generated futures whose action-space responses under the active IDM most closely match the recorded future, then replays them with gradients.Candidate generation and scoring are gradient-free; differentiable replay updates both the IDM and SVP while retaining their recorded-data pretraining losses.Source: §3.4.4 · pp. 12–13

3.4.5 Controlled Real-Robot Ablation

  • KASO raises single-object pick success from 12% to 40% (+28 points) and four-object macro-average pick success from 22.5% to 37.5% (+15 points) versus E2E+PT.The four-object follow score is 95% for both methods. These are 300-hour alignment stages atop full-scale pretrained components, without evaluation-specific SFT, in unseen evaluation environments and object instances.Source: §3.4.5 · pp. 14–15, Table 2 · p. 15

4 Training Data and Mixture Composition

GE-Act 2.0 uses separate data mixtures for SVP pretraining, IDM pretraining, and KASO co-training, matching each stage to its available supervision. The mixtures draw on robot, simulation, and human or egocentric data, with markedly different embodiment proportions.

4 Training Data and Mixture Composition

  • SVP learns from instruction–video data, IDM from action-labeled trajectories, and KASO co-training connects them with instruction–video–action data while retaining their separate pretrained objectives.SVP data can include manipulation recordings without robot action labels, including egocentric video; IDM data include demonstrations, failures, and deployment trajectories.Source: §4 · p. 15
  • The SVP, IDM, and co-training mixtures contain 39,000, 32,000, and 30,000 hours, respectively, while G1-OP exceeds 50% and G2-90D remains below 2% of co-training data.Sources span robot embodiments, simulation, open-source robot datasets, egocentric and human manipulation video, and rollout and failure data; Figure 8 summarizes stage-specific proportions.Source: §4 · p. 15, Figure 8 · p. 16

5 Evaluating Zero-Shot OOD Manipulation at Scale

GE-Act 2.0 is evaluated directly on held-out real-robot OOD tasks, with success increasing as co-training data scales across both embodiments. The section also measures fine-grained instruction grounding and conflict handling, while treating adapted simulation benchmarks as a separate evaluation.

5.1 Zero-Shot OOD Evaluation without Per-Task SFT

  • All main real-robot experiments deploy pretrained checkpoints without per-task fine-tuning, demonstrations, or task-specific checkpoint selection on held-out scenes, backgrounds, lighting, and object instances.One checkpoint is deployed across the full suite with fixed settings.Source: §5.1 · p. 16

5.2 Scaling Manipulation Skills

  • The real-robot suite evaluates 100 executable manipulation tasks across 20 skill groups on both G1-OP and G2-90D, focusing on local visuomotor competence rather than high-level planning.Each task is evaluated over 10 trials per embodiment and training-data scale, with success reported at task, skill-group, and suite levels.Source: §5.2 · p. 18

5.2.1 Scaling with Training Data Volume

  • G1-OP suite-level SR rises from 17.1% at 300 hours to 44.1% at 30,000 hours, while G2-90D rises from 13.4% to 31.1%.Both continue rising from 5,000 to 30,000 hours, with no evidence of saturation within the measured range.Source: Figure 10 · p. 18, §5.2.1 · p. 19
  • At 30,000 versus 300 hours, 19 of 20 G1-OP skill groups and 18 of 20 G2-90D groups improve, despite G2-90D contributing less than 2% of co-training data.Nonzero-SR tasks rise from 39 to 76 for G1-OP and from 24 to 72 for G2-90D. Five of six G2-90D skill groups with fewer than two hours of embodiment-specific data move from zero to nonzero SR.Source: Figure 9 · p. 17, §5.2.1 · p. 19, Figure 11 · p. 19

5.2.2 Scaling with Skill-Specific Data Volume

  • Across 19 nontrivial skill groups, skill-specific training coverage is strongly associated with G1-OP zero-shot OOD success (Pearson r = 0.80; Spearman ρ = 0.85).The fitted relationship corresponds to 1.94 logit units higher SR per tenfold increase in training hours; for example, Wipe reaches 76.7% SR with 824.1 hours, whereas Sweep reaches 3.3% with 64.6 hours.Source: §5.2.2 · pp. 20–21, Figure 12 · p. 20

5.3 Instruction Following in Complex Scenes

  • Complex-scene tests use the final full-KASO G1-OP checkpoint without further adaptation to assess instruction-dependent control over familiar pick-and-place behaviors.The tests examine choosing among visually plausible alternatives and responding when an instruction conflicts with an ongoing action or conventional scene relation.Source: §5.3 · p. 21

5.3.1 Fine-Grained Grounding across Instruction Dimensions

  • Follow Score is at least 90.0% for Pick and Place on object identity, color, direct position, and shape, but falls to 13.3%/26.7% for order.Across 295 rollouts, overall Follow Score is 83.1% versus full-task SR of 72.9%; order is both the least represented and weakest dimension, while shape reaches 90.0% Follow Score for both primitives despite its low occurrence. The results establish association, not causation, between instruction frequency and grounding performance.Source: §5.3.1 · p. 22, Figure 14 · p. 22

5.3.2 Instruction Following under Behavioral and Semantic Conflicts

  • In qualitative conflict tests, the policy redirects to a newly commanded target or arm and follows an unusual cup-into-shoebox instruction over a conventional scene association.Target changes near contact require recovery from the previous trajectory, whereas arm designation changes show little hesitation; the demonstrations are illustrative, not quantitative evaluation.Source: §5.3.2 · p. 23, Figure 15 · p. 23

6 Conclusion

GE-Act 2.0 unifies visual generation and action prediction through components designed for control-relevant compression, decoupled visual planning, and action-aligned co-training. Zero-shot OOD evaluations show broad gains from more manipulation data, transfer to a sparsely represented embodiment, and strong instruction grounding.

6 Conclusion

  • GE-Act 2.0 combines a control-oriented autoencoder, a single-step visual planner for decoupled pretraining, and KASO to align generated futures with compatible action supervision.The model was designed to study visual generation and action prediction as a unified system.Source: §6 · p. 24
  • Across 100 real-robot zero-shot OOD tasks on two embodiments, more manipulation data improved nearly all evaluated skill groups, including on the sparsely represented embodiment.Skill-specific data coverage strongly predicted downstream success, while complementary experiments showed fine-grained instruction grounding and adherence to explicit commands under behavioral and semantic conflicts.Source: §6 · p. 24

7 Limitations and Outlook … A.2 GenieSim-Instruction: OOD Instruction Following

Across OOD simulation benchmarks, GE-Act 2.0 performs strongly but not uniformly across conditions and instruction categories. The paper identifies scaling egocentric-video pretraining and coupling the model with deliberative System-2 capabilities as directions for future work.

7 Limitations and Outlook

  • The study has not scaled egocentric-video pretraining to its potential, and proposes testing how its volume, diversity, and composition affect visual-generation pretraining and robot capability.The authors view egocentric video as a promising source of diverse physical interactions and task semantics, with preliminary signals from training mixtures described as encouraging.Source: §7 · p. 24
  • GE-Act 2.0 focuses on System-1 instruction-conditioned low-level control; future work will couple it with System-2 deliberation for reasoning, long-horizon planning, task decomposition, memory, and self-correction.The proposed direction is to study how the two levels plan, act, and improve together in closed loop.Source: §7 · p. 24

A OOD Generalization in Simulation

  • The simulation protocol uses designated in-distribution SFT before evaluating only on held-out OOD conditions, enabling more comparable cross-model evaluation.It complements real-robot experiments with three standardized benchmarks spanning environment randomization, instruction following, and visual and embodiment perturbations.Source: A OOD Generalization in Simulation · p. 30

A.1 RoboTwin Clean-to-Random Generalization

  • RoboTwin Clean-to-Random tests whether Easy/Clean-adapted policies retain effectiveness under OOD changes to background, lighting, clutter, table height, and their combined Hard condition.GE-Act 2.0 is compared with StarVLA, GR00T N1.7, and π0.5.Source: §A.1 · p. 30, Table 3 · p. 30
  • 60.52% success under RoboTwin’s combined Hard randomization is 12.62 points above π0.5, and GE-Act 2.0 leads five of six conditions.Success declines by 16.19 points from Easy to Hard, versus 25.20 points for π0.5; lighting is the exception, where GE-Act 2.0 scores 65.92% versus π0.5’s 69.20%.Source: Table 3 · p. 30, §A.1 · p. 30

A.2 GenieSim-Instruction: OOD Instruction Following

  • GenieSim-Instruction evaluates ten language-following tasks under prescribed OOD scene variations after SFT on released training demonstrations.Evaluation varies background textures, object positions, and related scene factors; task categories include color, number, shape, size, object type, references, logical composition, and object straightening.Source: §A.2 · p. 30, Table 4 · p. 31
  • 0.770 average score on GenieSim-Instruction exceeds ACoT-VLA’s 0.757 and π0.5’s 0.746, with GE-Act 2.0 leading four tasks.Its clearest margins over the previous best are 0.08 on logical composition and 0.06 on common-sense reference; the strongest baseline remains ahead on number, shape, size, and several direct attribute-matching tasks.Source: Table 4 · p. 31, §A.2 · p. 31

A.3 LIBERO-Plus Robustness across Perturbation Axes

After adaptation on standard LIBERO, GE-Act 2.0 achieves 80.4% overall on LIBERO-Plus, with its strongest results under camera and sensor-noise perturbations; robustness remains uneven across the seven axes.

A.3 LIBERO-Plus Robustness across Perturbation Axes

  • The evaluation SFTs GE-Act 2.0 on standard LIBERO training data and tests seven held-out axes: viewpoint, robot initial state, language, lighting, background, sensor noise, and object layout.LIBERO-Plus tests whether a policy adapted on standard LIBERO remains reliable under controlled changes to observations, instructions, and execution conditions.Source: §A.3 · p. 31
  • GE-Act 2.0 scores 80.4% overall, 6.3 percentage points above StarVLA (StarVLA Community, 2026) but 4.0 below π0.5 (Physical Intelligence et al., 2025), and leads camera (94.1%) and sensor-noise (95.5%) robustness.Overall is computed across 10,030 task instances, weighted by axis instance counts under the benchmark protocol.Source: Table 5 · p. 31, §A.3 · p. 31
  • Robot-state and especially background shifts remain principal weaknesses, while GE-Act 2.0 stays close to π0.5 under language and layout changes.The per-axis results expose these benchmark-specific failure modes alongside broader OOD capability in simulation.Source: §A.3 · p. 31
  • Simulation results show the strongest generalization under compounded scene randomization and OOD instruction following, while per-axis results diagnose remaining limitations.These standardized comparisons complement real-robot atomic-task evaluation.Source: §A.3 · p. 31

B Training Recipes and Hyperparameters

The training recipe uses shared optimization and precision defaults across stages, with guidance and caption dropout disabled.

B Training Recipes and Hyperparameters

  • All stages use bf16 mixed precision, DeepSpeed ZeRO stage 2, AdamW (β1, β2) = (0.9, 0.95), weight decay 10^-5, clipping at 1.0, and constant learning rate after linear warmup, with no classifier-free guidance or caption dropout.Batch sizes are specified per GPU.Source: B Training Recipes and Hyperparameters · p. 32

B.1 Architectures

The architecture section reports parameter counts for trained components in the final co-trained checkpoint, excluding the frozen Qwen3.5-2B VLM.

B.1 Architectures

  • The component parameter counts come from the final co-trained checkpoint and exclude the frozen 2.72 B-parameter Qwen3.5-2B VLM.Source: Table 6 · p. 32

B.2 Temporal Schedule and Sample Construction … C Full Results

The appendices detail sample construction, component pretraining, KASO co-training, and deployment, then use a toy system to illustrate how an IDM can fail on generated off-manifold video despite strong recorded-pair performance.

B.2 Temporal Schedule and Sample Construction

  • Each sample aligns four dense conditioning frames, two sparse frames, and 52 dense actions over a 1.73-second horizon, with absolute joint positions standardized per source.Frames are 256 × 384; sparse action tokens receive loss weight 0.1.Source: §B.2 · p. 32

B.3 SVP Pretraining

  • The SVP is trained from random initialization with learning rate 10^-4, batch size 8 then 12 per GPU, progressively introduced data sources, and linear decay to zero.Training uses a 20,000-step warmup.Source: §B.3 · p. 32

B.4 IDM Pretraining

  • The IDM is pretrained on recorded futures from the 32,000-hour mixture using flow matching, unit loss weight, and light Gaussian blur and noise on visual latent tokens.It uses per-GPU batch size 16, learning rate 10^-4, and a 5,000-step warmup.Source: §B.4 · pp. 32–33

B.5 KASO Co-training

  • KASO co-training accumulates recorded-video SVP, recorded-future IDM, and selected-generated-future action losses, choosing the lowest-energy candidate from four and discarding the rest.The loss coefficients are (1, 0.1, 1), and the recorded and generated action losses contribute at a 1:1 ratio; the VLM and CoAE remain frozen.Source: §B.5 · p. 33

B.6 Deployment

  • At deployment, the SVP predicts future latents in one pass, the IDM integrates action flow in five Euler steps without guidance, and the first 30 dense actions are executed before replanning.Reported results use each run’s last checkpoint with fixed settings and no per-task checkpoint selection.Source: §B.6 · p. 33

B.7 Toy Two-Stage System

  • The toy system maps four video modes to two balanced action modes and compares decoupled pretraining, connected objectives, and selected-candidate training.Its recorded data have z3 = 0, while the shared sz = 0.2 setting amplifies the IDM’s response to that coordinate fivefold as an off-manifold stress test.Source: §B.7 · pp. 33–34
  • The Decoupled arm had the lowest video sliced-W1 distance among learned arms (0.068 versus 0.103 for KASO) but failed on generated off-manifold video; Figure 6 illustrates a mechanism, not a seed-averaged benchmark.Its recorded-pair IDM RMSE was 0.0112 for both sz = 0.2 and sz = 1.0. Across seeds 0–4, Decoupled’s maximum action-density height relative to ground truth ranged from 0.240 to 1.035.Source: §B.7 · p. 34

C.1 Atomic-Skill Taxonomy and Evaluation Details

The evaluation covers 100 atomic tasks in 20 skill groups on two robot embodiments, using fixed, observable, instruction-specific success criteria. Trials use held-out scenes and objects with matched resets across checkpoints.

C.1 Atomic-Skill Taxonomy and Evaluation Details

  • 100 atomic task instances span 20 skill groups, with each task evaluated on G1-OP and G2-90D using a fixed, observable, instruction-specific terminal success criterion.Table 7 gives the task taxonomy; examples include grasping and lifting the specified object for Pick and fully closing the target for Close.Source: §C.1 · p. 34, Table 7 · p. 35
  • 10 trials are run for every task, embodiment, and training-data scale, with reset configurations matched across checkpoints and test scenes, backgrounds, lighting, and object instances excluded from training.Success rate (SR) is the percentage of successful trials; evaluation uses the full instruction wording and indicated arm configuration.Source: §C.1 · p. 34

C.2 Task-Level Data-Scaling Results

Table 8 presents task-level zero-shot OOD success rates across data scales, reporting results separately for G1-OP and G2-90D. It covers all 100 tasks, with each rate computed over 10 trials.

C.2 Task-Level Data-Scaling Results

  • The reported data scales are the nested co-training pools from Section 5.2.1, and task descriptions follow the taxonomy in Table 7.Source: §C.2 · p. 35
  • Table 8 reports task-level zero-shot OOD success rates across data scales in separate G1-OP and G2-90D blocks.The continuation carries the same task-level results.Source: Table 8 · pp. 36–37
  • Every task-level success rate is computed over 10 trials, and the tables include all 100 tasks.Source: §C.2 · p. 35, Table 8 · p. 36

C.3 Instruction-Following Prompts and Results

The final unadapted KASO checkpoint shows instruction-following and complete-task success across a fine-grained G1-OP grounding suite, while instruction-level results vary by reference type. The section also characterizes descriptor frequencies in RefCOCOg and the training corpus.

C.3 Instruction-Following Prompts and Results

  • 83.1% Follow Score and 72.9% SR were achieved across 295 rollouts with the final KASO checkpoint on G1-OP without adaptation.The suite tested 59 instructions (33 Pick, 26 Place) over five trials each; Follow Score measures selecting only the specified object or destination, while SR requires the complete behavior.Source: §C.3 · pp. 37–38
  • The grounding suite used five physical scene families with visually feasible alternatives, testing object identity, color, size, shape, position, and order references.Size combines extreme and medium references; position combines left/right and near/far, while order combines middle and ordinal references. The scenes include cluttered objects, colored items or containers, varied sizes and shapes, and cup or container arrangements.Source: §C.3 · pp. 37–38
  • Ordinal references were challenging: both “second cup” Pick instructions had 0% Follow and 0% SR, while “middle cup” scored 40% on each measure.Other outcomes also varied; for example, the rectangular-object Pick instruction had 100% Follow but 20% SR.Source: §C.3 · p. 38
  • Size references accounted for 0.96% of training-corpus descriptors, comprising 0.68% extreme and 0.28% medium references.Descriptor rates are lexical matches over 95,010 RefCOCOg expressions and approximately 3.7 million training-corpus Pick-and-Place segments; non-object descriptor categories can overlap.Source: §C.3 · p. 39, Table 10 · p. 39

Related papers

Loading 2609.05588v1…