XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation

1 INTRODUCTION

• WAMs model scene evolution through future RGB and action prediction, but their spatial understanding is limited because they generally represent the world as 2D RGB video.Appearance prediction does not explicitly capture metric depth or surface orientation needed for physical interaction.Source: §1 · pp. 1–2
• Prior spatial WAMs commonly use auxiliary heads, gated residual modules, or modality-specific branches, and often omit surface orientation or task-dependent functional regions.The paper argues that these signals should be expressed in the same generative space as observations and actions.Source: §1 · p. 2
• XGenAct encodes RGB observations, metric depth, surface normals, functional role segmentation, and 7 DoF robot actions as RGB videos through deterministic codecs.Each stream uses the same frozen video VAE, diffusion transformer, output head, and flow-matching objective, without modality-specific learned heads, auxiliary losses, or manually balanced loss coefficients.Source: §1 · p. 2
• The model adds depth, surface normals, and functional role segmentation to image-space world action modeling, supplying complementary geometric and semantic information.Source: §1 · p. 2
• Structured geometric and semantic generation improves average closed-loop success by up to 28.75 percentage points over RGB-only training at matched compute and action budgets.Future RGB generation is retained.Source: §1 · p. 2

Structured world prediction for action learning

• Several WAMs add geometric or semantic supervision through modality-specific components, objectives, or action modules.Examples include PAD's separate modality encoders and decoders, auxiliary branches or residual modules, and RynnWorld-4D's downstream inverse-dynamics head.Source: §2 · p. 3
• SA-WAM shares a frozen video tokenizer for RGB and depth, but uses dedicated action latent positions and a fixed RGB-depth-action layout.XGenAct instead represents perception and action as video streams and samples multiple perception-action templates in one generative model.Source: §2 · p. 3

Perception as generation

• Perception-generation methods provide a common visual interface for multiple tasks, but do not test whether mixed structured perception tasks improve a robot action generator using that same output space.SUV retains a separate action expert.Source: §2 · p. 3

Visual representations of action

• Visual action methods either use motion cues while predicting action tokens, visualize candidate actions, or represent 7-DoF poses as multiview RGB canvases.XGenAct extends image-space action generation with metric depth, surface normals, and functional role segmentation, without modality-specific heads or auxiliary losses.Source: §2 · p. 3

3.1 OVERVIEW

• XGenAct extends the Action Images formulation by encoding RGB, metric depth, surface normals, functional role segmentation, and robot actions as videos processed by the same frozen VAE.The diffusion transformer is fine-tuned while the video VAE remains frozen.Source: §3.1 · p. 3

3.2 GEOMETRY ENRICHED VIDEO SPACE

• Deterministic codecs convert modality streams into three-channel videos accepted by the frozen VAE, with no learnable codec parameters and information preserved through VAE encoding and decoding.RGB uses identity mapping; depth is mapped along an RGB path, normals are mapped to RGB and renormalized on decoding, and functional roles receive fixed RGB codes.Source: §3.2 · p. 4

3.3 TASK TEMPLATES AND CONDITIONING PLANS

• Each sample pairs one perception modality with action streams from two camera views; sampled conditioning plans specify which frames are observed and which are generated.The joint plan predicts future perception and actions, the policy plan predicts future actions from current perception, and the inverse-dynamics plan predicts actions from fully observed perception.Source: §3.3 · p. 5

3.4 SHARED TRAINING OBJECTIVE

• The shared flow-matching objective trains the diffusion transformer to predict only positions designated for generation, while observed positions remain context.The same objective applies across modalities, templates, and plans, without modality-specific output heads, auxiliary objectives, or manually chosen loss weights.Source: §3.4 · p. 5

4.1 SETUP

• The studies compare policies, modality menus, and direct perception generation; all groups were evaluated on RLBench tasks absent from training.The menu and perception-study groups used a 70/30 RLBench–ManiSkill3 training mixture; the external-policy comparison used RLBench training.Source: §4 · p. 6, §4.1 · p. 6

4.2 COMPARISON WITH ROBOT POLICIES

• XGENACT reached 52% average success, versus 26% for the strongest baselines, and led on three of five tasks.Figure 3 pairs methods by scene seed and shows the first trial XGENACT solves; Table 1 reports success over 20 trials.Source: §4.2 · p. 7, Figure 3 · p. 7
• Baseline failure modes differed: PAD-Depth and UWM mostly timed out, while VLA-JEPA and MolmoAct often stopped after unreachable-pose IK failures; XGENACT failures were timeouts.XGENACT did not exhaust its stricter budget of 20 IK failures in total.Source: §4.2 · p. 7

4.3 STRUCTURED PERCEPTION IMPROVES THE ROBOT CONTROL

• Every perception menu raised average success from 32.5% for RGB to 52.5–61.25%; adding depth to RGB+N+S reduced success from 61.25% to 58.75%.The best menu, RGB+N+S, gained 28.75 points; segmentation and normals were most useful on tasks with corresponding object ambiguities.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 FUTURE PERCEPTION UNDER SIMULATOR REPLAY

• Direct generation lowered depth AbsRel from 0.41–0.43 to 0.28–0.33 and raised segmentation mIoU from 0.22–0.24 to 0.42–0.49; normals were near-tied except with RGB+D+N.Adding perception streams changed future-RGB LPIPS only from 0.39 for RGB to 0.40–0.43.Source: Figure 4 · p. 8, §4.4 · pp. 8–9
• Generated metric depth produced 3-D clouds whose table, fixtures, and object layout coincided with the demonstration; visible differences lay on the arm, whose predicted pose differed.Generated RGB frames also matched simulator frames reached by executing their decoded actions.Source: Figure 5 · p. 9, §4.4 · p. 9

5 CONCLUSION

• Structured perception improves held-out RLBench closed-loop success over RGB-only training, while direct generation predicts future depth and segmentation more accurately than evaluated RGB-to-expert pipelines.The best training menu remains task-dependent, motivating broader evaluations and adaptive task balancing.Source: §5 · p. 9
• Deterministic codecs let actions, future RGB, metric depth, surface normals, and functional-role segmentation share a frozen video VAE, diffusion transformer, output head, and objective without modality-specific learned heads or auxiliary losses.Source: §5 · p. 9
• XGENACT expresses robot actions and four perception streams through one video-generation interface.Source: §5 · p. 9

• Generative AI tools supported literature retrieval and discovery, research ideation, and experimental work, including baseline debugging and discussion of conditioning, ablations, and experimental design.Source: AI USE STATEMENT · p. 10
• The tools also helped draft parts of the paper and appendix, polish sections, and format the manuscript.Source: AI USE STATEMENT · p. 10
• The authors state that they reviewed and verified AI-assisted activities and generated content and take responsibility for the final methodology, experiments, results, and manuscript.Source: AI USE STATEMENT · p. 10

• The paper and appendices specify the model, objective, action representation, data construction and splits, modality codecs, training mixture, optimization settings, camera conventions, and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
• The planned release includes code for data preparation, training, inference, rollout, and evaluation, plus pinned environments, commands, configurations, task and split definitions, seeds, and checkpoints for every reported variant.Source: REPRODUCIBILITY STATEMENT · p. 10
• For simulator assets and third-party checkpoints governed by their own licenses, the authors plan to provide versioned download and reconstruction instructions rather than redistribute them.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 LATENT VIDEO-DIFFUSION WORLD ACTION MODEL

• The causal 3D VAE encoder and decoder remain frozen, while only the diffusion transformer is fine-tuned.The model uses two camera views conditioned with Plücker ray embeddings.Source: §A.1 · p. 14
• For 41-frame, 512² clips, the latent grid is 11 × 32 × 32 with 48 channels.Source: §A.1 · p. 14

A.2 ACTIONS AS VIDEO

• Each 7-DoF action frame is projected into image points and rendered as RGB-channel Gaussian blobs, so the frozen VAE and transformer process actions as ordinary video.The representation uses a fixed 0.1 m tool length; gripper encoding and trajectory decoding follow Action Images.Source: §A.2 · p. 14

B TRAINING DATA AND EVALUATION SPLITS

• The external-policy comparison uses RLBench training only, while the menu ablation and perception study use a 70/30 RLBench–ManiSkill3 mixture.RLBench provides 16 tasks, four camera views, and 3,960 training episodes; ManiSkill3 provides seven tasks and 1,750 episodes.Source: Table 3 · p. 14, B TRAINING DATA AND EVALUATION SPLITS · pp. 14–15
• Evaluation episodes are excluded from both training trees, and closed-loop trials use fresh scenes generated from disjoint seeds.The perception study covers four unseen RLBench tasks with five rollouts each; the internal ablation and external comparison use separate task sets and 20 trials per task.Source: B TRAINING DATA AND EVALUATION SPLITS · p. 15

C MODALITY CODECS AND ADMISSION-GATE FIDELITY

• Depth is transformed and mapped onto an RGB-cube path, then decoded by projecting generated colors back to that path and inverting the transform.Colors outside a fixed tolerance from the path are marked invalid; valid metric depth spans 0.05–10 meters.Source: C MODALITY CODECS AND ADMISSION-GATE… · p. 15
• Normals are computed from metric depth and decoded as unit vectors, while segmentation uses a shared RGB lookup for eight functional roles.The ninth unknown color is an annotation diagnostic rejected during validation, not a training class; the online-rendered action stream has no reported equivalent VAE admission test.Source: C MODALITY CODECS AND ADMISSION-GATE… · pp. 15–16

D TRAINING-MIXTURE DETAILS

• XGENACT samples video+action with probability 0.4 and each depth+action, segmentation+action, and normal+action template with probability 0.2.The RGB-only control uses video+action with probability 1.0; every template carries action and has the same shape and cost.Source: D TRAINING-MIXTURE DETAILS · p. 16
• The conditioning-plan distribution, optimization schedule, and action dropout are matched across runs, so their training distributions differ only in the template menu.An action stream is dropped with probability 0.1 before the plan is drawn.Source: D TRAINING-MIXTURE DETAILS · p. 16

E FLOW-MATCHING AND OPTIMIZATION DETAILS

• The matched mixed-source runs use 6,000 optimization steps, an effective batch of four, and learning rate 5 × 10^-7; the VAE remains frozen.All diffusion-transformer parameters are fine-tuned, and the reported perception comparison uses step 4,000 checkpoints.Source: E FLOW-MATCHING AND OPTIMIZATION DETAILS · p. 17

F CAMERA CONDITIONING AND COORDINATE CONVENTIONS

• Camera rays are embedded as Plücker coordinates relative to the first frame of view 1, then encoded by the frozen VAE for transformer conditioning.Absolute camera extrinsics and intrinsics are used only to project and decode action images; one projection routine serves both data sources.Source: F CAMERA CONDITIONING AND COORDINATE… · p. 17

G EVALUATION PROTOCOL

• Perception scoring excludes the anchor and first four ramp poses, and uses simulator-depth changes greater than 1 cm to define the dynamic region.A frame is scored only when the region exceeds 100 pixels, preventing static background from dominating the metric.Source: G EVALUATION PROTOCOL · p. 18, Figure 6 · p. 18

H EXTERNAL POLICY BASELINES

• The external-policy settings table summarizes camera inputs, action chunk sizes, and the numbers of actions executed before replanning.XGENACT uses 50 denoising steps, classifier-free guidance 7.5, temporal stride 3, and seed 42.Source: G EVALUATION PROTOCOL · p. 17, Table 6 · p. 19
• External policies are evaluated on five tasks with 20 trials each, using task success and specified episode limits for success, timeouts, or inverse-kinematics failures.Most baselines are fine-tuned on the 16 RLBench training tasks; MolmoAct is evaluated zero-shot, and Evo-1’s result reflects a documented data-conversion defect.Source: H EXTERNAL POLICY BASELINES · pp. 19–20

J ADDITIONAL QUALITATIVE RESULTS

• The qualitative-results section presents unedited simulator renders and raw generated canvases, with trials selected by caption-specific fixed rules rather than baseline outcomes.For imagined-versus-executed comparisons, generated chunks are aligned with simulator frames after the decoded actions are executed.Source: J ADDITIONAL QUALITATIVE RESULTS · p. 20, §J.3 · p. 24

I PERCEPTION RESULTS UNDER SIMULATOR REPLAY

• Table 7 compares direct modality generation with RGB followed by a frozen expert on four unseen RLBench tasks, using five trials per task at step 4,000.Scores use each route’s own simulator replay and dynamic-region pixels; gated averages exclude a task for every route when any shown cell has a commanded–achieved gap of at least 10 mm.Source: I PERCEPTION RESULTS UNDER SIMULATOR… · p. 20, Table 7 · p. 21

J.1 CLOSED-LOOP ROLLOUTS AGAINST EXTERNAL POLICIES

• The qualitative rollout figures show matched-scene successes and failures, with each external-policy panel reporting its method’s success rate over 20 trials.The close-microwave example shows XGENACT closing the door while PAD-Depth, UWM, and MolmoAct time out and VLA-JEPA stops after repeated inverse-kinematics failures.Source: §J.1 · p. 21, Figure 7 · p. 22, Figure 8 · p. 23

J.2 PAIRED ROLLOUTS OF RGB AND XGENACT

• Paired RGB-only and XGENACT rollouts start from the same task instance and scene seed; imagined and executed meat-on-grill sequences show similar grasp, transport, and placement poses.The 40-step chunk has whole-frame PSNR of 21–24 dB.Source: §J.2 · p. 21, Figure 10 · p. 24

J.4 OPEN-LOOP GENERATION ACROSS TASKS

• The all-modality checkpoint receives only frame-0 observations and current end-effector poses from two views, then generates 40 future frames for each output template.The examples use plan IIII with 50 flow steps, guidance 7.5, and seed 42.Source: §J.4 · p. 24
• Figure 12 compares each output space’s generated frame 40 with the demonstration, while noting that separate prompts can yield slightly different arm poses across columns.The figure displays view 1 for the final generated frame and the demonstration.Source: §J.4 · p. 25, Figure 12 · p. 26
• Figure 13 overlays decoded action-image end-effector paths with demonstration paths on training and held-out tasks; its corner value is mean 2-D pixel distance, not 3-D pose error.Path points are position-blob centroids, with start, end, and first gripper-closed frames marked distinctly.Source: §J.4 · p. 25, Figure 13 · p. 27