PaperlayerView exact revision on arXiv

Source-linked AI summary

An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM

Tianheng Wang, Zhou Xie, Heng Jia, Jianhua Xu, Tong Zhang, Kaicheng Yu

arXiv:2609.25961v1cs.RO

TL;DR

Continuous control raises the question of whether action generation needs a separate pathway from visual prediction. PatchWAM uses a fixed codec to represent actions as visual-latent tokens for joint prediction, and its reported evaluations include gains over a matched dual-expert control. The evidence is bounded by simulation-only experiments and does not establish physical-robot performance or safe deployment.

  • Problem

    The paper asks whether continuous control can use the visual generative pathway directly, rather than requiring a separate action expert or learned action-specific interface.

    Source: §1 · p. 1

  • Method

    PatchWAM uses a fixed Action-as-Patch codec to represent each action step as a visual-latent token, which a shared transformer denoises jointly with future-image tokens.

    Source: §1 · pp. 1–2

  • Results

    With one of every 20 RoboTwin 2.0 training-window starts retained, PatchWAM achieves 88.0% average success versus 78.4% for the matched dual-expert control.

    Source: §1 · p. 2, §7 · p. 16

  • Takeaways & Limitations

    The matched comparison shows that a separate action expert is not required in this sampling regime, though it does not establish that every dedicated action head is unnecessary.

    Source: §6 · p. 15

    All experiments are in simulation, so the results do not establish real-time operation, physical-robot performance, or safe deployment.

    Source: §6 · p. 15

Abstract

from arXiv · show

Generative visual models offer a foundation for learning representations of physical dynamics, yet their extension to continuous control raises a fundamental question: do visual prediction and action generation require separate computational pathways? Existing approaches usually introduce trainable action heads or separate action experts to bridge low-dimensional states and high-dimensional visual representations. In this work, we explore whether the visual backbone's existing capacity can also support control when actions are expressed in a compatible representation. Thus, we introduce PatchWAM (Patch World-Action Model), which treats continuous actions as another type of patch through a fixed mapping called Action-as-Patch. This allows a single model to predict both how the robot should move and what the scene may look like afterward. Visual prediction and action generation become parts of the same generative process, without a dedicated action head or separate action expert. Experiments with subsampled training windows show gains over a matched dual-expert control, while benchmark evaluations reach 91.8% success rate on LIBERO-Plus and 96.12% on RoboTwin 2.0 in a full-data setting with additional augmented demonstrations. More broadly, the result suggests that capability need not be added where it can be inherited: the constraint on extending a generative backbone is the interface a new signal is written in, not the capacity to model it.

1 INTRODUCTION

The introduction asks whether continuous control can reuse a pretrained visual generative pathway rather than require a separate action expert. PatchWAM tests this with a fixed action-to-visual-token codec and shared joint denoising, then evaluates it against a matched dual-expert control and across benchmarks.

1 INTRODUCTION

  • Because actions specify robot motion while future images capture resulting scene configurations, their shared denoising formulation motivates testing whether continuous control needs a separate action expert.The introduction frames the fixed action representation as a way to test reuse of the visual pathway without a learned action-specific interface.Source: §1 · pp. 1, 3
  • Action-as-Patch maps each action step to a visual-latent token through a fixed codec, allowing the visual backbone to process actions without a trainable action encoder, action-specific output projection, or separate action expert.For the considered action spaces, repetition and zero-padding preserve information, and a fixed decoder recovers actions by group averaging.Source: §1 · pp. 1–2
  • Across distinct data regimes, PatchWAM reaches 96.12% average success on RoboTwin 2.0 with augmented demonstrations and 91.8% on LIBERO-Plus with augmented training.With one of every twenty RoboTwin 2.0 training-window starts retained, it scores 88.0% average success versus 78.4% for the matched dual-expert control; that comparison does not isolate parameter sharing from attention organization.Source: Figure 1 · p. 2, §1 · p. 2
  • Conditioned on language, proprioception, and current images, a shared diffusion transformer jointly denoises action and future-image tokens under a joint flow-matching objective.The action and image tokens receive independent Gaussian noise at a shared noise level; the fixed decoder turns predicted action tokens into commands for receding-horizon execution.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work connects robot policies to pretrained vision-language and generative visual models, but many approaches retain action-specific pathways or use different visual representations for actions. PatchWAM instead encodes each action step as a single token in the backbone’s latent space.

Vision–language–action models

  • Most recent vision-language-action models generate continuous actions through a learned action head or separate action expert, with task-specific physical dynamics typically learned primarily from robot data.Some models, such as OpenVLA, represent actions as discrete tokens.Source: §2 · p. 3

World action models

  • Many world action models jointly predict future observations and actions yet retain a separate action stream or expert; other methods reduce inference cost by bypassing, distilling, or using latent-space future prediction.ImageWAM predicts a single target frame using an image-editing model, and PatchWAM’s implementation is based on ImageWAM.Source: §2 · p. 3

Actions in the visual domain

  • Visual-domain action methods include pixel-grounded action videos, pixel-motion representations, partially revealed trajectories, and latent-frame encodings; Action-as-Patch instead uses one latent token per action step without rendering pixels.Hydra-0 still uses a trained action head to map resulting features to executable actions.Source: §2 · p. 3

Joint modeling across modalities

  • Unified multimodal models process modalities as tokens, and Action-as-Patch uses the backbone’s native token format without a learned action tokenizer or dedicated action expert.Unlike approaches assigning distinct noise levels or schedules across modalities, PatchWAM uses a single noise level for action and future-image tokens.Source: §2 · pp. 3–4

3 METHOD

PatchWAM treats actions as tokens in the visual backbone, using a fixed codec so one transformer jointly predicts an action chunk and a future visual endpoint. The method specifies how these targets are represented, trained, attended to, and generated at inference.

3 METHOD

  • PatchWAM encodes each action step as a token in the backbone’s native visual format, making action prediction token prediction within the existing visual path.The model predicts a single future frame rather than a full video, and the future-frame latent need not be decoded into RGB.Source: §3 · p. 4

3.1 PROBLEM SETUP

  • Conditioned on language, proprioception, and current images, the policy predicts an H-step action chunk and the observation at its endpoint; the future frame is a training target only.At inference, the future frame and action chunk are generated jointly, but only actions are executed.Source: §3.1 · p. 4, §3 · p. 4

3.2 ACTION-AS-PATCH

  • Action-as-Patch repeats each normalized action component into a visual-width token, zero-pads the remainder, and averages copies; its fixed decoder is an exact left inverse, not a learned head.Actions do not pass through the image autoencoder or a discrete vocabulary.Source: §3.2 · p. 6

3.3 TRAINING OBJECTIVE

  • Training independently noises future-frame and action tokens at a shared noise level, then supervises both predicted velocities with weighted, group-averaged losses.The action loss excludes padded steps when a chunk extends past the demonstration’s end.Source: §3.3 · p. 7

3.4 TOKEN SEQUENCE AND ATTENTION

  • The clean conditioning prefix precedes a joint noisy block; prefix tokens attend only to the prefix, while future-frame and action tokens attend to the prefix and the entire noisy block.Positional coordinates distinguish current-frame, future-frame, and action groups; their first-axis values are structural, not physical time.Source: §3.4 · p. 7

3.5 INFERENCE

  • Inference initializes future-frame and action tokens with Gaussian noise, jointly denoises them along a shifted Euler schedule, then decodes actions for execution and replanning.Evaluations use 10 solver steps; the future-frame latent need not be decoded into RGB.Source: §3.5 · pp. 7–8

4 EXPERIMENTS

Experiments compare PatchWAM with prior methods across RoboTwin 2.0, LIBERO, LIBERO-Plus, and RoboDojo, alongside a matched dual-expert comparison. Results span full-data and clean-to-random training protocols and include additional benchmark settings.

4.1 EXPERIMENTAL SETUP

  • RoboTwin comparisons span full-data and clean-to-random protocols; matched runs retain one in twenty window starts, not one in twenty demonstrations.Full-data training includes clean and randomized demonstrations; clean-to-random training uses clean demonstrations only.Source: §4.1 · p. 8

4.2 MAIN RESULTS ON ROBOTWIN 2.0

  • 96.12% average success (96.04% clean; 96.20% randomized) puts PatchWAM 0.18 percentage points above MotuBrain under full-data training.The PatchWAM configuration includes 2,100 augmented demonstrations, so this comparison does not isolate the effect of the architecture.Source: §4.2 · p. 9, Table 1 · p. 10
  • 79.14% average success is highest under clean-to-random training, though PatchWAM's 66.72% randomized-scene rate trails the listed best of 67.90%.PatchWAM's 24.84-point clean-to-random drop cannot be attributed to architecture alone because its configuration uses photometric, AdaIN, and FFT-based augmentation.Source: §4.2 · p. 9, Table 2 · p. 10

4.3 DOES ACTION PREDICTION NEED A DEDICATED PATH?

  • 88.0% average success versus 78.4% for the dual-expert control shows PatchWAM's advantage after 41,940 updates with one in twenty window starts retained.The models share training data and optimization settings, but differ in token interaction and parameterization.Source: §4.3 · p. 9, Table 3 · p. 11
  • 60.7% at 20k updates matched the control's 60.4% only at 30k; by 30k, PatchWAM reached 81.0%, above the control's final 78.4%.The learning curves are based on single runs evaluated with 10 episodes per task and condition, so checkpoint differences are subject to sampling error.Source: §4.3 · pp. 9–10, Figure 3 · p. 11

4.4 ADDITIONAL BENCHMARKS

  • 98.4% average success on LIBERO is below QuoVLA's 99.6%, while PatchWAM matches the best listed Object-suite result at 100.0%.The benchmark is close to saturation, with three listed methods exceeding 99% average success.Source: §4.4 · pp. 11–12, Table 4 · p. 12
  • 91.8% total success on augmented LIBERO-Plus data is highest listed, 0.41 percentage points above Hermite-VLAReg; original LIBERO training yields 80.2%.The 91.8% result reflects training on perturbed data rather than zero-shot robustness.Source: §4.4 · p. 12, Table 5 · p. 13
  • 29.39 Score and 23.60% average success were leaderboard highs on RoboDojo as of September 15, 2026, but PatchWAM trailed on memory and long-horizon tasks.The next entry, DM0.5, had a Score of 24.90 and success rate of 19.34%.Source: §4.4 · p. 13

5 ANALYSIS

PatchWAM inference is slower than the dual-expert control, while full-data mechanism controls find no measurable gain from a learned interface and a small success-rate drop when action and future-frame tokens cannot attend to each other. Variants with separate noise levels do not indicate an improvement in the tested setting.

5.1 INFERENCE COST

  • 1.87×: PatchWAM takes 0.446 s per inference call versus 0.239 s for the dual-expert control, with full-sequence processing and repeated prefix computation contributing to the cost.These timings use 20 solver steps; the 10-step RoboTwin evaluation setting was not timed.Source: §5.1 · pp. 13–14

5.2 MECHANISM CONTROLS

  • 94.57% versus 94.50% average success: the learned-linear interface adds 3,726 trainable parameters but brings no measurable gain over the fixed codec.The comparison uses the full-data protocol; the learned interface does better in clean scenes and worse in randomized scenes.Source: Table 7 · p. 14, §5.2 · p. 14
  • 0.58 percentage points: preventing noisy action and future-frame tokens from attending to each other lowers average success from 94.57% to 93.99%.The isolated-attention control also drops 0.76 points in clean scenes and 0.40 points in randomized scenes; a single run per model cannot establish the interaction’s effect.Source: §5.2 · p. 14, Table 8 · p. 15
  • 88.7%, 88.3%, and 85.5%: three decoupled-noise variants compare with 88.0% for shared noise, but their changed objectives prevent isolating the noise-level effect.The first two variants’ average gains are at most 0.7 points, comparable to the uncertainty of a 10-episode evaluation; the third is 2.5 points lower.Source: §5.2 · p. 15

6 LIMITATIONS AND DISCUSSION

The matched comparison does not isolate which architectural changes explain PatchWAM’s advantage, and the paper’s results leave both seed-to-seed robustness and physical deployment unresolved.

6 LIMITATIONS AND DISCUSSION

  • The matched dual-expert comparison changes action-specific capacity, token visibility, and processing path together, so it does not establish that every dedicated action head is unnecessary.Full-data controls do not decompose the 9.6-point matched-comparison gap, and no comparison uses independent training seeds.Source: §6 · p. 15
  • Joint denoising and the isolated-attention control do not establish that future-frame predictions inform action selection or explain PatchWAM’s performance.Cross-modal coupling and pretrained-weight reuse remain motivations for the design rather than established explanations.Source: §6 · p. 15
  • All experiments are in simulation, so the results establish neither real-time operation, physical-robot performance, nor safe deployment.The 20-step inference call is 1.87 times as long as the dual-expert control, while the 10-step evaluation setting was not timed.Source: §6 · p. 15

7 CONCLUSION

The paper asks whether continuous control needs a separate action expert and tests a fixed action-to-visual-token interface with shared visual and action denoising. PatchWAM outperforms a matched dual-expert control in the subsampled-window setting and reports strong benchmark results, while identifying unresolved architectural and physical-robot questions.

7 CONCLUSION

  • 88.0% average success versus 78.4% for the dual-expert control: PatchWAM gains 9.6 percentage points with one of every 20 training-window starts retained.PatchWAM also reaches the control’s final success rate with fewer training updates.Source: §7 · p. 16
  • PatchWAM maps each action step to a visual latent token and jointly denoises action and future-frame tokens without a trainable action encoder, action-specific output projection, or separate action expert.The approach uses a fixed codec and shared transformer.Source: §7 · p. 16
  • With augmented demonstrations, PatchWAM reports 96.12% average success on RoboTwin 2.0 and 91.8% on LIBERO-Plus; its RoboDojo Score of 29.39 led the September 15, 2026 leaderboard entries.The RoboTwin result uses the full-data setting with augmented demonstrations; the LIBERO-Plus result uses augmented training data.Source: §7 · p. 16
  • The paper says further matched-setting controls and independent training seeds are needed to identify the factors behind the advantage, and physical-robot testing remains outstanding.Joint denoising also makes inference slower than in the dual-expert control.Source: §7 · p. 16

REPRODUCIBILITY STATEMENT

The reproducibility statement points readers to sections and appendices for method, run, and evaluation details, while noting that results lack independent-seed repetitions.

REPRODUCIBILITY STATEMENT

  • Section 3 and Appendix E describe the Action-as-Patch codec, training objective, token sequence, attention mask, and an example RoboTwin 2.0 sequence.Appendix E shows the sequence and mask for one sample.Source: REPRODUCIBILITY STATEMENT · p. 16
  • Appendix F lists shared model settings, run schedules, data filtering, augmentations, model inputs, and RoboDojo’s text encoder.It also describes the dual-expert control architecture, initialization, and parameter count.Source: REPRODUCIBILITY STATEMENT · p. 16
  • Appendix F reports evaluation seeds for RoboTwin 2.0 and LIBERO, LIBERO trial counts, and benchmark replanning intervals, but no result is repeated with independent training or evaluation seeds.Source: REPRODUCIBILITY STATEMENT · p. 16

A LIBERO-PRO COMPARISON

On LIBERO-Pro, PatchWAM’s average success rate exceeds π0 and OpenVLA but trails the other three listed methods. Its results vary substantially by perturbation, with especially low success on changed initial positions and changed tasks.

A LIBERO-PRO COMPARISON

  • 8.8% on changed initial positions and 5.0% on changed tasks show low PatchWAM success in both categories, which are difficult for every listed method.Only QuoVLA exceeds 21% in either category.Source: A LIBERO-PRO COMPARISON · p. 22
  • 80.8% under object perturbations is the lowest success rate among the listed methods.Source: A LIBERO-PRO COMPARISON · p. 22
  • 98.4% on unperturbed tasks and 96.8% under paraphrased instructions are respectively 1.4 and 2.7 percentage points below the best listed results.Source: A LIBERO-PRO COMPARISON · p. 22
  • 57.9% average success rate places PatchWAM above π0 and OpenVLA but below the other three listed methods.Table 9 averages success rates over the four LIBERO suites and sorts methods by average SR.Source: Table 9 · p. 22, A LIBERO-PRO COMPARISON · p. 22

B TRAINING DYNAMICS UNDER THE FULL-DATA PROTOCOL

Under the full-data protocol without augmented demonstrations, PatchWAM’s success rate rises rapidly early in training and later exceeds the released ImageWAM checkpoint. The final checkpoint’s 100-episode result differs from the separate augmented-demonstration configuration reported elsewhere.

B TRAINING DYNAMICS UNDER THE FULL-DATA PROTOCOL

  • Success rises from 65.8% after 10k updates to 80.5% after 20k and 92.0% after 40k, with most improvement in the first two epochs.Source: B TRAINING DYNAMICS UNDER THE… · p. 22
  • From 70k updates onward, every evaluated checkpoint exceeds the released ImageWAM checkpoint, while neighboring-checkpoint differences are comparable to 10-episode evaluation uncertainty.The final checkpoint is 95.45% with 10 episodes and 94.57% with 100 episodes per task and condition.Source: B TRAINING DYNAMICS UNDER THE… · p. 22
  • 94.57% final-checkpoint success under the 100-episode evaluation exceeds the released ImageWAM checkpoint by 1.80 percentage points.Figure 5 traces a 104,850-update run without augmented demonstrations; the 96.12% result comes from a separate configuration with 2,100 augmented demonstrations and 110k updates.Source: B TRAINING DYNAMICS UNDER THE… · p. 22, Figure 5 · p. 23

C PER-TASK RESULTS UNDER THE CLEAN-TO-RANDOM PROTOCOL

The clean-to-random results vary by task: PatchWAM performs strongly on many randomized-scene tasks, while a few remain difficult, and methods with similar aggregate success can solve different tasks. The appendix also pairs successful and failed rollouts and details the token sequence and attention mask used for a RoboTwin example.

C PER-TASK RESULTS UNDER THE CLEAN-TO-RANDOM PROTOCOL

  • In randomized scenes, PatchWAM reaches at least 80% success on 16 tasks and at most 20% on two; against OLA-Sem and GigaBrain-0.7, it wins on 23 and 24 tasks, respectively, and loses on 26 each.In clean scenes, it succeeds in every episode on 11 tasks and falls below 80% on five.Source: C PER-TASK RESULTS UNDER THE… · p. 23, Figure 6 · p. 24

D QUALITATIVE CLEAN-TO-RANDOM ROLLOUTS

  • The four paired randomized-scene rollouts come from tasks with success rates of 4%, 9%, 25%, and 48%, and each row contrasts a successful and failed outcome from the same task.The examples are selected from 100 evaluation episodes per task; the RoboTwin evaluator supplies the success and failure labels.Source: D QUALITATIVE CLEAN-TO-RANDOM ROLLOUTS · p. 24, Figure 7 · p. 25

E TOKEN SEQUENCE AND ATTENTION MASK

  • A RoboTwin sample contains 721 tokens, including 16 action tokens that account for about 2% of the sequence; each frame contributes 288 latent tokens.The instruction is padded to 128 text tokens, with a proprioceptive token inserted after valid text.Source: E TOKEN SEQUENCE AND ATTENTION MASK · p. 25, Figure 8 · p. 26
  • Future-frame and action queries attend to the prefix and the entire noisy block, whereas prefix queries attend only to prefix keys.The isolated-attention control prevents future-frame and action tokens from attending to each other while preserving access to the prefix and their own group.Source: E TOKEN SEQUENCE AND ATTENTION MASK · p. 25, Figure 8 · p. 26

F IMPLEMENTATION DETAILS

The implementation appendix specifies how training windows, image augmentation, model configurations, and evaluation procedures differ across runs. The matched RoboTwin comparison retains a subsampled set of windows, while other runs use distinct augmentation and inference settings.

F IMPLEMENTATION DETAILS

  • RoboTwin 2.0 has about 5.37 million training windows; the matched comparison retains one start in twenty, about 268,000 windows, and trains both runs over each retained window ten times.Each sample spans 17 time steps, covering the current observation, 16 actions, and the observation at the end of the chunk.Source: F IMPLEMENTATION DETAILS · p. 26
  • For the clean-to-random run, stronger randomization is applied to 80% of samples, with color, exposure, blur, noise, AdaIN, and FFT-based transforms but no crops or rotations.The transforms are shared across both frames and drawn separately for each camera view.Source: F IMPLEMENTATION DETAILS · p. 27
  • The RoboDojo run conditions on 20 slots of past head-camera frames and adds 320 pooled history tokens without adding parameters.Its text encoder is Qwen3-VL-4B-Instruct; its vision encoder remains frozen, while rank-64 LoRA adapters add 132.1M trainable parameters.Source: F IMPLEMENTATION DETAILS · pp. 27–28
  • The matched dual-expert control uses ImageWAM with the same FLUX.2 [klein] 4B backbone and an action expert that adds 0.64B trainable parameters.The expert starts from random weights and uses linear layers to embed action steps and read out action velocity.Source: F IMPLEMENTATION DETAILS · p. 28
  • RoboTwin 2.0 executes each predicted 16-action chunk in full before replanning, whereas LIBERO and LIBERO-Plus replan after 12 actions.LIBERO evaluations use 20 trials per task; all evaluations use 10 solver steps and no classifier-free guidance.Source: F IMPLEMENTATION DETAILS · p. 28

G SUPPLEMENTARY SELF-FLOW RESULTS

The supplementary self-flow results compare three variants that alter the training objective, rather than independent PatchWAM repetitions. Variants 1 and 2 succeed more often in clean scenes, while all three succeed less often in randomized scenes.

G SUPPLEMENTARY SELF-FLOW RESULTS

  • Table 12 reports success rate percentages after 41,940 updates, retaining one of every 20 window starts and using 10 episodes per task and condition.Source: Table 12 · p. 28
  • Variants 1 and 2 succeed more often in clean scenes, while all three variants succeed less often in randomized scenes.These are self-flow variants, not independent training repetitions of PatchWAM, because they change the training objective.Source: G SUPPLEMENTARY SELF-FLOW RESULTS · p. 28
  • Variant 1 combines dual-timestep scheduling with an EMA-teacher representation loss; Variant 2 adds modality-structured timestep masks.Source: G SUPPLEMENTARY SELF-FLOW RESULTS · p. 28
  • Variant 3 additionally withholds action labels for 25% of samples and substitutes teacher-generated labels.Source: G SUPPLEMENTARY SELF-FLOW RESULTS · p. 28
Loading 2609.25961v1…