UniAction: Unified Action Conditioning for World Models

1 INTRODUCTION

• Existing action-conditioned world models typically use action-specific mechanisms and data, fragmenting scarce annotations and isolating knowledge about scenes, objects, and world dynamics.Action-annotated data are limited in scale and concentrated in a narrow range of scenes and interactions.Source: §1 · p. 1
• Existing generic conditioning either renders actions into visual conditions, which not every action can provide, or uses tokens whose action-to-video correspondence remains implicit.The passage contrasts visual rendering with token conditioning as alternatives for accommodating heterogeneous actions.Source: §1 · pp. 1–2
• UniAction represents each action as sparse elements combining a discrete semantic identity with a continuous spatiotemporal coordinate in the pretrained video model’s positional geometry.Cardinality-calibrated attention addresses conditioning-strength bias from differences in the number of elements.Source: §1 · p. 2
• A single jointly trained UniAction model outperforms action-specific experts across all four action forms, while also outperforming conditioning alternatives in action following and visual quality.The evaluated forms are force, camera control, hand pose, and robot pose.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

• Video world models typically define controls for specific environments, whereas UniAction lets actions from different environments and structures condition one model through a shared interface.The cited examples of specific controls include keyboard inputs and camera movement signals.Source: §2 · pp. 2–3
• Action-conditioned video models commonly use representations and conditioning mechanisms tailored to game, camera, navigation, force, hand-motion, or robot-manipulation actions.The cited approaches focus on specific action spaces.Source: §2 · p. 3
• Prior unification approaches connect related actions through shared kinematic structures or common physical-control variables, while UniAction accommodates distinct structures and parameterizations.The shared formulation uses action semantics and spatiotemporal grounding to support joint learning within one world model.Source: §2 · p. 3
• Controllable video methods unify controls through visual conditions or learned encoders, while UniAction represents actions directly without a separate renderer or learned encoder for each action type.Visual-condition approaches require a suitable visual encoding for each control signal.Source: §2 · p. 3

3 METHOD

• Each action element pairs a discrete semantic identity with a continuous video-space grounding coordinate, and all elements condition the shared model through one mechanism.The model generates future video conditioned on an initial frame, text prompt, and action.Source: §3 · pp. 3–4

3.1 ACTION SEMANTICS

• A shared semantic vocabulary maps each action element’s identity to a learned action token; identities are reused across locations and times, while grounding represents those differences.New semantic identities can be added to the vocabulary for new action forms.Source: §3.1 · p. 4

3.2 SPATIOTEMPORAL GROUNDING

• Grounding uses continuous time and video-space coordinates, then reuses the pretrained backbone’s positional system so action and video tokens interact through native self-attention.This places both token types in the same spatiotemporal geometry, reusing the backbone’s pretrained spatiotemporal modeling capabilities.Source: §3.2 · p. 4

3.3 CARDINALITY-CALIBRATED ATTENTION

• Cardinality-calibrated attention offsets action logits using a shared reference count, making total action attention mass independent of actual token count while preserving relative action-token attention.The correction addresses bias introduced because representation granularity changes the number of action tokens.Source: §3.3 · p. 4

4.1 BACKBONE INSTANTIATION

• In Wan2.2, action tokens are concatenated after video tokens and interact with them through the backbone’s existing self-attention.Action locations are mapped into the latent coordinate system, applying the same rotary positional encoding as video tokens.Source: §4.1 · p. 5

4.2 ACTION INSTANTIATION

• The four action forms use sparse elements to encode distinct structures, including force vectors, framewise pose joints, and globally acting camera commands.Force uses elements at the vector’s tail and head; camera commands receive fixed canonical frame locations, while visible hand and robot joints are located at their image positions.Source: §4.2 · p. 5

4.3 EXPERIMENTAL SETUP

• The setup samples 10K clips per action form, totaling 40K clips, and evaluates counterfactual action responses with motion estimators and human ratings.The main backbone is Wan2.2-I2V-A14B’s high-noise expert adapted with LoRA; human evaluation covers 60 cases and 100 participants.Source: §4.3 · p. 6

4.4 COMPARISON WITH GENERIC CONDITIONING MECHANISMS

• UniAction achieves the best action following and lowest LPIPS across four action forms, while human ratings favor its action following on every form.Cross-Attn performs worse for spatially localized hand and robot pose; VACE has its largest gap on force and camera control, which lack natural visual forms.Source: Table 1 · p. 6, §4.4 · p. 7, Figure 3 · p. 7

4.5 BENEFITS OF UNIFIED TRAINING

• Jointly trained UniAction outperforms action-specific experts in action following across all four action forms, with lower or comparable LPIPS.The experts use the same samples and total compute, but together have four times the parameters and hard-route samples by action form.Source: §4.5 · p. 8, Table 2 · p. 8

4.6.1 SPATIOTEMPORAL GROUNDING

• Under the same training budget, replacing shared pretrained positional geometry or self-attention yields low action following and near failure on hand and robot pose.The ablations replace the shared attention pathway or the pretrained 3D RoPE with absolute positional encodings; the study reports these variants in Table 3.Source: §4.6.1 · pp. 8–9, §4.6 · p. 8, Table 3 · p. 9
• Training forces only at t = 0 still lets UniAction start induced motion at unseen test times t = 8 and t = 24.Learned APE changes little as force time shifts; the comparison is shown for models trained on force data with all forces applied at t = 0.Source: §4.6.1 · p. 9, Figure 5 · p. 9

4.6.2 CARDINALITY CALIBRATION

• Removing cardinality calibration lowers action following for force and camera control, while pose actions remain largely unaffected.The reported pattern is consistent with attention mass growing with token count: many-token pose actions dominate while sparse force and camera tokens are suppressed without calibration.Source: §4.6.2 · p. 9

5 CONCLUSION

• Across force, camera control, hand pose, and robot pose, UniAction outperforms visual- and token-based alternatives, while one jointly trained model surpasses action-specific experts.The paper presents this as evidence that heterogeneous action data can benefit one another within a shared world model.Source: §5 · p. 9
• Cardinality-calibrated attention prevents differences in action representation granularity from biasing conditioning strength.Action and video tokens interact in a shared space, with action coordinates mapped to the pretrained model’s native positional geometry.Source: §5 · p. 9
• The authors report using AI tools to polish writing, create decorative figure icons, and help produce the supplementary introductory video, while experimental results were not AI-generated, modified, or selected.The authors state that they reviewed the AI-assisted content and take responsibility for the paper and supplementary materials.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The human evaluation recruited 100 Prolific participants, collected informed consent, and gathered no personally identifiable information; training data came from publicly available datasets used under their licenses.Source: ETHICS STATEMENT · p. 10

A SEMANTIC REPRESENTATION AND EXTENSIBILITY

• In high-dimensional feature space, distinct semantic identities are initialized with low correlation and approximate orthogonality, while their correspondence to scene evolution is learned through joint action-conditioned training.Source: A SEMANTIC REPRESENTATION AND… · p. 13
• The semantic vocabulary extends by appending independently initialized tokens, allowing additional action forms to be incorporated and learned through the shared world model.Source: A SEMANTIC REPRESENTATION AND… · p. 13

B.1 DERIVATION

• Actions with comparable average video-query compatibility can receive different total attention shares solely because their representations contain different numbers of tokens.The uncalibrated action contribution scales with action cardinality, which is determined by representation granularity.Source: §B.1 · p. 13
• Calibration replaces actual action cardinality with a fixed reference cardinality in the total action contribution, while preserving average action–video compatibility and relative attention relationships among action tokens.The same offset is added to every action logit.Source: §B.1 · pp. 13–14

B.2 FORWARD, BACKWARD, AND INFERENCE

• In the forward pass, a scalar offset is applied to the action-logit block before the shared softmax; the output combines video and action values using their calibrated attention weights.Source: §B.2 · p. 14
• The calibrated action attention weight controls each action token’s forward contribution and the training signal propagated through the action branch.The reference cardinality is fixed, calibration adds no learnable parameters, and the same computation is used at training and inference.Source: §B.2 · p. 14

B.3 ALGORITHM

• The algorithm computes video and action logits, offsets action logits by log(Nref/Na), applies a joint softmax, and combines video and action values to produce video-token outputs.It retains the original attention computation while adding a cardinality-dependent action-logit offset.Source: §B.3 · p. 14

C EXPERIMENTAL DETAILS

• The main training set contains 40K clips, with 10K action-annotated clips selected from each of GoalForce, SolarWM, EgoDex, and DROID.Clips were sampled to 81 frames and resized and cropped to 480 × 832 pixels.Source: C EXPERIMENTAL DETAILS · p. 15
• Compared methods share data preprocessing, per-task training-data budgets, inference configuration, and evaluation protocols; UniAction uses a fixed reference cardinality of 784.Training updates the action branch while keeping the video branch frozen, over two full passes through the training data.Source: C EXPERIMENTAL DETAILS · p. 15
• The test set fixes each source sample’s initial frame and prompt, varies only the action condition, and uses 107 held-out source samples to form 410 test conditions per model.Conditions include original actions and intervention variants; Table 4 is identified as the test-set construction table.Source: C EXPERIMENTAL DETAILS · p. 15, Table 4 · p. 15
• Action-following scores range from 0 to 1, average action variants equally, and further average Force ball, domino, and flower subcategories equally.The evaluation also uses camera-trajectory alignment and tracked hand or robot displacement; invalid force responses score zero.Source: C EXPERIMENTAL DETAILS · pp. 15–16
• In human evaluation, 100 participants compared UniAction, VACE, and Cross-Attention videos side by side, rating action following and visual quality on 1–5 scales.Each participant saw eight video groups, with two groups per action form.Source: C EXPERIMENTAL DETAILS · p. 16

D EFFICIENCY ANALYSIS

• UniAction incurs the lowest additional latency across all four action forms, while its overhead varies with the number of action tokens.The comparison measures one denoising forward pass against the backbone without action input; Table 5 reports trainable parameters, additional TFLOPs, and relative latency increases.Source: D EFFICIENCY ANALYSIS · p. 16, Table 5 · p. 16

E ADDITIONAL QUALITATIVE COMPARISONS

• Figures 6–8 provide 13 additional comparisons across four action forms, with six frames per method and overlays indicating input conditions.Figure 6(b) uses W to indicate forward camera motion.Source: E ADDITIONAL QUALITATIVE COMPARISONS · p. 17, Figure 6 · p. 17, Figure 7 · p. 18

• Training only with forces applied at t = 0, UniAction delays object motion when force application is shifted to t = 8 or t = 24.Figures 9 and 10 show temporal generalization on dominoes and balls, respectively; yellow arrows indicate the input force.Source: F ADDITIONAL TEMPORAL GENERALIZATION… · p. 20, Figure 9 · p. 20, Figure 10 · p. 20

G SCENE GENERALIZATION

• Figure 11 presents 24 generations from the jointly trained model; all but two specified examples use initial frames generated by GPT Image 2 and unseen during training.Each strip shows an initial frame and two later frames; Figure 11’s caption describes scene and appearance generalization across four action forms.Source: G SCENE GENERALIZATION · p. 21, Figure 11 · p. 21

H COMPOSITIONAL ACTION GENERALIZATION

• At inference, tokens for force, camera control, hand pose, and robot pose can be concatenated to generate simultaneous actions or transitions without additional training.Figure 12 shows three long videos; each training sample contains only one action form, and the initial frames were generated by GPT Image 2.Source: H COMPOSITIONAL ACTION GENERALIZATION · p. 22, Figure 12 · p. 22