CustomX: Unified Character, Action, and Scene Customization in Video World Models

1 Introduction

• CustomX combines explorable 3D environments with user-specified characters capable of open-ended actions, addressing limits of static-world and controllable-entity approaches.Static-world methods lack active agents, while controllable-entity methods support only limited actions in otherwise uncontrollable environments.Source: §1 · p. 2
• Users supply a 3DGS scene and a 3D or multi-view character, then use natural language to direct behavior and explore the environment through coherent video clips.Scenes may represent synthetic environments or real-world reconstructions; clips capture evolving states of both character and environment.Source: §1 · p. 2
• CustomX supports up to hundreds of actions, including locomotion, gestures, and object-centric interactions, alongside user-directed camera behavior and iterative long-horizon interaction.Camera control uses a rendered projection of the 3DGS scene along the specified path as an explicit conditioning input.Source: §1 · pp. 2–3
• Fine-tuning a pre-trained video generator on a small basic-locomotion dataset enhances motion quality while preserving generalization across actions and characters.The paper evaluates visual quality, character consistency, action-control success across up to around 150 actions, and long-horizon generation; CustomX consistently outperforms video-generation foundation models and dedicated world models across nearly all metrics.Source: §1 · p. 3

Controllable Video Generation

• Controllable video generation has advanced through subject and camera controls; CustomX instead conditions camera motion on a rendered 3DGS projection along the requested path.Typical subject control uses reference-image embeddings, while common camera control injects Plücker embeddings.Source: §2 · p. 4

Memory Mechanism in Video Generation

• Video-generation memory methods retrieve prior content or maintain point-cloud maps, while CustomX uses the 3DGS scene as spatial memory and the character as a source of dynamic motion and interaction cues.Prior methods retrieve RGB or depth content, select views by surfel indexing or camera overlap, or maintain a global point-cloud map.Source: §2 · p. 4

World Models for Static Scene Creation

• Static-world methods either represent worlds implicitly in video models or explicitly as manipulable 3DGS scenes; CustomX lets users specify a 3DGS scene or uses Marble to generate one when absent.Other developments generate 3DGS from panoramic inputs, text, or a single image, or integrate video generation to streamline 3DGS creation.Source: §2 · p. 4

Method overview

• The training overview maps character-action videos through segmentation, inpainting, and token encoding before predicting target video tokens from scene, mask, text, and multi-view character conditions.The model uses a Multi-Modal Diffusion Transformer trained with Flow Matching [55].Source: Figure 2 · p. 5

3 Method

• CustomX formulates interaction as conditional autoregressive video generation, synthesizing each clip from the preceding clip, character and scene representations, and current text instruction.The scene and character provide spatial and visual grounding, while the instruction specifies behavior and preceding clips provide temporal references.Source: §3 · p. 5

3.1 Training Data Pre-Processing

• Training samples use fixed-length, single-action GTA-V clips, with characters segmented, removed to create inpainted scene videos, and paired with action labels and four-view character renders.Each clip contains 129 frames; the processed sample is represented as {V, S, M, T, C}.Source: §3.1 · pp. 5–6

3.2 Architecture

• The training pipeline encodes video, scene, mask, text, and multi-view character inputs, then uses Flow Matching to predict target video tokens from noise under multiple conditions.Scene and mask tokens are projected and added to noisy target tokens; text, character, and conditional noisy tokens are concatenated for the Transformer backbone.Source: §3.1 · p. 6, §3.2 · pp. 6–7
• View-specific shifted 3D-RoPE assigns the front, left, right, and back character tokens temporal shifts of −1, −2, −3, and −4 while keeping their spatial shift the same.Text tokens receive no positional embeddings, while conditional noisy tokens use standard 3D-RoPE.Source: §3.2 · p. 7

3.3 Auto-Regressive Mode

• Autoregressive training conditions new target tokens on preceding video tokens, using the first quarter of a target sequence as context and the remaining three quarters as the prediction target.Training adds small Gaussian noise to ground-truth preceding tokens to mitigate their mismatch with model-generated preceding tokens at inference.Source: Figure 3 · p. 7, §3.3 · p. 8

3.4 Inference and Acceleration

• At inference, users specify the character, scene, camera viewpoint, and character anchor; the instruction determines a camera path used to render a scene video before model generation.Instructions are parsed into locomotion, gesture, object-centric action, and camera behavior; locomotion and camera instructions determine trajectories, while gesture and object-centric instructions use a stationary camera.Source: §3.4 · pp. 8–9
• Iterating Steps 2 and 3 enables temporally consistent, long-horizon generation; for faster inference, DMD2 distills the original 30-step denoising model into a more efficient 4-step version.Source: §3.4 · p. 9

4.1 Experimental Settings

• The 13B-parameter model is trained at 360P and 720P using LoRA-style fine-tuning, with a training set of 2,084 samples featuring five characters.Both resolution-specific models are trained for 5,000 steps; the 360P version is the default unless otherwise specified.Source: §4.1 · p. 9
• Evaluation spans WorldScore visual-quality metrics, character similarity, action control, and generalization to as many as 142 novel actions.The comparison includes dedicated world models and video-generation base models; the three dedicated models support locomotion only, and models requiring an initial image use Gemini-rendered character-scene images.Source: §4.1 · p. 10, §4.2 · pp. 10–11

4.2 Main Results

• CustomX outperforms its base model, with fine-tuning reported to improve motion fidelity and behavioral coherence while retaining broad pretrained generalization.The authors interpret the locomotion-focused training data as refining motion dynamics and human-embodiment representations rather than redefining the model’s generative space.Source: §4.2 · p. 11

4.3 Ablation Study

• Character consistency improves with more character views, while per-frame character anchors are reported to raise DINOv2 and CLIP consistency scores.The four-view model is compared with single-view and front–back-view baselines; the anchor comparison contrasts per-frame anchors with no anchor and a first-frame-only anchor.Source: Table 3 · p. 13, Table 4 · p. 13, §4.3 · p. 13

4.2 Main Results

• A four-interaction example maintains temporal coherence across generated clips, alongside visualizations of different characters performing varied actions in two scenes.The paper also presents a long-horizon exploration example and additional results in the appendix and supplementary videos.Source: Figure 5 · p. 12, Figure 6 · p. 12, §4.2 · p. 13

4.3 Ablation Study

• Using both the 3DGS scene and multi-view character conditions improves long-horizon visual quality over variants that replace either condition with text descriptions.The comparison uses CLIP-Aesthetic and DINOv2 scores over autoregressive sequences of 10 clips per trial.Source: §4.3 · pp. 13–14, Figure 7 · p. 14
• Hybrid training with 400 real-world videos improves realism and captures physical effects such as dynamic clothing wrinkles absent from game footage.Training solely on GTA-V tends to produce stylized, game-engine rendering even for real-world character inputs.Source: §4.3 · p. 14, Figure 8 · p. 14
• DMD2 distillation reduces 360P generation latency from 121s to 21s, with slight drops in DINOv2 and CLIP-Aesthetic scores.The distilled model uses four denoising steps rather than the original 30-step schedule.Source: §4.3 · p. 14

• CustomX lets users provide a character and 3DGS scene, then iteratively control the character and explore the world.It supports open-ended natural-language action control rather than only a small set of predefined actions.Source: §5 · p. 15
• The model improves motion dynamics and character consistency over its base model across evaluations of visual quality, action control, character fidelity, and long-horizon generation.Source: §5 · p. 15
• Inference remains non-real-time, and the current model is limited to videos with one active subject.It cannot generate videos involving multiple simultaneously active characters.Source: §5 · p. 15

• The prompt pairs an action description with encoded front, left, back, and right character views, separated by modality tokens.The LLaVA image encoder produces the view embeddings, and <SEP> distinguishes text from visual modalities.Source: §A.1 · p. 16
• The prompt’s [Action] placeholder denotes the action description supplied for the character.Source: §A.1 · p. 16

A.2 More Training Details

• Training starts from HunyuanCustom, freezes the LLaVA encoder and scene-condition projector, and applies rank-64 LoRA to the backbone.The scene condition is randomly dropped with probability 0.3; optimization uses AdamW with a 1e-4 learning rate and 500 warm-up steps.Source: §A.2 · p. 16
• First-clip generation produces 93 frames, while autoregressive generation conditions on the preceding clip’s last 33 frames and generates 96 new frames.Source: §A.2 · p. 16

A.3 Inference Acceleration

• DMD2 distillation trains student and fake-score LoRA parameters alongside a frozen teacher, requiring three 13B models and substantial GPU memory.The approach is applied only to the 360P model, trained for 4,000 steps on eight NVIDIA B200 GPUs.Source: §A.3 · p. 16
• Hybrid training with game and real-world data is reported to improve character consistency, as measured by DINOv2 and CLIP scores.Source: Table A1 · p. 17

B.1 Game-Real Hybrid Data Enhances Real-World Character Fidelity and Physics

• The added real-world dataset contains 400 clips of people performing the same locomotion actions as the GTA-V data, processed at 360P.The motivation is that GTA-V-only training tends to make real-world characters look stylized; hybrid training aims to improve realism while retaining motion patterns from game data.Source: §B.1 · p. 17
• Training distinguishes visual domains by tagging GTA-V samples “rendered” and real-world samples “real.”The remaining training procedure is unchanged.Source: §B.1 · p. 17
• Hybrid-data training produces measurable improvements in real-world character fidelity for two unseen individuals.The evaluation uses DINOv2 and CLIP scores.Source: §B.1 · p. 17

B.2 Inference Acceleration

• The qualitative comparison includes the original 30-step model, the accelerated four-step model, and the original model restricted to four steps.The third variant tests whether fewer steps alone account for the accelerated model's result.Source: §B.2 · p. 17, Figure A1 · p. 18
• The accelerated four-step model maintains visual fidelity on par with the original framework while delivering a 7.5× inference speedup.The comparison uses the original model with a 30-step denoising schedule.Source: §B.2 · p. 18, Figure A1 · p. 18
• Restricting the non-accelerated original model to four denoising steps causes a pronounced visual-quality degradation.This contrasts with the accelerated four-step model's visual fidelity, which is on par with the original framework.Source: §B.2 · pp. 17–18

B.3 Latencies for Higher-Resolution Inference

• DINOv2 rises from 0.698 to 0.704 and CLIP-Aesthetic rises from 5.665 to 5.887 for 720P versus 360P.These are the reported metric pairs for higher-resolution inference.Source: §B.3 · p. 18
• Generating a 93-frame video clip at 720P takes 159 seconds on 8× NVIDIA H100 GPUs.Source: §B.3 · p. 18
• The 720P model outperforms the 360P model on both reported metrics, DINOv2 and CLIP-Aesthetic.Source: §B.3 · p. 18

B.4 Overall Inference Latencies

• For a cold-start 96-frame 360P video on one H100, accelerated inference takes approximately 21 seconds; without acceleration, the same generation takes approximately 121 seconds.Cold-start total latency also includes a one-time 3DGS scene-generation phase of approximately 300 seconds with World Labs Marble or roughly 9 seconds with FlashWorld, and approximately 0.695 seconds for rendering; pre-existing 3DGS assets bypass scene generation.Source: §B.4 · p. 18

B.5 Computational Cost of 720P Video Generation

• A 96-frame 720P video requires approximately 85,800 spatiotemporal tokens, with latency of roughly 151 seconds for HunyuanCustom-13B and 182 seconds for Wan2.2-14B on 8× NVIDIA H100 GPUs.The authors identify high-resolution generation as a limitation for existing generators and their method; 360P is the default, while 720P is reserved for enhanced visual quality.Source: §B.5 · pp. 19–20

C.1 Action Control and Generalization

• The paper reports quantitative control success for 142 novel actions and visualizes 84 sampled actions with one character and 25 annotated actions with a different character.Source: Figure A2 · p. 19, §C.1 · p. 20, Figure A3 · p. 21

C.2 Scene Customization

• CustomX scenes include varied generated worlds and a publicly available reconstructed real-world 3DGS scene.Most scenes in the work are sourced from World Labs Marble.Source: §C.2 · p. 20

C.3 Character Customization

• The model is demonstrated controlling previously unseen characters, including diverse characters sourced from generation tools or online repositories.The examples show these characters performing locomotion actions.Source: §C.3 · p. 20

C.4 Long-Horizon Generation

• Autoregressive generation builds temporally coherent sequences from previously generated clips, supporting extended user–model interactions.The appendix presents two long-horizon examples.Source: §C.4 · p. 20

D Potential Negative Societal Impact

• The framework may be repurposed to produce malicious content, motivating the importance of ethical safeguards.Source: D Potential Negative Societal Impact · p. 21