Source-linked AI summary
H3-World: Turning Language Understanding into World Control
Danze Chen, Zeqing Wang, Ziyue Lin, Xingyi Yang, Yeying Jin
TL;DR
Pretrained video generators offer coarse language-based control, but precise temporal control for interactive world modeling remains limited. H3-WORLD converts character and camera actions into temporally grounded language instructions and applies lightweight adaptation, achieving effective control with preserved generation quality and generalization to unseen scenarios.
Problem
Pretrained video generators do not by themselves expose the precise action interface required for interactive world modeling.
Method
H3-WORLD represents character and camera actions as compositional text, aligns them with video latent intervals, and uses temporal attention routing with lightweight LoRA adaptation.
Results
With 8,000 gameplay samples and 0.199% trainable parameters, H3-WORLD achieves effective character and camera control while preserving generation quality and generalizing to unseen actions and visual scenarios.
Takeaways & Limitations
The results show that coarse language-based control emerging in large video generators can be transformed into interactive world control.
Takeaways & Limitations
The study focuses on short-horizon generation, while persistent world state, real-time interaction, planning, and policy learning remain unsupported.
Abstract
from arXiv · showhide
We present H3-World, an efficient framework that turns the 33B MiniMax-H3 video generator into an interactive world model. Our key finding is that, as large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3, for example, already supports zero-shot control of character behavior and camera motion through natural-language instructions. Building on this, H3-World turns this coarse language interface into precise, temporally grounded world control, without introducing dedicated action modules. Specifically, we represent each action as a structured combination of character and camera instructions, and align them with the corresponding temporal video latents. To make the control temporally precise, we further introduce temporal attention routing, which restricts each instruction to its intended time interval and reduces control leakage across actions. Importantly, H3-World directly reuses the semantic representations learned during large-scale video pretraining and requires only lightweight adaptation. With only 8,000 gameplay samples, 10,000 LoRA optimization steps, and 0.199% trainable parameters, H3-World achieves effective character and camera control while preserving strong generation quality. It also generalizes to unseen scenarios. These results show that the control capabilities emerging in large video generators can be efficiently transformed into interactive world control.
1 Tencent 2 National University of Singapore 3 The Hong Kong Polytechnic University
Figure 1 presents H3-WORLD controlling both characters and cameras across diverse visual environments.
- H3-WORLD enables character control across diverse visual environments.
- H3-WORLD enables camera control across diverse visual environments.
- Figure 1 summarizes control across diverse visual environments.
1 Introduction
The introduction argues that pretrained video generators already expose coarse language-based control, but precise interactive control remains underdeveloped. H3-WORLD addresses this gap by temporally grounding language actions with lightweight adaptation, achieving effective control and generalization while preserving generation quality.
- Motivation: Existing control systems add action pathways and adaptation on top of pretrained video generators, increasing supervision, computation, and storage demands.The introduction also notes that extensive adaptation may disturb capabilities acquired during pretraining.
- Motivation: MiniMax-H3 already follows coarse textual instructions for character and camera motion without action-conditioned training.These instructions produce roughly correct responses with realistic visual dynamics.
- Approach: H3-WORLD converts character and camera actions into compositional text, aligns them with video latent intervals, and routes them temporally without a dedicated action module.The framework injects controls through MiniMax-H3’s native text pathway and uses lightweight LoRA adaptation.
- Results: 8,000 gameplay samples and 0.199% trainable parameters suffice for effective control while preserving the pretrained model’s generative capabilities.The reported control covers both character and camera behavior.
- Results: H3-WORLD transfers beyond training observations, including unseen action compositions and distinct initial observations.The introduction presents this transfer as evidence that pretrained video generators contain structure useful for interactive control.
- Implications: The paper concludes that coarse language-based control in large video generators provides a basis for interactive world modeling.
2 Related Work
Related work spans action-conditioned simulators, language-based interventions, and controllable video generation. H3-WORLD differs by grounding language actions inside MiniMax-H3’s packed bidirectional single-stream attention.
- World Models: Earlier world models condition visual dynamics on recorded actions, while Genie learns a latent action interface from video.
- World Models: Recent systems extend video generators toward open worlds, device control, causal rollout, persistent history, and real-time inference.
- Language Actions and Temporal Grounding: H3-WORLD focuses on temporal grounding because semantic conditions and video latents coexist in MiniMax-H3’s packed single-stream self-attention.Unlike systems with separate text cross-attention, MiniMax-H3 jointly denoises the full future horizon.
3 Method
H3-WORLD converts compositional character–camera commands into temporally aligned textual prompts and routes them through MiniMax-H3’s native sequence architecture. Lightweight LoRA adaptation preserves the model’s shared visual processing while assigning each action a direct entry point to its matched video latent.
- 3.1 Overview: H3-WORLD generates future video latents from an initial observation, static semantic condition, and scheduled action sequence.
- 3.4 Single-Egress Routing and Low-Rank Adaptation: LoRA adapts MiniMax-H3’s attention projections while the model processes the packed multimodal sequence through single-stream self-attention.The native audio stream is retained while the method focuses its notation on visual action control.
- 3.2 Semantic Action Interface: The semantic action interface represents each action as a character–camera pair and converts the pair into a short concatenated textual instruction.The representation preserves the factorization of the action space within H3’s native text-conditioning space.
- 3.2 Semantic Action Interface: Training covers 83 of 135 structurally valid character–camera combinations, with the 20 and 40 most frequent pairs accounting for 71.4% and 95.4% of prompts.This sparse, imbalanced support motivates evaluation of compositional control on unseen joint commands.
- 3.3 Latent-Aligned Temporal Binding: Each action prompt is independently encoded, assigned a mirrored temporal position, and paired with the corresponding video latent interval in the packed H3 sequence.The packing preserves text-before-video ordering while providing a consistent temporal alignment cue.
- 3.4 Single-Egress Routing and Low-Rank Adaptation: Single-egress routing masks unmatched action and video connections while retaining bidirectional attention among all video latent spans.Each action reaches the visual stream directly through its matched latent, after which information can propagate through video-to-video attention for continuity and scene consistency.
4 Experiments
Experiments evaluate H3-WORLD’s temporally grounded text interface through controlled interventions, held-out recorded controls, and generalization tests. The results show coordinated character and camera control, temporal responsiveness, and transfer to unseen action compositions and visual domains.
- Experimental Setup: 7,872 gameplay clips train H3-WORLD, while 128 separate clips are held out for evaluation.Each clip contains 124 frames at 24 fps and provides 37 latent-aligned action prompts.
- Pretrained Action Prior and Adaptation: H3-WORLD follows both directions of a scheduled camera reversal, producing +52.7 before and −106.0 after the switch.The reversed instruction order yields −58.7 before and +121.0 afterward, whereas global prompting misses the scheduled interval binding.
- Pretrained Action Prior and Adaptation: LoRA adaptation enables temporal bindings: H3-WORLD follows changing schedules, while frozen global prompting and zero-LoRA span-specific conditioning remain insufficient.Under constant action, global prompting and H3-WORLD yield nearly identical directional separation, 301.8 and 300.5, respectively.
- Action Controllability: Text-based H3-WORLD produces coordinated character and camera changes, unlike direct additive-bias and FiLM conditioning, which respond weakly or inconsistently.The comparison uses a held-out clip with recorded controls and the same action-conditioning interfaces.
- Action Controllability: On held-out clips, H3-WORLD follows recorded character displacement and viewpoint changes while preserving scene layout and subject appearance.The generations are conditioned on each clip’s first frame and recorded action sequence.
- Action Controllability: Changing only the action instruction produces distinct lateral motion and stronger changes for fast than slow camera commands under fixed initial conditions and sampling.The controlled comparisons include stationary, forward, strafing, and bidirectional slow and fast panning actions.
- Generalization: H3-WORLD composes unseen character-camera pairs, following both components on held-out gameplay and out-of-distribution observations.The evaluated unseen pair combines clauses observed separately during training but never jointly.
- Generalization: Across six visually diverse initial observations, the interface preserves scene layout, subject identity, and visual style while producing requested character or camera responses.The observations span first- and third-person views, indoor and outdoor scenes, and fantasy, science-fiction, and varied rendering styles.
5 Conclusion
H3-WORLD adapts MiniMax-H3 into an interactive world model using compositional language instructions aligned to latent intervals and low-rank adaptation. It supports controlled and generalized character-camera behavior, but remains limited to short, fixed-length generation without persistent state or real-time interaction.
- Conclusion: H3-WORLD combines compositional character-camera text commands, latent-interval alignment, single-egress routing, and low-rank adaptation.The design reuses control capabilities already present in the pretrained video generator.
- Conclusion: The framework supports recorded and intervened control while generalizing to unseen action pairs and visually distinct initial observations.These results establish temporally grounded language conditioning as a route from pretrained video generation to low-level world control.
- Limitations: Current evaluation focuses on short-horizon generation, with generalization assessed mainly through representative examples rather than systematic coverage.More evaluation across action combinations, scenes, and random seeds is needed to quantify control reliability.
- Limitations: The model generates fixed-length segments and lacks persistent world state, real-time interaction, planning, and policy learning.Extending the interface in these directions is identified as important for longer-horizon interactive world models.