Plan-X: Instruct Video Generation via Semantic Planning

1. Introduction

• Diffusion Transformers can produce visually plausible but semantically inconsistent results for complex instructions, including misaligned prompts, hallucinations, and incorrect action ordering.The stated difficulty includes reasoning jointly over textual guidance and visual context and planning semantic change over extended horizons.Source: §1 · p. 2
• Plan-X separates semantic reasoning from visual synthesis: a multimodal Semantic Planner produces spatio-temporal visual tokens that guide a video DiT.The tokens act as structured sketches of what happens and when, while the DiT focuses on high-fidelity, temporally consistent video generation.Source: §1 · p. 2
• This division treats semantic understanding as a language-modeling problem while retaining diffusion transformers’ strengths in realistic, temporally coherent rendering.The authors describe the combined system as capable of interpreting, planning, and executing complex video-generation tasks.Source: §1 · p. 2
• Across text-to-video, image-to-video, and video continuation, Plan-X improves prompt alignment, semantic coherence, and motion naturalness.The reported improvements cover human–object interactions, multi-stage actions, and long-horizon planning.Source: §1 · p. 2

Video Generation

• Recent video-generation work has shifted from U-Net backbones to Diffusion Transformers, while prompt enhancers expand or reformulate instructions but do not eliminate semantic misalignment on complex prompts.Existing systems are also described as lacking explicit semantic reasoning and temporal planning mechanisms.Source: §2 · pp. 2–3

Unified MultiModal Visual Generation

• TA-Tok converts SigLIP2 embeddings into discrete text-aligned tokens, creating a shared semantic space for visual and textual representations.Plan-X uses this tokenizer to represent video frames and text prompts for cross-modal reasoning.Source: §2 · p. 3
• Prior multimodal generation methods primarily target static images, whereas Plan-X connects multimodal language models and video DiTs for temporally consistent video synthesis.The overview describes the planner’s semantic keyframes guiding a DiT for high-fidelity video realization.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

• Plan-X assigns semantic intent and temporal logic to a multimodal language model, while the diffusion transformer realizes the resulting blueprint as video.The design separates these roles rather than asking one DiT to perform both semantic planning and video generation.Source: §3 · pp. 3–4

3.1. Text-Grounded Semantic Planning

• Text-grounded visual tokens provide explicit spatio-temporal structure across variable video durations, unlike global text descriptions or implicit embeddings.The paper notes that text descriptions can drift from prompts and implicit embeddings lack explicit spatio-temporal structure.Source: §3.1 · p. 4
• Plan-X encodes uniformly sampled keyframes with SigLIP2 and uses TA-Tok to quantize them into text-aligned spatial tokens.Keyframes are sampled at 2 FPS, resized to 384 × 384, and represented by 81 spatial tokens each.Source: §3.1 · p. 4
• A Qwen-2.5-Instruct planner autoregressively predicts semantic keyframes at 2 FPS from prompts and, optionally, an encoded initial frame.The unified prompt format separates static scene content, intended video dynamics, and target duration; training supports text-to-video, image-to-video, and video continuation.Source: §3.1 · pp. 4–5

3.2. Semantics-Instructed Video Synthesis

• The DiT attends to global text and spatio-temporal semantic tokens through a dedicated semantic branch, treating tokens as complementary context rather than deterministic controls.Three-dimensional time-aligned RoPE provides positional correspondence across video latents and semantic tokens with differing spatial and temporal resolutions.Source: §3.2 · p. 5
• Training first fits the new semantic branch while freezing pretrained parameters, then gradually restores text conditioning and fine-tunes with planner-generated tokens.The end-to-end stage is intended to improve adaptation to imperfect semantic predictions during inference.Source: §3.2 · p. 5

4.1. Implementation Details

• The training corpus combines 4.5M video clips from public datasets and in-house collections, including human–object interaction data, with Qwen3-VL-32B-generated unified captions.The cited interaction datasets include Taste-Rob and HOIGen-1M.Source: §4.1 · pp. 5–6

4.2. Evaluation

• The benchmark contains 50 test cases each for T2V and I2V, emphasizing human–object interactions and multi-step actions; comparisons include five video-generation baselines and a 16-participant preference study.MLLM scoring covers interaction accuracy and fidelity, motion completeness and naturalness, scene consistency, and visual quality.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.1. Implementation Details

• The default setup pairs a 7B Semantic Planner with Wan 2.2 (5B), trains them separately, then applies one epoch of joint end-to-end fine-tuning.The planner uses a 4K-token context window supporting generations of 20 seconds; diffusion inference uses 50 denoising steps.Source: §4.1 · p. 6

4.2. Evaluation

• Plan-X achieves the highest scores on nearly all MLLM-evaluated dimensions, including 0.7971 Accuracy versus Seedance’s 0.7114 and 0.8571 Completeness versus 0.7943.Plan-X-Wan also has the top Visual score of 0.9737 and Naturalness score of 0.9394; its human preference score is 0.262 versus Seedance’s 0.218.Source: Table 1 · p. 7, §4.2 · p. 7
• Qualitative examples span T2V, I2V, video continuation, and semantic transfer; Plan-X variants follow intended actions, while baselines and planner ablations show misalignment or hallucinations.The reported baseline failures include hallucinated or misplaced objects and incorrect spatial interactions.Source: Figure 3 · p. 6, Figure 4 · p. 8
• Ablations report that explicit semantic planning outperforms text-based prompt enhancement and query conditioning; larger planners improve planning, and 3D RoPE improves prompt alignment.The planner also improves prompt following across Wan 2.2 and Seedance 1.0 backbones without compromising pretrained visual fidelity.Source: §4.2 · p. 8

Conclusion

• Plan-X uses a multimodal Semantic Planner to turn instructions and visual context into spatio-temporal tokens that guide a DiT in high-fidelity video generation.The paper presents this separation as a way to improve alignment, coherence, controllability, and reduce hallucination across video-generation tasks.Source: §5 · p. 8
• The pretrained TA-Tok tokenizer has limited expressiveness for complex concepts and abstract reasoning, including mathematical or symbolic content.The paper proposes developing more expressive text-aligned visual semantic tokenizers as future work.Source: §5 · p. 8
• The authors note that the method could potentially extend to controversial applications such as fake videos and say synthesized videos should identify themselves as synthetic.Source: §5 · p. 9

Supplementary Material

• Additional implementation and training details are provided in supplementary Section 6.Source: Supplementary Material · p. 12
• Supplementary Section 7 describes the MLLM-based evaluation and user-study setup.Source: Supplementary Material · p. 12
• Supplementary Sections 8 and 9 examine end-to-end training through ablations and show representative failure cases.Source: Supplementary Material · p. 12

6.1. Semantic Planner Training

• TA-Tok represents frames at three adaptive scales, with 729 tokens for the initial frame and 81 tokens for each target frame in the assistant response.Training computes cross-entropy loss over the assistant’s TA-Tok tokens; randomly zeroing user-prompt TA-Tok embeddings with 50% probability supports both text-to-video and image-to-video capabilities.Source: §6.1 · p. 12

6.2. Semantics Guidance in DiT

• The DiT receives semantic guidance through a semantic branch whose keys and values are concatenated with the text keys and values.For Wan2.2, text and semantic attention logits are computed separately, then concatenated before softmax.Source: §6.2 · p. 12

6.3. End-to-End Training.

• The final stage jointly fine-tunes the Semantic Planner and DiT, injecting the planner’s last hidden states as semantic conditioning.The weighted objective combines diffusion loss and TA-Tok prediction loss with weights 1.0 and 0.1, respectively.Source: §6.3 · p. 12

• Gemini-2.5 evaluates generated videos across multiple dimensions.The detailed evaluation system prompt is provided in Figure 8.Source: §7 · p. 12
• Each model variant is evaluated on 20 randomly selected test samples by 16 independent human raters.Source: §7 · p. 12
• Raters choose the top two videos based on overall quality and prompt alignment.Source: §7 · p. 12
• The evaluation reports each result’s share of total top-two selections as a percentage.Source: §7 · p. 12

8. Additional Ablations

• End-to-end training improves prompt-following behavior, while retaining the DiT text-conditioning branch helps preserve pretrained visual quality.The captioned qualitative ablation associates removing text conditioning with oilpainting-like textures and reduced temporal consistency, and omitting joint fine-tuning with lower robustness to semantic planning noise.Source: §8 · p. 12, Figure 5 · p. 13
• Without joint end-to-end fine-tuning, Plan-X is less robust to semantic planning noise, including an incorrectly placed white peeler in the illustrated example.Source: Figure 5 · p. 13
• Figure 6 presents the Semantic Planner training data format.Source: Figure 6 · p. 13

9. Limitations

• Reliance on a pretrained DiT leaves Plan-X susceptible to physical-fidelity and visual-consistency artifacts.Source: §9 · p. 13
• Training on approximately 4.5M text–video pairs leaves the Semantic Planner without strong abstract reasoning and commonsense intelligence.Source: §9 · p. 13
• Illustrated failures include a spoon morphing out of a cup, disappearing red paint, and mismatches in a flag or the number of children.Source: Figure 7 · p. 14