WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

• Open-loop generation cannot adapt to visual progress, while closed-loop systems with frozen executors may fail to realize correct actions; errors in long-horizon procedures can affect subsequent steps.Source: §1 · pp. 1–2
• Closed-loop training requires planning from generated visual state, predicting completion from progress, maintaining bounded visual context, and step-level supervision pairing atomic actions with visual outcomes.Source: §1 · p. 2
• WorldGuide trains its ContextPlanner and Executor on the same demonstrations, with the Executor conditioned on planner action embeddings, and introduces a benchmark of approximately 59K procedural videos.The benchmark covers 245 tasks and 27 categories, with temporally grounded atomic-action clips and explicit completion signals.Source: §1 · p. 2
• Closed-loop execution improves Task Success by 21.62% over open-loop; visual feedback improves Task Success by 18.61% and Plan Success by 5.34% while reducing Repeat/Skip by 4.46%.Source: §1 · p. 3
• Memory provides gains of 5.77% on WorldGuide Bench and 17.91% on Video-CraftBench, while WorldGuide achieves the highest Aesthetic Quality and Overall Consistency on WorldGuide Bench.Source: §1 · p. 3

2.1 Video Generation and World Models

• Existing video generators and world models address visual synthesis, memory, or decision-making, but none select actions from visual progress, directly supervise video execution, and decide completion within one closed loop.Source: §2.1 · p. 3

2.2 Planner–Executor Video Generation

• Planner–executor systems vary in their use of latent planning, fixed temporal spans, simulation, retries, or critics; WorldGuide learns action selection, atomic-action execution, and completion from the same demonstrations.Source: §2.2 · p. 3

3.1 Overview and Problem Formulation

• WorldGuide repeatedly predicts an atomic action or DONE from the goal, current visual state, and history, then generates the next clip or terminates.The new clip becomes the next visual state and is added to the history; rollout also stops when the step budget is reached.Source: §3.1 · pp. 3–4

3.2 ContextPlanner

• The ContextPlanner uses the latest three clip–action pairs to predict one next-action instruction or completion token from observed progress.It is trained by supervised fine-tuning on demonstration histories and next-action or completion targets.Source: §3.2 · p. 4

3.3 Executor

• The Executor is trained on ground-truth action captions and target clips, then uses the planner’s predicted action through the same embedding pathway at inference.Executor training updates its parameters while the ContextPlanner remains frozen.Source: §3.3 · pp. 4–5

3.4 Hierarchical Memory Compression

• Hierarchical memory gives recent latent frames finer spatial embeddings than older ones, bounding the Executor’s history-token count.The memory includes an oldest-frame anchor; its capacity is an upper limit rather than the length used at every step.Source: §3.4 · p. 5

3.5 Dataset for WorldGuide Training

• WorldGuide Bench combines ordered atomic-action clips, task goals, and explicit completion signals; it contains 58,679 videos across 245 tasks and 27 procedural categories.Gemini 2.5 Flash was used to filter web instructional videos and annotate actions and completion.Source: §3.5 · p. 6

4.1 Experimental Setup

• The setup fine-tunes Qwen2.5-VL-7B-Instruct as ContextPlanner and HunyuanVideo-1.5 as Executor, using K=3 recent clip-action pairs and evaluating 980 WorldGuide Bench and 294 Video-CraftBench samples.Video-CraftBench models receive the initial image and goal without reference actions.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 Does Closed-Loop Execution Improve Procedural Task Completion?

• 33.33% Task Success exceeds MiniMax-H3’s 29.90% on WorldGuide Bench despite reference actions, and 47.69% exceeds 32.73% on goal-only Video-CraftBench; WorldGuide also outperforms closed-loop peers.The MiniMax-H3 comparison is reported in the respective benchmark settings.Source: §4.2 · p. 8, Table 3 · p. 8
• 33.33% Task Success and 62.85 Plan Accuracy with closed-loop execution exceed the open-loop variant’s 11.71% and 22.87, respectively.Open-loop generation fixes the action sequence and rollout length before execution; Table 4 reports the ablation.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 Does Generated Visual Feedback Improve Replanning?

• Restoring generated visual feedback raises Task Success from 14.72% to 33.33% and Plan Accuracy from 29.29 to 62.85 compared with action-history-only planning.Action history alone cannot reveal whether an action completed, was partially realized, or drifted visually.Source: §4.3 · pp. 8–9, Table 4 · p. 9

4.4 Can WorldGuide Recognize Completion and Stop?

• Visual feedback raises autonomous Plan Success from 86.41% to 91.75% and reduces Repeat/Skip from 20.19 to 15.73; at K=3, ground-truth histories yield 79.66 Plan Accuracy versus 64.03 autonomously.Plan Success measures whether the predicted plan would achieve the goal if executed correctly, while video-judged Task Success is separate.Source: §4.4 · p. 9, Table 5 · p. 9

4.5 How Do Planning and Execution Failures Interact?

• Oracle reference actions add +5.49% Task Success; memory adds +5.77% on WorldGuide Bench, +17.91% on Video-CraftBench, and +7.49% video-judged Plan Accuracy.The oracle uses the same trained Executor and memory; the memory comparison keeps ContextPlanner and Executor checkpoints unchanged.Source: §4.5 · pp. 9–10

4.6 Does Procedural Improvement Compromise Visual Quality?

• On Video-CraftBench, 47.69% Task Success accompanies lower Imaging Quality (64.63 vs. 69.65) and Overall Consistency (22.09 vs. 23.18) than MiniMax-H3; WorldGuide leads Aesthetic Quality (49.36) and Overall Consistency (31.13) on WorldGuide Bench.TempAct and SkyReels score higher on some individual low-level visual metrics on WorldGuide Bench.Source: §4.6 · p. 10

4.7 Qualitative Results

• In the fried-rice example, WorldGuide orders aromatics, vegetables, and rice before seasoning and mixing, while MiniMax repeats rice and changes cookware and other models stall or lose scene consistency.The comparison samples frames uniformly from the trajectories.Source: §4.7 · p. 10, Figure 5 · p. 10

• WorldGuide substantially improves task completion, though a remaining performance gap indicates room for more robust long-horizon execution.Source: §5 · p. 10
• Premature or redundant planner actions and clips that incompletely realize state transitions are typical failures, and their effects can propagate into later decisions.Source: §5 · pp. 10–11
• Future work could add richer state-validity signals and explicit recovery, jointly optimize Planner and Executor, and selectively retain task-relevant memory.Selective retention is proposed as a way to improve long-horizon consistency while keeping memory efficient.Source: §5 · p. 11

• WorldGuide achieves the highest Task Success among evaluated video generators and world models on both WorldGuide Bench and Video-CraftBench.Source: §6 · p. 11
• The method repeatedly selects a semantic action, renders its consequence, and uses the resulting visual state to guide subsequent execution.Source: §6 · p. 11
• WorldGuide Bench comprises approximately 59K annotated videos across 245 tasks and 27 procedural categories.Source: §6 · p. 11

A Design Rationale

• The ContextPlanner shares Qwen2.5-VL-7B-Instruct initialization with HunyuanVideo-1.5’s prompt encoder, replacing a separate planning network and preserving the pretrained action-embedding pathway.The supplement also motivates HunyuanVideo-1.5 as a compact Executor starting point with strong procedural execution among baselines.Source: A Design Rationale · p. 16
• The Planner is trained first and frozen during Executor fine-tuning to preserve readable natural-language actions, which support interpretable guidance, separate diagnosis, and independent text-based evaluation.Jointly optimizing both modules while preserving clear language outputs is left as future work.Source: A Design Rationale · p. 16

B WorldGuide Bench

• WorldGuide Bench adds task-completion supervision to step-level atomic-action clips and task goals, which existing procedural video datasets do not provide.The supplement says existing datasets instead target planning QA, localization or grounding, segmentation or recognition, or video generation and evaluation.Source: B WorldGuide Bench · pp. 16–17

B.1 Collection and Filtering

• Filtering retained 58,679 of 133,360 retrieved videos (44.0%) across 245 tasks and 27 categories, after Gemini 2.5 Flash assigned each video a score of 5.Videos were sourced from public instructional material and retrieval used task-specific queries expanded with short-form and multilingual variants in 15 languages.Source: §B.1 · p. 17

B.2 Temporal Action Annotation; B.3 Training Samples

• Videos are annotated as timestamped atomic actions, and step records train the Planner to predict the next caption or completion and the Executor to generate clips from captions and visual context.Action durations target 1–5 seconds; each planner record can include up to K=3 preceding clip–caption pairs, and an extra post-final record supervises stopping without a target clip.Source: §B.2 · p. 17, §B.3 · p. 17

B.4 Statistics and Evaluation Split; B.5 Annotation Example

• Holding out four videos per task yields 980 test and 57,699 training videos, evaluating unseen demonstrations of known tasks; repeated actions in examples show why progress requires visual state.The origami example contains 21 aligned clip–caption pairs, with completion supervision after the final step.Source: §B.4 · p. 17, §B.5 · p. 19

B.6 Human Audit of Annotations

• The human audit reviews 245 task-balanced videos containing 3,168 action steps, using six visual-evidence criteria and excluding “Cannot determine” ratings from correctness scores.Three independent annotators rate each video; step-level scores are averaged within and then across videos so each task contributes equally, with majority-vote scores also reported.Source: §B.6 · pp. 19–20

C Hierarchical Memory; C.1 History and Reference Conditioning

• Hierarchical memory keeps recent latent frames at higher spatial resolution and progressively compresses older ones, bounding memory-token count as video grows; a separate reference slot holds the latest frame.The history includes an anchor and up to 1,365 recent latent frames, while the reference can duplicate the newest history frame in separate token positions.Source: C Hierarchical Memory · p. 20, §C.1 · p. 21

C.1 History and Reference Conditioning

• History concatenates recent latent frames from previous clips, is updated after each generated clip, and is left-padded when shorter than required; the reference frame occupies a reserved conditioning slot.The visual conditioning contains history followed by the reference frame, and the memory branch is skipped when history is absent.Source: §C.1 · p. 21

C.2 Multi-Scale Embedding and Injection

• Multi-scale spatial compression adds history as conditioning tokens without increasing denoised video tokens or changing the training objective, while preserving all temporal latent frames.The coarsest compression uses an effective spatial stride of 128; temporal kernel size and stride remain one.Source: §C.2 · p. 21

C.3 Compression Schedule and Token Budget

• History, anchor, and reference cost at most eight full-resolution latent-frame equivalents, with context-token count bounded independently of rollout length for fixed history capacity and resolution.The compression schedule varies with history length; spatial padding can exceed nominal counts when dimensions are not divisible by strides.Source: §C.3 · p. 22

D Training and Inference

• The ContextPlanner is trained first, then the Executor is trained with the ContextPlanner frozen.Source: D Training and Inference · p. 22

D.1 ContextPlanner Training

• Qwen2.5-VL-7B-Instruct [3] is fine-tuned to predict the next action or completion from the goal, visual state, and up to three preceding clip–action pairs.Cross-entropy loss applies only to response tokens.Source: §D.1 · p. 22

D.2 Executor Training

• The Executor is fine-tuned on HunyuanVideo-1.5 with precomputed VAE latents, frozen ContextPlanner action embeddings, and jointly trained multi-scale memory embedders.Source: §D.2 · p. 22
• The ContextPlanner training configuration fine-tunes Qwen2.5-VL-7B-Instruct to predict next-step actions from the goal, visual state, and three clip–action history pairs.Source: Table 11 · p. 23

D.3 Rollout and Termination

• Rollouts stop immediately when the planner emits the completion token; otherwise, generation is capped at 80 steps, above the benchmark’s longest 62-step reference procedure.Source: §D.3 · p. 23
• Sampling uses 10 denoising steps, guidance scale 7.5, and flow shift 5.0; training clips use 24 FPS and generated videos play at 16 FPS.Source: §D.3 · p. 23

E.1 Benchmarks, Baselines, and Ablations

• Comparisons span video generators, video world models, and planner–executor methods; WorldGuide and starred systems use goal-only planning, unlike unstarred reference-plan baselines.Video-CraftBench gives every model only the initial image and task goal, accepting different valid procedures and step granularities.Source: §E.1 · pp. 23–24
• WorldGuide ranked first in the independent human study, followed by MiniMax-H3.The study is cited as a validation of the automatic video-judge evaluation.Source: §E.1 · p. 25

E.2 Video-Judge Metrics

• The video judge scores visible Plan Accuracy, Order, Repeat/Skip, and Task Success from generated video, task description, and reference plan, never ground-truth video.Equivalent actions, different step granularities, and valid reorderings of independent steps receive credit.Source: §E.2 · pp. 25–26, Figure 7 · p. 26

E.3 Planner Evaluation

• Text-plan evaluation uses GPT-5.2 without video and accepts equivalent actions, different granularities, and valid action reordering.Plan Success assesses whether the predicted sequence would achieve the goal if executed correctly, whereas visual stopping is assessed separately.Source: §E.3 · pp. 26–27

E.4 Paired Alignment

• Paired Alignment samples both videos at 16 relative positions and combines appearance, state, and temporal similarity with weights 0.40, 0.30, and 0.30.The measure complements rather than replaces task metrics.Source: §E.4 · pp. 27–28

E.5 VBench Video-Quality Metrics

• VBench reports perceptual and temporal video-quality measures, none of which assesses procedural correctness.The five dimensions use the official implementation and are reported on a 0–100 scale, with higher scores better.Source: §E.5 · p. 28, Table 14 · p. 28

F Human Evaluation of Generated Videos

• Evaluators scored whether necessary actions appeared in sensible order, essential steps were completed, and the final goal was visibly achieved.Each video received a task-completion score from 1 to 5.Source: F Human Evaluation of Generated Videos · p. 28
• 2.750 average task-completion score for WorldGuide led MiniMax-H3 at 2.542 and LTX-2.5 at 2.472.Four evaluators rated 162 videos across 18 task cases; the study is presented as independent validation of automatic video-judge results.Source: F Human Evaluation of Generated Videos · p. 28, Table 15 · p. 29

G.1 WorldGuide Qualitative Comparisons

• Qualitative comparisons show WorldGuide progressing through assembly and preparation steps, while baselines sometimes substitute objects or omit visible fillings.TempAct changes supplied blocks and folds a tortilla without visible fillings; Bernini introduces floating blocks, and PhysAgent shifts to stacking round objects.Source: §G.1 · p. 29

G.2 Video-CraftBench Results

• In the block-horse example, WorldGuide adds a yellow bar and orange cylinder to the starting arch, while MiniMax-H3 introduces shaped toy pieces and LTX-2.5 substitutes different blocks.All compared models receive the initial image and task goal without reference action captions.Source: §G.2 · p. 29
• WorldGuide shows successive folds in the airplane and boat examples, but late paper appearance and scene texture deteriorate; TempAct substitutes white aircraft- and boat-shaped objects.The benchmark judge accepts different valid procedures and step granularities.Source: §G.2 · p. 29

H Limitations and Failure Analysis

• A planner may repeat an action, skip a prerequisite, or stop too early, while the executor may distort geometry or appearance even when given a correct instruction.Because each generated clip becomes the next planning observation, replanning can respond to errors but cannot undo an incorrect transition.Source: H Limitations and Failure Analysis · p. 29
• The modules are trained sequentially without a joint task-completion objective, and compressed older frames can discard details needed later.Enabling memory also increases repetition.Source: H Limitations and Failure Analysis · p. 29
• Outcome verification, explicit recovery from failed steps, and joint training are identified as next steps.Folding examples show correct action progression alongside late visual degradation.Source: H Limitations and Failure Analysis · p. 29