PaperlayerView exact revision on arXiv

Source-linked AI summary

The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Text

Hanlin Wang, Hao Ouyang, Qiuyu Wang, Yue Yu, Yihao Meng, Wen Wang, Ka Leong Cheng, Shuailei Ma, Qingyan Bai, Yixuan Li, Cheng Chen, Yanhong Zeng, Xing Zhu, Yujun Shen, Qifeng Chen

arXiv:2512.16924v1cs.CV

TL;DR

Text-only prompts and existing trajectory-controlled video methods lack aspects of the control needed for complex, compositional events. WorldCanvas combines trajectories, language, and reference images to specify motion, intent, and appearance. The authors report coherent, finely controlled events that retain consistency through temporary object exits and re-entries.

  • Problem

    Text-only prompts and existing trajectory-controlled methods have limitations in specifying localized actions, trajectory timing and visibility, and reference-based appearance for complex events.

    Source: §1 · p. 2

  • Method

    WorldCanvas adapts Wan 2.2 I2V with curated trajectory-video-text data, trajectory injection, and spatial-aware cross-attention that binds phrases to trajectory regions.

    Source: §1 · p. 2

  • Results

    WorldCanvas generates complex, coherent events with fine-grained control while maintaining consistency when objects temporarily leave and re-enter the scene.

    Source: §1 · p. 2

  • Takeaways & Limitations

    WorldCanvas provides a practical interface for specifying and simulating user-defined events, a step toward interactive world models that let users shape events.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

We present WorldCanvas, a framework for promptable world events that enables rich, user-directed simulation by combining text, trajectories, and reference images. Unlike text-only approaches and existing trajectory-controlled image-to-video methods, our multimodal approach combines trajectories -- encoding motion, timing, and visibility -- with natural language for semantic intent and reference images for visual grounding of object identity, enabling the generation of coherent, controllable events that include multi-agent interactions, object entry/exit, reference-guided appearance and counterintuitive events. The resulting videos demonstrate not only temporal coherence but also emergent consistency, preserving object identity and scene despite temporary disappearance. By supporting expressive world events generation, WorldCanvas advances world models from passive predictors to interactive, user-shaped simulators. Our project page is available at: https://worldcanvas.github.io/.

1. Introduction

WorldCanvas extends promptable world events beyond text prompts by combining trajectories, reference images, and language to specify when, where, who, and what. It targets complex event control and reports coherent generation with consistency through object exits and re-entries.

1. Introduction

  • Trajectories specify motion and timing through positions and point spacing, visibility flags mark occlusion or entry/exit, reference images ground identity, and text supplies semantic intent.Together, these modalities address when, where, who, and what in an event.Source: §1 · p. 2
  • The multimodal triplet is presented as a complete, unambiguous specification for creating controllable promptable events on an interactive canvas.The introduction frames world models as moving from passive simulators toward user-directed creation.Source: §1 · p. 2
  • Existing methods lack localized text-to-trajectory links, discard trajectory timing and visibility information, and provide incomplete reference-based control.These shortcomings make complex, compositional event generation difficult.Source: §1 · p. 2
  • WorldCanvas combines multimodal data curation, entry/exit augmentation, trajectory injection, and spatial-aware cross-attention binding textual phrases to trajectory regions.The framework adapts Wan 2.2 I2V and provides an interface for entering trajectories and reference images.Source: §1 · p. 2
  • WorldCanvas generates complex, coherent events with fine-grained control while maintaining consistency when objects temporarily leave and re-enter the scene.The authors describe these results as evidence of structured world dynamics beyond surface-level generation.Source: §1 · p. 2

2. Related Work

The related work frames text-only promptable events as lacking the precision needed for complex control, then positions WorldCanvas as a multimodal extension. It also contrasts trajectory-controlled video methods with a pipeline that fuses text, trajectories, and reference images.

2.1. Promptable World Events

  • Text-only prompts can lack the spatial, temporal, and behavioral precision needed for complex events; WorldCanvas adds trajectories and reference images to textual instructions.Promptable world events are described as semantically meaningful environmental changes unfolding over time.Source: §2.1 · p. 3

2.2. Trajectory-Controlled Video Generation

  • Trajectory-controlled video methods often use global captions and coarse positional cues, while WorldCanvas combines curated multimodal data with spatial-aware weighted cross-attention in pretrained Wan.The related-work discussion also identifies limited customizability and notes Frame-In-Out's lack of joint text-trajectory-image modeling.Source: §2.2 · p. 3

3. Method

WorldCanvas builds on Wan2.2 I2V and combines curated trajectories, reference images, and motion captions with model controls for video generation.

3.1. Data Curation Pipeline

  • The curation pipeline tracks foreground and background points, filters near-static clips, and creates motion captions tied to colored object trajectories.Random cropping and transformed reference images support objects entering from off-screen and references starting at varied positions.Source: §3.1 · pp. 3–4

3.2. Video Generation with Multimodal Triplet

  • The multimodal triplet pairs each foreground trajectory’s coordinates and visibility with its initial-frame bounding box and motion-focused caption.Background points carry coordinates and visibility, without the foreground object box and caption.Source: §3.2 · p. 4

3.2.1. Trajectory Injection

  • Trajectory Injection adds Gaussian heatmaps and point VAE maps as conditional channels, guiding generation along user-specified trajectories.The point VAE map propagates the first-frame keypoint’s image VAE feature to later points on that trajectory.Source: §3.2.1 · p. 4

3.2.2. Spatial-Aware Weighted Cross-Attention

  • Spatial-Aware Weighted Cross-Attention boosts attention between a trajectory’s caption and visual tokens within its bounding-box-sized coverage area.The weighting strengthens trajectory-caption correspondence while allowing attention to other regions.Source: §3.2.2 · p. 5

3.3. Training and Inference

  • Training uses flow matching with L1 reconstruction loss; inference exposes timing, shape, visibility, text binding, and reference placement controls.Point spacing controls speed, while users can specify visible trajectory segments and adjust reference-image position and size.Source: §3.3 · p. 5

4. Experiments

WorldCanvas is evaluated against video-generation baselines for promptable event control, consistency, and trajectory–text alignment. The reported results show stronger performance across these capabilities, including qualitative alignment and persistence through off-screen intervals.

4.1. Implementation Details

  • In the implementation, the model is trained on 280k trajectory–video–text triplets for 9k steps using 64 NVIDIA H800 GPUs.Videos are processed at 480 × 832 resolution, with a learning rate of 1×10−5 and total batch size of 64.Source: §4.1 · p. 5

4.2. Comparison

  • The comparison evaluates Wan2.2 I2V, ATI, and Frame In-Out for semantic understanding, trajectory following, and reference-based generation, respectively.The experiments section frames these comparisons as tests of promptable world-event capabilities.Source: §4.2 · p. 5, §4 · p. 5

4.2.1. Qualitative Results

  • In the tested examples, WorldCanvas aligns text, trajectories, and reference images, while the baselines fail to correspond to those inputs.Wan and ATI fail to understand some intended events; Frame In-N-Out fails to preserve reference-image identity and misinterprets the event.Source: Figure 3 · p. 6, §4.2.1 · pp. 6–7
  • For multi-subject events, WorldCanvas binds each trajectory to its caption and generates the intended interactions, including objects entering from off-screen.The baselines fail to capture the full semantic intent in the off-screen-entry example.Source: §4.2.1 · p. 7, Figure 4 · p. 7

4.2.2. Quantitative Results

  • WorldCanvas consistently outperforms the baselines in trajectory following, semantic alignment, and generated-video quality.The evaluation uses 100 image–trajectory pairs and reports ObjMC, Appearance Rate, Subject & Background Consistency, and CLIP-T; the supplied passage does not include the table’s values.Source: §4.2.2 · p. 7, Table 1 · p. 7

4.3. Consistency Maintenance Capability

  • WorldCanvas maintains scene and object consistency when subjects temporarily leave the frame and later reappear.The reported examples include preserved appearance, identity, and spatial relationships across occlusions or off-screen intervals.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4. Ablation Study

  • Without spatial weighting or with hard cross-attention, captions can mismatch trajectories; the spatial-aware variant preserves their intended correspondences.The passage reports that supplementary quantitative results also show higher alignment accuracy and video quality for the proposed method.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

WorldCanvas combines trajectories, text, and reference images to generate promptable world events. The authors report support for controllable interactions and consistency across temporary disappearance, positioning the framework as a pathway toward interactive world models.

5. Conclusion

  • WorldCanvas combines trajectories, text, and reference images to support controllable, semantically grounded video simulation.The framework supports semantic actions, complex interactions, object entry and exit, and reference-guided appearance.Source: §5 · p. 8
  • The model exhibits emergent object consistency and scene memory, including when objects temporarily disappear.The authors suggest this reflects structured world dynamics beyond surface-level generation.Source: §5 · p. 8
  • The authors position WorldCanvas as a practical and scalable pathway toward interactive world models that simulate user-defined events.They describe the intended simulations as combining realism and reliability.Source: §5 · p. 8

A. Details on Quantitative Metrics

The appendix describes a 100-pair benchmark and metrics for trajectory accuracy, visibility, temporal consistency, and semantic alignment. It distinguishes global text-video similarity from local prompt-to-region alignment.

A. Details on Quantitative Metrics

  • The benchmark uses 100 image–trajectory pairs and tracks user-specified points in generated videos with CoTracker3.The resulting trajectories are evaluated against the input trajectories.Source: A. Details on Quantitative Metrics · p. 9
  • ObjMC measures mean Euclidean distance between generated and user-defined trajectories, with lower values indicating more precise motion control.Appearance Rate measures the proportion of frames where tracked points are predicted visible when the input trajectory marks them visible.Source: A. Details on Quantitative Metrics · p. 9
  • Subject & Background Consistency assesses feature-level stability across consecutive frames for both subject and background.The metric reflects temporal smoothness and visual consistency.Source: A. Details on Quantitative Metrics · p. 9
  • Global CLIP-T measures whole-video alignment with the global text description, while local CLIP-T compares cropped video patches with associated local text.Local scores are averaged across multiple window sizes.Source: A. Details on Quantitative Metrics · p. 9

B. More Quantitative Results

The appendix section presents additional quantitative experimental results, including an ablation study of Spatial-Aware Weighted Cross-Attention and a user study.

B. More Quantitative Results

B.1. Quantitative Ablation Study

The ablation compares standard full cross-attention, hard cross-attention, and Spatial-Aware Weighted Cross-Attention. On 100 image–trajectory pairs, the proposed method performs best across video-quality and semantic-alignment metrics.

B.1. Quantitative Ablation Study

  • The comparison evaluates standard full cross-attention without spatial weighting, hard cross-attention, and Spatial-Aware Weighted Cross-Attention.Source: §B.1 · p. 9
  • Spatial-Aware Weighted Cross-Attention achieves the best results across all video-quality and semantic-alignment metrics in the 100-pair evaluation.The evaluation compares standard full cross-attention, hard cross-attention, and the proposed mechanism.Source: §B.1 · p. 9, Table S1 · p. 9
  • The quantitative comparison uses an evaluation dataset of 100 image–trajectory pairs.Source: §B.1 · p. 9

B.2. Human Evaluation

The human evaluation compares WorldCanvas with three baselines across five criteria using challenging motion, interaction, and reference-image cases. WorldCanvas receives the highest share of best votes across all five criteria, while automatic metrics have stated limitations for fine motion and unstable video quality.

B.2. Human Evaluation

  • CLIP-T is not particularly accurate for fine-grained motion, and unstable Frame In-N-Out outputs can cause inaccurate CoTracker3 tracking.The resulting ObjMC and Appearance Rate scores can favor Wan2.2 I2V [32], which does not provide trajectory control, over Frame In-N-Out [34].Source: §B.2 · p. 9
  • Fifteen participants evaluated videos from WorldCanvas, Wan 2.2 I2V 14B [32], ATI [33], and Frame In-N-Out [34].The cases included 20 challenging motion or interaction examples and 10 reference-image cases, used only for reference-fidelity evaluation.Source: §B.2 · p. 9
  • Participants judged trajectory following, prompt adherence, text–trajectory alignment, reference fidelity, and overall video quality.Source: §B.2 · p. 10
  • WorldCanvas receives the highest percentage of best votes across all five human-evaluation criteria.Table S2 reports the percentage of best votes received by each model.Source: §B.2 · p. 10, Table S2 · p. 10

C. World-Model Related Capabilities

WorldCanvas is assessed for physical plausibility, causal reasoning, and prediction by prompting causes without specifying their effects. Additional examples test whether it can render counterfactual events while maintaining causal and semantic consistency.

C. World-Model Related Capabilities

D. Counterfactual generation

  • Counterfactual examples include a shark leaping from sand and a flying dog whose leash snaps while its owner remains on the ground.The authors report that these generated events retain causal relationships and semantic fidelity despite their impossible premises.Source: D. Counterfactual generation · p. 10, Figure S2 · p. 11

E. Failure cases

WorldCanvas sometimes fails in particularly challenging scenarios involving complex spatial transformations or logical reasoning. The examples expose blur and inconsistency during a camera rotation, and insufficient water-level increase after an out-of-view interval.

E. Failure cases

  • The failures occur in particularly challenging scenarios involving complex spatial transformations or logical reasoning.Source: E. Failure cases · p. 11
  • During a requested 360-degree camera rotation back to the original view, books on the shelf become blurry and inconsistent.The figure caption identifies blurring and distortion in case (a).Source: E. Failure cases · p. 11, Figure S3 · p. 12
  • After a camera pans away from a filling cup and returns, the water level shows no significant increase despite the expected continued filling.The caption identifies insufficient water-level rise as one of the illustrated failure cases.Source: E. Failure cases · p. 12, Figure S3 · p. 12
  • The examples point to challenges in maintaining consistency under drastic motion and reasoning about content that is out of view.Source: E. Failure cases · p. 12
Loading 2512.16924v1…