PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring MLLM-Diffusion Information Transfer with MetaCanvas

Han Lin, Xichen Pan, Ziqi Huang, Ji Hou, Jialiang Wang, Weifeng Chen, Zecheng He, Felix Juefei-Xu, Junzhe Sun, Zhipeng Fan, Ali Thabet, Mohit Bansal, Chu Wang

arXiv:2512.11464v1cs.CVcs.AIcs.LG

TL;DR

MLLMs can interpret structured visual content, but diffusion generation systems struggle with similarly precise spatial, attribute, and knowledge-intensive control. MetaCanvas uses MLLM-planned latent canvases to guide diffusion, and reports gains across diverse image and video tasks. Its results support latent-space planning as a promising direction for transferring MLLM capabilities into generation.

  • Problem

    In visual generation, MLLMs are often used as global conditions, limiting fine-grained spatial and temporal control despite their ability to interpret structured visual content.

    Source: §1 · pp. 1–2

  • Method

    MetaCanvas appends learnable multidimensional canvas tokens to MLLM inputs and uses a lightweight connector to inject their embeddings patch-wise into diffusion latents.

    Source: §1 · p. 2

  • Results

    MetaCanvas reports gains across image and video tasks, including improved video-editing accuracy over baselines while maintaining comparable video quality and achieving the highest human preference across evaluation dimensions.

    Source: §1 · p. 2, §5.3 · p. 9

  • Takeaways & Limitations

    Across three backbones and six diverse tasks, MetaCanvas delivers strong performance while maintaining training efficiency relative to prior architectures.

    Source: §6 · p. 11

    Training-data quality is not optimal and data scale is limited for some tasks; success is not high for in-context video generation from three or more reference images.

    Source: K Limitations and Future Works · p. 30

Abstract

from arXiv · show

Multimodal learning has rapidly advanced visual understanding, largely via multimodal large language models (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation.

1 Introduction

MetaCanvas addresses the gap between MLLMs’ structured visual understanding and diffusion systems’ difficulty with precise spatial, temporal, and knowledge-intensive generation. It uses MLLM-planned latent canvases to guide diffusion and reports gains across image and video tasks.

1 Introduction

  • Global conditioning constrains fine-grained spatial and temporal control, while MLLMs can interpret structured visual content that generation systems struggle to produce.The paper motivates investigating whether MLLMs can plan directly in spatial and spatiotemporal latent spaces.Source: §1 · pp. 1–2
  • MetaCanvas appends learnable multidimensional canvas tokens to the MLLM input and uses a lightweight connector to inject their embeddings patch by patch into diffusion latents.Zero initialization keeps the injected signals from changing diffusion inputs at the start of training.Source: §1 · p. 2, Figure 1 · p. 3
  • MetaCanvas improves video-editing accuracy over baselines while maintaining comparable video-quality scores, and earns the highest human preference across evaluation dimensions.It also reports competitive or superior performance on in-context video generation and preserves original video-generation capabilities.Source: §1 · p. 2
  • Multidimensional canvas queries let MLLMs plan object layouts and temporal evolution in latent space, supporting structural fidelity and fine-grained editing control.The canvas tokens are described as implicit visual sketches for guiding generation.Source: §1 · p. 2
  • Across six image and video tasks and three diffusion backbones, MetaCanvas surpasses baselines without canvas tokens and matches state-of-the-art performance while preserving MLLM understanding.The framework also maintains training efficiency relative to prior architectures.Source: §1 · p. 3

2 Related Work

Prior interfaces range from text prompts to dense MLLM embeddings, but both leave limitations in the control available to diffusion generators. Related work motivates structured spatial or spatiotemporal latent plans as a more direct interface.

2 Related Work

  • Text-only signals provide limited guidance for dense layouts, precise attribute bindings, and long-range temporal structure in complex scenes.Prior methods use LLM-generated captions, layouts, or scripts to condition diffusion models.Source: §2 · p. 3
  • Dense MLLM embeddings are more expressive than text and largely preserve reasoning ability, but compress geometry, layout, and motion into a single 1D sequence.Such embeddings may be autoregressively predicted visual representations or continuous query-token embeddings.Source: §2 · p. 3
  • Because sequence-level embeddings provide no explicit spatial or temporal handles, related work motivates a latent canvas for patch-wise plans to the diffusion generator.Source: §2 · p. 3

3 MetaCanvas

MetaCanvas connects pretrained MLLMs to diffusion models using learnable multidimensional canvas tokens and a lightweight connector. The design supports structured spatial and temporal guidance across image and video generation tasks.

3 MetaCanvas

  • MetaCanvas uses learnable multidimensional canvas tokens as spatiotemporal priors and a lightweight connector to bridge pretrained MLLMs and diffusion models.The paper frames the approach as a unified multimodal learning framework for improving information transfer with efficient training.Source: §3 · p. 3

3.1 Preliminaries

  • Canvas tokens appended to the MLLM’s multimodal input are processed jointly, and their output embeddings serve as structural visual-semantic priors added to noisy latents.Source: §3.1 · p. 4

3.2 Overall Framework for MetaCanvas

  • The framework sends MLLM context embeddings through an MLP for DiT conditioning, while canvas embeddings are fused with noisy latents through a Canvas Connector.Images and videos are encoded by both the MLLM visual encoder and the diffusion model’s VAE; training keeps the MLLM frozen, with optional LoRA.Source: §3.2 · p. 4

3.3 Multi-dimensional Canvas Tokens

  • For images, canvas tokens adapt spatially to resolution; for videos, sparse keyframe tokens are interpolated across latent frames to provide temporal information compactly.For video tasks, reference and condition latents are concatenated with noisy latents, and keyframe canvases are not applied to reference frames.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 MetaCanvas Connector Design

  • The connector aligns canvas tokens to DiT latent space, fuses them after patchification, and processes the fused tokens with an efficient DiT block; zero-initialized projections make the initial canvas residual null.For video, canvas tokens modify noisy latent frames but not reference frames; AdaLN modulates their influence across timesteps.Source: Figure 2 · p. 4, §3.4 · p. 5

4 Experiment Setup

The experiments use task-specific training configurations, including a three-stage video training procedure. Evaluation covers image generation and editing, video generation and editing, and reference-guided in-context video generation.

4.1 Training Setup

  • Exploratory text-to-image training pairs frozen Qwen2.5-VL-3B with SANA-1.6B, retaining T5 text conditioning while training connectors, canvas or query tokens, and the DiT on BLIP3o-60k.Source: §4.1 · p. 6
  • Image editing starts from UniWorld-V1’s MLP connector, finetunes the diffusion visual branch with canvas tokens and a transformer connector, and trains on O(1M) editing samples.The MLLM uses rank-64 LoRA.Source: §4.1 · p. 6
  • Video generation and editing replace Wan2.2’s default T5 encoder with Qwen2.5-VL-7B and use three training stages, from connector alignment to joint image-video multitask training.The stages use O(40M) image-text pairs, then O(3M) video-text pairs at a 4:1 video-to-image ratio, before unfreezing diffusion parameters and adding MLLM LoRA.Source: §4.1 · p. 6

4.2 Evaluation Setup

  • Evaluation uses GenEval for exploratory text-to-image, GEdit and ImgEdit for image editing, and VBench for text-to-video and image-to-video tasks.Video editing uses a curated balanced set of 300 prompts; in-context reference image-to-video is evaluated with OmniContext-Video across single-ID and multi-ID cases.Source: §4.2 · pp. 6–7

5 Results and Discussion

Across image and video tasks, MetaCanvas improves image-editing benchmark results and achieves competitive video-generation performance while adding video-editing capabilities. Video editing retains comparable quality scores while outperforming baselines in editing accuracy, and the experiments also examine T2I guidance, connector design, and video keyframes.

5.1 Exploratory Experiments on T2I Generation

  • In exploratory T2I experiments, combining global text guidance with MetaCanvas as a visual prior yields consistent gains and the fastest GenEval convergence among tested variants.Even without text conditioning, 2D canvas tokens added to noisy DiT latents provide meaningful structural guidance.Source: §5.1 · pp. 7–8
  • Connector ablations find that timestep conditioning enables dynamic canvas influence, while DiT and transformer blocks fuse canvas information with latents.Avoiding early projection into the low-dimensional VAE space yields additional gains; three keyframes outperform using more, with only a 3.1% increase in training-step time.Source: §5.1 · p. 8, Table 3 · p. 8, §5.3 · pp. 9–10

5.2 Results on Image Editing Task

  • FLUX.1-Kontext [Dev] with MetaCanvas improves results on both ImgEdit-Bench and GEdit-Bench, with steady gains during training.The comparison adds only lightweight connector modules, with minimal parameter and computational overhead.Source: §5.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8

5.3 Results on Video Generation and Editing Tasks

  • Video-generation performance is comparable to open-source models, while MetaCanvas also provides video-editing capabilities.The comparison includes CogVideoX-5B, HunyuanVideo, Wan2.1-14B, and Wan2.2-5B.Source: §5.3 · p. 9, Table 4 · p. 9
  • Video editing achieves comparable quality scores while outperforming all baselines in editing accuracy and earning the highest human preference across evaluated dimensions.The comparison uses VBench and GPT-4o quality measures; the human evaluation covers editing accuracy, spatio-temporal consistency, and overall user preference.Source: §5.3 · p. 9, Table 5 · p. 9
  • MetaCanvas performs competitively with Wan-VACE on in-context video generation, particularly for human-object interaction tasks.The task generates videos from reference images.Source: §5.3 · p. 10, Table 7 · p. 11, Figure 8 · p. 11

6 Conclusion

MetaCanvas bridges MLLMs and diffusion models with multidimensional canvas tokens and a lightweight connector, and experiments span three backbones and six image and video tasks. The related-work discussion contrasts this latent-space interface with existing approaches, while the preliminaries describe the flow-matching process used for training and inference.

6 Conclusion

  • Across three backbones and six tasks, MetaCanvas delivers strong performance while maintaining training efficiency relative to prior architectures.Its multidimensional canvas tokens act as spatiotemporal priors that fuse multimodal input conditions and guide media synthesis.Source: §6 · p. 11

A.1 Extended Related Works

  • Unlike native unified generation approaches that require substantial resources to learn generation from scratch, MetaCanvas bridges existing MLLMs and diffusion generators without extensive retraining.The related-work discussion also describes the potential higher upper bound of end-to-end designs.Source: §A.1 · p. 18
  • MetaCanvas injects learnable multidimensional canvas tokens into diffusion latents, aiming to encode spatial-temporal structure without the challenges of 1D token conditioning.The related-work discussion notes that 1D tokens make structural priors difficult to encode and can require large connectors and substantial data.Source: §A.1 · p. 18

A.2 Extended Preliminaries

  • The preliminaries define a rectified-flow latent path from data toward noise, train the model to match its target velocity, and use denoising inference from t = 1 to t = 0.MetaCanvas uses the same flow-matching objective without extra auxiliary losses and retains the same denoising strategy at inference.Source: §A.2 · p. 19

B.1 Architecture Details of Other Design Choices

The appendix reports exploratory canvas-conditioning findings, setup details, a controlled query-based comparison, and additional ablations. It also documents the connector design and training configuration used in those experiments.

B.1 Architecture Details of Other Design Choices

  • Canvas tokens transfer MLLM information into DiT; combining them with text adds gains and converges faster than the tested alternatives, including MetaQuery.The exploratory setup retains SANA’s T5 text encoder rather than replacing it with MLLM text embeddings.Source: §B.1 · pp. 19–20

B.2 Ablation for Canvas Connector Design

  • Table 3 quantifies the connector-component ablation, while Figure 10 visualizes its architecture variants; the appendix omits further illustration as redundant.The quantitative component results are reported in Table 3 of the main paper.Source: Figure 10 · p. 20, §B.2 · p. 21

B.3 Experiment Setup Details

  • The exploratory runs freeze the MLLM, train DiT from scratch, and use 256 canvas tokens for 16 × 16 latents; training takes about one day for 20k steps on one A100 node.These settings apply to the Figure 4 and Table 3 experiments.Source: §B.3 · p. 21

B.4 Additional Comparison with Query-Based Architecture

  • Under matched Qwen2.5-VL-7B/SANA and BLIP3o-60k training, MetaCanvas outperforms MetaQuery and BLIP-3o on GenEval.On uncommon color combinations, unusual spatial relationships, and counts above three, MetaCanvas succeeds where BLIP-3o fails.Source: §B.4 · pp. 21–22, Table 8 · p. 22

B.5 Additional Ablation Results

  • Rank-32 MLLM LoRA raises GenEval from 0.86 to 0.87; multimodal RoPE slightly improves results at no additional computational cost.Without multimodal RoPE, results are slightly worse, although both variants outperform MetaQuery and BLIP-3o.Source: §B.5 · p. 22

C Image Editing Task

This appendix details image-editing setup and video-task connector, conditioning, and data choices. It also reports a 2D-versus-3D canvas comparison, including an early-frame flickering issue and its mitigation.

C Image Editing Task

  • Image editing uses FLUX.1-Kontext-Dev with Qwen2.5-VL-7B; training unfreezes the visual branch, canvas tokens, and connector, with rank-64 MLLM LoRA.Training uses about 1M OmniEdit and HQEdit image-editing samples; the appendix identifies GEdit-EN-full and ImgEdit evaluation tables.Source: C Image Editing Task · pp. 23–24, Table 11 · p. 23, Table 12 · p. 24

D.1 Context and Canvas Connectors Design

  • The video canvas connector uses a transformer block to align MLLM embeddings with DiT features, then a DiT block fuses them with latents through AdaLN.It is applied per frame, reducing GPU memory consumption.Source: §D.1 · p. 25

D.2 Interface Design for Canvas Keyframe and Reference/Condition Frames Injection

  • The video interface combines canvas embeddings with latents after patchification and supports reference-frame and source-video conditioning across four video tasks.The tasks include text-to-video, image-to-video, video editing, and reference-to-video generation.Source: §D.2 · p. 25

D.3 2D canvas vs. 3D canvas

  • 2D canvas achieves comparable optical-flow-error and PSNR alignment and the best GPT-4o score, but three 3D keyframes alleviate early-frame flickering.The authors attribute the flickering to inconsistencies in Wan2.2 VAE’s initial latent frames and note that the current design may remain suboptimal.Source: §D.3 · pp. 25–26

E Text/Image-to-Video Generation Tasks

  • For image-to-video generation, the clean first-image latent replaces the first noisy latent frame and receives timestep zero to preserve input-image information.The remaining latent frames receive noise according to their corresponding timesteps.Source: E Text/Image-to-Video Generation Tasks · p. 26

F.1 Training dataset details.

  • The appendix constructs video-editing samples from raw videos or paired image-editing data, using generated edited first frames and depth frames to produce edited videos.The two data-generation routes use Qwen-Image-Edit and depth ControlNet, with Wan2.2-5B animating the image-based route.Source: §F.1 · p. 26

F.2 Evaluation benchmark details.

The benchmark materials describe a curated video-editing evaluation set and complementary automatic and human assessments. They also detail a pipeline for constructing 70k in-context video training samples from reference images.

F.2 Evaluation benchmark details.

  • The evaluation set contains 300 prompts for high-resolution, long videos, balanced across local object editing, background changes, and style transfer.Videos are sampled from 3k held-out in-house videos; GPT-4o generates and filters prompts, followed by manual verification, and half the videos contain people.Source: §F.2 · p. 26
  • VACE-14B tends to over-copy reference images into generated videos, making user-instruction following harder than with MetaCanvas.Table 15 also includes detailed prompt-following and subject-consistency scores alongside the overall score.Source: Table 15 · p. 27
  • Evaluation combines VBench quality scores, GPT-4o semantic and quality scoring on frames sampled at 1 FPS, and blind human comparisons of MetaCanvas against two strong baselines.VBench averages subject and background consistency, motion smoothness, aesthetic quality, dynamic degree, and imaging quality; 50 evaluators judge editing accuracy and consistency.Source: §F.2 · pp. 26–27

G.1 Training Dataset Details

  • For in-context video training data, Qwen2.5-VL extracts dominant objects, characters, and background scenes from each video's first frame, and Qwen-Image-Edit generates reference images.Source: §G.1 · p. 27
  • To counter the white backgrounds often produced for extracted objects, Qwen2.5-VL imagines natural backgrounds and Qwen-Image-Edit generates new edited references, yielding 70k training samples.Source: §G.1 · p. 27

G.2 Evaluation Benchmark Details

The appendix adapts OmniContext’s image benchmark into OmniContext-Video for reference-conditioned video generation. It retains the original prompts, references, and evaluation dimensions while evaluating generated videos.

G.2 Evaluation Benchmark Details

  • OmniContext-Video adapts OmniContext’s prompts and reference images to video generation, preserving its evaluation protocol for prompt-following and subject-consistency.Generated video is sampled at 1 FPS for GPT-4.1 evaluation; the appendix provides an overall score and separate dimension scores.Source: §G.2 · pp. 27–28

H Video Tasks Training Details

Video training first aligns Qwen2.5-VL-7B with Wan2.2-5B on basic tasks, then adds canvas components for multi-task training. The authors report cross-task benefits from the third stage.

H Video Tasks Training Details

  • Figure 15 visualizes the trainable components in each stage, while Table 16 summarizes stage-wise training hyperparameters, model parameters, and data-sampling ratios.The overview is provided to support reproducibility.Source: H Video Tasks Training Details · p. 28
  • Stages 1 and 2 align Qwen2.5-VL-7B with Wan2.2-5B on text-to-image generation, image reconstruction, and text-to-video generation, without canvas tokens or connector training.The authors allocate relatively large compute budgets to these stages, saying sufficient alignment training is crucial for capturing fine-grained prompt information in Stage 3.Source: H Video Tasks Training Details · p. 28
  • Stage 3 multi-task training adds model-specific canvas components, takes 10k steps, and is reported to provide mutual benefits across tasks.The authors cite improved style understanding as an example, which they suggest may come from image and video style-editing data.Source: H Video Tasks Training Details · p. 28

I Visualizations

The appendix visualizes canvas features and task outputs, reporting improved prompt interpretation and grounding across video tasks. It also points readers to the simplified canvas-connector implementation.

I Visualizations

  • Joint training on diverse tasks, including video global-style editing, is reported to improve MetaCanvas’s understanding of stylistic variations in text-to-video prompts.The examples focus on VBench’s appearance-style category.Source: I Visualizations · p. 29
  • On VBench camera-motion examples, MetaCanvas interprets image-to-video instructions more effectively than the base Wan2.2-5B model.The appendix points to Table 14 for expanded results corresponding to main-paper Table 4.Source: I Visualizations · p. 29
  • Across video-editing visualizations, MetaCanvas more precisely grounds target objects and backgrounds than other methods, and matches Ditto on global-style editing where Lucy-Edit-Dev-v1.1 fails.The authors report these comparisons in Figures 20–24.Source: I Visualizations · pp. 29–30
  • For in-context video generation, MetaCanvas better follows prompts and composes reference images into scenes, while VACE can copy reference backgrounds or fail to compose multiple references correctly.Both methods produce imperfect videos when the number of reference images reaches three.Source: I Visualizations · p. 30

J Simplified Code Implementation

  • The appendix describes a simplified PyTorch implementation of the canvas-connector DiT block and directs readers to Section 4.1 and Table 9 for training procedures and hyperparameters.The authors say they will release the code upon acceptance.Source: J Simplified Code Implementation · p. 30

K Limitations and Future Works

The authors identify duplicated visual conditioning and limited, non-optimal training data as boundaries of the current setup. They specifically note low success for in-context generation from three or more reference images.

K Limitations and Future Works

  • The setup provides images and videos to both the MLLM and diffusion model, leaving an MLLM-only visual-input pathway as future work.The authors frame their focus as testing information transfer between MLLMs and diffusion Transformers through MetaCanvas.Source: K Limitations and Future Works · p. 30
  • Curated training-data quality is non-optimal, and data scale is limited for some tasks; expanding task-specific datasets could further improve performance.The limitation is reported for a system evaluated across tasks with text, image, video, or combined inputs.Source: K Limitations and Future Works · p. 30
  • In-context video generation has low success when using three or more reference images, and both MetaCanvas and VACE produce imperfect videos at three references.The authors suggest more and higher-quality data, with a larger share of multi-reference examples, as a possible way to alleviate the issue.Source: K Limitations and Future Works · p. 30
Loading 2512.11464v1…