PaperlayerView exact revision on arXiv

Source-linked AI summary

Canvas-to-Image: Compositional Image Generation with Multimodal Controls

Yusuf Dalva, Guocheng Gordon Qian, Maya Goldenberg, Tsai-Shien Chen, Kfir Aberman, Sergey Tulyakov, Pinar Yanardag, Kuan-Chieh Jackson Wang

arXiv:2511.21691v1cs.CV

TL;DR

Diffusion models still struggle to honor multiple compositional controls together, including references, pose, and layout. Canvas-to-Image encodes heterogeneous controls in one canvas and trains across control types. The model generalizes from single-control training to multi-control inference and reports strong results across identity preservation and control adherence.

  • Problem

    Diffusion models have limited flexibility for simultaneous control, and existing approaches do not jointly handle heterogeneous references, layouts, and poses.

    Source: §1 · p. 2

  • Method

    Canvas-to-Image encodes heterogeneous controls in one RGB canvas and jointly trains a diffusion model across canvas types for compositional reasoning.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    Experiments report improved identity preservation and control adherence, while the model generalizes from single-control training to multi-control inference.

    Source: §1 · p. 2, §3.2 · p. 4, §5 · p. 8

  • Takeaways & Limitations

    A unified model can combine reference, pose, and layout controls at inference, including combinations not seen together during training.

    Source: §3.2 · p. 4, §5 · p. 8

    A single RGB canvas limits how many concepts can be interpreted simultaneously because increasing concept density crowds the canvas.

    Source: C. Limitations · pp. 16–17

Abstract

from arXiv · show

While modern diffusion models excel at generating high-quality and diverse images, they still struggle with high-fidelity compositional and multimodal control, particularly when users simultaneously specify text prompts, subject references, spatial arrangements, pose constraints, and layout annotations. We introduce Canvas-to-Image, a unified framework that consolidates these heterogeneous controls into a single canvas interface, enabling users to generate images that faithfully reflect their intent. Our key idea is to encode diverse control signals into a single composite canvas image that the model can directly interpret for integrated visual-spatial reasoning. We further curate a suite of multi-task datasets and propose a Multi-Task Canvas Training strategy that optimizes the diffusion model to jointly understand and integrate heterogeneous controls into text-to-image generation within a unified learning paradigm. This joint training enables Canvas-to-Image to reason across multiple control modalities rather than relying on task-specific heuristics, and it generalizes well to multi-control scenarios during inference. Extensive experiments show that Canvas-to-Image significantly outperforms state-of-the-art methods in identity preservation and control adherence across challenging benchmarks, including multi-person composition, pose-controlled composition, layout-constrained generation, and multi-control generation.

1. Introduction

Diffusion models offer realistic, diverse images but lack unified control over multiple compositional signals. Canvas-to-Image combines those controls in a canvas representation and trains jointly across tasks, supporting mixed-control generation.

1. Introduction

  • Canvas-to-Image lets users combine positioned or posed subjects, tagged bounding boxes, and pose overlays in one canvas interface.These complementary controls define both semantics and composition.Source: §1 · p. 2
  • Existing models handle isolated compositional controls, while jointly interpreting subject references, boxes, and textual tags remains challenging.Prior subject-injection methods often lack spatial control, while layout-guided methods may lack specific poses or subjects.Source: §1 · p. 2
  • The Multi-Task Canvas encodes heterogeneous controls in a composite RGB image, giving the model a common pixel space without extra modules or architectural changes.The framework also curates aligned multi-task datasets and jointly fine-tunes the diffusion model across control tasks.Source: §1 · p. 2
  • Joint training supports mixed controls at inference, and experiments report improved identity preservation and control adherence against existing methods.The reported generalization includes combinations of controls not seen together during training.Source: §1 · p. 2

2. Related Work

Prior work has advanced image quality, personalization, and individual forms of compositional control, but combining heterogeneous controls remains difficult. Canvas-to-Image addresses this with a unified visual canvas.

2. Related Work

  • Diffusion and multimodal models improve image synthesis, yet fine-grained control across multiple compositional constraints remains difficult.Personalization methods often focus on reference injection; multi-concept approaches face disentanglement or embedding-complexity challenges.Source: §2 · pp. 2–3
  • ControlNet and T2I-Adapter use structural cues such as poses, while GLIGEN, LayoutDiffusion, and CreatiDesign target boxes or segmentation masks.These lines of work generally address isolated tasks rather than unifying heterogeneous controls.Source: §2 · p. 3
  • Methods supporting subject injection often lack fine-grained spatial control, while layout-guided methods cannot incorporate specific poses or subject identities, making unified compositional control highly challenging.Source: §2 · p. 3
  • Canvas-to-Image reformulates layouts, pose guidance, and subject appearance injection as controls in one visual canvas.The framework is described as maintaining constant computation cost while supporting multi-control, multi-subject personalization.Source: §2 · p. 3

3. Methodology

Canvas-to-Image represents heterogeneous compositional controls in one RGB canvas and trains a VLM–Diffusion model across canvas types. Single-control training supports combinations of controls at inference, including combinations not seen during training.

3. Methodology

  • The Multi-Task Canvas is a shared RGB input format for heterogeneous controls, with variants that express different forms of compositional reasoning.The framework jointly optimizes the VLM–Diffusion architecture across control types.Source: §3 · p. 3, §3.1 · p. 4

3.1. Multi-Task Canvas

  • Spatial Canvas places segmented subject cutouts on a masked background; Pose Canvas overlays a pose skeleton and may drop subject segments during training.Cross-frame subject-background pairing is used to avoid copy-pasting artifacts; dropping segments supports pose control without reference injection.Source: §3.1 · p. 4
  • Box Canvas encodes layout with annotated bounding boxes that identify subjects and their spatial regions, without reference images.Person identifiers are ordered from left to right.Source: §3.1 · p. 4

3.2. Model and Multi-Task Training

  • The VLM encodes canvas information, which is combined with canvas VAE latents, text embeddings, and noisy latents for diffusion-model conditioning.Flow matching trains velocity prediction, while a task indicator distinguishes the control type represented by each canvas.Source: §3.2 · p. 4
  • Single-control canvas training generalizes to combinations of controls at inference, including combinations absent from training.A task-indicator prompt specifies the control type and helps prevent mode blending.Source: §3.2 · p. 4, §3.1 · p. 4

4. Experiments

Experiments compare Canvas-to-Image across four compositional-control benchmarks and report qualitative advantages in identity, pose, and layout adherence. Ablations show gains as Pose Canvas and Box Canvas tasks are added to training, including generalization to multi-control inference.

4.1. Experiment Details

  • Comparisons cover four control benchmarks and baselines including Qwen-Image-Edit, Nano-Banana, CreatiDesign, and Overlay Kontext, using identity, image-quality, text-alignment, and Control-QA metrics.Training fine-tunes attention and modulation layers with rank-128 LoRA for 200K steps on 32 NVIDIA A100 GPUs.Source: §4.1 · pp. 5–6

4.2. Qualitative and Quantitative Results

  • On the Multi-Control Benchmark, Canvas-to-Image achieves the highest compositional fidelity while jointly satisfying identity, pose, and box constraints.The same model generalizes from single-control training samples to complex control scenarios at inference.Source: Figure 6 · p. 7, §4.2 · p. 8
  • Canvas-to-Image follows target poses while maintaining identity fidelity, and outperforms baselines on 4P composition identity preservation and spatial alignment.The 4P comparisons also report copy-pasting artifacts from Nano-Banana and identity failures from Overlay Kontext and Qwen-Image-Edit.Source: Figure 3 · p. 5, §4.2 · pp. 6–7, Figure 4 · p. 6
  • On layout-guided composition, Canvas-to-Image adheres to box constraints and surpasses CreatiDesign, while Nano-Banana and Qwen-Image-Edit often ignore structural signals or render annotations poorly.Canvas-to-Image maintains image quality comparable to Qwen-Image-Edit while improving spatial condition alignment.Source: §4.2 · pp. 7–8, Figure 5 · p. 7

4.3. Ablation Studies

  • Adding Pose Canvas and Box Canvas tasks to Spatial Canvas training yields consistent gains in HPSv3 and Control-QA and improves performance on multi-control inputs.Each training sample contains only one control type, while the final model handles multi-control scenarios at inference.Source: §4.3 · p. 8, Table 2 · p. 8, Figure 7 · p. 8

5. Conclusion

Canvas-to-Image unifies reference subjects, pose signals, and layout constraints in a canvas-conditioned framework that generalizes from single-control training to multi-control inference. The paper identifies the information density of a single RGB interface as a current boundary.

5. Conclusion

  • Canvas-to-Image unifies reference subjects, pose signals, and layout constraints in a single canvas-conditioned paradigm.The framework is presented as a scalable paradigm for multimodal guidance.Source: §5 · p. 8
  • Multi-Task Canvas training generalizes from single-control training samples to complex multi-control scenarios at inference.The unified model is reported to achieve strong identity preservation, pose fidelity, and structural coherence.Source: §5 · p. 8
  • The approach is currently bounded by the information density of a single RGB interface.The paper presents the formulation as a foundation for future, richer visual and semantic controls.Source: §5 · p. 8

Supplementary Material

Training dynamics show early gains in Control-QA and convergence around 50K iterations. Training continues to 200K to refine local details and improve generation robustness.

Supplementary Material

  • Control-QA improves steadily during early training.The curve indicates that the model learns consistent control and composition.Source: Figure I · p. 12
  • Control-QA converges around 50K iterations.The passage characterizes this as the point by which consistent control and composition have been learned.Source: Figure I · p. 12
  • Training continues to 200K iterations to refine local details and enhance generation robustness.Source: Figure I · p. 12

A. Comparisons with Personalization Methods

Supplementary comparisons evaluate Canvas-to-Image against personalization methods across composition, pose-guided composition, and ID–Object interaction. The reported results include strong metric performance and qualitative identity, pose, and interaction fidelity.

Comparison scope

4P Composition Benchmark

Pose-Guided 4P Composition Benchmark

ID-Object Interaction Benchmark

4P Composition Benchmark

  • Qualitative 4P examples show more consistent identity preservation and realistic multi-subject composition than personalization baselines and Qwen-Image-Edit.The comparisons include DreamO, OmniGen2, and UNO.Source: Figure II · p. 13

ID-Object Interaction Benchmark

  • On ID-Object examples, Canvas-to-Image preserves human identity and object fidelity with correct proportions and natural interactions.Existing baselines often fail to integrate the human and object realistically.Source: A. Comparisons with Personalization… · p. 14

B. Supplementary Ablations

Supplementary analyses track convergence and probe trainable-block choices. They report convergence near 50K iterations, along with performance effects from branch selection, feed-forward layers, and the task indicator.

Convergence Behavior of Canvas-to-Image

  • Control-QA improves rapidly early and largely converges by 50K; training continues to 200K for detail refinement and robustness, using that checkpoint for ablations.Source: B. Supplementary Ablations · p. 14

Ablations of Trainable Blocks

  • Omitting either the text or image branch lowers identity fidelity, while unfreezing feed-forward layers worsens visual quality and prompt alignment.The final model excludes feed-forward layers from optimization.Source: B. Supplementary Ablations · pp. 14–15
  • Removing the task indicator degrades all metrics; the authors identify explicit control-type signaling as crucial for resolving ambiguity and switching reasoning modes.Source: B. Supplementary Ablations · p. 15

C. Limitations

Canvas-to-Image’s canvas has a pixel-space capacity limit, while the paper also reports background-aware composition and favorable human-study comparisons. The section additionally presents qualitative pose results and an architecture ablation.

C. Limitations

  • In pose-overlaid 2P and 1P compositions, the paper reports identity preservation and pose alignment; the 2P results follow target poses from FLUX-Dev priors.The 2P examples are described as coherent and high-quality.Source: Figure V · p. 16, Figure VI · p. 16
  • The architecture ablation’s highest-performing configuration fine-tunes both text and image branches, excludes the feed-forward layer, and includes task indicators.Source: Table IV · p. 17
  • As concepts increase, the single RGB canvas becomes crowded and harder to interpret; the authors suggest layered controls such as an additional alpha channel.The stated constraint is the available pixel space, and the paper reports handling occluding entities up to 4P composition.Source: C. Limitations · pp. 15–17

D. Additional Applications

  • Canvas-to-Image can insert humans or objects into a background through reference-image pasting or box annotations, with natural scene interaction.The paper describes this as background-aware composition and provides qualitative examples.Source: D. Additional Applications · p. 17

E. User Study

  • The human evaluation uses separate studies for pose plus box layout and pose plus identity, with 30 participants and 30 examples per study.Participants select their preferred output in A/B comparisons.Source: E. User Study · p. 17
  • Canvas-to-Image outperforms both baselines overall; among competitors, performs stronger on identity preservation, whereas [45] performs better on control following, aligning human preference with reported metrics.Source: E. User Study · p. 17

F.1. Evaluation Metrics

This section specifies identity, image-quality, prompt-alignment, and control-adherence metrics, alongside benchmark construction procedures. It also reports user preference for Canvas-to-Image in control following and identity preservation.

F.1. Evaluation Metrics

  • For background-aware composition, the canvas can integrate humans or objects into a scene using reference-image pasting or bounding-box annotations.The figure describes natural spatial alignment and consistent lighting with the surrounding environment.Source: Figure VIII · p. 18
  • Canvas-to-Image is significantly preferred over strong baselines in both control following and identity preservation user studies.Source: Table V · p. 18
  • ArcFace similarity measures human identity preservation across specified benchmarks, while DINOv2 similarity measures object consistency in ID-Object Interaction.Human identities are sampled from FFHQ-in-the-wild.Source: §F.1 · p. 18
  • HPSv3 evaluates visual quality and prompt adherence, and VQAScore further assesses prompt alignment.HPSv3 is computed using the original user-provided prompt, without implicit prompt rewrites.Source: §F.1 · p. 18
  • Control-QA uses GPT-4o to score generated compositions against control images across pose, spatial-layout, box-layout, and multi-control settings.PoseAP and human user studies provide additional evaluations.Source: §F.1 · p. 18

F.2. Evaluation Benchmarks

  • Benchmark canvases use sampled identities and detected positions for 4P composition, extracted poses for pose-guided composition, filtered CreatiDesign samples for layout, and combined pose, identity, and annotations for multi-control.An automated pipeline constructs benchmark canvases.Source: §F.2 · pp. 18–19

F.3. Dataset Details

The training data combines an internal human-centric cross-frame dataset with CreatiDesign samples. The supplied evaluation protocols score compositional fidelity and joint control through holistic criteria and one-to-five ratings.

F.3. Dataset Details

  • Training uses about 6M human-centric images from about 1M cross-frame scenes, augmented with CreatiDesign named-annotation and text-rendering samples.Legal constraints prevent open-sourcing the internal dataset; the authors note a similar multi-frame dataset could be built from public video datasets.Source: §F.3 · p. 19
  • The compositional-fidelity rubric jointly evaluates identity, relative order, and realism, accepting minor artistic differences when those core properties remain preserved.Its instructions call for a single holistic score based on the three criteria.Source: §F.3 · p. 20
  • The accompanying composition rubric defines a 1–5 score, from excellent integration to failure, and asks evaluators to consider identity, spatial order, and realism together.Source: §F.3 · p. 20
  • The pose-control rubric adds pose fidelity to identity, order, and realism, with examples of minor and major flaws mapped to its rating levels.A score of 5 requires correct identities, order, poses, and realistic integration.Source: Table VII · p. 21, §F.3 · p. 21
  • The joint-control rubric scores whether identity, pose, layout, and realism are all satisfied, treating failure in one criterion as a compositional failure.The protocol calls for one holistic Joint Control Fidelity Score.Source: §F.3 · p. 23, Table IX · p. 23
Loading 2511.21691v1…