PaperlayerView exact revision on arXiv

Source-linked AI summary

From Prompting to Composing: A Spatial Canvas Interface for Poster Generation

Yitong Wang, Fangyun Wei, Jinjing Zhao, Sirui Zhang, Hongyang Zhang, Dong Chen, Bo Dai, Yan Lu

arXiv:2610.12230v1cs.CVcs.AI

TL;DR

Text prompting makes it indirect to specify poster composition, which is inherently two-dimensional. The paper introduces a Spatial Canvas and Compo for structured, direct or agentic generation; experiments report stronger compositional controllability than general-purpose models and dedicated poster systems while maintaining high visual quality.

  • Problem

    Text-only prompting makes it indirect and ambiguous to specify posters’ intentional spatial, identity, and visual constraints.

    Source: §1 · p. 2

  • Method

    The Spatial Canvas combines semantic, identity, text, and pixel bindings with Text Specifications, and Compo uses automatically generated supervision to support direct and agentic poster generation.

    Source: §1 · p. 2

  • Results

    Compo achieves stronger compositional controllability than general-purpose image generation models and dedicated poster systems while maintaining high visual quality.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The Spatial Canvas serves both as a human-facing composition interface and as a structured intermediate representation for agentic generation.

    Source: §1 · p. 2

    The authors urge responsible use of the models and data that respects privacy and image owners’ intellectual-property rights, particularly in commercial applications.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Text prompting is an indirect interface for poster generation, requiring users to encode inherently two-dimensional composition intent into a one-dimensional sequence of words. We introduce a Spatial Canvas Interface that enables users to directly compose generation intent in space through four complementary binding types: semantic, identity, text, and pixel, together with Text Specifications for individual elements and global appearance. Based on this interface, we develop Compo, a poster generation model adapted from a pretrained image editing model to understand Spatial Canvas inputs and Text Specifications. Compo supports both direct inference, where users explicitly construct the canvas, and agentic mode, where a high-level request is automatically translated into a planned Spatial Canvas. To train Compo, we develop a scalable pipeline that automatically constructs supervision data for different binding types and their combinations, enabling efficient adaptation without training a specialized poster generator from scratch. We further introduce a benchmark that evaluates adherence to individual binding types and their joint composition. Experiments show that Compo achieves stronger compositional controllability than both general-purpose image generation models and dedicated poster generation systems while maintaining high visual quality. By decoupling intent specification from visual generation, our work shifts poster generation from prompting toward composing.

1 INTRODUCTION

The paper replaces text-only poster prompting with a Spatial Canvas that lets users specify composition spatially, using bindings and Text Specifications. Compo supports direct and agentic generation, and experiments report stronger compositional controllability than general-purpose models and dedicated poster systems while maintaining high visual quality.

1 INTRODUCTION

  • The Spatial Canvas represents element locations and bindings, while Text Specifications give element-level instructions and global descriptions such as background and style.Together, they form a structured representation of the target poster.Source: §1 · p. 2
  • Semantic, identity, text, and pixel bindings can be combined on one canvas to control semantics, identity, text, pixels, and layout.They respectively associate a textual concept, reference image, exact text string, or preserved visual content with a target region.Source: §1 · p. 2
  • Compo supports direct inference from user-constructed canvases and agentic generation, which converts high-level requests into the same structured representation.The agent extracts elements and text, plans layout, selects bindings, and retrieves reference images when needed.Source: §1 · p. 2
  • A scalable pipeline automatically creates supervision for individual binding types and their combinations, enabling adaptation of a pretrained image-editing model without training a specialized poster generator from scratch.The paper also introduces a benchmark measuring adherence to individual bindings and their joint composition.Source: §1 · p. 2
  • Compo achieves stronger compositional controllability than general-purpose image generation models and dedicated poster systems while maintaining high visual quality.The benchmark evaluates adherence to individual binding types and their joint composition.Source: §1 · pp. 2–3

2 RELATED WORKS

Related work spans image-generation foundations, reference- and layout-driven control, and poster-specific systems. The paper identifies fine-grained multimodal binding and extensible control for structured graphic design as remaining gaps.

2 RELATED WORKS

  • Despite advances in image-generation and editing models, jointly expressing multiple multimodal references and precise layout constraints remains underexplored for complex graphic design.Design-oriented systems offer more structured inputs, while other systems support generation, editing, and sometimes image understanding.Source: §2 · p. 3
  • Reference- and layout-driven methods offer identity or spatial control, but joint-control methods often use task-specific modules and existing approaches lack fine-grained bindings between regions and text descriptions.Existing work mainly targets portraits or general image synthesis, leaving structured graphic design underexplored.Source: §2 · p. 3
  • Poster-generation methods have expanded from layout and text rendering to references, layout conditions, background control, and reinforcement learning, yet often retain restricted reference types or task-specific control modules.These restrictions limit extensibility and require complex user inputs.Source: §2 · p. 3

3 METHOD

Compo takes a Spatial Canvas and Text Specifications as input to generate posters satisfying spatial, semantic, identity, text, and pixel constraints. Its method combines scalable paired-data construction, two-stage adaptation of a pretrained model, and direct or agentic inference.

3.1 TRAINING DATA GENERATION

  • Compo’s training pipeline first generates posters, then automatically recovers visual and text elements to create paired Spatial Canvases and Text Specifications.The dataset provides scalable supervision for training Compo.Source: Figure 2 · p. 4
  • The pipeline creates varied poster concepts from sampled themes, items, and styles, then rewrites structured specifications into prompts while preserving planned information.Posters are rendered using multiple text-to-image models.Source: §3.1 · p. 4

3.2 ARCHITECTURE

  • Compo adapts a pretrained multimodal diffusion transformer with LoRA, jointly encoding the Spatial Canvas and Text Specifications with the target poster during supervised training.Reinforcement learning then optimizes complementary rewards for text accuracy, instance similarity, and layout adherence.Source: Figure 3 · p. 5, §3.2 · pp. 5–6

3.1 TRAINING DATA GENERATION

  • Automatic annotation segments visual instances, processes assets into references, recognizes text and locations, and pairs the poster with canvas annotations and Text Specifications.The resulting Compo-200K dataset contains roughly 200K samples with fine-grained structured annotations.Source: §3.1 · p. 5

3.2 ARCHITECTURE

  • The reinforcement-learning rewards measure text correctness and character fidelity, instance visual and semantic similarity, and the IoU of intended and generated element regions.Sentence accuracy is aggregated with a logical AND, while other reward metrics use arithmetic averaging.Source: §3.2 · p. 6

3.3 INFERENCE

  • Direct inference lets users specify canvas elements and requirements, while Agentic Mode converts a high-level request into a planned canvas and detailed Text Specifications for Compo.The agent extracts elements, plans layout, selects bindings, and searches for visual references when needed.Source: §3.3 · p. 7

4 EXPERIMENTS

Compo is evaluated on a 500-sample benchmark using binding-specific metrics and comparisons with general image-editing models and dedicated poster systems. Both Compo variants outperform all baselines on nearly all metrics, while ablations show complementary contributions from reward components and model branches.

4 EXPERIMENTS

  • The 500-sample benchmark uses non-overlapping data and structured annotations; its metrics assess binding-specific preservation and aggregate scores across evaluated regions.Detected regions are matched to ground truth by IoU, with unsuccessful matches receiving zero scores.Source: §4 · pp. 7–8

4.1 MAIN RESULTS

  • Figure 5 qualitatively compares Compo with four representative baselines, with transparent pink boxes marking noticeable errors.The comparisons include three dedicated poster methods and HiDream-O1-Image.Source: Figure 5 · p. 8, §4.1 · p. 8
  • Both Compo variants outperform all baselines on nearly all metrics, yielding consistent overall gains.Comparisons include general text-and-image-to-image models and dedicated poster methods.Source: §4.1 · p. 8

4.2 ABLATION STUDY

  • Removing all reinforcement-learning rewards gives the lowest overall performance, while individual removals chiefly affect their corresponding binding requirements.Text reward removal degrades text binding, instance-similarity removal affects pixel and identity consistency, and layout reward removal lowers IoU-related metrics.Source: §4.2 · p. 9, Table 2 · p. 9
  • A text-reward weight of 0.75 achieves the highest sentence accuracy and second-highest NED without compromising layout adherence.Lower weights tolerate minor rendering errors, whereas a weight of 1 can produce sparse rewards early in reinforcement learning when all text is rarely correct.Source: §4.2 · p. 9, Table 3 · p. 9
  • Fine-tuning both text and image branches achieves the best overall performance; either branch alone leaves the other insufficiently adapted and weakens binding performance.The text branch adapts to structured visual-language tokens, while the image branch learns to interpret noisy target and spatial canvas tokens.Source: §4.2 · pp. 9–10, Table 4 · p. 9

5 CONCLUSION

The conclusion presents Compo as a poster-generation model that replaces text-only composition intent with a spatial canvas and text specifications. It reports stronger compositional controllability than general-purpose and dedicated systems while noting data provenance and reproducibility practices.

5 CONCLUSION

  • Compo uses a Spatial Canvas Interface to specify poster intent spatially through semantic, identity, text, and pixel bindings with Text Specifications.The interface is paired with direct composition or agentic generation from high-level requests.Source: §5 · p. 10
  • Experiments show stronger compositional controllability than general-purpose image generation models and dedicated poster systems while maintaining high visual quality.Source: §5 · p. 10
  • The authors report using large language models and image-generation models in data generation and annotation, and manually reviewed generated text assets for harmful content.They state that authors reviewed and verified manuscript content refined with language models.Source: AI USE STATEMENT · p. 11
  • The data uses images from publicly available image-generation models, and the authors encourage privacy- and intellectual-property-respecting use, particularly commercially.Source: ETHICS STATEMENT · p. 11
  • The paper identifies sections and appendices covering dataset construction, training, inference, evaluation, and additional implementation details to facilitate reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 COMPO-200K DATASET

Compo-200K is built through an automated pipeline that plans poster content, generates images, recovers annotations, and renders Spatial Canvas inputs with Text Specifications.

Text Asset Collection

  • The content repository contains roughly 1,000 instances across 12 poster themes, while the style repository contains around 250 visual style descriptions.Large language models assist in curating both repositories.Source: §A.1 · p. 17

LLM Planning

  • GPT-5.5 combines sampled themes, 3–8 instances, and a style with planned design elements to produce approximately 200K structured plans.The plans are rewritten as natural-language generation prompts while preserving sampled assets and planned attributes.Source: §A.1 · p. 17

Text-to-Poster Generation

  • Three text-to-image models are randomly selected as generation backends to render posters at 1024² resolution, increasing diversity and mitigating model-specific biases.The pipeline runs on approximately 32 NVIDIA A100 80GB GPUs.Source: §A.1 · p. 17

Annotation and Canvas Construction

  • The pipeline segments visual instances, normalizes identity poses or object lighting, and uses OCR-recognized text and locations as final ground-truth annotations.Canvas regions are populated with text, identity, pixel, or semantic bindings, with descriptions recorded in Text Specifications.Source: §A.1 · p. 17

A.2 BENCHMARK

The benchmark contains 500 non-overlapping samples with structured annotations and evaluates models through input formats suited to their interfaces using automatic, model-agnostic metrics.

Benchmark Data Creation

  • The benchmark samples 4–9 instances per poster, exceeding the 3–8 instances used in training to make evaluation more challenging.Source: §A.2 · p. 17
  • It contains 500 samples with no training-set overlap, each paired with a Spatial Canvas and structured annotations.Source: §A.2 · p. 18

Evaluation Metrics

  • Evaluation converts inputs to each model type’s expected format and computes metrics automatically from generated posters and structured annotations.The protocol is model-agnostic and requires neither model-specific input designs nor access to internal representations.Source: §A.2 · p. 18

B.1 SUPERVISED TRAINING

Compo is supervised from pretrained image-editing models, then post-trained with binding-specific rewards; its agentic mode uses a planner and verifier to construct and check canvas inputs.

B.1 SUPERVISED TRAINING

  • Compo is initialized from LongCat-Image-Edit and Qwen-Image-Edit-2511, then fine-tuned with rank-128 LoRA at approximately 1024^2 resolution.LongCat uses 8 A100 GPUs and batch size 64; Qwen uses 16 A100 GPUs and batch size 32.Source: §B.1 · p. 18

B.2 REINFORCEMENT LEARNING

  • RL rollouts generate 10 candidates per sample using 10 denoising steps; Qwen runs at 512^2 to reduce GPU memory overhead, while LongCat runs at 1024^2.Both models use guidance scale 1.0 and 16 A100 GPUs during rollouts.Source: §B.2 · p. 18
  • Candidates are matched to ground-truth regions by maximum IoU, with below-threshold matches receiving zero reward and valid matches scored by binding-specific metrics.Text uses sentence accuracy and normalized edit distance; semantic uses CLIP-T; identity uses CLIP-I, DINO-I, and CLIP-T; pixel uses CLIP-I and DINO-I.Source: §B.2 · p. 18
  • Image-level rewards combine text, similarity, and layout signals to optimize LoRA modules with DiffusionNFT, using logical AND for text accuracy and averages for other metrics.The weights are 0.4 for text, 0.4 for instance similarity, and 0.2 for layout.Source: §B.2 · p. 18

B.3 AGENTIC MODE

  • GPT-5.6 serves as planner and verifier in agentic mode, extracting instances, assigning binding types, inferring spatial relations, and checking retrieved images.Source: §B.3 · p. 19

C.1 QUANTITATIVE RESULTS

The results compare Compo before and after RL post-training, which improves most evaluation metrics consistently across both model variants.

C.1 QUANTITATIVE RESULTS

  • Table 5 compares Compo-LongCat-Base and Compo-Qwen-Base against their respective RL-post-trained versions.Source: §C.1 · p. 19, Table 5 · p. 19
  • RL post-training yields consistent improvements across most evaluation metrics for both Compo model variants.Table 5 compares supervised-only Base models with their corresponding RL-post-trained models.Source: Table 5 · p. 19, §C.1 · p. 19
  • Table 5 covers pixel, identity, semantic, and text binding metrics, with higher values indicating better performance.It defines NED as normalized edit distance and Sen. Acc. as sentence accuracy.Source: Table 5 · p. 19

C.2 QUALITATIVE RESULTS

This section presents qualitative Compo poster results across unseen themes, elements, and resolutions, alongside the LLM prompts and constraints used for dataset planning and agentic Spatial Canvas planning. It also describes how agentic plans bind poster elements and how candidate references are checked for usability and subject match.

C.2 QUALITATIVE RESULTS

  • Figures 6 and 7 provide additional visual comparisons of Compo with dedicated poster-generation methods and HiDream-O1-Image.The comparisons include PosterMaker, PosterOmni, and CreatiDesign; the captions identify transparent pink boxes as highlighting noticeable errors without specifying a winner.Source: Figure 6 · p. 20, Figure 7 · p. 21
  • Across diverse unseen themes and visual elements, Compo produces coherent, plausible posters at various resolutions, indicating robust generalization.The examples include cutting-edge technology, Disney and Japanese animation, and AI industry news.Source: §C.2 · p. 19, Figure 8 · p. 22

D.1 PROMPT FOR LLM PLANNING IN DATASET GENERATION

  • The identity-object interaction prompt requires a structured JSON plan with a scenario, background prompt, visual text, and one action-and-target entry per humanoid.It restricts background descriptions to background content and requires target objects to come from the input Things list.Source: §D.1 · pp. 23–24
  • The object-only prompt instead specifies one visual description for each input Thing, alongside a coherent scenario, background prompt, and short poster text.The object descriptions focus on the objects themselves rather than interactions, and the prompt says every input Thing should remain visually relevant.Source: §D.1 · pp. 25–27
  • Dataset planning prompts request concise, concrete object descriptions and coherent poster plans while enforcing safety, valid JSON, and exact input names.The example plan includes a scenario, background prompt, visual text, and descriptions for each requested object.Source: §D.1 · pp. 27–28

D.2 PROMPT FOR LLM REWRITING IN DATASET GENERATION

  • Rewriting prompts turn structured plans into fluent image prompts, preserving specified content while requiring readable text, varied layouts, visible subjects, and cohesive background integration.Visual text should be horizontal and legible; layouts should not default to centered hero compositions.Source: §D.2 · p. 29

D.3 PROMPT FOR AGENTIC MODE

  • Agentic planning retains a minimum sufficient set of independently controllable elements, capped at 10, and assigns each element one of four binding types.Pixel binding preserves reference appearance, identity binding preserves subject identity, semantic binding specifies a generic category and location, and text binding specifies exact copy.Source: §D.3 · p. 31
  • The planner prioritizes references and exact copy, retains elements that affect core meaning or need independent control, and reserves clear space for readable subjects and text.Semantic elements are normally subordinate; explicitly requested copy must retain its exact wording, and unboxed scenery belongs in the background.Source: §D.3 · pp. 31–32
  • Named or identity-critical subjects use identity_binding, exact-reference subjects use pixel_binding, and generic objects use semantic_binding; matching user references take priority, while missing references trigger retrieval or generation as specified.If no useful reference can be retrieved or generated, the planner falls back to semantic_binding.Source: §D.3 · pp. 32–33
  • Generated reference prompts isolate one subject, include global style and solid-black studio lighting, and use neutral views for identity_binding while preserving applicable pixel_binding pose or orientation.The planner leaves reference paths and metadata null for the resolver to fill later.Source: §D.3 · pp. 34–35

Image Search Verification System Prompt

  • The validator checks subject presence, correspondence to the expected subject and reference text—including distinctive traits for named identities—and sufficient visibility for reference use.It rejects empty or unrecognizable images, unrelated or incorrect identities, and subjects that are tiny or obscured.Source: Image Search Verification System Prompt · p. 37

Related papers

Loading 2610.12230v1…