PaperlayerView exact revision on arXiv

Source-linked AI summary

MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis

Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, Yi Yang

arXiv:2402.05408v2cs.CV

TL;DR

Existing text-to-image systems mainly handle single-instance descriptions, while generating multiple instances with coordinated positions and attributes remains challenging. The paper proposes MIGC, which decomposes multi-instance generation into instance-shading subtasks, enhances each instance, and aggregates the results. Across COCO-MIG, COCO, and DrawBench, MIGC improves control performance while retaining inference speed close to stable diffusion, but remains limited by instances that stable diffusion cannot generate accurately.

  • Problem

    Single-instance generation does not adequately address simultaneous multi-instance control of quantity, position, attributes, and interactions.

  • Method

    MIGC divides multi-instance generation into single-instance shading subtasks, enhances them with Enhancement Attention, and aggregates them through Layout Attention and a Shading Aggregation Controller.

  • Results

    MIGC substantially improves control across COCO-MIG, COCO, and DrawBench, including Instance Success Rate from 32.39% to 58.43% on COCO-MIG, with speed close to original stable diffusion.

  • Takeaways & Limitations

    MIGC extends the strong single-instance capability of pre-trained stable diffusion to multi-instance generation with control over quantity, position, attributes, and interactions.

  • Takeaways & Limitations

    MIGC also encounters difficulties when stable diffusion cannot generate a specific instance accurately, such as individual letters.

Abstract

from arXiv · show

We present a Multi-Instance Generation (MIG) task, simultaneously generating multiple instances with diverse controls in one image. Given a set of predefined coordinates and their corresponding descriptions, the task is to ensure that generated instances are accurately at the designated locations and that all instances' attributes adhere to their corresponding description. This broadens the scope of current research on Single-instance generation, elevating it to a more versatile and practical dimension. Inspired by the idea of divide and conquer, we introduce an innovative approach named Multi-Instance Generation Controller (MIGC) to address the challenges of the MIG task. Initially, we break down the MIG task into several subtasks, each involving the shading of a single instance. To ensure precise shading for each instance, we introduce an instance enhancement attention mechanism. Lastly, we aggregate all the shaded instances to provide the necessary information for accurately generating multiple instances in stable diffusion (SD). To evaluate how well generation models perform on the MIG task, we provide a COCO-MIG benchmark along with an evaluation pipeline. Extensive experiments were conducted on the proposed COCO-MIG benchmark, as well as on various commonly used benchmarks. The evaluation results illustrate the exceptional control capabilities of our model in terms of quantity, position, attribute, and interaction. Code and demos will be released at https://migcproject.github.io/.

1. Introduction

The paper introduces Multi-Instance Generation (MIG), which requires simultaneous control of multiple instances’ quantity, position, attributes, and interactions. MIGC addresses this challenge by dividing instance shading, enhancing each subtask, and aggregating the results, with experiments showing improved control and near-original inference speed.

  • Motivation: MIG extends single-instance generation to simultaneous generation of multiple instances with diverse quantity, position, attribute, and interaction controls.Single-instance methods align generated content with one description, whereas MIG requires multiple instances to satisfy corresponding descriptions and layouts together.
  • Challenges: MIG is challenging because text encoders can mix multiple attributes and Cross-Attention lacks precise positional control.These limitations can cause attribute leakage and make instances affect one another’s regions.
  • MIGC: MIGC divides MIG into single-instance shading subtasks, conquers them with Enhancement Attention, and combines them using Layout Attention and Shading Aggregation Controller.The approach applies divide-and-conquer within stable diffusion’s Cross-Attention space and aggregates shading results into the final result.
  • Evaluation: COCO-MIG evaluates whether generation models simultaneously control position, attributes, and quantity.The benchmark is based on COCO and is designed to assess inherent MIG capabilities.
  • Results: 32.39% to 58.43%: MIGC increased Instance Success Rate on COCO-MIG while maintaining inference speed close to original stable diffusion.Experiments also covered COCO and DrawBench and reported improvements across position, attribute, and count.

2. Related work

Text-to-image generation has shifted from GANs toward diffusion and autoregressive models because of more stable training and higher image quality. Related layout-to-image methods improve positional control but struggle to isolate attributes across multiple instances.

  • Text-to-image generation: Diffusion and autoregressive models increasingly replaced conditional GANs as foundational text-to-image generators because of more stable training and higher image quality.The related-work passage describes this as a progression in text-to-image generation methods.
  • Layout-to-image generation: Layout-to-image methods integrate layout information for positional control but struggle to isolate multiple instances’ attributes, causing mixed attributes.MIGC is presented as addressing precise position-and-attribute control together.

3. Method

MIGC decomposes multi-instance generation into masked instance-shading subtasks, enhances each instance's semantics and position, then aggregates the results for coherent generation. Its design addresses attribute leakage, instance merging, and instance missing while evaluating performance across increasing instance counts.

  • Divide MIG into Instance Shading Subtasks: MIGC frames multi-instance generation as producing correct feature shading for each described instance within its corresponding bounding-box mask.The method operates in Cross-Attention space, where text and image features interact and generated content is determined.
  • Divide MIG into Instance Shading Subtasks: MIGC combines instance descriptions with Fourier-embedded box coordinates to create grounded phrase tokens that distinguish similarly described instances.This design targets instance merging when boxes are close or overlapping.
  • Conquer Instance Shading: A trainable Enhancement-Attention layer strengthens each masked shading result, helping generate an instance when the initial Cross-Attention result is weak.The instance mask restricts the enhancement to the correct region and guides semantic satisfaction for the instance description.
  • Combine Shading Results: MIGC obtains shading backgrounds from the global prompt and uses Layout Attention to construct a template that bridges independently shaded instances and background.The layout attention mask restricts pixels to attending within the same instance region, avoiding attribute leakage.
  • Evaluation Setup: COCO-MIG evaluates simultaneous control of position, attribute, and quantity, with levels L2 through L6 corresponding to the required number of generated instances.The method is evaluated alongside qualitative comparisons against existing baselines.
  • Combine Shading Results: The Shading Aggregation Controller dynamically fuses instance, background, and layout-template residuals through sequential intra-attention, inter-attention, and softmax-normalized weights.The aggregation weights sum to 1 at each spatial pixel and produce the final shading.

4. Experiments

Experiments evaluate MIGC across COCO-MIG, COCO-Position, and DrawBench, using quantitative, qualitative, and ablation analyses. MIGC improves multi-instance control while retaining near-stable-diffusion speed and image quality.

  • Benchmarks and metrics: Models are evaluated on COCO-MIG, COCO-Position, and DrawBench using position, attribute, quantity, image-quality, and human-evaluation measures.COCO-MIG uses 8 seeds per prompt and 6,400 generated images; DrawBench uses 64 prompts and 512 images.
  • Quantitative results: MIGC increases COCO-MIG Instance Success Rate from 32.39% to 58.43% and mIoU from 32.25 to 51.48.Improvements persist across all count-division levels, covering position, quantity, and attributes.
  • Quantitative results: MIGC raises COCO-Position Success Rate from 70.52% to 80.29%, mIoU from 71.61 to 77.38, and AP from 40.68/68.26/42.85 to 54.69/84.17/61.71.FID remains similar to stable diffusion, indicating improved position control without destroying image quality.
  • Quantitative results: MIGC achieves the best performance on DrawBench in both mechanical metrics and human evaluation.Human evaluation judges whether images correctly satisfy color, position, and count requirements without relying on IoU for position correctness.
  • Qualitative results: Qualitative results show accurate position-and-attribute control in COCO-MIG and precise box-constrained generation in COCO-Position without missing or merged instances.DrawBench qualitative results are reported as supplementary material.
  • Ablation study: MIGC applies its controller during the first 25 of 50 denoising steps, shifting from instance-focused foreground shading early to global context later.SAC gives more early weight to EA instance shading and LA background templates, then increases attention to global background context.
  • Ablation study: Ablations show that EA substantially improves Success Rate from 12.10% to 80.16%, mIoU from 29.55 to 76.63, and AP from 1.89 / 7.64 / 0.49 to 53.03 / 84.05 / 58.67.LA improves image quality and AP, SAC improves shading aggregation, and inhibition loss improves control, especially with many instances.

5. Conclusion

The paper defines MIG and introduces MIGC to improve stable diffusion’s multi-instance generation ability. It divides generation into single-instance shading subtasks, enhances each instance, and combines results through layout attention and shading aggregation.

  • Conclusion: MIGC divides complex MIG into single-instance shading subtasks, conquers each with Enhancement Attention, and combines results using Layout Attention and Shading Aggregation Controller.Experiments on COCO-MIG, COCO-Position, and DrawBench support the approach’s reported efficiency and effectiveness.

Supplementary Material

COCO-MIG extends COCO-Position by assigning each instance a specified color while preserving layouts. Its construction tests simultaneous control of position, attributes, and quantity.

  • Benchmark construction: COCO-MIG requires each generated instance to satisfy both its prescribed position and its assigned color attribute.The benchmark uses COCO-Position layouts and adds a specific color to each instance.
  • Benchmark construction: Layouts are filtered for sufficiently large instances and at least two instances, then divided into levels L2-L6 by target instance count.Each level contains 160 sampled layouts.
  • Benchmark construction: Each instance receives one of eight specified colors, and the global prompt lists the colored objects together.The prompt follows the form “a <attr1> <obj1>, a <attr2> <obj2>, ..., and a ...”.
  • Benchmark motivation: GLIGEN can satisfy the donut layout in the example but fails the color-attribute requirement that COCO-MIG evaluates.Position-only correctness in COCO-Position is therefore insufficient for COCO-MIG.

C. More MIG Results

Additional MIG results show that MIGC handles complex layouts and rich attribute descriptions while preserving instance placement and attributes. It can also control specified interactions between instances.

  • More MIG results: MIGC generates instances at correct positions with correct attributes even under complex layouts and rich attribute descriptions.When the global prompt specifies relationships such as actions, MIGC can further control interactions between instances.

D. More Qualitative Results on COCO-MIG

On COCO-MIG, MIGC better controls instance position, attributes, and quantity simultaneously than previous state-of-the-art methods.

  • MIGC better controls position, attributes, and quantity simultaneously than previous state-of-the-art methods on COCO-MIG.

E. Qualitative Results on DrawBench

On DrawBench, MIGC provides precise control over instance attributes, positions, and quantities, including multi-category cases, while reducing attribute leakage and extra generation.

  • MIGC can control interactions between instances by specifying their relations through the global prompt.
  • MIGC precisely controls each instance’s attributes, avoiding attribute leakage such as confusion between yellow and red.
  • MIGC generates counterfactual attributes such as a black apple that previous state-of-the-art methods cannot correctly generate.
  • MIGC controls positions more accurately and reduces extra generation, including excessive carrot generation by Multi-D and GLIGEN.The authors attribute this improvement mainly to inhibition loss.
  • MIGC achieves accurate quantity control, including cases with multiple categories where stronger attribute control prevents cat attributes from leaking into dogs’ regions.

G. Details of Evaluation Pipeline

The evaluation pipeline checks whether generated instances satisfy target position, using Grounding-DINO detection and an IoU threshold.

  • The pipeline detects the generated object’s bounding box with Grounding-DINO and compares it with the target layout box using IoU.
  • An instance is position-correct when its detected bounding-box IoU with the target layout is at least 0.5.
  • Benchmarks requiring only position control require every instance to be position-correctly generated.

H. Manual Evaluation on DrawBench

Manual DrawBench evaluation measures whether generated images globally satisfy text descriptions for color, position, and count, using judgments from ten people.

  • Ten people judge each DrawBench image as correctly or wrongly generated for color, position, and count adherence.
  • The reported manual-evaluation result is the average accuracy across the ten participants’ judgments.
  • Unlike automated mIoU evaluation, manual evaluation checks whether the image satisfies the text description globally.

I. More Implementation Details

The paper reports implementation details, divide-and-conquer baselines, and a limitation arising when Stable Diffusion cannot generate a specific instance accurately. MIGC is evaluated against alternative pipelines that trade attribute correctness against inference cost or visual harmony.

  • Implementation Details: MIGC is deployed in selected mid-layer and lowest-resolution decoder Cross-Attention layers, while other layers use global-prompt shading.The selected layers are described as determining layout and semantic information.
  • Baselines: The PCT-Net pipeline independently generates instances and the background before fusion, preserving attributes but increasing inference time and potentially reducing image harmony.The passage presents this pipeline as evidence supporting the divide-and-conquer idea.
  • Limitation: MIGC extends Stable Diffusion’s single-instance generation capability to multi-instance generation through divide and conquer.This design inherits difficulties from Stable Diffusion when individual instances are hard to generate.
  • Limitation: When Stable Diffusion generates individual letters inaccurately, MIGC can preserve per-letter color control while producing incorrect letter content.The CVPR example shows that incorrect individual-instance content can cause the complete multi-instance sample to fail.
Loading 2402.05408v2…