PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Programming for Text-to-Image Generation and Evaluation

Jaemin Cho, Abhay Zala, Mohit Bansal

arXiv:2305.15328v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Prior T2I work had not combined LMs and visual modules for generation, while existing evaluation often relied on single models that can be unreliable for some skills and do not explain scores. This paper introduces VPGEN’s staged generation and VPEVAL’s modular evaluation, reporting stronger layout control and higher human correlation than single-model evaluation methods.

  • Problem

    Prior work had not combined LMs and visual modules for T2I generation, and single-model evaluation methods can be unreliable on some skills and do not explain their scores.

    Source: §1 · pp. 1–2

  • Method

    VPGEN uses an LM for object/count and layout generation before image generation, while VPEVAL creates evaluation programs that invoke skill-specific visual modules and explain results.

    Source: §1 · pp. 1–2, §3 · p. 4, §4 · p. 4

  • Results

    VPGEN improves control of object counts, spatial relations, and sizes over strong T2I baselines, while VPEVAL correlates more highly with human evaluation than single-model methods on both prompt types.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors hope the work fosters future progress on interpretable and explainable T2I generation and evaluation.

    Source: §1 · p. 3

    GLIGEN sometimes fails to generate images properly even when Vicuna produces correct layouts, so VPGEN’s results depend partly on the layout-to-image model.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGen has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEval produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEval provides a more human-correlated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.

1 Introduction

The introduction presents VPGEN for step-by-step T2I generation and VPEVAL for interpretable, explainable evaluation. It reports stronger layout control from VPGEN and higher human correlation from VPEVAL than single-model evaluators.

1 Introduction

  • The authors identify a gap: prior work combined LMs and visual modules for vision-and-language tasks, but not for T2I generation.They propose visual programming frameworks for both T2I generation and evaluation.Source: §1 · p. 1
  • VPGEN decomposes generation into object/count, layout, and image steps, using an LM for the first two and a layout-to-image model for the last.The introduction identifies Vicuna and GLIGEN as examples of those models.Source: §1 · pp. 1–2
  • VPGEN improves control of object counts, spatial relations, and sizes over strong T2I baselines, including Stable Diffusion.In skill-based prompts, it scores higher on Count, Spatial, and Scale; it is competitive on open-ended prompts and performs better when precise layouts are needed.Source: §1 · p. 2
  • VPEVAL uses evaluation programs that combine visual modules for different skills and explain results visually and textually.GPT-3.5-Turbo dynamically generates programs for open-ended prompts without finetuning on evaluation-program annotations.Source: §1 · pp. 2–3
  • Earlier work used single modules for T2I evaluation, which the authors describe as unreliable for some skills and unable to explain score reasoning.Examples include difficulty counting objects or parsing text rendered in images.Source: §1 · p. 2
  • VPEVAL correlates more highly with human evaluation than single-model methods on both skill-based and open-ended prompts.Its evaluation programs invoke modules specialized for different skills and provide visual and textual explanations.Source: §1 · p. 3

2 Related Works

Related work covers T2I models, layout-guided generation, language models paired with visual modules, and automated T2I evaluation. The paper positions VPGEN and VPEVAL as addressing limitations in these approaches.

Text-to-image generation models

  • Recent T2I systems include multimodal language models [43] [22] and diffusion models [44], with systems such as Parti, MUSE, Stable Diffusion, UnCLIP, and Imagen [24] demonstrating photorealism in zero-shot generation.Source: §2 · p. 3

Bridging text-to-image generation with layouts

  • Prior layout-guided T2I methods [25] [26] [27] focus on predefined object classes and cannot place objects unseen during training.VPGEN instead uses an LM to generate objects, counts, and positions in text, adapting pretrained knowledge to diverse region descriptions.Source: §2 · p. 3

Language models with visual modules

  • Prior vision-and-language work uses external visual modules [9] [11], while this paper applies visual programming to interpretable and explainable T2I generation and evaluation.The related-work passage describes earlier work as converting visual inputs into text for LMs to understand.Source: §2 · p. 3

Evaluation of text-to-image generation models

  • Automated T2I evaluation commonly measures visual quality with IS [29] and FID [30], or image-text alignment with a single model [31] [33] [25] [34] [19] [36] [37].VPEVAL instead uses diverse visual modules and evaluation programs with visual and textual explanations.Source: §2 · p. 3

3 VPGEN: Visual Programming for Step-by-Step Text-to-Image Generation

VPGEN represents T2I generation as an interpretable program with object/count, layout, and image-generation stages. Its LM-generated text layouts support objects unseen during layout training and feed a layout-to-image model.

3 VPGEN: Visual Programming for Step-by-Step Text-to-Image Generation

  • VPGEN decomposes T2I generation into object/count generation, layout generation, and image generation.The three modules execute step by step as an interpretable generation program.Source: §3 · p. 3, Figure 2 · p. 4
  • An LM generates objects, counts, and positions as text, enabling layouts for objects unseen during text-to-layout training.The paper uses bounding boxes for layout representation because they generally require fewer tokens than other formats.Source: §3 · p. 4
  • Layout generation has two LM stages: enumerate objects and counts, then generate their bounding boxes using coordinates quantized into 100 bins.Each object is represented by four normalized bounding-box coordinates.Source: §3 · p. 4
  • Vicuna 13B is finetuned with LoRA on 1.2M text-layout examples from Flickr30K Entities, MS COCO Instances 2014, and PaintSkills.The authors state that LoRA preserves the LM’s original knowledge and saves memory during training and inference.Source: §3 · p. 4
  • For image generation, GLIGEN receives bounding boxes, text descriptions for regions, and the original prompt.Source: §3 · p. 4

4 VPEVAL: Visual Programming for Explainable Evaluation of Text-to-Image Generation

VPEVAL evaluates text-to-image outputs through interpretable programs that call visual modules and return scores with visual and textual explanations. It covers five generation skills and extends to open-ended prompts by selecting prompt elements and modules to evaluate.

4.1 Evaluation Skills

  • VPEVAL defines five skills—Object, Count, Spatial, Scale, and Text Rendering—and supports zero-shot evaluation, free-form text prompts, and additional 3D spatial, scale, and text-rendering skills.These latter skills were not supported in PaintSkills [19].Source: §4.1 · p. 5

4.2 Visual Evaluation Modules

  • Eight expert visual modules provide scores and visual-plus-textual explanations, using bounding boxes and templated text.The modules include object detection and OCR, plus VQA and skill-specific checks for objects, counts, spatial relations, scale, and text.Source: §4.2 · pp. 5–6

4.3 Skill-based Evaluation with Visual Programs

  • Skill-based evaluation programs call relevant visual modules for each skill-specific prompt and explain the results visually and textually.The visual explanations show detected-object or text bounding boxes; scores are binary.Source: Figure 3 · p. 5, §4.3 · p. 6

4.4 Open-ended Evaluation with Visual Program Generator LM

  • For open-ended prompts, an LLM generates programs that select prompt elements and matching modules, then average module scores and return explanations.ChatGPT generates programs via in-context learning; the evaluated prompt set includes 160 TIFA v1.0 prompts.Source: §4.4 · p. 6

5 Experiments and Results

VPGEN improves performance on challenging count, spatial, and scale skills, while VPEVAL correlates more strongly with human judgments than single-module evaluation methods. On open-ended prompts, VPGEN performs similarly to Stable Diffusion overall, and GLIGEN can still fail to render images correctly even when layouts are right.

5.1 Evaluated Models

  • The evaluated systems include VPGEN and five public T2I models; Stable Diffusion v1.4 is the closest baseline because GLIGEN uses its frozen backbone with inserted spatial-control adapters.The comparison includes diffusion models and multimodal autoregressive language models.Source: §5.1 · p. 7

5.2 Evaluation on Skill-based Prompts

  • All five baseline models score above 93% on Object except minDALL-E, while Count, Spatial, Scale, and Text Rendering remain challenging.The reported low accuracies on these other skills indicate persistent difficulty for recent T2I models.Source: §5.2 · pp. 7–8
  • +12.7% Count than Karlo, +24.8% Spatial than Stable Diffusion v2.1, and +9.9% Scale than Karlo for VPGEN on skill-based prompts.These results are reported alongside high Object-skill accuracy; all models, including VPGEN, score low on Text Rendering.Source: §5.2 · p. 8
  • VPGEN outperforms Stable Diffusion v1.4 on every fine-grained Count, Spatial, and Scale split.VPGEN exceeds 50% accuracy at each tested count; it does better on 2D than 3D spatial relations and on different-size than same-size pairs.Source: Figure 6 · p. 8, §5.2 · p. 8

5.3 Evaluation on Open-ended Prompts

  • On open-ended prompts, VPGEN performs similarly to Stable Diffusion overall; on PaintSkills, it scores 71.0 versus 63.5 for Stable Diffusion v1.4 and 71.0 versus 68.4 for v2.1.Object descriptions and attributes make up 86.4% of the evaluated prompt elements, while spatial layouts account for 13.6%.Source: Table 2 · p. 8, §5.3 · p. 8
  • GLIGEN sometimes fails to generate a proper image even when Vicuna produces a correct layout, identifying image generation as a remaining error source.The authors state that better layout-to-image models would improve VPGEN results.Source: §5.3 · p. 9

5.4 Human Evaluation of VPEVAL

  • 66.6 overall Spearman’s ρ for VPEVAL, higher than single-module CLIP, captioning, and VQA evaluations on skill-based prompts.VPEVAL is especially strongly correlated with human judgments on Count and Text Rendering.Source: §5.4 · p. 9, Table 3 · p. 10
  • VPEVAL correlates better with human judgments than TIFA (BLIP-2) on open-ended prompts, and VPEVAL† achieves an even higher correlation.The study also finds high prompt-element coverage and high per-module accuracy compared with human judgment.Source: Table 4 · p. 10, §5.4 · p. 10

6 Conclusion

The paper presents VPGEN for interpretable text-to-image generation and VPEVAL for explainable evaluation; its qualitative examples compare VPGEN and Stable Diffusion on skill-based and open-ended prompts.

6 Conclusion

  • VPGEN decomposes text-to-image generation into three steps, while VPEVAL uses diverse visual modules and provides visual and textual explanations of evaluation results.The conclusion reports that VPGEN better follows descriptions, especially for object counts, spatial relations, and sizes, while VPEVAL correlates more highly with human judgments on skill-specific and open-ended prompts.Source: §6 · p. 10

Appendix

  • The appendix includes qualitative examples, additional VPGEN analyses, error analyses for VPGEN and VPEVAL, implementation details, and limitations and broader impacts.It also lists TIFA prompt-element analysis details and license information.Source: Appendix · p. 15

A Qualitative Examples

  • Figures 8 and 9 compare VPGEN and Stable Diffusion v1.4 on skill-based and open-ended prompts, respectively.Figure 10 shows VPGEN generations involving objects absent from Vicuna 13B’s text-to-layout fine-tuning annotations.Source: A Qualitative Examples · p. 15, Figure 8 · p. 15

B Additional VPGEN Analysis

Additional analyses examine VPGEN’s counting, layout quantization, layout-generation LM, image quality, and training data; they report skill and data-dependent trade-offs.

Counting prompts: More than 4 objects & Unspecified number of objects

  • VPGEN generates layouts and images from prompts without explicit counts, producing two Pikachus for “Pikachus on the table”; counted prompts include counts up to 7, with higher counts more challenging.Source: B Additional VPGEN Analysis · p. 15

Qualitative examples

  • Figure 10 shows VPGEN generations from prompts containing Pikachu, Link, and master sword, objects absent from the text-to-layout fine-tuning annotations.Source: Figure 10 · p. 16

Bounding box quantization

  • Increasing bounding-box quantization from 100 to 1000 bins does not increase accuracy on skill-based or open-ended prompts.The authors infer that the current 100-bin quantization does not hurt object-placement accuracy.Source: B Additional VPGEN Analysis · p. 16

GPT-3.5-Turbo v.s. Vicuna 13B

  • Vicuna 13B-based VPGEN has higher skill-based and open-ended VPEval accuracies than GPT-3.5-Turbo-based VPGEN.The comparison uses Vicuna trained on Flickr30k+COCO+PaintSkills and GPT-3.5-Turbo with 36 in-context examples.Source: B Additional VPGEN Analysis · p. 16

Visual quality metric (FID)

  • Both VPGEN checkpoints have better skill-based accuracy than Stable Diffusion v1.4 and comparable open-ended accuracy; FID ranks VPGEN (Flickr30k) < Stable Diffusion v1.4 < VPGEN (Flickr30k+COCO+Paintskills).The authors attribute the combined-training checkpoint’s FID increase to PaintSkills layouts differing from natural-scene layouts.Source: B Additional VPGEN Analysis · pp. 16–17

Training with real v.s. synthetic images

  • PaintSkills-only training scores 46.0 average accuracy on skill-based prompts versus Flickr30K’s 43.9, while Flickr30K scores 71.0 on open-ended prompts versus PaintSkills’ 65.6.The authors interpret this as synthetic training injecting specific skills such as counting and spatial relations, while not covering diverse objects and attributes as well on its own.Source: B Additional VPGEN Analysis · p. 17

C VPGEN Error Analysis

VPGEN’s error analysis finds that image rendering is a major source of errors, especially for counting and spatial skills, while identifying distinct layout and rendering failures.

Layout v.s. Image generation on skill-based prompts

  • The analysis tracks object recall, object count, layout accuracy, and final-image accuracy to locate errors across VPGEN’s generation steps.Object recall and count assess generated layouts; layout accuracy evaluates layouts, while image accuracy evaluates GLIGEN’s final images.Source: C VPGEN Error Analysis · p. 17
  • Layout accuracy is much higher than image accuracy, especially for Count and Spatial skills, implicating GLIGEN’s rendering step as a major error source.The authors suggest a more accurate layout-to-image model could improve VPGEN’s pipeline accuracy.Source: C VPGEN Error Analysis · p. 17

Human evaluation on open-ended prompts

  • On 50 randomly sampled TIFA160 prompts, Vicuna 13B layout accuracy is 92%, above GLIGEN image correctness at 65%.Two expert annotators reported Cohen’s κ and Krippendorff’s α of 0.73 for layout ratings and 0.87 for image ratings.Source: C VPGEN Error Analysis · p. 17

Common error categories

  • Vicuna 13B sometimes adds or misses prompted objects, while GLIGEN can generate wrong objects, omit objects, or miss object details.Examples include a suitcase instead of a bench and a closed oven instead of an oven with its door open.Source: C VPGEN Error Analysis · p. 18

Challenging, non-canonical prompts

  • VPGEN captures some elements of challenging non-canonical prompts but can miss exact relations, such as a circle being twice another’s size.The authors suggest training prompts may not include many examples in these styles.Source: C VPGEN Error Analysis · p. 18

D VPEVAL Error Analysis

Human annotators found that VPEVAL programs covered most prompt elements and that sampled evaluation modules were often accurate.

Evaluation program analysis: coverage and accuracy

  • On average, VPEVAL programs cover 94% of prompt elements, and evaluation modules are accurate 83% of the time.Two expert annotators assessed 50 randomly sampled TIFA160 prompts; module accuracy checks covered three randomly sampled modules per program.Source: D VPEVAL Error Analysis · p. 18

E VPGEN/VPEVAL Implementation Details

VPGen implementation uses a finetuned Vicuna model and GLIGEN, while VPEVAL details specify benchmark prompts, baseline procedures, runtime, and program-generation cost. The appendix also documents representative generation errors and limits on depth information.

E.1 VPGEN Details

  • 1.2M text-layout examples from Flickr30K Entities, COCO 2014, and PaintSkills are used to train Vicuna 13B; the maximum count per object class is 7.The validation set uses 2,000 randomly selected examples; training on all three datasets runs for 2 epochs, while Flickr30K-only training runs for 6 epochs.Source: §E.1 · pp. 18–19
  • Table 8 compares VPGen with Stable Diffusion v1.4 using VPEval accuracy and FID, while Table 9 compares Vicuna training with real versus synthetic data.The captions identify the tables’ scope but do not report their cell values.Source: Table 8 · p. 19, Table 9 · p. 19
  • 6s: VPGEN takes around 6s to generate an image on one A6000 GPU, with 2s for Vicuna 13B and 4s for GLIGEN.This is the reported end-to-end inference time.Source: §E.1 · p. 19

E.2 VPEVAL Details

  • VPEVAL’s skill-based benchmark uses prompts for object, count, spatial, size, and text skills, with 400, 1,000, 1,000, 1,000, and 403 prompts, respectively.Templates combine objects, counts, relations, scale, and text; the benchmark is described as extendable to additional elements.Source: §E.2 · p. 19
  • For VQA baselines, BLIP-2 answers yes/no questions about each object and their relation; CLIP scores image-prompt cosine similarity.Spatial and scale examples require yes answers for both object-presence questions and the relation question.Source: §E.2 · pp. 19–20
  • Vicuna sometimes adds or omits prompted objects, while GLIGEN can substitute objects, omit them, or miss requested details; Vicuna does not generate depth information for front/behind splits.The examples include an unwanted chair, a missing building, a suitcase instead of a bench, and an oven with the wrong door state.Source: Table 10 · p. 19, Figure 12 · p. 20, Figure 13 · p. 20
  • Each VPEVAL module typically runs in less than 1s, and GPT-3.5 Turbo program generation costs about $0.007 per prompt under the stated token setting.The cost estimate uses 4K input tokens and 0.5K output tokens.Source: §E.2 · p. 20

F TIFA Prompt Element Analysis Details

A sample of open-ended TIFA prompts contained relatively few elements requiring spatial-layout understanding. The supplied figures also show VPGen outputs for challenging prompts and VPEVAL programs with module outputs.

F TIFA Prompt Element Analysis Details

  • 11/81 elements (13.6%) in 20 sampled TIFA prompts concerned object layouts; the other 70 concerned objects or attributes.Each prompt could contain multiple elements, including objects, attributes, and spatial relations.Source: F TIFA Prompt Element Analysis Details · p. 20
  • Figure 14 presents VPGen images generated from challenging, non-canonical prompts.Source: Figure 14 · p. 21
  • Figure 15 shows VPEVAL evaluation programs and module outputs, with modules covering different parts of prompts.The caption directs readers to Appendix D for discussion of module coverage.Source: Figure 15 · p. 21

G Limitation and Broader Impacts

The authors identify language and image-domain boundaries for their modules, note the cost of generating evaluation programs, and describe plans for public release and licensing information.

G Limitation and Broader Impacts

  • Because the LMs were trained mainly on English-heavy data and modules mainly on natural images, non-English prompts and domains such as fine arts or medical images may be poorly handled.The framework supports updating modules or adding domain-specific ones.Source: G Limitation and Broader Impacts · p. 21
  • LLM-based evaluation-program generation can cost computational resources and API fees; releasing programs and a locally runnable finetuned LM are proposed ways to avoid regeneration.Source: G Limitation and Broader Impacts · p. 21
  • VPGen may support user-controlled image generation or data augmentation, but could also be used to create false or misleading images.The passage frames this risk as shared with other image-generation frameworks.Source: G Limitation and Broader Impacts · p. 21

H License

  • Table 11’s prompt templates use grammar tokens such as <a>, replaced with “an” or “a” to maintain correct grammar.Source: Table 11 · p. 22
  • The authors plan to make code and models public and provide licenses for the datasets, code, and models; listed examples include Apache 2.0 for FastChat (Vicuna), Grounding DINO, and Diffusers.The listed LAVIS (BLIP-2) license is BSD 3-Clause.Source: H License · p. 22

Related papers

Loading 2305.15328v2…