PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMGA: Multimodal Large Language Model based Generation Assistant

Bin Xia, Shiyin Wang, Yingfan Tao, Yitong Wang, Jiaya Jia

arXiv:2311.16500v4cs.CV

TL;DR

Existing MLLM approaches use fixed-size visual embeddings to control image generation, while LLMGA aims to assist users with generation and editing through detailed language prompts. It combines task-specific MLLM training, Stable Diffusion adaptation, and reference-based restoration; the authors report capabilities across generation and editing tasks.

  • Problem

    Existing approaches use MLLMs to output images rather than assist users in generating or editing images, and some control Stable Diffusion through fixed-size visual embeddings.

    Source: §1 · p. 2

  • Method

    LLMGA trains an MLLM on generation and editing tasks, adapts Stable Diffusion to its detailed language prompts, and uses DiffRIR with masked-image references for restoration.

    Source: §1 · p. 3

  • Results

    The authors report a unified interactive framework with generation and editing capabilities, and visual results demonstrating DiffRIR’s handling of texture, brightness, and contrast disparities.

    Source: §1 · pp. 3, 15

  • Takeaways & Limitations

    The framework lets users interactively refine image generation and editing and can integrate with external plugins such as ControlNet.

    Source: §1 · p. 3

Abstract

from arXiv · show

In this paper, we introduce a Multimodal Large Language Model-based Generation Assistant (LLMGA), leveraging the vast reservoir of knowledge and proficiency in reasoning, comprehension, and response inherent in Large Language Models (LLMs) to assist users in image generation and editing. Diverging from existing approaches where Multimodal Large Language Models (MLLMs) generate fixed-size embeddings to control Stable Diffusion (SD), our LLMGA provides a detailed language generation prompt for precise control over SD. This not only augments LLM context understanding but also reduces noise in generation prompts, yields images with more intricate and precise content, and elevates the interpretability of the network. To this end, we curate a comprehensive dataset comprising prompt refinement, similar image generation, inpainting \& outpainting, and instruction-based editing. Moreover, we propose a two-stage training scheme. In the first stage, we train the MLLM to grasp the properties of image generation and editing, enabling it to generate detailed prompts. In the second stage, we optimize SD to align with the MLLM's generation prompts. Additionally, we propose a reference-based restoration network to alleviate texture, brightness, and contrast disparities between generated and preserved regions during inpainting and outpainting. Extensive results show that LLMGA has promising generation and editing capabilities and can enable more flexible and expansive applications in an interactive manner.

1 Introduction

LLMGA is an interactive assistant for image generation and editing that guides Stable Diffusion with detailed language prompts rather than fixed-size visual embeddings. The paper also introduces task-specific training data and a reference-based restoration network for edited images.

1 Introduction

  • Unlike methods that use fixed-size visual embeddings for implicit Stable Diffusion control [30] [55], LLMGA supplies detailed language prompts for explicit generation guidance.Source: §1 · p. 2
  • The authors identify MLLM instruction refusal and imprecise prompts as challenges, alongside selecting prompt text and fitting detailed prompts into Stable Diffusion’s 75-token CLIP encoding limit.LLMGA prompts typically exceed 150 tokens, while Stable Diffusion was trained on short captions.Source: §1 · p. 3
  • DiffRIR uses masked images as references and contrast and brightness perturbations during training to alleviate texture, contrast, and brightness disparities between generated and preserved regions.Source: §1 · pp. 3–4
  • LLMGA is presented as a unified interactive framework for image generation and editing, with plugin integration and user interaction for flexible image design.Source: §1 · p. 3
  • The training data covers prompt refinement, similar-image generation, inpainting and outpainting, and instruction-based editing; a two-stage scheme trains the MLLM and then adapts Stable Diffusion to its prompts.Source: §1 · p. 3
  • The authors state that they release generated data, model-training code, a checkpoint, and a demo.Source: §1 · p. 4

2 Related Work

LLMGA uses detailed language prompts to guide image generation and editing, supported by training for both the MLLM and Stable Diffusion. Its restoration network uses preserved image content to address differences between generated and preserved regions.

2 Related Work

  • Prior MLLMs encode visual inputs or invoke pretrained tools, but are not tailored to image generation and editing assistance with prompts intended to enhance results.LLMGA is designed for these tasks and flexible user interaction.Source: §2 · pp. 4–5

3.1 Overview of LLMGA

  • LLMGA uses detailed language prompts rather than fixed-size visual embeddings to control Stable Diffusion, making its prompts interpretable and dynamically sized.The paper also presents language prompts as better suited to the MLLM’s text pretraining and context comprehension.Source: §3.1 · p. 5

3.2 MLLM Training

  • The MLLM is trained on prompt-refinement, similar-image-generation, inpainting/outpainting, and instruction-based-editing data to improve its understanding of generation and editing tasks.The training also includes question-answering and visual question-answering data.Source: §3.2 · pp. 6–7

3.3 Stable Diffusion Training

  • Because Stable Diffusion’s text encoder handles 75 tokens while LLMGA prompts can exceed 150, the authors repeatedly encode prompts and train text-to-image and inpainting models on them.They optimize the Stable Diffusion U-Net while freezing its other networks; inpainting and outpainting conditioning also includes the mask and VAE-encoded masked image.Source: §3.3 · pp. 7–8

3.4 Restoration Network Training

  • DiffRIR restores edited images using both the low-quality image and preserved masked-image content as inputs.Its training adds random brightness and contrast disturbances, and its structure is based on DiffIR.Source: §3.4 · p. 8

4 Experiments

Experiments evaluate LLMGA’s generation and editing performance, the two-stage training scheme, its ControlNet integration, and DiffRIR restoration. The reported comparisons cover T2I generation, instruction-based editing, inpainting and outpainting, and image restoration.

4.1 Implementation Details

  • Implementation uses LLaVA-1.5-7B or -13B for stage one, SD1.5 for stage two, and DIV2K and Flickr2K to train DiffRIR.The reported training runs are 1 × 10^5 SD1.5 iterations and 4 × 10^5 DiffRIR iterations.Source: §4.1 · p. 9

4.2 Experimental Results

  • LLMGA-7B with SD1.5-ft reports 5.0712 FID and 3.92 IS improvements, and LLMGA-7B significantly outperforms GILL in T2I generation.The paper also reports 5.4847 FID and 5.05 IS improvements for LLMGA.Source: Table 1 · p. 9, Table 5.4847
  • T2I visual comparisons show LLMGA refining prompts with scene details, including battle attire and a monster-battling scenario for a husky.The authors also describe generation of concepts such as a space elevator.Source: §4.2 · p. 9, Figure 3 · p. 10
  • In zero-shot MagicBrush editing, LLMGA surpasses InstructPix2Pix and achieves performance similar to MagicBrush without training SD for editing.The authors describe the visual editing results as more accurate and reasonable.Source: §4.2 · p. 10, Table 2 · p. 10, Figure 4 · p. 11
  • LLMGA-7B improves outpainting FID by 2.5681 over SD1.5, while SD1.5-ft further improves FID over SD1.5 in both inpainting and outpainting.LLMGA-13B also outperforms LLMGA-7B in these comparisons.Source: §4.2 · pp. 10–11, Table 3 · p. 11
  • DiffRIR’s reference scheme improves restoration, and color degradation during training helps alleviate brightness and contrast distortion in LLMGA-generated outpainting.The authors report that DiffRIR alleviates texture, brightness, and contrast discrepancies while generating realistic details.Source: §4.2 · p. 12, Table 4 · p. 12, Figure 7 · p. 13
  • LLMGA prompts used with ControlNet enhance the diversity and richness of picture-reference-guided generation outcomes.The visual comparison also describes LLMGA plus ControlNet as enhancing generated-image details.Source: §4.2 · p. 12, Figure 6 · p. 13
  • Embedding-based image quality deteriorates as conversation turns increase, while LLMGA’s language-prompt control remains unaffected.The comparison uses repeated MSCOCO prompts in conversational form.Source: §4.2 · pp. 13–14, Table 5 · p. 13

5 Conclusion

LLMGA uses detailed language prompts and two-stage training to support image generation and editing, alongside DiffRIR for restoration. The conclusion presents the system as an interactive assistant that offers design suggestions and enhances results based on user requests.

5 Conclusion

  • LLMGA guides Stable Diffusion with detailed language prompts rather than fixed-size visual embeddings, using two training stages to align the MLLM and SD.The first stage trains the MLLM to understand generation and editing properties; the second optimizes the SD UNet to adapt to its prompts.Source: §5 · p. 14
  • The training dataset covers prompt refinement, similar image generation, inpainting and outpainting, and instruction-based editing.The conclusion describes these capabilities as supporting fitting responses to image-generation and editing requests.Source: §5 · p. 14
  • DiffRIR is proposed to mitigate texture, contrast, and brightness disparities between generated and preserved regions during image editing.The conclusion says LLMGA can offer design suggestions and enhance results based on user requests during interactions.Source: §5 · p. 14

1 Overview

The supplemental materials outline implementation, dataset construction, and evaluations, followed by additional comparisons and visual examples. They also cover interactive generation and editing, including examples using SDXL.

Supplemental Material

  • The supplement provides implementation details and a comprehensive account of training-dataset creation.It also includes a discussion comparing BLIP-diffusion with LLMGA and reports VQA performance.Source: §1 · p. 15
  • Additional material explains instruction-based editing, training and evaluation, and SDXL training.It also expands the comparison between LLMGA and LLMGA Embedding.Source: §1 · p. 15
  • Supplemental examples cover picture-book creation, T2I generation, instruction-based editing, and inpainting and outpainting.These are listed as additional visual results and capabilities.Source: §1 · p. 15
  • Further examples address DiffRIR’s handling of brightness, contrast, and texture differences, plus interactive generation and editing with SDXL.The overview identifies these as material in Sections 13 and 14.Source: §1 · p. 15

2 Implementation Details

The implementation trains an MLLM first, then fine-tunes SD1.5 for generation or inpainting and outpainting, and trains DiffRIR separately for restoration. The passages specify the models, datasets, and key training settings for these components.

Implementation Details

  • Stage one starts from pretrained LLaVA-1.5-7B or LLaVA-1.5-13B and trains with AdamW at 2×10−5, batch size 16, and one epoch.The training ratios are 1 for VQA and 0.3 each for general/design QA, prompt refinement, similar image generation, inpainting and outpainting, and instruction-based editing.Source: §2 · pp. 15–16
  • Stage two starts from SD1.5 and trains for 1 × 10^5 iterations with AdamW at 1 × 10−5 and batch size 32.This stage targets image generation or inpainting and outpainting models.Source: §2 · p. 16
  • DiffRIR is trained on DIV2K and Flickr2K with a GAN-based loss, batch size 64, 64 × 64 LQ patches, and 4 × 10^5 iterations.The optimizer is Adam with learning rate 2 × 10−4.Source: §2 · p. 16

3 Data

LLMGA’s training data combines detailed image descriptions, instruction-based editing examples, and design and graphic-story material. These sources support image understanding, editing, and varied design tasks.

3 Data

  • MSCOCO captions paired with GPT4-V-generated detailed descriptions provide training data intended to help the LLM understand image composition and support generation and editing.GPT4-V expands short captions into visually compelling descriptions.Source: §3 · p. 16
  • Mixtral7Bx8 was fine-tuned on InstructPix2Pix editing data to generate editing examples and target descriptions, which were filtered for valid output formats.The resulting instruction-based editing dataset was created from detailed descriptions of MSCOCO images.Source: §3 · p. 16
  • Design books and graphic-story datasets supply QA training data for tasks including logo, fashion, infographic, and poster design, as well as illustrated storybooks.Storybooks intertwine text and images and are tailored to user specifications.Source: §3 · p. 16

4 Discuss BLIP-diffusion and LLMGA.

LLMGA is presented as a conversational designer that retains LLM cognitive abilities, unlike approaches that primarily use MLLMs as image and text encoders. The authors also report better subject-driven generation outcomes with IP-Adapters integrated into their SD fine-tuning setup.

4 Discuss BLIP-diffusion and LLMGA.

  • Unlike BLIP-diffusion, which primarily uses MLLMs as image and text encoders, LLMGA retains multi-turn conversation and general LLM cognitive ability for conversational image design.The authors frame LLMGA as a designer and assistant that produces images through conversation.Source: §4 · pp. 16–17
  • Subject-driven generation can use IP-Adapters integrated into SD-ft, and the authors report better outcomes for LLMGA in Fig. 3.The authors say they will add discussion of these methods to the paper.Source: §4 · p. 17

5 VQA Evaluation.

The section reports that LLMGA performs better than other MLLMs on VQA and image generation, then details instruction-based editing and SDXL training. It also contrasts language-prompt control with embedding-based control, which the authors say lags and degrades over dialogue turns.

5 VQA Evaluation.

  • LLMGA performs much better than other MLLMs on both VQA and image generation, according to the reported comparison.Table 1 is identified as the VQA comparison; the supplied passage does not give its cell values.Source: §5 · p. 17, Table 1 · p. 18

6 More Details on Instruction-based Editing

  • For instruction-based editing, LLMGA produces an image description and edited target description, then uses Direct Inversion and prompt-to-prompt to edit the image.The initial noise map comes from Direct Inversion on the original image.Source: §6 · pp. 17–18

7 More Training and Evaluation Details

  • LLMGA (SDXL) follows a similar training process to SD1.5 and is reported to generate higher-quality images.Its SDXL diffusion model is trained on LAION-Aesthetic prompts from the first-stage MLLM, with 1024 × 1024 input patches.Source: §7 · pp. 18–19

8 More Details on Control Scheme

  • The control comparison considers language-generated prompts and visual embeddings, with embedding-based approaches reported to lag behind LLMGA and decline in accuracy and quality as dialogue turns increase.The authors attribute the decline to added noise in predicted embeddings as more prior conversation is incorporated.Source: §8 · pp. 19–20
  • LLMGA’s detailed language prompts are presented as more transparent and interactive than implicit embeddings, while embedding noise can be filtered by mapping to a fixed language domain.The embedding-based alternative uses a three-stage training scheme: text and embedding training, projection-layer optimization, then SD U-Net optimization.Source: §8 · pp. 19–21
  • Dynamic-sized language prompts can accommodate added generation requests during interaction, and the authors describe their training as simpler and more efficient.The paper also argues that explicit prompts better suit MLLM context comprehension than implicit embeddings.Source: §8 · p. 21

9 Showcasing LLMGA’s Integrated Image and Text Picture Book Creation and Design Capabilities

LLMGA is shown creating picture books that intertwine text and illustrations from a single instruction, and generating story illustrations interactively from user-provided snippets. It is also presented across a range of image-design tasks.

9 Showcasing LLMGA’s Integrated Image and Text Picture Book Creation and Design Capabilities

  • From a single instruction, LLMGA can create a storybook interweaving text and illustrations, and interactively generate illustrations from user-input story snippets.The paper presents these capabilities as a picture-book showcase.Source: §9 · p. 21
  • LLMGA’s design examples span logo, game-character, poster, T-shirt, and infographic design.The showcase is identified in Fig. 6.Source: §9 · p. 21, Figure 6 · p. 22

10 More Results on T2I Generation

LLMGA uses its comprehension, reasoning, and knowledge to generate detailed visuals and refine prompts to users’ requirements. It also produces accurate, visually pleasing instruction-based edits.

10 More Results on T2I Generation

  • LLMGA leverages its comprehension, reasoning, and knowledge to generate visuals with more details.The visual results are shown in Fig. 7.Source: §10 · p. 22
  • LLMGA refines prompts based on user requirements.Source: §10 · p. 22

11 More Results on Instruction-based Editing

  • LLMGA realizes more accurate and visually pleasing edits according to user requirements.The visual results are shown in Fig. 8.Source: §11 · p. 22

12 More Results on Inpainting and Outpainting

LLMGA infers plausible complete images from masked inputs and supports edits according to user specifications. DiffRIR addresses discrepancies between generated and preserved regions, while interactive examples show LLMGA assisting users in image design.

12 More Results on Inpainting and Outpainting

  • LLMGA infers plausible complete images from masked inputs and edits images according to user specifications and masks.Visual results are shown in Fig. 9 and paper Fig. 1.Source: §12 · p. 22, Figure 9 · p. 26

13 More Results on Image Restoration

  • DiffRIR alleviates texture, brightness, and contrast discrepancies and generates more realistic details.The restoration results are shown in Fig. 10.Source: §13 · p. 23, Figure 10 · p. 27

14 More Results on Interactive Generation and Editing

Related papers

Loading 2311.16500v4…