PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, William Yang Wang

arXiv:2305.15393v2cs.CVcs.AI

TL;DR

Fine-grained layouts improve visual control but burden users, while existing LLM-based systems do not reliably produce compositional layouts from language alone. LayoutGPT uses CSS-style in-context demonstrations to plan layouts, improving 2D image composition and matching supervised methods in 3D indoor scene synthesis.

  • Problem

    Existing visual generators struggle with specified counts, positions, attributes, and coherent room-scale furniture arrangements, while layout-based control burdens users with fine-grained inputs.

  • Method

    LayoutGPT uses training-free in-context learning with CSS-style structured programs and retrieved demonstrations to generate text-conditioned layouts across 2D and 3D domains.

  • Results

    LayoutGPT improves counting and spatial faithfulness in 2D image generation and achieves performance comparable to supervised methods in 3D indoor scene synthesis.

  • Takeaways & Limitations

    LLMs can serve as visual planners for compositional layout generation across image-space and 3D indoor-scene tasks.

  • Takeaways & Limitations

    Limited in-context demonstrations restrict LayoutGPT’s coverage of furniture distributions, causing it to avoid rarely observed objects.

Abstract

from arXiv · show

Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

1 Introduction

LayoutGPT is a training-free, program-guided method that uses in-context demonstrations to enable LLMs to plan visual layouts from text across 2D images and 3D indoor scenes. It addresses compositional failures in existing text-to-image and scene-synthesis systems while improving numerical and spatial faithfulness and achieving strong 3D performance.

  • Motivation: Existing text-to-image models struggle with specified object counts, positions, and attributes, while 3D scene synthesis models struggle to preserve furniture within predefined room sizes [10] [24] [30].
  • Motivation: Visual layouts provide symbolic representations of visual composition, but existing layout models are restricted to discrete categories or lack reasoning for complicated text conditions [33] [45] [34].
  • LayoutGPT: LayoutGPT is a training-free approach that injects visual commonsense into LLMs through in-context demonstrations, enabling desirable layout generation from text without image-data training.The method represents image layouts in a style sheet language format compatible with LLMs.
  • 3D Extension: LayoutGPT extends from 2D layout planning to 3D indoor scene synthesis, handling depth, furniture sizes, and coherent arrangements across room types, with performance comparable to a state-of-the-art supervised method.
  • Contributions: LayoutGPT improves counting and spatial-relation faithfulness in 2D image generation and achieves strong 3D indoor scene-synthesis performance; the NSR-1K benchmark evaluates counting and positional relations [10, 24].The benchmark contains prompts characterizing numerical and positional relations for text-to-image generation.

2 Related Work

Prior work addresses layout generation, compositional image generation, indoor scene synthesis, and adapting LLMs to vision-language tasks. These directions span trained layout models, methods for correcting compositional failures, 3D furniture synthesis, and multimodal LLM adaptation.

  • Image Layout Generation: Layout generation methods target indoor scenes [40], document layouts [15], and graphical user interfaces, often using models trained from scratch.LayoutGAN generates class and geometric labels for wireframe boxes with a fixed number of scene elements.
  • Compositional Image Generation: Text-to-image models exhibit compositional failures including missing objects, incorrect spatial relations, and incorrect attributes [24] [2].StructureDiffusion [10] adjusts text embeddings using linguistic structure, while Attend-and-Excite [4] optimizes attention regions so objects occupy separate regions.
  • Indoor Scene Synthesis: Indoor scene synthesis seeks functional 3D furniture layouts, progressing from supervised visual-map prediction [40] to transformer-based scene generation [47] [38].SceneFormer [47] adds furniture with transformers, while ATISS [38] uses a single transformer to generate multiple object attributes.
  • LLMs for Vision: LLMs have become important in vision-language tasks [11] [28] [14], motivating adaptations for multimodal tasks [31] [55].Multimodal chain-of-thought [58] incorporates visual inputs as rationales for question answering, while learns translation parameters between visual and language embeddings.

3 Method

LayoutGPT formulates layout generation as predicting object tuples for 2D image planning and 3D scene synthesis, using structured CSS-based prompts with task instructions, normalized values, and retrieved in-context exemplars. Generated layouts are then converted into images or rendered 3D scenes through downstream models and retrieved objects.

  • Layout generation: LayoutGPT predicts object-layout tuples conditioned on text for 2D image planning and on room specifications for 3D scene synthesis.For images, each tuple contains an object category, 2D location, and 2D size; 3D tuples include category, location, orientation, and size.
  • Task Instructions & Normalization: LayoutGPT prompts combine task instructions, normalized values capped at 256px, and in-context CSS exemplars.Instructions specify the task goal, standard format, and units; property values are normalized using a fixed scalar before rescaling.
  • CSS Structures: The framework structures layouts as CSS-like representations so language models can interpret the physical meaning of spatial values rather than an underspecified value sequence.The method replaces plain autoregressive sequences such as (c1, x1, y1, w1, h1, c2, x2, ...) with a format motivated by CSS structures.
  • Demonstration retrieval: For in-context learning, the method retrieves supporting demonstrations by measuring distances between test and demonstration conditions.For 2D text-conditioned layouts, CLIP features are used to compare the test caption with demonstration images through cosine similarity.
  • Downstream generation: Generated 2D layouts feed a layout-to-image model, while predicted 3D layouts guide object retrieval and direct scene rendering.The 3D pipeline retrieves objects according to predicted category, location, orientation, and size before rendering the scene.

4 LayoutGPT for Text-Conditioned Image Synthesis

LayoutGPT is evaluated for 2D text-to-image synthesis against end-to-end and layout-guided baselines, achieving stronger numerical, spatial, image-level, and attribute correctness while supporting diverse generation scenarios. Its CSS-style prompting improves spatial reasoning, and its generated layouts remain effective across downstream image-generation models.

  • Datasets & Benchmarks: The NSR-1K benchmark evaluates specified object counts and spatial locations using template-based and natural prompts derived from MSCOCO [29].Layout evaluation reports precision, recall, and accuracy for counts and spatial positions, while image evaluation uses GLIP [26] detections and average accuracy.
  • Quantitative Results: 20-40% higher GLIP-based accuracy and 1-6% higher CLIP similarity let LayoutGPT surpass end-to-end text-to-image models.Among LayoutGPT variants, GPT-3.5 performs best on numerical reasoning, whereas GPT-4 performs best on spatial-position correctness; LayoutGPT also outperforms LayoutTransformer by large margins.
  • Qualitative Results: LayoutGPT handles visual commonsense, complex multi-object relations, dense scenes, regional descriptions, and counterfactual prompts while producing layouts similar to human designs.It predicts numerous scene objects from sparse prompts, enriches objects with fine-grained regional details, and generates reasonable layouts for counterfactual relationships.
  • Quantitative Results: 100% attribute-binding accuracy on HRS color prompts enables LayoutGPT with GLIGEN/ReCo to improve multi-object color correctness over Stable Diffusion.The final-image attribute correctness evaluation uses HRS-Bench prompts and a hue-based classifier.
  • Component Analysis: Task instructions and CSS-formatted in-context demonstrations improve spatial layout accuracy, with CSS style identified as the most essential component.Normalization alone degrades accuracy but slightly improves performance when combined with other components.
  • Model-Agnostic Property: LayoutGPT is model-agnostic: its layouts work with Layout-Guidance, showing only a minor GLIP-accuracy gap and comparable CLIP similarity to ground-truth layouts.The comparison uses the same LayoutGPT-generated layouts with Layout-Guidance and evaluates image-level metrics.

5 LayoutGPT for Indoor Scene Synthesis

LayoutGPT outperforms ATISS on spatial validity and rendered-scene quality for bedroom and living-room synthesis, while ATISS better matches furniture-category distributions. Qualitative and conditional-generation results show that LayoutGPT captures 3D commonsense, follows enumerated furniture captions, and coherently completes partial scenes.

  • Quantitative Results: 43.26% vs. 49.88% and 64.16% vs. 83.02% out-of-bound rates, plus FID scores of 28.37 vs. 30.02 and 76.34 vs. 85.40, show LayoutGPT outperforming ATISS for bedrooms and living rooms.These results verify stronger spatial reasoning and higher rendered-scene quality; the living-room split contains more objects on average.
  • Qualitative Results: LayoutGPT understands 3D relations such as pendant lamps hanging from ceilings, nightstands beside headboards, and functional furniture arrangements across rooms.It also plans living-and-dining scenes with dining furniture on one side and seating and entertainment furniture on the other.
  • Quantitative Results: ATISS achieves lower KL divergence, whereas LayoutGPT avoids extremely rare furniture because its limited demonstrations do not fully represent furniture distributions.The distribution mismatch is attributed to the limited size of the in-context demonstrations.
  • Text-guided Synthesis: LayoutGPT follows captions enumerating complete furniture lists and achieves a KL divergence value close to zero.This demonstrates text-guided furniture arrangement in 3D scenes.
  • Partial Scene Completion: LayoutGPT autoregressively completes partial scenes coherently while learning symmetry, positional relations, and room-function commonsense from demonstrations.Examples include symmetric nightstands, a stool at the end of a bed, and desks and chairs in dining areas.

6 Conclusion

LayoutGPT turns LLMs into visual planners through in-context learning and CSS-style prompts, enabling collaboration with visual generative models across image space and 3D indoor scenes. It generates plausible visual arrangements and can improve image compositions through accurate layout generation.

  • Conclusion: LayoutGPT uses in-context learning and CSS-style prompts to turn LLMs into visual planners collaborating with visual generative models.The approach generates plausible visual arrangements in both image space and 3D indoor scenes and can improve image compositions through accurate layout generation.

A Implementation Details … B.3 GPT-3.5/4 Prompting

The paper specifies LayoutGPT’s prompting and evaluation setup for 2D and 3D layout planning, including CSS-structured instructions, task-specific data construction, and standardized metrics. It also examines prompt components, exemplar selection, and domain-specific instructions for challenging scene layouts.

  • A Implementation Details: LayoutGPT’s task instructions combine verbal descriptions with formal CSS definitions, while indoor synthesis additionally supplies furniture categories and normalized frequencies with little observed effect on generation.The frequency distribution produces only a trivial change in KL divergence according to the supplied passage.
  • A Implementation Details: The implementation evaluates Codex, GPT-3.5, GPT-3.5-chat, and GPT-4 as base LLMs, with conversational models receiving in-context exemplars as multiple turns.The supplied passage identifies the four model variants and their respective training or optimization characteristics.
  • A Implementation Details: All LLMs use temperature 0.7 without next-token penalties; image-layout evaluation uses 16 exemplars for numerical reasoning and 8 for spatial reasoning, while exemplar-count gaps are not significant.Five different layouts/images are generated per prompt, according to the supplied hyperparameter description.
  • B.1 NSR-1K Benchmark Construction: NSR-1K is constructed from MSCOCO captions and bounding-box annotations to test numerical counts from 1 to 5 and spatial relations including left, right, above, and below.Numerical prompts sample object combinations, while spatial prompts include template-based and natural COCO prompts.
  • B.1 NSR-1K Benchmark Construction: Numerical layout planning is evaluated with precision, recall, and accuracy, where accuracy requires exact category-count matching except in comparison tasks, which assess the predicted relation.Figure 9 illustrates the automatic numerical-reasoning evaluation procedure.
  • B.1 NSR-1K Benchmark Construction: Spatial accuracy is computed from generated-image detections using a COCO-finetuned GLIP model, while all benchmarks also measure cosine-based CLIP similarity.The same spatial-relation definition is applied to LLM-generated and GLIP-based layouts.
  • B LayoutGPT for 2D Layout Planning: For 2D layout planning, structured prompts vary instruction detail, CSS formatting of bounding-box attributes, and normalization, with supporting-example formats documented in Tables 7 and 8.The ablations target instructions, structure, and normalization settings for presenting height, width, top, and left attributes.
  • B.3 GPT-3.5/4 Prompting: Prompting is adapted for dense Panoptic scenes with overlapping or unmentioned likely objects, and GPT-4 generates counterfactual prompts describing rare spatial relationships among MSCOCO categories.The supplied example is a monkey riding on top of a bird.

B.4 Additional Experiments · B.5 Failure Cases · C LayoutGPT for 3D Scene Synthesis

Additional experiments show that exemplar count and numerical-task structure affect LayoutGPT’s reasoning, while failures arise when many objects must fit constrained image space. For 3D synthesis, LayoutGPT uses room type and room-size specifications converted from floor-plan images.

  • B.4 Additional Experiments: Fixed random in-context exemplars are sampled from the training support set and reused across test conditions despite being unrelated to each test condition.This contrasts with the retrieval-augmented exemplar selection described in Section 3.
  • B.4 Additional Experiments: Counting accuracy increases with the number of in-context exemplars, suggesting that additional few-shot examples improve challenging numerical predictions.The passage specifically cites comparison prompts as an example of challenging cases.
  • B.4 Additional Experiments: 86%: LayoutGPT achieves approximately 86% precision, recall, and accuracy on template-based Single Category numerical reasoning, while Two Category accuracy drops to 66%.The passage reports minimal precision and recall changes for Two Category tasks but lower accuracy.
  • B.4 Additional Experiments: Natural MSCOCO prompts yield high recall but lower precision because prompts can mention object classes absent from the 80-class ground-truth annotations.Predictions for unannotated classes lower precision while matching the intended behavior described in the passage.
  • B.4 Additional Experiments: LayoutGPT is evaluated on the HRS size-comparison benchmark [2], whose prompts contain counterfactual relations between randomly sampled common objects.The passage characterizes these size relations as rarely seen in ordinary examples.
  • B.5 Failure Cases: Two-object numerical and comparison prompts are harder because LayoutGPT predicts smaller bounding boxes, making GLIGEN’s image generation within those regions more difficult.These failures are illustrated in Figure 12 for numerical and spatial relations.
  • C LayoutGPT for 3D Scene Synthesis: For 3D synthesis, conditions use room type and room size; floor-plan images are converted into size specifications because LLMs do not accept image inputs, unlike ATISS [38].An example condition specifies a bedroom with maximum length and width of 256px.

C.1 Exemplar Selection · C.2 Failure Cases

LayoutGPT depends strongly on geometrically similar in-context exemplars for indoor-scene synthesis, while still producing characteristic spatial errors. Random exemplar selection substantially worsens boundary violations, and failures include out-of-bound or overlapping furniture and implausible object placement.

  • C.1 Exemplar Selection: 85.58% out-of-bound rate with eight random bedroom exemplars, versus 43.26% in Table 5, shows that LayoutGPT relies on similar floor plans.The difference indicates that similar exemplar rooms help keep generated objects within scene boundaries.
  • C.1 Exemplar Selection: LayoutGPT’s exemplar analysis evaluates each generated bedroom against eight in-context exemplars using the minimum scene difference, with distances computed in meters.The method compares generated and training scenes using object categories, locations, sizes, and orientations.
  • C.1 Exemplar Selection: Scene differences below 1.0 indicate likely duplication, values below 6.0 indicate moderate object changes, and values above 6.0 indicate new objects or major differences.These thresholds are applied to all 423 bedroom testing samples plotted in Fig. 13.
  • C.1 Exemplar Selection: The scene-difference analysis tests whether generated layouts reproduce or modify their in-context exemplars across 423 bedroom testing samples.The analysis distinguishes duplication, modification, and substantial scene changes using the reported thresholds.
  • C.2 Failure Cases: Despite comparable results with ATISS, LayoutGPT produces typical failures including out-of-bound furniture and overlapped objects.These failure cases are shown in Fig. 14.
  • C.2 Failure Cases: LayoutGPT may place both nightstands on the same side of the bed instead of on opposite sides of the headboard.The paper identifies more sophisticated in-context learning or fine-tuning as potential future improvements for 3D understanding.

D LayoutGPT for 2D Keypoint Planning · E Ethical Statement

LayoutGPT is explored as a 2D keypoint planner that predicts MSCOCO2017-compatible human keypoints for GLIGEN-based image generation, showing promising action control but substantially greater planning difficulty than layouts. The section also describes human-planned bounding-box layouts as a comparative baseline.

  • D LayoutGPT for 2D Keypoint Planning: LayoutGPT predicts 17 MSCOCO2017 [29] human keypoint distributions from text, which GLIGEN [27] then uses for keypoint-to-image generation.This extends LayoutGPT beyond 2D and 3D layout planning to text-conditioned image generation.
  • D LayoutGPT for 2D Keypoint Planning: Preliminary examples show promising potential for controlling specific movements or actions through LayoutGPT-planned keypoints.The examples compare LayoutGPT-conditioned images with StableDiffusion-v2.1 and Attend-and-Excite [4].
  • D LayoutGPT for 2D Keypoint Planning: Keypoint planning is considerably harder than bounding-box planning because it predicts 17 node positions instead of four 2D layout aspects or seven 3D layout aspects.The larger number of predicted nodes increases the planning complexity.
  • D LayoutGPT for 2D Keypoint Planning: The broad spatial relations among body movements in MSCOCO further challenge reliable keypoint planning, so the authors leave this direction for future research.The section identifies keypoint distributions and body-motion relations as unresolved reliability challenges.
  • E Ethical Statement: Human-planned layouts serve as a natural comparative baseline alongside layouts predicted by GPT-3.5/4.Annotators receive a blank square canvas and the prompt’s noun words or phrases, then draw a bounding box for each corresponding element.
  • E Ethical Statement: The human-baseline interface asks annotators to draw bounding boxes for prompt elements while intentionally avoiding additional constraints.The supplied passage states that no additional constraints were imposed, without specifying further ethical procedures.

F Limitations · G Broader Impact

LayoutGPT’s visual planning remains limited in its control mechanisms and task coverage, while its use of LLMs may reduce design effort, improve fine-grained control, and support broader future applications. The work is presented as an initial step toward visual planning across domains such as architecture, virtual reality, and computer-aided design.

  • F Limitations: The main limitations are reliance on 2D and 3D bounding-box layouts and only preliminary exploration of keypoint-based spatial control.Figure 15 shows plausible keypoint distributions generated by LayoutGPT with GPT-4 before text-conditioned image generation.
  • F Limitations: Future research could integrate LLMs with alternative visual controls, including segmentation masks and depth maps, to broaden visual planning capabilities.
  • F Limitations: The framework primarily targets visual generation and does not yet provide a unified approach for tasks such as classification or visual understanding.
  • G Broader Impact: LLMs can reduce human designers’ burden, improve productivity, and support scalable handling of large-scale compositional 2D or 3D planning tasks.
  • G Broader Impact: By conditioning on textual inputs, LLMs enable fine-grained visual control in compositional generation.
  • G Broader Impact: As an initial step in LLM-based visual planning, this work suggests future applications in architecture, virtual reality, and computer-aided design.

H Additional Qualitative Examples

Additional visual showcases illustrate LayoutGPT’s capabilities in 2D numerical reasoning, 2D spatial reasoning, and 3D bedroom-scene synthesis, highlighting its effectiveness and versatility across domains.

  • H Additional Qualitative Examples: Figure 16 presents qualitative examples of LayoutGPT variants on 2D numerical-reasoning prompts.
  • H Additional Qualitative Examples: Figure 17 illustrates qualitative examples of LayoutGPT variants on 2D spatial-reasoning prompts.
  • H Additional Qualitative Examples: Figure 18 displays additional qualitative examples of LayoutGPT variants in bedroom-scene synthesis.
Loading 2305.15393v2…