PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding

Jingyu Liu, Wenhan Xiong, Ian Jones, Yixin Nie, Anchit Gupta, Barlas Oğuz

arXiv:2303.03565v2cs.CV

TL;DR

Indoor scene synthesis needs furniture arrangements that are both functionally plausible and visually coherent, but category-level methods miss instance-specific style and color. CLIP-Layout autoregressively predicts furniture instances using multi-view CLIP embeddings, and it improves 3D-FRONT completion metrics while supporting zero-shot text-guided synthesis and editing.

  • Problem

    Existing indoor-scene methods model furniture mainly by category, limiting their ability to capture instance-level attributes such as color and style.

    Source: §1 · p. 2

  • Method

    CLIP-Layout uses an autoregressive scene model with semantic furniture embeddings derived from multi-view CLIP image features and a permutation-invariant transformer.

    Source: §3.2.2 · p. 4, §3.2.4 · p. 5

  • Results

    22% - 129% relative improvements in auto-completion precision and recall over previous state-of-the-art are reported on 3D-FRONT, with stronger precision and recall in bedroom completion.

    Source: §1 · p. 2, §4.2.1 · p. 7

  • Takeaways & Limitations

    The embedding-based model generates style-consistent scenes, generalizes to new furniture without retraining, and supports zero-shot text-guided scene synthesis and editing.

    Source: §1 · p. 2, §4.2.2 · p. 8, §4.3.1 · p. 8

    With a relatively small training dataset, the model can place furniture outside floor plans, duplicate furniture, or stop early.

    Source: §4.1 · p. 6, §5 · p. 9

Abstract

from arXiv · show

Indoor scene synthesis involves automatically picking and placing furniture appropriately on a floor plan, so that the scene looks realistic and is functionally plausible. Such scenes can serve as homes for immersive 3D experiences, or be used to train embodied agents. Existing methods for this task rely on labeled categories of furniture, e.g. bed, chair or table, to generate contextually relevant combinations of furniture. Whether heuristic or learned, these methods ignore instance-level visual attributes of objects, and as a result may produce visually less coherent scenes. In this paper, we introduce an auto-regressive scene model which can output instance-level predictions, using general purpose image embedding based on CLIP. This allows us to learn visual correspondences such as matching color and style, and produce more functionally plausible and aesthetically pleasing scenes. Evaluated on the 3D-FRONT dataset, our model achieves SOTA results in scene synthesis and improves auto-completion metrics by over 50%. Moreover, our embedding-based approach enables zero-shot text-guided scene synthesis and editing, which easily generalizes to furniture not seen during training.

1 Introduction

Indoor scene synthesis must combine functional plausibility with visual consistency, but prior category-level methods cannot represent distinctive furniture attributes. CLIP-Layout addresses this gap with instance-level CLIP embeddings and reports substantial improvements on 3D-FRONT auto-completion.

Motivation

  • Indoor scene synthesis aims to create rooms with furniture that is physically and functionally plausible while also visually consistent and aesthetically pleasing.Examples include placing beds on floors, fridges against walls, and furniture combinations that fit the room type while matching in style and color.Source: §1 · p. 2

Limitations of Prior Work

  • Hand-crafted procedural and scene-graph methods rely on finite rules and heuristics, limiting the diversity of generated scenes and requiring attributes to be explicitly encoded in metadata.These methods can capture aesthetic constraints only when the relevant attributes are manually represented.Source: §1 · p. 2
  • Learned category-level models can place furniture types contextually but cannot recommend particular instances or learn visual correspondences such as color and style.This limits their ability to model the variations humans consider when composing coordinated interiors.Source: §1 · p. 2

Approach

  • CLIP-Layout uses pooled CLIP image embeddings from multiple object renders in an autoregressive model to capture instance-level visual attributes and enable text-guided editing.The web-trained image encoder is intended to represent diverse real-world objects and styles while supporting text-based control.Source: §1 · p. 2

Results

  • 22% - 129% relative improvements in auto-completion precision and recall over previous state-of-the-art are reported on 3D-FRONT.The model generates natural-looking, diverse, style-consistent scenes and can generalize to new furniture without retraining.Source: §1 · p. 2

2 Related Work

Prior indoor-scene methods range from hand-crafted placement rules and procedural or probabilistic generation to implicit and text-to-image representations. CLIP-Layout is motivated by the need for semantic, visual 3D object representations that support aesthetics-aware composition without large dedicated 3D datasets.

Rule-Based and Procedural Methods

  • Early indoor-scene synthesis methods rely on hand-crafted rules and constraints, followed by procedural modeling and probabilistic grammars.These approaches can produce impressive samples but may ignore visual coherence during composition.Source: §2.1 · pp. 2–3

Implicit and Image-Based Methods

  • Implicit-function and text-to-image approaches model occupancy, attributes, or rooms continuously rather than placing explicit furniture objects.Their realism is accompanied by limitations for downstream tasks requiring explicit object representations.Source: §2.1 · p. 3

3D Shape Representations

  • Existing 3D shape representations emphasize topology or reconstruction and generally require large amounts of 3D or multi-view training data.This motivates using semantic and visual representations better suited to aesthetics-aware indoor scenes.Source: §2.2 · p. 3

3 Method

CLIP-Layout models a floor-plan-conditioned distribution over unordered furniture scenes, autoregressively predicting whether to add an object, its semantic instance embedding, and its transformation. It uses a permutation-invariant transformer and multi-view CLIP embeddings to represent furniture appearance, while retaining pose and layout information.

Problem Setup

  • The model represents each scene as a floor layout plus an unordered set of furniture objects and models their conditional distribution with an autoregressive likelihood objective.The floor plan supplies a prior, while each object is conditioned on the floor plan and previously selected partial-scene objects.Source: §3.1 · p. 3

Scene Representation

  • Each furniture instance combines a semantic embedding with translation, size, and vertical rotation, while the floor plan is encoded as a binary placement mask.The model therefore represents both instance appearance and spatial configuration.Source: §3.1 · p. 3, §3.2.1 · p. 4

Transformer Decoder

  • A decoder-only transformer without positional encoding produces permutation-invariant features for the furniture set and predicts the next instance through learned queries and encoded scene inputs.The architecture includes floor-plan, transformation, and embedding encoders, a transformer decoder, and next-instance decoders.Source: §3.2.2 · p. 4, §3.2 · p. 4

Autoregressive Prediction

  • At each step, the model predicts a stop token, decodes an instance embedding, retrieves furniture through embedding-space scores, and then predicts translation, rotation, and size.Transformation attributes are modeled with mixtures of logistics distributions.Source: §3.2.3 · p. 4

Object Embedding

  • Each object embedding averages normalized CLIP image features from eight canonical renderings of a normalized mesh, producing a representation that captures fine visual details without depending on object categories.Fixed rendering conditions support consistent encoding, and the resulting embedding is used for image- or text-based retrieval.Source: §3.2.4 · pp. 4–5, Figure 2 · p. 5

Training and Inference

  • Training randomly selects and permutes scene-object subsets, while inference feeds predicted furniture and transformations back into the autoregressive process to construct scenes from floor plans or partial scenes.Data augmentation includes random rotations and mirroring, and inference can begin with optional existing furniture.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate CLIP-Layout against ATISS on partial completion, unconditional synthesis, and text-guided applications using filtered 3D-FRONT scenes. CLIP-Layout improves quantitative completion results, produces more style-consistent scenes, and supports text-guided synthesis and furniture replacement.

4.2.1 Quantitative Evaluations

  • 53% average precision and 34% average recall improvements are reported for bedroom partial scene completion over the baseline.The evaluation removes furniture while varying the number of existing objects, with results summarized in Table 3.Source: §4.2.1 · p. 7, Table 3 · p. 7
  • FID improvements generally increase with more pre-populated furniture, but CLIP-Layout trails the baseline in the largely incomplete one-furniture setting.The authors hypothesize that the model is over-calibrated for precision and may stop early when scenes are highly incomplete.Source: §4.2.1 · p. 7, Table 3 · p. 7
  • CLIP-Layout outperforms the baseline substantially across all room types and metrics in one-step partial scene completion.The summary averages quantitative results over five fixed-seed runs, while full room-type metrics appear in Appendix Table 4.Source: §4.2.1 · p. 7, Table 2 · p. 7

4.2.2 Qualitative Results

  • Qualitative comparisons show that CLIP-Layout captures style and color patterns that the baseline misses despite producing plausible layouts.Examples compare unconditional synthesis from floor plans and partial scene completion; the model generates patterns such as consistently modern furniture or a dominant color.Source: Figure 3 · p. 6, §4.2.2 · pp. 7–8, Figure 4 · p. 8

4.3 Text-guided Applications

  • CLIP-Layout supports zero-shot text-conditioned scene synthesis and furniture replacement through semantic CLIP embeddings.Text prompts guide scene style or target replacement objects, whose placement can be adjusted using instance-level information rather than copied blindly.Source: §4.3 · p. 8, §4.3.1 · p. 8, §4.3.2 · p. 8

5 Limitations

With a relatively small training dataset, CLIP-Layout sometimes produces unreasonable scenes through malformed placement, duplicated furniture, or premature stopping.

  • Small training data leads to three common failure modes: furniture outside the floor plan, duplicated furniture, and early stopping.The authors relate these issues to malformed training examples, repetitive-generation behavior, and an imbalanced stop-token objective.Source: §5 · p. 9

6 Conclusions

CLIP-Layout models furniture instances with semantic embeddings to synthesize and complete style-consistent scenes while preserving furniture configurations. Its multimodality supports controlled tasks, and the model outperforms baselines across settings.

  • CLIP-Layout synthesizes and auto-completes scenes that follow dataset style and aesthetics while preserving furniture configurations.Source: §6 · p. 9
  • Semantic furniture embeddings and multimodality support text-guided furniture replacement and zero-shot scene synthesis.Source: §6 · p. 9
  • The model outperforms the baseline in different qualitative and quantitative settings.Source: §6 · p. 9

A Full Scene Completion Metrics

The full evaluation reports metrics for all room types in Table 2, extending the bedroom analysis to other room categories. The authors state that the bedroom conclusions also hold for other room types.

  • Table 2 provides the full evaluation metrics for all room types, beyond the bedroom-focused analysis.Bedrooms contain the most data with correct layouts and furniture counts, motivating their use for the primary metric discussion.Source: A Full Scene Completion Metrics · p. 12
  • The authors state that the conclusions from bedroom scenes hold for other room types.Source: A Full Scene Completion Metrics · p. 12

B Rendering Settings

The experiments standardize mesh materials and render objects and scenes from eight canonical directions using specified camera settings. Evaluation uses EGL-based rendering, while paper figures use Blender ray tracing and a fixed floor texture.

  • The reproducibility settings cover mesh material standardization followed by camera settings for eight-view object and scene rendering.Source: B Rendering Settings · p. 12
  • Materials are manually adjusted because abnormal dataset properties can produce overly dark or bright renders; the chosen ambient, diffuse, and specular ratio is (0.4, 0.4, 0.1).Source: B Rendering Settings · p. 12
  • Objects and scenes are rendered from eight canonical directions with and EGL, with camera parameters summarized in Table 5.Source: B Rendering Settings · p. 12
  • Evaluation uses, whereas paper figures use Blender ray tracing with the CYCLE engine and a three-point-light setup; floor textures are fixed during evaluation.Source: B Rendering Settings · p. 12

C More Visual Examples

This section presents visual examples of CLIP-Layout for text-guided furniture replacement, text-guided synthesis, floor-layout synthesis, and partial scene completion.

Text-guided Furniture Replacement

Text-guided Synthesis

Scene Synthesis from Floor Layouts

Partial Scene Completion

Related papers

Loading 2303.03565v2…