HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation

1. Introduction

• Prior methods often neglect small objects or place them randomly or by predefined rules, limiting realism and downstream applications.Small objects also depend on larger furniture for support and must satisfy both physical constraints and user specifications.Source: §1 · p. 1
• HSM decomposes scene generation into support-region identification and compositional object arrangement, from room-level furniture to small objects on furniture.The unified approach uses text-explicit and implicit relationships to align the generated scene with user intent.Source: §1 · pp. 1–2
• HSM’s key insight is that indoor object arrangements contain reusable spatial patterns across scales, which can be composed into complex scene motifs.The paper describes motifs as learnable from a few examples and applicable across different scales.Source: §1 · p. 2
• HSM generates realistic, physically constrained scenes with dense object arrangements that align with user expectations and outperform prior methods on complex arrangements.The paper highlights plausible, text-consistent small-object placements and more realistic hierarchical structures.Source: §1 · p. 2

2. Related Work

• Many methods neglect small objects or simplify their placement, while HSM conditions arrangements at all scales on precise language descriptions.The paper contrasts HSM’s recurring motifs with Architect’s 2D-inpainting approach [75] and SceneFunctioner’s LLM-predicted anchor rules.Source: §2 · p. 2
• Prior support-surface prediction methods use 2D images, whereas HSM extracts valid support regions through geometric reasoning on synthetic 3D objects.The paper notes that image-based approaches require viewpoint selection and can encounter occlusion issues.Source: §2 · p. 2

3. Method

• At each hierarchical level, HSM uses a VLM to decompose the text into room type and object requirements, then extracts support regions, generates motifs, and optimizes placements.The stages repeat from room-level furniture to small objects on furniture surfaces.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1. Scene Motifs

• A motif is an atomic spatial pattern, while a scene motif composes learned motifs to represent more complex relationships between objects.Motifs are visual programs in a pre-learned library; scene motifs can combine patterns such as a book stack with a lamp to its left and a laptop in front.Source: §3.1 · p. 3

3.2. Input Description Requirement Decomposition

• A VLM extracts room type and object requirements, groups objects by their supporting elements, and infers quantities, appearances, and dimensions when unspecified.If the input omits a room boundary, the VLM is prompted to generate one with door and window placement and room height.Source: §3.2 · p. 3

3.3. Support Region Extraction

• For furniture support regions, HSM clusters mesh faces into planar surfaces, filters horizontal surfaces by height clearance, and segments them using vertical surfaces.The resulting regions represent areas where objects can be placed, including compartments on multi-level furniture.Source: §3.3 · p. 4, Figure 4 · p. 5

3.4. Scene Motif Generation

• For each region, a VLM groups objects and decomposes their arrangement into a hierarchy of motifs, which are instantiated with retrieved meshes and checked against the input.Generation begins from a primary motif, adds the remaining objects iteratively, and applies spatial optimization to resolve collisions, move objects closer, and simulate gravity while respecting the hierarchy.Source: §3.4 · pp. 4–5

3.5. Layout Optimization

• A VLM proposes initial placements, a grid-based DFS solver enforces region, wall-alignment, and non-overlap constraints, and a scene-level optimizer refines physical support and collisions.These steps repeat at each hierarchy level to preserve spatial coherence from room-scale furniture to small objects.Source: §3.5 · p. 5

4.1. Scene Generation

• HSM is evaluated on 100 SceneEval-500 descriptions spanning easy, medium, and hard cases, against LayoutGPT, InstructScene, LayoutVLM, and Holodeck.The study uses HSSD-200 assets; non-Holodeck methods receive a standardized 6 m × 6 m floor plan with walls.Source: §4.1 · pp. 5–6
• Scene generation is assessed with text-image similarity and SceneEval fidelity metrics, alongside collision, support, navigability, accessibility, and out-of-bound plausibility metrics.Scene-level text-image similarity uses BLIP-2, Long-CLIP, and VQAScore; small-object alignment is evaluated on 30 furniture arrangements.Source: §4.1 · pp. 6–7
• A 25-participant perceptual study compares HSM with Holodeck for fidelity and plausibility at both scene and small-object levels.It uses 25 scene pairs and 30 furniture pairs, with preferences reported as percentages.Source: §4.1 · p. 6

4.2. Support Region Extraction

• Support-region extraction is evaluated on 529 manually annotated regions from 100 HSSD-200 furniture items, using volume IoU and F1 at a 0.5 threshold.The baseline predicts regions only on object top surfaces; IoU applies a 10 cm height-alignment threshold.Source: §4.2 · p. 7

5.1. Scene Generation

• HSM outperforms baselines on text-image scores and SceneEval fidelity, with qualitative examples showing more accurate specified object relationships and room-contained placements.LayoutGPT, InstructScene, and LayoutVLM sometimes place objects outside room boundaries; HSM alone matches two cited scene arrangements.Source: §5.1 · p. 7, Figure 6 · p. 8
• HSM outperforms prior work on small-object placements and is preferred over Holodeck for fidelity and plausibility at both scene and small-object levels.Its support-region analysis identifies valid placement regions across furniture surfaces, supporting denser and more realistic small-object arrangements.Source: §5.1 · pp. 7–8

5.3. Ablation

• Removing scene motifs, the scene spatial optimizer, or the DFS solver worsens performance; the DFS solver ablation causes the largest drop across most metrics.The ablation results associate solver removal with lower fidelity and plausibility from invalid placements outside support regions.Source: Table 4 · p. 8, §5.3 · p. 8

5.2. Support Region Extraction

• Support-region extraction reaches 60.27% IoU and 48.54% F1, versus 32.54% IoU and 16.80% F1 for the top-surface-only baseline.Figure 7 provides examples of extracted support regions.Source: §5.2 · p. 8, Figure 7 · p. 8

6. Conclusion

• HSM models indoor scenes as a hierarchy of support regions, each populated with objects.Source: §6 · p. 8
• Scene motifs exploit recurring spatial patterns to arrange objects from room-level furniture layouts to fine-grained small-object placements.Source: §6 · p. 8
• The authors characterize HSM’s unified hierarchical framework as a step toward densely populated, realistic indoor environments.Source: §6 · p. 8

A.1. Motif Library

• The motif library contains 17 named arrangements, including stacks, rows, grids, surrounds, bed setups, and relative-placement patterns.Source: §A.1 · p. 13
• Motifs are learned from arrangements extracted from HSSD scenes, except bed setup and pyramid, whose examples were created manually in Blender.Source: §A.1 · p. 13
• Each example is manually annotated with a text description, and the learning phase creates reusable meta programs following SMC.Source: §A.1 · p. 13

A.2. Support Region Extraction

• Implementation settings include normal thresholds of 0.9 and 0.95, 0.05 m clearance between horizontal supports, 0.5 m top-surface clearance, and 80% horizontal-surface segmentation.Source: §A.2 · p. 13
• The clustering algorithm traverses neighboring faces, adding faces whose normals meet the cluster threshold and queuing neighbors that meet the adjacency threshold.Source: §A.2 · p. 13
• Surface orientation is estimated by fitting a plane to face clusters, using an upright-oriented bounding box unless its volume is significantly larger than an unconstrained box.The upright box is preferred when its volume is no more than 1% larger; a planar surface must have a smallest dimension below 0.1 of the other two.Source: §A.2 · p. 13
• A plane is classified as horizontal when its y-normal is at least 0.95 and vertical when it is below 0.05.Source: §A.2 · p. 13

A.3.1 DFS Solver

• The DFS solver searches motif poses within a support region, seeded by VLM positions and orientations, while accounting for fixed objects and door clearances.For floor regions, a VLM also determines whether motifs should align to a wall.Source: §A.3.1 · p. 14
• Candidate placements lie on a filtered support-surface grid, sampled at 0.1 m for floors, walls, and ceilings and 0.01 m for furniture.Source: §A.3.1 · p. 14
• Motifs are considered in decreasing footprint area, with candidate poses constrained to avoid collisions, remain within support-region bounds, and meet designated wall alignment.Source: §A.3.1 · p. 14
• The solver scores candidates by distance from the initial pose and, for edge-constrained motifs, distance from the target wall; it explores up to 10 candidates per step and stops at the first complete layout.It backtracks when placements fail, has a 10-second time limit, and leaves explicit objects at their initial positions if no solution is found.Source: §A.3.1 · p. 14

A.3.2 Scene Spatial Optimization

• Scene-level optimization checks for mesh intersections and unsupported objects, preserving placements that already pass those checks.When needed, it resolves collisions through a small vertical lift followed by horizontal displacement, with movement constraints based on the support region.Source: §A.3.2 · pp. 14–15
• Downward rays test support beneath motif centers and corners; unsupported objects are moved to the nearest supporting surface with minimal vertical adjustment.The stated support-test threshold is 0.01 m; ablations report artifacts such as colliding motifs and reduced fidelity when walls and ceilings are removed.Source: §A.3.2 · p. 15, Figure 8 · p. 15, Table 6 · p. 15

A.4. Asset Retrieval

• Asset retrieval filters by object category, ranks candidates by OpenCLIP text-image similarity, then selects among the top five by bounding-box dimension match.The VLM extracts dimensions and style attributes from the description or predicts them when unspecified.Source: §A.4 · p. 15

A.5. Additional Quantitative Analysis

• HSM maintains consistent SceneEval performance across difficulty levels, while LayoutGPT and InstructScene fidelity metrics CNT and ATR drop as prompt complexity increases.Source: §A.5 · p. 15
• Removing scene motifs increases object count but reduces fidelity and arrangement coherence; grouped motifs fill surfaces faster.Source: §A.5 · p. 15
• Removing the DFS solver produces the largest OOR drop among ablations, increases collisions, and lowers support by bypassing geometric constraints.Source: §A.5 · p. 15
• Removing the spatial optimizer increases collisions and lowers support, while removing wall and ceiling regions decreases fidelity but also lowers OOB rate.Source: §A.5 · p. 15

A.6. Computational Cost and Runtime Analysis

• Generation takes about 8 minutes per scene, with VLM calls costing approximately US$0.50 per scene, or roughly US$0.02 per object.Source: §A.6 · p. 16
• VLM inference latency, rather than geometric computation, is the primary speed limit, and scene motif generation is the main bottleneck.Source: §A.6 · p. 16
• In a representative 8-minute-37-second scene with 15 scene motifs, generation averaged 35 seconds per motif.Source: §A.6 · p. 16

A.7. Open Source VLM

• With InternVL, HSM scores 45.40% CNT, 47.81% ATR, and 23.78% OOR, matching or surpassing Holodeck’s 44.64% CNT, 39.42% ATR, and 20.92% OOR.Source: §A.7 · p. 16
• InternVL HSM records 72.69% SUP versus Holodeck’s 62.12%, with comparable collision rates of 17.01% versus 17.32%.Source: §A.7 · p. 16
• InternVL fidelity scores are lower than GPT-4o’s but follow the same general trends; the open-source model shows reduced instruction following in complex spatial reasoning.Source: §A.7 · p. 16
• HSM with InternVL remains comparable to Holodeck, which uses GPT-4o, supporting the authors’ conclusion that HSM is not overly dependent on one proprietary model.Source: §A.7 · p. 16

A.8. Limitations and Future Work

• Motifs operate within individual subtrees, so HSM cannot express relations spanning subtrees, such as a desk facing a bed.Source: §A.8 · p. 16
• The manually curated library covers a limited set of atomic patterns, constraining unusual or highly specific configurations outside its primitives.Source: §A.8 · p. 16
• Support-region extraction assumes well-curated 3D geometry with correct surface normals; malformed meshes can produce misplaced objects.Source: §A.8 · p. 16
• Geometric analysis does not distinguish accessible from enclosed support regions, so small objects may be placed in unreachable spaces such as inside cabinets.Source: §A.8 · p. 16
• Multiple VLM queries and optimization steps increase runtime, especially for larger scenes.Source: §A.8 · p. 16
• Proposed future work includes identifying inaccessible support regions, parallel or GPU processing, and automatically learning additional motif types.Source: §A.8 · p. 16

B. Support Region Dataset Details

• The support-region dataset covers 100 HSSD-200 assets across 12 categories, selected for multi-layer or irregular surfaces and manually annotated with regions, heights, and top-surface flags.Top surfaces without an obstructing surface above receive a height of htop = 1.0 m.Source: B. Support Region Dataset Details · pp. 16–17

C. User Study Instructions

• The user-study instructions outline an overall study description plus separate guidelines for scene-level and small object-level evaluations.Source: C. User Study Instructions · p. 17

C.1. Overall Description of the Study

• Participants choose between two generated scenes in scene-level and small object-level evaluations, using top-down and close-up views, respectively.Each comparison is expected to take approximately 20 seconds.Source: §C.1 · p. 17

C.2. Instructions for Scene-level Evaluation

• Scene-level evaluation asks which scene better matches the text and has more realistic furniture placement, while using consistent object colors and red to mark objects near the ceiling.Source: §C.2 · p. 17

C.3. Instructions for Object-level Evaluation

• Object-level evaluation judges text-description fit and realistic small-object placement on furniture, with consistent coloring to help identify repeated objects.Source: §C.3 · p. 17

D. VLM Prompts

• The appendix provides VLM prompts for scene-motif decomposition and generation, as well as scene-, furniture-, and small object-level tasks.Source: D. VLM Prompts · p. 17

D.1.1 Scene Motif Hierarchy Decomposition

• The decomposition prompt maps descriptions into hierarchical motifs, selecting among predefined arrangements according to object types, counts, and relationships such as perimeter, on_top, and next_to.The motif set includes single-object patterns and two-object arrangements, with constraints such as wall placement or assumed central-object shape for some motifs.Source: §D.1.1 · p. 18
• For arrangements exceeding motif limits, the prompt advises splitting objects into valid motifs and grouping similar objects, while requiring complete, non-invented object coverage and exactly-once use.Source: §D.1.1 · pp. 18–19
• The VLM identifies a dominant arrangement from the objects’ core functional relationship, choosing a motif that fits its type constraints and typical spatial use.For larger descriptions, it prioritizes a core pair or triplet and handles other objects in secondary arrangements or nested structures.Source: §D.1.1 · p. 19
• Remaining objects are grouped into logical secondary arrangements with suitable clearance from the primary arrangement, and each object must appear exactly once.The rules also require accounting for all remaining objects without reusing objects.Source: §D.1.1 · p. 20
• The VLM combines arrangements into a hierarchical JSON using valid motif types, without adding objects or relationships absent from the identified arrangements.The hierarchy is validated for physical feasibility, motif correctness, completeness, and simplicity; invalid motifs trigger a retry from primary-arrangement identification.Source: §D.1.1 · p. 20

D.1.2 Scene Motif Generation

• After validation, the VLM generates arrangements using meta-programs, reasoning about object relationships and maintaining consistent orientations and spacing.Generation guidelines call for sensible clearance and human access, functional positioning, and consideration of object size and default orientation.Source: §D.1.2 · p. 21
• Motif placement uses a right-handed coordinate system: X is left/right, Y is vertical, and Z is front/back; objects default to facing +Z.Source: §D.1.2 · p. 21
• The VLM generates motifs iteratively, inferring parameters from object sizes and existing arrangements until the hierarchy is complete.Source: §D.1.2 · p. 21
• The VLM validates each motif from top-down and front orthographic views, assessing adherence to the description and providing corrective feedback.Source: §D.1.2 · p. 21

D.2. Scene Level Prompts

• Scene-level prompting first establishes task instructions and infers a canonical room type from the textual description.Source: §D.2 · p. 22
• When no room boundary is given, the VLM proposes room vertices, height, and door location, sizing the room for the described furniture and usable movement.Source: §D.2 · p. 22
• Before object decomposition, the VLM analyzes the floor plan and room shape, describing functional zones and suggesting their locations.Source: §D.2 · p. 22
• The VLM decomposes the description into floor, furniture-top, wall, and ceiling objects, specifying each object's style, dimensions, count, and parent for small objects.Source: §D.2 · pp. 22–23
• Scene-level inventories separate each object type, decompose composite sets into individual objects, and associate small objects with a parent furniture or wall object.Entries include names, descriptions, dimensions, and amounts for large, wall, ceiling, and small objects.Source: §D.2 · p. 23

D.3. Furniture Level Prompts

• Furniture motifs group spatially related large objects and encode their relationships, footprint, and clearance; the VLM then places motifs using room views and layout constraints.Placement guidance includes coordinates and rotation, wall alignment, even distribution, and keeping motifs clear of doors.Source: §D.3 · pp. 23–24
• When room occupancy is below tocc, the VLM suggests additional objects while avoiding objects already present to improve diversity.The prompt specifies adding 1–3 large floor-placed furniture pieces, excluding furniture already in the room.Source: §D.3 · p. 24

D.4. Small Object Level Prompts

• For small-object placement, the VLM selects floor or wall-mounted objects for population with commonly co-occurring items, using otherwise similar procedures.The prompt provides a top-down 2D visualization of the furniture surface.Source: §D.4 · p. 25
• The 2D visualizations use the origin as the plot center, mark usable surfaces and surrounding objects, and show furniture orientation and measurements on a grid.The x-axis represents width and the z-axis depth; the highlighted area indicates usable placement space.Source: §D.4 · p. 25
• Placement prompts use surrounding objects to map interaction zones and organize surface space for accessible individual setups and shared items.They also call for functional access, clear reaching paths, and consideration of object removal or replacement.Source: §D.4 · p. 25
• Motif instructions prefer single objects, grouping only functionally dependent items, with realistic dimensions, spatial relationships, footprint, and clearance specified.Each small object is used once per motif, and motifs are limited to four object types before being split.Source: §D.4 · p. 25
• Layered placement uses the provided layer and surface information to distribute objects according to real-world use and vertical-space constraints while meeting exact total quantities.The prompts require the amounts for each object type to sum to its specified total across all furniture surfaces.Source: §D.4 · pp. 26–27

• Figure 10 reports that HSM produces the most consistent room layouts and object arrangements relative to input descriptions among the compared methods.Source: Figure 10 · p. 28
• Figure 11 shows realistic, densely populated HSM scenes that include smaller objects and align with user input.Source: Figure 11 · p. 29