PaperlayerView exact revision on arXiv

Source-linked AI summary

From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries

Joy Hsu, Emily Jin, Jiajun Wu, Niloy J. Mitra

arXiv:2510.10292v1cs.CVcs.AI

TL;DR

Realistic lived-in scene generation is difficult because real 3D data are limited and object poses vary with room use. FactoredScenes factorizes scenes into learned room programs and hierarchical object poses, using synthetic data, LLMs, and limited real scans. It improves oriented-layout generation and produces rooms that humans have difficulty distinguishing from real ScanNet scenes.

  • Problem

    Realistic generation of varied object poses in real-world scenes remains challenging because high-quality 3D data such as ScanNet are scarce.

    Source: §1 · p. 1

  • Method

    FactoredScenes learns reusable room programs from synthetic scenes, uses LLMs to generate layouts, and hierarchically predicts object poses from programs using limited ScanNet data.

    Source: §1 · pp. 1–2

  • Results

    FactoredScenes significantly improves prior work on realistic ScanNet oriented layouts by FID and KID, and its generated scenes are difficult for humans to distinguish from real scenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FactoredScenes provides a factored route to synthesizing realistic rooms by combining structural knowledge from synthetic data with pose variation learned from lived-in scenes.

    Source: §1 · p. 2, §5 · p. 10

    The method remains limited by inconsistent LLM generation of valid room programs and by heuristics used to generate ground-truth oriented bounding boxes.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.

1 Introduction

FactoredScenes addresses realistic scene generation from limited real-world data by separating reusable room structure from lived-in object-pose variation. It combines learned program libraries, LLM-generated layouts, and hierarchical pose prediction, improving realism and human indistinguishability.

1 Introduction

  • Real-world rooms are noisy and lived-in, with object arrangements shaped by interaction, making high-quality 3D data scarce and realistic generation difficult.Examples include chairs arranged around seating interactions and monitors angled toward specific seats.Source: §1 · p. 1
  • The framework decomposes scene generation into learned room programs, LLM-based program generation, layout execution, program-conditioned pose prediction, and object retrieval.This factorization allows different data sources and methods to model distinct components instead of learning the full scene distribution solely from limited ScanNet data.Source: Figure 1 · p. 1, §1 · p. 2
  • FactoredScenes learns reusable room-structure programs from professionally designed 3D-Front scenes and uses them to guide diverse layouts generated by large language models.The learned library captures structural relationships between objects rather than relying only on an inference-based library.Source: §1 · p. 2
  • A program-conditioned pose model trained on limited ScanNet data predicts orientations hierarchically, first for primary objects and then for dependent objects such as chairs around tables.The predicted dimensions and poses are used to retrieve object instances and complete the 3D scene.Source: §1 · p. 2
  • FactoredScenes improves realistic ScanNet oriented-layout generation over prior work by FID and KID, while its scenes are difficult for humans to distinguish from real rooms.The evaluation also reports a 644.1% relative improvement in function use and an 11.4% relative improvement in dependent-object pose prediction.Source: §1 · p. 2

2 Related Works

Prior indoor-scene methods span image, latent-variable, diffusion, autoregressive, and language-model approaches, but FactoredScenes targets lived-in unconditional synthesis with oriented object poses. Its learned program library differs from fixed domain-specific languages by learning reusable abstractions from example scenes.

Indoor scene synthesis

  • Existing indoor-scene methods include 2D-to-3D generation, VAE and GAN models, hierarchical diffusion, autoregressive transformers, and LLM-based planners.Representative methods include ATISS, DiffuScene, Sync2Gen, and LayoutGPT.Source: §2 · pp. 2–3
  • FactoredScenes focuses on unconditional real-world scene synthesis with lived-in variation and models object orientations in addition to bounding-box coordinates and sizes.Prior methods commonly predict axis-aligned labeled boxes before retrieving meshes or point clouds.Source: §2 · p. 3

Program-based generation

  • Program-based prior work often relies on predefined languages requiring domain-specific knowledge, whereas FactoredScenes learns its program library from a large-scale scene dataset.The learned library improves compact scene representation over an inference-only library and lets LLMs select suitable abstraction levels.Source: §2 · p. 3

3 Method

FactoredScenes factorizes scene generation into learned room programs, LLM-based layout generation, program-conditioned pose prediction, and object retrieval. It learns reusable structure from 3D-Front and realistic pose variation from limited ScanNet data.

3 Method

  • FactoredScenes factorizes scene generation into program, layout, pose, and object-instance components, allowing different data sources and modeling methods for each stage.The factorization combines synthetic library learning, LLM program generation, real-world pose modeling, and retrieval.Source: §3 · pp. 3–4

3.1 Library Learning on 3D-Front

  • The learned library captures reusable room relationships, using wake-stage program reconstruction and sleep-stage abstraction proposals to discover functions such as align, grid, symmetrical, and cluster_placement.The library is learned from professionally designed 3D-Front scenes rather than requiring real-world examples.Source: §3.1 · pp. 4–5, Figure 3 · p. 4

3.2 Program Generation via an LLM

  • An LLM generates diverse scene programs from learned-library functions and parsed ScanNet examples, then a deterministic interpreter executes them into axis-aligned layouts.The learned library regularizes generation while allowing novel scene constructions and commonsense room structures.Source: §3.2 · p. 5

3.3 Object Pose Prediction on ScanNet

  • The pose model predicts oriented bounding-box rotations hierarchically: primary objects are predicted first, while dependent objects additionally use their dependency target’s predicted orientation and program function.Training uses ScanNet scenes and discretizes orientations modulo 180° into 36 five-degree classes.Source: §3.3 · pp. 5–6

3.4 Object Retrieval of ScanNet Objects

  • Object retrieval selects nearest-neighbor 3D instances by class and dimensions, then scales, translates, rotates, and assigns facing directions using program-defined region boundaries.The retrieval design remains agnostic to whether objects are represented as meshes or point clouds.Source: §3.4 · p. 6, Table 1 · p. 7

4 Experiments

FactoredScenes improves realistic ScanNet-like scene generation by combining learned room-structure programs with hierarchical pose prediction. Experiments show stronger layout realism, reusable program libraries, and pose predictions, while remaining limited by LLM program validity and noisy ScanNet orientations.

4.1 FactoredScenes Evaluation

  • 40.1% FID and 79.5% KID improvements on living rooms, alongside 38.3% FID and 80.4% KID improvements on bedrooms, establish stronger layout realism than prior works.The full framework significantly outperforms prior methods across both evaluated room types.Source: §4.1 · p. 7
  • Human evaluation achieved 0.67 mean accuracy for choosing real ScanNet scenes, indicating generated rooms were difficult to distinguish from real rooms.The study collected 400 answers across 20 scene pairs and participant judgments.Source: §4.1 · p. 8

4.2 Learned Program Library Evaluation

  • The learned program library improved function use by 170.1% on 3D-Front and 644.1% on ScanNet while outperforming inference-only libraries on diversity and compression accuracy.The library was evaluated on 100 3D-Front and 150 ScanNet scenes; its learned functions generalized to real scenes.Source: §4.2 · pp. 8–9, Table 2 · p. 8

4.3 Pose Prediction Evaluation

  • The pose model reached 0.745 mIoU on unseen ScanNet scenes, while removing dependent-object conditioning substantially reduced dependent-object performance.Primary-object orientation accuracy remained similar between the full model and the ablated variant.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Discussion

  • The framework remains constrained by occasional invalid LLM-generated programs and inaccurate ScanNet orientations caused by partial objects and heuristic bounding-box labels.These issues can produce unnatural placements or illogical object overlaps.Source: §4.4 · p. 9

5 Conclusion

FactoredScenes addresses real-world scene generation under limited data by separating room structure from varied object poses. Its learned programs and hierarchical pose model produce scenes that quantitative experiments and human studies find difficult to distinguish from real ones.

  • FactoredScenes learns reusable room programs from professionally designed synthetic data and trains a hierarchical object-pose model on limited real data.The factored representation decomposes scenes into underlying layout programs and varied object poses.Source: §5 · p. 10

Supplementary for Factored Real-World Scene Generation via Learned Program Libraries

The supplementary material extends evaluation, documents the learned library and parsed programs, and provides implementation and human-study details.

  • The appendix covers prompt-variation robustness, office-room generation, function signatures, compressed ScanNet programs, orientation analyses, rendered scenes, implementation details, human studies, and broader impact.These materials span Appendices A–F.Source: Supplementary for Factored Real-World… · p. 13

A Additional Results

Additional experiments test prompting robustness and transfer to office scenes. FactoredScenes remains strong across few-shot prompt variants and is evaluated on a new indoor room category.

A Additional Results

  • FactoredScenes shows consistently strong generation performance across 5-shot, 3-shot, and 1-shot prompting configurations.The experiments assess robustness and sensitivity to prompt variations.Source: A Additional Results · p. 13, Table 4 · p. 13
  • Office-room results extend evaluation beyond the main bedroom and living-room focus using the same FID and KID protocol against ScanNet office scenes.The authors expect learned ScanNet distributions to transfer across indoor scene types.Source: A Additional Results · p. 13, Table 5 · p. 13

B Program Library

FactoredScenes’s learned library provides reusable functions for expressing structured object layouts, including primary-object furniture placement and dependent-object arrangements.

B Program Library

  • The function signatures define the reusable program primitives used throughout FactoredScenes’s learned library.The library’s documented functions cover both primary-object placement and dependent-object structure.Source: B Program Library · p. 14, Figure 14 · p. 16
  • The library includes furniture placement for primary objects and parallel, align, grid, grid_with_offset, symmetrical, and cluster_placement functions for dependent objects.These functions encode reusable layout relationships for different object configurations.Source: Figure 8 · p. 14, Figure 9 · p. 14, Figure 10 · p. 14

C ScanNet Program Parsing

FactoredScenes uses learned-library functions to parse complex ScanNet scenes into high-level programs, with LLMs selecting appropriate structured relationships.

C ScanNet Program Parsing

  • GPT-o1 parses bounding boxes in text form into programs using FactoredScenes’s learned library and handles complex ScanNet scenes with high proficiency.Figure 15 shows programs reconstructing ScanNet scenes, including parallel relationships among beds, desks, and dressers.Source: C ScanNet Program Parsing · p. 17, Figure 15 · p. 17

D Predicted Object Poses

FactoredScenes predicts object poses on unseen ScanNet scenes and also evaluates probabilistic pose prediction, while revealing failures in rotations and dependent-object relationships.

D Predicted Object Poses

  • The pose model produces predictions on unseen ScanNet scenes, but some examples contain infeasible rotations and lack logically dependent poses.The authors attribute these failures partly to limited quantity and accuracy of labeled data.Source: D Predicted Object Poses · p. 18, Figure 16 · p. 18
  • FactoredScenes’s probabilistic pose model improves NLL over random and informed fixed mean-and-variance baselines, while mean-based estimates outperform prior works but trail the predictive model.The evaluation is reported in Tables 6 and 7; independent sampling limits robust alignment of dependent-object orientations.Source: D Predicted Object Poses · p. 19, Table 6 · p. 19, Table 7 · p. 19

E Generated Scenes

Generated scenes capture natural object orientations and relationships in several cases, but errors arise when programs misidentify dependencies or objects occupy unfamiliar positions.

E Generated Scenes

  • Generated scenes can orient same-type chairs toward tables without explicitly specified orientation or chair-type constraints.The examples show natural facing directions emerging from the program-conditioned generation process.Source: E Generated Scenes · p. 20, Figure 17 · p. 20
  • Other examples produce natural couch, coffee-table, and stool arrangements, although some scenes contain unrealistic chair orientations toward beds.The examples illustrate both plausible relational layouts and localized pose failures.Source: E Generated Scenes · p. 21
  • FactoredScenes occasionally generates invalid dependent-object programs or unreasonable poses for objects in unfamiliar positions.The authors suggest better-performing LLMs and human-annotated orientation data as possible improvements.Source: E Generated Scenes · p. 21

F Details

The appendix provides implementation, human-study, broader-impact, reproducibility, experimental-detail, and ethics disclosures. The checklist records that the work discusses limitations and LLM use, while noting that error bars are not reported because language-model APIs are expensive.

F Details

  • FactoredScenes trains its library on 3D-Front and its orientation model on ScanNet, using 581 parsed scenes split into 523 training and 58 test scenes.Training used Adam with learning rate 0.0001 on a single Titan RTX GPU with 24 GB memory.Source: F Details · p. 22
  • The human study compared generated rooms with ScanNet rooms using 20 participants, 20 scene pairs each, shuffled questions and answers, and $25-per-hour compensation.The authors report no potential participant risks and no IRB approval because the task used rendered 3D scenes without personal or sensitive content.Source: F Details · p. 22
  • The authors identify potential misuse for fake-content generation and bias inherited from the large language model, encouraging future bias-mitigation work.The stated intended purpose is creating real-world, object-centric datasets.Source: F Details · p. 22

NeurIPS Paper Checklist

  • The paper states that it provides information sufficient to reproduce its main experimental results, releases code for reproduction, and discusses limitations in Section 4.4.Source: NeurIPS Paper Checklist · pp. 23–25
  • The authors do not report error bars because both FactoredScenes and some baselines rely on expensive API-based language models.This limits statistical-significance reporting for the experiments supporting the main claims.Source: NeurIPS Paper Checklist · p. 25
  • The checklist records discussion of societal impacts, asset licensing, human-subject procedures, LLM usage, and reproducibility expectations.The paper cites ScanNet and 3D-Front licenses and provides human-study instructions, compensation, risk, and IRB information in the appendix.Source: NeurIPS Paper Checklist · pp. 27–29

Related papers

Loading 2510.10292v1…