PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model

Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard Newcombe, Vasileios Balntas

arXiv:2403.13064v1cs.CV

TL;DR

SceneScript addresses scene representation by generating full indoor scenes as autoregressive structured-language command sequences rather than conventional geometric representations. It combines a video-to-language encoder-decoder with the 100k-scene Aria Synthetic Environments dataset, achieving state-of-the-art architectural layout estimation, competitive 3D object detection, and extensions to new tasks through language additions.

  • Problem

    Conventional scene representations have task-dependent limitations, motivating a compact, interpretable, editable, and extensible representation for full indoor scenes.

    Source: §1 · p. 2

  • Method

    SceneScript autoregressively predicts tokenized, hand-designed structured language commands from video using an encoder-decoder architecture, trained with Aria Synthetic Environments.

    Source: §1 · pp. 2–3, §4 · p. 6

  • Results

    SceneScript achieves state-of-the-art architectural layout estimation, competitive 3D object detection, and extensions to new tasks through simple language additions.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The command-based representation supports compact, editable, interpretable scene models and adaptation to elements such as object parts and Bezier curves with minimal changes.

    Source: §1 · p. 2, §9 · pp. 14–15

    The manually defined, high-level commands require human intervention and can miss fine-grained geometric details, producing simpler and coarser reconstructions.

    Source: §8 · p. 14

Abstract

from arXiv · show

We introduce SceneScript, a method that directly produces full scene models as a sequence of structured language commands using an autoregressive, token-based approach. Our proposed scene representation is inspired by recent successes in transformers & LLMs, and departs from more traditional methods which commonly describe scenes as meshes, voxel grids, point clouds or radiance fields. Our method infers the set of structured language commands directly from encoded visual data using a scene language encoder-decoder architecture. To train SceneScript, we generate and release a large-scale synthetic dataset called Aria Synthetic Environments consisting of 100k high-quality in-door scenes, with photorealistic and ground-truth annotated renders of egocentric scene walkthroughs. Our method gives state-of-the art results in architectural layout estimation, and competitive results in 3D object detection. Lastly, we explore an advantage for SceneScript, which is the ability to readily adapt to new commands via simple additions to the structured language, which we illustrate for tasks such as coarse 3D object part reconstruction.

1 Introduction

SceneScript represents full indoor scenes as autoregressively generated structured language commands, addressing limitations of conventional scene representations. It is paired with the 100k-scene Aria Synthetic Environments dataset and supports layout estimation, object detection, and extensions to new tasks.

1 Introduction

  • The approach is motivated by transformer and LLM next-token prediction methods and aims to infer metrically accurate full-scene representations as specialized text sequences.This departs from meshes, voxel grids, point clouds, and implicit representations discussed as alternative scene representations.Source: §1 · p. 2
  • SceneScript autoregressively predicts compact, crisp, interpretable, editable, semantically rich scene commands that can incorporate new geometric entities through added commands.The representation uses high-level parametric commands such as make_door(*door_parameters).Source: §1 · p. 2
  • Aria Synthetic Environments provides 100k unique synthetic indoor scenes with egocentric trajectories, photorealistic sensor renderings, depth and instance ground truth, and SceneScript layout annotations.The dataset addresses the lack of paired scene walkthroughs and structured language commands for training.Source: §1 · p. 3
  • The method targets architectural layout estimation and object detection while extending the same language to tasks including coarse 3D object reconstruction and entity-state representation.These extensions can retain the visual input and network architecture while modifying the structured language.Source: §1 · pp. 2–3
  • SceneScript training on Aria Synthetic Environments leads to generalization on real scenes.The paper points to videos and demonstrations for this result.Source: §1 · p. 3

2 Related Works

Related work spans architectural layout inference and transformer-based geometric sequence modeling. SceneScript differs by learning high-level structured commands without explicit architectural heuristics or priors, while emphasizing interpretability and semantic richness.

2.1 Layout Estimation

  • Prior layout methods infer architectural elements using wall heuristics, integer-linear constraints, bird’s-eye-view paths, segmentation, Manhattan-world assumptions, or object-layout relationships.These approaches include Scan2BIM, Ochmann et al., Furukawa et al., RoomNet, LayoutNet, AtlantaNet, and SceneCAD.Source: §2.1 · pp. 3–4
  • The structured language includes commands such as make_wall, make_door, make_window, and make_bbox for architectural layouts and object bounding boxes.Table 1 also describes extensible commands such as make_prim and make_pillar and parameters including int, float, and bool types.Source: Table 1 · p. 4, §2.1 · p. 4
  • SceneScript requires neither layout heuristics nor explicitly defined architectural priors, yet learns geometric scene understanding by predicting ground-truth scene-language token sequences.This contrasts with prior layout methods using heuristics, integer programs, Manhattan assumptions, or graph-based relationships.Source: §2.1 · pp. 3–4

2.2 Geometric Sequence Modelling

  • Transformer-based sequence methods model meshes, CAD sketches, vector graphics, and CAD models as text-like sequences, but SceneScript uses higher-level commands for interpretability and semantic richness.Low-level commands can represent complex geometry but may require prohibitively long sequences for full scenes.Source: §2.2 · p. 4
  • Pix2Seq uses a similar architecture for 2D object detection, while Point2Seq autoregressively regresses continuous 3D bounding-box parameters and outperforms anchor- and center-based detection architectures.Source: §2.2 · p. 4

3 SceneScript Structured Language Commands

SceneScript defines indoor scenes as arbitrary-length sequences of structured commands parsed into 3D geometry, and trains an encoder-decoder on paired egocentric walkthroughs and command sequences. Its command language is intentionally extensible, supported by the 100k-scene Aria Synthetic Environments dataset.

3 SceneScript Structured Language Commands

  • SceneScript represents indoor layouts and objects with structured commands, then uses a video-to-tokenized-language encoder-decoder pipeline to reconstruct scenes.The architecture consumes a video sequence and returns SceneScript language in tokenized form.Source: §3 · p. 5

3.1 Commands and Parameters

  • The initial layout parameterization uses make_wall, make_door, and make_window, with parameters producing well-defined gravity-aligned planes and box-based wall cutouts.The parameterization is a proof-of-concept choice prioritizing ease of use and research iteration speed.Source: §3.1 · p. 5
  • The make_bbox command represents gravity-aligned oriented 3D bounding boxes using object identity, class, position, angle, and scale parameters.The command jointly supports object inference alongside architectural layout commands.Source: §3.1 · p. 5
  • The command language can add geometric or semantic parameters, such as open_degree for door states, enabling extensions including coarse 3D object reconstruction.The paper describes these additions as requiring extensions to the language rather than a new visual input or network architecture.Source: §3.1 · p. 6

3.2 Scene Definition

  • A scene is an arbitrary-length, unordered sequence of commands that a custom interpreter parses into a 3D scene.The representation is serialized into integer tokens through a bijective mapping for transformer decoding.Source: §3.2 · p. 6

3.3 Training Dataset

  • Aria Synthetic Environments is a publicly released dataset pairing egocentric scene walkthroughs with corresponding ground-truth command sequences and containing 100k synthetic scenes, each with specified simulated-scene resources.Source: §3.3 · p. 6

4 SceneScript Network Architecture

SceneScript encodes visual scene inputs into a latent sequence and autoregressively decodes structured language commands. Its tokenization supports compact, interpretable, variable-length scene descriptions across input modalities.

4 SceneScript Network Architecture

  • SceneScript uses an encoder-decoder pipeline that converts video or scene observations into tokenized structured language commands.The decoder autoregressively generates the command sequence from the encoded scene representation.Source: §4 · p. 6, §4.2 · p. 8

4.1 Input Modalities and Encoders

  • The system supports point-cloud, posed-image, and combined encoders while keeping the transformer decoder unchanged.Point-cloud features are produced with sparse 3D convolutions, while the image-only variant uses bidirectional image-voxel attention guided by camera geometry.Source: §4 · p. 6, Point Clouds · p. 7, End-to-end Encoding of Posed Views · p. 8
  • Point-cloud encoding condenses N input points into K ≪ N latent feature vectors augmented with active-site coordinates.The resulting F_geo representation has K × 512 features and is ordered for decoder use.Source: Point Clouds · p. 7
  • Image features can be lifted into 3D by projecting points into keyframe cameras, averaging retrieved features, and concatenating them with point coordinates.This creates point features that combine visual evidence with geometric location.Source: Point Clouds with Lifted Features · pp. 7–8

4.2 Language Decoder

  • The transformer decoder applies causal attention to generate structured commands autoregressively from embedded tokens and the scene code.A causal mask ensures each prediction depends only on preceding tokens and the encoded scene context.Source: §4.2 · p. 8

4.3 Language Tokenization

  • Tokenization establishes a bijection between structured commands and integer sequences, using PART delimiters to pack variable-length and hierarchical scene descriptions.The scheme avoids fixed-size slots or padding and permits arbitrarily complex scene representations.Source: §4.3 · pp. 8–9

5 Results

SceneScript is evaluated on layout estimation and 3D object detection using geometric and F1-based metrics. It substantially outperforms layout baselines and reaches competitive object-detection performance, while encoder choice affects performance as scene complexity grows.

5.1 Metrics

  • Layout evaluation matches entities with Hungarian assignment and reports F1 at distance thresholds plus average F1 across thresholds from 1 to 100 cm.Scores are averaged across entity classes and scenes.Source: §5.1 · p. 9

5.2 Layout Estimation

  • SceneScript shows a substantial performance advantage over state-of-the-art layout-estimation baselines across multiple F1-based metrics on Aria Synthetic Environments.Table 2 reports the main quantitative comparison, while qualitative results show baseline degradation on finer details.Source: §5.2 · pp. 10–11, Table 2 · p. 10
  • SceneScript remains accurate across tested encoder configurations and outperforms the included layout baselines by a significant margin.Lifted image features widen the advantage over the RayTran-based encoder, especially as room count and scene complexity increase.Source: Encoder Ablation · p. 11

5.3 Object Detection

  • SceneScript does not predict confidence scores for make_bbox commands, so conventional mAP cannot be fairly computed and F1-based metrics are used instead.The resulting mAP issue includes order variance across scene evaluation.Source: §5.3 · p. 11
  • By adding the make_bbox command, SceneScript achieves object-detection performance on par with state-of-the-art baselines across multiple datasets.On Aria Synthetic Environments, 3DETR performs poorly under noisier semi-dense point clouds, while SceneScript is similar to 3DETR and SoftGroup in the reported comparison.Source: Table 3 · p. 11, §5.3 · p. 12

6 Extending the SceneScript Structured Language

SceneScript extends its structured language to increase reconstruction expressiveness, using volumetric primitives for coarse multi-category object reconstruction and additional commands for future extensions.

6.1 Objects as Volumetric Primitives

  • Volumetric primitives preserve object semantics while representing arbitrary categories with simple cuboids and extruded cylinders.For example, tabletops can be represented by a single cuboid, allowing one language to describe many object categories simultaneously.Source: §6.1 · p. 12
  • The make_prim command encodes each primitive through its center, rotation, scale, and optional semantic primitive number.Its parameters include a bounding-box identifier, primitive number, class, 3D center, 3D rotation, and 3D scale.Source: §6.1 · p. 13
  • Adding one make_prim command enables coarse reconstruction of multiple object categories, including qualitative inferences on real scenes despite synthetic-only training.The proof-of-concept covers tables, chairs, and sofas using cuboids and extruded cylinders.Source: Dataset · p. 13, §6.1 · p. 13, Figure 6 · p. 13

6.2 Further Extensions

  • SceneScript’s structured language can be extended beyond volumetric primitives to curved walls, object states, and parametric geometry models.These further explorations aim to increase reconstruction fidelity and accuracy.Source: §6.2 · pp. 13–14

7 Interactive Scene Reconstruction

SceneScript supports interactive scene reconstruction by streaming live predictions into a VR headset, allowing users to inspect and refine reconstructions while exploring an environment.

7 Interactive Scene Reconstruction

  • 2–5s inference enables SceneScript reconstructions to stream into a modern VR headset at an interactive rate.The system overlays the reconstruction onto the environment being explored in real time.Source: §7 · p. 14
  • Users can refine reconstructions by more thoroughly exploring areas that may have been missed.The interface visualizes this refinement workflow in Figure 1.Source: §7 · p. 14

8 Limitations and Future Work

SceneScript’s manually defined, high-level commands limit automation and fine-grained geometric precision, producing simpler reconstructions that may miss intricate details.

8 Limitations and Future Work

  • Manual command design currently requires human intervention, while high-level commands make millimeter-scale geometric detail difficult to capture.The authors identify automating command definition and improving fine-detail representation as future work.Source: §8 · p. 14

9 Conclusion

SceneScript represents indoor scenes as compact, editable, interpretable sequences of structured language commands predicted autoregressively from video. The representation can be extended with minimal changes to support arbitrary elements, connecting 3D reconstruction with language-model-style scene representations.

9 Conclusion

  • SceneScript autoregressively predicts compact, editable, and interpretable structured scene-language commands from an indoor-space video stream.The method is presented as a tokenized scene representation for reconstruction.Source: §9 · p. 14
  • Minimal language changes allow SceneScript to extend to arbitrary elements such as Bezier curves and object-part decomposition.The conclusion frames these extensions as new directions for representing 3D scenes as language.Source: §9 · p. 15

A Aria Synthetic Environments

  • Aria Synthetic Environments contains randomly selected indoor scenes with bird’s-eye layout views and egocentric close-ups of scene details.The dataset visualizations show room layouts, furniture arrangements, and rendered scene details.Source: Figure 7 · p. 19

A.1 Large Scale Training Dataset

SceneScript is trained on Aria Synthetic Environments, a large synthetic egocentric-scene dataset, and uses structured language commands to represent increasingly detailed scenes. Its layout performance remains strong across encoders, while image-only encoding degrades as scene complexity increases.

A.1 Large Scale Training Dataset

  • Aria Synthetic Environments provides 100k training pairs of photorealistically rendered indoor scenes with structured language commands and an additional 1k test scenes.It includes layout ground truth and is publicly available for non-commercial research use.Source: §A.1 · pp. 19, 21
  • Each training example pairs a rendered egocentric video with ground-truth SceneScript commands generated from a 3D indoor scene and simulated Aria trajectory.The rendering pipeline also produces RGB, depth, instance segmentation, and a SLAM-based point cloud.Source: §A.1 · pp. 19–20

B Structured Language Commands

F.1 Visualization of High-Level Commands

  • Adding make_bbox and make_prim increases reconstruction detail from room layouts to object boxes and volumetric primitives without changing the network architecture.The primitive proof of concept covers chairs, sofas, and tables; unsupported object classes retain empty boxes.Source: §F.1 · p. 23, Figure 9 · p. 23

F.2 Model Performance with respect to Scene Complexity

  • Point-cloud and lifted-feature encoders remain consistently strong as room count increases, whereas image-only performance drops drastically with more rooms.The authors attribute this pattern to weaker occlusion reasoning when image rays traverse multiple rooms without wall intersections.Source: §F.2 · p. 24, Figure 10 · p. 24

F.3 Failure Cases

SceneScript failures arise from incomplete scene observation and incorrect vertical placement, while layout accuracy improves when image features are lifted onto semi-dense point clouds. Object-detection implementation uses the same core architecture, and sparse convolution improves a 3DETR-style baseline on Aria data.

F.3 Failure Cases

  • Incomplete exploration can leave substantial scene structure unobserved, making the ground-truth layout nearly unpredictable.This is identified as the more common notable failure mode.Source: §F.3 · p. 24, Figure 11 · p. 25
  • Accurate room structure is sometimes reconstructed at an incorrect Z-value, with suspected sensitivity to point outliers and greater Z-direction sensitivity than XY.The failure appears in both point-cloud and image-only encoder configurations.Source: §F.3 · p. 24
  • The reported failure visualizations use the worst 10 predictions from the 1,000-scene test set and therefore are not representative of general prediction quality.The selection is based on wall entity distance.Source: §F.3 · pp. 24–25

F.4 Quantitative Evaluation of Layout Predictions

  • Lifting image features onto semi-dense point clouds provides much of its accuracy improvement for windows and doors, while window predictions remain the most challenging class.For every tested encoder setup, 90% of window predictions fall within 0.5 m of ground truth.Source: §F.4 · p. 25, Table 5 · p. 26

G.1 Implementation Details

  • Object detection requires no architectural changes: SceneScript adds make_bbox while retaining the point-cloud encoder, language decoder, and single cross-entropy objective.The experiment uses a point-cloud version of SceneScript trained for approximately 200k iterations.Source: Training Details of SceneScript · p. 26

G.2 Sparse Encoder with 3DETR Head

  • Replacing 3DETR’s Transformer encoder with a sparse CNN produces stronger performance on Aria Synthetic Environments.The authors hypothesize that sparse convolutions better handle the non-uniformity of Project Aria’s semi-dense point clouds.Source: §G.2 · pp. 27–28, Table 6 · p. 27

G.3 Qualitative Results on Aria Synthetic Environments

On Aria Synthetic Environments, large objects are easier to detect than small ones, and average precision is problematic when SceneScript cannot rank detections across scenes. Structured-language extensions support curved walls, composite walls, door states, and Blender procedural models.

G.3 Qualitative Results on Aria Synthetic Environments

  • Large categories such as beds and sofas are usually detected correctly, whereas small objects such as jars and flower pots are substantially harder.Class-averaged F1 can therefore be low even when qualitative predictions appear reasonable.Source: §G.3 · pp. 28–29, Figure 13 · p. 29

G.5 Discussion of Average Precision Metric

  • SceneScript uses F1-based detection metrics because average precision is order-sensitive when predictions lack comparable confidence scores across scenes.The same detections can yield different precision-recall curves when scene evaluation order changes.Source: §G.5 · pp. 30–31, Figure 14 · p. 30, Table 8 · p. 31

H.1 Extending SceneScript with Curved Entities

  • A new make_curved_wall command extends the language to extruded Bezier walls, whose predicted geometry is described as nearly indistinguishable from ground truth.The extension adds four control-point parameters for wall curvature.Source: §H.1 · pp. 31–32, Figure 15 · p. 32

H.2 Extending SceneScript to Compositions of Wall Primitives

  • Composite walls are represented by combining a base wall with make_wall_prim cuboids, and the network infers these compositions from noisy surface point clouds in a proof of concept.The evaluation reports correctly predicted composite-wall parameters within specified bounds.Source: §H.2 · p. 33, Table 10 · p. 33

H.3 Extending SceneScript to Object States

  • Door-state estimation adds open_degree, hinge_side, and open_direction to the door command while keeping the input and network architecture unchanged.The qualitative experiment estimates opening angle, hinge position, and opening side.Source: §H.3 · p. 34

H.4 Extending SceneScript to Blender Parametric Object Models

  • A Blender Geometry Node experiment shows reasonable real-world table reconstructions from predicted procedural-language parameters, including an unseen high sides_num prediction for a circular tabletop.Training uses synthetic RGB images of randomly parameterized tables composited on random backgrounds.Source: §H.4 · pp. 34–35, Figure 17 · p. 34

Related papers

Loading 2403.13064v1…