PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialLM: Training Large Language Models for Structured Indoor Modeling

Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou

arXiv:2506.07491v2cs.CV

TL;DR

Structured indoor modeling seeks to recover architectural layouts and 3D objects from unstructured point clouds, but large, high-quality training data and general LLM-based approaches remain limited. SpatialLM addresses this by fine-tuning open-source LLMs with a large synthetic dataset and structured text outputs, achieving competitive public-benchmark performance and demonstrating feasibility for future 3D scene foundation models.

  • Problem

    Structured indoor modeling aims to recover architectural layouts and 3D objects from raw point clouds, but real datasets are small and laborious to annotate, with most lacking structural annotations.

  • Method

    SpatialLM represents layouts and objects as general-purpose-language scripts, fine-tunes open-source LLMs, and studies point-cloud features and their integration into LLMs using a synthetic dataset.

  • Results

    SPATIALLM achieves competitive performance on public layout-estimation and 3D object-detection benchmarks and shows resilience in zero-shot experiments on diverse point clouds.

  • Takeaways & Limitations

    The study demonstrates the feasibility of training LLMs for structured indoor modeling as a step toward foundation models for understanding, reasoning about, and interacting with structured 3D scenes.

  • Takeaways & Limitations

    SPATIALLM is not universal across arbitrary point clouds, and dataset-specific fine-tuning remains necessary for best performance.

Abstract

from arXiv · show

SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.

1 Introduction

SPATIALLM frames structured indoor scene descriptions as editable Python-like text and predicts them with a standard multimodal LLM pipeline. The work addresses limited 3D modeling data by introducing a large synthetic dataset and reports competitive benchmark performance.

  • Representation: Structured scene descriptions compactly represent indoor layouts and objects while remaining human interpretable, editable, and extensible.The text representation also leverages pretrained LLM coding capabilities.
  • Modeling approach: The method directly fine-tunes open-source LLMs for structured indoor modeling from point cloud inputs.Its output includes structured scene descriptions generated in text form.
  • Dataset: 12,328 distinct scenes and 54,778 rooms form the synthetic dataset, paired with 3D structure information.The dataset supports study of point-cloud feature alignment with LLMs.
  • System: SPATIALLM processes point clouds and generates structured scene descriptions using an Encoder-MLP-LLM architecture.The reconstructed 3D structure can be overlaid on the input point cloud for visualization.
  • Evaluation: The model achieves competitive performance on public layout estimation and 3D object detection benchmarks.The paper also reports proof-of-concept zero-shot handling of point clouds from monocular video sequences.

2 Related Work

Prior work uses geometric methods, specialized networks, scene encoders, or custom language interfaces for 3D scene understanding. SPATIALLM instead applies a standard Encoder-MLP-LLM design to structured reconstruction from point clouds.

  • Layout estimation: Room layout methods range from RANSAC-based wall-plane detection to joint CAD alignment and Transformer-based corner and room prediction.These approaches focus primarily on recovering layout structure from point clouds.
  • Sequence modeling: SceneScript models structured indoor reconstruction as sequence prediction but requires domain-specific language-command tokens and a specialized decoder.SPATIALLM instead fine-tunes modern foundation models within the standard Encoder-MLP-LLM architecture.
  • 3D-LLM integration: Several 3D-LLM systems first detect objects and encode object-level features before passing them to an LLM.Other systems directly align scene-level point-cloud features with LLMs.
  • 3D-LLM integration: Scene-level integration methods include image-feature back-projection, Q-former token mapping, fine-grained 3D encoding, and voxel-grid aggregation.These designs differ in how point-cloud information is converted into LLM inputs.
  • Alternative 3D integration: Other approaches enhance 3D reasoning through synthetic VQA data, 3D position embeddings, image-derived scene graphs, or video-frame representations.These methods target broader spatial understanding rather than the same structured reconstruction formulation.

3 SPATIALLM

SPATIALLM formulates structured indoor modeling as predicting general-purpose language scripts for layouts and objects from point clouds, using an Encoder-MLP-LLM pipeline. The section develops a large human-authored synthetic dataset, evaluates layout and detection with IoU-based F1 scores, and studies encoder and tokenization choices.

  • Problem formulation: Given a point cloud P, SPATIALLM predicts language scripts describing 3D layouts of walls, doors, windows, and scene objects.The method fine-tunes LLMs to generate structured scene descriptions in text form.
  • Dataset and modeling questions: The authors investigate point-cloud features and their integration into LLMs after introducing a dataset for structured scene reconstruction.The study explicitly examines encoder selection and feature integration strategies.
  • Dataset motivation: Real datasets are limited in scale and annotation coverage, while synthetic datasets trade realism or diversity depending on how scenes are constructed.Real collections require laborious RGBD scanning and annotation; procedural generation increases scale but reduces scene quality.
  • Dataset construction: 12,328 scenes and 54,778 rooms are collected from professionally authored interior designs for training structured indoor modeling LLMs.The dataset uses an online repository of interior designs and is filtered into individual rooms.
  • Evaluation: Layout estimation and 3D object detection are evaluated with Hungarian matching and F1-scores at IoU2D or IoU3D thresholds of 0.25 and 0.5.Layout elements are projected onto ground-truth planes for IoU2D, while matched objects use 3D intersection-over-union.
  • Point cloud encoders: Scene-level point clouds produce too many point features for LLMs, motivating an encoder that maps N input points to K feature embeddings with K ≪N.Each point contains XYZ and RGB, while the output embeddings are consumed by the multimodal LLM architecture.
  • Encoder design: Random point sampling and voxelized image-feature alternatives yield extremely low F1-scores because naive down-sampling loses spatial information.The authors identify spatial information as essential for reconstructing geometric coordinates.
  • Encoder design: Learned encoders preserve semantic and geometric information, while adding DINOv2 features gives slightly better results through enhanced contextual information.Sonata achieves the best results among evaluated encoders with raw points as input.

4 Experiments

SPATIALLM is evaluated on layout estimation, 3D object detection, and zero-shot video-reconstructed point clouds using standard RGBD benchmarks and qualitative comparisons. Pretraining on the SPATIALLM dataset followed by downstream fine-tuning supports competitive benchmark performance and qualitative robustness to imperfect inputs.

  • Experimental Setup: The experiments evaluate layout estimation on Structured3D, 3D object detection on ScanNet, and zero-shot detection on reconstructed indoor-video point clouds.The study uses established RGBD benchmarks and additionally tests inputs reconstructed from RGB videos.
  • 4.1 Layout Estimation: RoomFormer uses specialized two-level queries, whereas SceneScript and SPATIALLM formulate structured prediction autoregressively.RoomFormer predicts room polygons and corners through separate query levels; autoregressive methods preserve structural relationships by design.
  • 4.1 Layout Estimation: Auto-regressive methods preserve wall-attachment relationships, while SceneScript tends to miss some architectural elements in qualitative comparisons.RoomFormer independently detects room elements, producing examples where doors and windows are not attached to walls.
  • 4.2 3D Object Detection: ScanNet-only training yields very low F1 scores, while dataset pretraining followed by ScanNet fine-tuning outperforms SceneScript and is competitive with V-DETR.The comparison includes variants trained on ScanNet only, the synthetic dataset only, and sequential pretraining followed by ScanNet fine-tuning.
  • 4.2 3D Object Detection: SPATIALLM improves detection accuracy over SceneScript but struggles with small picture and sink objects and the infrequent showercurtain category relative to V-DETR.Picture and sink occupy less than 0.1m3, while showercurtain is absent from the training dataset.
  • 4.3 Zero-shot Detection on Videos: Without fine-tuning, SPATIALLM preserves consistent layouts and object predictions on noisy, occluded, and geometrically imperfect video reconstructions.The model predicts object sizes and orientations from sparse inputs and plausibly completes partially observed room layouts.

5 Conclusion

The study demonstrates the feasibility of training LLMs for structured indoor modeling and frames this as a step toward foundation models for structured 3D scene understanding. The authors also identify boundaries in generality, retained language capabilities, and vocabulary scope.

  • Conclusion: The study demonstrates the feasibility of training LLMs for structured indoor modeling tasks.The conclusion presents this as a step toward future foundation models that understand, reason about, interact with, and create structured 3D scenes.
  • Limitations: SPATIALLM is not a universal state-of-the-art model for arbitrary point clouds, and dataset-specific fine-tuning remains necessary for best performance.Point clouds from monocular videos, RGBD scans, and LiDAR sensors differ substantially.
  • Limitations: The paper focuses on structured indoor modeling without evaluating effects on natural-language processing and reasoning skills.The authors identify broader benchmarking as a future need.
  • Limitations: Predefined object categories limit open-ended language capabilities, motivating future open-vocabulary detection and 3D visual question answering.These extensions are proposed to support more flexible and generalizable scene understanding.

A SPATIALLM Dataset

The SPATIALLM dataset provides large-scale indoor scene and room coverage with structural annotations and a curated category inventory. Its statistics indicate realistic object sizes, room-specific object correlations, and diverse scene content.

  • Dataset Scale: The dataset contains 12,328 distinct scenes and 54,778 rooms, including 403,291 walls, 123,301 doors, and 48,887 windows.Its total floor area is approximately 863,986 m2.
  • Object Categories: The dataset organizes 59 commonly occurring object categories by functional and semantic similarity.Category statistics are presented in Table 7 and Figure 7.
  • Room-Object Correlations: Object distributions correlate strongly with specific room types, while each room type contains diverse object categories.Figure 8 visualizes these relationships as an object-to-room correlation heatmap.
  • Dataset Quality: Visualizations showcase photo-realistic scans, realistic layout annotations, and diverse, high-quality rendered objects.The dataset visualizations include ground-truth point clouds, layout annotations, and rendered object images.

B Implementation Details of SPATIALLM

Implementation maps point clouds and structured outputs into a coordinate representation suitable for LLM prediction, while training uses augmentation, large-scale pretraining, and dataset-specific fine-tuning. The appendix also documents dataset statistics and room-object correlations.

  • Coordinate Representation: Coordinates are shifted to non-negative values, quantized into 1,280 bins at 2.5cm resolution, then mapped back to continuous original coordinates.Inverse transformation and normalization restore the predicted coordinates to the original coordinate system.
  • Data Augmentations: Point-cloud augmentation includes cuboid cropping with a minimum aspect ratio of 0.8 and at least 50,000 retained points.The crop center and dimensions are randomly selected relative to the input point cloud.
  • Data Augmentations: Four jitter types simulate sensor uncertainty, high-uncertainty regions, foreground-background edges, and distant reflective or floating points.These noises target distinct sources of imperfection in indoor point clouds.
  • Compute Resources: SPATIALLM training uses 4 epochs, total batch size 64, learning rate 10^-4, cosine scheduling, and 32 NVIDIA H20 GPUs for approximately one day.AdamW uses beta values 0.9 and 0.99, epsilon 1 × 10^-8, and a warm-up ratio of 0.03.
  • Fine-tuning: Structured3D fine-tuning runs for 50 epochs, while ScanNet training adds 3 epochs of mapped-label pretraining before 30 epochs of ScanNet fine-tuning.Experiments use batch size 8, and label mapping mitigates differences between dataset label spaces.

C Supplementary Results for SPATIALLM Experiments

This section reports per-category evaluation across layout estimation, 3D object detection, and zero-shot video detection, alongside point-cloud augmentations.

  • The section also includes per-category zero-shot detection results on videos.
  • Per-category results cover layout estimation at IoU2D thresholds of 0.25 and 0.5 on the Structured3D test set.
  • Layout estimation results are reported in Table 9, while point-cloud data augmentations are summarized in Table 8.

C.2 3D Object Detection

SPATIALLM is evaluated for 3D object detection across 18 ScanNet categories using F1 scores at two IoU3D thresholds. It substantially outperforms SceneScript and is near V-DETR at IoU3D@0.25, but trails V-DETR at IoU3D@0.5.

  • Evaluation reports F1 scores at IoU3D thresholds of 0.25 and 0.5 across 18 ScanNet object categories.
  • +16.5% overall gain over SceneScript and +0.5% difference from V-DETR at IoU3D@0.25.
  • +15.8% overall gain over SceneScript and -4.2% difference from V-DETR at IoU3D@0.5.
  • The largest gaps from V-DETR occur for picture, sink, and shower_curtain, which are among the smallest or least occurring ScanNet objects.
  • Tables 10 and 11 provide the quantitative results for IoU3D@0.25 and IoU3D@0.5, respectively.

C.3 Zero-shot Detection on Videos

The video experiment tests zero-shot detection on reconstructed point clouds from 107 virtual room-tour videos. Despite noisy, incomplete inputs and alignment errors, SPATIALLM is qualitatively robust across diverse point-cloud sources.

  • 107 virtual room-tour videos are reconstructed with MASt3R-SLAM and evaluated using aligned ground-truth annotations.
  • The video point clouds are often noisy and highly incomplete, while imperfect trajectory alignment can reduce correspondence with ground-truth annotations.
  • Table 12 reports zero-shot layout estimation and 3D object detection results at IoU@0.25.
  • The broader experiments consider text-to-3D, hand-held video, LiDAR-based reconstruction, and synthetic-mesh point clouds.
  • SPATIALLM is robust to variations in point-cloud origin, structure, and appearance.

D.2 Task Adaptation via Language-based Prompts

SPATIALLM represents scene structures as text and adapts to downstream tasks through natural-language instructions without changing its architecture. Demonstrations include category-conditioned detection and semantic label completion.

  • Task adaptation via language-based prompts: Text-based scene structures allow SPATIALLM to adapt to downstream tasks through natural-language instructions without architectural changes.
  • Detection with user-specified categories: User-specified detection prompts make the model predict oriented bounding boxes only for instances matching selected categories.
  • Semantic label completion: Semantic label completion supplies object boxes and poses while leaving categories unspecified for prediction from layout and point clouds.
  • Semantic label completion: 96.8% overall classification accuracy is achieved on the dataset’s semantic label completion test set.
Loading 2506.07491v2…