PaperlayerView exact revision on arXiv

Source-linked AI summary

TUN3D: Towards Real-World Scene Understanding from Unposed Images

Anton Konushin, Nikita Drozdov, Bulat Gabdullin, Alexey Zakharov, Anna Vorontsova, Danila Rukhovich, Maksim Kolodiazhnyi

arXiv:2509.21388v1cs.CVeess.IV

TL;DR

Indoor scene understanding seeks a compact description of room structure and major objects, but point-cloud input can require depth sensors or accurate reconstruction. TUN3D jointly estimates layouts and detects 3D objects from multi-view images, including unposed images. It reports state-of-the-art performance across point-cloud, posed-image, and unposed-image settings.

  • Problem

    Indoor scene understanding needs compact structural and object descriptions, while point-cloud inference can require depth sensors or accurate multi-view reconstruction.

    Source: §I · p. 1

  • Method

    TUN3D uses a lightweight sparse-convolutional model with separate heads for joint layout estimation and 3D object detection, plus a compact layout parameterization.

    Source: §I · p. 1, §VII · p. 7

  • Results

    TUN3D sets a new state-of-the-art in joint layout estimation and 3D object detection from ground-truth point clouds, posed images, and unposed images.

    Source: §VII · p. 8

  • Takeaways & Limitations

    A compact room-layout and major-object representation is especially beneficial for applications running on a device.

    Source: §I · p. 1

Abstract

from arXiv · show

Layout estimation and 3D object detection are two fundamental tasks in indoor scene understanding. When combined, they enable the creation of a compact yet semantically rich spatial representation of a scene. Existing approaches typically rely on point cloud input, which poses a major limitation since most consumer cameras lack depth sensors and visual-only data remains far more common. We address this issue with TUN3D, the first method that tackles joint layout estimation and 3D object detection in real scans, given multi-view images as input, and does not require ground-truth camera poses or depth supervision. Our approach builds on a lightweight sparse-convolutional backbone and employs two dedicated heads: one for 3D object detection and one for layout estimation, leveraging a novel and effective parametric wall representation. Extensive experiments show that TUN3D achieves state-of-the-art performance across three challenging scene understanding benchmarks: (i) using ground-truth point clouds, (ii) using posed images, and (iii) using unposed images. While performing on par with specialized 3D object detection methods, TUN3D significantly advances layout estimation, setting a new benchmark in holistic indoor scene understanding. Code is available at https://github.com/col14m/tun3d .

I. INTRODUCTION

TUN3D targets joint indoor layout estimation and 3D object detection from multi-view images, including unposed images, where point-cloud methods face input constraints. It introduces a joint architecture and compact layout parameterization, and reports state-of-the-art performance across three input scenarios.

I. INTRODUCTION

  • A compact description of room layout and major objects is lighter than dense 3D reconstruction and especially beneficial for on-device applications.Source: §I · p. 1
  • Earlier joint point-cloud methods were either extremely slow or less accurate than single-task methods, motivating TUN3D’s real-time-model foundation.Source: §I · p. 1
  • Point-cloud inference can require depth sensors or accurate multi-view reconstruction, limiting use where only video is available.Source: §I · p. 1
  • TUN3D relaxes input requirements to multi-view images, with or without camera poses, and introduces a compact layout parameterization and joint architecture.Source: §I · p. 1
  • TUN3D establishes state-of-the-art scene-understanding performance from ground-truth point clouds, posed multi-view images, and unposed multi-view images.Source: §I · p. 1

A. Scene Understanding from Point Clouds

Prior work spans point-cloud methods, posed-image methods, and single-view or panoramic approaches, with gaps in real-world joint layout and object prediction from unposed images. TUN3D uses a sparse-convolutional backbone and two task-specific heads to process multiple input types.

A. Scene Understanding from Point Clouds

  • Point-cloud object detectors use voting, transformers, or sparse 3D convolutions; TUN3D follows the sparse-convolutional paradigm for speed, accuracy, and scalability.Source: A. Scene Understanding from Point Clouds · p. 1
  • Layout estimation has been studied alone and jointly with object detection, while SceneScript and SpatialLM use synthetic data and report no layout results on real scans.Source: A. Scene Understanding from Point Clouds · pp. 1–2
  • TUN3D’s figure shows a sparse-convolutional backbone and neck feeding separate layout and object-detection heads, with unposed images, posed images, and point clouds as inputs.Source: Figure 1 · p. 2

B. Scene Understanding from Posed Images

C. Scene Understanding from Unposed Images

III. SCENE UNDERSTANDING FROM POINT CLOUD

The problem formulation represents a scene using a point cloud, detected 3D objects, and a layout made of walls. Objects are described by bounding boxes and categories, while walls are specified by their corner coordinates.

A. Problem Formulation

B. Network Architecture

TUN3D uses a sparse-convolutional backbone and separate heads for object detection and layout estimation. Its wall parameterization reduces layout prediction to the bird’s-eye-view plane while retaining scene-height information.

B. Network Architecture

  • The backbone voxelizes the point cloud at 2 cm and uses four sparse residual levels, while the neck aggregates their features and expands visibility at the 64 cm and 32 cm levels.The sparse-convolutional channels are capped at 128 for efficiency because larger values compromise inference speed.Source: B. Network Architecture · pp. 2–3
  • The detection head predicts, at each location, class logits, a three-dimensional offset for an object's center, and three-dimensional log-sizes for its 3D bounding box.Separately, the layout head takes 32-cm-level features as input, produces walls, predicts wall probability scores, and estimates wall parameters.Source: B. Network Architecture · p. 3

C. Wall Parameterization

  • The alternatives include four independent 3D corner offsets or two 3D lower-corner offsets plus height, while the proposed bird’s-eye-view formulation reduces degrees of freedom to encourage cleaner, more robust geometry.The paper notes that unconstrained 3D offsets can produce malformed scene geometry.Source: C. Wall Parameterization · p. 3
  • The proposed wall representation predicts two 2D offsets for lower corners and wall height, deriving upper corners by adding height along the global up axis.The layout head projects features onto the floor plane and appends an encoded scene-height distribution to restore height information lost during pooling.Source: C. Wall Parameterization · p. 3

D. Training

Training assigns predicted locations to ground-truth objects and walls using task-specific matching strategies, then applies separate classification and regression losses to the corresponding outputs.

D. Training

  • Object matching assigns each ground-truth object to six nearby locations at a category-specific feature level, typically 32 cm for large objects and 16 cm for smaller ones.Wall matching instead assigns each wall to the six nearest locations at the 32 cm level, or to their 2D floor-plane projections.Source: D. Training · p. 3
  • Object classification uses focal loss and 3D box regression uses DIoU loss, while wall classification uses focal loss and wall-parameter regression uses L1 loss.The losses train specific output layers in the two heads.Source: D. Training · p. 4

IV. TOWARDS SCENE UNDERSTANDING FROM UNPOSED IMAGES

The study evaluates scene understanding with progressively less input information, including unposed images, to examine visual-only settings relevant to casual-camera video.

IV. TOWARDS SCENE UNDERSTANDING FROM UNPOSED IMAGES

  • The experiments compare point clouds, posed images, and unposed images as scene-understanding inputs, moving beyond the most informative but resource-intensive point-cloud modality.The motivation includes customer devices without depth sensors or trackers and prerecorded video.Source: §IV · p. 4

A. Posed Images

For posed-image input, TUN3D uses DUSt3R to estimate dense depths, fuses them with images using ground-truth poses, and extracts a point cloud for the point-cloud pipeline.

A. Posed Images

  • DUSt3R supports both pose-aware and pose-agnostic operation, allowing the same image-to-point-cloud approach to serve posed-image and unposed-image settings.In the posed setting, camera intrinsics and extrinsics accompany the images; poses may come from specialized hardware or software trackers.Source: A. Posed Images · p. 4
  • The authors choose DUSt3R to avoid data leakage because it was not trained on ScanNet, unlike some competing methods, including Mast3R.They also state that this preserves the methodological purity of their experimental protocol.Source: A. Posed Images · p. 4
  • DUSt3R estimates dense depth maps, which are fused with the original images and ground-truth camera poses into a TSDF volume before point-cloud extraction.The remaining processing steps match the point-cloud-based scenario.Source: A. Posed Images · p. 4

B. Unposed Images

The unposed setting uses image collections without known camera intrinsics or extrinsics, a scenario relevant to casual-device captures and videos missing capture information. The experiments span several indoor-scene datasets with differing annotations and evaluation roles.

B. Unposed Images

  • Unposed input is an image collection without known camera intrinsics or extrinsics, matching casual-device captures and videos with missing capture information.DUSt3R jointly predicts depth maps and camera parameters; estimated poses are used for TSDF integration.Source: B. Unposed Images · p. 4

A. Datasets

  • The experiments cover ScanNet, S3DIS, ARKitScenes, and Structured3D, with ARKitScenes used only for cross-dataset experiments because its training subset lacks layout labels.ScanNet layouts come from SceneCAD, S3DIS wall bounds are annotated by the authors, and Structured3D uses SpatialLM annotations lifted from floorplans.Source: A. Datasets · p. 4
  • ScanNet has 1,201 training and 312 validation scans; S3DIS evaluation uses Area 5, with training on the remaining areas.ARKitScenes has 4,493 training and 549 validation scans, while Structured3D contains 3.5K house designs.Source: A. Datasets · p. 4

B. Implementation Details

The section specifies the model's layout-head design and evaluation metrics, then reports comparisons across point-cloud, posed-image, and unposed-image settings. TUN3D outperforms the stated baselines across these image settings and leads prior methods on four point-cloud benchmarks.

B. Implementation Details

  • The layout head encodes 10 z-quantiles as a 40-dimensional vector and concatenates it with 128-channel neck features to form 168-channel floor-projected features.The backbone and neck match TR3D; inference applies separate NMS to object and wall predictions.Source: B. Implementation Details · p. 5

C. Metrics

  • Object detection uses mAP at IoU thresholds 0.25 and 0.5; layout accuracy uses F1, with Structured3D matching walls by floorplan-projection IoU.Other layout benchmarks match walls by maximum corner-to-corner distance.Source: C. Metrics · p. 5

A. Comparison with Prior Approaches

  • Figures 3 and 4 show ground-truth and predicted layouts and objects on ScanNet and S3DIS.Source: Figure 3 · p. 6, Figure 4 · p. 6
  • Layout gains are +23.6 F1 over PQ-Transformer on S3DIS, +5.8 and +4.4 F1 over Omni-PQ on ScanNet and ARKitScenes, and +4.0 F1@0.25 over SpatialLM on Structured3D.The passage also reports state-of-the-art object detection on S3DIS and superiority over depth-agnostic methods, including depth-aware methods on mAP@0.5.Source: A. Comparison with Prior Approaches · pp. 5–6, Table II · p. 6, Table III · p. 6
  • TUN3D leads prior methods on four point-cloud benchmarks, beats the posed-image baseline on both tasks, and beats the unposed baseline, whose layouts are less accurate than the posed baseline.The unposed comparison indicates that layouts can be more accurate without camera poses than those from DUSt3R → PQ with poses.Source: A. Comparison with Prior Approaches · pp. 5–6

B. Ablation Experiments

The ablations compare TUN3D’s efficiency and accuracy with alternative methods and examine how pose estimation, image count, spatial encoding, and wall parameterization affect performance.

Inference time

  • TUN3D is 4x faster than PQ-Transformer and orders of magnitude faster than SpatialLM in the inference-time comparison.Table IV compares inference time with posed images from ScanNet and S3DIS.Source: Table IV · p. 7, B. Ablation Experiments · p. 7

Backbone architecture

  • TUN3D is 1.7x faster than UniDet3D while improving layout F1 by +4.4, but object-detection results vary by dataset.TUN3D trails UniDet3D on ScanNet and exceeds it on S3DIS by +1.2 mAP@0.5; the authors identify no ultimate leader.Source: B. Ablation Experiments · p. 7

Pose estimation

  • DUSt3R pose estimation yields a 2x improvement in both layout-estimation and 3D object-detection accuracy over DROID-SLAM.The comparison is for unposed-image processing on videos such as ScanNet.Source: B. Ablation Experiments · p. 7

Number of images

  • Layout and object-detection scores increase with the number of input frames; TUN3D uses 45 frames in its final version.The passage notes that ImVoxelNet uses 50 images and NeRF-Det uses 100.Source: B. Ablation Experiments · p. 7, Table VI · p. 7

Number of z-quantiles

  • 10 z-quantiles add +5.2 F1 on ScanNet and +4.3 F1 on S3DIS with negligible computation overhead and no effect on inference time.The model has reasonable performance even without this spatial information.Source: B. Ablation Experiments · p. 7, Table VII · p. 7

Wall parameterization

VII. CONCLUSION

TUN3D jointly predicts layouts and 3D objects from multi-view images in real scans, without ground-truth camera poses or depths. Across ground-truth point clouds, posed images, and unposed images, the authors report state-of-the-art performance.

VII. CONCLUSION

  • TUN3D uses a lightweight sparse-convolutional model with separate layout and object-detection heads and a novel layout parameterization.It is the first method described for joint layout prediction and 3D object detection in real scans from multi-view images.Source: §VII · pp. 7–8
  • TUN3D is trained without ground-truth camera poses or depths, relaxing input requirements to casually captured images or videos..Source: §VII · p. 7
  • Across ground-truth point clouds, posed images, and unposed images, TUN3D sets a new state of the art in joint layout estimation and 3D object detection.The authors describe this as a new milestone in holistic indoor scene understanding.Source: §VII · p. 8
Loading 2509.21388v1…