PaperlayerView exact revision on arXiv

Source-linked AI summary

Atlas: End-to-End 3D Scene Reconstruction from Posed Images

Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, Andrew Rabinovich

arXiv:2003.10432v3cs.CV

TL;DR

Existing RGB-only approaches often predict depth maps that remain less accurate than depth sensors and do not fuse satisfactorily into 3D models. This paper directly regresses a full 3D model from posed RGB sequences, significantly outperforming state-of-the-art multiview stereo baselines on ScanNet while also supporting semantic segmentation with minimal extra computation.

  • Problem

    RGB-only 3D reconstruction commonly relies on intermediate depth maps that are less accurate than depth sensors and do not produce satisfactory fused 3D models.

  • Method

    The method back-projects image features into a voxel volume and uses a 3D CNN to directly predict a full TSDF model end to end from posed RGB sequences.

  • Results

    On real indoor-room scans from ScanNet, the method significantly outperforms state-of-the-art multiview stereo baselines and produces accurate, complete meshes.

  • Takeaways & Limitations

    Direct 3D regression supports reconstruction from RGB alone, completion of unobserved geometry, and semantic labeling with little additional computation.

  • Takeaways & Limitations

    Dense ray-based feature volumes prevent sparse convolutions in the encoder, although the models train at 4cm3 voxel resolution without them.

Abstract

from arXiv · show

We present an end-to-end 3D reconstruction method for a scene by directly regressing a truncated signed distance function (TSDF) from a set of posed RGB images. Traditional approaches to 3D reconstruction rely on an intermediate representation of depth maps prior to estimating a full 3D model of a scene. We hypothesize that a direct regression to 3D is more effective. A 2D CNN extracts features from each image independently which are then back-projected and accumulated into a voxel volume using the camera intrinsics and extrinsics. After accumulation, a 3D CNN refines the accumulated features and predicts the TSDF values. Additionally, semantic segmentation of the 3D model is obtained without significant computation. This approach is evaluated on the Scannet dataset where we significantly outperform state-of-the-art baselines (deep multiview stereo followed by traditional TSDF fusion) both quantitatively and qualitatively. We compare our 3D semantic segmentation to prior methods that use a depth sensor since no previous work attempts the problem with only RGB input.

1 Introduction

The paper replaces depth-map-first reconstruction with direct, end-to-end prediction of a full 3D model from posed RGB sequences. Its feature-volume pipeline outperforms multiview-stereo baselines on ScanNet and adds 3D semantic segmentation with minimal extra computation.

  • Motivation: RGB-only reconstruction avoids special depth hardware but must address noisy, missing, and incomplete depth estimates from stereo methods.Depth sensors are effective but costly and can fail on low-albedo, glossy, or occluded surfaces; predicted depth maps remain less accurate and fuse poorly.
  • Approach: The method directly predicts a full 3D model from an RGB image sequence instead of first estimating and fusing per-image depth maps.The authors argue that direct 3D regression lets the network fuse more information, learn geometric priors, simplify frame handling, and amortize computation across the sequence.
  • Approach: Image features are extracted with a 2D CNN, back-projected into a canonical voxel volume, fused across frames, and refined by a 3D CNN that predicts TSDF values.Unlike cost-volume methods, the approach maps pixels to rays in a shared voxel volume rather than warping features into a target-view frustum.
  • Results: On real indoor ScanNet scans, the method significantly outperforms state-of-the-art multiview-stereo baselines and produces accurate, complete meshes.The evaluation compares direct regression against multiview stereo followed by conventional TSDF fusion.
  • Results: 3D semantic segmentation is added through an extra head with minimal computation, establishing a strong RGB-only baseline for a task previously assuming depth input.The paper positions this as an additional output of the reconstructed 3D model rather than a separate pipeline.

2 Related Work

Prior 3D reconstruction methods commonly fuse sensor depth or predicted depth, while direct image-to-3D methods largely target isolated objects. The paper extends direct 3D prediction to scene-scale RGB reconstruction and proposes an RGB-only semantic-segmentation baseline.

  • 2.1 3D reconstruction: Conventional scene reconstruction acquires depth for image sequences and fuses the maps into voxel or surfel-based 3D structures.TSDF fusion is common for voxel accumulation, while surfels provide an oriented-point-cloud alternative.
  • 2.1 3D reconstruction: Monocular, binocular, and multiview stereo estimate depth from RGB, but multiview systems require target-image and neighboring-frame selection.The paper contrasts these independently or weakly coupled depth predictions with a single coherent 3D model.
  • 2.1 3D reconstruction: Learned TSDF-refinement methods improve noisy or incomplete fused depth, including systems designed for objects, scan portions, and entire rooms.These methods generally begin with depth-derived TSDF volumes rather than predicting scene geometry directly from RGB.
  • 2.1 3D reconstruction: Earlier direct image-to-3D methods encode images into compact latent representations and mainly predict object-level occupancy or signed-distance outputs.Reported evaluations focus largely on single objects from ShapeNet rather than full indoor scenes.
  • 2.2 3D Semantic Segmentation: Existing semantic-labeling approaches either back-project 2D predictions or predict labels in 3D, generally assuming depth from a sensor.The paper identifies RGB-only 3D semantic segmentation as relatively untouched and proposes a strong baseline.

3 Method

The method directly regresses a scene TSDF from posed RGB-image features by back-projecting and averaging them in a voxel volume, then refining the volume with a 3D CNN. It also supports joint 3D semantic segmentation and coarse-to-fine TSDF prediction.

  • 3.1 Feature Volume Construction: The system extracts features from each RGB image with a 2D CNN, back-projects them into a canonical 3D voxel volume, and accumulates them over the sequence.Back-projection uses known camera intrinsics and extrinsics; each pixel feature fills the voxels along its camera ray.
  • 3.1 Feature Volume Construction: A weighted running average fuses feature volumes across frames, using a binary mask to indicate whether each voxel lies inside the camera frustum.The accumulation follows a TSDF-fusion-like update with frustum-based weights.
  • 3.2 3D Encoder-Decoder: A 3D convolutional encoder-decoder refines the accumulated features and directly regresses the output TSDF.The architecture uses residual blocks, stride-2 convolutions for downsampling, trilinear interpolation for upsampling, and additive skip connections.
  • 3.2 3D Encoder-Decoder: An additional 1x1x1 convolutional head predicts 3D semantic-segmentation logits alongside the TSDF reconstruction.This enables joint reconstruction and semantic segmentation within the same 3D network.
  • 3.2 3D Encoder-Decoder: Intermediate TSDF heads provide multiscale supervision and sparsify later resolutions by clamping voxels predicted near the truncation boundary.Loss is backpropagated only for nonclamped voxels, encouraging later stages to refine predictions near surfaces.
  • 3.2 3D Encoder-Decoder: The dense ray-based feature volume prevents sparse convolutions in the encoder, although multiscale decoder outputs can sparsify decoder feature volumes.The models were trained at 4cm3 voxel resolution without sparse convolutions.

4 Implementation Details

The implementation uses a ResNet50-FPN image backbone, a four-scale 3D resolution pyramid, multiscale TSDF supervision, and large voxel grids for training and testing.

  • 4 Implementation Details: The 2D backbone is a ResNet50-FPN with the merging method of and 32 output feature channels.The 3D CNN uses a four-scale resolution pyramid, doubling channels whenever resolution is halved.
  • 4 Implementation Details: The 3D encoder uses (1,2,3,4) residual blocks across scales, while the decoder uses (3,2,1) residual blocks.This specifies the residual-block allocation across the four-scale 3D architecture.
  • 4 Implementation Details: Multiscale TSDF predictions are supervised with log-transformed ℓ1 loss against observed ground-truth voxels.Voxels whose entire vertical column equals 1 are additionally penalized to reduce artifacts outside the room and behind walls.
  • 4 Implementation Details: Training uses 50 randomly selected images, 4cm3 voxels, and a 160×160×64 grid covering 6.4 × 6.4 × 2.56 meters.At test time, in-place accumulation supports arbitrary-length sequences and a 400×400×104 grid covering 16 × 16 × 4.16 meters.

5 Results

On ScanNet, the method produces accurate and complete reconstructions, fills unobserved geometry, supports RGB-only semantic labeling, and remains effective with fewer frames while becoming faster than DPSNet after approximately four frames.

  • Evaluation: The method is evaluated on ScanNet using 2D depth and 3D metrics, with qualitative comparisons against four state-of-the-art baselines.The baselines are COLMAP, MVDepthNet, GPMVS, and DPSNet; predicted depth maps are fused for 3D evaluation.
  • 3D reconstruction: Direct TSDF regression fills holes caused by sensor limitations, occlusions, and incomplete scans, unlike multiview stereo methods that cannot complete unobserved geometry.Evaluation removes geometry absent from the ground truth before point-cloud metrics are computed.
  • 3D reconstruction: The method achieves state-of-the-art performance on about half of the metrics and remains competitive on all metrics, with stronger qualitative results than previous methods.The authors note that the TSDF L1 metric better reflects the qualitative performance gap than the other metrics.
  • 3D semantic segmentation: RGB-only 3D semantic segmentation uses a single 1×1×1 convolutional head, while evaluation transfers predicted-mesh labels to the ground-truth mesh for standard IOU measurement.Prior methods in the comparison use depth input, making the comparison unfair.
  • Inference time: The approach shows almost no quality degradation with as few as 25 input frames, compared with 784 total frames.Because the large 3D CNN cost is amortized over frames, the method becomes faster than DPSNet after approximately four frames.

6 Conclusions

The paper concludes that direct regression from RGB sequences provides a depth-free route to complete 3D reconstruction, while also supporting semantic prediction with little additional computation.

  • Conclusions: The approach does not require depth inputs and can integrate long frame sequences while completing unobserved geometry.The conclusion describes the method as temporally unbounded and able to integrate long sequences.
  • Conclusions: The authors experimentally verify that direct regression to a 3D model is superior to per-view depth estimation followed by reconstruction.The comparison concerns the classical pipeline of estimating depth for each view and then fusing it.
  • Conclusions: A semantic segmentation objective can be added without significant additional computation to label the reconstructed surfaces.The conclusion also identifies learned back-projection and accumulation as future directions for handling occlusions and large mult room scenes.
Loading 2003.10432v3…