PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple and Effective Synthesis of Indoor 3D Scenes

Jing Yu Koh, Harsh Agrawal, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lee, Yinfei Yang, Jason Baldridge, Peter Anderson

arXiv:2204.02960v2cs.CVcs.AIcs.LG

TL;DR

Synthesizing high-resolution, 3D-consistent indoor views far beyond limited image context remains challenging. The paper maps reprojected point-cloud guidance directly to RGB-D with an image-to-image GAN, outperforming prior work in image evaluations and FID. Its synthesized views also augment VLN trajectories, with combined Habitat augmentation improving R2R success rate by 1.5%.

  • Problem

    The task is to synthesize high-resolution indoor images and videos from novel viewpoints, including far-extrapolated views, while maintaining 3D consistency from limited observations.

    Source: Introduction · p. 1, Approach · p. 3

  • Method

    An image-to-image GAN converts reprojections of accumulated RGB-D point clouds into high-resolution RGB-D views, accumulating predictions for multi-step 3D consistency.

    Source: Introduction · p. 1, Approach · p. 3

  • Results

    The model outperforms prior work in human evaluations and FID, while combined Habitat and model-based trajectory augmentation improves R2R success rate by 1.5%.

    Source: Introduction · p. 2, Conclusion · p. 8, Results · p. 8

  • Takeaways & Limitations

    The model’s visual augmentation is training-based and can be applied to train virtually any off-the-shelf VLN agent.

    Source: Results · p. 8

Abstract

from arXiv · show

We study the problem of synthesizing immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints, including viewpoints that extrapolate far beyond the input images while maintaining 3D consistency. Existing approaches are highly complex, with many separately trained stages and components. We propose a simple alternative: an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images. On the Matterport3D and RealEstate10K datasets, our approach significantly outperforms prior work when evaluated by humans, as well as on FID scores. Further, we show that our model is useful for generative data augmentation. A vision-and-language navigation (VLN) agent trained with trajectories spatially-perturbed by our model improves success rate by up to 1.5% over a state of the art baseline on the R2R benchmark. Our code will be made available to facilitate generative data augmentation and applications to downstream robotics and embodied AI tasks.

Introduction

The paper replaces complex novel-view synthesis pipelines with an image-to-image GAN that turns reprojections of accumulated point clouds into high-resolution RGB-D images. It reports better image-generation results than prior work and uses synthesized views to augment VLN training trajectories.

Introduction

  • Interactive, immersive imagery could support content creation, robotics, and embodied-AI applications such as navigation agents that predict around corners.The paper describes using such models as world models for planning in novel environments and synthesizing interactive training environments from static images or video.Source: Introduction · p. 1
  • An image-to-image GAN maps reprojected point-cloud guidance directly to high-resolution photorealistic RGB-D, avoiding several components used by prior systems.The approach drops Pathdreamer’s stochastic structure generator, spatially adaptive normalization, semantic-segmentation dependence, and multi-step training, as well as several PixelSynth components.Source: Introduction · p. 1, Figure 1 · p. 2
  • Human raters preferred the model to Pathdreamer in 60% of comparisons and to PixelSynth in 77%.On Matterport3D, FID improved over Pathdreamer from 27.2 to 19.6 for single-step predictions and from 65.8 to 58.0 over six-view trajectories; on RealEstate10K it improved over PixelSynth from 25.5 to 23.5.Source: Introduction · p. 2
  • Synthesized views spatially perturb VLN training panoramas by up to 1.5 m, improving unseen-environment success by 1% alone or 1.5% when combined with Habitat renders.The combined augmentation achieves state-of-the-art performance on the R2R test set.Source: Introduction · p. 2

Related Work

Related work spans multi-view novel-view synthesis, point-cloud rendering, and generative data augmentation. The paper focuses on large viewpoint changes and unseen indoor scenes, where sparse context requires filling missing regions while preserving scene consistency.

Related Work

  • NeRF methods learn continuous volumetric scene functions from multiple images, but the paper says they primarily focus on scene representation rather than generalizing high-resolution synthesis to unseen scenes.The passage also surveys extensions to large-scale environments, unconstrained photo collections, scene priors, multi-scale representations, and reduced input requirements.Source: Related Work · p. 2
  • Recent work addresses single-image synthesis and large viewpoint changes, but this task must inpaint missing pixels and outpaint large regions from limited context while maintaining scene consistency.The closest works identified are PixelSynth (Rockwell, Fouhey, and Johnson 2021) and Pathdreamer (Koh et al. 2021b).Source: Related Work · pp. 2–3
  • Prior point-cloud rendering methods often use smaller point clouds, which the paper says limits efficiency and viability for high-resolution imagery.Song et al. (2020) uses point clouds of size 4096 × 6, while this method accumulates RGB-D images of size 1024 × 512, at least 64× larger.Source: Related Work · p. 3
  • Generative augmentation adds synthetic samples matching a dataset’s distribution and characteristics; the paper investigates using novel views to create spatially perturbed navigation trajectories.The stated aim is post hoc perturbation of camera viewpoints.Source: Related Work · p. 3
  • In imitation learning, demonstrations can be augmented with recovery examples to reduce mismatch between training states and those encountered at inference.The paper frames novel-view synthesis as a flexible, hardware-free alternative to multi-camera recording and investigates it for photorealistic VLN on R2R.Source: Related Work · p. 3

Approach

The approach reprojects accumulated RGB-D context into a target-view guidance image, then uses a single-stage model to generate RGB-D output. Partial convolutions and a larger receptive field address sparse guidance, while the final model omits a stochastic structure generator and VGG perceptual loss.

Approach

  • Accumulated point clouds are reprojected at each target pose to form guidance images, and predictions are added back for multi-step 3D consistency.The same architecture supports equirectangular Matterport3D panoramas and perspective RealEstate10K images through the corresponding camera models; experiments use ground-truth or estimated depth and poses depending on the dataset.Source: Approach · p. 3
  • The single-stage, end-to-end encoder-decoder maps guidance images to photorealistic RGB-D, using a shared encoder for RGB and depth inputs and separate decoders for the two outputs.The architecture uses a ResNet-101 encoder, mirrored transposed-convolution decoders, and skip connections to preserve spatial information.Source: Approach · p. 3
  • Partial convolutions restrict convolution to valid input regions, and four added encoder-output layers expand the receptive field to propagate non-local information.The authors report that the added layers are essential; without them, the model does not learn meaningful outputs.Source: Approach · p. 3
  • The stochastic SVG structure generator reduced generation quality despite improving image diversity, so the final model omits it for simplicity and performance.The authors note it can be incorporated to trade generation quality for diversity.Source: Approach · pp. 3–4
  • Training combines an L1 loss on predicted depth with adversarial loss on RGB-D, whose discriminator helps enforce consistency between the separate RGB and depth predictions.The authors report that VGG perceptual loss is unnecessary for strong performance, based on ablation experiments.Source: Approach · p. 4
  • The model is trained end-to-end from scratch with random masking of up to 75% of guidance images, which the authors find significantly improves generation quality in unseen environments.Source: Approach · p. 4

View Synthesis Experiments

The experiments evaluate novel-view synthesis on Matterport3D and RealEstate10K using FID scores and human judgments. They compare the model with prior methods and test design choices through ablations.

View Synthesis Experiments

  • The study evaluates Matterport3D and RealEstate10K against prior methods using FID and pairwise human evaluations.Matterport3D comparisons follow Pathdreamer’s higher-resolution, multi-step evaluation procedure because the concurrent methods’ evaluation procedures are not comparable.Source: View Synthesis Experiments · p. 4, Matterport3D Experiments · p. 4

Matterport3D Experiments

  • On Matterport3D Val-Unseen, FID improves over Pathdreamer across settings, including 27.6 to 19.6 for one-step prediction with one context pano; our images are preferred in 60.4% of comparisons.For unanimous human preferences, our model is selected in 30.7% of cases versus 12.3% for Pathdreamer.Source: Table 1 · p. 4, Matterport3D Experiments · p. 5, Figure 3 · p. 6

Ablation Studies

  • Removing L1 depth reconstruction worsens one-step and multi-step FID, while adding VGG perceptual loss improves FID@1 but worsens FID@{1-6}.With KLD loss and noise conditioning, FID@1 changes from 19.6 to 22.6 and FID@{1-6} from 58.0 to 65.2.Source: Ablation Studies · pp. 5–6
  • Four added convolutional layers plus random guidance-pixel masking improve FID@1 from 24.2 to 19.6 and FID@{1-6} from 68.0 to 58.0; using ground-truth depth for GAN loss worsens both metrics.Ground-truth depth for GAN loss changes FID@1 from 19.6 to 24.3 and FID@{1-6} from 58.0 to 66.1; a shared decoder worsens FID@1 to 24.1 but improves FID@{1-6} to 56.9.Source: Ablation Studies · p. 6

RealEstate10K Experiments

  • On RealEstate10K, FID is 23.5 versus PixelSynth’s 25.5 and SynSin’s 34.7; human evaluators prefer our images over PixelSynth in 77.3% of cases.On the indoor-only subset, FID is 21.5 versus PixelSynth’s 23.6.Source: RealEstate10K Experiments · p. 6, Figure 5 · p. 6, Table 3

Trajectory Augmentation in VLN

The paper tests whether spatially perturbed training panoramas improve VLN generalization, using a strengthened VLN agent and comparisons with image augmentation and simulator rendering. Model-based trajectory augmentation improves Val-Unseen performance, and combining it with Habitat yields the largest reported Success Rate gain.

Trajectory Augmentation in VLN

  • The study hypothesizes that spatially perturbing captured panorama locations reduces overfitting to incidental trajectory details and improves generalization to unseen environments.The R2R training set contains 14K instruction-trajectory pairs defined by sequences of adjacent panoramas in a navigation graph.Source: Trajectory Augmentation in VLN · p. 7

VLN Agent and Metrics

  • The VLN baseline uses VLN⟳BERT, upgraded from ResNet-152 to MURAL-large, which raises Val-Unseen SR from 62.2% to 67.3%.SR is the proportion of trajectories ending within 3m of the target; NE measures final distance to the target, and SPL weights success by normalized inverse path length.Source: VLN Agent and Metrics · p. 7

Trajectory Augmentation

  • Training augmentation resamples panorama locations, perturbing ground-plane position by up to 1.5m and height by up to 0.1m, while rejecting moves beyond observed depth.The 1024×512 panorama is generated from the nearest two ground-truth RGB-D panoramas; alternatives include SimCLR image operations and Habitat mesh renders.Source: Trajectory Augmentation · pp. 7–8

Results

  • The model-plus-Habitat augmentation raises Val-Unseen SR by 1.5% and the combined model outperforms published prior work on the unseen test set.Model-based trajectory augmentation alone improves Val-Unseen SR and SPL by 1%, reducing the Val-Seen/Val-Unseen gap from 8% to 4%.Source: Results · p. 8, Table 4 · p. 8

Conclusion

The conclusion presents a simpler approach to immersive 3D scene synthesis and reports strong generation results alongside an application to navigation.

Conclusion

  • The paper proposes an image-to-image GAN trained with random input masking and other architectural changes for synthesizing immersive 3D environments from limited images.The approach is presented as an alternative to methods with many separately trained stages and components.Source: Conclusion · p. 8
  • The approach outperforms prior work in human evaluations and FID, though this passage reports no specific values.Source: Conclusion · p. 8
  • Spatially perturbing training images with the model achieves state-of-the-art R2R results and improves generalization to unseen environments.Source: Conclusion · p. 8

Implementation Details

The implementation uses a shared GAN training setup, with FID evaluation procedures specified separately for Matterport3D and RealEstate10K.

Implementation Details

  • Models are trained for 300K steps with batch size 128, using Adam, spectral normalization, and exponential moving average of generator weights for evaluation.The generator and discriminator learning rates are 1e−4 and 4e−4, respectively; the discriminator takes two steps per generator step, and early stopping uses validation FID.Source: Implementation Details · p. 12
  • Matterport3D FID uses 10,000 random samples per prediction-sequence step, with random horizontal rolls and flips for evaluation.Source: Implementation Details · p. 12
  • RealEstate10K FID is evaluated on 3,122 indoor images selected from 3,600 examples, using the same subset for the model and PixelSynth.Indoor selection uses image-classifier tags, with uncertain predictions manually inspected.Source: Implementation Details · p. 12

Architectural Details

The generator and discriminator architectures are detailed in separate tables, with the same model architecture used for both datasets apart from input dimensions.

Architectural Details

COLMAP sampling details

RealEstate10K sequences were extended and filtered to form an indoor training set with reconstructed camera trajectories and refined, denser depth maps.

COLMAP sampling details

  • Only 6,865 RealEstate10K videos remained available; sequences up to 1,000 frames yielded 108,275 subsequences.The remaining videos had been tracked as connected frame sequences by ORB-SLAM.Source: COLMAP sampling details · p. 12
  • Filtering out subsequences without indoor-category frames left 83,559 subsequences and over 15 million frames; COLMAP built dense reconstructions for 67,834 subsequences.Training frames were retained only when MVS depth covered at least 10% of the image.Source: COLMAP sampling details · p. 12
  • MVS depth was refined by filtering outliers and checking consistency against motion-parallax depth between adjacent frames.The passage notes that MVS estimates can be noisy due to blur, reflections, poor lighting, and other factors.Source: COLMAP sampling details · p. 12
  • Dense MiDaS predictions supplemented sparse MVS depth, with their scale fitted to valid COLMAP MVS pixels by least squares.MiDaS depths could not directly form a unified point cloud because their frame-to-frame scales were inconsistent and unaligned with camera pose.Source: COLMAP sampling details · p. 12

Human Evaluations

With guidance images shown, human raters preferred our model to PixelSynth on RealEstate10K. Additional examples and videos illustrate plausible novel views, including sequences that preserve scene geometry over extended viewpoints.

User Interface and Process

  • The human-evaluation interface anonymized the models, randomized their order, and collected five independent ratings for each of 1,000 image pairs on both datasets.Matterport3D comparisons were against Pathdreamer; RealEstate10K comparisons were against PixelSynth.Source: User Interface and Process · p. 12, Figure 7 · p. 14

Evaluation with Guidance Image

  • With guidance images shown, raters preferred our model in 88.0% of comparisons and unanimously in 48.4%, versus 12% and 2.0% for PixelSynth.The unanimous result means all five raters preferred the same model; Figure 8 presents the guidance-input evaluation and comparison.Source: Evaluation with Guidance Image · p. 13, Figure 8 · p. 14

Qualitative Evaluation

  • Qualitative examples include cherry-picked and random sequences for both datasets, plus side-by-side comparisons with Pathdreamer.Figures 9–13 show these examples, including Matterport3D Val-Unseen R2R sequences and RealEstate10K evaluation-split sequences.Source: Qualitative Evaluation · p. 13, Figure 9 · p. 15, Figure 10 · p. 16

Video Generation Results

  • Generated videos show plausible novel views at extended distances, with examples preserving doorway or hallway geometry and maintaining consistency across viewpoints.Examples describe preserved geometry at 6.3m, plausible rooms at 5.6m, plausible traversal at 6.8m and beyond, and a hallway result at 4.9m that assumes a wall rather than matching ground truth.Source: Video Generation Results · pp. 13, 15, 16
Loading 2204.02960v2…