PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image

Xuanchi Ren, Xiaolong Wang

arXiv:2203.09457v1cs.CV

TL;DR

Single-image view synthesis has been limited to relatively small camera motions, leaving long-term scene exploration beyond the input view challenging. The paper predicts video frames autoregressively using multiple preceding frames and camera-guided attention, and reports superior view-synthesis performance, especially on long-term indoor scenes.

  • Problem

    Single-image view-synthesis methods are limited in the camera-motion range they can handle, while long-term views also need to remain consistent.

    Source: §1 · p. 2

  • Method

    An autoregressive Transformer predicts future frames from multiple previous frames and cameras, with camera-aware locality bias guiding self-attention across patches.

    Source: §1 · p. 2

  • Results

    The method outperforms state-of-the-art view-synthesis approaches, with particularly large gains on long-term frames and a focus on indoor 3D scenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach synthesizes consistent long-term indoor scenes, including a described path from a room through its door into a hallway.

    Source: §1 · p. 2, §5 · p. 8

    Autoregressive inference is slightly slower than vanilla models, and PSNR and LPIPS are imperfect metrics for evaluating long-term view synthesis.

    Source: §5 · p. 9

Abstract

from arXiv · show

Novel view synthesis from a single image has recently attracted a lot of attention, and it has been primarily advanced by 3D deep learning and rendering techniques. However, most work is still limited by synthesizing new views within relatively small camera motions. In this paper, we propose a novel approach to synthesize a consistent long-term video given a single scene image and a trajectory of large camera motions. Our approach utilizes an autoregressive Transformer to perform sequential modeling of multiple frames, which reasons the relations between multiple frames and the corresponding cameras to predict the next frame. To facilitate learning and ensure consistency among generated frames, we introduce a locality constraint based on the input cameras to guide self-attention among a large number of patches across space and time. Our method outperforms state-of-the-art view synthesis approaches by a large margin, especially when synthesizing long-term future in indoor 3D scenes. Project page at https://xrenaa.github.io/look-outside-room/.

1. Introduction

Prior single-image view-synthesis methods struggle with large camera motions, while long-term synthesis also requires consistency across the generated views. The paper models multiple frames autoregressively and uses camera-based locality constraints, reporting stronger performance especially on long-range future frames.

Motivation

  • Existing 3D-geometric view-synthesis approaches produce high-quality views but are limited to relatively small camera motions.The introduction gives seeing outside a room's door as an example of the challenge.Source: §1 · p. 2

Task

  • The task is to generate a consistent video from one scene image and a long-term camera trajectory, including views beyond a room's door.The described path passes through the door and into a hallway; the task is also motivated by content generation and editing and robotics planning and control.Source: §1 · p. 2

Approach

  • Unlike prior autoregressive view synthesis that samples diverse views, the proposed model predicts each future frame from multiple prior frames and their cameras.At inference, predicted frames and previous frames can be added progressively to the inputs.Source: §1 · p. 2
  • Camera-Aware Bias uses relative-camera information to guide attention toward relevant patches, addressing difficult space-time dependencies and supporting inter-frame consistency.The bias is computed by an MLP and added to the self-attention affinity matrix.Source: §1 · p. 2

Results

  • The method outperforms state-of-the-art view-synthesis approaches, with a particularly large gain on long-term frames.Experiments include RealEstate10K and Matterport3D, which mainly focus on indoor 3D scenes.Source: §1 · p. 2

2. Related Work

Related work spans 3D-based and autoregressive single-image view synthesis, long-term video synthesis, and image outpainting. The paper positions its approach as targeting consistent long-term indoor scenes with large camera changes.

Novel View Synthesis

  • Single-image 3D-based methods work well for small camera changes but cannot outpaint far from the input view; autoregressive methods handle larger changes but may lack trajectory consistency.A hybrid approach achieves consistent large-change indoor synthesis but does not generate long-term views outside the room's door.Source: §2 · pp. 2–3

Video Synthesis

  • Prior long-term video synthesis from one image and a trajectory focuses on outdoor nature, whereas this work targets indoor 3D scenes requiring more structural reasoning for outpainting..Source: §2 · p. 3

Image Extrapolation and Outpainting

  • Autoregressive image outpainting predicts pixels sequentially, while latent-space modeling such as VQ-VAE reduces sequential complexity and supports Transformer-based high-resolution generation.The paper also uses cameras as tokens, following prior work.Source: §2 · p. 3

Transformers

  • Transformer self-attention is used across generation work, but optimizing it over long visual-token sequences remains challenging, motivating the paper's camera-aware locality bias..Source: §2 · p. 3

3. Method

The method autoregressively predicts a video from one image and a camera trajectory, using camera-aware self-attention to model cross-frame patch relationships. Its architecture combines image and camera tokens, while iterative inference strategies support long sequences.

3.1. Autoregressive Scene Synthesis

  • Given one scene image and a camera trajectory, the autoregressive model predicts future views from previous frames and cameras for unconstrained-length synthesis without 3D information.Sequential modeling raises challenges in learning cross-frame patch relationships and maintaining long-term consistency.Source: §3.1 · p. 3

3.2. Network Architecture

  • A pretrained VQ-GAN converts images into discrete latent tokens that a GPT-style Transformer models autoregressively before decoding predictions into images.The VQ-GAN is pretrained in a first training stage; its smaller latent token grid makes the representation tractable.Source: §3.2 · pp. 3–4
  • The first image is the canonical view, camera transformations are encoded relative to it, and decoupled spatial, camera, and temporal embeddings represent the input sequence.Spatial and camera positional embeddings are shared across images and cameras, respectively.Source: §3.2 · pp. 4–5

3.3. Camera-Aware Bias in Transformer

  • Camera-Aware Bias uses relative camera transformations to guide patch-to-patch attention, adding a 3D-aware inductive bias to Transformer self-attention.An MLP computes the bias from camera transformations; the design also applies to causal self-attention.Source: §3.3 · p. 5

3.4. Training and Inference Details

  • Overlapping iterative modeling reuses information from generated frames for subsequent predictions, while fine-tuning on its own outputs addresses error accumulation and beam search avoids greedy artifacts.The authors report that overlapping iteration supports consistent long-term synthesis even with a small training sequence length.Source: §3.4 · pp. 5–6

4. Experiments

The experiments evaluate short- and long-term view synthesis on indoor-scene datasets, with long-term tests emphasizing image quality and consistency. The method performs significantly better than baselines on both long-term measures.

4.1. Experimental setup

  • Experiments use Matterport3D and RealEstate10K, comparing against SynSin, PixelSynth, GeoGPT, and a larger-view-change SynSin-6x variant.Matterport3D long-term episodes are rendered by an embodied agent; the evaluation includes 6,000 training and 500 test videos.Source: §4.1 · p. 6

4.2. Evaluation on Short-Term View Synthesis

  • On short-term view synthesis, our method outperforms methods with explicit geometry and the geometry-free GeoGPT baseline.The authors attribute GeoGPT's weaker performance to its not ensuring consistency.Source: Table 1 · p. 6, §4.2 · p. 7

4.3. Evaluation on Long-Term View Synthes

  • For long-term view synthesis, our method is significantly better than geometry-based and geometry-free baselines in image quality and consistency.The reported gap is even more prominent on Matterport3D, where view-angle changes are more significant.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Ablation Study

  • Camera-aware bias improves quality and consistency; vanilla positional embeddings degrade both, while error-accumulation finetuning benefits long-term synthesis.These ablations are reported on Matterport3D.Source: Table 4 · p. 7, §4.4 · p. 8

4.3. Evaluation on Long-Term View Synthes

  • Long-term evaluation uses FID for image quality and an A/B user study for consistency on sequences with 20 subsequent ground-truth frames and significant camera motion.For each A/B comparison, users choose which generated video is more consistent.Source: §4.3 · p. 7

4.4. Ablation Study

  • Increasing clip length from 2 to 3 improves consistency; length 5 adds little consistency and significantly lowers image quality, so the authors use length 3.They cite computation and performance when selecting the default length.Source: Table 5 · p. 8, §4.4 · p. 8

5. Discussion

The paper concludes that its camera-locality autoregressive Transformer supports long-term novel-view synthesis in indoor 3D scenes. It also identifies slower inference and imperfect long-term evaluation metrics as remaining challenges.

5. Discussion

  • Camera-based locality in self-attention is used to ensure consistency among generated frames in long-term novel-view synthesis.The conclusion focuses especially on long-term future views in indoor 3D scenes.Source: §5 · p. 8
  • The method achieves superior novel-view-synthesis performance compared with state-of-the-art approaches.The conclusion highlights geometry-free synthesis of consistent, high-fidelity 3D scenes.Source: §5 · p. 8
  • Autoregressive inference is slightly slower than vanilla models, and PSNR and LPIPS are imperfect for evaluating long-term view synthesis.The authors call for further autoregressive-model advances and new metrics for the task.Source: §5 · p. 9

A.1. Our Qualitative Results

The supplementary qualitative results show synthesized views with large camera transformations while maintaining high fidelity. Comparisons with prior methods emphasize greater consistency and clarity.

A.1. Our Qualitative Results

  • The method synthesizes consistent novel views under large camera transformations.Additional long-term 3D scene videos are shown in Figures 7 and 8.Source: §A.1 · p. 12
  • The additional long-term scene videos maintain high fidelity.Source: §A.1 · p. 12

A.2. Qualitative Comparison with Baselines

  • Compared with SynSin [70], SynSin-6x [45], GeoGPT [46], and Appearance Flow [75], the method generates more consistent and clear results.The comparison is shown in Figure 9.Source: §A.2 · p. 12

A.3. Additional Visual Ablation Study

The additional visual ablation study uses Figure 6 to validate the beam-search strategy.

A.3. Additional Visual Ablation Study

B.1. Comparison with Additional Baselines

Additional baseline comparisons report significant image-quality improvements over an approximation of Infinite Nature and worse performance by Video Autoencoder. The appendix also reports average frame-generation inference time.

B.1. Comparison with Additional Baselines

  • On Matterport3D, the method achieves significant image-quality improvements over SynSin-Sequential, an approximation used for comparison with Infinite Nature [30].The paper focuses on indoor scenes, whereas Infinite Nature focuses on nature scenes; its training code was unavailable.Source: §B.1 · p. 12
  • Video Autoencoder [25] performs worse than the method on Matterport3D, despite not requiring camera ground truths during training.The authors note that this is a more challenging setting.Source: §B.1 · p. 12

B.2. Time consumption.

C. More Implementation Details

The implementation uses a GPT-2-style autoregressive Transformer and VQ-GAN, while the baseline discussion distinguishes methods by their limits on large camera changes and long-term consistency. Figures compare long-term synthesis on RealEstate10K and Matterport3D with baseline results.

C. More Implementation Details

  • The Transformer follows GPT-2, with hidden dimension 1024, 16 attention heads, and a two-layer MLP of hidden size 4096; training uses teacher forcing and autoregressive masks.The masks enable parallel computing during training.Source: C. More Implementation Details · p. 12
  • The VQ-GAN uses a downsampling factor of 16, encoding a 256 × 256 image as 16 × 16 tokens.Source: C. More Implementation Details · p. 12

D. Details of Baselines

  • SynSin and SynSin-6x are limited to inpainting, PixelSynth does not support long-term view synthesis, and GeoGPT does not ensure consistency or use locality constraints.Appearance Flow also struggles with large camera changes because warping leaves large missing areas.Source: D. Details of Baselines · p. 12
  • Previous methods cannot synthesize a consistent long-term scene video, while the proposed method produces perceptually consistent, high-fidelity long-term views.The figures show long-term view synthesis on RealEstate10K and Matterport3D and a comparison with baselines.Source: Figure 7 · p. 13, Figure 8 · p. 13, Figure 9 · p. 14
Loading 2203.09457v1…