PaperlayerView exact revision on arXiv

Source-linked AI summary

Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, Angjoo Kanazawa

arXiv:2012.09855v4cs.CVcs.GR

TL;DR

The paper addresses perpetual view generation: producing arbitrarily long camera-trajectory videos from a single image despite the limits of existing view- and video-synthesis methods. It combines disparity-based rendering with image-and-geometry refinement in a repeatable loop, and reports plausible hundreds-frame sequences over large trajectories. The method is designed for local consistency, but does not guarantee global consistency and does not model dynamic scenes.

  • Problem

    Perpetual view generation requires synthesizing unseen regions and new detail along arbitrary camera trajectories, beyond the limited horizons or geometric modeling of existing methods.

  • Method

    A render-refine-repeat framework uses disparity for geometric rendering, then refines image content and disparity before iterating autoregressively.

  • Results

    The system generates plausible sequences spanning hundreds of frames and longer horizons than recent view-synthesis and video-generation baselines.

  • Takeaways & Limitations

    The results indicate that combining geometric and generative techniques is a promising step toward perpetual view generation.

  • Takeaways & Limitations

    The memory-less loop targets local consistency and does not directly enforce long-term global consistency; the system also does not model dynamic scenes.

Abstract

from arXiv · show

We introduce the problem of perpetual view generation - long-range generation of novel views corresponding to an arbitrarily long camera trajectory given a single image. This is a challenging problem that goes far beyond the capabilities of current view synthesis methods, which quickly degenerate when presented with large camera motions. Methods for video generation also have limited ability to produce long sequences and are often agnostic to scene geometry. We take a hybrid approach that integrates both geometry and image synthesis in an iterative `\emph{render}, \emph{refine} and \emph{repeat}' framework, allowing for long-range generation that cover large distances after hundreds of frames. Our approach can be trained from a set of monocular video sequences. We propose a dataset of aerial footage of coastal scenes, and compare our method with recent view synthesis and conditional video generation baselines, showing that it can generate plausible scenes for much longer time horizons over large camera trajectories compared to existing methods. Project page at https://infinite-nature.github.io/.

1. Introduction

The paper introduces perpetual view generation: synthesizing a video along an arbitrary camera trajectory from one image. It combines geometry and image synthesis in a render-refine-repeat framework to produce plausible views over much longer horizons than prior methods.

  • Perpetual view generation synthesizes a video for an arbitrary camera trajectory from a single scene image.The task is motivated as a tractable alternative to generating an entire high-resolution 3D planet from one image.
  • The task must extrapolate unseen regions while adding detail to existing regions that become closer as the viewpoint moves.
  • Video synthesis methods generate limited numbers of novel frames and often omit scene geometry, while view synthesis methods fail beyond limited camera-motion ranges.The cited examples include 25 or 48 novel frames for video synthesis.
  • The proposed hybrid method renders views using disparity, refines images and geometry, and repeats the process for arbitrary trajectories.
  • A dataset of over 700 nature and coastal videos spanning 2 million frames supports training sequences of hundreds of frames.The trained model maintains the aesthetic feel of aerial coastal video even after the camera moves beyond the initial scene’s depicted limits.
  • Experiments report plausible frames for much longer time horizons than recent view-synthesis and video-generation baselines, while global consistency remains limited.

2. Related Work

Prior image-extrapolation, view-synthesis, and video-synthesis methods address parts of the problem but generally lack the combination of geometry, long-range camera motion, and long sequences targeted here.

  • Image extrapolation: Image-extrapolation methods extend images through stitching, patches, semantic inpainting, or adversarial synthesis, whereas this work also reasons about 3D geometry and temporal sequences.
  • View synthesis: Single-image and multi-view view-synthesis methods typically support only limited output viewpoints, including methods explicitly designed for view extrapolation.
  • View synthesis: Static scene representations such as depth images, multiplane images, point clouds, and radiance fields restrict viable camera positions through finite scene bounds.
  • View synthesis: Video-trained view-synthesis methods often use image pairs, while this model trains on several widely spaced frames to generate long-range video.
  • Video synthesis: Video-synthesis work commonly predicts dynamic objects with static cameras, whereas this paper targets mostly static natural scenes filmed by moving aerial cameras.
  • This work: The proposed overview combines disparity-based rendering with image refinement and outputs RGB plus geometry for repeated generation.

3. Perpetual View Generation

Perpetual view generation iteratively renders a new viewpoint from image and disparity, refines missing content and geometry, and repeats the process to produce long trajectories.

  • Problem formulation: Given one RGB image and an arbitrarily long camera trajectory, the system outputs an image sequence depicting a flythrough of the initial scene.
  • Approach: Render, Refine, Repeat: The framework combines 3D rendering, image-to-image translation, and autoregressive training in three render-refine-repeat steps.
  • Rendering: Differentiable rendering warps image content according to disparity, producing a new view, disparity, and a mask for missing regions.
  • Refinement: The refinement network inpaints, outpaints, and super-resolves the rendered image while producing updated RGB and disparity.
  • Repeat: Propagating refined disparity between frames enables recurrent generation and supports training with losses backpropagated over multiple steps.
  • Geometric Grounding: Geometric re-grounding rescales refined disparity to a consistent range and significantly reduces test-time drift.

4. Aerial Coastline Imagery Dataset

The Aerial Coastline Imagery Dataset uses long moving-camera drone footage of natural coastal scenes, processed with structure from motion and supplemented with disparity estimates.

  • Dataset motivation: Existing view-synthesis datasets lack long sequences or substantial camera movement, motivating a dataset of aerial nature footage with mostly static scenes.
  • Construction: Structure from motion processes coastline drone footage collected from YouTube to create the Aerial Coastline Imagery Dataset.
  • Dataset scale: The released dataset contains over 13,000 sequences totaling 2.1 million frames, along with video lists and SfM camera trajectories.
  • Disparity processing: MiDaS single-view depth predictions provide disparity maps for every frame, rescaled using sparse SfM point clouds.
  • Training preparation: Frames are strategically subsampled to maintain consistent camera speed across training sequences despite differing video motions.

5. Experimental Setup

The experiments use recurrent render-refine-repeat training, adversarial and perceptual image objectives, and FID-based evaluation of generated-image quality over time.

  • Training data: The training data comprise image sequences, with RGB and disparity outputs jointly supervised during autoregressive generation.The supplied passages specify sequence-based training and joint RGB-disparity losses but do not provide dataset size here.
  • Training objective: Training applies L1 reconstruction losses to RGB and disparity, alongside VGG perceptual, hinge adversarial, feature-matching, and KL-divergence losses.The complete objective sums reconstruction, perceptual, adversarial, feature-matching, and KL-divergence terms.
  • Training objective: The complete loss is computed across all render-refine-repeat iterations and all samples in each mini-batch.
  • Metrics: FID measures realism and diversity from Inception embeddings, while sliding-window FID tracks generated-image quality over time.Short-range evaluation also uses MSE and LPIPS when ground-truth frames are available.
  • Implementation: Models are trained for T = 5 render-refine-repeat steps at 160 × 256 resolution, using batch size 4 across 10 GPUs for 7M iterations.The five-step choice is limited by memory and available training-sequence lengths.

6. Evaluation

Evaluation compares the method with view-synthesis and video-generation baselines on short, medium, and long trajectories. The proposed framework maintains more plausible quality over extended camera motion, while ablations show that recurrent training, refinement, and geometric grounding matter.

  • Evaluation protocol: The study compares against 3D Photos, SynSin, single-view MPIs, and SVG-LP, plus iterated and autoregressively trained SynSin and MPI variants.The iterated variants generate each frame from the previous frame; repeat variants train across T = 5 steps.
  • Evaluation protocol: The short-to-medium benchmark contains 279 trajectories of 50 steps, with ground truth for the first 10 frames and extrapolated camera motion thereafter.Short-range metrics use the first 10 steps; medium-range FID is computed over all 50 generated frames.
  • Quantitative results: Our approach has a dramatically lower FID-50 than other methods over 50 generated frames, while baseline FID-20 deteriorates more rapidly over time.The method is also competitive with recent view-synthesis approaches on short-range LPIPS and MSE.
  • Qualitative results: Qualitatively, competing methods develop blur, stretched textures, visible MPI layers, holes, or missing detail, whereas the proposed method generates plausible views.SynSin produces plausible texture but still develops holes and does not add new detail.
  • Baseline analysis: Iterated and repeat SynSin and MPI variants perform consistently worse than their originals, indicating that repetition or autoregressive training alone is insufficient for large camera movement.The variants show more drifting artifacts, while repeated MPI application becomes blurry because it cannot add detail or propagate geometry.
  • Ablations: Removing recurrent training produces similar short-range LPIPS and MSE but lower FID that worsens substantially with increasing time.The ablation trains with T = 1 instead of the full multi-iteration setup.
  • Ablations: Removing refinement leaves increasing missing regions and blur, while disabling geometric grounding slightly lowers short-to-medium-range quality.The refinement network fills uncovered regions and super-resolves non-masked content.
  • Long-range results: Over 500 autopilot-generated frames, our FID-50 remains lower than baseline FID-50 measured over their first 50 steps.This compares a sequence an order of magnitude longer for the proposed method than for the baselines.

7. Discussion

The paper presents perpetual view generation as a new problem and combines geometric and generative techniques to generate hundreds of frames. It reports single-image operation with original disparity and identifies several limitations and implementation considerations.

  • Discussion: Perpetual view generation combines geometric and generative techniques to produce video sequences spanning hundreds of frames.The framework is presented as a first step toward generating arbitrarily long novel-view sequences.
  • Limitations: The memory-less render-refine-repeat loop supports arbitrarily long output with finite memory and compute, but targets local rather than global consistency.The authors identify long-term memory and global representation as open questions.
  • Disparity: Scaling and shifting MiDaS disparity aligns training geometry with SfM trajectories, while original MiDaS disparity can be used for sequence generation from one image.Scaled disparity is defined as D = a ˆD + b; scaling requires sparse SfM points during training but not ordinary sequence generation at test time.
  • Disparity: Original MiDaS outputs have a negligible effect on FID scores, and a smartphone photo example demonstrates long-sequence generation from unscaled disparity.The example assumes a single test image and uses MiDaS for its initial disparity.
  • Training: Training motion is normalized by subsampling sequences according to estimated camera speed.The sampling range is chosen from translation magnitudes between scale-normalized camera poses.

A.4. Network Architecture

The system uses a variational encoder and SPADE generator to refine RGB, disparity, and missing-region masks, with an auto-pilot for long trajectories. Additional procedures support smooth interpolation and user-controlled motion.

  • Network Architecture: The refinement generator encodes an initial RGBD frame into a Gaussian feature distribution and repeatedly samples or uses its mean to produce GAN noise.The SPADE generator receives RGB texture, disparity, and a missing-region mask.
  • Network Architecture: A Pix2Pix-based U-net struggled to converge and often failed beyond the initial five steps.This comparison motivated the SPADE-based refinement architecture.
  • Auto-pilot View Control: The auto-pilot jointly generates camera trajectories and images while using disparity thresholds to avoid ground and obstacles.It categorizes low-disparity regions as sky and high-disparity regions as near, then derives target viewing and movement directions.
  • Auto-pilot View Control: A time-varying offset to the target look direction can create meandering trajectories and provide user control.A horizontal sinusoidal variation is one example of such an offset.
  • Additional Frame Interpolation: Frame interpolation creates intermediate poses, rerenders both neighboring frames, and blends the results with weight λ.The supplementary video adds four frames between each pair; paper comparisons and metrics use raw outputs.

A.9. Aerial Coastline Imagery Dataset

ACID is an aerial nature-scene dataset organized with globally consistent structure-from-motion poses and intrinsics. Evaluation uses motion-matched forward trajectories, with distinct procedures for 50-frame and 500-generation FID tests.

  • Aerial Coastline Imagery Dataset: ACID provides train and test splits of aerial nature footage with multiple frames, structure-from-motion poses, and intrinsics in a globally consistent system.The dataset is released in a format matching RealEstate10K.
  • Evaluation: Test sequences use motion-based subsampling to match the training distribution across view synthesis methods.This procedure is applied consistently when preparing evaluation sequences.
  • Evaluation: Evaluation restricts trajectories to forward motion within a 90° frontal cone, uses extrapolated poses for 50-frame FID, and uses auto-pilot poses for 500-generation FID.Real inception statistics are computed from ACID images.

B.1. SynSin training

The SynSin baselines were adapted for longer camera strides and tested with iterative and repeat variants. Autoregressive SynSin performed worse than direct long-range synthesis, while the repeat comparison has implementation constraints.

  • SynSin training: SynSin training increased the upper camera-motion range by 5×, improving performance over default settings; larger increases showed no improvement.The modified model was used for all SynSin evaluations.
  • Qualitative Comparisons: Additional qualitative comparisons show stretched pixels, incomplete outpainting, and incomplete camera motion across several baselines, while further iterate-and-repeat variants do not improve results.The listed artifacts occur for specific methods including 3D Photos, MPI, SynSin, Ours no-repeat, and SVG-LP.
  • SynSin training: SynSin-iter applies the trained SynSin model autoregressively at test time, but performed worse than direct long-range synthesis.This result contrasts iterative reuse with direct long-range prediction.
  • SynSin training: SynSin-repeat trains the model to consume its own output for T = 5 steps, but reduced batch size was required because the original configuration exceeded memory limits.The authors frame this as evidence about approach suitability rather than proof that SynSin cannot perform perpetual view generation.

C.1. Limitations

The method’s long-range generation depends on geometric grounding, but its memory-less iterative representation does not guarantee global consistency in geometry or appearance.

  • A memory-less Markov process does not guarantee global consistency across multiple iterations.Past geometric structures may not reappear, and scene appearance can change over several iterations.
  • Refined disparity maps provide a visual representation of generated geometry alongside RGB texture.The discriminator is trained on concatenated RGB and disparity, making the disparity maps visually plausible as well.
  • Without geometric grounding, disparity drifts toward very small values before frame 150, followed by deterioration in RGB outputs.The ablation uses the same model checkpoint with geometric grounding disabled only at test time.

C.4. Results under Various Camera Motions

The method remains resilient to several turning camera motions, while geometric drift and memory-less generation create consistency failures on long or reversing trajectories.

  • Results under Various Camera Motions: Without geometric grounding, disparity drifts substantially by frame 150 and RGB output begins drifting after frame 175.The sequence initially appears plausible before the geometric and appearance failures emerge.
  • Results under Various Camera Motions: FID-50 scores remain comparable on long generations with no offset, gentle turns, or stronger turns.The auto-pilot varies the relative horizontal offset magnitude across 0.0, 0.5, and 1.0, producing alternating left-right turns.
  • Results under Various Camera Motions: Longer palindromic trajectories produce greater mismatch from the original image when the camera returns to its initial pose.The memory-less representation must regenerate content that left the field of view or became occluded during forward motion.
  • Results under Various Camera Motions: Strongly turning camera motion causes only a slight quality reduction relative to the full model evaluation.The camera-motion variants add different amounts of turning to the auto-pilot trajectory.
  • Results under Various Camera Motions: Forward-backward generation exposes the absence of persistent content memory when previously seen regions must be synthesized again.The resulting image need not match the original input after returning to the initial pose.
Loading 2012.09855v4…