PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressively Optimized Local Radiance Fields for Robust View Synthesis

Andreas Meuleman, Yu-Lun Liu, Chen Gao, Jia-Bin Huang, Changil Kim, Min H. Kim, Johannes Kopf

arXiv:2303.13791v1cs.CV

TL;DR

Reconstructing large scenes from casual handheld video requires estimating long camera trajectories while representing unbounded environments, despite the fragility of pre-estimated poses. The paper progressively jointly optimizes poses and radiance fields while adding local fields, and reports improved robustness and fidelity across two datasets. Its method assumes continuous video and does not handle dynamic elements or global loop closure.

  • Problem

    Large-scene reconstruction from handheld video requires accurate long-trajectory camera poses and scalable radiance fields, while SfM frequently fails on in-the-wild video.

    Source: §1 · p. 2

  • Method

    The method progressively introduces frames to jointly optimize camera poses and radiance fields, while dynamically allocating local fields supervised within a temporal window.

    Source: §3 · p. 3, §3.2 · p. 4

  • Results

    Across two datasets, the method substantially improves the robustness and fidelity of radiance-field reconstruction in this challenging scenario.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors report that local fields support large-scale representation and sharpness, with pose-misestimation effects locally bounded and arbitrarily long videos processed without loss of accuracy or memory limits.

    Source: §1 · p. 2

    The method assumes continuous video without shot changes, does not tackle dynamic elements, and omits global bundle adjustment and loop closure.

    Source: §1 · p. 2, §4.6 · p. 8

Abstract

from arXiv · show

We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art.

1. Introduction

Radiance fields offer photorealistic view synthesis, but large scenes captured with a handheld camera require both long-trajectory pose estimation and scalable scene reconstruction. The paper progressively estimates poses and radiance fields and allocates overlapping local fields, targeting robustness and large-scale representation.

1. Introduction

  • Single-handheld-camera reconstruction must handle long camera trajectories and large scenes, while SfM often fails on videos with view-dependent appearance, weak texture, or slight motion.The paper notes that jointly resolving camera motion and view-dependent appearance is difficult.Source: §1 · p. 2
  • Unlike NDC’s limited forward-facing frustum or Mip-NeRF360’s inward-facing contraction, the approach places local radiance fields along the camera trajectory to adapt to camera paths.Each field maps the entire space to a [−2, 2] cube and has its own contraction center.Source: Figure 2 · p. 2
  • Evaluation uses Tanks and Temples and Static Hikes, a collected dataset of twelve outdoor scenes captured with four consumer cameras.The sequences include long handheld trajectories, motion blur, and complex appearance.Source: §1 · p. 2
  • Progressive joint pose and radiance-field estimation, together with dynamically allocated local fields, is presented as a way to improve robustness and scale to long videos.The authors also claim locally bounded impact from pose misestimation and increased sharpness from modeling scene details with multiple fields.Source: §1 · p. 2
  • The method targets novel-view synthesis but does not perform global bundle adjustment or loop closure, so it is not a complete SLAM system.Source: §1 · p. 2

2. Related Work

Related work spans neural implicit view synthesis, scalable scene representations, and camera-pose estimation. The paper distinguishes its approach by targeting casually captured monocular first-person videos with long trajectories.

2. Related Work

  • Neural implicit methods produce high-quality view synthesis but can take days to train; alternatives include voxel-like, tensor-factorized, light-field, and hashed voxel/MLP representations.The authors also leverage TensoRF.Source: §2 · p. 3
  • Prior scalable methods impose input or scene constraints, while NeRFusion does not address camera-pose estimation or unbounded outdoor scenes.Examples include omnidirectional inputs, proxy geometry, specialized drone or satellite shots, or multiview observations.Source: §2 · p. 3
  • In contrast, the paper’s method is described as robust to arbitrary-length camera trajectories using casually captured monocular first-person videos.Source: §2 · p. 3
  • Visual-odometry methods use photometric consistency, hand-crafted features, or learned trajectories, while joint NeRF pose methods often struggle on large scenes and long monocular first-person videos.Vox-Fusion and Nice-SLAM achieve good pose estimation but require RGB-D inputs and accurate depth.Source: §2 · p. 3

3. Method

The method jointly estimates camera poses and radiance fields from a long monocular video, progressively adding frames and dynamically allocating local fields. It combines ray-based radiance-field rendering with depth and optical-flow supervision to support reconstruction across unbounded scenes.

3.1. Formulation and Preliminaries

  • TensoRF maps 3D positions to density and view-dependent color, while an L∞ contraction maps unbounded scene points into a bounded space for querying.The method samples points along camera rays, queries radiance fields for color and density, and optimizes them with camera poses using input colors as supervision.Source: §3 · p. 3, §3.1 · pp. 3–4

3. Method

  • The method incrementally updates camera poses and radiance fields using a sweeping temporal window, adding new local fields to scale to long videos with fixed memory.New frames are added to a well-converged solution, and local fields are supervised by frames within a temporal window.Source: §3 · p. 3

3.2. Progressive Joint Camera Pose and Radiance Field Optimization

  • Progressive optimization starts with five frames and adds subsequent frames, using the current endpoint pose to initialize each new pose and provide a locality prior.The new frame benefits from the current pose and radiance-field initialization, making it less prone to local minima.Source: §3.2 · pp. 4–5

3.3. Local Radiance Fields

  • A new local radiance field is created when the estimated trajectory leaves the current field’s uncontracted space, limiting pose-misestimation impact and keeping high-resolution representation near the trajectory.Each field uses a local subset of frames, including the preceding 30 frames for overlap; overlapping fields blend rendered colors, and previous fields are frozen when a new one is created.Source: §3.3 · p. 5

3.4. Implementation

  • Monocular depth and neighboring-frame optical flow provide additional optimization losses, while the implementation progressively registers frames and refines fields before allocating new ones.The schedule starts from five identity-initialized poses and allocates a new field when the last registered pose leaves the current field’s uncontracted space.Source: §3.4 · pp. 5–6

4. Experimental Results

Experiments on Tanks and Temples and Static Hikes evaluate reconstruction quality, pose estimation, and scalability across large-scale scenes. The results show stronger Static Hikes performance than BARF, competitive Tanks and Temples results, and contributions from both progressive optimization and local radiance fields.

4.1. Datasets

  • Tanks and Temples evaluation uses nine static scenes; Static Hikes adds twelve 1920×1080 handheld outdoor videos with longer trajectories, motion blur, and complex appearance.The Tanks and Temples sequences are downscaled to full HD and limited to the first 1,000 images.Source: §4.1 · p. 6

4.3. Quantitative Evaluation

  • On Static Hikes, our method consistently beats BARF; Mip-NeRF360 cannot produce results for 19.5% of test frames, while our rendered frames are higher quality.Figure 6 plots sorted frame-wise metrics across the dataset.Source: Figure 6 · p. 7, §4.3 · p. 7

4.4. Qualitative evaluation

  • Qualitative results show local fields retain sharpness along trajectories, while locality improves robustness to illumination changes and pose-estimation failures.The method also returns results for all frames in the reported qualitative comparisons.Source: Figure 7 · p. 7, §4.4 · pp. 7–8, Figure 8 · p. 8

4.2. Compared Methods

  • Comparisons include unbounded-scene methods, scalable Mega-NeRF, and self-calibrated SCNeRF and BARF; SCNeRF is excluded from fully self-calibrated evaluation because it requires COLMAP initialization.Other compared methods that require preprocessed poses use COLMAP poses.Source: §4.2 · pp. 6–7

4.3. Quantitative Evaluation

  • On Tanks and Temples, our method provides competitive results, matches Mip-NeRF360 with COLMAP poses, and outperforms self-calibrating methods when optimizing poses from scratch.The authors attribute the from-scratch comparison to progressive optimization, which estimates fewer parameters at once and adds a locality prior to camera pose.Source: §4.3 · p. 7

4.5. Ablation Study

  • Ablations indicate both progressive optimization and local radiance fields are necessary; progressive optimization is crucial for long-sequence pose estimation, and local fields support robustness and scalability.Table 2 reports PSNR, SSIM, and LPIPS on five Static Hikes scenes.Source: §4.5 · p. 8, Table 2 · p. 8

4.6. Limitations

  • The method assumes continuous video without shot changes, does not tackle dynamic elements, and sudden rotations can break pose estimation and yield poorly rendered images.It is therefore not suited to reconstructing scenes from unstructured frames without coherence.Source: §4.6 · p. 8

5. Conclusions

The paper presents radiance-field reconstruction for large scenes from casually captured video. It combines progressive joint pose and field optimization with dynamically instantiated local fields, and reports improved reconstruction robustness and fidelity across two datasets.

  • The method reconstructs radiance fields of large scenes from casually captured video.Source: §5 · p. 8
  • Its core ideas are progressive optimization for joint camera-pose and radiance-field estimation and dynamic instantiation of local radiance fields.Source: §5 · p. 8
  • Evaluation on two datasets shows substantially improved robustness and fidelity of radiance-field reconstruction in this challenging scenario.Source: §5 · p. 8

Related papers

Loading 2303.13791v1…