Source-linked AI summary
BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields
Peng Wang, Lingzhe Zhao, Ruijie Ma, Peidong Liu
TL;DR
Motion-blurred images violate NeRF’s sharp-image assumption and make accurate camera poses difficult to obtain. BAD-NeRF models blur formation while jointly learning the scene representation and exposure-time camera trajectories. The paper reports effective deblurring, novel-view rendering, and camera-trajectory recovery on synthetic and real data.
Problem
Motion blur violates NeRF’s sharp-image assumption, while blurred images make accurate camera poses difficult to obtain.
Source: §1 · pp. 1–2Method
BAD-NeRF models a blurred image as an average of rendered sharp images along an exposure trajectory and jointly estimates NeRF parameters and camera motion.
Source: §1 · p. 2Results
BAD-NeRF effectively deblurs images, renders novel views, and recovers camera motion trajectories accurately within exposure time.
Source: §5 · p. 8Takeaways & Limitations
The method acquires high-quality 3D scene representations from motion-blurred images.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) have received considerable attention recently, due to its impressive capability in photo-realistic 3D reconstruction and novel view synthesis, given a set of posed camera images. Earlier work usually assumes the input images are of good quality. However, image degradation (e.g. image motion blur in low-light conditions) can easily happen in real-world scenarios, which would further affect the rendering quality of NeRF. In this paper, we present a novel bundle adjusted deblur Neural Radiance Fields (BAD-NeRF), which can be robust to severe motion blurred images and inaccurate camera poses. Our approach models the physical image formation process of a motion blurred image, and jointly learns the parameters of NeRF and recovers the camera motion trajectories during exposure time. In experiments, we show that by directly modeling the real physical image formation process, BAD-NeRF achieves superior performance over prior works on both synthetic and real datasets. Code and data are available at https://github.com/WU-CVGL/BAD-NeRF.
1. Introduction
BAD-NeRF addresses the mismatch between NeRF’s sharp-image assumption and motion-blurred inputs, which also make camera poses difficult to estimate. It models blur formation and jointly estimates the scene representation and camera motion, reporting improved performance on synthetic and real data.
1. Introduction
- Motion blur violates NeRF’s sharp-image assumption and makes accurate camera poses difficult to obtain from blurred images.Blurred images encode motion during exposure and complicate keypoint detection and matching.Source: §1 · pp. 1–2
- BAD-NeRF models blur by averaging rendered sharp images along an exposure trajectory and jointly optimizes NeRF parameters and camera motion.The trajectory is represented by start and end poses, with intermediate poses linearly interpolated in SE(3).Source: §1 · p. 2
- BAD-NeRF reports superior performance to prior state-of-the-art methods on synthetic and real datasets.The approach explicitly models the motion-blurred image formation process.Source: §1 · p. 2
- The paper presents photometric bundle adjustment for motion-blurred images and reports high-quality 3D representations and novel-view images from blurred inputs.The formulation is described as potentially integrable with other vision pipelines.Source: §1 · p. 2
2. Related Work
Prior NeRF work improves rendering efficiency, handles inaccurate poses or challenging conditions, and extends scene scale and object types. BAD-NeRF differs from methods that estimate a single pose or fix poses during training, while related deblurring work includes optimization-based, learned, and multi-view approaches.
2. Related Work
- NeRF variants target rendering efficiency, inaccurate poses or challenging conditions, large scenes, and non-rigid reconstruction.NeRF has also been used in 3D-aware image generation.Source: §2 · p. 2
- BARF and Jeong et al. jointly learn scene and camera parameters, but estimate poses at particular timestamps rather than motion trajectories during exposure.DeblurNeRF trains on blurred images but fixes poses obtained from ground truth or COLMAP.Source: §2 · p. 2
- Motion deblurring includes optimization of latent sharp images and blur kernels and end-to-end learning for single images or video.Park et al. jointly recover camera poses, dense depth maps, and latent sharp images from multi-view blurred images; BAD-NeRF instead represents the scene implicitly with NeRF.Source: §2 · pp. 2–3
3. Method
BAD-NeRF jointly learns a NeRF scene representation and camera motion trajectories from blurry images by modeling how motion forms an image. It renders virtual sharp views over each exposure, combines them into blurry images, and optimizes photometric consistency with the captured images.
3.4. Loss Function
- BAD-NeRF jointly estimates NeRF parameters and each image’s motion trajectory by minimizing photometric loss between synthesized and captured blurry images.The trajectory is represented by poses at the exposure start and end, with intermediate poses used to synthesize virtual sharp images.Source: §3 · p. 3, §3.4 · p. 4
3.1. Neural Radiance Fields
- NeRF represents the 3D scene with MLPs and computes pixel intensity by volume-rendering sampled 3D points along camera rays.The rendered pixel intensity is a function of the learnable network parameters and camera pose.Source: §3.1 · p. 3
3.2. Motion Blur Image Formation Model
- Motion blur is modeled by integrating virtual sharp images captured during exposure, with the integral discretely approximated from sampled images.The captured image is formed from the virtual sharp images associated with camera motion during the exposure time.Source: §3.2 · p. 3
3.3. Camera Motion Trajectory Modeling
- Camera motion is represented by start and end poses, with intermediate poses linearly interpolated in the Lie algebra of SE(3).The authors also explore cubic B-spline trajectories, but report that linear interpolation delivers satisfying performance for typically short exposures.Source: §3.3 · p. 4
4. Experiments
Across synthetic and real data, BAD-NeRF outperforms prior methods on deblurring and novel-view synthesis, while recovering camera poses more accurately. Experiments also find that linear trajectory interpolation performs comparably to cubic B-Splines.
4.1. Experimental details
- The evaluation uses synthetic and real datasets, compares deblurring and novel-view synthesis methods, and assesses pose accuracy with ATE.Synthetic MBA-VO images use non-constant ETH3D motion trajectories; COLMAP poses initialize BAD-NeRF, and the model is trained for 200K iterations.Source: §4.1 · p. 5
4.2. Ablation study
- Performance saturates as the number of interpolated virtual cameras increases, and the experiments use seven virtual images.Source: Figure 3 · p. 5, §4.2 · p. 6
4.3. Results
- BAD-NeRF achieves the best deblurring performance on the Deblur-NeRF synthetic dataset, while DeblurNeRF is sensitive to whether poses are ground-truth or COLMAP-estimated.The comparison includes DeblurNeRF trained with ground-truth poses and with COLMAP-estimated poses.Source: Table 2 · p. 6, §4.3 · p. 6
- BAD-NeRF delivers state-of-the-art novel-view synthesis performance compared with prior works on the Deblur-NeRF synthetic dataset.Source: Table 3 · p. 6
4.2. Ablation study
- Spline-based trajectories outperform direct optimization of N poses; linear interpolation performs comparably to cubic B-Splines across the tested datasets.Linear interpolation is slightly better on Deblur-NeRF datasets, while cubic B-Splines are slightly better on MBA-VO datasets.Source: Table 1 · p. 5, §4.2 · p. 6
4.3. Results
- On MBA-VO, BAD-NeRF achieves the best deblurring performance despite non-constant camera motion during exposure.The dataset’s synthetic images use real motion trajectories from ETH3D.Source: Table 4 · p. 6, §4.3 · p. 7
- BAD-NeRF recovers camera poses more accurately than COLMAP and BARF on Deblur-NeRF sequences, measured using ATE after alignment with ground truth.The comparison with Park et al. is omitted because that method hardly converges.Source: §4.3 · p. 7, Table 5 · p. 7
- Qualitative results show consistent performance across blur levels and superior results on real data; DeblurNeRF leaves sharp edges unresolved near large depth changes.Source: Figure 4 · p. 7, §4.3 · p. 8, Figure 5 · p. 8
5. Conclusion
BAD-NeRF uses NeRF-based photometric bundle adjustment to jointly learn a 3D representation and optimize camera poses from blurry images and inaccurate initial poses. Evaluations on real and synthetic datasets report effective deblurring, novel-view rendering, and camera-motion recovery.
5. Conclusion
- BAD-NeRF jointly learns a NeRF-based 3D representation and optimizes camera poses from blurry images and inaccurate initial poses.The paper presents this as a photometric bundle adjustment formulation for motion-blurred images.Source: §5 · p. 8
- Evaluations on real and synthetic datasets report that BAD-NeRF effectively deblurs images and renders novel views.Source: §5 · p. 8
- The method also recovers camera motion trajectories accurately within exposure time.Source: §5 · p. 8