Source-linked AI summary
Deblur-NeRF: Neural Radiance Fields from Blurry Images
Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, Pedro V. Sander
TL;DR
Blurry inputs from motion or defocus degrade NeRF reconstruction and produce artifacts in novel views. Deblur-NeRF simulates the blurring process with a deformable sparse kernel while jointly optimizing a sharp NeRF, outperforming NeRF and image-space baselines on synthetic and real data. Its scope is limited when blur is view-consistent or severe enough to prevent reliable camera-pose reconstruction.
Problem
Motion and defocus blur significantly degrade NeRF reconstruction quality, while existing approaches do not address training NeRF from blurry input images.
Method
Deblur-NeRF jointly optimizes a sharp NeRF and a deformable sparse kernel that simulates spatially varying blur, including ray-origin changes.
Results
Deblur-NeRF outperforms original NeRF and image-space deblurring baselines on synthetic and real datasets with camera motion and defocus blur.
Takeaways & Limitations
The framework reconstructs a sharp NeRF from blurry input and models both camera motion blur and defocus blur within one approach.
Takeaways & Limitations
The method may fail for view-consistent blur and for severely blurred inputs when COLMAP cannot reconstruct camera poses.
Abstract
from arXiv · showhide
Neural Radiance Field (NeRF) has gained considerable attention recently for 3D scene reconstruction and novel view synthesis due to its remarkable synthesis quality. However, image blurriness caused by defocus or motion, which often occurs when capturing scenes in the wild, significantly degrades its reconstruction quality. To address this problem, We propose Deblur-NeRF, the first method that can recover a sharp NeRF from blurry input. We adopt an analysis-by-synthesis approach that reconstructs blurry views by simulating the blurring process, thus making NeRF robust to blurry inputs. The core of this simulation is a novel Deformable Sparse Kernel (DSK) module that models spatially-varying blur kernels by deforming a canonical sparse kernel at each spatial location. The ray origin of each kernel point is jointly optimized, inspired by the physical blurring process. This module is parameterized as an MLP that has the ability to be generalized to various blur types. Jointly optimizing the NeRF and the DSK module allows us to restore a sharp NeRF. We demonstrate that our method can be used on both camera motion blur and defocus blur: the two most common types of blur in real scenes. Evaluation results on both synthetic and real-world data show that our method outperforms several baselines. The synthetic and real datasets along with the source code is publicly available at https://limacv.github.io/deblurnerf/
1. Introduction
Deblur-NeRF addresses artifacts caused by motion and defocus blur in NeRF training by simulating image formation and jointly recovering a sharp scene representation.
- Motivation: Motion and defocus blur substantially degrade NeRF reconstruction, producing artifacts in rendered novel views.Motion blur can arise from camera shake during long exposures, while defocus blur occurs with large depth variation and apertures.
- Limitations of Existing Approaches: Existing image-space deblurring baselines cannot guarantee multi-view consistency or exploit 3D geometry across views.Single-image methods miss neighboring-view information, while video methods can produce inaccurate correspondences at large baselines.
- Approach: Deblur-NeRF explicitly models blurring by convolving a clean rendered image with a learned blur kernel during NeRF training.The framework jointly optimizes a sharp NeRF and blur parameters so synthesized blurry views match the inputs.
- Results: Experiments on synthetic and real datasets show improved performance over original NeRF and image-space baselines for camera motion and defocus blur.The evaluation covers the two blur types addressed by the framework.
- Approach: The deformable sparse kernel models blur efficiently by deforming a canonical sparse kernel and supports different blur types.The method avoids the computation and memory costs of dense kernels in NeRF rendering.
- Approach: The method extends blur-kernel modeling into 3D by jointly optimizing each kernel point’s ray origin according to physical blur formation.This design accounts for changing camera centers in motion blur and equivalent origin variation in defocus blur.
2. Related Work
Related work addresses non-ideal NeRF inputs and blur through calibration, alternative scene assumptions, or image-space processing, but these approaches have important scope limitations.
- NeRF with Non-Ideal Input: Prior NeRF extensions handle illumination changes, moving objects, scale variation, distortion, missing camera poses, or complex camera calibration.Examples include NeRF-W, Mip-NeRF, SCNeRF, BRAF, NeRF−−, and GNeRF.
- Image Deblurring: Blur-kernel methods use homographies, regional assumptions, depth, or optical flow, often imposing strong assumptions or targeting one blur type.These approaches reduce the space of spatially varying kernels through specialized parameterizations.
- Image Deblurring: Deblur-NeRF instead represents spatially varying kernels with an MLP that can generalize across different blur types.Image deblurring networks may also struggle to generalize to unseen real-world blur types because they depend heavily on training data.
3. Preliminary
NeRF represents a static scene as a continuous volumetric function and renders pixels by accumulating color and density samples along camera rays.
- NeRF Representation: A NeRF maps a 3D position and 2D viewing direction to color and volume density through an MLP.The scene function uses positional encoding to represent the input coordinates.
- NeRF Representation: Positional encoding maps vector elements into a higher-dimensional frequency space, with L controlling the highest frequency and scene smoothness.The encoding is part of the MLP-based scene representation.
- Volume Rendering: To render a pixel, NeRF emits a camera ray, samples sorted distances between near and far planes, and evaluates color and density at those samples.The ray is emitted from the camera projection center along the pixel’s viewing direction.
- Volume Rendering: The final pixel color is computed by volume-rendering the sampled colors and densities using inter-sample distances and accumulated transmittance.The rendering process is differentiable, allowing the scene function to be optimized from image observations.
4. Method
Deblur-NeRF trains a sharp NeRF by explicitly simulating blur with a deformable sparse kernel, jointly optimizing sparse rays, weights, ray origins, and the NeRF. Training renders and blends optimized rays to match blurry inputs, while testing directly renders sharp rays from the restored NeRF.
- 4. Method: Deblur-NeRF generates multiple optimized rays with a Deformable Sparse Kernel and blends their NeRF renderings to synthesize each blurry pixel.The synthetic blurry pixel is supervised by the corresponding ground-truth blurry color during training.
- 4.1. Deformable Sparse Kernel: A dense K × K blur kernel is approximated with N sparse points whose continuous locations and weights are jointly optimized with the NeRF.The sparse approximation avoids the computation and memory costs of rendering every ray in the supporting window.
- 4.4. Aligning the NeRF: At test time, kernel deformation is removed and the restored sharp NeRF directly renders sharp views with gamma correction.An alignment loss additionally constrains one optimized ray toward the input ray to reduce non-rigid NeRF–kernel deformation.
- 4.1. Deformable Sparse Kernel: An MLP deforms canonical kernel locations and predicts their weights using spatial coordinates and a learned view embedding for view-dependent blur patterns.The final sparse location is q = q′ + ∆q, and the model can fit a different kernel for each input view.
- 4.2. Convolution with Irradiance: The method assumes the sharp NeRF predicts linear-space colors, then applies a simple gamma correction to compensate for nonlinear camera response.The authors state that this scheme improves quality, while more complex calibrated or jointly optimized response functions are possible.
- 4.3. Optimizing the Ray Origin: Each sparse kernel point can jointly optimize ray-origin translation, modeling blur from camera motion and defocus more physically than shared-origin image convolution.This accounts for rays cast from different origins, including effects that are not well approximated by 2D pixel translation near depth discontinuities.
5. Experiments
Experiments evaluate Deblur-NeRF on synthetic and real scenes with camera motion and defocus blur, using ablations and comparisons against naive and image-space baselines. The full model achieves the best ablation results, while N = 5 balances quality and efficiency.
- Ablation study: The full model achieves the best ablation results across the reported PSNR, SSIM, and LPIPS evaluations.Other variants show larger errors, particularly near object boundaries.
- Ablation study: Increasing the number of sparse kernel points improves PSNR and SSIM across blur types and blur levels, but with diminishing improvement.The experiments use N = 5 for subsequent evaluations as a balance between rendering quality and efficiency.
- Experimental setup: The experiments cover synthetic and real scenes with camera motion blur and defocus blur, including quantitative and qualitative evaluations.Real-world scenes lack ground-truth images because of pose misalignment or exposure variation.
- Comparisons: On synthetic scenes, the full pipeline outperforms naive NeRF and image-space deblurring baselines for both blur types.The compared image-space methods include MPR + NeRF, PVD + NeRF, and KPAC + NeRF.
- Comparisons: On real scenes, Deblur-NeRF produces sharp edges, rich details, sharper depth maps, and more view-consistent results than the baselines.The reference columns may be misaligned or have different exposures from the ground truth.
6. Discussion and Conclusion
The discussion identifies an ill-posedness in jointly recovering a sharp NeRF and blur kernel from blurry images, while explaining how view inconsistency helps the framework compensate for blur. The method has explicit failure cases, including view-consistent or severe blur.
- Discussion: Jointly recovering a sharp NeRF and blur kernel from only blurry images is ill-posed because a blurry scene can also explain the observations.The framework uses NeRF’s view-consistency prior while the DSK module compensates for view-inconsistent blur.
- Limitations: The method can fail when blur is view consistent, such as repeated camera shake in one direction or a fixed focal point.The authors identify image priors as a possible direction for addressing consistent blur.
- Limitations: Severe blur can cause COLMAP to fail to reconstruct camera poses, although this was observed only in very blurry cases.The authors report robustness to inaccurate poses in their real-world experiments.
- Conclusion: Experiments on synthetic and real-world scenes demonstrate improvements over naive NeRF and image-space deblurring approaches.The conclusion presents the framework as a method for training a sharp NeRF under blurry input.