Source-linked AI summary
Selective Light Field Refocusing for Camera Arrays Using Bokeh Rendering and Superresolution
Yingqian Wang, Jungang Yang, Yulan Guo, Chao Xiao, Wei An
TL;DR
Camera-array refocusing must address aliasing while improving focused-region resolution and bokeh. The paper combines depth-based bokeh rendering with reconstruction-based superresolution, reporting strong visual performance, post-adjustable depth of field, and acceptable computational cost.
Problem
Camera-array refocusing faces bokeh aliasing, while existing approaches trade off computational cost, residual artifacts, and image resolution.
Source: §I · p. 1Method
The method renders bokeh with a depth-based anisotropic filter and uses the rendered bokeh image as a regularizer in reconstruction-based superresolution.
Source: §I · p. 1, §II · p. 2Results
Compared with,, and, the method achieves the highest PSNR in focused regions and promising bokeh performance, with execution time 5% to 10% of.
Source: A. Visual Performance Comparison · p. 3, D. Execution Time · p. 4Takeaways & Limitations
The method enables post-processing adjustment of both the focused depth and depth of field, which the paper says is unavailable with DSLRs and prior refocusing or LF reconstruction methods.
Source: B. Adjustable DoF · p. 3Disparity-estimation errors partially influence visual performance, and errors at focused-region boundaries can produce artifacts, although the method is robust to minor errors.
Source: C. Influence of Errors in Disparity… · p. 3
Abstract
from arXiv · showhide
Camera arrays provide spatial and angular information within a single snapshot. With refocusing methods, focal planes can be altered after exposure. In this letter, we propose a light field refocusing method to improve the imaging quality of camera arrays. In our method, the disparity is first estimated. Then, the unfocused region (bokeh) is rendered by using a depth-based anisotropic filter. Finally, the refocused image is produced by a reconstruction-based superresolution approach where the bokeh image is used as a regularization term. Our method can selectively refocus images with focused region being superresolved and bokeh being aesthetically rendered. Our method also enables postadjustment of depth of field. We conduct experiments on both public and self-developed datasets. Our method achieves superior visual performance with acceptable computational cost as compared to other state-of-the-art methods. Code is available at https://github.com/YingqianWang/Selective-LF-Refocusing.
I. INTRODUCTION
Camera arrays offer a route to high-quality light-field imaging, but sparse angular sampling creates aliasing and prior methods trade off bokeh quality, resolution, and computational cost. The proposed method integrates bokeh rendering with superresolution and allows depth of field to be adjusted after capture.
I. INTRODUCTION
- DSLRs produce high-resolution, high-SNR images with shallow depth of field, but their high-quality lenses and sensors make them bulky and expensive.Bokeh can blur foreground or background elements and guide attention to major subjects.Source: §I · p. 1
- Camera arrays improve image resolution over plenoptic cameras, but their wider baselines and sparse angular sampling make traditional refocusing prone to bokeh aliasing.Arrays use independent cameras in a regular grid; plenoptic cameras face a choice between dense angular and dense spatial sampling.Source: §I · p. 1
- LF reconstruction methods address aliasing but incur high computational burden or leave artifacts, while bokeh rendering improves bokeh without increasing image resolution.Some bokeh-rendering approaches also require costly depth sorting.Source: §I · p. 1
- The proposed method integrates bokeh rendering and superresolution, permits post-capture depth-of-field adjustment, and produces images reported as more DSLR-like than existing methods.The paper reports experiments on a self-developed dataset.Source: §I · p. 1
II. RECONSTRUCTION-BASED SR MODEL
The reconstruction model accounts for camera-array sub-image shifts, blur, down-sampling, and noise. It estimates a high-resolution image by fitting the observations while regularizing toward a bokeh image and using bilateral total variation.
II. RECONSTRUCTION-BASED SR MODEL
- The camera-array degradation model accounts for parallax-induced shearing shifts, optical-distortion blur, and down-sampling from low-resolution sensors.It models each camera's sub-image as a shifted, blurred, down-sampled high-resolution image plus noise.Source: §II · p. 2
- The SR objective combines observation fidelity with bokeh and bilateral total variation regularization.The depth-based, spatially varying bokeh weights are larger in unfocused regions, and the bokeh term uses the bokeh image.Source: §II · p. 2
- Gradient descent approximates the optimal solution to the reconstruction objective.The paper introduces step size and iteration settings in its experiments section.Source: §II · p. 2
III. BOKEH RENDERING
The method renders bokeh with a depth-based anisotropic filter, then upsamples the rendered center view to create a bokeh image. CoC radius and depth-based weights determine which regions blur and how strongly.
III. BOKEH RENDERING
- The bokeh intensity K reflects overall bokeh and depth of field; larger K produces stronger bokeh and shallower depth of field.Source: §III · p. 2
- The anisotropic filter combines contributions from surrounding points, with a point contributing only when Q lies within its CoC.The contributions are weighted when calculating the intensity at Q.Source: §III · p. 2, Figure 2 · p. 2
- Bicubic interpolation upsamples the rendered center view to the target resolution, producing the bokeh image xb.Source: §III · p. 2
- The depth-based weight vector ωb is calculated by normalizing CoC radii and applying a sigmoid to distinguish focus from bokeh.The sigmoid uses a decay factor and threshold; its output weights are reordered into a vector.Source: §III · p. 3
IV. EXPERIMENTS
The experiments compare refocusing, LF reconstruction, and bokeh-rendering methods on public and self-developed light-field datasets. Focused-region sharpness is assessed with PSNR, while bokeh is evaluated visually.
IV. EXPERIMENTS
- The Stanford LF test uses 3×3 sub-images down-sampled by a factor of 2, comparing methods,, and with the proposed method.Source: §IV · p. 3
- Focused-region PSNR is measured against the HR center-view groundtruth, while bokeh quality is evaluated by visual inspection.Source: §IV · p. 3
- The self-developed dataset scans nine camera positions on a 2×2 grid with an iPhone 6S, using Canon EOS 5D Mark IV images as groundtruth.The captured images are calibrated before evaluation.Source: §IV · p. 3
- The method uses 10 iterations as a compromise between visual performance and computational cost; tests ran in MATLAB on a 2.40 GHz CPU with 12 GB RAM.The authors report that the result is insensitive to most parameter settings except b, which strongly influences focus-versus-bokeh classification.Source: §IV · p. 3
A. Visual Performance Comparison
Across the compared methods, the proposed method achieves the highest focused-region PSNR and promising bokeh performance. Comparisons also show the trade-offs among refocusing, LF reconstruction, and bokeh rendering.
A. Visual Performance Comparison
- Method produces obvious aliasing despite shallow depth of field, while alleviates but does not eliminate aliasing and leaves the focused region blurred.Source: A. Visual Performance Comparison · p. 3
- Figure 3 compares Knights, Cards, and Chess across methods [9], [14], [20], and the proposed method (K=2); red-box focused-region sharpness is evaluated with PSNR.Source: Figure 3 · p. 3
- Compared with,, and, the proposed method achieves the highest focused-region PSNR and promising bokeh; Figure 3 shows three Stanford LF scenes.Source: A. Visual Performance Comparison · p. 3, Figure 3 · p. 3
B. Adjustable DoF
The method supports post-processing refocus to different depths and adjustment of depth of field using different K settings. The paper states that this adjustment is unavailable with DSLRs and several prior refocusing and LF reconstruction methods.
B. Adjustable DoF
- The method can refocus an image to different depths after capture.Source: B. Adjustable DoF · p. 3
- Different K settings allow post-processing adjustment of depth of field.Source: B. Adjustable DoF · p. 3
- The authors state that this DoF post-adjustment is unavailable with DSLRs, traditional refocusing methods,,, and LF reconstruction methods,,.Source: B. Adjustable DoF · p. 3
C. Influence of Errors in Disparity Estimation
Disparity-estimation errors partly affect refocused visual quality: boundary errors create artifacts, while minor errors are tolerated in focused and bokeh regions.
C. Influence of Errors in Disparity Estimation
- Boundary errors in estimated disparity create artifacts, although minor errors are tolerated in both focused and bokeh regions.The comparison uses ground-truth disparity and disparity estimated by on Box and Sideboard.Source: C. Influence of Errors in Disparity… · p. 3, Figure 6 · p. 4
- The comparison evaluates the method with ground-truth disparity against its results using disparity estimated by [23].Figure 6 shows Box and Sideboard from the HCI LF dataset [27].Source: Figure 6 · p. 4
D. Execution Time
The method runs in 5% to 10% of the time required by, while runtime depends more on iteration count than on bokeh intensity K.
D. Execution Time
- [9] is fastest because it requires no depth estimation, pixel mapping, or filtering; is most time-consuming, while is relatively fast but resolution-limited.The execution time of rises dramatically as image resolution increases.Source: D. Execution Time · p. 4
- 5% to 10% of 's execution time is required by the proposed method.Table II compares execution time.Source: D. Execution Time · p. 4, Table II · p. 4
- NoI affects overall execution time more than K, and the preferred K depends on the scene's distance.The paper suggests K=3 for close shots such as Knights and K=1 for distant scenes such as landscapes.Source: D. Execution Time · p. 4, Figure 7 · p. 4
V. CONCLUSION
The paper concludes that its camera-array refocusing method combines super-resolution with bokeh rendering, with experiments supporting its performance on public and self-developed datasets.
V. CONCLUSION
- The method formulates a super-resolution model and designs an anisotropic filter to increase image resolution and render bokeh.Source: §V · p. 4
- Experiments on public and self-developed datasets demonstrate the method's superiority.Source: §V · p. 4
- The method enables camera arrays to produce aesthetical photographs with acceptable computational cost.Source: §V · p. 4