Source-linked AI summary
AnyBokeh: Physics-Guided Any-to-Any Bokeh Editing with Optical Fingerprint Transfer
Xinyu Hou, Xiaoming Li, Zongsheng Yue, Chen Change Loy
TL;DR
Any-to-any bokeh editing must infer both the source image’s spatially varying blur and its target blur scale when camera parameters are incomplete. AnyBokeh transfers a source-specific optical fingerprint and uses dual-CoC conditioning for relative blur synthesis, achieving effective editing across any-to-any, AIF-to-bokeh, and bokeh-to-AIF tasks.
Problem
Any-to-any bokeh editing must infer spatially varying source blur and target blur scale when camera parameters are incomplete or unavailable.
Method
AnyBokeh estimates source CoC and disparity, transfers their optical-fingerprint slope to the target setting, and synthesizes relative blur with dual-CoC conditioning.
Results
AnyBokeh achieves the best performance across all metrics on both AIF-to-bokeh datasets and improves DISTS and FID over its single-stage baseline.
Takeaways & Limitations
AnyBokeh enables faithful, spatially adaptive bokeh editing across any-to-any editing, AIF-to-bokeh rendering, and bokeh-to-AIF deblurring without per-image calibration.
Takeaways & Limitations
Extremely defocused inputs can lack enough visual evidence for faithful sharp-detail reconstruction, while extreme bokeh changes may leave residual artifacts or produce hollows.
Abstract
from arXiv · showhide
Depth-of-field control is a fundamental tool in photography, yet post-capture bokeh editing from a single image remains challenging. A practical editor should handle images captured under arbitrary focus and aperture settings. Existing methods typically assume an all-in-focus input, or first recover an all-in-focus image before rendering new bokeh. Such pipelines can discard useful blur cues from the source image and propagate reconstruction artifacts into the final edit. We introduce AnyBokeh, a physics-guided framework for any-to-any bokeh editing. Instead of treating source blur merely as a degradation to be removed, AnyBokeh estimates the source blur state with a signed circle-of-confusion map and a disparity map. By modeling the linear relation between signed circle of confusion and disparity difference, AnyBokeh estimates a source-specific optical fingerprint and transfers the source optical characteristics to the desired focus and aperture setting. A generative editor conditioned on both source and target circle-of-confusion maps then performs relative blur synthesis, enabling spatially adaptive deblurring, preservation, and defocus rendering. To support physically supervised learning, we further construct a high-fidelity synthetic dataset with accurate depth, focus distance, and full EXIF metadata. Experiments on real-world benchmarks show that AnyBokeh achieves faithful and controllable editing across any-to-any bokeh editing, all-in-focus-to-bokeh rendering, and defocus deblurring, while avoiding all-in-focus reconstruction and test-time bokeh-level calibration commonly required by existing approaches. The code and dataset will be available at https://github.com/itsmag11/AnyBokeh.
1 Introduction
AnyBokeh addresses arbitrary source blur by estimating its optical state and transferring that information to a desired focus and aperture setting. Its dual-CoC editor enables relative, spatially adaptive blur editing without an all-in-focus bottleneck.
- The challenge: Any-to-any bokeh editing must infer both the source blur state and the target blur scale when camera parameters are incomplete.Source images may mix sharp, mildly blurred, and severely defocused regions, while focus distance, aperture, focal length, sensor size, and resolution are often unavailable.
- The challenge: Existing pipelines either assume all-in-focus inputs or reconstruct an all-in-focus image, potentially discarding source optical cues and propagating hallucinated details.They may also require per-image bokeh-level calibration to match a desired blur strength.
- Optical fingerprint transfer: AnyBokeh estimates a signed source CoC map and disparity, then models their linear relation to derive a source-specific optical fingerprint.The fingerprint captures the global optical and scaling factors that map disparity differences to image-space defocus.
- Optical fingerprint transfer: The method transfers the optical fingerprint to the desired focus and aperture setting to compute a target CoC map without AIF reconstruction or test-time calibration.This derives target blur scale from the observed source blur and the user-specified optical change.
- Dual-CoC editing: A generative editor conditioned on the source image, source CoC, and target CoC performs relative blur synthesis across spatially selective deblurring, preservation, and defocus rendering.AnyBokeh also introduces UnrealBokeh, a synthetic dataset with ground-truth depth, focus distance, aperture, and focal length.
2 Related Work
Related work spans physically based, image-space, layered, and neural bokeh rendering, alongside geometry-aware defocus deblurring. Persistent challenges include depth-discontinuity artifacts, complex occlusions, arbitrary refocusing, and accurate blur estimation.
- Bokeh rendering: Physically based depth-of-field rendering can produce realistic bokeh but is computationally expensive and requires complete 3D geometry.Image-space post-filtering improves efficiency but struggles with depth discontinuities, color bleeding, and partial occlusion artifacts.
- Bokeh rendering: Layered and neural rendering methods address geometric artifacts, yet complex occlusions remain a persistent challenge.The field progressed from all-in-focus natural camera-bokeh benchmarks toward depth-aware layered scene representations.
- Arbitrary refocusing: DiffCamera enables arbitrary refocusing with a DiT-based framework trained on simulated depth-of-field pairs, using stacking constraints and depth dropout.The passage raises potential exposure bias from its defocus maps used for bokeh-to-clear rendering.
- Defocus deblurring: Defocus deblurring seeks an all-in-focus image from spatially varying depth-of-field blur, whose magnitude changes across depth discontinuities and occlusion boundaries.Learning-based approaches use sensor sub-views, adaptive filters, disparity-aware kernels, and implicit inverse kernels to connect blur with scene geometry.
3 Methodology
AnyBokeh estimates the source image’s blur and geometry, transfers its optical scale to a requested setting, and edits directly between blur states. This avoids reconstructing an all-in-focus intermediate while supporting relative and absolute blur control.
- Circle of Confusion: The signed CoC magnitude gives blur diameter, while its sign indicates whether a scene point lies before or behind the focal plane.The paper uses signed CoC diameter throughout unless otherwise specified.
- Circle of Confusion: Resizing changes CoC magnitude in pixels because physical blur is converted using image width and sensor width.UnrealBokeh computes dense signed CoC supervision from depth and optical metadata.
- Optical fingerprint: A linear CoC–disparity relation yields a source-specific optical fingerprint κsrc that absorbs aperture, focal length, sensor, resolution, and disparity-scale factors.This makes the source blur scale estimable even when EXIF metadata is incomplete.
- Stage 1: CoC and disparity estimation: A signed CoC map records source defocus, while disparity supplies the geometry needed to transfer that blur state.Stage 1 jointly predicts CoCsrc and D from the input image.
- Target CoC calculation: The target CoC is obtained by scaling κsrc for the desired aperture and focus, avoiding test-time bokeh-level calibration.The aperture ratio is defined as ra = Nsrc/Ntgt when focal length and sensor scale are fixed.
- Dual-CoC editing: Dual-CoC conditioning gives Stage 2 both source and target blur maps, enabling relative deblurring, blur preservation, and defocus synthesis.The editor maps (Isrc, CoCsrc, CoCtgt) to the target image rather than requiring an all-in-focus input.
- UnrealBokeh dataset: UnrealBokeh provides physically supervised training data by pairing rendered RGB images with depth, optical metadata, and dense signed CoC maps.The dataset records focus distance, aperture, focal length, sensor size, and image resolution.
4 Experiments
AnyBokeh evaluates source-blur estimation, controllable editing, and restoration across multiple settings. Its physics-guided pipeline supports arbitrary aperture and focus changes while avoiding forced all-in-focus reconstruction and test-time bokeh calibration.
- Source CoC and disparity estimation: AnyBokeh estimates source CoC and disparity, then transfers a source-specific optical fingerprint to compute target blur for user-specified edits.Stage 1 predicts source CoC and disparity; Stage 2 conditions generation on source and target CoC.
- Any-to-any bokeh editing: AnyBokeh outperforms deblur-then-render baselines on EBB! and RealBokeh without test-time bokeh-level calibration.Its target CoC is derived from the input image and user-specified focus/aperture change rather than externally calibrated blur scale.
- Any-to-any bokeh editing: AnyBokeh adjusts bokeh strength with aperture changes and shifts the sharp region when users relocate focus from a single input image.Aperture-control examples keep focus fixed while varying the target f-number; focus-control examples keep aperture fixed while moving the target focus point.
- Defocus deblurring: AnyBokeh achieves the best DISTS and FID on both EBB! and RealBokeh for defocus deblurring.The authors report improved perceptual structure and distribution-level realism, although LPIPS is slightly worse than restoration baselines.
- Defocus deblurring: LPIPS can penalize plausible detail recovery when synthesized high-frequency structures differ from the exact reference.Restoration baselines may remain closer in LPIPS while producing smoother outputs with residual defocus blur.
- Ablation studies: The two-stage design predicts blur-aware geometry from defocused inputs, improving DISTS and FID over a single-stage baseline.The single-stage variant relies on depth estimated from severely defocused images, whereas Stage 1 jointly predicts CoCsrc and disparity.
- Ablation studies: Providing source CoC improves all ablation metrics by explicitly specifying the transition from source blur to target blur.Without CoCsrc, the editor must infer the input blur state from ambiguous image appearance, especially under spatially varying or severe defocus.
5 Discussion and Limitations
AnyBokeh treats existing blur as optical information, enabling relative edits without reconstructing an all-in-focus intermediate. Its main weaknesses emerge when severe blur has erased details or when bokeh changes are extreme.
- Core formulation: AnyBokeh transfers source-specific optical information and conditions editing on both source and target CoC maps.This supports spatially adaptive deblurring, blur preservation, and defocus synthesis without canonicalizing inputs to all-in-focus images.
- Failure cases: Severe source defocus can prevent faithful sharp reconstruction because high-frequency textures and fine structures are missing.Performance improves as source blur weakens, and the method remains clearer than the baseline in the cited extreme-blur cases.
- Failure cases: Very large bokeh balls may leave residual or hollow artifacts when the target requires substantially reducing their size.The failure is linked to rare extreme examples in training data and scene content occluded by large defocus disks.
- Scope: AnyBokeh remains effective across a wide range of practical post-capture bokeh editing scenarios despite limitations in extreme blur or high-resolution settings.The cited limitations primarily concern unusually demanding blur transformations and inputs with insufficient visual evidence.
6 Conclusion
AnyBokeh performs any-to-any bokeh editing by transferring a source image’s CoC-based optical fingerprint instead of forcing arbitrary inputs through an all-in-focus intermediate. Dual-CoC conditioning and the UnrealBokeh dataset support physically grounded editing across multiple tasks without per-image bokeh calibration.
- Method: AnyBokeh transfers the source blur state to a desired focus and aperture setting through a CoC-based optical fingerprint.The framework models editing as a relative transition between source and target blur states.
- Method: Dual-CoC conditioning enables spatially adaptive editing without per-image bokeh-level calibration.The editor receives both source and target CoC information to guide the relative blur transformation.
- Results: Experiments demonstrate effectiveness across any-to-any editing, AIF-to-bokeh rendering, and bokeh-to-AIF deblurring.Ablations further identify source CoC estimation and dual-CoC conditioning as important for faithful relative editing.
- Dataset: UnrealBokeh provides per-pixel depth and complete EXIF metadata for computing ground-truth signed CoC maps.This supports physically supervised learning of the blur and disparity estimation components.