PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashReFocus: Interactive Image Refocusing in Seconds

Ching-Heng Cheng, Chia-Ming Lee, Ming-Ching Chang, Xin Li, Yu-Lun Liu, Chih-Chung Hsu

arXiv:2610.23297v1cs.CVcs.AIeess.IV

TL;DR

Single-image refocusing must recover an all-in-focus image and render new blur, but existing generative methods rerun large diffusion models for every edit. FlashReFocus restores once with diffusion and uses a lightweight, depth-conditioned renderer for subsequent edits. It achieves the best perceptual bokeh fidelity on EBB! Val176 and 3CReal, with further edits taking about half a second.

  • Problem

    Single-image refocusing is interactive, but existing generative methods rerun large diffusion models for every edit even though rendering blur is largely determined by depth and the focal plane.

    Source: §1 · p. 1

  • Method

    FlashReFocus runs one-step diffusion once to restore an all-in-focus image, then uses a lightweight renderer conditioned on an explicit defocus map for each edit.

    Source: §1 · p. 2

  • Results

    FlashReFocus achieves the best perceptual bokeh fidelity on EBB! Val176 and unseen cameras in 3CReal, with further edits taking about half a second.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The explicit defocus map supports user-controlled focus, aperture-like strength, tilt-shift, and two-plane effects while in-focus content stays unchanged.

    Source: §1 · p. 2, §5 · p. 8

    The optional decoder refiner trades perceptual sharpness for fidelity.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Single-image refocusing changes the focal plane and depth of field of a photograph after capture. It is inherently interactive: users try many focal points and blur strengths on one image, so each edit should be fast and change only the blur. Recent generative methods, however, rerun large diffusion models for every edit. We observe that the two sub-problems differ in kind: recovering the all-in-focus image is ill-posed and needs a generative prior, whereas rendering blur is largely determined by depth and the focal plane. We therefore propose FlashReFocus, which runs diffusion once per photograph and renders every edit with a lightweight network. To address the ill-posed restoration, a one-step diffusion deblurrer with a decoder refiner recovers fine detail efficiently while balancing fidelity and perceptual quality. For precise control, the renderer is conditioned on an explicit defocus map and uses defocus-bias attention, which lets pixels at similar defocus share information without interference from pixels at other depths; in-focus content stays unchanged, and editing the map yields tilt-shift and two focal planes. To further test generalization to unseen cameras, we collect 3CReal, a three-camera benchmark of 89 pairs. FlashReFocus achieves the best perceptual fidelity for zero-shot bokeh rendering on EBB! and 3CReal, refocuses 148.3x faster than GenRefocus, and takes about half a second per further edit, 462x faster.

1. Introduction

Single-image refocusing must support repeated focal edits, but current generative methods rerun large diffusion models for each edit. FlashReFocus instead restores an all-in-focus image once and uses a lightweight renderer for subsequent edits.

1. Introduction

  • FlashReFocus separates ill-posed all-in-focus restoration from blur rendering, which is largely determined by depth and the focal plane.Prior generative approaches rerun a large diffusion model for every edit and can regenerate regions meant to stay sharp.Source: §1 · p. 1
  • Each session restores the all-in-focus image once, then uses a 1.3M-parameter renderer per edit while leaving in-focus content unchanged.In-focus regions stay at 44.4 dB PSNR against 29.0 dB for GenRefocus.Source: Figure 1 · p. 1, §1 · p. 2
  • An explicit defocus map and defocus-bias attention control blur by depth, enabling tilt-shift and two-focal-plane effects.Pixels at similar defocus share information without interference from pixels at unrelated depths.Source: §1 · p. 2, Figure 2 · p. 2
  • The authors introduce 3CReal, a benchmark of 89 pairs from three unseen cameras, and report leading perceptual fidelity on 3CReal and EBB!.The paper presents the benchmark as a test of cross-camera generalization.Source: §1 · p. 2
  • The method combines one-step diffusion restoration with a decoder refiner to recover detail while balancing fidelity and perceptual quality.The refiner preserves text and edges that VAE decoding distorts.Source: §1 · p. 2

2. Related Work

Prior work spans image restoration, bokeh rendering, and refocusing, including diffusion-based approaches to restoration and rendering. FlashReFocus uses an explicit all-in-focus and depth-based decomposition but replaces generative rendering with a lightweight renderer reused across edits.

2. Related Work

  • Restoration methods range from reconstruction-trained convolutional and Transformer models to classical or learned depth-varying blur approaches, while generative priors target perceptual quality.Diffusion provides a learned prior for ill-posed inverse problems.Source: §2 · p. 2
  • Learned bokeh renderers use varied controls and designs, while recent methods also apply diffusion to bokeh synthesis.Control signals include scalar strength, masks, depth maps, and multiplane images; FlashReFocus instead uses a depth-derived signed defocus map with global strength.Source: §2 · p. 3
  • Refocusing approaches include AiF restoration before focal blur, dual-camera capture, and two-diffusion-model pipelines; others generate focal stacks or edit images already containing bokeh.GenRefocus trains separate diffusion transformers for deblurring and bokeh rendering.Source: §2 · p. 3
  • FlashReFocus applies the AiF, depth, and lens-blur decomposition to real photographs, then reuses the AiF image across edits with a lightweight feed-forward renderer.This replaces generative rendering in the refocusing pipeline.Source: §2 · p. 3

3. Proposed Method

FlashReFocus separates restoration from blur rendering: it estimates an all-in-focus image once, then renders edits from explicit depth-derived controls. Its one-step diffusion restoration and lightweight renderer are designed to balance detail recovery, precise control, and reuse across focal settings.

3. Proposed Method

  • The method first estimates an all-in-focus image from the defocused photograph, then renders the requested depth of field from that intermediate.The explicit intermediate lets the stages be trained separately and reused across focal settings.Source: §3 · p. 3

3.1. Stage I: One-Step Diffusion Deblurring

  • A frozen Stable Diffusion U-Net maps the blurred-image latent directly to a sharp-image latent in one step, without a tagger, caption, or iterative sampling.Pixel and detail LoRA branches separate fidelity-oriented restoration from detail recovery.Source: §3.1 · pp. 3–4

3.2. Stage II: Defocus-Aware Bokeh Rendering

  • The renderer conditions a NAF encoder–decoder on depth, focal point, and blur strength, and predicts a residual added to the restored image.The residual stays near zero in focus, keeping in-focus regions intact; each new focal setting takes one forward pass.Source: Figure 4 · p. 4, §3.2 · pp. 4–6

3.1. Stage I: One-Step Diffusion Deblurring

  • Stage I uses mixed supervision from real lens-blur pairs, pixel-aligned light-field pairs, and synthetically blurred images with exact pixel correspondence.The sources combine different blur strengths and forms of supervision.Source: §3.1 · p. 4
  • A decoder-side refiner fuses multi-scale input features, guided by a Laplacian high-frequency map, to recover text and thin edges weakened by VAE decoding.The refiner is optional and trades perceptual sharpness for fidelity.Source: §3.1 · p. 4, Figure 5 · p. 4

3.2. Stage II: Defocus-Aware Bokeh Rendering

  • Defocus-bias attention uses strength-scaled defocus differences so tokens at similar defocus exchange information regardless of image distance.Training pre-trains on synthetic Hypersim scenes and adapts on RealBokeh to retain multi-focal control while adopting real-lens appearance.Source: §3.2 · pp. 5–6

4. Experiments

FlashReFocus is evaluated on defocus deblurring, bokeh rendering, controllability, and complete-refocus efficiency. It leads several perceptual rendering metrics on EBB! and 3CReal, while further edits reuse the restored image and depth.

4. Experiments

  • The evaluation covers deblurring on DPDD and RealDOF and bokeh rendering on EBB! Val176 and 3CReal, a benchmark of 89 aperture pairs from three camera systems unseen in training.For rendering, methods receive the same all-in-focus input and focal point, and are scored at the strength yielding their highest SSIM to the target.Source: §4 · p. 6

4.1. Defocus Deblurring

  • FlashReFocus with the refiner achieves the best LPIPS and BRISQUE on DPDD and RealDOF, while GenRefocus retains the best FID and regression restorers the highest PSNR.Stage I with the refiner needs 96 TFLOPs at 1536 × 1536 and ω=1, 181× fewer than the GenRefocus deblurring stage.Source: §4.1 · p. 6

4.2. Controllable Bokeh Rendering

  • FlashReFocus leads MS-SSIM, LPIPS, LPIPSVGG, DISTS, and CLIP-I on EBB! and 3CReal, and leads PSNR and SSIM on 3CReal.On EBB!, it improves LPIPS from 0.202 to 0.181 and DISTS from 0.111 to 0.104 over the next-best method; on 3CReal, it leads PSNR by 0.40 dB and improves LPIPS from 0.230 to 0.218.Source: Table 3 · p. 7, §4.2 · p. 7
  • Moving the focal point moves the sharp region, while increasing strength from 1× to 3× lowers background Laplacian variance from 237 to 29 and 11, with the subject retaining 65% of its sharpness.A vertical-ramp depth produces a tilt-shift look, raising the receding wall’s Laplacian variance from 7 to 321 and lowering the shirt’s from 362 to 24.Source: Figure 8 · p. 7, Figure 10 · p. 8, §4.2 · p. 8
  • On 3CReal, adapting synthetic pre-training to real pairs raises PSNR to 27.938 dB; differential learning rates add 0.043 dB and lower LPIPS to 0.218.Defocus-bias attention outperforms Bokehlicious relative-position attention by 0.084 dB PSNR and 0.005 LPIPS.Source: Table 5 · p. 7, §4.2 · p. 8

4.3. Complete Refocusing and Efficiency

  • At 1536 × 1536, a complete refocus takes 2.50 s, 148.3× faster than GenRefocus; each further edit takes 0.47 s.Further edits reuse the restored image and its depth; in-focus regions measure 44.4 dB PSNR against the restored image on 3CReal, versus 29.0 dB for GenRefocus.Source: Table 6 · p. 8, §4.3 · p. 8

5. Conclusion

FlashReFocus runs diffusion once per photograph, then uses a lightweight renderer for edits in about half a second while preserving in-focus content. It achieves the best perceptual bokeh fidelity on EBB! Val176 and unseen-camera 3CReal, with depth-related limits on blur placement and occlusion.

5. Conclusion

  • FlashReFocus achieves the best perceptual bokeh fidelity on EBB! Val176 and unseen-camera 3CReal, with each further edit taking about half a second.The renderer uses an explicit defocus map and leaves in-focus content unchanged.Source: §5 · p. 8
  • Diffusion runs once per photograph to restore an all-in-focus image, which a lightweight renderer reuses for subsequent edits.The renderer uses an explicit defocus map, which also supports aperture, tilt-shift, and two-plane control.Source: §5 · p. 8
  • Blur placement follows monocular depth, so depth errors misplace blur and occluded background cannot be revealed.Distilling Stage I and video refocusing are future work.Source: §5 · p. 8

Supplementary Material

The supplementary material details the defocus approximation, restoration implementation and data, and a decoder refiner that uses input guidance to recover detail lost in decoding.

A. Defocus Image Formation

  • The disk-blur approximation represents each blurred pixel using a disk kernel over its support, while omitting aperture shape, lens aberrations, occlusion, and camera processing.The text notes that equal depth offsets on opposite sides of the focal plane can yield different blur sizes because of inverse-depth dependence.Source: A. Defocus Image Formation · p. 13

B.1. Conditioning and Data Preparation

  • Stage I uses a deterministic one-step diffusion setup, and its mixed deblurring data combine real lens blur, refocused light-field pairs, and spatially varying synthetic blur.The synthetic pairs use disk, Gaussian, or motion kernels blended along a random gradient map; the 16,429 training samples also include filtered RealBokeh pairs and DPDD and LFDOF data.Source: §B.1 · p. 13, Figure B.1 · p. 14

B.2. High-Frequency Decoder Guidance

  • The decoder refiner fuses multi-scale input features, a high-frequency map, and blurred-image structure features into decoder blocks to refine the Stage-I latent.It is trained with the decoder while Stage I stays frozen; its input guidance uses detail from in-focus edges and text without copying defocused content.Source: §B.2 · p. 14
  • The refiner’s loss compares structure after an affine color fit, and the refiner is optional, trading perceptual sharpness for fidelity.The color fit is per image and treats its fitted parameters as constants during backpropagation.Source: §B.2 · p. 14

C.1. Defocus-Bias Attention

Defocus-bias attention steers renderer features toward tokens at similar defocus, helping preserve depth boundaries and in-focus regions. The appendix also reports improved depth estimates from restored images and describes synthetic multi-focal pre-training followed by real-data adaptation.

C.1. Defocus-Bias Attention

  • A learned head-wise bias based on pairwise defocus differences penalizes attention across different defocus levels, with its scale conditioned on global blur strength.The defocus embedding uses fixed logarithmic features, and the axial attention operates along rows and columns.Source: §C.1 · pp. 14–15
  • 0.64 versus 0.40 of attention falls on tokens with defocus within 0.1 of the query when the bias is used rather than removed.The comparison averages queries over 89 3CReal images; the figure visualizes attention-weight ratios with and without the bias.Source: §C.1 · p. 15, Figure C.1 · p. 16
  • The renderer’s residual leaves regions with defocus magnitude below 0.05 nearly unchanged, while changes elsewhere grow with defocus magnitude and blur strength.The preservation figure visualizes pixelwise change for several blur strengths and focal settings.Source: §C.1 · p. 15
  • On 3CReal, mean absolute depth difference to the deep-depth-of-field reference falls from 0.048 on the input to 0.027 on the restored image, lower for 74 of 89 views.Blurred backgrounds lose thin structures needed by monocular depth estimation.Source: §C.1 · p. 16

C.2. Synthetic Multi-Focal Supervision

  • Because real pairs provide only one focal setting per scene, renderer training first samples focal depths and blur strengths for synthetic Hypersim targets, then adapts on RealBokeh pairs.Pre-training uses 59,543 RGB-D frames from 367 Hypersim scenes; adaptation uses 20,554 pairs from 3,960 RealBokeh training scenes.Source: §C.2 · p. 16

D.1. 3CReal Benchmark

3CReal provides paired deep- and shallow-depth-of-field captures across three camera and lens systems unseen during training, addressing the single-camera scope of existing real-bokeh benchmarks. The appendix also describes evaluation data and notes residual alignment and capture differences that affect full-reference scores.

D.1. 3CReal Benchmark

  • 3CReal contains 89 paired captures from three camera and lens systems unseen during training, designed to test transfer beyond single-camera real-bokeh benchmarks.Scenes were captured at small and large apertures from a tripod, with automatic ISO and shutter speed used to reduce exposure differences.Source: §D.1 · pp. 16–17, Table D.1 · p. 17, Table D.2 · p. 17
  • The 3CReal scenes cover indoor and outdoor layouts, specular highlights, thin structures, and depth boundaries, and none is used for training.The pairs are aligned and cropped, but residual exposure differences, vignetting, moving objects, and aperture-dependent occlusion remain, so full-reference scores should be considered alongside visual comparisons.Source: §D.1 · p. 17

D.2. Other Evaluation Data

  • EBB! evaluation uses a filtered validation split of 176 pairs, while RealBokeh evaluation uses 219 validation scenes and measures in-domain rather than cross-camera performance.EBB! pairs may retain viewpoint, exposure, border, or pixel misalignment; RealBokeh shares its camera and lens with renderer adaptation data.Source: §D.2 · pp. 17–18
  • MODEST supplies 17 aligned aperture series from six scenes for continuous-aperture evaluation, while unpaired Unsplash and personal captures are used only for qualitative comparisons.The MODEST series pair an f/22 input with f/16, f/9, f/5, and f/2.8 targets.Source: §D.2 · p. 18

E. Experiment Settings

The experiments evaluate deblurring and bokeh rendering with shared inputs and method-specific inference settings. They also measure quality, controllability, and computational cost, including the cost of subsequent focal edits.

Bokeh rendering

  • Focal points are shared across bokeh methods, but each method is evaluated at its own best strength from a fixed search because strength scales differ.The search assumes SSIM is unimodal; deblurring outputs are evaluated at reference resolution unless otherwise specified.Source: E. Experiment Settings · p. 18, Table E.1 · p. 19

3CReal

  • 3CReal contains 89 aligned aperture pairs captured with three camera-and-lens systems from a fixed tripod.The collection procedure aligns, crops, and inspects the pairs; each shown pair differs only in aperture.Source: Figure D.1 · p. 19, Figure D.2 · p. 20

End-to-end refocusing

  • End-to-end qualitative comparisons resize inputs to a 768-pixel shorter side, while square-input DiffCamera receives reflection-padded rectangular images.Padding is removed after inference.Source: E. Experiment Settings · p. 19

Efficiency

  • Efficiency is measured at 1536 × 1536 on one NVIDIA RTX PRO 6000 GPU, with diffusion costs including denoising and VAE encoding and decoding.For a new focus, FlashReFocus reuses depth estimated once from the restored image; memory reports the joint peak of both stages.Source: E. Experiment Settings · p. 19

Metrics

  • Evaluation uses PSNR, SSIM, LPIPS, and DISTS for both stages, with additional rendering metrics and blur-strength controllability measured by LVCorr.For LVCorr, methods render ten strengths over their own ranges; GenRefocus and AnyBokeh are assessed on the first 20 scenes of each set.Source: E. Experiment Settings · p. 19

F. Additional Deblurring Results

Additional deblurring results examine texture recovery, reconstruction failures, and the effects of detail weighting and decoder refinement. They show that added detail can improve textured regions, but its weight and the input structure matter.

Qualitative comparisons

  • Beyond DPDD and RealDOF, FlashReFocus deblurs 3CReal images captured with real lenses unseen by Stage I.The comparisons use shallow-depth-of-field photographs alongside deep-depth-of-field captures of the same views.Source: F. Additional Deblurring Results · p. 21

Failure cases

  • The diffusion prior can add unsupported high-frequency patterns in textureless regions, while severely defocused foregrounds may become over-smoothed or ghosted.The detail-weight study reports that hallucination grows as ω increases beyond the selected range.Source: F. Additional Deblurring Results · p. 21

Detail weight

  • The detail weight balances fidelity and perceptual metrics: PSNR peaks at ω = 0.5–0.75, while LPIPS, DISTS, and FID improve through ω = 1.25–1.5.Beyond that range all metrics degrade as the extrapolated detail branch hallucinates texture; the selected ω = 1.25 has the lowest rank sum.Source: F. Additional Deblurring Results · p. 21
  • 32.5% of the detail-branch difference falls on the sharp target’s most-textured 20% of pixels, where error decreases from 18.09/255 to 17.66/255.Across 76 DPDD test images, the difference averages 4.3/255; error on the smoothest half stays at 7.3/255.Source: F. Additional Deblurring Results · p. 21

Decoder refiner

  • Decoder-side high-frequency injection recovers text that is relatively clear in the input but distorted by latent encoding and decoding.Severely blurred characters remain limited because their structure cannot be inferred reliably.Source: F. Additional Deblurring Results · p. 21

G. Additional Bokeh Results

Additional bokeh results test real-aperture behavior, cross-depth-model use, and rendering variants. They also describe the scope of the RealBokeh validation comparison and qualitative examples across real captures.

RealBokeh validation

  • On RealBokeh validation, FlashReFocus follows Bokehlicious on five full-reference metrics and has the second-best MUSIQ; on the ten GenRefocus pairs, it beats GenRefocus on every metric.For those ten pairs, PSNR is 25.46 vs. 23.38 dB and MUSIQ is 47.61 vs. 38.79; the validation shares cameras and lenses with training.Source: G. Additional Bokeh Results · p. 21

Real apertures

  • On MODEST, FlashReFocus’s selected strength rises with aperture in 16 of 17 series and has the highest rank correlation to 1/N (ρ = 0.985).It is on par with BokehMe (0.979) and above MagicBokeh (0.903); the strength is monotonic but not linear in 1/N, so each lens needs calibration.Source: G. Additional Bokeh Results · p. 21

Qualitative comparisons

  • Qualitative comparisons show subject preservation, highlight appearance, and boundary transitions across EBB!, 3CReal, and RealBokeh captures.The examples also place renders alongside real captures from two MODEST aperture series.Source: G. Additional Bokeh Results · p. 21

Ablation

  • Defocus-bias attention throughout is best on five metrics, while mixed relative-position and defocus-bias attention ties it on MS-SSIM and is marginally higher on CLIP-I.The CLIP-I values are 0.9811 vs. 0.9807, respectively.Source: G. Additional Bokeh Results · p. 21

Depth source

  • With the renderer unchanged, depth-model size spanning a 38× range changes PSNR by at most 0.21 dB and LPIPS by at most 0.009.Depth Pro improves LPIPS and DISTS on both benchmarks; using it takes about 1.6× as long as DA-V2-L for the first edit.Source: G. Additional Bokeh Results · pp. 21–22, Table G.4 · p. 22

H. Refocusing and Editing

FlashReFocus supports complete refocusing and interactive edits, with one all-in-focus restoration reused for changes in focus and blur. Its evaluations report strong full-reference results on LFDOF, improved in-focus preservation over GenRefocus, and controls including two focal planes.

Complete refocusing on LFDOF

  • FlashReFocus is best on all full-reference metrics in complete refocusing on 50 LFDOF test scenes, although Stage I training and refiner selection used related data.Each input and target is a different focal slice of the same light field, and blur strength is searched per image for highest SSIM.Source: H. Refocusing and Editing · p. 23, Table H.1 · p. 23

Refocusing examples

  • FlashReFocus recovers newly selected focal regions that were blurred in the input, but can recover a strongly blurred face less sharply than methods synthesizing facial detail.Examples include an out-of-focus cliff and foreground rock, and a face behind a globe.Source: H. Refocusing and Editing · pp. 23–24

Consistency across focal edits

  • 44.44 dB against 28.96 dB on 3CReal in setting (i): FlashReFocus preserves in-focus regions better than GenRefocus while matching its blur.Across consecutive edits, pixels whose blur should not change vary 2.7–3.7 times more with GenRefocus; each edit takes 0.10 s against 77 s.Source: H. Refocusing and Editing · p. 24

Restoration detail

  • The decoder refiner reinjects input high-frequency structure to recover legible text distorted by plain VAE decoding, but severely blurred characters remain hard to recover.The detail-branch comparison also visualizes its output difference from the pixel branch against the reference.Source: Figure F.4 · p. 24, Table F.2 · p. 24, Figure F.5 · p. 25

Controls and aperture

  • Increasing blur strength acts like opening the aperture, and editing depth to a vertical ramp produces tilt-shift effects.On MODEST, selected strength grows with aperture, though at f/2.8 the real lens blurs the near foreground more strongly than FlashReFocus in one series.Source: Figure G.2 · p. 26, Table G.2 · p. 26

Two focal planes

  • Editing the depth map keeps two selected focal planes sharp at once without retraining, while regions far from both remain blurred.Across three scenes, the two planes retain 66–84% of sharpness, compared with 9–21% for the non-focused plane when focusing on only one.Source: H. Refocusing and Editing · p. 27

I. Limitations and Broader Impact

The paper identifies limits in depth estimation, occlusion recovery, lens-specific rendering, restoration, cost, and frame-by-frame video use. It also cautions that generated restored detail should not be treated as photographic evidence.

Depth

  • Depth-estimation errors become blur errors, especially for thin structures, glass, or reflections; benchmark PSNR changes do not bound errors on individual images.Replacing DA-V2-L with Depth Pro or smaller DA-V2 models changes benchmark PSNR by at most 0.11 dB.Source: §I · p. 27

Occlusion

  • A single restored image contains no occluded background, so the renderer cannot reveal background hidden behind a foreground edge.A real large-aperture lens can see slightly around that edge.Source: §I · p. 27

Lens appearance

  • The renderer has no aperture-shape input or control for lens-specific effects such as cat-eye highlights, and strengths above 1 are extrapolated.Source: §I · p. 28

Restoration

  • Stage I may add unsupported texture or leave heavily blurred regions soft, including faces that generative refocusing methods render more sharply.Source: §I · p. 28

Cost

  • At 1536×1536, the first refocus takes 2.50 s, mostly for restoration, and restoration runs on a server-class GPU.Further edits use only the 1.3M-parameter renderer and take 0.47 s.Source: §I · p. 28

Video

  • Frame-by-frame application estimates restoration and depth independently, with no temporal constraint, so detail and blur can flicker.Video refocusing would need temporally consistent restoration and depth.Source: §I · p. 28

Broader impact

  • Because restored detail is partly generated by a diffusion prior, a refocused image should not be treated as a record of what the camera captured.Source: §I · p. 28

Related papers

Loading 2610.23297v1…