Source-linked AI summary
Learning to Refocus with Video Diffusion Models
SaiKiran Tedla, Zhoutong Zhang, Xuaner Zhang, Shumian Xin
TL;DR
Autofocus can miss the intended subject, and existing refocusing approaches may require specialized hardware or multiple inputs. This paper generates a focal stack from one defocused image using video diffusion models and reports perceptually superior refocusing for larger focus shifts, with robustness across challenging scenarios.
Problem
Autofocus can miss intended subjects, while post-capture refocusing often requires specialized hardware, multiple images, or depth maps.
Source: §1 · p. 1Method
Results
The method surpasses existing methods in perceptual quality under larger focal-position changes and demonstrates robustness across challenging scenarios.
Source: §6 · p. 9Takeaways & Limitations
The approach supports realistic refocusing, including defocus deblurring and natural defocus effects.
Source: §1 · p. 2The method does not generalize well to extreme defocus or bokeh absent from its iPhone training data, and video-frame use lacks enforced temporal consistency.
Source: §5.6 · p. 9
Abstract
from arXiv · showhide
Focus is a cornerstone of photography, yet autofocus systems often fail to capture the intended subject, and users frequently wish to adjust focus after capture. We introduce a novel method for realistic post-capture refocusing using video diffusion models. From a single defocused image, our approach generates a perceptually accurate focal stack, represented as a video sequence, enabling interactive refocusing and unlocking a range of downstream applications. We release a large-scale focal stack dataset acquired under diverse real-world smartphone conditions to support this work and future research. Our method consistently outperforms existing approaches in both perceptual quality and robustness across challenging scenarios, paving the way for more advanced focus-editing capabilities in everyday photography. Code and data are available at www.learn2refocus.github.io
1 Introduction
Autofocus can miss intended subjects, while existing post-capture refocusing often requires specialized hardware, multiple images, or depth maps. This work uses video diffusion models to generate realistic focal stacks from one defocused image and introduces a real-world smartphone dataset.
1 Introduction
- Earlier approaches often require specialized hardware, multiple images, or depth maps, while all-in-focus outputs do not reproduce natural optical defocus effects.These resource and output limitations motivate post-capture refocusing for aesthetic or corrective use.Source: §1 · p. 1
- Video diffusion models reconstruct a temporally coherent focal stack from a single defocused image, enabling interactive refocusing.The approach treats the task as multi-frame generation and makes a minimal modification to classifier-free guidance.Source: §1 · p. 1
- The method consistently produces perceptually realistic focal stacks across challenging scenarios.It is presented as resilient across challenging capture conditions and as supporting realistic refocusing, defocus deblurring, and natural defocus effects.Source: §1 · p. 2
- The introduced dataset contains 1,637 focal-stack samples captured on an iPhone 12 under varied real-world conditions.The paper presents it as a practical benchmark covering diverse scenes and lighting scenarios.Source: §1 · p. 2
2 Related work
Prior work addresses focus manipulation through synthetic-data refocusing, specialized capture, depth-aware editing, and diffusion-based generation. This paper positions video diffusion priors and real smartphone focal stacks as a route to generating complete stacks from one defocused image.
2.1 Focus and depth-of-field manipulation
- RefocusGAN predicts an all-in-focus image before generating defocused views and iteratively builds a stack, whereas this method generates a full stack in one pass from real smartphone captures.RefocusGAN is trained exclusively on synthetic data.Source: §2.1 · p. 2
- Light-field refocusing needs specialized hardware, while other approaches use multiple captures, sensor-specific data, or an all-in-focus image and depth map.The paper also contrasts with controllable image-generation approaches that may alter scene identity during refocusing.Source: §2.1 · p. 2
- Most prior depth-of-field datasets are synthetic or light-field based; the cited largest public real-world dataset contains 510 focal stacks, compared with this work’s 1,637 iPhone 12 stacks.The new dataset is presented as covering diverse real-world scenarios.Source: §2.1 · p. 2
2.2 Diffusion models
- Diffusion-based deblurring usually produces an all-in-focus image, so focal-stack prediction requires added conditioning or modifications.Diffusion models’ generative flexibility is also used for image editing and restoration.Source: §2.2 · p. 2
- Unlike video interpolation between given frames, refocusing extrapolates from one defocused image to an entire focal stack, using video models’ implicit priors.The paper relates its use of implicit priors to generative camera-control work.Source: §2.2 · pp. 2–3
3 Large focal stack dataset
The paper introduces a real-world smartphone focal-stack dataset and processes its captures to align focus positions and provide evaluation references. The resulting benchmark represents each scene with aligned focal-stack images and an all-in-focus image.
3 Large focal stack dataset
- The dataset is presented as the largest publicly available collection of real-world smartphone focal stacks, with 1,637 scenes.Each scene was captured using a custom rig of five iPhones, and the paper uses the central camera’s focal stack for this work.Source: §3 · p. 3, §3.1 · p. 3
3.1 Data collection
- The 1,637 scenes include 1,474 training and 163 test stacks, spanning controlled lightbox and in-the-wild settings.Captures are RAW 4032×3024 images with consistent exposure and white balance; the scenes include varied environments, lighting, and slight background motion.Source: §3.1 · p. 3
3.2 Data processing
- Processing corrects focal breathing through distortion correction and scale alignment; without the processing, the model and baselines fail to attain good refocusing results.The alignment transforms stack images to match the first frame, which has the smallest field of view.Source: §3.2 · pp. 3–4, Figure 4 · p. 3
- Each processed scene contains nine aligned focal-stack images and one all-in-focus reference image.The all-in-focus image is generated with Helicon Focus in depth mode for evaluating defocus-deblurring methods.Source: §3.2 · p. 4
4 Method
The method treats a focal stack as a video and uses a video diffusion model to reconstruct it jointly from one defocused image. Position-dependent conditioning adapts classifier-free guidance so the input focal position can vary across the stack.
4 Method
- The video diffusion model jointly reconstructs a perceptually realistic focal stack from one defocused image in a single sampling process.Unlike RefocusGAN, which refocuses one frame at a time, the method modifies classifier-free guidance for single-shot focal-stack generation.Source: §4 · p. 4
4.1 Preliminaries
- Latent diffusion reduces computation by operating in a compressed latent space, and Stable Video Diffusion conditions on the first frame by replicating its latent across frames.Classifier-free guidance combines conditional and unconditional noise estimates; the unconditioned estimate uses zeroed conditioning.Source: §4.1 · p. 4
4.2 Modified classifier-free guidance
- The model predicts all stack frames, including the selected input position, in one sampling pass, while training randomly samples focal positions across batches.The modified guidance contrasts predictions conditioned on the selected-position latent with predictions conditioned on zeros.Source: §4.2 · p. 4
- Replicating a randomly chosen focal-position latent across the stack creates ambiguity, so the method places that latent only at its matching position and zeros the others.This position-dependent conditioning lets the UNet reconstruct stacks aligned with the input focal position, unlike interpolation methods that assume fixed start and end frames.Source: §4.2 · p. 4
5 Experiments
The experiments evaluate refocusing quality on the authors’ dataset and everyday images, test camera generalization and downstream editing, and discuss limitations.
5.1 Implementation details
- Baselines included reimplemented RefocusGAN, modified deblurring networks, and image-diffusion methods; all networks trained for 200,000 epochs on the same dataset splits.The method retrained only its UNet transformer blocks and used classifier-free guidance at test time.Source: §5.1 · p. 5
5.2 Results on our dataset
- At larger shifts from positions 1 or 9, the SVD-based method achieves superior perceptual quality; at position 5, deblurring baselines better preserve high-frequency details.The method also outperforms IP2P and Swintormer. The table evaluates positions 1, 5, and 9.Source: Table 1 · p. 7, §5.2 · p. 7
- The method remains robust to tested tone-mapping gamma settings (γ=0.5 and γ=2) and additive Gaussian input noise (σ=0.05); quantitative and qualitative results are reported in the supplementary material.The authors attribute a significant part of the PSNR drop to VAE latent-space compression and prioritize perceptual plausibility over exact pixel fidelity, accepting modest losses in reconstruction accuracy.Source: §5.2 · p. 7
- For shifts greater than four positions, users preferred our method 88.25% to NAFNet’s 11.75%; motion-profile-ID conditioning failed to produce refocused stacks.The user study used 20 scenes and 20 users; the ablation reproduced the input in every frame.Source: §5.2 · p. 7
5.3 Results on everyday photos
- On everyday iPhone images, the method handles faces and thin plants and reconstructs details such as bread char and glass caustics.When focal-position metadata was unavailable, outputs were sampled at each position and selected by manual inspection.Source: §5.3 · p. 8, Figure 7 · p. 8
5.4 Generalization to other cameras
- The method qualitatively generalizes to smartphones, film cameras, and DSLRs, reconstructing eye, hair, and pottery details while preserving original color profiles.The authors note that these cameras’ focal-position metadata does not align with iPhone positions.Source: §5.4 · p. 8, Figure 8 · p. 9
5.5 Downstream tasks
- Generated focal stacks support all-in-focus and selective depth-of-field editing, and refocusing can reduce slight foreground motion blur.The authors attribute reduced blur to the model learning to generate images without motion blur from its training data.Source: §5.5 · p. 8
5.6 Failure cases and limitations
- The iPhone training data limits performance on extreme defocus and exaggerated DSLR bokeh; applying the image-refocusing model to video frames does not enforce temporal consistency.The authors identify temporal artifacts as a consequence of the lack of enforced consistency.Source: §5.6 · p. 9
6 Conclusions
The method applies video diffusion models to reconstruct realistic focal stacks from single defocused images, outperforming existing methods in perceptual quality under larger focal changes and generalizing beyond training cameras. The paper also releases a smartphone focal-stack dataset and identifies challenges at small focal shifts and opportunities to broaden defocus coverage.
6 Conclusions
- The method surpasses existing methods in perceptual quality under larger focal position changes, remains robust across challenging scenarios, and generalizes to film cameras, DSLRs, and other smartphones.It modifies the classifier-free guidance mechanism.Source: §6 · p. 9
- The approach also proves effective for depth-of-field editing and motion blur reduction.Source: §6 · p. 9
- The released dataset contains 1,637 real smartphone focal stacks with RAW images, realistic lighting variations, and everyday environments.It is intended as a benchmark for refocusing methods.Source: §6 · p. 9
- Latent diffusion limits detail reconstruction at small focal position changes; large-aperture data and additional sensors may improve generalization across defocus properties.The authors also suggest pixel diffusion, a better-autoencoder latent model, or a hybrid for the detail limitation.Source: §6 · p. 9