PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dual-Camera Smooth Zoom via Decoupled Geometric-Photometric Data Synthesis

Junjie Fan, Zhilu Zhang, Renlong Wu, Dazhi Zhang, Wangmeng Zuo

arXiv:2610.34854v1cs.CV

TL;DR

Smartphone dual-camera zoom can produce abrupt perspective and color discontinuities, while standard VFI models face a domain gap and continuous real-world ground truth is impractical to capture. ZoomGS v2 synthesizes DCSZ training data by decoupling camera transformations from scene modeling, and ZoomFI is designed for this task. Experiments report improvements from synthesized data and state-of-the-art performance for ZoomFI on evaluated DCSZ benchmarks.

  • Problem

    Dual-camera zoom transitions can cause perspective and color discontinuities, while standard VFI faces a domain gap and continuous real-world ground-truth capture is impractical.

    Source: §I · p. 1, A. Problem DCSZ Setup · p. 3

  • Method

    ZoomGS v2 decouples geometric and color transformations from 3D scene modeling to synthesize scalable DCSZ data, while ZoomFI uses complementary bidirectional warping for dual-camera information fusion.

    Source: §I · p. 2, B. Motivation of Methods · p. 3

  • Results

    Models fine-tuned on ZoomGS v2 data improve over baselines and ZoomGS, and ZoomFI outperforms state-of-the-art VFI models on evaluated DCSZ benchmarks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The experiments support using ZoomGS v2 data to improve VFI performance in real-world DCSZ scenarios.

    Source: §I · p. 2

Abstract

from arXiv · show

When zooming between dual cameras on smartphones, prominent discontinuities in both geometric perspective and color rendition inevitably degrade the visual experience of users. To address this issue, we introduce the dual-camera smooth zoom (DCSZ) task, aiming to synthesize continuous intermediate frames for fluid zooming. However, directly applying existing video frame interpolation (VFI) models to this task is severely hindered by substantial domain gaps and the practical intractability of capturing real-world continuous ground-truth. To this end, we propose a novel data synthesis framework to generate high-fidelity DCSZ training sequences, named ZoomGS v2. It explicitly decouples geometric and color transformations from 3D scene representation modeling, thereby enabling extensive data generation from easily accessible single-camera 3D scenes. Furthermore, we develop ZoomFI, a specialized VFI network tailored to the DCSZ task, which leverages complementary bidirectional warping for comprehensive dual-camera information fusion. Extensive experiments on test datasets captured using two mobile phones demonstrate that training with the constructed DCSZ data significantly improves the performance of various VFI models. Moreover, ZoomFI achieves state-of-the-art results in both quantitative metrics and visual quality. The datasets, code, and pretrained models are available at https://hitjunjief.github.io/DCSZ-v2

I. INTRODUCTION

Smartphone dual-camera zoom can produce abrupt perspective and color shifts when switching from the ultra-wide to wide camera, while existing VFI models and real-world capture do not readily provide a solution. ZoomGS v2 synthesizes DCSZ training sequences from single-camera scenes, and experiments report gains from this data and from ZoomFI.

I. INTRODUCTION

  • At the UW-to-W transition, cropping and scaling the UW view gives way to an abrupt lens switch that can cause perspective and color discontinuities.The cited example switches from ×0.9 to ×1.0.Source: §I · p. 1, Figure 1 · p. 2
  • DCSZ aims to synthesize continuous intermediate frames for fluid zoom previews, but standard VFI models face a domain gap and dense real-world ground-truth capture is physically impractical.The gap includes larger viewpoint displacements and more severe color variations than standard VFI benchmarks.Source: §I · p. 1
  • Earlier ZoomGS lacked intermediate-view supervision, causing temporal instability, and required dual-camera captures for each scene, limiting dataset scalability.Source: §I · p. 1
  • ZoomGS v2 decouples geometry and color transformations from scene modeling, enabling DCSZ data generation using 3D scenes reconstructed from single-camera multi-view data.ZoomFI is tailored to DCSZ and uses complementary bidirectional warping to fuse information from both cameras.Source: §I · p. 2
  • Models fine-tuned on ZoomGS v2 data improve quantitatively and qualitatively over baseline models and ZoomGS, while ZoomFI outperforms state-of-the-art VFI models on evaluated DCSZ benchmarks.Evaluation uses real-world dual-camera image pairs from Xiaomi and Huawei phones.Source: §I · p. 2

II. RELATED WORK

The related work covers frame interpolation and 3D reconstruction as foundations for DCSZ. It identifies domain and motion challenges for existing VFI, and rendering-quality and speed constraints for NeRF-based reconstruction.

A. Frame Interpolation

  • Standard VFI methods face a domain gap on DCSZ data, and larger inter-frame motion makes accurate motion modeling difficult.The review describes kernel-, hallucination-, flow-, and state-space-model-based approaches, among others.Source: A. Frame Interpolation · p. 2

B. 3D Reconstruction

  • NeRF-based methods struggle to achieve visually realistic rendering and real-time speed simultaneously, while 3DGS uses explicit scene representation and differentiable rasterization.This work further specializes 3DGS for each virtual camera's imaging characteristics to generate DCSZ data.Source: B. 3D Reconstruction · p. 3

III. PROPOSED PROBLEM AND METHOD

DCSZ frames are intermediate images synthesized between ultra-wide and wide-camera inputs. The setup proposes virtual cameras to render synthetic training targets, avoiding the need for physically capturing continuous ground truth.

A. Problem DCSZ Setup

  • DCSZ uses a VFI model to synthesize N intermediate images between the ultra-wide and wide-camera images.Source: A. Problem DCSZ Setup · p. 3
  • Existing pretrained VFI models often underperform because standard benchmarks differ substantially from dual-camera zoom, while continuous real-world ground truth is impractical to capture.Capturing it would require a dense array of sensors between the dual cameras.Source: A. Problem DCSZ Setup · p. 3
  • The proposed setup constructs virtual cameras along the transition between the two inputs and renders synthetic intermediate views for VFI training.Source: A. Problem DCSZ Setup · p. 3
  • Virtual-camera rendering combines camera intrinsics and extrinsics for perspective with ISP parameters for color, interpolated smoothly from calibrated endpoint cameras.Source: A. Problem DCSZ Setup · p. 3

B. Motivation of Methods

ZoomGS v2 separates scene reconstruction from device-specific camera transformations, enabling DCSZ data synthesis from single-camera scenes. It models geometric disparity through calibrated camera parameters and derives the W-camera extrinsics from an UW view.

B. Motivation of Methods

  • ZoomGS v2 calibrates geometric and color gaps from a small dual-camera dataset, then applies them to large-scale single-camera scenes to synthesize DCSZ data.Decoupling scene representation learning from cross-camera transformation modeling avoids ZoomGS’s lack of intermediate-view supervision.Source: B. Motivation of Methods · p. 3

C. Geometric Transformation Calibration of Dual-camera

  • ZoomGS v2 models the UW-to-W intrinsic relationship using a fixed effective focal-length ratio, Kw = Kuw/s.The paper treats s as a constant for a smartphone.Source: C. Geometric Transformation… · p. 4
  • Because the dual-camera sensors are rigidly fixed to the same planar substrate, their relative orientation is approximated as identical, Rw ≈ Ruw.The measured mean absolute error between rotation parameters is less than 3.5×10−3.Source: C. Geometric Transformation… · p. 5
  • ZoomGS v2 fits a multivariate Gaussian to measured relative displacement vectors, accounting for fluctuations from camera shake, sensor thermal variation, and SfM estimation.A displacement vector is sampled from this distribution to derive the W-camera translation from a single-camera UW view.Source: C. Geometric Transformation… · p. 5
  • Under the shared-orientation approximation, the W-camera extrinsics are systematically derived from the UW-camera extrinsics using the sampled relative displacement.The geometric formulation is summarized in Fig. 3; its camera-parameter calibration uses UW-W pairs.Source: Figure 3 · p. 4, C. Geometric Transformation… · p. 5

D. Color Transformation Prediction of Dual-camera

ZoomGS v2 models cross-camera color differences through an ISP-inspired transformation from the UW image to the W image. A CNN predicts transformation parameters from geometrically aligned UW-W pairs.

D. Color Transformation Prediction of Dual-camera

  • The color transformation maps the UW RGB image through inverse ISP to linear RAW, transforms the RAW representation, then applies forward ISP to produce the W RGB image.The cross-sensor spectral discrepancy is represented by a linear transformation with a weight matrix and bias.Source: D. Color Transformation Prediction of… · p. 5
  • The modeled ISP stages are color correction, auto white balance, and channel-wise gamma correction.Color correction uses full matrices, while auto white balance is represented by diagonal gain matrices.Source: D. Color Transformation Prediction of… · p. 5
  • A CNN predicts color-transformation parameters from geometrically aligned UW-W pairs, supervised by a ΔE loss in CIELAB space.Pairs are aligned by field-of-view cropping followed by optical-flow-based warping.Source: D. Color Transformation Prediction of… · p. 5
  • At inference, the estimated photometric parameters transform the UW view’s color profile into that of the W camera.The intermediate linear operations are unified as a compact affine mapping.Source: D. Color Transformation Prediction of… · pp. 5–6

E. Data Generation from ZoomGS v2

ZoomGS v2 generates intermediate dual-camera views by interpolating camera geometry and photometric transformations over reconstructed 3D scenes. The intermediate geometry is rendered first, then the interpolated photometric transformation is applied.

E. Data Generation from ZoomGS v2

  • The target W-camera translation is determined using a sampled ΔV drawn from a fitted multivariate Gaussian distribution.The sampled original capture defines the reference UW view and its intrinsic and extrinsic matrices.Source: E. Data Generation from ZoomGS v2 · p. 6
  • Intermediate camera intrinsics are linearly interpolated between the UW and W camera parameters.The reference view uses K_uw, while K_w is obtained according to Sec. III-C.Source: E. Data Generation from ZoomGS v2 · p. 6
  • Intermediate extrinsic parameters are computed by geodesic interpolation in the Lie algebra between the endpoint camera poses.The progression factor τ_i is normalized over the virtual camera views, and exp and log map between SE(3) and se(3).Source: E. Data Generation from ZoomGS v2 · p. 6
  • Intermediate geometric views are rendered from the reconstructed 3DGS representation using interpolated camera parameters, then photometric transformations are modulated by progression and applied to generate continuous zoom sequences.The photometric transformations are predicted by a CNN.Source: E. Data Generation from ZoomGS v2 · p. 6

F. Pipeline and Training of ZoomFI

ZoomFI combines forward and backward warping to fuse dual-camera information under the large motion and scale differences of DCSZ. It aligns both input images to the intermediate state and fuses the aligned features and flows to synthesize a frame.

F. Pipeline and Training of ZoomFI

  • The DCSZ task's large motion and scale disparity can cause sampling ambiguities and peripheral distortions when backward warping the broad-FoV UW view toward the localized W view.ZoomFI couples forward warping, which preserves peripheral content under scale expansion, with backward warping and a specialized flow network for the scale discrepancy.Source: F. Pipeline and Training of ZoomFI · p. 6
  • Intermediate backward flows are approximated via forward warping, while intermediate forward flows are derived through linear trajectory interpolation to provide coarse motion initialization.The bidirectional input flows are computed by a pretrained optical-flow network at 4× downsampled resolution; forward warping splats tensors along a flow field.Source: F. Pipeline and Training of ZoomFI · pp. 6–7
  • ZoomFI concatenates four warped frames and their intermediate flow fields, then feeds them to a multi-scale fusion module to synthesize the intermediate frame.The fusion module uses initial convolutional feature extraction layers followed by a multi-scale U-Net architecture.Source: F. Pipeline and Training of ZoomFI · p. 7
  • ZoomFI-Lite estimates optical flow at an 8× downsampling factor to accelerate inference and reduce FLOPs relative to standard ZoomFI.The variant is introduced to facilitate real-time deployment on resource-constrained mobile hardware.Source: F. Pipeline and Training of ZoomFI · p. 7
  • ZoomFI is optimized with a joint loss combining multi-scale Laplacian-pyramid reconstruction and perceptual losses.The perceptual features come from a pretrained VGG-19 network, and λ_per is set to 1.0.Source: F. Pipeline and Training of ZoomFI · p. 7

IV. EXPERIMENTS

The experiments evaluate two dual-camera zoom settings on Xiaomi and Huawei phones. The ×0.85 setting is formed by center-cropping the UW image before the transition to the W view.

IV. EXPERIMENTS

  • Experiments assess generalization across Xiaomi and Huawei phones using zoom transitions between ×0.6 and ×1.0 and between ×0.85 and ×1.0.The datasets for both devices are constructed in the ×0.6 ↔ ×1.0 setting.Source: §IV · p. 7
  • For ×0.6 ↔ ×1.0, continuous geometric and color transformations are applied directly from the UW view to the W view.The UW and W zoom factors are ×0.6 and ×1.0, respectively.Source: §IV · p. 7
  • For ×0.85 ↔ ×1.0, the UW image is center-cropped to approximately ×0.85 before continuous geometric and color transformations lead to the W view.The crop scale is 0.6/0.85 for the UW image and 1 for the W image; intermediate training crops increase linearly to 1.Source: §IV · p. 7

A. Experimental Settings

The evaluation uses synthetic training sequences and real dual-camera test pairs from Xiaomi and Huawei phones. Because suitable reference sequences are impractical to obtain, test performance is assessed with no-reference quality metrics and a color-continuity protocol.

A. Experimental Settings

  • ZoomGS v2 produces 8,704 Xiaomi and 7,686 Huawei sequences after filtering synthetic sequences with poor 3DGS reconstruction quality or severe rendering artifacts.The sequences come from 4,000 DL3DV-10K scenes, with four sampled dual-camera parameter sets per scene and device.Source: A. Experimental Settings · p. 7
  • The test sets contain 100 Xiaomi scenes and 50 Huawei scenes, each with a UW image and corresponding W image, and use no-reference metrics rather than rendered ground-truth references.Rendered sequences may have reconstruction artifacts or camera poses that do not match real dual-camera systems, so they may not faithfully reflect real-world zoom performance.Source: A. Experimental Settings · p. 7
  • ZoomFI is pretrained for 200k iterations on ZoomGS v2 data without color transformation, then fine-tuned for 100k iterations on color-transformed data.The 3DGS representations are optimized for 50k iterations.Source: A. Experimental Settings · p. 7
  • Tables I and II compare VFI models on Xiaomi and Huawei test datasets, distinguishing fine-tuning on ZoomGS data from ZoomGS v2 data.The captions note that unfine-tuned models can appear better on color-continuity metrics because excessive blur suppresses frame-to-frame color variation.Source: Table I · p. 8, Table II · p. 8
  • Evaluation uses MUSIQ, MANIQA, and TOPIQ for image quality, DOVER and FasterVQA for video quality, and Smooth-ΔE measures for color continuity.Smooth-ΔE_mean and Smooth-ΔE_max are the mean and maximum absolute differences between a fitted cubic color-transition curve and measured values across frames.Source: A. Experimental Settings · p. 8

B. Comparison with State-of-the-Art Methods

Fine-tuning with ZoomGS v2 data improves VFI results on real-world DCSZ benchmarks, and ZoomFI performs best on most evaluated metrics. ZoomFI also offers a faster inference-time trade-off than some compared models, while ZoomFI-Lite has the lowest reported computational cost.

B. Comparison with State-of-the-Art Methods

  • Fine-tuning on ZoomGS v2 data outperforms fine-tuning on ZoomGS data and non-fine-tuned models on most evaluation metrics, with notable perceptual-quality gains.Visual comparisons on both devices show more photorealistic detail and fewer artifacts with ZoomGS v2 training.Source: B. Comparison with State-of-the-Art… · pp. 8–9, Figure 7 · p. 9, Figure 8 · p. 9
  • ZoomFI achieves the best performance on most evaluation metrics among the compared methods on the Xiaomi and Huawei DCSZ benchmarks.Visual comparisons also show ZoomFI preserving more photorealistic detail and exhibiting fewer artifacts than non-fine-tuned models.Source: B. Comparison with State-of-the-Art… · p. 9
  • ZoomFI has shorter inference time than VFIMamba and AMT, while ZoomFI-Lite has the lowest inference time and FLOPs among compared methods.At 1632 × 1216 resolution, ZoomFI-Lite maintains competitive generation quality relative to other methods except ZoomFI; both models are described as promising mobile-deployment candidates.Source: B. Comparison with State-of-the-Art… · p. 9, Table III · p. 9

C. Ablation Study

Ablations show that Gaussian translation-offset sampling, color transformation, more ZoomGS v2 training scenes, and bidirectional warping each improve aspects of DCSZ interpolation. The full data-scale condition and the combined warping directions yield the strongest reported outcomes within their respective studies.

Effect of Sampling Strategy of ∆V in ZoomGS v2

  • Gaussian sampling improves all image and video quality metrics and one color-continuity metric over uniform sampling, with fewer misalignment artifacts.The paper attributes the improvement to Gaussian sampling better matching the empirical distribution of real dual-camera translations.Source: C. Ablation Study · p. 10, Figure 11 · p. 10, Table IV · p. 10

Effect of Color Transformation in ZoomGS v2

  • Adding color transformation improves all video-quality and color-continuity metrics, and the full framework beats variants omitting gamma correction or linear mapping on all seven metrics.Color-transformed sequences show smoother ∆E variations, while transformed UW images are more color-consistent with W images and exhibit less chromatic aberration.Source: C. Ablation Study · p. 11, Figure 12 · p. 11, Figure 13 · p. 11

Effect of Training Data Scale from ZoomGS v2

  • The VFI model achieves its best overall performance on the full ZoomGS v2 dataset, and performance improves as the synthetic training data increases.At 1.0% of ZoomGS v2 scenes, performance is comparable to or better than training on ZoomGS data, despite a similar scene count.Source: C. Ablation Study · p. 11, Table VI · p. 11

Effect of Bidirectional Warping in ZoomFI

  • Using both forward and backward flows consistently improves all evaluation metrics, with fewer blurring artifacts and structural distortions than either direction alone.The two directions have complementary sampling properties that can improve robustness in occluded and large-motion regions.Source: Figure 14 · p. 11, Table VII · p. 11, C. Ablation Study · p. 12

V. CONCLUSION

The conclusion presents DCSZ as a task for addressing abrupt dual-camera zoom transitions, using ZoomGS v2 to synthesize training data and ZoomFI to interpolate frames. Experiments on Xiaomi and Huawei devices support the approach, while more accurate camera and ISP modeling and efficient mobile deployment remain future work.

V. CONCLUSION

  • ZoomGS v2 decouples geometry and photometry to synthesize scalable DCSZ training data from single-camera datasets, and ZoomFI uses bidirectional warping to generate intermediate frames.Experiments on Xiaomi and Huawei test datasets demonstrate data-synthesis quality and verify extension to various mobile devices.Source: §V · p. 12

Related papers

Loading 2610.34854v1…