PaperlayerView exact revision on arXiv

Source-linked AI summary

DuSRFlow: Dual-Lens Super-Resolution via Parallax-Aware Flow Estimation and Complementary Reference Warping

Fangpu Zhang, Huanjing Yue, Zifan Cui, Shaoxiong Tu, Qirui Yang, Jingyu Yang

arXiv:2610.14630v1cs.CV

TL;DR

Dual-lens super-resolution uses a telephoto reference to enhance a wide-angle LR image, but large parallax and cross-camera differences make alignment unreliable. DuSRFlow combines dedicated correspondence learning with reliability-aware reference aggregation and constructs DuSR-RealV2; across four benchmarks, it outperforms 11 state-of-the-art methods and generalizes well to challenging real-world scenarios.

  • Problem

    Large parallax, occlusion, and cross-camera imaging differences undermine reliable reference alignment, while monocular flow assumptions of photometric consistency and smooth motion do not fit dual-lens images.

    Source: §I · p. 1

  • Method

    DuSRFlow trains a dedicated dual-lens flow estimator with real–synthetic composites and coordinates patch-based and kernel-free-guided deformable warping according to local reference reliability.

    Source: §I · pp. 1–2

  • Results

    DuSRFlow consistently outperforms 11 state-of-the-art methods across four benchmark datasets and generalizes well to challenging real-world scenarios.

    Source: §VI · p. 10

  • Takeaways & Limitations

    Patch warping preserves transfer in highly similar regions, while KF-guided deformable warping improves aggregation in weakly correlated regions.

    Source: §VI · p. 10

    Foreground simulation may not cover the extreme geometric variations of real scenes, and extremely large cross-view displacement can leave residual alignment errors that propagate into visible fusion artifacts.

    Source: §VII · pp. 10–11

Abstract

from arXiv · show

Dual-lens super-resolution (DuSR) aims to enhance a wide-angle low-resolution (LR) image by exploiting a telephoto reference (Ref) image. A central challenge in DuSR is reliable reference utilization under large parallax, occlusion, and cross-camera imaging discrepancies. Existing methods often employ off-the-shelf monocular optical flow models to provide geometric priors. However, their assumptions of photometric consistency and smooth motion are violated in dual-lens scenarios, leading to unreliable alignment and degraded reference-based reconstruction. To address these issues, we propose DuSRFlow, a dual-lens super-resolution framework that improves reference utilization from both correspondence learning and reference aggregation. First, we introduce DuFlowNet, a dedicated dual-lens flow estimator for robust cross-view alignment. To train DuFlowNet without ground-truth flow annotations, we develop a real--synthetic data fusion strategy that composites synthetic foreground objects onto real dual-lens image pairs, generating controllable large-parallax samples while preserving realistic cross-camera imaging characteristics. Second, we observe that reference reliability is highly spatially uneven in DuSR, where unreliable local reference responses may suppress the utilization of reliable cues. We therefore propose a Complementary Reference Warping strategy that coordinates patch-based warping and kernel-free-guided deformable warping according to local reference reliability, producing a more reliable reference representation before LR--Ref fusion. In addition, we construct DuSR-RealV2, a challenging real-world benchmark with complex depth variations, large parallax, and occlusion. Extensive experiments on multiple DuSR benchmarks demonstrate that DuSRFlow achieves state-of-the-art performance in reconstruction quality, perceptual quality, and cross-dataset generalization.

I. INTRODUCTION

DuSR uses a telephoto reference to enhance a wide-angle LR image, but parallax and cross-camera imaging differences make correspondence and reference use difficult. DuSRFlow addresses these challenges through dedicated flow learning, complementary reference warping, and a real-world benchmark.

I. INTRODUCTION

  • Dual-lens parallax, occlusion, and cross-camera differences challenge reliable alignment, while uneven reference quality can cause weak local responses to limit use of reliable cues.Off-the-shelf flow assumptions about appearance consistency and smooth motion do not hold well in this setting.Source: §I · pp. 1–2
  • DuFlowNet learns dual-lens correspondence from controllable large-parallax composites of synthetic foreground objects and real camera pairs, without ground-truth flow annotations.The fusion strategy preserves realistic backgrounds and cross-camera discrepancies while adding view-dependent occlusion.Source: §I · pp. 1, 3
  • The figure contrasts cleaner DuFlowNet alignment with SPyNet and shows fewer warping artifacts and fewer reference-imprinted artifacts than KeDuSR.It links improved alignment to more faithful structures and reference use.Source: Figure 1 · p. 2, §I · p. 2
  • Complementary Reference Warping coordinates patch-based and deformable warping by local reliability before LR–Ref fusion.Patch warping preserves structures in highly similar regions, while DCN-based warping compensates in weakly correlated or partially occluded regions.Source: §I · pp. 2–3
  • DuSR-RealV2 adds a real-world benchmark with complex depth, large parallax, and occlusion, including aligned paired data and challenging unpaired data.The paired subset supports quantitative evaluation, while the unpaired subset assesses generalization in more realistic settings.Source: §I · pp. 2–3

II. RELATED WORK

The related work reviews reference matching and alignment approaches, then identifies challenges in dual-lens correspondence and real-world evaluation. DuSR-RealV2 addresses the dataset gap with paired and unpaired subsets spanning reliable alignment and stronger parallax.

A. Reference-Based Super-Resolution

  • RefSR methods establish correspondence through dense patch matching,,,, optical flow, deformable convolution, or combined and coarse-to-fine approaches –,.Their assumptions of relatively consistent appearance and geometry become unreliable under dual-lens discrepancies and large parallax.Source: A. Reference-Based Super-Resolution · p. 3

B. Dual-Lens Super-Resolution

  • DuSR methods explore self-supervision, optical-flow pre-alignment, online feature adaptation, geometric correspondence, and kernel-free matching –,, [25], but reliable correspondence remains difficult under large parallax and varying reference similarity.The challenge combines cross-camera discrepancies with spatially varying reference similarity.Source: B. Dual-Lens Super-Resolution · p. 3

C. Image Alignment

  • Generic flow models such as SPyNet, PWC-Net, and RAFT assume photometric consistency and relatively smooth motion, assumptions violated by dual-lens resolution, color, brightness, blur, and viewpoint differences.These discrepancies motivate task-specific dual-lens flow learning.Source: C. Image Alignment · p. 3

III. DUSR-REALV2 DATASET

  • DuSR-RealV2-Paired has the most compact low-OR distribution, while Unpaired shifts toward higher AFM, indicating reliable pairs and stronger parallax.The 4K dataset uses iPhone 15 Pro wide-angle and telephoto cameras and contains 725 image triplets, including paired and unpaired samples.Source: §III · pp. 3–4

IV. METHOD

DuSRFlow aligns the telephoto reference to the LR center, aggregates reference features using complementary warping, and fuses them with LR features. DuFlowNet is trained with real–synthetic samples designed to add controllable parallax while retaining realistic dual-lens imaging characteristics.

A. Framework Overview

  • The framework aligns Ref to the LR center, combines flow alignment with KF matching and LR features in complementary warping, then fuses the aggregated Ref and LR features.The result is the final super-resolved image.Source: A. Framework Overview · p. 4

B. Dual-Lens Flow Based Alignment

1) Real–Synthetic Data Fusion Strategy:

  • Synthetic foregrounds transformed differently across reference and LR/HR-center branches create controllable parallax and view-dependent occlusion while real backgrounds preserve dual-lens imaging characteristics.The LR foreground is blurred to mimic the resolution gap between LR and HR.Source: 1) Real–Synthetic Data Fusion Strategy · p. 5, Figure 5 · p. 5

2) DuFlowNet for Dual-Lens Alignment:

DuFlowNet estimates cross-view flow from the reference to the LR-center domain and uses the warped reference image for supervision rather than ground-truth flow. A toy experiment reports more reliable matching with DuFlowNet, while Complementary Reference Warping coordinates patch and KF-guided deformable warping.

2) DuFlowNet for Dual-Lens Alignment:

  • The network concatenates downsampled Ref with the LR-center image to predict flow, which backward-warps Ref for supervision against the downsampled HR-center image.The HR-center target shares the LR-center view and the telephoto camera response of Ref, providing a photometrically consistent target.Source: 2) DuFlowNet for Dual-Lens Alignment · p. 5
  • Flow-guided deformable convolution refines the warped reference features, aligning them to the LR-center domain and producing the aligned reference feature.The refinement addresses sensitivity of direct flow-based feature warping to local errors.Source: 2) DuFlowNet for Dual-Lens Alignment · p. 5

3) Alignment Effectiveness Validation:

  • DuFlowNet yields fewer blocky artifacts than common matching, fewer line-like and structural artifacts than KF matching with SPyNet, and higher hit rate at equal patch-similarity thresholds.The toy experiment compares reconstruction from matched patches and correspondence statistics.Source: 3) Alignment Effectiveness Validation · pp. 5–6, Figure 6 · p. 6

C. Complementary Reference Warping

  • Complementary Reference Warping coordinates patch-based and KF-guided DCN-based warping to build a more reliable reference representation before LR–Ref fusion.KF matching between LR and its center region supplies an index and confidence score for subsequent reference warping and fusion.Source: C. Complementary Reference Warping · p. 6

1) Kernel-Free Matching:

Patch-based warping transfers features using the KF index, while KF-guided deformable warping provides local compensation where rigid patch retrieval is less reliable. Its learned offsets and masks are conditioned on LR and reference features.

2) Patch-Based Warping:

3) KF-Guided DCN-Based Warping:

  • KF-guided DCN uses the KF index as a coarse correspondence anchor for deformable aggregation around the matched location.This branch addresses weakly correlated regions where patch warping is less reliable.Source: 3) KF-Guided DCN-Based Warping · p. 6
  • A lightweight block conditioned on LR, patch-warped reference, and aligned reference features predicts learnable deformable offsets and modulation masks for aggregation.The aggregation also uses shared kernel weights and predefined sampling offsets.Source: 3) KF-Guided DCN-Based Warping · p. 6

4) Complementary Reference Fusion:

Complementary Reference Warping combines patch-based and KF-DCN warping before LR–Ref fusion, using local confidence to preserve reliable reference cues. Patch warping suits highly similar regions, while KF-DCN supports weakly correlated or occluded areas.

4) Complementary Reference Fusion:

  • Simply mixing the branches can let unreliable local reference features suppress reliable cues, making spatially adaptive aggregation important.This concern arises because reference reliability varies across regions with overlap, correspondence, and occlusion.Source: 4) Complementary Reference Fusion · p. 6
  • Patch warping emphasizes highly similar regions, while KF-DCN warping is relied on more in weakly correlated or occluded areas.The branches have complementary advantages rather than uniform reliability across the image.Source: 4) Complementary Reference Fusion · p. 6
  • Confidence-aware aggregation before LR–Ref fusion coordinates the two warped reference features to preserve reliable cues.A spatially adaptive modulation map weights patch-warped and KF-DCN-warped features.Source: 4) Complementary Reference Fusion · pp. 6–7
  • KF confidence has a Spearman correlation of 0.637 with ideal reference confidence, supporting its use as an inference-time reliability proxy.Ideal reference confidence requires the HR image and is unavailable during inference; the analysis bins valid patches by KF confidence.Source: 4) Complementary Reference Fusion · p. 7, Figure 7 · p. 7

D. LR–Ref Fusion

DuSRFlow extracts and upsamples LR features, then fuses them with the aggregated reference representation before reconstruction. Across the reported real-world benchmarks, it leads the paired and unpaired comparisons and generalizes well.

D. LR–Ref Fusion

  • The LR feature is upsampled to the aggregated reference feature's resolution, then fused using spatial and channel attention before convolutional reconstruction.The method uses a standard SISR encoder to extract the LR feature.Source: D. LR–Ref Fusion · p. 7

A. Quantitative Comparison of SR Methods

  • Comparisons cover four real-world DuSR datasets, with full-reference metrics for paired sets across full, center, and corner image scopes.DuSR-RealV2-Unpaired is assessed with no-reference metrics because well-aligned HR images are unavailable.Source: A. Quantitative Comparison of SR Methods · pp. 7–8
  • DuSRFlow achieves the best overall PSNR, SSIM, and LPIPS on paired benchmarks and the best results across no-reference metrics on DuSR-RealV2-Unpaired.The unpaired subset lacks well-aligned HR ground truth, so it is evaluated with no-reference metrics.Source: Table I · p. 7, Table II · p. 7, A. Quantitative Comparison of SR Methods · p. 8
  • Visual comparisons report faithful texture details and closer resemblance to HR images, including detail recovery from the reference that is absent in LR inputs.The cited visual results include DuSR-RealV2; additional datasets are shown in the supplementary material.Source: A. Quantitative Comparison of SR Methods · p. 8
  • DuSRFlow achieves the best cross-dataset generalization and the second-fastest inference speed among the compared RefSR methods.The detailed results are provided in the supplementary material.Source: A. Quantitative Comparison of SR Methods · p. 8

B. Quantitative Comparison of Flow Methods

DuFlowNet is evaluated as a dual-lens alignment model, and the ablations examine its flow prior, KF-DCN aggregation, and complementary reference warping. The results support dedicated flow adaptation and confidence-aware coordination of the warping branches.

B. Quantitative Comparison of Flow Methods

  • Real–synthetic adaptation improves most evaluated flow architectures on both paired and unpaired subsets; DuFlowNet offers 0.65 M parameters and 5 ms latency.No real dual-lens ground-truth flow is available, so alignment is evaluated by reference-to-target similarity proxies.Source: B. Quantitative Comparison of Flow… · pp. 8–9
  • DuFlowNet produces the cleanest visual alignment, with the fewest artifacts especially under severe parallax and occlusion.The comparison visualizes structural discrepancies as color artifacts.Source: B. Quantitative Comparison of Flow… · p. 9, Figure 9 · p. 9

1) Ablation on DuFlowNet Prior and KF-DCN:

  • Explicit flow guidance improves over the flow-free baseline, and DuFlowNet brings more consistent gains, especially in the center region.The ablation compares SPyNet and DuFlowNet as flow priors and finds the dedicated dual-lens prior more suitable for highly correlated regions.Source: 1) Ablation on DuFlowNet Prior and… · p. 9
  • KF-guided deformable aggregation improves on direct LR–Ref matching indices, and the variant combining it with DuFlowNet achieves the best overall performance.Direct-index DCN improves corner reconstruction but remains affected by cross-camera discrepancies.Source: 1) Ablation on DuFlowNet Prior and… · p. 10

2) Ablation on Complementary Reference Warping:

  • Patch warping better preserves center details, while KF-DCN performs better in low-similarity corners; rigid stitching and channel concatenation do not fully retain these regional advantages.The compared naive strategies lack adaptive spatial reliability coordination.Source: 2) Ablation on Complementary Reference… · p. 10
  • Confidence-aware aggregation before LR–Ref fusion achieves the best overall performance, better preserving reliable cues than uniform mixing with later confidence modulation.The visualization reports spatially coherent, high-quality features for the complementary fusion variant.Source: 2) Ablation on Complementary Reference… · p. 10
  • Removing parallax simulation notably degrades alignment on the large-parallax unpaired subset, supporting its importance for robust dual-lens flow estimation.Source: 2) Ablation on Complementary Reference… · p. 10

VI. CONCLUSION AND DISCUSSION

DuSRFlow combines real–synthetic-trained DuFlowNet with complementary reference warping, and the experiments report strong results across four benchmarks. The paper also identifies residual artifacts under extreme cross-view displacement and a boundary in the foreground simulation's coverage.

VI. CONCLUSION AND DISCUSSION

  • Figure 10 marks center and corner regions; Figure 11 illustrates the extreme-parallax failure case.The latter's color artifacts visualize large LR–Ref displacement.Source: Figure 10 · p. 10, Figure 11 · p. 10
  • Across four benchmark datasets, DuSRFlow consistently outperforms 11 state-of-the-art methods and generalizes well to challenging real-world scenarios.Its reference aggregation coordinates patch warping for highly similar regions and KF-guided deformable warping for weakly correlated regions.Source: §VI · p. 10

VII. LIMITATIONS

  • Foreground simulation may not fully cover the extreme geometric variations encountered in real scenes, and very large cross-view displacement can still produce alignment errors and fusion artifacts.Fig. 11 reports fewer residual fusion artifacts than KeDuSR under extreme parallax.Source: §VII · pp. 10–11, Figure 11 · p. 10

S1. IMPLEMENTATION DETAILS A. Training Settings

  • Training uses batch size 4 for both models; LR patches are 448 × 448 for DuFlowNet and 256 × 256 for DuSRFlow.DuSRFlow training follows DuFlowNet convergence, then fine-tunes DuFlowNet.Source: A. Training Settings · p. 13

B. Loss Functions

DuFlowNet uses photometric reconstruction and edge-aware smoothness losses, while DuSRFlow uses a hybrid reconstruction, perceptual, and adversarial objective.

1) DuFlowNet:

  • DuFlowNet's photometric term uses Charbonnier loss to supervise the warped reference against the downsampled HR-center image.The downsampled HR-center image is the target described alongside the warped reference.Source: 1) DuFlowNet · p. 13
  • A second-order edge-aware smoothness loss suppresses flow variations while preserving motion boundaries.Its edge-weighting strength is α = 10.Source: 1) DuFlowNet · p. 13
  • DuFlowNet's overall training objective combines its loss terms with λf = 1 and λS = 0.5.Source: 1) DuFlowNet · p. 13

2) DuSRFlow:

  • DuSRFlow uses a hybrid objective with reconstruction, perceptual, and adversarial terms, including Charbonnier reconstruction loss and VGG19 feature maps for perceptual loss.The adversarial loss is used to enhance visual realism; λ1 = 10−3 and λ2 = 10−4, and both reconstruction-only and hybrid training are evaluated.Source: 2) DuSRFlow · p. 13

S2. ADDITIONAL EXPERIMENTAL RESULTS

Supplementary experiments report strong quantitative performance on two real-world datasets, visual comparisons across real-world datasets and DuSR-RealV2 subsets, and model-complexity comparisons.

A. Additional Quantitative Results

B. Additional Visual Results

  • Visual comparisons report sharper structures and more faithful details across three real-world datasets, with robustness across paired and unpaired DuSR-RealV2 subsets.The unpaired subset's pseudo ground truth is provided for reference only.Source: B. Additional Visual Results · pp. 13–14

C. Model Complexity

  • At 1792 × 896 output, DuSRFlow reduces parameter count and inference latency versus KeDuSR and is the second-fastest compared RefSR method.FLOPs and latency are measured on a single NVIDIA RTX 3090 GPU.Source: C. Model Complexity · p. 14

D. Cross-Dataset Generalization

The supplementary results report best cross-dataset generalization, examine the real–synthetic data fusion ablation, and quantify the complementary warping branches' memory overhead.

D. Cross-Dataset Generalization

S3. ABLATION ON DUFLOWNET

  • Combining real dual-lens backgrounds with controllable foreground parallax simulation improves alignment across large-parallax foreground and background regions.Without foreground simulation, alignment is poor on the Unpaired subset; synthetic-only backgrounds perform worst on the Paired subset.Source: S3. ABLATION ON DUFLOWNET · p. 14, Table S4 · p. 14, Figure S1 · p. 14

S4. MEMORY OVERHEAD

  • The complete model uses 3600.75 MiB peak GPU memory, only 98 MiB (2.8%) more than KF-DCN warping alone.The warping branches execute sequentially and share KF indices and a confidence map, keeping peak memory close to the more demanding branch rather than their sum.Source: Table S5 · p. 14, S4. MEMORY OVERHEAD · p. 14

Related papers

Loading 2610.14630v1…