PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images

Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, Yao Zhao

arXiv:2106.12859v1cs.CV

TL;DR

Feature-based stitching depends heavily on feature detection and homography assumptions that can fail in challenging scenes. The paper proposes an unsupervised two-stage alignment-and-reconstruction framework with a real-world benchmark dataset, achieving state-of-the-art performance and user-preferred visual quality over supervised methods.

  • Problem

    Feature-based stitching relies heavily on feature detection and can fail with few features or at low resolution, while a single homography cannot model multiple depth levels and may cause ghosting.

  • Method

    An unsupervised framework combines coarse homography alignment using an ablation-based loss and stitching-domain transformer with feature-to-pixel image reconstruction and a real-world benchmark dataset.

  • Results

    The method outperforms state-of-the-art homography and stitching solutions, and users prefer its visual quality even compared with supervised stitching methods.

  • Takeaways & Limitations

    The released real-world dataset provides a benchmark for unsupervised deep image stitching and supports evaluation of methods that avoid labeled training data.

  • Takeaways & Limitations

    The reconstruction network may fail under extremely large parallax because the first stage remains based on a single homography and reconstruction capacity is limited.

Abstract

from arXiv · show

Traditional feature-based image stitching technologies rely heavily on feature detection quality, often failing to stitch images with few features or low resolution. The learning-based image stitching solutions are rarely studied due to the lack of labeled data, making the supervised methods unreliable. To address the above limitations, we propose an unsupervised deep image stitching framework consisting of two stages: unsupervised coarse image alignment and unsupervised image reconstruction. In the first stage, we design an ablation-based loss to constrain an unsupervised homography network, which is more suitable for large-baseline scenes. Moreover, a transformer layer is introduced to warp the input images in the stitching-domain space. In the second stage, motivated by the insight that the misalignments in pixel-level can be eliminated to a certain extent in feature-level, we design an unsupervised image reconstruction network to eliminate the artifacts from features to pixels. Specifically, the reconstruction network can be implemented by a low-resolution deformation branch and a high-resolution refined branch, learning the deformation rules of image stitching and enhancing the resolution simultaneously. To establish an evaluation benchmark and train the learning framework, a comprehensive real-world image dataset for unsupervised deep image stitching is presented and released. Extensive experiments well demonstrate the superiority of our method over other state-of-the-art solutions. Even compared with the supervised solutions, our image stitching quality is still preferred by users.

I. INTRODUCTION

Image stitching aims to build wider-field-of-view panoramas, but feature dependence and planar homography assumptions limit conventional methods. The paper proposes an unsupervised two-stage framework, a real-world dataset, and reports superiority over existing solutions.

  • Motivation: Image stitching constructs wider-field-of-view panoramas from images captured at different viewing positions.The task supports applications including biology, medicine, surveillance, autonomous driving, and virtual reality.
  • Motivation: A single homography cannot model multiple depth levels, often producing ghosting when scenes violate the planar assumption.Homography handles transformations between 2D planes but cannot account for all alignments at different depths.
  • Limitations of existing methods: Feature-based stitching can sharply degrade or fail in scenes with few features or low resolution because it depends heavily on feature detection.Existing methods reduce or hide artifacts through spatially varying warps or optimized seams, but retain this dependence.
  • Proposed framework: The proposed unsupervised framework combines coarse homography alignment with feature-to-pixel image reconstruction.An ablation-based loss targets large-baseline scenes, while a stitching-domain transformer warps inputs with less occupied space.
  • Proposed framework: The reconstruction network uses feature-level processing to reduce visible misalignment, with low-resolution deformation and high-resolution refinement branches.The method reconstructs stitched features close to the warped inputs before reconstructing the final image at pixel level.
  • Evaluation: The paper introduces a large real-world dataset and reports performance superior to state-of-the-art methods, with users preferring its quality over supervised solutions.The dataset includes varying overlap rates, parallax levels, and indoor, outdoor, night, dark, snow, and zooming scenes.

II. RELATED WORK

Prior stitching research uses adaptive warping, seam-driven artifact concealment, or learning-based pipelines. These approaches remain constrained by feature scarcity, restricted viewpoints, limited training data, or difficult real-world scenes.

  • Feature-based methods: Feature-based artifact reduction methods either align images through multiple spatially varying transformations or hide artifacts with optimized seams.Adaptive warping combines multiple alignment models, while seam-driven methods select or optimize seams for natural results.
  • Feature-based methods: Feature-based algorithms often fail with few features or low resolution because their performance depends heavily on feature detection quality.They can produce perceptually natural results in many scenes but remain vulnerable when reliable features are unavailable.
  • Learning-based methods: Learning-based stitching is limited by scarce real datasets, low overlap rates, and large parallax in practical scenes.These constraints keep learning-based image stitching in development.
  • Learning-based methods: View-fixed methods use end-to-end networks for specific applications but cannot extend to arbitrary camera views.Examples include autonomous driving and surveillance-video settings.
  • Learning-based methods: View-free approaches either use CNNs only for feature detection or employ complete pipelines with limitations in resolution handling and real-scene quality.Later work removes the fully connected-layer resolution restriction and improves real-scene stitching performance.

C. Deep Homography Schemes

The framework uses unsupervised deep homography estimation for coarse alignment, targeting the larger baselines encountered in image stitching. An ablation-based constraint replaces padding-based supervision to retain meaningful overlap information.

  • Deep Homography Schemes: The homography network estimates a coarse alignment before images are warped by the stitching-domain transformer layer.This two-step arrangement separates unsupervised homography estimation from subsequent image warping in the stitching domain.
  • Deep Homography Schemes: Padding-based constraints can fail when large-baseline input patches contain no overlap, producing meaningless homography estimates.Existing methods pad invalid warped pixels with extra target-image contents, which works for smaller baselines but not sufficiently large ones.
  • Deep Homography Schemes: The proposed ablation-based strategy uses full images and removes corresponding reference contents where warped target pixels are invalid.Pixel-wise multiplication with an all-one matrix defines the ablation-based objective without padding invalid warped pixels.

B. Stitching-Domain Transformer Layer

The stitching-domain transformer warps input images into the smallest rectangle containing the stitched image, reducing wasted spatial area while preserving image contents.

  • Stitching-Domain Transformer Layer: Existing spatial-transformer stitching methods use a fixed maximum output resolution, leaving much of the surrounding black-pixel area unused.The fixed-resolution design accommodates varying overlap rates but wastes space outside the valid stitched region.
  • Stitching-Domain Transformer Layer: The stitching-domain is defined as the smallest bounding rectangle of the stitched image, preserving content integrity while minimizing spatial waste.The layer computes warped vertices and the resulting output size before assigning warped-image pixels.
  • Stitching-Domain Transformer Layer: The layer assigns warped-image pixels from the inputs using identity and homography transformations within the computed stitching domain.Warping uses a 3×3 transformation matrix and produces warped images IA W and IB W at the stitching-domain resolution.
  • Stitching-Domain Transformer Layer: Transforming inputs in stitching-domain space reduces subsequent feature-map occupancy and supports larger-resolution images under limited GPU memory.This is the stated advantage over the transformer layers used in earlier deep image-stitching methods.

IV. UNSUPERVISED IMAGE RECONSTRUCTION

The unsupervised reconstruction stage addresses residual misalignment from single-homography alignment by reconstructing stitched images from features to pixels. It combines low-resolution deformation learning with mask-based content and seam constraints.

  • IV. UNSUPERVISED IMAGE RECONSTRUCTION: Because a single homography cannot fully align real-world images with multiple depth levels, reconstruction breaks the limitation after coarse alignment.The reconstruction network uses low-resolution deformation and high-resolution refinement branches.
  • A. Low-Resolution Deformation Branch: The low-resolution branch first learns stitching deformation rules with a broad receptive field before high-resolution reconstruction.Warped images are down-sampled to 256×256 in the implementation, then processed by an encoder-decoder network.
  • A. Low-Resolution Deformation Branch: Content masks constrain reconstructed features toward warped images, while seam masks encourage natural and continuous overlap boundaries.The masks are used to formulate content and seam losses for unsupervised deformation learning.
  • A. Low-Resolution Deformation Branch: The reconstruction process progresses from overlapping regions toward non-overlapping regions as encoder-decoder features become semantically deeper and then clearer.Visualization shows overlapping areas emphasized initially, followed by attention to non-overlapping areas during decoding.

B. High-Resolution Refined Branch

After low-resolution deformation initialization, the high-resolution refined branch increases output resolution and refines the stitched image using convolutional processing.

  • B. High-Resolution Refined Branch: The branch consists entirely of convolutional layers, allowing it to process images of arbitrary resolution.Its architecture contains three separate convolutional layers and eight residual blocks.
  • B. High-Resolution Refined Branch: The high-resolution branch upsamples the low-resolution stitched image, concatenates it with the warped inputs, and outputs the high-resolution stitched image.Its high-resolution target exceeds 512×512 in the dataset.
  • B. High-Resolution Refined Branch: High-resolution content and seam losses reuse the low-resolution formulations with high-resolution outputs and masks.The high-resolution perceptual loss uses VGG-19 layer ‘conv3 3’, which is shallower than the low-resolution ‘conv5 3’ layer.

C. Objective Function

The reconstruction objective addresses the high-resolution branch’s artifact tendency while learning feature-to-pixel stitching behavior. Feature visualization shows the network progressively reconstructing semantic and non-overlapping content.

  • Objective Function: The high-resolution branch can introduce artifacts because increased resolution reduces its receptive field.A content consistency loss is proposed to enhance resolution while reducing parallax artifacts.
  • Objective Function: The objective combines low-resolution, high-resolution, and content-consistency components with separately weighted contributions.The weights are denoted ωLR, ωHR, and ωCS.
  • Objective Function: During encoding, features focus first on overlapping areas, while deeper layers extract and reconstruct increasingly semantic features.Decoder features later attend to non-overlapping regions as well as overlapping regions.

A. Dataset and Implement Details

The study uses a real-world, unlabeled dataset spanning overlap rates, parallax levels, and varied scenes, with separate training and testing cases. Homography comparisons evaluate synthetic and real-data performance using established baselines and overlap-region metrics.

  • Dataset: The proposed dataset is collected from moving videos and contains varying overlap rates, non-planar scenes, and different degrees of parallax.Its scenes include indoor, outdoor, night, dark, snow, and zooming conditions.
  • Dataset: The dataset contains 10,440 training cases and 1,106 testing cases, with overlap-rate and parallax distributions reported across defined levels.High, middle, and low overlap rates are greater than 90%, 60%-90%, and lower than 60%, respectively.
  • Dataset: Because the dataset has no ground truth, its testing results are released as a benchmark for comparison.The benchmark is intended to support subsequent unsupervised image-stitching research.
  • Implement Details: The unsupervised training procedure uses synthetic pretraining, real-dataset homography finetuning, and real-dataset reconstruction training.The stages run for 150, 50, and 20 epochs, respectively.
  • Homography Estimation: On the synthetic dataset, Ours v1 outperforms existing unsupervised deep homography methods, whereas the ablation-based loss is unsuitable for the no-parallax setting.Applying the loss to LB-DHN increases 4pt-Homography RMSE on this synthetic dataset.
  • Homography Estimation: On the real dataset, Ours v2 outperforms all compared methods, including supervised solutions, while Ours v1 generalizes better than LB-DHN.Ours v2 is Ours v1 finetuned on the proposed real dataset; evaluation uses overlap-region PSNR and SSIM.

C. Comparison of Image Stitching

The proposed framework is compared with feature-based and learning-based stitching methods across robustness and visual-quality studies. It remains robust in challenging low-light, indoor, and low-resolution settings and is preferred by users over supervised alternatives.

  • Robustness: The robustness comparison evaluates 1,106 test samples resized to 512 × 512, 256 × 256, and 128 × 128 resolutions.The setup tests sensitivity to changing feature quantity through input resolution.
  • Robustness: Ours is more robust than feature-based methods, especially in low-light and indoor scenes where their error and failure cases are concentrated.The method successfully stitches challenging indoor and dark examples.
  • Robustness: As resolution decreases, learning-based methods’ success rates decrease while ours remains robust.The robustness study includes the 128 × 128 condition used for comparison with VFISNet.
  • Learning-Based Solutions: Compared with supervised learning-based methods, ours shows superiority in robustness, continuity, illumination, and visual quality.The comparison includes VFISNet and EPISNet.
  • Robustness: The reconstruction model reduces artifacts, while the overall method also addresses inaccurate homography estimation as a source of failure cases.These mechanisms are reported as reasons for improved robustness.
  • Visual Quality: In user studies, participants preferred the proposed stitched images over EPISNet and VFISNet+Bicubic in visual quality.The studies average percentage preferences across 20 participants; Bicubic resizing is used for VFISNet.
  • Visual Quality: Visual comparisons cover varying parallax, with green rectangles marking severe blur and red rectangles marking discontinuous edges.The examples include both proposed-dataset samples and classic instances outside the dataset.

D. Ablation Studies

Ablations isolate the effects of the low- and high-resolution branches and the content, seam, and content-consistency losses. The components progressively improve resolution, suppress seam distortions, and remove artifacts.

  • Network and Loss Ablations: The LR branch with content loss enables image stitching but leaves seam distortions and limited resolution.The seam distortion is identified as a side effect of the content loss.
  • Network and Loss Ablations: Adding the HR branch enhances stitched-image resolution but introduces some artifacts because its convolutional receptive field is too small.This compares framework v2 with v1.
  • Network and Loss Ablations: The ablation figure compares framework outputs and enlarged patches for artifacts, definition, and seam distortions.The columns organize full outputs followed by diagnostic enlarged regions.
  • Network and Loss Ablations: Adding seam loss removes seam distortions by imposing pixel-level similarity at the overlapping-area edge.Artifacts remain after this modification.
  • Network and Loss Ablations: Adding content consistency loss removes the remaining artifacts and acts as an enhancer of the HR branch’s receptive field.It promotes the HR branch receptive field from that of the LR branch.

VI. LIMITATION AND FUTURE WORK

The reconstruction network can remove parallax artifacts only within limits. As parallax grows, alignment worsens and reconstruction becomes more difficult, potentially misinterpreting misalignments as objects.

  • As parallax increases, first-stage alignment degrades and reconstruction becomes more burdensome.
  • With extremely large parallax, the reconstruction network may treat residual misalignments as new objects.
  • Future work targets stronger alignment and a larger reconstruction receptive field for remaining large misalignments.

VII. CONCLUSION

The paper presents an unsupervised two-stage stitching framework with coarse homography alignment and feature-to-pixel reconstruction, alongside a benchmark dataset. Experiments report superiority over state-of-the-art methods, while extremely large parallax remains a limitation.

  • The framework combines unsupervised coarse alignment with unsupervised image reconstruction for deep image stitching.
  • An ablation-based loss constrains homography estimation, while a stitching-domain transformer warps input images.
  • The reconstruction network generates stitched images from features to pixels and removes artifacts without supervision.
  • A real dataset is released as a benchmark for unsupervised deep image stitching methods.
  • The method outperforms other state-of-the-art solutions, but may fail in scenes with extremely large parallax.
Loading 2106.12859v1…