Source-linked AI summary
Parallax-Tolerant Unsupervised Deep Image Stitching
Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, Yao Zhao
TL;DR
Traditional stitching relies on complex geometric features, while deep methods still struggle with large parallax under homography-based alignment. UDIS++ combines a flexible homography-to-TPS warp, iterative adaptation, and unsupervised seam-driven composition; experiments report superiority over state-of-the-art solutions.
Problem
Traditional stitching depends on complex geometric features, while deep stitching remains limited by large parallax under homography-based alignment.
Source: §1 · p. 2Method
UDIS++ jointly parameterizes homography and thin-plate spline motion, adapts its warp iteratively across datasets and resolutions, and learns seam-driven composition masks.
Source: §1 · p. 2Results
Experiments report that UDIS++ is superior to other state-of-the-art solutions.
Source: §1 · p. 2Takeaways & Limitations
The proposed solution addresses challenging scenes and large-parallax cases while exhibiting efficiency with GPUs.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Traditional image stitching approaches tend to leverage increasingly complex geometric features (point, line, edge, etc.) for better performance. However, these hand-crafted features are only suitable for specific natural scenes with adequate geometric structures. In contrast, deep stitching schemes overcome the adverse conditions by adaptively learning robust semantic features, but they cannot handle large-parallax cases due to homography-based registration. To solve these issues, we propose UDIS++, a parallax-tolerant unsupervised deep image stitching technique. First, we propose a robust and flexible warp to model the image registration from global homography to local thin-plate spline motion. It provides accurate alignment for overlapping regions and shape preservation for non-overlapping regions by joint optimization concerning alignment and distortion. Subsequently, to improve the generalization capability, we design a simple but effective iterative strategy to enhance the warp adaption in cross-dataset and cross-resolution applications. Finally, to further eliminate the parallax artifacts, we propose to composite the stitched image seamlessly by unsupervised learning for seam-driven composition masks. Compared with existing methods, our solution is parallax-tolerant and free from laborious designs of complicated geometric features for specific scenes. Extensive experiments show our superiority over the SoTA methods, both quantitatively and qualitatively. The code is available at https://github.com/nie-lang/UDIS2.
1. Introduction
Traditional stitching relies on hand-crafted geometry that struggles in weakly structured scenes, while deep methods remain limited by large parallax and generalization. UDIS++ combines a flexible warp, iterative adaptation, and learned seam-driven composition to address these challenges.
1. Introduction
- Traditional stitching increasingly relies on complex geometric features, which adapt poorly to scenes lacking sufficient structure and can run intolerably slowly in richly structured scenes.The cited challenging settings include medical, industrial, low-texture, low-light, and low-resolution images.Source: §1 · pp. 1–2
- Deep stitching learns semantic features and handles varied conditions, but homography-based registration struggles with large parallax and cross-dataset or cross-resolution generalization.A reconstruction-based approach can also introduce blur around parallax regions.Source: §1 · p. 2
- UDIS++ parameterizes global homography and local thin-plate spline motion together, jointly optimizing alignment and distortion for content alignment and shape preservation.The warp is intended to align overlapping regions while preserving non-overlapping shapes.Source: §1 · p. 2
- Experiments report that UDIS++ is superior to other state-of-the-art stitching solutions.The reported experiments cover both warp and composition.Source: §1 · p. 2
- An iterative strategy adapts the warp across datasets and resolutions, while unsupervised seam-driven masks composite stitched images to reduce parallax artifacts without the stated blur side effect.The composition approach is contrasted with reconstruction-based composition.Source: §1 · p. 2
2. Related Work
Traditional stitching combines geometric cues for alignment and shape preservation, while seam cutting composites results using photometric costs. Deep stitching learns semantic features but remains constrained by homography-based alignment in large-parallax cases.
2.1. Traditional Image Stitching
- Traditional stitching uses SIFT keypoints for global and adaptive warps, with line, edge, depth, and semantic-plane cues also used to improve alignment or preserve structure [33].Other cited methods use line segments and line-preserving terms [31, 49].Source: §2.1 · p. 2
- Traditional seam cutting assigns labels along a seam using photometric costs, including color, gradient, motion- and exposure-aware, and saliency differences.These energy functions can be minimized with graph-cut optimization.Source: §2.1 · p. 3
- Geometric-feature methods can produce poor stitching or fail without sufficient scene structure, while computational cost rises sharply when structures are abundant.Source: §2.1 · p. 3
2.2. Deep Image Stitching
- Deep stitching learns high-level semantic features and is robust to varied scenes, but homography-based alignment cannot handle large parallax and reconstruction can blur parallax regions.Source: §2.2 · p. 3
3. Methodology
UDIS++ uses a two-stage warp-and-composition framework to align overlapping images while preserving shape, then produce a seamless stitched image. Its method combines global homography with local deformation, seam-driven composition, and iterative warp adaptation.
3. Methodology
- The framework first warps overlapping reference and target images, then predicts composition masks to produce a seamless stitched image.The warp aims to align images while preserving shape.Source: §3 · p. 3, Figure 2 · p. 4
3.1.1 Warp Parameterization
- For non-planar scenes, UDIS++ bridges global homography and local TPS deformation by initializing control-point motions with homography and predicting residual motions.TPS is used for efficient local deformation when homography cannot accurately align images captured from different centers.Source: §3.1.1 · p. 3
3.1.2 Pipeline of Warp
- The warp pipeline extracts semantic features, estimates a four-point homography, then predicts residual control-point motions for a flexible TPS warp.The homography supplies initial control-point motions, and feature maps are warped before the residual motions are estimated.Source: §3.1.2 · p. 4
3.1.3 Optimization of Warp
- Warp optimization combines pixel-level alignment in overlapping regions with mesh constraints that preserve non-overlapping structures and reduce projective distortion.The inter-grid constraint encourages successive edges to be colinear only in non-overlapping regions; the intra-grid constraint prevents dramatic grid scaling.Source: §3.1.3 · p. 4
3.2.1 Motivation
- Because graph-cut labels truncate backpropagation gradients and dynamic programming is non-differentiable, composition learns soft seam-inspired masks under boundary and smoothness constraints.The masks use floating-point values rather than hard labels.Source: §3.2.1 · p. 5
3.2.2 Pipeline of Composition
- The composition network extracts features from each warped image with shared encoder weights, subtracts features for skip connections, and predicts a mask for the reference image.The target-image mask is obtained by post-processing.Source: §3.2.2 · p. 5
3.2.3 Optimization of Composition
- Composition losses use boundary constraints to set seam endpoints at warped-image boundary intersections and smoothness constraints to shape the seam route.The network learns semantic differences and does not rely on photometric differences during inference.Source: §3.2.3 · p. 5
3.3. Iterative Warp Adaption
- Unsupervised iterative adaptation adjusts local alignment on new datasets or resolutions, retaining pretrained homography and distortion constraints and supporting even a single sample.It stops at a maximum iteration count or when consecutive errors fall below a threshold; the example reports reduced artifacts as iterations increase, at about 0.1 seconds per iteration.Source: §3.3 · pp. 5–6, Figure 3 · p. 5
4. Experiments
Experiments evaluate the warp, composition, and complete stitching framework across UDIS-D and traditional datasets. They report qualitative alignment and composition comparisons, speed comparisons, and ablations of warp constraints and composition smoothness.
4.1. Dataset and Implement Details
- The model is trained on UDIS-D and evaluated on that dataset and traditional datasets; the warp and composition networks are trained for 100 and 50 epochs, respectively.The warp adaption strategy is also tested; implementation details specify Adam and a single NVIDIA RTX 3090 Ti GPU.Source: §4.1 · p. 6, §4.2.1 · p. 6
4.2.1 Comparisons of Warp
- On UDIS-D, the proposed warp outperforms other solutions by a large margin in qualitative alignment comparisons across near and far regions.The comparison highlights parallax artifacts at different depth surfaces.Source: §4.2.1 · p. 6, Figure 4 · p. 7
- With GPU acceleration, the proposed warp runs far faster than the compared methods, while traditional warps cannot use GPU acceleration.Traditional mesh-based warp runtime does not vary linearly with resolution and can become especially slow in scenes rich in geometric features.Source: §4.2.1 · p. 6, Table 2 · p. 8
4.2.2 Comparisons of Composition
- Composition is evaluated qualitatively because traditional seam-cutting masks are integer-valued while the proposed masks are float-valued; its GPU-accelerated implementation is reported as significantly faster than the alternatives.The speed comparison uses inputs first warped by the proposed method.Source: §4.2.2 · p. 6, Table 3 · p. 8
4.2.3 More Comparisons
- The complete stitching framework is compared with state-of-the-art methods in Fig. 9, with additional qualitative, user-study, challenging-case, and cross-dataset results in the supplementary material.The supplied passage identifies the comparisons but does not state their outcomes.Source: §4.2.3 · p. 6
4.3. Ablation studies
- In the warp ablation, inter-grid constraints preserve structures and intra-grid constraints reduce projective distortions, with little adverse impact on alignment.Quantitative ablation results are reported in the supplementary material.Source: §4.3 · pp. 6, 8, Figure 7 · p. 8
- In the composition ablation, smoothness constraints on the difference map and stitched image significantly improve discontinuous regions.The discontinuities are indicated by rectangles in Fig. 7(bottom).Source: §4.3 · p. 8, Figure 7 · p. 8
5. Conclusion
The conclusion presents a parallax-tolerant stitching method combining an adaptively learned warp, seam-inspired composition, and iterative warp adaptation. Its supplementary discussion explains how TPS and alignment and distortion constraints support the warp’s alignment, shape preservation, and efficient computation.
5. Conclusion
- The method combines an adaptively learned warp for alignment and shape preservation, seam-inspired composition to reduce artifacts, and iterative warp adaptation for cross-dataset and cross-resolution generalization.The conclusion says the method addresses challenging scenes and large-parallax cases and exhibits efficiency with increasingly popular GPUs.Source: §5 · p. 8
A. Supplemental Material
- The supplemental material omits network architecture details such as layers and channels, while promising code for reference.The supplement lists additional warp, composition, robustness, distortion, and results material.Source: A. Supplemental Material · p. 11
B.1. Physicality of TPS
- The warp objective combines alignment and distortion energies, with λ balancing smoothness; the implementation sets λ = 0 to constrain control-point motion to predicted motion.The supplement describes TPS as a flexible 2D deformation and says the warp function is derived by minimizing the objective.Source: §B.1 · p. 11
B.2. Discussion of Alignment and Distortion
- Intra-grid and inter-grid constraints address distortion without increasing λ, supporting alignment in overlapping regions and distortion elimination in non-overlapping regions.The approach assumes control points are evenly distributed in the target image and their motion is smooth.Source: §B.2 · p. 11
B.3. Multiple Homography vs. TPS
- TPS lets all pixels share one warp function, enabling parallel backward interpolation without determining each pixel’s grid, unlike multi-homography deformation.The supplement identifies efficient parallel interpolation as a reason TPS training is faster than multi-homography.Source: §B.3 · pp. 11–12
B.4. Difference to Stitching Methods using TPS
The paper distinguishes its deep TPS warp from feature-matched TPS methods and details a boundary constraint for composition. It uses floating-point masks and smoothness to make seam-driven composition trainable and produce smooth transitions.
B.4. Difference to Stitching Methods using TPS
- The proposed method predicts motions for evenly distributed control points without matched keypoints, then uses intra-grid and inter-grid constraints; compared with ELA, it achieves better alignment, fewer distortions, and better efficiency.The paper describes this as the first deep learning-based stitching scheme using TPS transformations.Source: §B.4 · p. 12
C.1. Boundary Term
- The boundary constraint encourages pixels along the two warped-image boundaries to come from their respective images; shared boundary intersections help determine seam endpoints.The boundary masks lie inside the warped reference and target images.Source: §C.1 · p. 12
C.2. Difference to Seam Cutting
- Floating-point composition masks preserve backpropagation and, with an image smoothness constraint, encourage smooth transitions across a wider learned seam than seam-cutting masks.Traditional seam-cutting masks use only 0 or 1, so the paper says its composition cannot be evaluated quantitatively with traditional indicators; Figure 10 compares the masks.Source: §C.2 · p. 12, Figure 10 · p. 13
D.1. Analysis on Robustness
The robustness analysis contrasts the proposed semantic-feature approach with traditional feature-based stitching in challenging conditions. It highlights low-light examples for both warping and composition.
D.1. Analysis on Robustness
- In an extremely low-light case, APAP’s SIFT keypoints are sparse and sometimes mismatched, while the proposed warp extracts semantic feature maps that become more evident with network depth.The paper presents this as a robustness advantage over traditional solutions.Source: §D.1 · p. 13
- In a low-light composition example, the color-difference map misses a tree, whereas the composition network’s semantic difference maps reveal it despite training with color difference.The network is trained on both simple and challenging samples.Source: §D.1 · p. 13
D.2. Analysis on Projective Distortion
The distortion analysis attributes reduced projective distortion to pixel-level alignment across overlap regions and an intra-grid constraint. The supplement also points to additional warp and large-parallax composition comparisons.
D.2. Analysis on Projective Distortion
- The proposed warp produces fewer projective distortions by aligning all overlapping pixels rather than only feature-rich areas, while an intra-grid constraint prevents dramatic mesh scaling.Figure 13 reports visual-preference percentages averaged over 20 participants.Source: §D.2 · pp. 13–14, Figure 13 · p. 14
E.1. Results of Warp
- The supplement directs readers to additional warp results on UDIS-D and other datasets in Figures 18 and 19.Figures 14 and 15 show the input images for Figures 4 and 5 in the manuscript.Source: §E.1 · p. 14, Figure 14 · p. 14, Figure 15 · p. 14
E.2. Results of Composition
- Additional large-parallax composition results compare the proposed method with seam cutting and reconstruction on UDIS-D and other datasets, using SIFT+RANSAC alignment and average fusion as a reference.The supplement points to Figure 20 for these comparisons.Source: §E.2 · p. 14
E.3. Results of Complete Solutions
The complete framework is compared qualitatively with LPC and UDIS, and additional results show natural, seamless stitching across traditional datasets and varied scenes.
E.3. Results of Complete Solutions
- Figure 21 presents qualitative comparisons of the complete framework with LPC and UDIS, using seam cutting or reconstruction as their post-processing.Source: §E.3 · p. 14
- The user study compared the method with LPC and UDIS on 1,106 UDIS-D test images, with 20 participants choosing preferred results.Participants could zoom in; when they reported no preference, they indicated whether both results were good or both bad.Source: §E.3 · p. 14
- Across traditional datasets, the method produces natural and seamless results across scenes with varying resolutions and parallax.The paper also reports varied occlusions in these scenes.Source: §E.3 · p. 14
E.4. Results of Challenging Scenes
The proposed method succeeds on challenging scenes where traditional stitching fails for lack of geometric features; supplied captions also describe shape-preservation and warp-combination ablations.
E.4. Results of Challenging Scenes
- The proposed method succeeds on challenging low-texture and low-light scenes where traditional methods fail because geometric features are lacking.The paper refers to Section D.1 for discussion of the reason.Source: §E.4 · p. 14
- With the distortion term, alignment performance decreases little in the UDIS-D ablation.Source: Table 4 · p. 15
- Table 5 compares combining TPS with homography on the UDIS-D dataset.The caption identifies the comparison's scope but does not provide cell values or a winner.Source: Table 5 · p. 15
- Figure 16 shows an example of stitching four images from a traditional dataset.Source: Figure 16 · p. 15
E.5. Ablation Studies
The ablations examine the alignment–distortion trade-off and the combination of homography with TPS; additional examples cover multi-image stitching, challenging scenes, cross-dataset warping, and composition.
E.5. Ablation Studies
- The distortion constraints preserve shape, while SSIM decreases by only 0.03 and alignment is affected little.The paper reports that combining TPS with homography significantly improves alignment over homography alone and is slightly better than TPS alone with less computational cost.Source: §E.5 · p. 15
E.6. Multi-Image Stitching
- Multiple-image stitching is generalized by repeatedly stitching pairs, illustrated with a four-image example.Source: §E.6 · p. 15
- Traditional methods fail on the challenging scenes shown because they lack geometric features.Source: Figure 17 · p. 15
- The UDIS-D warp comparison marks artifacts with red arrows; the cross-dataset comparison marks distortions with arrows.Source: Figure 18 · p. 16, Figure 19 · p. 17
- Composition methods are compared on large-parallax images warped with SIFT+RANSAC, with red circles marking seam discontinuity or blur.Source: Figure 20 · p. 18
- The complete-framework comparison includes LPC and UDIS with seam cutting or reconstruction as composition methods.Source: Figure 21 · p. 18
- Results on traditional datasets demonstrate generalization to scenes with varied occlusions and parallax.Source: Figure 22 · p. 19
- The pipeline figure shows an intermediate result for a large-parallax cross-dataset case; UDIS-D images use no warp-adaptation iterations.Source: Figure 23 · p. 20