Source-linked AI summary
Single-Perspective Warps in Natural Image Stitching
Tianli Liao, Nan Li
TL;DR
Single-perspective stitching preserves perspective consistency but can suffer projective distortion, motivating better natural-image warps. The paper combines APAP, quasi-homography, and dual features in a parametric warp, and optimizes a mesh warp over alignment, naturalness, distortion, and saliency. Comprehensive evaluation reports improvements over several state-of-the-art warps.
Problem
Single-perspective stitching offers perspective consistency but suffers projective distortion, while existing approaches also face alignment and naturalness issues.
Method
The paper proposes an APAP-and-QH parametric warp coupled through dual features, plus a mesh warp optimized with a total energy over alignment, naturalness, distortion, and saliency.
Results
Comprehensive evaluation demonstrates that the proposed warp outperforms homography, APAP, AutoStitch, SPHP, and GSP, while yielding low alignment errors and natural-looking results.
Takeaways & Limitations
The proposed single-perspective designs jointly address alignment, naturalness, distortion, and saliency in natural image stitching.
Abstract
from arXiv · showhide
Results of image stitching can be perceptually divided into single-perspective and multiple-perspective. Compared to the multiple-perspective result, the single-perspective result excels in perspective consistency but suffers from projective distortion. In this paper, we propose two single-perspective warps for natural image stitching. The first one is a parametric warp, which is a combination of the as-projective-as-possible warp and the quasi-homography warp via dual-feature. The second one is a mesh-based warp, which is determined by optimizing a total energy function that simultaneously emphasizes different characteristics of the single-perspective warp, including alignment, naturalness, distortion and saliency. A comprehensive evaluation demonstrates that the proposed warp outperforms some state-of-the-art warps, including homography, APAP, AutoStitch, SPHP and GSP.
I. INTRODUCTION
Image stitching combines overlapping images into a wider view, but existing warps trade alignment, distortion, naturalness, and perspective consistency. The paper proposes two single-perspective warps to improve these properties.
- I. INTRODUCTION: Image stitching composes overlapping images into a wider field of view and evaluates warps by alignment, distortion, and naturalness.
- I. INTRODUCTION: Global, spatially varying, and seam-driven warps improve alignment but commonly retain single-perspective projective distortion outside the overlap.
- I. INTRODUCTION: Combining alignment-oriented warps with similarity warps reduces distortion but usually produces multiple-perspective results with perspective inconsistency.
- I. INTRODUCTION: Dual-feature warping emphasizes points and lines to improve naturalness because human vision is especially sensitive to line structure.
- I. INTRODUCTION: The paper proposes a parametric APAP-plus-QH warp via DF and a mesh-based warp optimized with a sparse quadratic total energy.
A. Mathematical Setup
The mathematical setup models stitching with homography and spatially varying warps, then analyzes their alignment, naturalness, and distortion behavior. Quasi-homography and dual-feature constraints are used to preserve perspective while reducing distortion and line misalignment.
- A. Mathematical Setup: A homography is a planar transformation mapping target pixel coordinates to reference coordinates, estimated from feature correspondences by DLT.
- B. Alignment Issue: For non-planar scenes, homography alignment is most accurate near the common dominant plane and worsens for content farther from it.
- B. Alignment Issue: APAP replaces one global homography with spatially varying local homographies in the overlap, evolving toward the dominant plane outside it.
- C. Naturalness Issue: Homography and APAP can create naturalness problems through line-orientation violations, while dual-feature estimation emphasizes line consistency alongside point alignment.
- D. Distortion Issue: Quasi-homography regularization mitigates distortion while preserving perspective, and its construction preserves length ratios along a selected family of parallel lines.
- D. Distortion Issue: The slope function s(x, y, k) combines directional derivatives of the warp components, with fx, fy, gx, and gy denoting their partial derivatives.
E. Composite Warp
The composite parametric warp combines QH with APAP to produce a more natural-looking single-perspective stitching result.
- Composing the QH warp H† with the APAP warp H∗ produces a more natural-looking single-perspective result.The homography warp H is extrapolated from H∗ into H†.
IV. SINGLE-PERSPECTIVE MESH DEFORMATION
The mesh-based warp models deformed sample points through bilinear interpolation and optimizes a total energy covering alignment, naturalness, distortion, and saliency.
- The deformation optimizes the reshaped vector of mesh vertices after constructing indexed grids for the target image.The original and deformed vertices are represented as coordinate vectors V and V-hat.
- Each sample point is represented by bilinear interpolation of its four enclosing grid vertices, transferring point constraints to vertex correspondences.The same representation is used after deformation with fixed interpolation coefficients.
- The mesh-based warp addresses alignment, naturalness, distortion, and saliency by minimizing a combined total energy.Its energy contains point, line, cross-line, and saliency terms.
C. Naturalness Term
The naturalness term strengthens line correspondences within the mesh-based warp, while cross-line constraints preserve perspective and mitigate projective distortion.
- Line correspondences represent target line segments through their endpoints and reference-image line equations.These correspondences provide the basis for line-based mesh constraints.
- The cross-line energy preserves perspective from a homography prior while mitigating projective distortion.It is divided into perspective-preserving and distortion-mitigating components.
- Cross-line features are uniformly sampled in overlapping and non-overlapping regions to construct distortion-related constraints.The figure distinguishes overlapping samples in red from non-overlapping samples in green.
- The distortion formulation preserves slopes of corresponding cross-lines and ratios of lengths along the associated lines.The construction uses line directions calculated from a homography warp prior.
E. Saliency Term
The saliency term protects salient lines from bending in the mesh-based warp.
- Salient-line constraints use line normals calculated from a homography warp prior.The saliency formulation is expressed through a matrix of bilinear-interpolation coefficient vectors.
- The saliency term protects salient lines from being bent during mesh deformation.Its capability is demonstrated by comparing different values of λ_s.
F. Optimization
The two-image optimization computes feature-based correspondences, extracts salient lines, and minimizes a total energy function to produce and composite the warped image.
- The procedure takes a target image and a reference image as input and returns a stitched image.
- The algorithm matches point and line features, estimates a dual-feature homography, and derives cross-line correspondences.
- It extracts salient line segments and uniformly samples the resulting correspondences before optimization.
- The method solves for mesh vertices by minimizing the total energy function, then warps the target image by bilinear interpolation and blends it with the reference.
B. Multiple-Image Stitching
For multiple-image stitching, the method establishes a reference perspective and jointly optimizes homographies and feature correspondences, using line correspondences to improve naturalness.
- A reference image defines the standard perspective, while homographies for all images are estimated and adjusted in a common coordinate system.
- The saliency weight balances bent salient lines against increasing projective distortion as it becomes too small or too large.
- Line correspondences supplement point correspondences during bundle adjustment, producing more natural-looking results.
- The bundle-adjustment formulation jointly optimizes homographies, point variables, and line endpoints while penalizing correspondence transfer errors.
- The formulation uses DLT initialization and binary correspondence indicators, while normalization prevents frequently matched points or lines from dominating.
2) Simultaneous Optimization:
After bundle adjustment, the method maps target images into the reference perspective and simultaneously optimizes alignment, naturalness, distortion, and saliency terms for multiple-image stitching.
- 1) Bundle Adjustment:: Bundle-adjusted homographies transform each target image into the reference perspective and provide cross-line correspondences.
- 2) Simultaneous Optimization:: Figure 7 compares bundle adjustment without and with line correspondences for stitching ten images.
- 2) Simultaneous Optimization:: For multiple-image stitching, alignment and naturalness terms use bundle-adjusted line normals and intermediate point and line correspondences.
- 2) Simultaneous Optimization:: The distortion and saliency terms are determined from homography chains and inverse homographies, respectively.
- 2) Simultaneous Optimization:: The resulting total energy remains sparse and quadratic, allowing efficient minimization with a sparse linear solver.
VI. EXPERIMENTS
The experiments evaluate alignment accuracy quantitatively and naturalness qualitatively against established warps, with the proposed mesh-based warp yielding the lowest errors for most tested image pairs.
- VI. EXPERIMENTS: The broader evaluation compares alignment, naturalness, and time efficiency against homography, APAP, AutoStitch, SPHP, and GSP.
- VI. EXPERIMENTS: The experiments use SIFT features, RANSAC outlier removal, LSD line detection, and a 40 × 40 deformation grid with fixed energy weights.
- A. Quantitative Evaluation of Alignment: Alignment accuracy is measured by RMSE on point correspondences, with separate training and testing sets used to compare the proposed warp against homography and APAP.
- A. Quantitative Evaluation of Alignment: The evaluation also reports the percentage of overlapping-region outliers using a four-pixel neighborhood and a ten-gray-level intensity threshold.
- A. Quantitative Evaluation of Alignment: The proposed warp yields the lowest RMSE and percentage of outliers for most pairs across nine challenging image pairs and twenty repetitions.
B. Qualitative Evaluation of Naturalness
The proposed warp produces more natural-looking stitching than the compared methods across qualitative examples, while its runtime is comparable to GSP and convergence depends on initialization.
- C. Time Efficiency: Homography initialization yields much faster convergence than the other tested initializations.This comparison is shown by the residual-iteration curve in Figure 10(a).
- C. Time Efficiency: Our warp and GSP have comparable elapsed times because both solve mesh deformation simultaneously.The timing includes mesh optimization, texture mapping and linear blending, but excludes feature detection and matching.
- D. Failure Cases: The proposed warp can fail when excessive overlap-region parallax makes a common dominant plane inadequate for representing the perspective transformation.This is the stated failure condition for the method.