Source-linked AI summary
RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration
Lei Han, Mengqi Ji, Lu Fang, Matthias Nießner
TL;DR
Direct image alignment has limited convergence and often needs good initialization, while learned-feature methods retain optimization limitations. RegNet jointly learns features and the Jacobian; ablations report increased convergence range, especially for poor initial alignment.
Problem
Direct methods need good initialization to avoid local minima, and direct optimization remains limited in convergence range despite learned features.
Source: §1 · p. 1, §3.3 · p. 4Method
RegNet jointly learns per-pixel feature representations and the Jacobian, replacing numerical differentiation in an end-to-end image-to-image pose optimization system.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The reported improvement concerns direct image-to-image alignment optimization under challenging initial alignment.
Source: §5 · p. 8Numerical differentiation assumes feature smoothness between initialization and the ground-truth match, while enforcing smoothness across large baselines blurs object boundaries.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions. Deep learning approaches has been applied to address this problem by learning better feature representations using convolutional neural networks, yet still require a good initialization. In this paper, we demonstrate that the inaccurate numerical Jacobian limits the convergence range which could be improved greatly using learned approaches. Based on this observation, we propose a novel end-to-end network, RegNet, to learn the optimization of image-to-image pose registration. By jointly learning feature representation for each pixel and partial derivatives that replace handcrafted ones (e.g., numerical differentiation) in the optimization step, the neural network facilitates end-to-end optimization. The energy landscape is constrained on both the feature representation and the learned Jacobian, hence providing more flexibility for the optimization as a consequence leads to more robust and faster convergence. In a series of experiments, including a broad ablation study, we demonstrate that RegNet is able to converge for large-baseline image pairs with fewer iterations.
1. Introduction
Direct image alignment can be undermined by poor initialization, while learned-feature optimization still faces limitations in its optimization step. RegNet jointly learns features and the Jacobian, and experiments report better accuracy and a larger convergence range.
1. Introduction
- Direct methods minimize photometric error but usually need good initialization to avoid local minima, while feature-based methods can struggle in texture-less or repetitive environments..Source: §1 · p. 1
- Learned-feature optimization can improve representations, but conventional Gauss–Newton optimization may favor smooth features and over-smooth object boundaries.Source: §1 · p. 2
- RegNet learns the Jacobian under re-projection-error supervision, allowing it to reshape the cost landscape and support more flexible optimization steps.The authors report more robust and faster convergence.Source: §1 · p. 2
- RegNet jointly learns feature representations and the Jacobian for direct image-to-image registration, with experiments reporting better accuracy and a larger convergence range.The approach targets inaccurate conventional Jacobians under poor initialization and trains the optimization step end to end.Source: §1 · p. 2
2. Related Work
Prior approaches include feature matching, pose regression, and direct image alignment, each with different constraints. RegNet builds on direct optimization by learning both features and the Jacobian while jointly refining depth and relative motion.
2. Related Work
- Image-to-image pose estimation is grouped into feature-based, regression-based, and direct image alignment approaches.Source: §2 · p. 2
- Feature-based methods can be robust to initialization and efficient, but depend on feature detection and may overfit feature-rich areas.Source: §2 · p. 2
- Regression-based methods predict pose or homography from image pairs, while Demon uses per-pixel optical flow to regress motion and dense depth.The paper adopts Demon's large-baseline indoor and outdoor dataset for fair comparison.Source: §2 · p. 2
- Direct alignment methods minimize photometric or feature-metric error, but monocular warping requires depth and conventional direct optimization can be sensitive to initialization.RegNet uses predicted depth, jointly refines depth and motion, and replaces the handcrafted numerical Jacobian with a learned one.Source: §2 · pp. 2–3, Figure 2 · p. 3
3. RegNet
RegNet estimates relative pose by minimizing feature-metric residuals, jointly optimizing motion and a compact depth representation. Its multi-scale feature network and learned Jacobian target the limited convergence of direct methods, with training designed to handle large motions.
3.1. Problem Formulation
- Relative motion T and depth weights w are estimated by minimizing the L2 norm of per-pixel feature-metric residuals, with depth represented as D = ReLU(w^T B).The residual compares learned features at each pixel with features at its warped coordinate; motion and depth weights are optimized using LM.Source: §3.1 · pp. 3–4
3.3. Jacobian Prediction Network
- When initialization is far from ground truth, a learned Jacobian replaces numerical differentiation to improve nonlinear optimization without requiring smooth feature representations across large baselines.The feature network aligns images from coarse to fine; the learned Jacobian uses features from both the first image and the warped second image.Source: §3.1 · p. 4, §3.3 · pp. 4–5
3.2. Feature Learning Network
- The U-Net-like feature network builds multi-resolution maps for coarse-to-fine alignment, using l0–l2 for coarse registration and l3 for precise registration with depth optimization.The maps progress from 16 × 12 with 256 dimensions at l0 to 192 × 128 with 32 dimensions at l3.Source: §3.2 · p. 4
3.4. Training details
- Training proceeds hierarchically: the first layer trains features with the learned Jacobian, while later layers refine alignment using numerical Jacobians after rough convergence.Random pose perturbations train for large motions; bootstrapping switches from log(L + 1) back to L to handle large-baseline pairs, and final-layer training combines alignment and depth losses.Source: §3.4 · pp. 5–6
4. Experiments
Experiments evaluate RegNet on RGBD, MVS, and SUN3D datasets, including quantitative comparisons, ablations, and qualitative examples. Results favor joint feature and Jacobian learning, particularly for challenging initial alignments.
4. Experiments
- The evaluation uses Demon-provided RGBD, MVS, and SUN3D data, compares RegNet with Demon and conventional approaches, and refines correspondence-based estimates by minimizing reprojection error.The data description excludes synthetic Scenes11 because it is not applicable to the depth prediction networks.Source: §4 · p. 6, §4.1 · p. 6
4.1. Quantitative Evaluation
- Across all three datasets, joint feature and Jacobian learning (Ours-LJ) considerably outperforms feature learning with a numerical Jacobian (Ours-NJ).Ours-NJ uses the same network structure but retains numerical rather than learned Jacobians.Source: §4.1 · p. 6
- RegNet (Ours-LJ) performs considerably better on RGBD and SUN3D and comparably on MVS in mean error; median errors also show significantly higher performance on MVS.The reported metrics are rotation and translation-vector error, in degrees.Source: Table 1 · p. 6, Table 2 · p. 6, §4.1 · p. 6
4.2. Ablation Study
- Learned Feature is comparable to RegNet at a large error threshold but fails at more precise alignments, and its success-rate advantage over RegNet with a numerical Jacobian reverses between large and small initial errors.The paper attributes this behavior to the learned features being over-smoothed and less precise than RegNet's features.Source: §4.2 · p. 7
- On SUN3D, RegNet produces the smallest reprojection errors among the four tested methods, while conventional numerical-Jacobian alignment is only slightly better than the unoptimized initialization.Each test sample is run 100 times with random initialization distortions and five optimization iterations.Source: §4.2 · p. 7
- RegNet has the best registration success ratio at every initialization error; above 30% initial reprojection error, it still registers 40% of pairs while most direct methods fail completely.A registration counts as successful when post-optimization reprojection error is below 5%.Source: §4.2 · p. 7, Figure 5 · p. 7
4.3. Qualitative Evaluation
- Across randomly selected RGBD, MVS, and SUN3D pairs, RegNet produces lighter photometric-error maps than Demon in the qualitative comparison.The visualization warps I2 into I1's coordinates using predicted depth and motion parameters.Source: §4.3 · p. 8, Figure 6 · p. 8
5. Conclusion
RegNet jointly learns the Jacobian and feature space for image-to-image alignment. Ablations show improved direct-alignment optimization, especially with poor initial alignment.
- RegNet jointly learns the Jacobian and feature space, increasing convergence range for image-to-image alignment.The ablation study reports improved direct-alignment optimization, especially for challenging poor initial alignments.Source: §5 · p. 8