Source-linked AI summary
Online Learning of Correspondences between Images
Michael Felsberg, Fredrik Larsson, Johan Wiklund, Niclas Wadströmer, Jörgen Ahlberg
TL;DR
The paper tackles point correspondence between image views when projection and scene geometry are unknown. It introduces an online CMap method that learns mappings between channel-based density representations using Neyman’s chi-square divergence, and reports fast convergence, high accuracy, and real-time operation across experiments.
Problem
The central problem is finding corresponding points between image sequences when point-set correspondences, projection geometry, and scene geometry are unknown.
Method
CMap iteratively updates a fixed-size linear mapping between channel-vector probability hypothesis densities using Neyman’s chi-square divergence.
Results
The method is reported to converge quickly, achieve high accuracy, run in real time, and outperform state-of-the-art methods in convergence and accuracy across experiments.
Takeaways & Limitations
CMap supports online correspondence learning from incoming image sequences without storing previous learning data.
Takeaways & Limitations
Accuracy and object separability are constrained by channel-representation resolution, basis-function width, and highly correlated object motion.
Abstract
from arXiv · showhide
We propose a novel method for iterative learning of point correspondences between image sequences. Points moving on surfaces in 3D space are projected into two images. Given a point in either view, the considered problem is to determine the corresponding location in the other view. The geometry and distortions of the projections are unknown as is the shape of the surface. Given several pairs of point-sets but no access to the 3D scene, correspondence mappings can be found by excessive global optimization or by the fundamental matrix if a perspective projective model is assumed. However, an iterative solution on sequences of point-set pairs with general imaging geometry is preferable. We derive such a method that optimizes the mapping based on Neyman's chi-square divergence between the densities representing the uncertainties of the estimated and the actual locations. The densities are represented as channel vectors computed with a basis function approach. The mapping between these vectors is updated with each new pair of images such that fast convergence and high accuracy are achieved. The resulting algorithm runs in real-time and is superior to state-of-the-art methods in terms of convergence and accuracy in a number of experiments.
1 INTRODUCTION
The paper addresses correspondence learning between image views when projection and scene geometry are unknown, using an online mapping learned from sequences of point sets. CMap represents uncertain point sets as channel-based densities and updates the mapping incrementally, with reported real-time performance and strong accuracy.
- Approach: CMap maps channel-vector representations of probability hypothesis densities between views, implicitly capturing surface geometry and projection distortions.Each point-set pair produces input and output vectors that are used to update the mapping.
- Problem: Unknown projection geometry, scene geometry, and point correspondences make learning mappings between image point sets a multimodal problem.Correspondences may be many-to-many because of occlusion, transparency, and uncertain feature detections.
- Motivation: The method is designed for settings where geometric models or visual similarity are unavailable, including uncalibrated overlapping cameras and differing sensors.The paper contrasts this scope with geometric methods requiring known projection models and appearance methods requiring comparable visual information.
- Main contributions: CMap uses Neyman’s chi-square divergence and fixed-size state to perform proper online learning without storing previous learning data.The update uses the model, the empirical output density, and the weighted residual of incoming data.
- Reported results: Across surveillance datasets and standard benchmarks, CMap is reported to achieve good accuracy, few position-estimate outliers, low residuals, and real-time operation.The introduction summarizes low computational cost on benchmarks and real-time performance on surveillance datasets.
2 METHODS
The method represents point-set uncertainty as channel-based PHDs and learns a sparse CMap that maps one image representation to another, including multimodal and nonlinear correspondences. Online updates optimize Neyman’s chi-square distance without accumulating data, enabling constant-size real-time learning.
- 2.1 Representations of PHDs with channels: Channel representations encode PHDs as smooth soft-histograms, assigning samples to several nearby bins with distance-dependent weights.The basis functions are non-negative, smooth, compactly supported, and arranged on a regular 2D grid.
- 2.1 Representations of PHDs with channels: Cos^2 basis functions with compact support are used to construct channel vectors from image coordinates on the regular grid.The grid uses spacings d1 and d2, with support size 3d1 × 3d2.
- 2.1 Representations of PHDs with channels: The channel representation reduces histogram quantization effects by up to a factor of 20, allowing fewer bins, higher accuracy, or a combination of both.This benefit follows from replacing hard binning with smooth bins.
- 2.1 Representations of PHDs with channels: A channel vector is decoded by filtering its reshaped 2D array with a 3 × 3 box filter and refining the maximum within the selected window.The maximum-response block determines the coarse location, while channel coefficients determine the sub-bin position.
- 2.2 CMap estimation using PHDs: The CMap maps an input PHD channel vector w to an output vector v, preserving multimodal or non-continuous mappings that ordinary regression can average away.The output estimate is decoded to a position from the predicted channel vector.
- 2.2 CMap estimation using PHDs: CMap learning uses Neyman’s chi-square divergence, which yields a weighted least-squares objective and an exact minimizer for the update.For channel vectors, the Mahalanobis interpretation is only approximate because overlapping basis functions weakly correlate bins.
- 2.3 The new online CMap learning algorithm: The online update keeps all entities at constant size, does not accumulate learning data, and can operate for hours in real time.The method can also combine noninterfering detections when they are sufficiently separated and move independently.
- 2.4 Properties of CMap learning: CMap learning produces the simplest mapping consistent with the data, yielding unimodal output when the learned problem is unimodal.This behavior is linked to the equivalence between PHD evolution and single-target correspondence learning.
3 EXPERIMENTS
CMap learning was evaluated across five datasets representing planar, curved, discontinuous, synthetic, and real surveillance settings. It generally converged quickly, maintained real-time performance, handled outliers and nonplanar surfaces, and often outperformed or matched comparison methods.
- Datasets: CMap was evaluated on five datasets covering cross 2D data, synthetic camera projections, and three surveillance settings.The surveillance datasets include PETS2001, PROMETHEUS, and a spiral staircase sequence.
- Cross 2D data: On cross 2D data, CMap converged fastest and was at least three times more accurate than LWPR across the tested sample counts.The evaluation used 500 to 10,000 learning samples and normalized mean square error.
- Cross 2D data: CMap and LWPR ran at 30 fps with stable computation time, whereas ROGER slowed substantially as samples accumulated.Learning 4,000 samples took about 3 hours for ROGER and less than a minute for the other methods.
- Synthetic projections: On synthetic projections, CMap accuracy degraded only slightly without correspondences and as surface complexity increased, while handling discontinuities and nonunique solutions.The evaluation included plane, hyperbolic, and two-plane discontinuous surfaces.
- Synthetic projections: For the synthetic flat2 evaluation, the proposed update scheme performed best overall, while ROGER’s performance degraded in the presence of outliers.The comparison examined mean and median absolute error across noise levels and update schemes.
- Surveillance datasets: On PETS2001, CMap converged to about half ROGER’s error and reached essentially the accuracy of normalized DLT homography estimation despite unknown correspondences.CMap required about twice as many samples as ROGER to converge on this varying-local-behavior data.
- Surveillance datasets: CMap found correct correspondences after about 150 learning frames on PROMETHEUS and in nearly all tested frames on the curved, discontinuous spiral-staircase data.PROMETHEUS errors were limited to cases with too few examples or nearby detection outliers, and D5 used about 150 training frames.
4 CONCLUSION
The paper introduces an online CMap learning method and identifies channel-representation resolution as its main limitation. The method also extends beyond 2D correspondence problems to tracking and future higher-dimensional applications.
- Neyman’s chi-square divergence trains a linear model on density representations, providing the basis for an iterative online CMap algorithm.The fixed-size model stores previously acquired data, supporting online learning without retaining all past samples.
- The main limitation is the resolution of the underlying channel representation, which can confuse objects closer than the representation distinguishes.Position accuracy is also limited by basis-function width, although it is reported as an order of magnitude better than channel spacing.
- Excessively high channel resolution can reproduce noise and unnecessarily increase computational burden.The paper gives approximately 20-pixel channel spacing as a compromise among accuracy, confusion, noise suppression, and generalization.
- The CMap learning algorithm is not restricted to 2D correspondence problems and has also been used for tracking.Future work targets combined mapping and tracking, higher-dimensional problems, and robot control.