PaperlayerView exact revision on arXiv

Source-linked AI summary

All Graphs Lead to Rome: Learning Geometric and Cycle-Consistent Representations with Graph Convolutional Networks

Stephen Phillips, Kostas Daniilidis

arXiv:1901.02078v1cs.CV

TL;DR

Multi-image feature matching must find correspondences across images, but labeled matches are difficult and expensive to obtain and sparse graph constraints challenge conventional convolutional networks. The paper learns graph embeddings with a GCN trained using cycle consistency and optional geometric losses, and reports promising comparisons with optimization-based baselines, with faster inference at comparable accuracy.

  • Problem

    Multi-image matching seeks corresponding 2D features across images, but hand-labeled correspondences are difficult and expensive to obtain and sparse multi-view constraints challenge conventional convolutional networks.

    Source: §1 · p. 1

  • Method

    A GCN learns embeddings on the correspondence graph using an unsupervised cycle-consistency loss, with geometric consistency losses available during training.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    The method shows promising results compared with cycle-consistency optimization baselines and is faster to comparable accuracy using a single forward pass at inference.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The graph-based approach is agnostic to how correspondences are computed, allowing it to work across a broad class of environments.

    Source: §1 · p. 2

    The embedding dimension must be at least the number of unique scene points, a quantity that is difficult to choose a priori for cycle-consistency methods.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Image feature matching is a fundamental part of many geometric computer vision applications, and using multiple images can improve performance. In this work, we formulate multi-image matching as a graph embedding problem then use a Graph Convolutional Network to learn an appropriate embedding function for aligning image features. We use cycle consistency to train our network in an unsupervised fashion, since ground truth correspondence is difficult or expensive to aquire. In addition, geometric consistency losses can be added at training time, even if the information is not available in the test set, unlike previous approaches that optimize cycle consistency directly. To the best of our knowledge, no other works have used learning for multi-image feature matching. Our experiments show that our method is competitive with other optimization based approaches.

1. Introduction

Multi-image matching can resolve the global inconsistencies that arise from matching image pairs independently, but learning for this task faces scarce labels and graph-structured sparse features. The paper uses a GCN and cycle-consistency training, with geometric losses available during training, and reports promising comparisons with optimization-based baselines.

1. Introduction

  • Pairwise matching can produce global inconsistencies, while sparse multi-view constraints are difficult for conventional convolutional networks to handle.Hand-labeled point correspondences are also difficult and expensive to obtain.Source: §1 · p. 1
  • The method shows promising results against cycle-consistency optimization baselines and is faster to comparable accuracy because inference uses one network forward pass.Experiments compare the method with optimization-based approaches on Rome16K.Source: §1 · p. 2
  • Geometric consistency losses, including epipolar constraints, can aid training even when that information is unavailable at test time.The approach uses cycle consistency to train a neural network to improve pairwise matches in multi-view matching.Source: Figure 1 · p. 1, §1 · p. 2
  • A GCN operates on the correspondence graph and learns graph embeddings to address multi-image feature matching.The learned similarity representation is used to prune erroneous matches.Source: §1 · p. 2, Figure 2 · p. 2
  • Cycle consistency provides an unsupervised training loss, avoiding the need for expensive hand-labeled correspondences.Ground-truth correspondences are difficult and expensive to obtain.Source: §1 · pp. 1–2

2. Related Work

Related work covers established feature-matching pipelines, optimization and clustering for multi-image matching, learning for pairwise matching, and graph neural network architectures.

2.1. Feature Matching

  • SIFT and other hand-crafted descriptors with RANSAC have been widely used, and graph matching can provide a final step for more robust matches.Source: §2.1 · p. 2

2.2. Multi-image Matching

  • Multi-image methods optimize cycle consistency, including spectral embeddings and, more robust matching-matrix optimization [31] and [26], distributed optimization [17], and density-based clustering.The text notes that the low Gaussian noise assumption of spectral approaches is unrealistic.Source: §2.2 · pp. 2–3

2.3. Deep Learning for Matching

  • Prior learning approaches use ground-truth correspondences [30] [28] or address pairwise matching [9] and [29], while Zhu et al. [32] use pairwise rather than multi-image cycle consistency.Source: §2.3 · p. 3

2.4. Graph Neural Networks

  • Graph neural networks include spectral methods requiring a known graph structure [7] and non-spectral methods that do not [6] [16] [12], with neighborhood aggregation and global information passing among approaches.Gama et al. 2018 [13] [12] generalize neighborhood averaging beyond the graph Laplacian.Source: §2.4 · p. 3

3. Method

The method learns feature embeddings from a correspondence graph so pairwise similarities better reflect true matches. It combines graph convolutions with cycle consistency and an optional geometric loss.

3. Method

  • The method learns new feature embeddings from noisy correspondences, aiming for pairwise similarities that reflect true matches.The correspondence graph is the input representation for filtering noisy feature matches.Source: §3 · p. 3

3.1. Correspondence Graph

  • A correspondence graph represents features as vertices and putative matches as weighted edges in its adjacency matrix.The initial embedding concatenates vertex descriptors and can include additional information such as scale or orientation.Source: §3.1 · p. 3

3.2. Graph Neural Networks

  • Graph convolutions learn representations on the irregular correspondence graph, while retaining an embedding for every feature vertex.The method avoids graph coarsening because it needs an output embedding for each vertex; its receptive field grows through multiple layers.Source: §3.1 · p. 4, §3.2 · p. 4

3.3. Cycle Consistency

  • Cycle consistency requires pairwise matches to remain the same regardless of the path between images, and a shared feature universe expresses this with O(n^2) rather than O(n^3) constraints.The network learns embeddings to approximate the universe representation using the noisy correspondence graph; the work uses an L1 loss.Source: Figure 3 · p. 3, §3.3 · pp. 4–5

3.4. Geometric Consistency Loss

  • An epipolar-constraint loss adds geometric consistency during training, including when geometric information is unavailable at test time.The formulation assumes calibrated feature locations and applies the loss to low-rank embeddings, with c(i) denoting the camera for point index i.Source: §3.4 · p. 5

4. Experiments

Experiments test the GCN on synthetic graphs and Rome16K correspondence graphs, where its performance is compared with optimization-based methods. On Rome16K, it is competitive with MatchALS at 25 iterations and faster than PGDDS, while geometric training loss improves testing performance.

4.1. Synthetic Graph Dataset

  • On synthetic graphs, the GCN recovered the true descriptor structure well from noisy inputs, without geometric losses during training.The normalized noisy input descriptors correlated with the true values but did not preserve their structure well.Source: §4.1 · p. 5

4.2. Rome 16K Graph Dataset

  • The method performs better than 25 iterations of MatchALS, but not 50; it also trails PGDDS while running significantly faster.The Rome16K comparison evaluates L1 and L2 errors against ground-truth correspondence graph adjacency matrices.Source: Table 2 · p. 6, §4.2 · p. 7
  • Geometric training loss improves testing performance; Figure 5 also shows state-of-the-art optimization-based baselines as horizontal lines.Even with a simple network, the method compares well with those baselines.Source: Figure 5 · p. 6
  • Rome16K experiments use image triplets and quadruplets with at least 80 overlapping points, with correspondences from bundle adjustment.The dataset contains 16,000 images and was selected because other datasets had insufficient data for network training.Source: §4.2 · pp. 5–6
  • The Rome16K evaluation uses soft labels and compares predictions with ground-truth adjacency matrices rather than standard precision and recall.The network is trained with L1 and geometric consistency losses.Source: §4.2 · pp. 6–7

5. Conclusion

The conclusion presents GCN feature matching trained with unsupervised cycle-consistency and geometric-consistency losses, and reports effectiveness on Rome16K against traditional optimization-based baselines. Future work includes stronger outlier rejection, higher-order geometric constraints, image-level features, and distributed operation.

5. Conclusion

  • The paper reports effective Rome16K feature matching against traditional optimization-based baselines using a simple GCN.The method combines GCNs with unsupervised cycle-consistency and geometric-consistency losses.Source: §5 · p. 7
  • Future work includes robust losses for outlier rejection, higher-order geometric constraints such as the tri-focal tensor, and learned image-level features.The authors also identify adaptation to a distributed setting as a long-term goal.Source: §5 · p. 7

A. Relative Efficiency Against Optimization Based Methods

The appendix compares optimization methods across iteration counts with 12 iterations of the GCN method, reporting error distributions and runtime. It notes that the L1-optimized method has substantially better L1 loss than L2 losses.

A. Relative Efficiency Against Optimization Based Methods

  • Figure 8 compares optimization methods across iterations with 12 iterations of the GCN method, showing mean errors and one standard deviation.The rightmost plot reports time on a log scale.Source: Figure 8 · p. 9, A. Relative Efficiency Against… · p. 9
  • The GCN method's L1 loss is substantially better than its L2 losses, consistent with optimizing for L1 error.Source: Figure 8 · p. 9

B. Effect of Group Normalization

Group normalization improves training and generalization error, but is difficult to parallelize across graph nodes and is identified as the main barrier to parallelization.

B. Effect of Group Normalization

  • Group normalization improves training and generalization error, although the train/test curves are fairly similar with and without it.Figure 9 reports that distance to ground truth improves greatly with group normalization.Source: B. Effect of Group Normalization · p. 9, Figure 9 · p. 9
  • Because group normalization is not easily parallelized across graph nodes, it is the main barrier to parallelization.The authors identify more-parallelizable alternatives as necessary for improving generalization error.Source: B. Effect of Group Normalization · p. 9

Related papers

Loading 1901.02078v1…