PaperlayerView exact revision on arXiv

Source-linked AI summary

GIM: Learning Generalizable Image Matcher From Internet Videos

Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, Cheng Wang

arXiv:2402.11095v1cs.CV

TL;DR

Learning-based image matchers often generalize poorly to in-the-wild scenes, and standard data pipelines limit dataset diversity. GIM self-trains a single matcher on internet videos, while ZEB evaluates cross-domain generalization; GIM improves zero-shot matching and downstream-task performance.

  • Problem

    Learning-based image matchers typically need separate indoor and outdoor models and generalize poorly to unseen scenarios, while existing data construction limits dataset diversity and scale.

    Source: §1 · p. 2

  • Method

    GIM self-trains an image matching architecture on internet videos using complementary matching methods, robust filtering, correspondence propagation, and strong augmentations; ZEB provides a diverse zero-shot benchmark.

    Source: §1 · p. 2

  • Results

    GIM's single model improves downstream tasks over in-domain baselines; with 50 hours of YouTube videos, relative zero-shot gains are 9.9% for SuperGlue, 18.1% for LoFTR, and 8.4% for DKM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ZEB's diverse domains expose cross-domain generalization gaps, including cases where advanced hand-crafted methods outperform recent learning-based methods.

    Source: §1 · p. 2

Abstract

from arXiv · show

Image matching is a fundamental computer vision problem. While learning-based methods achieve state-of-the-art performance on existing benchmarks, they generalize poorly to in-the-wild images. Such methods typically need to train separate models for different scene types and are impractical when the scene type is unknown in advance. One of the underlying problems is the limited scalability of existing data construction pipelines, which limits the diversity of standard image matching datasets. To address this problem, we propose GIM, a self-training framework for learning a single generalizable model based on any image matching architecture using internet videos, an abundant and diverse data source. Given an architecture, GIM first trains it on standard domain-specific datasets and then combines it with complementary matching methods to create dense labels on nearby frames of novel videos. These labels are filtered by robust fitting, and then enhanced by propagating them to distant frames. The final model is trained on propagated data with strong augmentations. We also propose ZEB, the first zero-shot evaluation benchmark for image matching. By mixing data from diverse domains, ZEB can thoroughly assess the cross-domain generalization performance of different methods. Applying GIM consistently improves the zero-shot performance of 3 state-of-the-art image matching architectures; with 50 hours of YouTube videos, the relative zero-shot performance improves by 8.4%-18.1%. GIM also enables generalization to extreme cross-domain data such as Bird Eye View (BEV) images of projected 3D point clouds (Fig. 1(c)). More importantly, our single zero-shot model consistently outperforms domain-specific baselines when evaluated on downstream tasks inherent to their respective domains. The video presentation is available at https://www.youtube.com/watch?v=FU_MJLD8LeY.

1 INTRODUCTION

Learning-based image matchers perform strongly on existing benchmarks but generalize poorly to unseen scene types, while current data pipelines constrain dataset diversity and scale. GIM uses internet videos for self-training, and ZEB evaluates zero-shot generalization across diverse domains.

1 INTRODUCTION

  • Image matching supports applications including 3D reconstruction, visual localization, and autonomous driving.The introduction frames image matching as a fundamental computer vision task.Source: §1 · p. 2
  • Learning-based matchers typically require separate indoor and outdoor models, limiting generalization to unseen scenarios and practicality when the scene type is unknown.Existing RGBD- and SfM/MVS-based data construction methods also have limited efficiency and applicability for scaling training data.Source: §1 · p. 2
  • GIM self-trains a single generalizable matcher using internet videos, combining a pretrained architecture with complementary matching methods, robust filtering, correspondence propagation, and strong augmentations.The framework is intended to work with any image matching architecture.Source: §1 · p. 2
  • ZEB is a zero-shot benchmark combining data from 8 real-world and 4 simulated domains to assess cross-domain generalization.On several ZEB domains, advanced hand-crafted methods outperform recent learning-based methods.Source: §1 · p. 2
  • 9.9%, 18.1%, and 8.4% relative zero-shot performance improvements for SuperGlue, LoFTR, and DKM, respectively, follow training with 50 hours of YouTube videos.The paper also reports downstream-task improvements, including visual localization, homography estimation, and 3D reconstruction, even against in-domain baselines on their respective domains.Source: §1 · p. 2

2 RELATED WORK

Prior work spans hand-crafted matching, learned features and matchers, and data strategies for generalizable vision models. The paper positions GIM and ZEB as responses to domain-specific training and limited cross-domain data.

Image matching methods

  • Hand-crafted methods can be sparse and yield outliers on challenging inputs, while learning-based matchers improve accuracy and density but often generalize poorly beyond indoor or outdoor training domains.RootSIFT performs better than recent learning-based methods in many in-the-wild scenarios; SGP uses self-training but remains domain-specific, and robust fitting alone does not effectively improve generalization.Source: §2 · p. 3

Image matching datasets

  • Existing datasets and evaluation are domain-limited: MegaDepth centers on tourist landmarks, ScanNet covers indoor school scenes, and in-domain tests omit diverse cross-domain conditions.The paper proposes internet videos as a diverse complement and ZEB as a zero-shot benchmark for measuring generalization.Source: §2 · p. 3

Zero-shot computer vision models

  • Other generalizable vision efforts use large, diverse training data, and this work applies that data-generation strategy to image matching through self-training on internet videos.The cited examples include CLIP's internet image-text corpus, mixed depth datasets and 3D-movie labels, and SAM's data engine.Source: §2 · p. 3

3 METHODOLOGY

GIM uses internet videos to self-train a single matcher, enhancing labels from nearby frames for training on distant pairs. ZEB supplies diverse zero-shot evaluation data and reports pose-based metrics across domains.

3 METHODOLOGY

  • GIM starts from a standard-trained architecture and uses internet-video frames to generate, filter, propagate, and augment matching supervision for a generalizable model.The framework is presented as applicable to any image matching architecture.Source: §3 · p. 3, Figure 2 · p. 4

3.1 SELF-TRAINING

  • Nearby-frame labels combine a standard-trained base matcher with complementary methods; robust fitting filters outliers before the labels are propagated to distant frames.Base correspondences are generated at 20-, 40-, and 80-frame intervals, then fused with complementary matching outputs.Source: §3.1 · p. 4
  • Correspondences are propagated through shared matched pixels, extending supervision until pairs have no more than 1024 correspondences; strong random perspective augmentations are used during final training.GIM keeps the most distant pairs meeting the correspondence threshold; strong augmentations were found empirically to provide better supervision signals.Source: §3.1 · pp. 4–5
  • With 16 A100 GPUs, GIM processes 12.5 hours of video per day, and average processing time per frame does not increase significantly with video length.The paper attributes video-data processing workload primarily to running matching methods.Source: §3.1 · p. 5

3.2 ZEB: ZERO-SHOT EVALUATION BENCHMARK FOR IMAGE MATCHING

  • ZEB combines 8 real-world and 4 simulated datasets; its approximately 46K image pairs span scenes and overlap ratios, and evaluation reports pose-error AUC within 5° and average ranking across twelve datasets.The pose error is the maximum of rotation and translation angular error; relative poses are estimated using correspondences and RANSAC.Source: §3.2 · p. 5, Table 1 · p. 6

4 EXPERIMENTS

The experiments evaluate GIM on zero-shot and in-domain image matching, ablate its components, and test downstream applications across three architectures.

4.1 MAIN RESULTS

  • On ZEB, GIM raises AUC from 31.2 to 34.3 for SuperGlue, 33.1 to 39.1 for LoFTR, and 46.2 to 49.4 for DKM.Each GIM model is a single zero-shot model and significantly outperforms the best in-domain baseline.Source: §4.1 · p. 6
  • GIMDKM finds reliable correspondences and reconstructs dense, accurate point clouds where DKM (IN) fails on large view changes or small overlaps.This improvement is shown for challenging indoor and outdoor scenes.Source: §4.1 · p. 7, Figure 3 · p. 7
  • GIMDKM registers all three point-cloud pairs from BEV images, although those images were never seen during training.The pairs come from a real mapping application aligning point clouds of different building levels in the same horizontal plane.Source: §4.1 · pp. 7–8, Figure 4 · p. 7
  • In internet-video multi-view reconstruction, GIM lets DKM recover more of the scene with denser, less noisy point clouds.The experiment replaces SIFT matches in COLMAP reconstructions with matches from the tested models.Source: §4.1 · p. 8, Figure 5 · p. 8
  • On standard in-domain datasets, GIM performs best on average across indoor and outdoor scenes, though its gain is smaller than on ZEB.The authors attribute the smaller improvement to individual baselines fitting in-domain data well.Source: §4.1 · p. 8

4.2 ABLATION STUDY

  • GIM performance falls as video data shrinks, and label propagation matters substantially: without it, 50 hours underperform full GIM on 12.5 hours.Using only RootSIFT for video labels reduces performance slightly; generating labels on more diverse images matters more than using advanced base label generators.Source: §4.2 · pp. 8–9
  • With roughly one day of matched compute, COLMAP labels only 2.2 video hours and improves DKM AUC from 46.2% to 46.5%, versus 12.5 labeled hours from GIM.COLMAP processes 3.9 hours and fails to reconstruct 44.3% of them.Source: §4.2 · p. 9

4.3 APPLICATIONS

  • On downstream tasks, LoFTR gains 4.7%, 4.2%, and 3.4% across homography metrics, while GIMDKM improves DKM pose accuracy by >5% at (0.25m, 2°) indoors and outdoors.The visual localization results support deploying one GIM model across indoor and outdoor environments.Source: Table 3 · p. 8, §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

The conclusion presents GIM as a way to learn generalizable image matching from internet videos and introduces ZEB for zero-shot evaluation. Across three architectures, performance improves with more video data, with benefits reported for downstream tasks and cross-domain deployment.

5 CONCLUSION

  • GIM uses abundant internet videos for self-training, using enhanced outputs of domain-specific models to train the same architecture.The authors report applying it to three state-of-the-art architectures.Source: §5 · p. 10
  • ZEB is a zero-shot benchmark designed to evaluate image matching models in in-the-wild environments.The conclusion describes it as enabling thorough evaluation of generalization.Source: §5 · p. 10
  • The authors report that performance improves steadily with video data size and that one GIM model generalizes across applications and domains.Improved matching also benefits visual localization and 3D reconstruction.Source: §5 · p. 10

A DETAILS OF VIDEO DATA

The video-data appendix describes diverse downloaded training videos and gives examples of the images and correspondences used to generate labels.

A DETAILS OF VIDEO DATA

  • Figure 6 samples the training images and generated correspondences, covering indoor and outdoor, urban and natural scenes, and varied illumination.The appendix describes the examples as drawn from the video data.Source: A DETAILS OF VIDEO DATA · p. 10, Figure 6 · p. 10
  • The downloaded videos span scene types from 26 countries around the world.The table caption links this range to diversity in GIM's training data.Source: Table 6 · p. 11

B DETAILS OF ZEB

The ZEB appendix describes a benchmark assembled from datasets spanning varied domains, scene conditions, and image resolutions.

B DETAILS OF ZEB

C IN-DOMAIN EVALUATION RESULT

On MegaDepth-1500 and ScanNet-1500, GIM achieved the best average in-domain performance across indoor and outdoor scenes. The smaller gap between methods in-domain than in zero-shot evaluation underscores ZEB’s value for assessing generalization.

C IN-DOMAIN EVALUATION RESULT

  • GIM achieved the best average performance over indoor and outdoor scenes on in-domain evaluation data.The evaluation used MegaDepth-1500 and ScanNet-1500; Table 8 reports the in-domain results.Source: C IN-DOMAIN EVALUATION RESULT · p. 11, Table 8 · p. 12
  • The smaller performance gap in-domain than in the zero-shot scenario shows the importance of ZEB for reflecting generalization performance.Source: C IN-DOMAIN EVALUATION RESULT · p. 11
  • ZEB samples cover varied scene types, viewpoints, and lighting conditions to evaluate matching robustness.Source: Figure 7 · p. 12

D FURTHER QUALITATIVE RESULTS

Qualitative evaluations show that in-domain LoFTR and SuperGlue generalize poorly to challenging in-the-wild images and BEV point-cloud registration.

D FURTHER QUALITATIVE RESULTS

Loading 2402.11095v1…