PaperlayerView exact revision on arXiv

Source-linked AI summary

MatcherCompass: A Deployment-Aware Benchmark to Guide Image Matcher Selection in the Wild

Hyunwoo Kim, Giseop Kim

arXiv:2609.25688v1cs.RO

TL;DR

Field robots need image matching that works across sensing conditions while staying within onboard resource budgets, but single-device reports provide limited guidance for choosing configurations. MATCHERCOMPASS benchmarks matchers across visual conditions and hardware, linking accuracy with resource costs. Its findings show that matcher rankings and feasible choices depend on modality, resolution, hardware, and precision.

  • Problem

    Single-device accuracy and runtime reports provide limited guidance for choosing matcher configurations for field robots operating across visual conditions and onboard resource budgets.

    Source: §I · p. 1

  • Method

    MATCHERCOMPASS evaluates nine pipelines across visual tasks, resolutions, supported precisions, and four GPU platforms, linking pose AUC with runtime, GPU memory, and energy per pair.

    Source: §I · pp. 1–2, B. Accuracy and Resource Metrics · p. 3, E. Runtime, Memory, and Energy… · p. 4

  • Results

    Matcher rankings differ across modalities: at 1024 px on visible–thermal pairs, MatchAnything leads with pose AUC@10° of 22.9% versus ELoFTR’s 6.7%.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The measurement-based guide lets users compare configurations meeting time and resource constraints for a selected visual condition and platform.

    Source: §I · p. 2, §VII · p. 8

Abstract

from arXiv · show

Field robots operating across time of day and sensing modalities require accurate image correspondences within onboard time and resource budgets. However, accuracy and runtime reported for individual methods on a single device provide limited guidance for choosing a matcher and its configuration on a target platform. We present MatcherCompass, a deployment-aware benchmark for choosing local feature matchers in field robotics. Under common input and pose-evaluation procedures, we compare nine classical and learned matching pipelines across four image resolutions and supported numerical precisions. Four visual conditions cover viewpoint variation, day--night matching in visible and thermal imagery, and daytime visible--thermal matching. We evaluate pose accuracy using the area under the error--recall curve (AUC) at $5^\circ$, $10^\circ$, and $20^\circ$, and measure runtime, GPU memory, and energy per image pair on four GPU platforms spanning workstation and onboard computers. The results show that changes in hardware, input resolution, and numerical precision can move a matcher across a runtime budget boundary, altering the feasible choices. We organize the measurements into a selection guide that returns all configurations satisfying user-specified time and resource limits, together with their accuracy under the selected visual condition. MatcherCompass provides measured evidence for choosing matching pipelines that fit a robot's sensing conditions and computing hardware. Project page: https://matchercompass.github.io/.

I. INTRODUCTION

Field-robot matcher selection must account for changing viewpoint, time of day, and sensing modality while meeting onboard time, memory, and energy budgets. MATCHERCOMPASS compares configurations under shared evaluation procedures and provides a guide for finding those that meet deployment limits.

I. INTRODUCTION

  • A deployment-oriented comparison must connect condition-specific pose accuracy with runtime, memory, and energy for each matcher configuration on target hardware.Individual examples and single-device accuracy or runtime reports do not establish performance across intended conditions or compliance with onboard budgets.Source: §I · p. 1
  • MATCHERCOMPASS evaluates nine matching pipelines across four visual tasks and four GPU platforms, varying resolution and supported precision under common pose-evaluation procedures.It links pose AUC at 5°, 10°, and 20° to runtime, GPU memory, and energy per pair.Source: §I · pp. 1–2
  • The selection guide returns all measured configurations that satisfy user-specified time and resource limits, together with their accuracy and costs.The measurements support comparisons under 20, 50, and 100 ms runtime budgets.Source: §I · p. 2

II. RELATED WORK

Local feature matchers differ in correspondence density, computational structure, and training coverage, so accuracy alone does not determine their fit to field-robot conditions. Existing benchmarking and deployment tools provide partial support, while MATCHERCOMPASS connects condition-specific accuracy to configuration costs on target hardware.

A. Local Feature Matchers

  • Classical, learned sparse, semi-dense, and dense pipelines offer different correspondence densities, computational structures, and training coverage.Examples include attention-based SuperGlue, adaptive LightGlue, semi-dense LoFTR and ELoFTR, and dense DKM and MASt3R.Source: A. Local Feature Matchers · p. 2

B. Benchmarking and Deployment Support

  • Prior benchmarks and tools assess selected combinations of accuracy, cost, or matcher behavior, but MATCHERCOMPASS connects condition-specific accuracy to individual configuration costs and constraint-based selection.The guide covers matcher, resolution, and precision configurations on a target platform.Source: B. Benchmarking and Deployment Support · p. 2

III. MATCHERCOMPASS

MATCHERCOMPASS links benchmark measurements to deployment-aware matcher selection. It evaluates defined configurations with pose-accuracy and resource metrics, then lets users filter results against deployment constraints.

A. Benchmark Configurations

  • The benchmark evaluates four tasks covering visible-image geometric variation, day–night matching within each modality, and daytime visible–thermal matching.The tasks test accuracy across viewpoint, time-of-day, and modality changes.Source: A. Benchmark Configurations · p. 2
  • Each configuration combines a matcher, input resolution, and supported numerical precision; nine pipelines yield 84 configurations per task.Resolutions are 128, 256, 512, and 1024 px, with supported Native, FP32, MP, and FP16 modes.Source: A. Benchmark Configurations · p. 2

B. Accuracy and Resource Metrics

  • Pose accuracy uses relative-pose rotation and translation-direction errors, summarized as AUC at 5°, 10°, and 20°; failed pose estimates receive infinite error.AUC integrates the cumulative fraction of pairs up to each angle and is reported on a 0–100 scale.Source: B. Accuracy and Resource Metrics · p. 3
  • Computational costs comprise runtime, peak GPU memory, and energy per pair, with resource metrics summarized as medians over four pairs per task and platform.OOM failures are distinguished from unsupported precision modes and other missing measurements.Source: B. Accuracy and Resource Metrics · p. 3

C. MATCHERCOMPASS Web GUI

  • Users select a visual condition, target platform, and resource limits; the Web GUI returns all measured configurations meeting those constraints with their accuracy and costs.A runtime limit can filter the results to configurations that meet the specified budget.Source: C. MATCHERCOMPASS Web GUI · p. 3

IV. EXPERIMENTS

The experiments combine RUBIK geometric evaluation with STheReO temporal and cross-modal matching tasks. They compare two workstation and two onboard GPU platforms with differing compute, memory, and power budgets.

A. Datasets

  • RUBIK supplies geometric-variation pairs, while STheReO supports day–night and daytime visible–thermal tasks using repeated traversals, calibration, and reference trajectories.The evaluation uses 100 pairs per task; STheReO locations are shared across tasks and selected for at least 1° median pairwise parallax.Source: A. Datasets · p. 3, Table I · p. 3

B. Hardware Platforms

  • The evaluation spans two workstations, RTX 5070 Ti and RTX 5060 Ti, and two onboard platforms, Jetson AGX Thor and Orin Nano.The platforms span different compute, memory, and power budgets.Source: B. Hardware Platforms · p. 3, Table II · p. 3

C. Matcher Configurations

MatcherCompass evaluates fixed matcher configurations under common pose-evaluation procedures and measures deployment costs separately. The measurements connect pose accuracy and resource use to platform, resolution, and supported precision choices.

C. Matcher Configurations

  • Matcher configurations use public implementations and pretrained weights without additional training, with fixed settings across datasets and platforms.FP32 and FP16 use corresponding weight and input types without autocasting; MP uses FP32 weights with FP16 autocasting, and TF32 is disabled for learned pipelines.Source: C. Matcher Configurations · pp. 3–4

D. Matching Accuracy via Pose Estimation

E. Runtime, Memory, and Energy Measurement

  • Runtime, peak GPU memory, and energy per pair are measured across the same 16 image pairs on all platforms.Timed calls exclude model loading and pose estimation; runtime is measured separately from memory and energy to avoid tracing overhead.Source: E. Runtime, Memory, and Energy… · p. 4
  • Energy per pair includes idle power, while platform measurements use different power scopes that must be considered when comparing energy.RTX reports GPU board power, Thor the GPU rail, and Orin Nano the combined CPU, GPU, and CV rail.Source: E. Runtime, Memory, and Energy… · p. 4

V. RESULTS

Pose accuracy varies with visual condition, matcher, and resolution. The results show especially low scores for day–night visible matching and different resolution trends across matchers in visible–thermal matching.

A. Accuracy across Visual Conditions and Resolutions

  • On RUBIK, MASt3R reaches 74.5% pose AUC@10° at 512 px, but declines to 28.8% at 1024 px while ELoFTR, MatchAnything, and DKM improve.The resolution yielding the highest accuracy therefore depends on the matcher.Source: A. Accuracy across Visual Conditions… · p. 4
  • At 512 px, no evaluated matcher exceeds 10% pose AUC@10° for day–night visible matching.For thermal day–night matching, SuperPoint+SuperGlue reaches 23.9%, followed by DKM at 20.1%; MASt3R reaches 3.1%.Source: A. Accuracy across Visual Conditions… · p. 4
  • For daytime visible–thermal matching, ELoFTR falls from 26.6% pose AUC@10° at 512 px to 6.7% at 1024 px, while MatchAnything rises from 10.1% to 22.9%.LoFTR and MASt3R also decline as resolution increases.Source: A. Accuracy across Visual Conditions… · p. 4

B. Accuracy–Runtime Trade-offs across Hardware

The reported accuracy–runtime trade-offs on RUBIK differ across hardware platforms. A configuration that meets a workstation runtime budget can exceed it on an onboard platform.

B. Accuracy–Runtime Trade-offs across Hardware

  • Fig. 4(b) compares accuracy–runtime trade-offs across platforms on RUBIK at 256 and 512 px, using each platform’s runtime with accuracy evaluated on RTX 5070 Ti.The plot shows which matcher and resolution combinations meet a target platform’s runtime budget at the desired accuracy.Source: B. Accuracy–Runtime Trade-offs across… · p. 4

C. Computational Cost across Hardware Platforms

  • SuperPoint+LightGlue’s median runtime rises from 12.2 ms on RTX 5070 Ti and 18.3 ms on RTX 5060 Ti to 57.2 ms on Thor and 113.3 ms on Orin Nano.Thus, a configuration that meets a runtime budget on a workstation can exceed it on an onboard platform.Source: C. Computational Cost across Hardware… · p. 4
  • The reported cross-platform runtime comparison uses FP32 for learned matchers and Native precision for ORB (CUDA) and SIFT (GPU).The comparison concerns RUBIK at 512 px.Source: C. Computational Cost across Hardware… · p. 4

SIFT (GPU)

Reduced precision changes runtime, memory, energy, and which configurations fit within resource budgets. Its effects vary by matcher, and the reported accuracy changes must be weighed against resource savings.

SIFT (GPU)

  • On RTX 5070 Ti, SuperPoint+SuperGlue uses 0.330 GiB and 3.00 J per pair, while DKM uses 13.44 GiB and 163.46 J.These are 40.7 and 54.4 times the respective SuperPoint+SuperGlue costs.Source: SIFT (GPU) · p. 5
  • From 512 to 1024 px on RTX 5070 Ti, ELoFTR’s GPU memory rises from 2.19 to 10.24 GiB, while MASt3R’s energy per pair rises from 44.21 to 330.60 J.These cost changes can be compared with resolution-dependent accuracy when choosing input resolution.Source: SIFT (GPU) · p. 5

D. Resource Savings through Reduced Precision

  • On Thor, switching SuperPoint+SuperGlue from FP32 to FP16 reduces runtime by 68%, GPU memory by 65%, and energy per pair by 73%.Its runtime falls from 65.7 ms to 20.9 ms, meeting a 50 ms budget that FP32 exceeds.Source: D. Resource Savings through Reduced… · p. 5
  • On RUBIK at 512 px, SuperPoint+SuperGlue in MP has the largest absolute pose AUC change from FP32 across the reported thresholds: 4.42 percentage points at 5°.The reported accuracy changes must be weighed against runtime savings when selecting precision.Source: D. Resource Savings through Reduced… · p. 5
  • On Orin Nano at 1024 px, ELoFTR and MatchAnything fail with out-of-memory errors in FP32 and MP but run in FP16 with peak CUDA allocations of 4.29 and 4.33 GiB.Whether a configuration can run therefore depends on precision as well as matcher and resolution.Source: D. Resource Savings through Reduced… · p. 5

E. Qualitative Matching Results

The qualitative examples show matcher outputs and pose-estimation outcomes can differ on the same image pair. Deployment examples then connect feasible configurations to accuracy and resource costs under a selected platform and budget.

E. Qualitative Matching Results

  • SuperPoint+SuperGlue estimates a pose within 10° on the displayed daytime visible–thermal example, while SIFT (GPU) returns few or no correspondences in the displayed day–night examples.The figure compares matchers on the same pairs and overlays runtime, peak GPU memory, and energy per pair.Source: E. Qualitative Matching Results · p. 5, Table IV · p. 6

F. Deployment Examples

  • On RUBIK at 512 px, a 20 ms budget on Thor admits ORB (CUDA) in Native and SP+LG in FP16.The guide returns candidates for users to compare by accuracy and resource cost.Source: F. Deployment Examples · pp. 5–6
  • Table IV reports runtime, peak GPU memory, and energy per pair across hardware, using FP32 for learned matchers and Native for ORB (CUDA) and SIFT (GPU).Table V compares pose AUC and runtime across FP32, MP, and FP16 on RUBIK at 512 px.Source: Table IV · p. 6, Table V · p. 6

VI. FINDINGS AND DEPLOYMENT GUIDELINES

The findings show that deployment choices depend jointly on matcher, sensing condition, resolution, precision, and hardware. Accuracy must be weighed alongside runtime, memory, and energy because these factors alter which configurations meet a budget.

Hardware-specific budget choices

  • On Orin Nano at 512 px, SP+LG FP16 offers more timing margin at 77.1 ms, while SP+SG FP16 has higher pose AUC@10° (33.0% versus 27.5%) at 98.7 ms.Both are learned candidates within the 100 ms budget; SP+LG FP16 also meets 20 ms on Thor.Source: §VI · p. 6

Practical benefits of reduced precision

  • On Thor at 512 px, switching SP+SG from FP32 to FP16 reduces runtime, peak GPU memory, and energy by 68%, 65%, and 73%, respectively.Across the stated RUBIK conditions, MP/FP16 pose AUC changes are at most 4.42 percentage points from FP32; LoFTR peak memory changes by less than 1%.Source: §VI · p. 6

Deployment costs beyond runtime

  • On RTX 5070 Ti, DKM uses 40.7 times SP+SG’s GPU memory and 54.4 times its energy; on Orin Nano at 1024 px, ELoFTR and MatchAnything are OOM in FP32/MP but run in FP16.Their FP16 peak memories are 4.29 and 4.33 GiB, respectively, so memory and energy matter alongside latency.Source: §VI · p. 7

Different rankings for cross-modal matching

  • For visible–thermal matching, ELoFTR exceeds MatchAnything in pose AUC@10° at 512 px, but at 1024 px MatchAnything leads, 22.9% versus 6.7%.MASt3R leads on RUBIK at 512 px but not on visible–thermal pairs.Source: §VI · p. 7

Uncertain returns from higher resolution

  • At 1024 versus 512 px, RUBIK accuracy improves for ELoFTR, MatchAnything, and DKM but declines for MASt3R.On RTX 5070 Ti, ELoFTR memory rises from 2.19 to 10.24 GiB and MASt3R energy from 44.21 to 330.60 J; ELoFTR is slower than LoFTR at 128 px but faster at 1024 px.Source: §VI · p. 7

Geometric validity over correspondence count

  • On visible–thermal pairs, ELoFTR FP32 returns more correspondences at 1024 than 512 px, yet pose AUC@10° decreases; pose estimates are returned for all 100 pairs at both resolutions.Correspondence count or successful pose recovery alone does not establish matching quality; the resulting pose errors are the selection criterion.Source: §VI · p. 7

VII. CONCLUSION

MatcherCompass benchmarks nine matching pipelines across visual conditions, resolutions, precisions, and hardware, linking pose accuracy to resource costs. The results show that resolution, hardware, and precision affect accuracy and feasibility, and the guide compares configurations that satisfy resource constraints.

VII. CONCLUSION

  • MatcherCompass evaluates nine pipelines across four visual conditions and four input resolutions, linking pose accuracy to runtime, GPU memory, and energy per pair on four hardware platforms.The benchmark also varies supported numerical precisions.Source: §VII · pp. 7–8
  • The GUI displays feasible configurations as tiles; selecting one reveals its accuracy and resource costs, with the shown example using RUBIK on Jetson AGX Thor under a 100 ms budget.The feasibility figure includes RTX 5070 Ti, RTX 5060 Ti, Jetson AGX Thor, and Jetson Orin Nano.Source: Figure 6 · p. 8, Figure 7 · p. 8
  • Increasing resolution does not always improve accuracy, while hardware and precision change which configurations satisfy time and memory constraints.The conclusion presents a measurement-based guide for comparing feasible configurations by accuracy and resource cost.Source: §VII · p. 8

Related papers

Loading 2609.25688v1…