PaperlayerView exact revision on arXiv

Source-linked AI summary

WovenAnchor Matcher: Specialized Intra- and Inter-Image Context Modeling for Feature Matching

Zhiyang Li, Ruijiang Jin, Thibaut Klenke, Yusuke Sekikawa, Nakamasa Inoue

arXiv:2610.30799v1cs.CV

TL;DR

Semi-dense matching aggregates context within and across images, but these roles call for different kinds of evidence: spatially structured propagation and retrieval from plausible correspondences. WAM specializes them with WovenMamba and AnchorCrossAttention; on MegaDepth, it raises pose AUC@5° from 64.1 with JamMa to 66.1.

  • Problem

    Intra-image and inter-image context have different requirements: one should preserve spatial evidence, while the other should retrieve plausible correspondences and limit unrelated evidence.

    Source: §1 · p. 1

  • Method

    WAM uses horizontal-then-vertical WovenMamba for intra-image propagation and locally anchored cross-attention for inter-image retrieval.

    Source: §1 · p. 2

  • Results

    Pose AUC@5° on MegaDepth increases from 64.1 with JamMa to 66.1 with WAM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ablations and shared/reversed-role experiments support specializing the operators for intra-image propagation and inter-image retrieval.

    Source: §1 · p. 2

    WovenAnchorModule has higher measured runtime than JointMamba, and AnchorCrossAttention interaction complexity remains quadratic in the number of local windows.

    Source: §5 · p. 10

Abstract

from arXiv · show

Semi-dense local feature matching commonly aggregates contextual information before coarse-to-fine correspondence estimation. However, intra-image and inter-image aggregation serve different roles: the former should propagate spatial evidence within each image while retaining structured two-dimensional dependencies, whereas the latter should gather information from plausible corresponding regions while limiting noise from unrelated locations. We propose WovenAnchor Matcher (WAM), a semi-dense matching framework based on role-specialized context aggregation. WAM introduces two complementary operators. WovenMamba performs intra-image aggregation through horizontal-then-vertical state-space propagation, allowing vertical updates to operate on horizontally contextualized features and thereby encouraging a structured two-dimensional receptive-field bias. AnchorCrossAttention performs inter-image aggregation by using cross-attention to gather context from plausible corresponding regions across the image pair. It performs this exchange over local representative anchors that aggregate feature evidence within each window. On MegaDepth, WAM achieves 66.1 pose AUC@5$^\circ$, improving on the JamMa baseline by 2.0 AUC points under the same evaluation setup. Ablations show that removing or replacing either specialized component reduces accuracy, providing evidence that intra-image propagation and inter-image retrieval benefit from role-specific aggregation designs. Code and pretrained weights are available at https://github.com/mmaiLab/WovenAnchorMatcher.

1 Introduction

WAM treats intra-image propagation and inter-image retrieval as distinct context-aggregation roles, specializing its operators accordingly. On MegaDepth, it improves pose AUC@5° over JamMa, and ablations support the proposed role assignments.

1 Introduction

  • Intra-image aggregation should preserve spatial layout, while inter-image aggregation should retrieve plausible correspondences without importing unrelated evidence.Dense affinities can be unreliable under repetition, viewpoint changes, and local ambiguity.Source: §1 · p. 1
  • WovenMamba composes horizontal then vertical state-space propagation so the vertical stage processes horizontally contextualized features.This encourages composed two-dimensional dependencies that preserve where supporting evidence appears.Source: §1 · p. 2
  • AnchorCrossAttention exchanges context through local representative anchors rather than dense point-wise tokens, allowing different local structures to receive distinct cross-image updates.The design aims to reduce reliance on ambiguous token-level affinities.Source: §1 · p. 2
  • Pose AUC@5° rises from 64.1 with JamMa to 66.1 with WAM on MegaDepth.Removing or replacing either specialized component lowers accuracy.Source: §1 · p. 2
  • WAM frames semi-dense context aggregation as spatially structured propagation within images and correspondence-aware retrieval across images.Its contributions include horizontal-then-vertical WovenMamba and locally grouped AnchorCrossAttention.Source: §1 · p. 2

2 Related work

Related work spans sparse, dense, and semi-dense matching, with WAM situated in the semi-dense, coarse-to-fine family. Its distinction from prior context aggregation designs is to specialize within-image propagation and cross-image retrieval.

Local feature matching

  • Semi-dense matchers balance dense coarse features with refinement of selected correspondences; WAM is most directly related to JamMa’s coarse-to-fine framework.Sparse methods detect keypoints, while dense methods estimate correspondences over denser grids and can incur higher computational cost.Source: §2 · p. 2

Context aggregation for semi-dense matching

  • LoFTR uses self- and cross-attention, JamMa uses shared JointMamba, and WAM separates intra-image and inter-image aggregation into specialized components.Figure 2 is a schematic of coarse context blocks, not a layer-by-layer implementation.Source: Figure 2 · p. 3
  • WAM follows JamMa’s setting but replaces a shared mechanism with structured spatial propagation within images and correspondence-aware aggregation across images.Earlier attention-based methods retain attention aggregation, while JamMa uses a Mamba-based module.Source: §2 · pp. 2–3

State-space models and compact cross-image interaction

  • WovenMamba composes scan orders rather than summing independent directional responses, while compact local representations motivate correspondence-aware cross-image aggregation.Scan order determines the spatial inductive bias of image-feature SSMs; dense cross-attention can mix non-corresponding regions.Source: §2 · p. 3

3 Method

WAM keeps a coarse-to-fine matching framework while replacing generic context aggregation with specialized intra-image propagation and inter-image retrieval. WovenMamba composes horizontal and vertical scans, while AnchorCrossAttention exchanges context through local anchors.

3.1 Overview

  • Each WovenAnchorModule stage applies WovenMamba independently to both images, then AnchorCrossAttention and a feed-forward block; updated coarse features support coarse matching and fine features support local refinement.WAM retains a coarse-to-fine framework while separating intra-image propagation from inter-image context aggregation.Source: §3.1 · p. 3

3.2 WovenMamba for intra-image aggregation

  • WovenMamba composes horizontal-then-vertical state-space scans, so the vertical scan processes features already contextualized horizontally rather than adding independent directional responses.The method uses row-major and column-major flattenings for the horizontal and vertical scans, respectively.Source: §3.2 · p. 4, Figure 3 · p. 4
  • The simplified receptive-field analysis distinguishes WovenMamba’s direction- and distance-dependent two-dimensional support from the additive scan-and-sum baseline.The dominant-path approximation is for interpretation; the stated main comparison is between the additive baseline and the exact composed kernel.Source: §3.2 · p. 5, Figure 4 · p. 5

3.3 AnchorCrossAttention for inter-image aggregation

  • AnchorCrossAttention pools local-window features into multiple anchors, exchanges cross-image context between anchors with matching IDs, then projects updates to features and fuses overlapping windows.Soft-clustering assignment probabilities are reused for anchor-to-feature projection, and normalized overlap weights combine window updates on the dense grid.Source: §3.3 · pp. 5–6, Figure 5 · p. 6

4 Experiments

WAM is evaluated on MegaDepth, zero-shot ScanNet and HPatches, controlled component and operator ablations, and visualizations. The results compare its benchmark performance and role-specialized modules, while visualizations illustrate learned assignments, receptive fields, and matching failures.

4.1 Experimental setup

  • With LO-RANSAC and pose AUC on MegaDepth-1500 and ScanNet-1500, evaluation also covers zero-shot HPatches homography and batch-1 runtime.MegaDepth images are resized and padded to 832 × 832; ScanNet images are resized to 640 × 480.Source: §4.1 · p. 7

4.2 Benchmark comparison

  • 66.1 AUC@5◦ for WAM† versus 64.1 for JamMa and 64.5 for EDM; full-matcher runtime is 35.4 ms versus 43.8 ms for JamMa.WAM† is the checkpoint whose AUC@5◦ equals the median over seven runs.Source: §4.2 · p. 7

4.3 Controlled ablations

  • Removing WovenMamba lowers AUC@5◦ from 66.1 to 64.6, removing AnchorCrossAttention to 64.0, and removing both to 62.0.Table 3 reports pose-error AUC under WAM component removals on MegaDepth-1500.Source: Table 3 · p. 8, §4.3 · p. 8

4.2 Benchmark comparison

  • MegaDepth-trained weights improve over JamMa on zero-shot ScanNet relative pose and HPatches homography, and remain competitive with EDM.Table 2 reports relative pose AUC at angular-error thresholds and homography AUC at corner-error thresholds in pixels.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Controlled ablations

  • With ACA fixed, all directional-scan alternatives trail WovenMamba, while FourMamba adds runtime and Self-Attention is slower and less accurate.The comparison supports a better accuracy–efficiency trade-off for woven horizontal-then-vertical propagation than independent scans or self-attention.Source: §4.3 · p. 8, Table 4 · p. 9
  • With WovenMamba fixed, AnchorCrossAttention has the best accuracy among inter-image alternatives; dense CrossAttention has lower AUC and higher full-matcher runtime.CrossMamba uses paired-image context but lacks explicit correspondence-aware retrieval based on cross-image feature affinities.Source: Table 5 · p. 9, §4.3 · p. 9
  • Using one operator family for both roles or reversing the assignment reduces pose AUC relative to WovenMamba intra-image and ACA inter-image aggregation.The tested alternatives are AnchorSelfAttention and CrossWovenMamba.Source: Table 6 · p. 9, §4.3 · p. 9

4.4 Visualizations

  • Visualizations show consistent matches under appearance or viewpoint changes, but errors in repetitive or weakly constrained regions; learned anchors and receptive fields show differentiated spatial responses.WovenMamba emphasizes a coherent two-dimensional neighborhood, while OrthMamba shows stronger row-and-column ridges; ACA-2 anchors show distinct spatial preferences.Source: §4.4 · pp. 9–10

5 Discussion and limitations

The discussion identifies computational cost, scalability, and the matching strategy as limitations of WAM in the evaluated settings.

5 Discussion and limitations

  • WovenAnchorModule has higher measured runtime than JointMamba, and ACA interaction complexity remains quadratic in local-window count, limiting scalability at high resolutions.Although ACA reduces interactions relative to dense token-wise cross-attention, the paper identifies its remaining complexity as a scalability limit.Source: §5 · p. 10
  • The JamMa-style nearest-neighbor strategy may produce many-to-one correspondences and may contribute to WAM’s localization gap with LoFTR in Aachen.The reported comparison does not isolate the effect of match filtering.Source: §5 · p. 10
  • WAM generally improves over JamMa under the same matching strategy, but the larger localization evaluation is diagnostic rather than an isolation of match-filtering effects.Appendix G evaluates WAM in a larger localization pipeline.Source: §5 · p. 10

6 Conclusion

The conclusion presents WAM as a semi-dense matcher with specialized intra-image and inter-image context aggregation. Its controlled ablations and benchmark results support matching the aggregation design to each role.

6 Conclusion

  • WovenMamba uses horizontal-then-vertical SSM propagation, while AnchorCrossAttention retrieves cross-image context through locally grouped anchors.The conclusion describes these as WAM’s specialized components for intra-image and inter-image modeling.Source: §6 · p. 10
  • Controlled ablations show that both forms of context are useful and that the proposed designs are better suited to their respective roles.The simplified receptive-field analysis suggests a structural bias toward composed two-dimensional dependencies for WovenMamba.Source: §6 · p. 10
  • Benchmark results show that WAM improves over JamMa under the evaluated setup.The conclusion supports designing semi-dense matchers with context modules matched to intra-image and inter-image requirements.Source: §6 · p. 10

A Implementation and evaluation details

The implementation details specify WAM’s feature grids, module settings, matching and evaluation protocols, and runtime measurements. They also describe separator-token handling and its reported effect across benchmarks.

A Implementation and evaluation details

  • WAM uses two module stages by default, with coarse features at Cc = 256, fine features at 64, and a 512-dimensional WovenMamba branch.ACA uses overlapping 8 × 8 windows at stride 4, K = 4 anchors, and anchor dimension D = 256.Source: A Implementation and evaluation details · p. 13
  • Separator tokens are inserted at scan boundaries, processed by the SSM, then removed after unscanning; they have little MegaDepth effect but improve zero-shot ScanNet results.The separator step-size parameter receives a fixed offset of 6.0; the paper treats this as an implementation detail.Source: A Implementation and evaluation details · p. 13, Table 7 · p. 13
  • Coarse matching unions directional nearest-neighbor matches above probability 0.2; fine matching uses 5 × 5 windows and a 0.1 confidence threshold.The matching thresholds are shared across the main benchmark results and ablations unless otherwise stated.Source: A Implementation and evaluation details · p. 13
  • Runtime is measured on one H100 at batch size 1 with 832 × 832 MegaDepth pairs, after 50 warm-up steps; end-to-end timing excludes image I/O and RANSAC.WovenAnchorModule takes 14.0 ms versus 2.9 ms for JointMamba; coarse matching takes 4.7 ms for WAM versus 20.9 ms for JamMa.Source: A Implementation and evaluation details · p. 14, Table 8 · p. 14

B Spatial resolutions in WAM

  • Both module stages preserve the dense 1/8 coarse grid; overlapping ACA windows sample centers four coarse cells apart and create 2,500 anchors per 832 × 832 input.Cross-attention is applied separately to 625 anchors in each anchor-ID group, and padded image regions are masked.Source: B Spatial resolutions in WAM · p. 14

C Trained-model effective receptive fields

The trained-model ERFs show WovenMamba’s coherent two-dimensional neighborhood responses recurring across image pairs and spatial blocks, unlike OrthMamba’s more row- and column-aligned responses. The section also reports anchor-count and window-size sensitivity, with the default configuration balancing accuracy and computation within the tested range.

C Trained-model effective receptive fields

  • The ERF maps show relative spatial influence, using a shared color range of [−6, 0] rather than absolute gradient magnitudes.Gradients are taken with respect to each block’s input, and padded positions are masked.Source: C Trained-model effective receptive… · p. 15
  • WovenMamba’s coherent two-dimensional neighborhood responses contrast with OrthMamba’s more pronounced row- and column-aligned responses across both image pairs and spatial blocks shown.The ERFs use trained-model parameters, nonlinearities, gating, and normalization; second-block maps measure local sensitivity at a later contextualized representation, not gradients through the full matcher to the image.Source: C Trained-model effective receptive… · pp. 14–15, Figure 7 · p. 16, Figure 8 · p. 16

D AnchorCrossAttention details

  • AnchorCrossAttention uses overlapping windows with multiple anchors, whose appearance summaries pool raw and normalized features using soft assignment weights.The assignment-weighted center and covariance encode each anchor’s spatial location and spread; a learned linear layer maps these statistics to geometry features.Source: D AnchorCrossAttention details · p. 18
  • Removing learned assignments, geometry features, or adaptive overlap weighting reduces AUC@5° in the evaluated MegaDepth-1500 configuration.The overlap weights use assignment entropy, with learned parameters rather than a fixed preference for lower-entropy assignments.Source: D AnchorCrossAttention details · pp. 18–19, Table 10 · p. 19
  • ACA-1 has nearly balanced mean assignments, whereas ACA-2 has less uniform means and larger population variances for all four anchors.The assignment statistics pool valid token-window entries across both images and evaluation pairs, counting occurrences in overlapping windows separately; they do not establish fixed semantic categories.Source: D AnchorCrossAttention details · p. 19, Table 11 · p. 19
  • 66.3 AUC@5° at 41.3 ms for window size 4 compares with 66.1 at 35.4 ms for the default window size 8; K = 4 has the highest AUC@5° among tested anchor counts.Increasing K beyond 4 adds runtime without improving accuracy, so the default balances accuracy and computation within the tested range.Source: D AnchorCrossAttention details · p. 19

D.1 Anchor assignment visualizations and analysis

The assignment visualizations show recurring anchor preferences across the two views of each image pair, with more distinct spatial responses in ACA-2. They display individual anchor probabilities and selected windows across four window-offset subsets.

D.1 Anchor assignment visualizations and analysis

  • Figures 11–14 show image pair 1 and Figures 15–18 show image pair 2, with rows for the two ACA blocks and columns for selected windows, the highest-probability anchor, and probabilities for anchors 0–3.Heatmaps use a common [0, 1] scale; categorical maps show the highest-probability anchor, with padded positions and locations outside selected windows masked.Source: §D.1 · p. 19, Figure 11 · p. 20, Figure 15 · p. 22
  • The four visualized window subsets use offsets (0, 0), (0, 4), (4, 0), and (4, 4), with stride 8 within each subset.The underlying windows are 8×8 with stride 4, and each subset is visualized separately to retain its window-specific assignments.Source: §D.1 · p. 19, Figure 12 · p. 21, Figure 13 · p. 21
  • Recurring anchor preferences appear across the two views of each pair, with more distinct spatial responses in ACA-2.These maps complement assignment statistics and do not establish fixed semantic categories.Source: §D.1 · pp. 19–20

E Repeated-run variability

Seven independent WAM training runs on MegaDepth-1500 assess repeated-run variability. The default checkpoint, Run D, has the median AUC@5° across the runs, and all non-default component variants fall below their minimum observed AUC@5°.

E Repeated-run variability

  • Run D is the default checkpoint, and its AUC@5° equals the median across the seven independent MegaDepth-1500 training runs.The run identifiers are arbitrary; “std” in Table 13 denotes sample standard deviation.Source: E Repeated-run variability · p. 25, Table 13 · p. 25
  • All non-default component-removal and replacement variants in Tables 3–5 are below the minimum WAM AUC@5° observed across seven independent runs.The runs use the same MegaDepth-1500 configuration; Run D, used in the main tables, has the median AUC@5° among them.Source: E Repeated-run variability · p. 25, Table 13 · p. 25

F Input-resolution scalability

Across tested MegaDepth input resolutions, WAM obtains higher pose AUC than JamMa. Its context module has higher measured runtime than JointMamba at every tested resolution, and AnchorCrossAttention retains quadratic interaction complexity in the number of local windows.

F Input-resolution scalability

  • WAM obtains higher pose AUC than JamMa at all tested input resolutions.Table 14 reports pose AUC and runtime across resize-and-pad resolutions; the checkpoints are trained on MegaDepth.Source: Table 14 · p. 25, F Input-resolution scalability · p. 25
  • WovenAnchorModule has higher measured context-module runtime than JointMamba at all tested resolutions, while AnchorCrossAttention retains quadratic interaction complexity in the number of local windows.Context time isolates aggregation-module cost; full-matcher time includes the complete implementation, including coarse-matching optimizations.Source: F Input-resolution scalability · pp. 25–26, Table 14 · p. 25

G Additional visual localization evaluation

WAM is evaluated for Aachen visual localization under a shared HLoc-based pipeline, where it generally improves over JamMa but does not outperform LoFTR. The comparison is controlled within this evaluation setup rather than a reproduction of each method’s official leaderboard submission.

G Additional visual localization evaluation

  • All methods use the same localization settings and HLoc-based evaluation code, so results are controlled comparisons rather than reproductions of official leaderboard submissions.The paper notes that detailed settings for some published matcher baselines are not always fully specified.Source: G Additional visual localization… · p. 26
  • Aachen localization recall is reported at (0.25 m, 2°), (0.5 m, 5°), and (5.0 m, 10°).Source: Table 15 · p. 26
  • WAM generally improves over JamMa but does not outperform LoFTR on Aachen Day-Night v1.1 under the evaluated HLoc pipeline.The authors suggest match-selection strategy as one possible reason for the gap.Source: G Additional visual localization… · p. 26

H Limitations and broader impact

The section discusses matching and computational limitations, the potential benefits and harms of local feature matching, and the paper’s reproducibility and evaluation disclosures, including the absence of formal theoretical results and incomplete variability analysis for baselines and ablations. It also notes that human-subjects review was not applicable and that LLMs were not an important, original, or non-standard component of the work.

H Limitations and broader impact

  • Local feature matching may support localization, SfM, SLAM, mapping, image alignment, augmented reality, and robotic perception, but also enable privacy-invasive localization, surveillance, or reconstruction without consent.Incorrect matches may propagate errors to robotic or mapping systems; responsible deployment should consider privacy, consent, benchmark limitations, and downstream failure handling.Source: H Limitations and broader impact · p. 26
  • Nearest-neighbor matching can yield many-to-one correspondences that may contribute to Aachen’s localization gap, and the comparison does not isolate match filtering from feature effects.Localization may depend on both learned features and how correspondences are filtered and aggregated; proposed future directions include mutual filtering, one-to-one assignment, and localization-specific pruning.Source: H Limitations and broader impact · p. 26
  • WovenAnchorModule has higher measured context-module runtime than JointMamba at default resolution, while AnchorCrossAttention remains quadratic in local-window count and limits high-resolution scalability.Local anchors reduce interactions relative to dense token-wise cross-attention; the authors suggest more scalable anchor retrieval as future work.Source: H Limitations and broader impact · p. 26

NeurIPS Paper Checklist

  • The paper reports no formal theoretical results or theorems; its receptive-field discussion is an illustrative simplified linear analysis with stated assumptions.Source: NeurIPS Paper Checklist · p. 29
  • The release provides WAM code and a pretrained checkpoint with setup and evaluation instructions, but omits baseline-specific scripts, the seven-run seed/checkpoint mapping, and Aachen/HLoc integration.Dataset access remains subject to original terms.Source: NeurIPS Paper Checklist · p. 30
  • Seven WAM training runs on MegaDepth-1500 are summarized with mean, sample standard deviation, median, and min–max, but not all baselines and ablations were repeated.Ablation variants are controlled single-run comparisons unless otherwise stated, due to computational cost.Source: NeurIPS Paper Checklist · p. 31
  • The work involved no crowdsourcing or human-subjects research, so IRB approval or equivalent review was not applicable.Source: NeurIPS Paper Checklist · p. 34
  • LLMs were not an important, original, or non-standard component of the core method, experiments, or analysis.Any writing or editing assistance did not affect the methodology, scientific rigor, or originality of the research.Source: NeurIPS Paper Checklist · p. 34

Related papers

Loading 2610.30799v1…