PaperlayerView exact revision on arXiv

Source-linked AI summary

GlueStick: Robust Image Matching by Sticking Points and Lines Together

Rémi Pautrat, Iago Suárez, Yifan Yu, Marc Pollefeys, Viktor Larsson

arXiv:2304.02008v3cs.CV

TL;DR

Line segments offer structural cues, including in textureless areas, but can be difficult to describe and match. GlueStick jointly matches points and lines in a GNN that exploits wireframe connectivity. The paper reports improved matching across feature types and downstream homography and camera-pose estimation.

  • Problem

    Line matching is challenging because segments can be occluded, poorly textured, or repetitive, and their descriptors are less reliable than point descriptors.

    Source: §1 · p. 1

  • Method

    GlueStick jointly processes points and line endpoints as wireframe nodes in a GNN, enriching their descriptors with within-image, cross-image, and line-connectivity context.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 3

  • Results

    GlueStick improves matching for both points and lines across the experiments, with a significant leap over descriptor-based state of the art for line matching.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The improved matches directly translate to better homography and camera-pose estimation.

    Source: §5 · p. 8

    Performance degrades when low texture coincides with symmetric structures, which make descriptors less reliable and spatial-context disambiguation harder.

    Source: §D.3 · p. 17

Abstract

from arXiv · show

Line segments are powerful features complementary to points. They offer structural cues, robust to drastic viewpoint and illumination changes, and can be present even in texture-less areas. However, describing and matching them is more challenging compared to points due to partial occlusions, lack of texture, or repetitiveness. This paper introduces a new matching paradigm, where points, lines, and their descriptors are unified into a single wireframe structure. We propose GlueStick, a deep matching Graph Neural Network (GNN) that takes two wireframes from different images and leverages the connectivity information between nodes to better glue them together. In addition to the increased efficiency brought by the joint matching, we also demonstrate a large boost of performance when leveraging the complementary nature of these two features in a single architecture. We show that our matching strategy outperforms the state-of-the-art approaches independently matching line segments and points for a wide variety of datasets and tasks. The code is available at https://github.com/cvg/GlueStick.

1. Introduction

Line matching is difficult because segments can be poorly localized, occluded, repetitive, and hard to describe, while lines can provide useful structure where point matching struggles. GlueStick jointly matches points and lines in one network, using their complementary cues and local connectivity; the paper reports improved performance over prior point and line matchers across datasets and tasks.

1. Introduction

  • Line segments are difficult to match because they are poorly localized, less repeatable, and vulnerable to occlusion, perspective effects, and repetition.Classical matching heuristics such as mutual nearest neighbor and Lowe’s ratio test are often less effective for lines.Source: §1 · p. 1
  • GlueStick jointly matches keypoints and line segments in one GNN, using nearby distinctive features to resolve ambiguous matches in either feature type.It represents keypoints and line endpoints as wireframe nodes, with self-attention, cross-attention, and line message passing before separate point and line matching.Source: §1 · p. 2
  • The architecture exploits local feature connectivity and replaces heuristic geometric strategies for line matching with a data-driven approach.The paper presents joint point-and-line matching as a way to leverage the features’ complementary nature.Source: §1 · p. 2
  • GlueStick improves over previous state-of-the-art point and line matchers across a wide range of datasets and tasks.The conclusion also reports improved matching performance for both feature types and a significant leap for line matching over descriptor-based state of the art.Source: §1 · p. 2

2. Related Work

Prior work detects and describes line segments with classical or learned methods, and often supplements descriptor matching with geometric heuristics. GlueStick’s approach represents line endpoints as separate GNN nodes, enabling points and lines to contribute to matching together.

2. Related Work

  • Line detection ranges from classical Hough-transform methods to local gradient-based detectors [46]; this work trains with LSD.Deep line detectors can offer better detection results at greater computational cost, particularly for specialized wireframe-parsing tasks.Source: §2 · p. 2
  • Line descriptors use local image gradients, neural patches, or sampled points along a line [47].SOLD2 [47] jointly detects and describes lines and handles partial occlusion; this work interpolates SuperPoint [16] dense descriptors at line endpoints.Source: §2 · p. 2
  • Because line descriptor matching is difficult, prior methods add geometric cues such as rotation, line-pair properties, invariants, cross-ratio, or fundamental-matrix consistency.The paper notes that estimating a fundamental matrix for matching creates a chicken-and-egg problem and that these heuristics can fail in realistic scenarios.Source: §2 · p. 2
  • SuperGlue [54] uses a GNN and keypoint positional encoding to disambiguate repetitive patterns; later variants improve efficiency or use dense predictions [12] [26] [13] [17].Source: §2 · p. 2
  • Prior line and mixed-feature GNN matchers include WGLSM, LineTR, and HDPL, which represent each line with a single node.WGLSM and HDPL use a single Sinkhorn matrix; HDPL allows point-line assignments, while LineTR attends over sampled points within each line.Source: §2 · p. 2
  • GlueStick instead models each line endpoint as a separate GNN node, exploiting their usual consistency with epipolar geometry to disambiguate matches.The ablation reports that matching points and line endpoints together already greatly improves matching performance.Source: §2 · pp. 2–3

3. GlueStick

GlueStick combines points and line segments in a wireframe so a GNN can use visual, spatial, and connectivity context to update descriptors before matching. It builds point and line assignments separately and defines training labels and a joint assignment loss for both feature types.

3. GlueStick

  • The three-block pipeline combines detected points and lines in a wireframe, enriches their descriptors with a GNN, then assigns point and line matches separately.The motivation is to let each feature use cues from neighboring features rather than rely on heuristic combinations.Source: §3 · p. 3

3.1. From Points and Lines to Wireframes

  • The wireframe represents keypoints and line endpoints as nodes, with edges connecting endpoints of detected lines.Close endpoints are merged to create connectivity; endpoint positions are adjusted, but the number of lines is unchanged.Source: §3.1 · p. 3

3.2. Attention-based Graph Neural Network (GNN)

  • The GNN initializes node descriptors with visual and spatial information, then enriches them through self-attention, cross-attention, and line message passing.Line message passing aggregates information from neighboring nodes connected by line edges; the authors use a simple neighbor average because attention added complexity without performance gain.Source: §3.2 · pp. 3–4

3.3. Dual-Softmax for Points and Lines

  • Points and lines receive separate dual-softmax assignments, and line scores take the maximum across the two possible endpoint orderings.The final matches are mutual nearest neighbors above a score threshold.Source: §3.3 · p. 5

3.4. Ground Truth Generation

  • Point labels use pose-and-depth reprojection, while line labels sample segment points, exclude invalid or occluded samples, and track proximity to reprojected lines.Segments with more than 50% invalid samples are ignored during loss computation.Source: §3.4 · p. 5

3.5. Loss Function

  • Training minimizes the negative log-likelihood of point and line assignments rather than using triplet-ranking loss with hard-negative mining.The authors cite repetitive structures along lines as a reason hard negatives may be detrimental.Source: §3.5 · pp. 5–6

4. Experiments

The experiments evaluate GlueStick across line matching, homography estimation, camera rotation estimation, and visual localization. They test the contributions of wireframe connectivity and line message passing alongside the combined use of points and lines.

4. Experiments

  • GlueStick is pretrained on synthetically homography-warped image pairs and then fine-tuned on MegaDepth; training uses nine GNN blocks and takes 10 days on two RTX2080 GPUs.The training setup limits inputs to at most 1,000 keypoints and 250 line segments per image.Source: §4 · p. 6

4.1. Baselines

  • The line-matching comparisons include LBD, SOLD2, LineTR, L2D2, and PL-Loc, with ClusterGNN and LoFTR also included where possible.SOLD2 uses its own detector; the other listed line methods use LSD.Source: §4.1 · p. 6

4.2. Ablation Study

  • Wireframe preprocessing and Line Message Passing each boost the SuperGlue baseline, their combination performs best, and the lines-only GlueStick-L remains competitive.The ablation uses 13 ETH3D training scenes and image pairs with at least 500 ground-truth keypoints in common.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3. Line Matching Evaluation on ETH3D

  • On ETH3D, GlueStick recovers almost 80% of ground-truth line matches, while the best prior methods stay below 50% recall, and it exceeds LineTR by more than 15% in precision at equal recall.GlueStick almost doubles AP versus the previous best method. Its 258 ms average runtime is close to SuperGlue’s 235 ms, versus 419 ms for SOLD2 and 304 ms for LineTR.Source: §4.3 · pp. 6–7, Figure 5 · p. 6

4.4. Homography Estimation

  • GlueStick ranks first on HPatches, but its saturation and scarcity of structural lines motivate additional real-world planar-surface and pure-rotation homography experiments.The paper evaluates relative pose from planar surfaces and relative pose with pure rotations.Source: §4.4 · p. 7

4.4.1 Dominant Plane on ScanNet

  • On ScanNet, points-only GlueStick outperforms SuperGlue, lines-only GlueStick exceeds prior line-matching state of the art, and combining points and lines gives the best overall results.The network may rely more on accurate points in well-textured images and on lines when points are scarce.Source: §4.4.1 · p. 7, Table 1 · p. 7

4.4.2 Pure Rotations on SUN360

  • On SUN360 rotation estimation, GlueStick-PL performs best because lines help match pairs without enough texture; point-based SG and GlueStick-P obtain 3 points less AUC.Long lines can be detected very precisely and contribute to accurate estimation.Source: §4.4.2 · pp. 7–8, Figure 6 · p. 7

4.5. Visual Localization

  • On Stairs, GlueStick improves pose accuracy by 44% over SuperGlue; adding lines also improves Stairs and brings a small improvement on InLoc.Stairs is challenging for point methods because of low texture and repeated steps; InLoc includes low-texture images and large viewpoint changes.Source: §4.5 · p. 8

5. Conclusion

GlueStick jointly matches points and lines, allowing strong matches of either type to support neighboring features with less distinctive appearance. The conclusion reports improved matching and downstream performance, while the ground-truth procedure addresses ambiguity from occlusion and uncertain depth.

5. Conclusion

  • GlueStick ranks first for both points-only and point-line visual localization results on 7Scenes Stairs and InLoc.Table 2 reports translation, rotation, and pose-accuracy measures for Stairs, and pose AUC measures for InLoc.Source: Table 2 · p. 8
  • GlueStick jointly matches points and lines, propagating strong matches to neighboring features with less discriminative appearance.The conclusion reports improved matching performance for both feature types.Source: §5 · p. 8
  • The conclusion reports improved matching across points and lines and better results in downstream homography and camera pose estimation.The authors identify line connectivity as a key architectural signal.Source: §5 · p. 8

A. Ground Truth Generation

  • The ground-truth generation labels line pairs matched, unmatched, or ignored when depth is uncertain or reprojection is occluded.Compared with 's robust 3D line reconstruction, the method is faster and simpler, avoids that reconstruction step, and supports training on larger scenes with potentially noisy depth.Source: A. Ground Truth Generation · p. 9

B. Additional Insights on GlueStick

The additional analyses examine detector choice, detector generalization, and how keypoint availability, line length, and image overlap relate to GlueStick's line-matching performance. They report robust matching without keypoints and strong performance across line lengths and overlap levels.

Detector choice

  • The authors select LSD as a generic detector because learned detectors may not generalize across contexts, tasks, or image types.They also suggest that connecting line endpoints may improve their repeatability across views.Source: B. Additional Insights on GlueStick · pp. 9–10
  • LSD offers a useful trade-off in repeatability and number of detected lines, alongside low localization error and versatility.The learned F-Clip baseline has the highest repeatability but detects few lines; the authors choose LSD for its accuracy and versatility.Source: B. Additional Insights on GlueStick · p. 9

Detector generalization

Dependence on Point Matches

Impact of the Line Length

Robustness to Small Image Overlaps

  • Performance decreases as image overlap shrinks, but GlueStick maintains strong performance even for small-overlap pairs.The ETH3D analysis divides image pairs into small, medium, and large overlap categories.Source: B. Additional Insights on GlueStick · p. 12

C. Experimental Details

The section outlines four supplementary analyses and results reported in the appendix.

C. Experimental Details

  • The appendix adds baselines for ScanNet homography and relative-pose estimation.It also includes details and visualizations for pure-rotation estimation and its use in image stitching.Source: C. Experimental Details · p. 12
  • The appendix compares homography estimation methods on HPatches.The stated scope also includes full visual-localization results on 7Scenes.Source: C. Experimental Details · p. 12
  • The 7Scenes localization results are provided in full, though performance on most scenes is already saturated.Source: C. Experimental Details · p. 12

C.1. Relative Pose through Homographies

The relative-pose comparison finds GlueStick remains the top-performing method when essential matrices replace homographies. The pure-rotation procedure estimates rotation from point, line, or mixed feature matches within Hybrid RANSAC.

C.1. Relative Pose through Homographies

  • GlueStick remains the top-performing baseline when relative pose is estimated from essential matrices rather than homographies.Essential matrices yield higher relative-pose quality than homographies; the comparison uses outdoor models for fairness because GlueStick was trained only on outdoor data.Source: §C.1 · pp. 12–13, Table 4 · p. 13

C.2. Pure Rotation Estimation

  • The pure-rotation method uses minimal and least-squares solvers inside Hybrid RANSAC for point, line, or combined matches.It calibrates features with K^-1 and estimates the three degrees of freedom of rotation; its minimal solver aligns two feature-match vectors using the Kabsch algorithm.Source: §C.2 · p. 13
  • Feature sampling is weighted by match counts, while line segments are sampled proportionally to the square root of their length.This gives larger lines priority in sampling.Source: §C.2 · p. 13

C.3. Homography Estimation in HPatches

GlueStick ranks first among the considered methods on HPatches, but its advantage in homography estimation is small because the dataset is saturated and contains few structural lines. The section also describes the evaluation measures and shows matching examples on SUN360.

C.3. Homography Estimation in HPatches

  • Homographies are estimated from point and/or line correspondences with RANSAC and scored by corner reprojection-error AUC at 3, 5, and 10 pixels.The evaluation also reports precision and recall of matches obtained from the ground-truth homography.Source: §C.3 · pp. 13–14
  • SUN360 examples show GlueStick point and line matches and the resulting stitching of each image pair.Source: Figure 15 · p. 14
  • GlueStick ranks first by a very small margin in HPatches homography estimation, while line features bring no significant improvement to homography fitting.The authors attribute the limited gain to HPatches being saturated and containing few structural lines.Source: §C.3 · p. 14

C.4. Visual Localization on the Full 7Scenes Dataset

On the full 7Scenes dataset, GlueStick is slightly ahead of baselines and performs best on the challenging Stairs scene. Visual examples show more precise line matching in repeated structures and close alignment of pose-based reprojections with detected features.

C.4. Visual Localization on the Full 7Scenes Dataset

  • On 7Scenes, GlueStick is slightly ahead of baselines overall and largely outperforms them on the most challenging scene, Stairs.The dataset is small-scale and largely saturated for point-based methods; lines improve results in only a few scenes, especially Stairs.Source: §C.4 · pp. 14–15

D.1. Feature Matches

  • On ETH3D examples, GlueStick consistently matches more lines at higher precision than prior works, particularly in repeated structures.SOLD2 and L2D2 descriptors lack context from neighboring lines and match only a few lines in the repeated-structure example.Source: §D.1 · p. 15

D.2. Visualization of the Camera Pose Estimation

  • In Stairs pose visualizations, GlueStick’s reprojected 3D features align almost perfectly with the original 2D detections, unlike hloc’s reprojections.Source: §D.2 · p. 15

D.3. Failure Cases and Limitations

The authors identify line detection and separate feature pipelines as important limitations, alongside supervision requirements and challenging visual conditions. Failure cases include ambiguous repeated patterns, rotations beyond 45°, and low-texture symmetric scenes.

D.3. Failure Cases and Limitations

  • Line detection is GlueStick’s main performance bottleneck because current line detectors are less repeatable and accurate than point features.Endpoint-based line representation can also be affected by partial occlusion, though neighboring unoccluded features provide observed robustness; sampling along lines modestly helps occluded areas at higher runtime cost.Source: §D.3 · p. 15
  • Points and lines still use different detectors, requiring three algorithms, while discrete feature extraction prevents a fully differentiable pipeline.Joint point-and-line detection and description and end-to-end training are identified as future directions.Source: §D.3 · p. 15
  • Current training supervision requires ground-truth point and line correspondences across images, usually obtained by reprojection with depth and camera poses.Epipolar constraints and two-view geometry are proposed as possible alternative supervision signals.Source: §D.3 · p. 15
  • GlueStick may fail on repeated patterns without disambiguating cues, often fails beyond 45° rotations, and degrades when low texture coincides with symmetric structure.The authors suggest rotating one image by 0°, 90°, 180°, and 270° and keeping the best match for large-rotation cases.Source: §D.3 · pp. 15, 17

E. Attention visualization

Attention visualizations show how the network distributes attention across image features and progressively focuses on corresponding nodes. They illustrate complementary use of point and line structure, while the paper’s failure examples identify scenarios that remain difficult.

E. Attention visualization

  • Figure 20 averages attention across heads at cross layers and shows the top 20 activated nodes; green denotes connected line endpoints and cyan isolated keypoints.Source: E. Attention visualization · p. 17
  • Attention first ranges broadly across the image, then focuses on the corresponding node in the other image; keypoints and line endpoints can use each other’s context.Keypoints use line structure to seek points along a line, while line endpoints draw on both keypoint and endpoint attention.Source: E. Attention visualization · p. 17
  • The failure examples include perfectly repeated patterns, rotations greater than 45°, and low-texture symmetric structures.Source: Figure 19 · p. 18

Related papers

Loading 2304.02008v3…