PaperlayerView exact revision on arXiv

Source-linked AI summary

SOLD2: Self-supervised Occlusion-aware Line Description and Detection

Rémi Pautrat, Juan-Ting Lin, Viktor Larsson, Martin R. Oswald, Marc Pollefeys

arXiv:2104.03362v2cs.CV

TL;DR

Line segments complement feature points, but their detection and description face challenges from occlusion, unstable endpoints, and viewpoint changes. SOLD2 jointly learns line detection and description through self-supervision, and its full pipeline improves repeatability, localization accuracy, and matching performance over previous baselines.

  • Problem

    Line description is complicated by partial occlusion, endpoints that are not repeatable across views, and appearance changes under viewpoint variation.

    Source: §1 · p. 1, §3.5 · p. 4

  • Method

    SOLD2 jointly learns line-segment detection and description using self-supervision, with a matching procedure designed to handle occlusion.

    Source: §1 · p. 2

  • Results

    SOLD2 improves repeatability, localization accuracy, and matching performance over previous baselines across indoor and outdoor datasets.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Lines provide geometric scene cues and appear in poorly textured areas, complementing feature points for geometric tasks.

    Source: §1 · p. 1, §5 · p. 8

    Generic line predictions may need application-specific tuning, since localization may favor short stable lines while reconstruction or wireframe parsing may favor longer ones.

    Source: §5 · p. 8

Abstract

from arXiv · show

Compared to feature point detection and description, detecting and matching line segments offer additional challenges. Yet, line features represent a promising complement to points for multi-view tasks. Lines are indeed well-defined by the image gradient, frequently appear even in poorly textured areas and offer robust structural cues. We thus hereby introduce the first joint detection and description of line segments in a single deep network. Thanks to a self-supervised training, our method does not require any annotated line labels and can therefore generalize to any dataset. Our detector offers repeatable and accurate localization of line segments in images, departing from the wireframe parsing approach. Leveraging the recent progresses in descriptor learning, our proposed line descriptor is highly discriminative, while remaining robust to viewpoint changes and occlusions. We evaluate our approach against previous line detection and description methods on several multi-view datasets created with homographic warps as well as real-world viewpoint changes. Our full pipeline yields higher repeatability, localization accuracy and matching metrics, and thus represents a first step to bridge the gap with learned feature points methods. Code and trained weights are available at https://github.com/cvg/SOLD2.

1. Introduction

Lines complement feature points with structural cues and remain available in poorly textured areas, but detecting and describing them repeatably across views presents distinct challenges. SOLD2 jointly learns line detection and description without annotated line labels, with matching designed to handle occlusion.

1. Introduction

  • Lines complement points because they occur in poorly textured areas and collections of lines provide geometric clues about scene structure.Source: §1 · p. 1
  • Existing learned line detectors often target wireframe parsing, but have not been optimized for repeatability across images and require cumbersome manual line labels.Source: §1 · p. 1
  • Line descriptors must contend with partial occlusion, uncertain endpoints, changing support-region scale, and perspective or distortion changes.Existing learned descriptors are not designed for occlusion and remain sensitive to poorly localized endpoints.Source: §1 · pp. 1–2
  • SOLD2 jointly learns line-segment detection and description in a self-supervised network that can train on real-image datasets without line labels.The detector is pretrained on synthetic data and then generalized to real images.Source: §1 · p. 2
  • Its line-matching procedure is designed for occlusion and achieves state-of-the-art results on image-matching tasks.Source: §1 · p. 2

2. Related work

Prior work addresses line detection, description, and matching through distinct approaches, with learned methods often retaining limitations in repeatability or viewpoint robustness. The supplied training-pipeline overview shows SOLD2’s progression from synthetic detector training to real-image pseudo-labeling and full-model training.

Line detection

  • Wireframe-parsing detectors produce line segments but were not trained for viewpoint-repeatable lines, motivating generic line segments for varied scenes.Wireframes are collections of segments connected by endpoints, usually labeled by humans.Source: §2 · p. 2

Line description

  • Handcrafted descriptors summarize gradient information in line-support regions, while learned DLD and WLD embed patches and LLD samples points along lines.Source: §2 · p. 2
  • LLD is conceptually closest to SOLD2, but is not rotation invariant and degrades under large viewpoint changes.Source: §2 · p. 2

Joint detection and description of learned features

  • Learned point-feature methods share computation or use dense descriptors, varied supervision, and multi-task distillation; SOLD2 extends synthetic-pretrained SuperPoint to line segments.Source: §2 · p. 2

Line matching

  • Earlier line-matching methods use neighboring structures, coplanar line-point cues, or sampled points; SOLD2 instead flexibly matches points along lines to handle occlusions.Source: §2 · p. 2

Training pipeline

  • The pipeline first trains a detector on synthetic data, generates pseudo ground truth on real images through homography adaptation, then trains the full model with descriptors.Source: Figure 2 · p. 3

3. Method

SOLD2 jointly detects and describes line segments through a self-supervised pipeline: synthetic pretraining supplies a detector, then its predictions provide pseudo-labels on real images. It represents line descriptions as ordered point descriptors and aligns them with sequence matching.

3.4. Self-supervised learning pipeline

  • A synthetic-pretrained detector generates pseudo-ground-truth lines on real images; homography adaptation aggregates predictions, and one adaptation round sufficed.The predicted segments are used as ground truth for the next training round.Source: §3 · p. 3, §3.4 · p. 4

3.1. Problem formulation

  • Line segments are represented by endpoint pairs, chosen for simplicity and compatibility with the self-supervision process.Source: §3.1 · p. 3

3.2. Junction and line heatmap inference

  • A shared grayscale-image encoder feeds junction-probability, line-heatmap, and pixel-wise descriptor outputs.The junction map predicts line endpoints, while the heatmap predicts whether each pixel lies on a line.Source: §3.2 · p. 3

3.3. Line detection module

  • Candidate segments are sampled uniformly, locally re-centered on heatmap maxima, then filtered by mean activation and the fraction of samples above threshold.The search radius for local maxima is proportional to segment length.Source: §3.3 · pp. 3–4

3.5. Line description

  • Each line is described by an ordered sequence of sampled point descriptors, allowing visible portions to match when other portions are occluded.The descriptor is trained with a triplet loss that minimizes distances between matching points and maximizes distances to non-matching points.Source: §3.5 · p. 4

3.6. Multi-task learning

  • Dynamic weights optimized during training balance junction, line, and descriptor losses because the tasks have distinct uncertainties and loss scales.The weighting follows the multi-task learning approach of Kendall et al..Source: §3.6 · p. 4

3.7. Line matching

  • Needleman-Wunsch aligns ordered samples by scoring descriptor-similarity matches against gaps, accommodating occlusion and length-induced misalignment.The score matrix uses dynamic programming to maximize the total match and skip score.Source: §3.7 · pp. 4–5, Figure 3 · p. 5

3.8. Implementation details

  • A stacked-hourglass backbone supports comparison with wireframe parsers; training uses synthetic data and 5,000 Wireframe images while ignoring their provided line labels.The Wireframe split uses 462 images for testing.Source: §3.8 · p. 5

4. Experiments

Experiments compare line repeatability and localization on Wireframe and YorkUrban, and matching on warped and real-camera image pairs; descriptor choices are also ablated.

4.1. Line segment detection evaluation

  • Detection is evaluated on homography-warped Wireframe and YorkUrban pairs using repeatability and localization error; matching spans hand-labeled and predicted lines, including real-camera ETH3D pairs.For ETH3D, image pairs are selected by shared covisible 3D points.Source: §4.1 · pp. 5–6, §4.2 · pp. 6–7
  • SOLD2 achieves the best repeatability and localization error on Wireframe and YorkUrban; candidate selection slightly lowers performance but yields fewer segments and faster inference.Candidate selection removes segments with other junctions between their endpoints.Source: Table 1 · p. 6, §4.1 · p. 6

4.2. Line segment description evaluation

  • On ground-truth Wireframe lines, SOLD2's descriptor improves over WLD by 5%; LLD performs poorly on rotated pairs because its training frames had little rotation.The ground-truth-line evaluation uses exact one-to-one correspondences.Source: Figure 4 · p. 7, §4.2 · p. 7
  • Across tested occlusion levels, SOLD2 outperforms baselines by a large margin; WLD drops significantly, consistent with its line-patch representation.The evaluation adds random textured ellipses and shortens lines at occluder boundaries.Source: Figure 5 · p. 7, §4.2 · p. 7
  • On warped Wireframe and real-camera ETH3D images, SOLD2 beats descriptor baselines in precision and recall on LSD lines; its own lines with line NMS perform best.The best performance is reported when overlapping segments are removed with line NMS.Source: Table 2 · p. 7, §4.2 · p. 7, Figure 6 · p. 8

4.3. Ablation study

  • Endpoint SIFT matching is quickly limited, while averaging descriptors or nearest-neighbor scores slightly reduces accuracy, highlighting the importance of ordered points in NW matching.The ablation compares these alternatives on Wireframe ground-truth lines, with and without occluders.Source: §4.3 · p. 8, Table 3 · p. 8
  • D2-Net and ASLFeat saliency-based sampling perform slightly worse than regular sampling; five points balance too-few-point robustness loss against too many similar descriptors.The authors report five samples as a good overall trade-off.Source: §4.3 · p. 8

5. Conclusion

The conclusion presents SOLD2 as a self-supervised pipeline for joint line detection and description, reporting improved repeatability, localization accuracy, and matching performance over previous baselines. It notes that generic line predictions need application-specific tuning and outlines the appendix contents.

5. Conclusion

  • Across indoor and outdoor datasets, SOLD2 improves repeatability, localization accuracy, and matching performance over previous baselines.The conclusion presents this as an evaluation of the full pipeline.Source: §5 · p. 8
  • Generic line predictions still need application-specific tuning: localization may favor short, stable lines, while reconstruction and wireframe parsing may favor longer ones.The conclusion says adjusting line parameters can steer the output segments toward these different needs.Source: §5 · p. 8

Appendix

  • The appendix covers synthetic-data generation, network architecture, multi-task losses, detection details, evaluation metrics, statistical significance, ETH3D preprocessing, homography estimation, and qualitative examples.Source: Appendix · p. 9

A. Synthetic dataset examples and homography generation

The appendix describes a synthetic dataset of rendered shapes and the homographies used for augmentation and homography adaptation. These homographies combine scaling, translation, rotation, and perspective transforms.

A. Synthetic dataset examples and homography generation

B. Network architecture

The network uses a stacked hourglass backbone with separate junction, line-heatmap, and descriptor branches. The appendix specifies their output resolutions and training settings.

B. Network architecture

  • For a fair comparison with most wireframe parsing methods, the architecture uses the same stacked hourglass network as in for its backbone encoder.Source: B. Network architecture · p. 9
  • The junction branch produces an h/8 × w/8 × 65 map, while the heatmap branch upsamples by ×4 with subpixel shuffles to output a full-resolution line heatmap.The junction branch uses 3 × 3 and 1 × 1 convolutions; the heatmap branch ends with a sigmoid output.Source: B. Network architecture · p. 9
  • The descriptor branch applies 3 × 3 and 1 × 1 convolutions to produce an h/4 × w/4 × 128 semi-dense descriptor map, bilinearly interpolated at point locations.The triplet loss uses margin M = 1 and hardest-negative distance T = 8 pixels.Source: B. Network architecture · p. 9
  • Training uses Adam, ReLU after each convolution, and grayscale images resized to 512 × 512.Source: B. Network architecture · p. 9

C. Multi-task learning

The multi-task setup dynamically balances junction, line, and descriptor losses. The detection details describe adaptive heatmap search around sampled line locations and candidate filtering to reduce overlapping segments.

C. Multi-task learning

  • Dynamic weighting addresses losses with different uncertainty, scale, and changing relative values during training.The losses correspond to junctions, lines, and descriptors.Source: C. Multi-task learning · p. 9
  • With equal loss weights, detection and description results are at least 10% and 5.5% worse; using fixed weights from dynamic weighting makes them at least 6.7% and 76.2% worse.The final learned weights converge toward inverse loss values, leaving each weighted loss around 1.Source: C. Multi-task learning · p. 9

D. Line segment detection details

  • Detection converts junction and heatmap predictions into segments through sampling, adaptive local-maximum search, score and inlier checks, and optional candidate selection.Candidate selection can pre-filter line candidates.Source: D. Line segment detection details · p. 9
  • The adaptive search selects the highest nearby heatmap activation because pixel-limited heatmaps can give low values to samples beside the actual line.Its radius depends on segment length; the minimum radius can remain constant across image resolutions, and λ = 3 pixels was used.Source: D. Line segment detection details · p. 10
  • Candidate selection removes segments containing another junction between their endpoints; Figure 8 shows rendered images above estimated junctions and line-segment labels.The junction-to-line distance threshold is ξ_cs = 3 pixels.Source: D. Line segment detection details · p. 10, Figure 8 · p. 10

E. Detector evaluation metrics

The detector evaluation defines repeatability and localization error by comparing line detections across images related by a homography.

E. Detector evaluation metrics

  • Both metrics compare line segments detected in an image pair where the second image is a homographic warp of the first.The evaluation associates each image with a set of detected line segments.Source: E. Detector evaluation metrics · p. 10
  • Repeatability measures how often a line is re-detected across views, using a tolerance and one of two line distances.The distances are structural distance and orthogonal distance.Source: E. Detector evaluation metrics · p. 10
  • Localization error is the average line distance between a detected line and its re-detection in another view, under a tolerance.The passage does not specify which distance is used for this metric.Source: E. Detector evaluation metrics · p. 10
  • Set cardinality notation |·| denotes the number of elements in a set.This notation is used in the metric definitions.Source: E. Detector evaluation metrics · p. 10

F. Statistical evaluation of our method

The evaluation reports five-run statistics for the method and describes preprocessing used to evaluate it on ETH3D.

F. Statistical evaluation of our method

G. ETH3D dataset preprocessing

  • ETH3D evaluation uses images downsampled by a factor of 8 and reprojects detected lines into 3D using camera intrinsics and depth maps.The 13-scene dataset includes indoor and outdoor scenes, with provided 3D models built using COLMAP.Source: G. ETH3D dataset preprocessing · p. 11
  • Depth holes, especially near discontinuities where lines often occur, are inpainted up to 10 pixels from valid depth using NLSPN.The passage identifies NLSPN as the current state of the art in deep RGB-guided depth inpainting.Source: G. ETH3D dataset preprocessing · p. 11

H. Homography estimation experiment

The homography experiment estimates transformations from line matches and compares the line pipeline with SuperPoint, while also reporting descriptor and localization comparisons against line baselines.

H. Homography estimation experiment

  • The experiment estimates homographies from line matches with RANSAC, sampling four lines and accepting estimates below 3 pixels average corner reprojection error.It uses up to 1,000,000 LORANSAC iterations and orthogonal line distance for reprojection error.Source: H. Homography estimation experiment · p. 11
  • Table 4 reports Wireframe homography estimates from line matches, using a 5-pixel orthogonal line-distance threshold for inliers.The provided caption specifies the evaluation scope and threshold, but not table values.Source: Table 4 · p. 11
  • On LSD lines, the proposed descriptors have the highest accuracy among baselines, while the full pipeline performs similarly.The authors note learned features remain on par or slightly worse than handcrafted detections in localization error.Source: H. Homography estimation experiment · p. 11
  • SuperPoint performs significantly worse than the line method on Wireframe homography estimation, where textureless scenes and repeated structures are numerous.The authors also report SuperPoint is not robust to rotations above 45 degrees, while ordered line descriptors support invariance to any rotation.Source: H. Homography estimation experiment · p. 11, Table 4 · p. 11

I. Qualitative results of line segment detections and matches

The qualitative figures compare line detection and matching with prior methods and point matching, highlighting line matches in challenging image regions.

I. Qualitative results of line segment detections and matches

  • In challenging Wireframe images, the method matches enough lines for accurate pose estimation where SuperPoint fails.The examples include low-texture and repeated-structure scenes.Source: §I · p. 11, Figure 12 · p. 14
  • Figure 10 compares the full pipeline against LBD, LLD, and WLD descriptors computed on LSD lines, marking correct, wrong, and unmatched lines.Only lines shared between the views are displayed.Source: Figure 10 · p. 12
  • Figure 11 shows line detections from LCNN, HAWP, DeepHough, LSD, TP-LSD, and the proposed method on Wireframe images.Source: Figure 11 · p. 13
  • Figure 12 compares line and SuperPoint matches using a 5-pixel correctness threshold, with orthogonal line distance used for line matches.The caption identifies textureless areas, repeated structures, and symmetry as settings where lines can be matched.Source: Figure 12 · p. 14

Related papers

Loading 2104.03362v2…