PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients

Rémi Pautrat, Daniel Barath, Viktor Larsson, Martin R. Oswald, Marc Pollefeys

arXiv:2212.07766v3cs.CV

TL;DR

Handcrafted line detectors are accurate but vulnerable to noisy conditions, while learned detectors can lack accuracy and be biased toward wireframe lines. DeepLSD predicts attraction fields to guide handcrafted detection and refines segments with optimization; the paper reports a new record on several downstream tasks and improved line accuracy.

  • Problem

    Handcrafted detectors lack robustness to noisy gradients, while learned detectors can be less accurate and biased toward wireframe lines.

    Source: §1 · pp. 1–2

  • Method

    DeepLSD learns line attraction fields from bootstrapped detector outputs, converts them to a surrogate image gradient for handcrafted detection, and optimizes segments with vanishing points.

    Source: §1 · p. 2

  • Results

    DeepLSD sets a new record in several downstream tasks requiring line segments, and its refinement can improve the accuracy of existing deep line detectors.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The hybrid detector can be trained without ground-truth lines on any dataset and is presented as suitable for most line-segment tasks.

    Source: §1 · p. 2, §5 · p. 8

    Pseudo-ground-truth generation trades off recovering low-contrast lines against avoiding noisy background lines, including lines in the sky.

    Source: F. Limitations · pp. 12–13

Abstract

from arXiv · show

Line segments are ubiquitous in our human-made world and are increasingly used in vision tasks. They are complementary to feature points thanks to their spatial extent and the structural information they provide. Traditional line detectors based on the image gradient are extremely fast and accurate, but lack robustness in noisy images and challenging conditions. Their learned counterparts are more repeatable and can handle challenging images, but at the cost of a lower accuracy and a bias towards wireframe lines. We propose to combine traditional and learned approaches to get the best of both worlds: an accurate and robust line detector that can be trained in the wild without ground truth lines. Our new line segment detector, DeepLSD, processes images with a deep network to generate a line attraction field, before converting it to a surrogate image gradient magnitude and angle, which is then fed to any existing handcrafted line detector. Additionally, we propose a new optimization tool to refine line segments based on the attraction field and vanishing points. This refinement improves the accuracy of current deep detectors by a large margin. We demonstrate the performance of our method on low-level line detection metrics, as well as on several downstream tasks using multiple challenging datasets. The source code and models are available at https://github.com/cvg/DeepLSD.

1. Introduction

Line detectors must balance robustness to challenging images with accurate localization, but handcrafted methods struggle with noisy gradients while learned methods remain less accurate and biased toward wireframe lines. DeepLSD combines a learned attraction field with handcrafted detection, bootstraps training fields without ground-truth lines, and adds line refinement; the paper reports a new record in several downstream tasks.

1. Introduction

  • Handcrafted detectors are fast and accurate but can fail under noisy gradients, while learned detectors can generalize poorly beyond Wireframe’s indoor structural lines and lack accuracy.Source: §1 · pp. 1–2
  • DeepLSD uses deep learning to process images and suppress unnecessary detail before handcrafted methods detect line segments.The hybrid is intended to retain robustness to illumination and noise alongside classical detection accuracy.Source: §1 · p. 2
  • The method bootstraps line-attraction-field training targets from existing detectors, avoiding the need for ground-truth lines and allowing training on any dataset.Source: §1 · p. 2
  • A separate optimization jointly refines line segments and vanishing points and can improve any existing deep line detector.Source: §1 · p. 2
  • DeepLSD sets a new record in several downstream tasks requiring line segments by combining deep-learning robustness with handcrafted precision.Source: §1 · p. 2

2. Related Work

Prior line detectors use either handcrafted image-gradient heuristics or learned representations, each with different accuracy, robustness, and training-data constraints. DeepLSD combines these approaches by predicting attraction fields for classical line extraction and refinement.

Handcrafted Line Detectors

  • Handcrafted detectors such as LSD and EDLines grow aligned image-gradient regions to extract segments, making them fast and accurate but vulnerable to noise and low illumination.Source: §2 · p. 2

Learned Line Detectors

  • Learned detectors represent lines with endpoints, attraction fields, graphs, or transformers, but many are trained on Wireframe and favor indoor structural lines.Source: §2 · p. 2
  • Joint detection-and-description methods include self-supervised SOLD2 [39] and LiDAR-ground-truth L2D2, but these approaches still lack accuracy.Source: §2 · p. 2

Attraction Fields

  • Attraction fields encode discrete lines as continuous 2D vector fields suitable for deep networks, and DeepLSD predicts a distance field rather than a binary mask.The method uses the fields with classical line extractors; Figure 2 outlines bootstrapped distance and angle fields, prediction, conversion to a surrogate gradient, extraction, and refinement.Source: §2 · pp. 2–3, Figure 2 · p. 3

3. Hybrid Line Detector

DeepLSD predicts line attraction fields, converts them into a surrogate image gradient for handcrafted line detection, and refines the resulting segments using the fields and vanishing points.

3. Hybrid Line Detector

  • DeepLSD converts predicted attraction fields into a surrogate image gradient for LSD, combining learned noise suppression and low-contrast line detection with handcrafted segment extraction.An attraction-field optimization step can further refine segments.Source: §3 · p. 3, §3.4 · p. 5

3.1. Line Attraction Field

  • Separating the closest-line distance and orientation fields avoids noisy angles from small offset vectors and the long-range, unstable endpoint predictions required by direct endpoint regression.The network predicts these fields while traditional heuristics recover segment endpoints.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2. Ground Truth Generation

  • Homography adaptation generates pseudo-ground-truth fields by detecting lines in warped images and taking the per-pixel median of their distance and angle fields.Median aggregation removes noisy lines detected in only a few warped images.Source: §3.2 · p. 4

3.3. Learning the Line Attraction Field

  • A U-Net-like network predicts distance and angle fields, with loose supervision limited to pixels within r of a line so undetected low-contrast lines are not penalized as background.This loose supervision can yield more complete predictions than the pseudo ground truth.Source: §3.3 · p. 4

3.4. Extracting Line Segments

  • The method orients predicted line angles using image-gradient polarity, enabling detection of both boundaries of bright-dark-bright edges before applying LSD and filtering segments by field agreement.This preserves double edges that wireframe methods may treat as one line.Source: §3.4 · p. 5, Figure 5 · p. 5

3.5. Line Segment Refinement with Optimization

  • Refinement minimizes attraction-field angle and distance costs, with a vanishing-point cost when the associated VP is sufficiently close; line length stays fixed.Lines and VPs are refined in alternating iterations rather than jointly, which empirically performed worse.Source: §3.5 · pp. 5–6

3.6. Implementation Details

  • The reported implementation trains indoor and outdoor models and uses LSD with r = 5 pixels, 50 filtering samples, and five alternating refinement iterations.The indoor model uses Wireframe images without ground-truth lines; the outdoor model uses MegaDepth.Source: §3.6 · p. 6

4. Experiments

Experiments assess line detection on challenging image pairs and downstream tasks, then test refinement and design choices. DeepLSD combines strong repeatability with accurate localization, performs well in reconstruction and localization, and refinement improves several learned detectors.

4.1. Evaluation on Low-Level Metrics

  • The low-level evaluation uses homography-related image pairs, one-to-one line matching, and repeatability and localization-error metrics; homographies are estimated from LBD matches with LO-RANSAC.HPatches has 580 image pairs with ground-truth homographies and viewpoint and illumination changes; RDNIM pairs include challenging day–night variation.Source: §4.1 · p. 6
  • DeepLSD offers a strong trade-off between learned and handcrafted methods and ranks first in homography estimation; its largest improvement over LSD is on RDNIM’s day–night pairs.The evaluation covers HPatches and RDNIM, using repeatability, localization error, and homography estimation; learned methods have good repeatability but lower localization accuracy and weaker homography estimation.Source: §4.1 · p. 7, Table 1 · p. 7

4.2. 3D Line Reconstruction

  • DeepLSD achieves the best overall 3D line recall and second-best precision on the first four Hypersim scenes.TP-LSD ranks first in precision, but its average recall is 71% lower than DeepLSD’s; DeepLSD also reconstructs more lines with higher precision than LSD.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Visual Localization

  • DeepLSD performs best on the challenging 7Scenes Stairs localization task, where adding line features substantially boosts performance over points alone.The evaluation uses an RGB-D dataset with ground-truth poses and depth; lines can be matched in low-texture scenes.Source: §4.3 · pp. 7–8, Figure 7 · p. 8

4.4. Impact of the Line Refinement

  • Refinement improves line-localization and homography metrics, with up to 32% lower orthogonal localization error for HAWP and TP-LSD and homography-score gains of 27% and 39%.Gains are smaller for DeepLSD because its raw lines are already sub-pixel accurate; refinement time grows linearly with line count and requires two networks.Source: §4.4 · p. 8, Table 3 · p. 8

4.5. Ablation Study

  • The ablation compares double-edge detection, DF normalization, and retrained HAWP; retrained HAWP performs poorly because generic ground-truth lines have noisy endpoints and are numerous.The comparisons use low-level detector metrics on HPatches.Source: §4.5 · p. 8, Table 4 · p. 8

5. Conclusion

The conclusion presents DeepLSD as a hybrid detector that combines learned robustness with handcrafted accuracy. It can be trained without ground-truth lines, and the proposed refinement improves line accuracy and narrows the localization gap between deep and handcrafted detectors.

  • DeepLSD combines deep-learning robustness and handcrafted-detector accuracy through a learned surrogate image gradient.Source: §5 · p. 8
  • Without ground-truth lines, DeepLSD can be trained on any dataset and is intended for most line-segment tasks.Source: §5 · p. 8
  • The proposed refinement improves DeepLSD accuracy and bridges the localization gap between deep line detectors and handcrafted ones.Source: §5 · p. 8

A. Network Architecture

The attraction-field predictor uses a U-Net-like encoder–decoder, with the image downsampled by eight and then restored to its original resolution. Its two output branches use different activations for the distance and angle fields.

  • The predictor uses a U-Net-like architecture that downsamples the image by a factor of 8 through three average-pooling stages, then upsamples it bilinearly.Source: A. Network Architecture · p. 9
  • Skip connections concatenate features from each downsampling stage with the corresponding upsampling output.Source: A. Network Architecture · p. 9
  • The distance-field and angle-field branches end with ReLU and Sigmoid activations, respectively, without batch normalization.Source: A. Network Architecture · p. 9

B. Additional Ablation Studies

The additional ablation applies the learned processing to ELSED as well as LSD. Deep versions of both traditional detectors improve most low-level metrics on HPatches and RDNIM.

B.1. Generalization to Other Traditional Detectors

  • DeepELSED applies the method with ELSED as its base detector, showing that the approach extends beyond LSD.Source: §B.1 · p. 9
  • For both LSD and ELSED, learned processing improves most raw-line detection metrics on HPatches and RDNIM.Source: §B.1 · p. 9
  • The comparison evaluates unrefined detections on the HPatches and RDNIM datasets.Source: §B.1 · p. 9

B.2. Line Refinement on Traditional Methods

The refinement is not a reliable low-level accuracy improvement for traditional detectors, which are already sub-pixel accurate. It can still benefit downstream homography estimation for ELSED.

B.2. Line Refinement on Traditional Methods

  • The approach also applies to ELSED, with comparisons on HPatches and RDNIM reported without the final line refinement.The table compares the proposed approach with the original detectors.Source: Table 5 · p. 9
  • Traditional-detector refinement produced no low-level metric gains except a boost in ELSED homography estimation.The comparison covers LSD and ELSED on Wireframe; Table 6 reports the refinement results.Source: §B.2 · p. 9, Table 6 · p. 9
  • Already sub-pixel-accurate traditional lines can drift because the distance field has limited resolution and some lines lie in high-distance areas.The authors identify these as reasons refinement may not optimize such lines correctly.Source: §B.2 · p. 9

B.3. Training Learned Baselines with our Supervision Strategy

The authors compare DeepLSD with a HAWP detector retrained using their pseudo-ground-truth lines. DeepLSD remains the more satisfactory approach for generic lines, where endpoint prediction is challenging at high line densities.

B.3. Training Learned Baselines with our Supervision Strategy

  • DeepLSD remains more satisfactory than HAWP retrained with the proposed ground-truth lines.The comparison favors predicting distance and angle fields and applying LSD over directly supervising an end-to-end detector with extracted lines.Source: §B.3 · p. 10
  • Predicting line endpoints with an additional attraction field is unsuitable for generic images with many line segments.The authors identify endpoint prediction at high line density as one reason retrained HAWP produces lower-quality lines.Source: §B.3 · p. 10
  • The retrained-H AWP approach works better for sparser wireframe lines, which require less accuracy.This qualification contrasts wireframe lines with generic lines.Source: §B.3 · p. 10

C. Additional Visual Localization Results

On the full 7Scenes dataset, DeepLSD achieves the best translation error across scenes and the best overall metrics. The gains over earlier methods are small because visual localization on 7Scenes is already saturated.

C. Additional Visual Localization Results

D. Vanishing Point Estimation

DeepLSD performs strongly in vanishing-point estimation, while refinement improves most tested methods’ VP results. In qualitative Hypersim reconstructions, its 3D lines are more complete and cleaner than LSD’s.

D. Vanishing Point Estimation

  • VP candidates come from pairs of detected lines, then Progressive-X fitting and weighted least squares estimate VPs from inlier lines.The line length is used as the weight in the least-squares step.Source: D. Vanishing Point Estimation · p. 10
  • The evaluation uses YorkUrbanDB and NYU-VP, with VP consistency, angular VP error, and recall AUC as metrics.YorkUrbanDB has 102 urban images; NYU-VP has 1,449 images, with 49 held out for parameter tuning.Source: D. Vanishing Point Estimation · pp. 10–11
  • DeepLSD outperforms all baselines in VP error and AUC, and its lines are most consistent with ground-truth VPs at small thresholds.Table 8 compares median VP error and average recall AUC; Figure 10 shows consistency on York Urban.Source: Table 8 · p. 12, D. Vanishing Point Estimation · p. 12, Figure 10 · p. 12
  • Except for HAWP, all evaluated methods improve in VP consistency and error after refinement with VPs included in the optimization objective.The comparison covers YorkUrbanDB consistency and YUD+ VP error.Source: D. Vanishing Point Estimation · p. 12, Figure 11 · p. 12

E. Line 3D Reconstruction

  • On the first four Hypersim scenes, DeepLSD produces more complete, cleaner 3D reconstructions than LSD, whose high-resolution segments tend to break.DeepLSD’s longer lines make it easier to merge track lines into a long 3D line.Source: E. Line 3D Reconstruction · p. 12

F. Limitations

DeepLSD has limitations in training supervision, semantic filtering, differentiability, and refinement speed, while visual comparisons report more complete and accurate results than baselines.

F. Limitations

  • The pipeline is not fully differentiable, and making it end-to-end would require making LSD differentiable, which the authors say is unclear how to do.Source: F. Limitations · p. 12
  • Pseudo-ground-truth quality remains bounded by LSD: lines it rarely detects during homography adaptation are likely absent, while noisy repeatable lines are retained.Source: F. Limitations · p. 12
  • Ground-truth generation trades off detecting low-contrast lines against rejecting background noise; aggregation and background distance constraints can shift that balance.The paper uses median distance and angle fields; stronger background distance constraints reduce noisy lines but also ignore low-contrast lines.Source: F. Limitations · pp. 12–13

G. Additional Visualizations

  • DeepLSD produces more complete and accurate reconstructions than all baselines on Hypersim's first four scenes, and more complete or numerous lines in visual comparisons on YorkUrbanDB and Day-Night.The Day-Night examples include nighttime, over-exposure, and low image quality.Source: Figure 12 · p. 13, G. Additional Visualizations · p. 14

F. Limitations

  • DeepLSD lacks semantic understanding and detects lines of any kind, including lines in the sky or on dynamic objects such as humans.Source: F. Limitations · p. 13
  • Line refinement is slow, especially for other deep detectors, because it requires running two networks; the authors identify offline high-precision applications such as 3D reconstruction as a fit.Source: F. Limitations · p. 13

Related papers

Loading 2212.07766v3…