PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Hough Transform for Semantic Line Detection

Kai Zhao, Qi Han, Chang-Bin Zhang, Jun Xu, Ming-Ming Cheng

arXiv:2003.04676v4cs.CV

TL;DR

Semantic line detection seeks meaningful, structure-revealing lines, but prior object-detection formulations neglect lines’ simpler geometry and contextual features. The paper integrates CNNs with Hough transform in an end-to-end framework, adds a line-similarity metric and large dataset, and reports better detection quality and speed than prior methods.

  • Problem

    Prior methods treat semantic line detection as object detection, neglecting lines’ compact geometric structure and richer contextual features.

  • Method

    The paper combines CNN feature learning with Hough transform end-to-end, detecting lines in a parametric domain and introducing a line-similarity metric and NKL dataset.

  • Results

    Quantitative and qualitative experiments on the proposed and public datasets show significantly better detection quality and speed than previous methods.

  • Takeaways & Limitations

    The framework provides an efficient semantic line detector while supporting evaluation with a principled metric and a larger, diverse dataset.

  • Takeaways & Limitations

    The paper identifies prior single-line feature aggregation as producing sub-optimal results because richer contextual information is important for line detection.

Abstract

from arXiv · show

We focus on a fundamental task of detecting meaningful line structures, a.k.a. semantic line, in natural scenes. Many previous methods regard this problem as a special case of object detection and adjust existing object detectors for semantic line detection. However, these methods neglect the inherent characteristics of lines, leading to sub-optimal performance. Lines enjoy much simpler geometric property than complex objects and thus can be compactly parameterized by a few arguments. To better exploit the property of lines, in this paper, we incorporate the classical Hough transform technique into deeply learned representations and propose a one-shot end-to-end learning framework for line detection. By parameterizing lines with slopes and biases, we perform Hough transform to translate deep representations into the parametric domain, in which we perform line detection. Specifically, we aggregate features along candidate lines on the feature map plane and then assign the aggregated features to corresponding locations in the parametric domain. Consequently, the problem of detecting semantic lines in the spatial domain is transformed into spotting individual points in the parametric domain, making the post-processing steps, i.e. non-maximal suppression, more efficient. Furthermore, our method makes it easy to extract contextual line features eg features along lines close to a specific line, that are critical for accurate line detection. In addition to the proposed method, we design an evaluation metric to assess the quality of line detection and construct a large scale dataset for the line detection task. Experimental results on our proposed dataset and another public dataset demonstrate the advantages of our method over previous state-of-the-art alternatives.

1 INTRODUCTION

Semantic line detection targets meaningful, structure-revealing lines in natural scenes, but prior object-detection adaptations overlook line geometry and contextual features. The paper combines CNN representations with Hough-transform processing, and contributes a dataset, evaluation metric, and refinement module.

  • Semantic lines capture conceptual image structure and support applications including photographic composition, image processing, aesthetics, lane detection, and artistic creation.
  • Hough transform converts spatial evidence into a parametric domain by representing lines with parameters such as slope and offset, enabling local-maximal response detection.
  • Prior CNN methods adapt object detectors for line detection, but ROI pooling and non-maximal suppression reduce efficiency while single-line feature aggregation omits richer context.
  • The proposed end-to-end framework combines CNN feature learning with Hough transform so both components share one optimization target.
  • It also proposes a simpler, principled line-similarity metric and an edge-guided refinement module, while reporting a significant-margin improvement over prior methods on an open benchmark.
  • The paper introduces the 6,500-image NKL dataset, which is larger and more diverse than the previous SEL dataset.

2 RELATED WORK

This section reviews the evolution and limitations of Hough-transform and CNN-based approaches to line detection. CNN methods adapt object-detection pipelines, while line-wise pooling can leave contextual information inadequate.

  • Hough Transform: Hough transform parameterizes lines and has been extended from straight-line detection to localizing arbitrary shapes.The angle-radius formulation addresses the unbounded slope-offset space; later work generalized Hough transform to shapes such as ellipses and circles.
  • Hough Transform: High computational cost and unstable performance motivated probabilistic, gradient-guided, and kernel-based Hough-transform variants.These methods accelerate voting through random sampling, image-gradient directions, or elliptical-Gaussian kernels on collinear pixels.
  • Line Segment Detection: Hough transform cannot directly determine line-segment endpoints, and probabilistic sampling still favors long straight lines.Other line-segment detectors therefore use approaches beyond Hough transform, including edge-orientation-based methods.
  • CNN-based Line Detection: CNN line-detection methods adapt object detectors for semantic lines or wireframe parsing, using branches that verify lines and refine their positions.Examples include Faster R-CNN-style pipelines for meaningful straight lines and methods targeting line segments.
  • CNN-based Line Detection: LoI pooling aggregates deep features solely along each line, resulting in inadequate contextual information for line detection.The semantic-line framework extracts line proposals with LoI pooling before classification and regression, and introduces an intersection-over-union metric for straight lines.

3 APPROACH

The approach integrates a CNN encoder, Deep Hough Transform, parametric-space line detection, and Reverse Hough Transform in an end-to-end framework. It aggregates line and contextual features in the parametric domain, trains predictions there, and refines mapped-back lines with edge information.

  • 3 APPROACH: The proposed framework uses a CNN encoder, DHT, a parametric-space line detector, and RHT to detect semantic lines end to end.DHT converts spatial representations into the parametric domain, while RHT maps detected lines back to image space.
  • 3.1 Line Parameterization and Reverse: Lines are represented by orientation θ_l and distance r_l, with quantized parameters defining discrete candidate locations in the parametric space.The line-to-parameter and inverse mappings are bijective before r and θ are quantized into computer-processable bins.
  • 3.2 Deep Hough Transform: DHT aggregates features along each candidate line and assigns them to the corresponding location in the parametric feature space.With Θ·R candidate lines, the order-agnostic transform aggregates features for all candidates and is highly parallelizable.
  • 3.2.2 Multi-scale DHT with FPN: FPN features are transformed independently at multiple resolutions, resized to a common size, concatenated, and processed with convolutional layers for contextual line aggregation.Nearby lines become neighboring parametric points, allowing convolutional operations to capture contextual information around a central line.
  • 3.3 Loss function and 3.4 Reverse Mapping: Training converts ground-truth lines into the parametric space, smooths and expands them with a Gaussian kernel, and computes cross-entropy against predictions there.The detector predicts line-existence probabilities in the parametric space before thresholding, centroid extraction, and reverse mapping.
  • 3.5 Edge-guided Line Refinement: Edge-guided refinement searches nearby lines and selects the one with the highest average edge density.The candidate set is formed by moving both endpoints of a detected line by δr pixels in clockwise and anticlockwise directions.

4 THE PROPOSED EVALUATION METRIC

The section introduces EA-score, a line-similarity metric designed to avoid ambiguous IOU behavior by combining Euclidean and angular distances. It operates in parametric space and is reported as more efficient than pixel-rasterization metrics while producing similar results.

  • 4.2 The Proposed Metric: EA-score measures line agreement using both Euclidean distance and angular distance between the lines.The Euclidean component uses midpoint distance after normalizing the image to a unit square.
  • 4.1 Review of Existing Metrics: IOU can assign very different scores to similarly positioned line pairs and leaves the intersection area ambiguous.These issues are illustrated in Fig. 5(a) and (b).
  • 4.1 Review of Existing Metrics: EMD and Chamfer distance require rasterizing lines into pixels, making them less efficient for line-similarity evaluation.
  • 4.2 The Proposed Metric: EA-score is computed in parametric space and is more efficient than EMD and Chamfer distance.
  • 4.2 The Proposed Metric: Quantitative comparisons show that EA-score produces results very similar to EMD and Chamfer distance.
  • 4.2 The Proposed Metric: The EA-score is squared to increase sensitivity and discrimination when similarity values are high.

5 NKL: A SEMANTIC LINE DETECTION DATASET

NKL is a large-scale semantic line detection dataset created to address the limited scale and diversity of existing data. It contains diverse scenes, richer line annotations, and multi-annotator verification.

  • Dataset Overview: NKL contains 6,500 images with richer diversity in scenes and numbers of lines than the existing SEL dataset.Each image is annotated by multiple skilled human annotators.
  • Annotation: NKL annotations use three knowledgeable annotators per image, with inconsistent lines reviewed by two additional annotators.
  • Dataset Statistics: NKL contains 13,148 semantic lines, compared with 2,791 semantic lines in SEL.
  • Line Distribution: 67% of NKL images contain more than one semantic line, compared with 45.5% of SEL images.The NKL count is 4,356 of 6,500 images.
  • Scene Diversity: NKL has 327 unique scene labels, versus 167 in SEL, with a more even category distribution.The top three SEL categories exceed one-quarter of the dataset, while NKL’s top three comprise less than one-fifth.

6 EXPERIMENTS

The experiments compare methods on the SEL and NKL datasets using implementations based on PyTorch and CUDA. The setup uses standard CNN backbones, FPN multi-scale features, and fixed training procedures.

  • Implementation: The system is implemented in PyTorch, with a Jittor implementation also available.
  • Implementation: Deep Hough transform is implemented with native CUDA programming because it is highly parallelizable.Other components use framework-level Python APIs.
  • Network Architecture: ResNet50 and VGGNet16 backbones use FPN to extract multi-scale deep representations.The ResNet setup also uses dilated convolution in its last layer and batch normalization.
  • Training Configuration: Images are resized to 400 × 400, trained in batches of 8 for 30 epochs with Adam, and use left-right flip augmentation.The learning rate is 2 × 10^-4 and momentum is 0.9.
  • Experimental Setup: Experiments are conducted on both the public SEL dataset and the proposed NKL dataset.

6.2 Evaluation Protocol

The evaluation protocol matches predicted and ground-truth lines with bipartite matching before computing precision, recall, and F-measure. Performance is summarized across thresholds using average scores and multiple similarity metrics.

  • Metrics: Precision, recall, and F-measure are computed after matching the line sets.
  • Similarity Graph: Each bipartite-graph edge represents the similarity between one predicted line and one ground-truth line.The protocol evaluates EA-score alongside EMD and Chamfer distance.
  • Line Matching: Predicted and ground-truth lines are matched using maximum bipartite matching so each line is paired at most once.The matching can be solved with the Hungarian method in polynomial time.
  • Detection Counts: Matched lines count as true positives, unmatched predictions as false positives, and unmatched ground-truth lines as false negatives.
  • Evaluation Procedure: Thresholds from 0.01 through 0.99 produce score series that are summarized as average precision, recall, and F-measure.EMD, Chamfer distance, and EA-score are used for quantitative comparisons.

6.3 Tuning the Quantization Intervals

Quantization intervals trade off detection performance against computational efficiency. Experiments identify ∆r and ∆θ settings near performance turning points, with ∆θ = π/100 selected as appropriate.

  • The quantization intervals ∆θ and ∆r determine both the number of quantization levels and the performance-efficiency balance.
  • Larger ∆θ and ∆r produce fewer quantization levels and faster models, whereas smaller intervals increase computational overhead.
  • With ∆θ = π/100 fixed, performance changes as ∆r varies, revealing a turning point for selecting the distance interval.
  • Performance first improves smoothly as ∆θ decreases, then drops quickly with vibration; ∆θ = π/100 is therefore selected.

6.4 Quantitative Comparisons

The method is compared with SLNet and classical Hough transform using HED, with evaluation based on matched line predictions and multiple performance views. It achieves stronger detection performance and substantially faster inference across the reported comparisons.

  • Our method consistently outperforms SLNet and HT+HED on SEL with either VGG16 or ResNet50, including across a wide range of thresholds.
  • On NKL, our method outperforms the HED edge detector plus Hough transform baseline with a clear margin.
  • A single forward pass and centroid-based suppression in parametric space simplify inference compared with edge preprocessing and iterative refinement.

6.5 Qualitative Comparisons

Qualitative comparisons show detections that better agree with ground truth and human perception. The reported runtime comparison also characterizes the method as substantially faster than competing approaches.

  • Compared with SLNet and HED+HT, our detections are more compatible with ground truth and human cognition on example results.
  • The reported 49 FPS speed is markedly higher than the two competing methods.

6.6 Ablation Study

Ablation experiments evaluate multi-scale features, context-aware aggregation, and edge-guided refinement. Each component improves detection, while the complete method achieves the best reported combination performance.

  • Deep Hough transform: The ablation study explicitly compares DHT, multi-scale DHT, and the context-aware line detector as separate components.
  • Deep Hough transform: Both multi-scale features and context-aware aggregation improve the baseline model’s performance.
  • Deep Hough transform: Combining all components produces the best performance among the tested combinations.
  • Edge-guided Refinement: Performance with edge-guided refinement first increases and then saturates as δr grows, peaking at δr = 5.
  • Edge-guided Refinement: Edge-guided refinement improves detection results across different backbone architectures and datasets.

7 CONCLUSIONS

The paper presents an effective semantic line detector that combines CNN learning with the classical Hough transform. It also introduces a line-similarity metric and a larger dataset, with results showing improved detection quality and speed over previous methods.

  • The method incorporates CNN representations into the classical Hough transform for semantic line detection in natural scenes.This combination captures complex textures and rich contextual semantics of lines.
  • The paper designs a new evaluation metric that considers both Euclidean and angular distances between lines.
  • The authors construct a new semantic line detection dataset to address the gap between existing dataset scale and modern CNN complexity.
  • Quantitative and qualitative results show significantly better detection quality and speed than previous methods.
Loading 2003.04676v4…