PaperlayerView exact revision on arXiv

Source-linked AI summary

Line Segment Detection Using Transformers without Edges

Yifan Xu, Weijian Xu, David Cheung, Zhuowen Tu

arXiv:2101.01909v2cs.CV

TL;DR

Line segment detection remains difficult in cluttered, ambiguous scenes, while existing systems depend on heuristic intermediate processing and bounding-box formulations that poorly fit elongated segments. LETR addresses this with a joint end-to-end Transformer using multi-scale refinement and direct endpoint-distance learning, achieving state-of-the-art results on Wireframe and YorkUrban.

  • Problem

    Line segment detection is an important but unsolved mid-level vision problem, and existing heuristic pipelines and bounding-box representations are poorly suited to its geometric structure.

  • Method

    LETR jointly detects line segments with a Transformer that uses tokenized entities, self-attention, multi-scale encoder-decoder refinement, and direct endpoint-distance loss.

  • Results

    LETR achieves state-of-the-art results on the Wireframe and YorkUrban benchmarks.

  • Takeaways & Limitations

    The approach directly models and predicts geometric line segments without explicit edge, junction, region, proposal, or heuristic perceptual-grouping stages.

  • Takeaways & Limitations

    Directly applying DETR to line segment detection is unsatisfactory because elongated line segments are not well represented by bounding boxes.

Abstract

from arXiv · show

In this paper, we present a joint end-to-end line segment detection algorithm using Transformers that is post-processing and heuristics-guided intermediate processing (edge/junction/region detection) free. Our method, named LinE segment TRansformers (LETR), takes advantages of having integrated tokenized queries, a self-attention mechanism, and an encoding-decoding strategy within Transformers by skipping standard heuristic designs for the edge element detection and perceptual grouping processes. We equip Transformers with a multi-scale encoder/decoder strategy to perform fine-grained line segment detection under a direct endpoint distance loss. This loss term is particularly suitable for detecting geometric structures such as line segments that are not conveniently represented by the standard bounding box representations. The Transformers learn to gradually refine line segments through layers of self-attention. In our experiments, we show state-of-the-art results on Wireframe and YorkUrban benchmarks.

1. Introduction

LETR reframes line segment detection as a joint, end-to-end Transformer problem without heuristic edge, junction, region, proposal, or grouping stages. It extends DETR with multi-scale encoding/decoding and a direct endpoint-distance loss for geometric line structures.

  • Line segment detection remains difficult because meaningful segments can be locally ambiguous, partially occluded, and embedded in cluttered backgrounds.
  • Existing systems rely on heuristics-guided edge, junction, region, grouping, and post-processing modules that limit end-to-end development.
  • LETR uses tokenized entities, self-attention, integrated encoder-decoder processing, and joint query inference to detect line segments directly.
  • LETR achieves state-of-the-art results on the Wireframe and YorkUrban benchmarks without explicit edge, junction, region, or heuristic grouping processes.
  • Its multi-scale encoder/decoder and direct endpoint distance loss adapt DETR to fine-grained geometric structures that bounding boxes represent poorly.

2. Related Works

Prior line-segment detectors typically construct segments through edge, junction, attraction-field, or other intermediate representations and post-processing. LETR instead directly predicts vectorized segments in a general-purpose Transformer framework with multi-scale encoding and decoding.

  • Traditional methods detect edges, group low-level cues into line-support regions, and approximate line segments from those regions.
  • Junction-based methods predict junction and line heatmaps or junction proposals, then merge, verify, or filter resulting line proposals.
  • Dense-prediction approaches use attraction-field maps as surrogate representations before recovering vectorized line segments through post-processing.
  • LETR removes heuristic edge, junction, region-proposal, and surrogate-map stages while directly predicting vectorized line segments.
  • LETR augments DETR with a multi-scale encoder-decoder and a direct distance loss for line segments.

3. Line Segment Detection with Transformers

LETR directly predicts vectorized line segments with a Transformer encoder-decoder, replacing heuristic intermediate representations and adapting DETR to fine-grained geometric prediction. It uses multi-scale coarse-to-fine refinement, line entities, bipartite matching, and endpoint-based losses.

  • 3. Line Segment Detection with Transformers: LETR directly predicts vectorized line segments from images, avoiding heuristic edge, junction, region, proposal, and perceptual-grouping stages.The approach is built purely on a Transformer encoder-decoder structure rather than explicit intermediate detections or surrogate maps.
  • 3. Line Segment Detection with Transformers: Multi-scale decoders progressively refine line entities, combining fine-grained line details with global scene structure and communication among entities.The coarse-to-fine process uses interactions with multi-scale encoded features to produce final line entities.
  • 3. Line Segment Detection with Transformers: Each final line entity predicts a confidence and two endpoint coordinates, parameterizing one line segment.A feed-forward network contains classification and regression modules for these predictions.
  • 3. Line Segment Detection with Transformers: Hungarian bipartite matching assigns predictions to ground-truth segments using both confidence and coordinate distance, with unmatched predictions handled during inference by confidence thresholding.The matching objective balances distance and confidence, and the optimal permutation maps positive predictions to targets.
  • 3. Line Segment Detection with Transformers: Training combines focal-inspired classification weighting with an L1 endpoint distance loss instead of GIoU, which is designed for bounding boxes rather than line segments.The distance loss is applied to positive predictions and measures coordinate differences between predicted and target endpoints.

4. Experiments

LETR is evaluated on Wireframe and YorkUrban using multi-scale Transformer processing, structural and heatmap-based metrics, qualitative comparisons, and precision-recall curves. It achieves strong benchmark performance, with its main limitation appearing when predictions are heavily filtered.

  • Experimental Setup: LETR uses Wireframe and YorkUrban, with 5,000 training and 462 testing images for Wireframe and 102 testing images for YorkUrban.
  • Implementation: The model combines coarse and fine multi-scale encoder-decoder structures, each with 6 encoder and 6 decoder layers, using ResNet features at different resolutions.The coarse encoder uses Conv5 features, while the fine encoder uses higher-resolution Conv4 features.
  • Evaluation Metrics: Evaluation uses heatmap-based APH and FH together with structural sAP and sF metrics based on geometric line matching.Structural matches require L2 distance below a threshold of 10 or 15.
  • Results and Comparisons: LETR reaches state-of-the-art performance on YorkUrban and is consistently better than other models on heatmap-based metrics across both benchmarks.It outperforms HAWP by 1.5 APH on YorkUrban.
  • Results and Comparisons: When fewer predictions are retained, LETR has lower precision than HAWP, but including all predictions makes LETR slightly better than HAWP and other leading methods.

5. Ablation Study

The ablations show that LETR depends on geometric modeling, staged coarse-to-fine training, many line entities, image upsampling, and pretraining. These choices improve performance or address limitations of treating line segments as bounding-box objects.

  • Object Detection Baselines: Bounding-box diagonals in Faster R-CNN and DETR show clear limitations for line segment detection on Wireframe.
  • Multi-Stage Training: Fine decoding improves sAP10 and sAP15 from 63.8 and 66.5 to 64.7 and 67.4, while focal-loss tuning reaches 65.2 and 67.7.These results follow coarse decoding with C4 features.
  • Multi-Stage Training: Training the fine stage after coarse-stage convergence is necessary, because joint one-stage training produces significantly worse performance after 400 epochs.
  • Number of Queries: Using 1000 line entities is optimal on Wireframe, which contains 74 line segments on average, showing that many predictions are needed for this task.
  • Image Upsampling: Higher training upsampling improves both HAWP and LETR, while LETR gains further from higher test upsampling resolution.
  • Pretraining: COCO-pretrained DETR weights enable faster convergence and stronger results than ImageNet pretraining, while no pretraining makes LETR difficult to train on Wireframe.

6. Visualization

LETR progressively refines line segments from coarse global structure to fine detailed predictions through multi-scale decoding and attention. The fine decoder uses higher-resolution features and coarse line entities to improve precision.

  • Coarse-to-Fine Decoding: Coarse decoding captures global scene structure efficiently, but low-resolution features prevent precise line predictions.
  • Coarse-to-Fine Decoding: Fine decoding receives higher-resolution C4 features and coarse line entities, enabling more detailed image-space relations through attention.
  • Coarse-to-Fine Decoding: Decoded outputs improve consistently across layers, while attention heatmaps show increasingly detailed relations during coarse-to-fine refinement.

7. Conclusion

LETR is a multi-scale encoder/decoder Transformer for holistic, end-to-end line segment detection. It performs set prediction without heuristic intermediate stages and uses endpoint distance loss for geometric structures beyond bounding boxes.

  • LETR uses a multi-scale encoder/decoder Transformer to perform end-to-end line segment set prediction without explicit edge, junction, or region detection.The method also removes heuristics-guided perceptual grouping processes from the detection pipeline.
Loading 2101.01909v2…