Source-linked AI summary
End-to-End Wireframe Parsing
Yichao Zhou, Haozhi Qi, Yi Ma
TL;DR
Wireframe parsing systems commonly separate heat-map prediction from heuristic vectorization, while existing metrics overlook overlapped lines and connectivity. This paper presents an end-to-end approach with dynamic and static line sampling, introduces structural average precision, and reports stronger line-detection results than prior state-of-the-art methods.
Problem
Existing wireframe systems rely on complex heuristic vectorization, while heat-map-based metrics do not properly evaluate overlapped lines or wireframe connectivity.
Method
The method uses an end-to-end wireframe pipeline with static and dynamic line samplers, and evaluates vectorized outputs using structural average precision.
Results
13.3 points in APH and 4.0 points in FH separate the algorithm from previous state-of-the-art line detectors.
Takeaways & Limitations
Structural AP evaluates vectorized line segments and supports assessing connectivity relevant to downstream 3D-geometry inference.
Takeaways & Limitations
Existing boundary-detection metrics are problematic for wireframe detection.
Abstract
from arXiv · showhide
We present a conceptually simple yet effective algorithm to detect wireframes in a given image. Compared to the previous methods which first predict an intermediate heat map and then extract straight lines with heuristic algorithms, our method is end-to-end trainable and can directly output a vectorized wireframe that contains semantically meaningful and geometrically salient junctions and lines. To better understand the quality of the outputs, we propose a new metric for wireframe evaluation that penalizes overlapped line segments and incorrect line connectivities. We conduct extensive experiments and show that our method significantly outperforms the previous state-of-the-art wireframe and line extraction algorithms. We hope our simple approach can be served as a baseline for future wireframe parsing studies. Code has been made publicly available at https://github.com/zhou13/lcnn.
1. Introduction
Wireframe parsing targets the junctions and straight lines that form a scene’s global geometric structure. This paper addresses the limitations of heuristic, two-stage systems with an end-to-end network and a structural evaluation metric.
- Straight lines and their junctions form a wireframe, fundamental high-level geometric elements for assembling a scene’s 3D structures.
- Existing systems struggle with complex scenes containing complicated line connectivity.
- Prior methods predict junction and line heat maps, then use complex heuristic algorithms to vectorize segments and their connectivity.
- The central problem is learning a vectorized wireframe representation in an end-to-end trainable fashion.
- L-CNN uses one unified neural network to directly generate vectorized wireframes from RGB images without heuristics.
- The paper introduces a metric based on vectorized line matching because pixel-based metrics miss overlapped lines and incorrect connectivity.
2. Related work
Related work covers line detection and wireframe parsing, distinguishing methods that output lines alone from systems that model junctions and their connectivity. L-CNN contrasts with prior two-stage wireframe parsing by directly producing vectorized outputs through one trainable network.
- Line Detection: Traditional line detectors produce vectorized lines from local edge features but omit junctions and line–junction connectivity.
- Line Detection: Deep line detection combines learned image features with a line vectorization algorithm, while wireframe parsing additionally represents structural connections.
- Wireframe Parsing: Prior wireframe parsing predicts junction and line heat maps with separate networks before heuristic fusion into a vectorized output.
- Wireframe Parsing: L-CNN uses a single end-to-end trainable network that directly delivers a vectorized wireframe representation.
- L-CNN generates line proposals by connecting salient junctions rather than using sliding-window candidate generation.
3. Methods
L-CNN parses wireframes end to end by predicting junctions, sampling candidate lines, and verifying their connectivity in a unified differentiable network. Its modules use shared backbone features and complementary static and dynamic line sampling to produce vectorized wireframes without heuristic post-processing.
- 3.3. Backbone Network: L-CNN uses a stacked hourglass backbone to extract and refine shared semantic feature maps for subsequent wireframe modules.The backbone downsamples inputs twice, applies multiple U-Net-like hourglass modules, and uses intermediate supervision.
- 3.4. Junction Proposal Module: The junction proposal module divides each image into bins and predicts junction likelihoods and relative offsets for bins containing junctions.It uses binary cross entropy for likelihood classification, L2 regression for offsets, and non-maximum suppression to remove nearby duplicate responses.
- 3.5. Line Sampling Module: The line sampling module generates candidate segments by connecting proposed junctions and balances positive and negative training examples with static and dynamic samplers.Static sampling supplies ground-truth positives and hard negatives, while dynamic sampling uses predicted junctions and matched ground-truth connectivity.
- 3.5. Line Sampling Module: Static sampling supports training cold starts, while dynamic sampling adapts line endpoints to predicted junction locations and improves line detection.The two samplers complement each other during joint training.
- 3.6. Line Verification Network: The line verification module classifies candidate lines using backbone features pooled along each segment and outputs the wireframe’s junction positions and connectivity.During evaluation, it considers every pair of predicted junctions; training uses sampled candidates and balanced positive-negative losses.
- 3.6. Line Verification Network: LoI pooling samples uniformly spaced points along each line, bilinearly interpolates backbone features, and max-pools them into a fixed-length line representation.This differentiable operation allows gradients to back-propagate to the backbone while avoiding quantization artifacts.
4. Experiments
Experiments evaluate L-CNN’s implementation, structural metrics, ablations, and comparisons with prior line and wireframe detectors. L-CNN directly models junctions and line connectivity, achieving stronger performance across in-domain, structural, junction, and cross-dataset evaluations.
- Datasets and implementation: Experiments use ShanghaiTech for training and testing, with additional evaluation on York Urban to assess generalizability.ShanghaiTech contains 5,462 man-made-environment images: 5,000 training and 462 testing images.
- Evaluation metrics: Structural AP evaluates vectorized line segments through precision-recall curves, addressing overlapped lines and incorrect wireframe connectivity.Each ground-truth segment can be matched at most once, penalizing double-predicted lines.
- Ablation study: 58.9 sAP5 is achieved by the ablation configuration combining random dynamic sampling, static sampling, dynamic candidates, and hard examples.The ablation identifies carefully selected dynamic line candidates as vital to performance.
- Comparison with other methods: 13.3 points in APH and 4.0 points in FH separate L-CNN from previous state-of-the-art line detectors on heat map-based metrics.Figure 5a reports the strongest advantage particularly in the high-recall region.
- Comparison with other methods: 40-point sAP improvement over the previous state-of-the-art method is obtained when evaluating vectorized structural accuracy.The paper attributes this gap to L-CNN’s line verification network penalizing incorrect structures while heuristic heat-map vectorization loses junction connectivity.
- Generalization and visualization: L-CNN outperforms prior methods in junction mAP and remains best on York Urban without finetuning, while visualization shows fewer texture-driven and incorrectly connected lines.The York Urban scores are affected by duplicated and texture lines and by missing long lines in the annotations.
5. Supplementary Material
The supplementary material describes post-processing for overlapping lines, ablates coordinate-based line features, and specifies visualization settings for sampled outputs.
- 5.1. Post Processing: L-CNN post-processing removes or cuts overlapping lines caused by collinear junctions, improving visual appearance and selected metrics.Lines are compared in verification-score order; lower-ranked lines are deleted, shortened, or retained based on projection overlap.
- 5.1. Post Processing: 76.9 to 81.2 FH and 80.3 to 82.8 APH after post-processing on ShanghaiTech.The authors omit post-processing when structural correctness is evaluated with sAP.
- 5.2. Line Features: The manual six-dimensional line feature stores endpoint coordinates in its first four dimensions and normalized line directions in its last two.The coordinate-based feature is evaluated through separate endpoint-coordinate and slope components.
- 5.2. Line Features: Coordinate-based manual features do not improve performance, trigger premature validation-loss increases indicating overfitting, and are excluded from final L-CNN.The authors attribute the lack of improvement to the strength of features from LoIPool layers.
- 5.3. Qualitative Results: Randomly sampled wireframe results display L-CNN lines with scores above 0.98, using line colors to indicate confidence.The comparisons include LSD, Wireframe parser, and AFM under their reported hyper-parameters and cutoffs.