Source-linked AI summary
EDTER: Edge Detection with Transformer
Mengyang Pu, Yaping Huang, Yuming Liu, Qingji Guan, Haibin Ling
TL;DR
Edge detection must preserve precise local boundaries while representing broader image context. EDTER uses global and local transformer stages with high-resolution BiMLA decoding and feature fusion, and experiments on three benchmarks report superiority over state-of-the-art methods. Its extracted edges still occupy multiple pixels, leaving thin edge generation as future work.
Problem
Edge detection requires accurate boundaries and salient edges, but CNNs can suppress local details as receptive fields enlarge, while transformers commonly use coarse patches that are unfavorable for precise edges.
Method
EDTER uses a two-stage transformer framework that captures global context on coarse patches, local cues on fine-grained patches, and combines them through BiMLA decoding and feature fusion.
Results
EDTER outperforms state-of-the-art edge detectors across BSDS500, NYUDv2, and Multicue.
Takeaways & Limitations
Combining global context with fine-grained local cues yields clear, crisp, and less noisy edge maps for natural-image edge detection.
Takeaways & Limitations
EDTER’s extracted edges occupy multiple pixels, so generating clear and thin edges without post-processing remains future work.
Abstract
from arXiv · showhide
Convolutional neural networks have made significant progresses in edge detection by progressively exploring the context and semantic features. However, local details are gradually suppressed with the enlarging of receptive fields. Recently, vision transformer has shown excellent capability in capturing long-range dependencies. Inspired by this, we propose a novel transformer-based edge detector, \emph{Edge Detection TransformER (EDTER)}, to extract clear and crisp object boundaries and meaningful edges by exploiting the full image context information and detailed local cues simultaneously. EDTER works in two stages. In Stage I, a global transformer encoder is used to capture long-range global context on coarse-grained image patches. Then in Stage II, a local transformer encoder works on fine-grained patches to excavate the short-range local cues. Each transformer encoder is followed by an elaborately designed Bi-directional Multi-Level Aggregation decoder to achieve high-resolution features. Finally, the global context and local cues are combined by a Feature Fusion Module and fed into a decision head for edge prediction. Extensive experiments on BSDS500, NYUDv2, and Multicue demonstrate the superiority of EDTER in comparison with state-of-the-arts.
1. Introduction
EDTER addresses edge detection’s need to preserve local detail while capturing global context by combining two transformer stages with high-resolution decoding and feature fusion.
- 1. Introduction: Edge detection seeks accurate object boundaries and salient edges but remains challenging because of complex backgrounds and inconsistent annotations.
- 1. Introduction: EDTER extracts clear boundaries and meaningful edges by exploiting both global and local cues.
- 1. Introduction: CNNs improve semantic representation through larger receptive fields, but progressively enlarging receptive fields suppresses local details needed for precise edges.
- 1. Introduction: BiMLA decoders generate edge-aware high-resolution features, while an FFM integrates global and local information before edge prediction.
- 1. Introduction: EDTER combines global context and fine-grained local cues in a two-stage transformer framework for edge detection.Stage I processes coarse-grained patches with a global transformer, while Stage II processes fine-grained local patch sequences.
2. Related Work
Prior edge detectors range from hand-crafted gradient methods to learning-based and CNN-based hierarchical feature models. EDTER differs by applying transformers to generic edge detection through a two-stage global-context and fine-grained-local-cue design.
- 2. Related Work: Earlier edge detectors analyze image gradients, while learning-based methods combine low-level features and train classifiers for boundaries and edges.
- 2. Related Work: CNN-based methods improve edge detection through object-aware cues and hierarchical multi-scale features learned across levels.
- 2. Related Work: EDTER is presented as the first transformer-based model for generic edge detection, distinguishing it from prior CNN-centered approaches.
- 2. Related Work: EDTER’s key design learns global image context and fine-grained local cues in two stages while maintaining affordable computational cost.
3. Edge Detection with Transformer
EDTER combines global context from coarse patches with fine-grained local cues through two transformer stages, high-resolution BiMLA decoders, and feature fusion for edge prediction.
- Overview: EDTER uses Stage I global transformer encoding on coarse-grained patches and Stage II local transformer encoding on fine-grained patches to capture complementary context cues.The global stage models long-range context, while the local stage computes short-range cues from sampled image regions.
- Feature Fusion: The Feature Fusion Module uses global context as prior knowledge to modulate local context before a decision head predicts edge maps.Fusion features combine global context with fine-grained local details and are then passed to the local decision head.
- BiMLA Decoder: BiMLA decoders convert transformer representations into high-resolution, edge-aware features through top-down and bottom-up multi-level aggregation.The decoder uses bidirectional feature paths and learnable deconvolutional upsampling rather than only bilinear interpolation.
- Stage II: Local Refinement: Stage II samples non-overlapping windows with fixed patch counts, making attention complexity linearly related to image size while preserving fine-grained context.The local encoder processes 8×8 patches within sampled windows and concatenates their attentions before local BiMLA decoding.
- Network Training: Training addresses inconsistent BSDS500 annotations by converting multiple labels into edge-probability maps and thresholding pixels into positive or negative samples.The loss uses class weighting based on the proportions of negative and positive pixels.
4. Experiments
EDTER is evaluated on BSDS500, NYUDv2, and Multicue through component ablations and comparisons with state-of-the-art edge detectors. Results support the two-stage design, BiMLA decoder, and strong benchmark performance.
- Ablation Study: The two-stage strategy improves edge detection by combining global and local information, with Stage II significantly outperforming Stage I under either decoder.Removing FFM lowers ODS, OIS, and AP by 0.4%, 0.6%, and 1.3%, respectively.
- Ablation Study: BiMLA improves ODS, OIS, and AP by about 2.5%, 3%, and 3% over SETR-MLA in both stages while producing less noisy local edges.The qualitative decoder comparison is shown in Figure 4.
- Ablation Study: Using 4×4 rather than 8×8 Stage II patches slightly improves performance, while adding a third stage yields only marginal gains; the final setting uses 16×16 and 8×8 patches.The selected patch sizes balance computational efficiency and performance.
- Comparison with State-of-the-arts: 84.8% ODS, 86.5% OIS, and 90.3% AP on BSDS500 with extra training data and multi-scale testing surpass all compared state-of-the-art edge detectors.With single-scale testing, EDTER achieves 0.824 ODS, rising to 0.840 with multi-scale inputs.
- Comparison with State-of-the-arts: 77.4% ODS, 78.9% OIS, and 79.7% AP on NYUDv2 are the best reported scores, exceeding the second-best method by 2.6%, 2.6%, and 2.7%.All NYUDv2 results use single-scale input.
- Comparison with State-of-the-arts: EDTER produces competitive Multicue results, including 86.1% F-measure ODS on Multicue Boundary, higher than all other compared methods.The paper reports competitive performance on Multicue Edge.
5. Conclusion and Limitation
The paper concludes that EDTER combines coarse global and fine local context through a two-stage transformer framework with BiMLA and FFM, achieving competitive edge-detection results. A remaining limitation is that its extracted edges occupy multiple pixels rather than the ideal one-pixel width.
- Conclusion: EDTER combines coarse-grained global context and fine-grained local context through two transformer stages, BiMLA high-resolution features, and FFM fusion.The fused representations are used to predict final edge maps.
- Conclusion: EDTER achieves competitive results compared with state-of-the-art edge detectors across the reported experiments.
- Limitation: EDTER edges occupy multiple pixels, so generating clear and thin edges without post-processing remains an open direction.