PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Edge Detection with Diverse Deep Supervision

Yun Liu, Ming-Ming Cheng, Deng-Ping Fan, Le Zhang, JiaWang Bian, Dacheng Tao

arXiv:1804.02864v5cs.CV

TL;DR

Semantic edge detection must locate fine edges and recognize their categories, but those distinct supervision targets challenge deep supervision. DDS uses information converters to separate supervision across bottom and top network sides. The authors report state-of-the-art performance on SBD and Cityscapes.

  • Problem

    SED requires both fine edge localization and high-level category recognition, yet these distinct targets have hindered effective use of deep supervision.

    Source: §1 · p. 2

  • Method

    DDS uses information converters to separate feature representations and apply category-agnostic supervision at bottom sides and semantic-edge supervision at top sides.

    Source: §1 · p. 2

  • Results

    DDS achieves state-of-the-art performance on the SBD and Cityscapes datasets.

    Source: §6 · p. 16

  • Takeaways & Limitations

    The results support the authors' conclusion that deep supervision can improve SED when the architecture addresses its distinct supervision targets.

    Source: §6 · p. 16

Abstract

from arXiv · show

Semantic edge detection (SED), which aims at jointly extracting edges as well as their category information, has far-reaching applications in domains such as semantic segmentation, object proposal generation, and object recognition. SED naturally requires achieving two distinct supervision targets: locating fine detailed edges and identifying high-level semantics. Our motivation comes from the hypothesis that such distinct targets prevent state-of-the-art SED methods from effectively using deep supervision to improve results. To this end, we propose a novel fully convolutional neural network using diverse deep supervision (DDS) within a multi-task framework where bottom layers aim at generating category-agnostic edges, while top layers are responsible for the detection of category-aware semantic edges. To overcome the hypothesized supervision challenge, a novel information converter unit is introduced, whose effectiveness has been extensively evaluated on SBD and Cityscapes datasets.

1 Introduction

Semantic edge detection (SED) must locate detailed edges while recognizing their categories, but these distinct targets have made deep supervision difficult to apply. DDS uses information converters to support different supervision at bottom and top network sides, and is reported to achieve state-of-the-art performance on SBD and Cityscapes.

1 Introduction

  • SED combines fine edge localization with high-level category recognition, making it a useful component in applications such as object proposals and recognition.The two targets rely mainly on low-level features for edge detail and high-level features for category semantics.Source: §1 · pp. 1–2
  • Directly supervising bottom sides for category-agnostic edges and top sides for semantic edges can degrade performance, motivating indirect separation of the targets.The authors hypothesize that distinct supervision targets hinder existing SED methods' use of deep supervision.Source: §1 · p. 2
  • DDS uses information converters to create different feature representations and loss functions for low-level and high-level feature learning.The converters act as buffers between distinct targets, assigning separate parameters and losses to network sides.Source: §1 · p. 2
  • The paper analyzes why distinct SED supervision targets challenge deep supervision and proposes DDS with information converters as its response.It also reports detailed ablation studies of the proposed method.Source: §1 · p. 2
  • DDS achieves state-of-the-art performance on SBD and Cityscapes, according to the authors.The paper presents this result as evidence that deep supervision can improve SED with an appropriately redesigned architecture.Source: §1 · p. 2

2 Related Work

Related work spans classical and deep category-agnostic edge detection, semantic edge detection, and deep supervision. Prior SED methods largely avoided bottom-side deep supervision because of the distinct targets, while this paper seeks to address that challenge.

Classical category-agnostic edge detection

  • Classical edge detection progressed from designed filters and models to data-driven methods using image cues, clustering, sketches, and learned edge-patch structure.Examples include Sobel and Canny filters, gPb, sketch tokens, and random decision forests.Source: §2 · p. 3

Deep category-agnostic edge detection

  • Deep category-agnostic detectors include HED, which fuses bottom and top convolutional features using a deeply supervised fully convolutional network.Later work introduced training strategies and real-time detectors; the cited account reports higher F-measure than average human annotators on BSDS500 for Liu et al. (2019; 2017).Source: §2 · p. 3

Semantic edge detection

  • Semantic edge detection methods combine contours or edge maps with category information, including approaches based on object detectors, segmentation, and weak supervision.Gated-SCNN converts semantic edge representations from ResNet layers into a representation for segmentation.Source: §2 · p. 3
  • CASENet, SEAL, DFF, and STEAL advance semantic edge detection, but prior methods give up bottom-layer deep supervision because of SED's distinct targets.SEAL training is reported to need over 16 days for CASENet on SBD, despite the cited powerful CPU.Source: §2 · pp. 3–4

Deep supervision

  • Deep supervision has supported multiple vision tasks, but applying different category-agnostic and category-aware targets to bottom and top sides may be suboptimal for SED.Traditional deep supervision generally uses the same supervision type at all layers.Source: §2 · p. 4

3 Distinct Supervision Targets in SED

SED must localize fine edges while recognizing their categories, but deep supervision at bottom and top sides can impose conflicting targets. The discussion traces this challenge in CASENet and motivates using information converters to apply distinct supervision indirectly.

3.1 A Typical Deep Model for SED

  • CASENet connects bottom-side feature maps with Side-5 semantic activations to produce category-specific edge maps, while DDS adds supervision across all network sides.Fig. 2 contrasts CASENet’s top-only supervision with DDS’s all-side supervision.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Discussion

  • CASENet’s earlier deep-supervision attempts performed worse than its basic architecture, and adding binary-edge supervision at bottom sides led to divergence in top-side semantic classification.The authors attribute these outcomes to bottom features being pulled toward edge/non-edge distinction while also serving top-layer semantic representations.Source: §3.2 · p. 4
  • The authors hypothesize that distinct supervision targets cause the failure of direct deep supervision and propose information converters to separate backbone information using unique parameters and separate losses.They report 70.9% mean F-measure for CASENet +S4 versus 71.4% for original CASENet, suggesting that treating res4 as a buffer may be reasonable.Source: §3.2 · p. 5

4 Methodology

DDS uses different supervision for low-level edge details and high-level semantic edges, with information converters intended to buffer their distinct targets. The methodology describes the network, converter, multi-task losses, and implementation settings.

4.1 Diverse Deep Supervision

  • DDS supervises Side-1 through Side-4 with category-agnostic binary edges and Side-5 with category-specific semantic edges, then fuses their upsampled outputs.The final fused semantic edges are supervised by semantic-edge ground truth.Source: §4.1 · p. 6

4.2 Information Converter

  • The information converter uses two residual modules to transform each side’s representation and buffer distinct supervision targets through separate parameters and losses.The authors emphasize the converter’s existence rather than its specific format.Source: §4.2 · p. 6
  • The authors report that DDS combines fine details from bottom sides with top-side semantic information, avoids divergence in Side-5 classification, and improves localization quality of fused semantic edges.They attribute the fine-detail contribution to binary edges produced by bottom sides.Source: §4.2 · p. 6

4.3 Multi-task Loss

  • DDS uses category-agnostic edge losses for Side-1 through Side-4 and multi-label semantic-edge losses for Side-5 and the fused output, summed for end-to-end optimization.The authors describe both reweighted and unweighted loss variants; the latter uses edges aligned once before training.Source: §4.3 · p. 7

4.4 Implementation Details

  • Implementation uses a ResNet backbone with modified strides and dilated convolutions, and trains for 25k iterations on SBD and 80k on Cityscapes.Training uses SGD on 352×352 crops; the paper reports an NVIDIA TITAN Xp GPU.Source: §4.4 · pp. 7–8

5 Experiments

The experiments test DDS under standard SBD and Cityscapes protocols and ablate supervision and converter designs. The ablations support coordinating deep supervision with information converters rather than adding supervision or layers alone.

5.1 Experimental Settings

  • Evaluation uses SBD training/test splits and Cityscapes training/validation splits, with ODS F-measure and both Thin and Raw protocols.The Raw protocol matches unthinned predictions against unthinned ground truth; the Thin protocol thins predictions before matching.Source: §5.1 · p. 8

5.2 Ablation Studies

  • The ablation variants are compared with DDS and CASENet on SBD using the original benchmark protocol; all but Softmax use the reweighted loss.Source: §5.2 · pp. 8–9, Table 1 · p. 9
  • Basic achieves 70.6% ODS F-measure, 0.3% above DSN; CASENet+S4 beats DSN but remains below CASENet.These results support the authors’ view that bottom features suit binary edge detection better than semantic classification.Source: §5.2 · p. 9, Table 1 · p. 9
  • Softmax degrades substantially because semantic edges can overlap across classes, making a single pixel label inappropriate; the study therefore uses multi-label loss.Source: §5.2 · p. 9
  • DDS’s improvement is attributed to coordinating deep supervision with information converters, not merely adding convolutional parameters.Adding layers without deep supervision may make convergence more difficult; the authors also report that direct binary-edge supervision at bottom sides is not useful.Source: §5.2 · pp. 9–10
  • DDS-U performs better than DDS-R when using unweighted loss and aligned edges.Source: §5.2 · p. 10
  • Progressive training performs significantly worse and cannot simulate information converters because it does not optimize the DCNNs well.The variant trains and freezes backbone blocks with their side outputs separately.Source: §5.2 · pp. 9–10

Discussion about the design of the information

The converter design is tested through architectural and size ablations, and DDS is evaluated on SBD and Cityscapes. DDS-U leads SBD competitors, while both DDS variants outperform competitors on Cityscapes under Thin and Raw metrics.

Discussion about the design of the information

  • Three residual converter units perform best, only slightly better than two; two are the default trade-off between model complexity and performance.Performance remains robust across converter sizes from 1/16 to 4 times the default, which the authors attribute mainly to converter presence rather than format.Source: Discussion about the design of the… · p. 11
  • Class-agnostic evaluation finds that DDS significantly improves edge localization accuracy on SBD.The evaluation takes the maximum edge probability across categories at each pixel and compares the resulting map with class-agnostic ground truth.Source: Discussion about the design of the… · p. 11

5.3 Evaluation on SBD

  • DDS-U’s ODS F-measure is 1.7% higher than SEAL and 3.4% higher than CASENet on SBD under the original benchmark metric.DDS-R also improves DFF, and DDS produces state-of-the-art semantic edges with only a slight reduction in speed.Source: §5.3 · p. 12
  • On re-annotated SBD, DDS-U’s ODS F-measure is 3.0% and 2.7% above SEAL on Thin and Raw metrics, respectively.With the same training strategy, DDS-R is 4.3% and 8.0% above CASENet on Thin and Raw metrics, respectively.Source: §5.3 · pp. 12–13, Table 5 · p. 12
  • DDS-R side features show clearer category-agnostic edges and clearer horse/person separation than DSN and CASENet activations.The figure shows predicted edges, enlarged regions, and horse- and person-only boundaries; activations are shown before sigmoid non-linearity.Source: Figure 4 · p. 11, Figure 5 · p. 12, §5.3 · p. 13
  • Qualitative examples show DDS edges as clearer and smoother, with strong responses on a small cat boundary where other detectors are weak or noisy.On Cityscapes, DDS-R/DDS-U also produce smoother, clearer edges in complicated scenarios.Source: §5.3 · p. 13, Figure 7 · p. 15, §5.4 · p. 16

5.4 Evaluation on Cityscapes

  • On Cityscapes, DDS-R is 2.8% above CASENet on Thin ODS F-measure, and DDS-U is 4.7% above SEAL on Thin ODS F-measure.Both variants significantly outperform other methods on both Thin and Raw metrics.Source: §5.4 · p. 13, Table 7 · p. 13

6 Conclusion

The conclusion argues that properly redesigned deep supervision can improve semantic edge detection, with DDS achieving state-of-the-art performance on SBD and Cityscapes. It presents information conversion as a perspective for multi-task learning with distinct supervision targets.

6 Conclusion

  • DDS achieves state-of-the-art performance on the SBD and Cityscapes datasets.The authors argue that architecture redesign makes deep supervision useful for SED.Source: §6 · p. 16
  • The information converter mediates distinct supervision targets, applying category-aware edges at top sides and category-agnostic edges at bottom sides.The authors describe this design as resolving the challenge of applying deep supervision to SED.Source: §6 · p. 16
  • The authors identify semantic segmentation, object detection, and instance segmentation as potential high-level tasks for leveraging deep supervision.These are proposed directions, not reported evaluations in this conclusion.Source: §6 · p. 16
  • DDS offers a perspective on multi-task learning, where distinct supervision targets can hinder networks for related tasks.The authors plan to apply information converters to other relevant task pairs.Source: §6 · p. 16
Loading 1804.02864v5…