PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Feature Integration for Simultaneous Detection of Salient Object, Edge and Skeleton

Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng

arXiv:2004.08595v1cs.CV

TL;DR

The paper asks how salient object segmentation, edge detection, and skeleton extraction can be learned together despite differing feature and optimization requirements. It introduces dynamic feature integration and task-adaptive attention in a unified end-to-end framework. Across multiple datasets, the method performs comparably to or better than task-specific state-of-the-art methods while running all three tasks at 40 FPS.

  • Problem

    Jointly learning saliency, edge, and skeleton prediction is challenging because the tasks require different features and have divergent optimization targets.

  • Method

    A shared-backbone framework uses dynamic feature integration to select task- and input-dependent features and task-adaptive attention to allocate information across branches.

  • Results

    The method outperforms existing single-purpose methods on multiple benchmarks, including average saliency F-measure gains of 1.2% over six datasets and a 1.9% skeleton F-measure gain on SK-LARGE.

  • Takeaways & Limitations

    The framework shows that the three tasks can be solved together in one end-to-end network, with simultaneous inference at 40 FPS.

Abstract

from arXiv · show

In this paper, we solve three low-level pixel-wise vision problems, including salient object segmentation, edge detection, and skeleton extraction, within a unified framework. We first show some similarities shared by these tasks and then demonstrate how they can be leveraged for developing a unified framework that can be trained end-to-end. In particular, we introduce a selective integration module that allows each task to dynamically choose features at different levels from the shared backbone based on its own characteristics. Furthermore, we design a task-adaptive attention module, aiming at intelligently allocating information for different tasks according to the image content priors. To evaluate the performance of our proposed network on these tasks, we conduct exhaustive experiments on multiple representative datasets. We will show that though these tasks are naturally quite different, our network can work well on all of them and even perform better than current single-purpose state-of-the-art methods. In addition, we also conduct adequate ablation analyses that provide a full understanding of the design principles of the proposed framework. To facilitate future research, source code will be released.

I. INTRODUCTION

The paper develops an end-to-end unified framework for simultaneously solving salient object segmentation, edge detection, and skeleton extraction despite conflicting task requirements. Dynamic feature integration and task-adaptive attention enable task-specific processing, while experiments report strong benchmark performance and 40 FPS inference.

  • I. INTRODUCTION: The framework jointly solves salient object segmentation, edge detection, and skeleton extraction with a shared backbone and three task branches.The tasks have different feature preferences and optimization targets, creating challenges for simultaneous learning.
  • I. INTRODUCTION: Dynamic feature integration lets each task select appropriate backbone features automatically, while task-adaptive attention exchanges information across branches.The design aims to adapt feature selection and information allocation to each task rather than fixing one architecture for all tasks.
  • I. INTRODUCTION: The unified system performs all three pixel-wise tasks at 40 FPS on 300 × 400 images without reported performance sacrifice.The framework is trained end-to-end on a single GPU.
  • I. INTRODUCTION: Across multiple benchmarks, the method outperforms existing single-purpose methods, improving saliency F-measure by 1.2% on average over six datasets and skeleton F-measure by 1.9% on SK-LARGE.The paper also evaluates the proposed components through extensive ablation experiments.

II. RELATED WORK

Prior work commonly fixes task-specific architectures or assumes jointly learned tasks are complementary. This paper instead uses dynamic feature integration in a unified network trained from multiple individual datasets.

  • II. RELATED WORK: Unlike methods with fixed feature-integration strategies, the proposed approach dynamically adjusts network connections according to training data, task, and input.It supports joint learning from multiple individual datasets rather than requiring all annotations in one dataset.
  • II. RELATED WORK: The framework uses a shared ResNet-50 backbone with pyramid pooling, followed by DFIMs and TAMs that produce task-specific predictions.DFIMs integrate backbone features at multiple output scales, while TAMs allocate information across tasks.

B. Dynamic Feature Integration

Dynamic Feature Integration Modules resize multi-level backbone features, estimate task-specific selection probabilities, and retain favored connections to construct flexible task-dependent feature maps.

  • B. Dynamic Feature Integration: DFIM receives six backbone feature stages, aligns their channels and resolution, summarizes them globally, and predicts task-specific feature-selection probabilities.Independent fully connected layers and softmax operations produce selection indicators for saliency, edge, and skeleton tasks.
  • B. Dynamic Feature Integration: TAM visualizations show task-tailored features, including more distinct saliency regions, sharper edges, and stronger skeletons.The figure compares features before TAM, attention maps, features after TAM, and predictions.
  • B. Dynamic Feature Integration: The modules retain only the top half of candidate feature connections instead of using dense connections.The selection is based on the probabilities generated for each task and input.
  • B. Dynamic Feature Integration: DFIMs with multiple down-sampling rates generate task-specific integrated maps whose feature combinations are learned end-to-end from the input and task type.The resulting feature space is broader and more flexible than a manually fixed integration strategy.

C. Task-Adaptive Attention

Task-Adaptive Attention addresses conflicting gradients after task-specific feature integration by allocating shared information across tasks. Visualizations show clearer saliency regions and thinner, sharper edge activations after TAM.

  • C. Task-Adaptive Attention: TAM allocates information globally across tasks by adaptively adjusting task gradients toward the shared backbone.Its parameters are shared across tasks to support information exchange after DFIM produces task-specific features.
  • C. Task-Adaptive Attention: After TAM, saliency features suppress background activation and edge features become thinner and sharper around likely boundaries.The visualized changes connect task-adaptive attention to more task-specific intermediate representations.

IV. EXPERIMENT SETUP

The experiments use separate datasets and task-specific annotations for three pixel-wise tasks, with end-to-end joint training and standard task losses and evaluation criteria.

  • The network is trained in PyTorch with Adam for 12 epochs on a single NVIDIA RTX-2080Ti GPU, using an ImageNet-pretrained ResNet-50 backbone.
  • The study uses individual datasets with one annotation type per task and no extra preprocessing.
  • Training samples one image-groundtruth pair per task, forwards the three pairs sequentially, sums their losses, and performs one optimization step.
  • The losses use binary cross-entropy for saliency and balanced binary cross-entropy for edge detection and skeleton extraction, summed as the overall loss.
  • The parameter composition table reports the feature extractor and shared components as the dominant parts of the proposed network.
  • Evaluation covers F-measure, MAE, PR curves, and S-measure for saliency, ODS and OIS for edges, and NMS-thinned skeleton maps.

V. ABLATION STUDIES

The ablation studies examine parameter composition, dynamic feature integration, and task-adaptive attention to assess the framework’s design choices.

  • The ablation plan analyzes parameter composition, tests DFIM in single- and multi-task settings, and evaluates TAM through convergence and performance.

A. Composition of Parameters

The parameter and dynamic-integration analyses show that most parameters are shared, while task-dependent feature selection preserves performance across heterogeneous tasks.

  • A. Composition of Parameters: 91.34% of parameters are shared, with ResNet-50 and PPM comprising 91.71% of the feature-extractor parameters and task-specific branches 2.87% each.
  • B. Dynamic Feature Integration: 1.7% higher performance on skeleton extraction appears in the single-task DFIM setting, while saliency and edge detection remain comparable to state of the art.
  • B. Dynamic Feature Integration: Joint training improves salient object segmentation on three datasets in nearly all terms and also increases edge-detection performance.
  • B. Dynamic Feature Integration: 0.7% and 3.6% drops occur for edge detection and skeleton extraction when identity fusion replaces sparse feature selection, while dynamic integration keeps all tasks comparable to separate training.
  • B. Dynamic Feature Integration: The selected backbone stages vary greatly across tasks, indicating that feature preferences depend on task characteristics.
  • B. Dynamic Feature Integration: Dense connections perform worse on nearly all three tasks, supporting selective use of backbone stages rather than retaining every feature map.
  • B. Dynamic Feature Integration: A wider range of DFIM down-sampling rates provides a better overall equilibrium, especially for salient object segmentation and edge detection.

C. Task-Adaptive Attention

Task-adaptive attention improves joint optimization by allocating shared information across branches, particularly for skeleton extraction under conflicting task annotations.

  • C. Task-Adaptive Attention: 0.7% higher skeleton-extraction performance and slightly better saliency and edge results are obtained with TAM than without it.
  • C. Task-Adaptive Attention: TAM adjusts task gradients toward the shared backbone because saliency and edge annotations emphasize edge pixels differently from skeleton annotations.
  • C. Task-Adaptive Attention: Adding 1.66M unshared TAM parameters performs worse overall than shared TAM, although skeleton extraction improves slightly while the other tasks decline greatly.
  • C. Task-Adaptive Attention: The final saliency results achieve the best outcomes on nearly all datasets for F-measure, MAE, and S-measure.

VI. COMPARISONS TO THE STATE-OF-THE-ARTS

DFI is compared with state-of-the-art methods for salient object segmentation across six datasets, using quantitative, curve-based, and visual evaluations.

  • Salient Object Segmentation: DFI outperforms the second-best salient-object methods across six datasets, improving average F-measure by 1.2% and S-measure by 1.0%.On DUTS-TE, the gains reach 2.1% in F-measure and 1.8% in S-measure.
  • Salient Object Segmentation: DFI’s precision-recall curves are comparable or superior to prior methods, especially on PASCAL-S and DUTS-TE.Its precision remains higher as recall approaches 1, indicating fewer false positives in the saliency maps.
  • Salient Object Segmentation: Visual comparisons show DFI recovering occluded, tiny, irregular, and low-contrast salient objects with more precise boundaries and details.The authors associate these results with better edge-pixel discrimination and whole-object segmentation.

B. Edge Detection

DFI is evaluated against established edge-detection and skeleton-extraction methods, with quantitative and visual results favoring the unified framework.

  • B. Edge Detection: DFI achieves ODS 0.819 and OIS 0.836 for edge detection, exceeding prior methods designed specifically for edges.The authors attribute the result to DFIM and TAM allowing information from other tasks to help rather than hurt edge detection.
  • B. Edge Detection: On BSDS 500, DFI’s precision-recall curve is comparable to previous methods and exceeds human performance in some cases.The reported advantage is especially apparent in precision.
  • B. Edge Detection: For edge detection, DFI produces cleaner backgrounds and more confident object boundaries than several recent detectors.Its visual comparisons also report stronger boundary responses, including on the wolf example.
  • C. Skeleton Extraction: DFI’s skeleton extraction improves F-measure by 1.9 points on SK-LARGE and 0.9 points on SYM-PASCAL.These comparisons use separate models trained on the two datasets, following prior methods.
  • C. Skeleton Extraction: Skeleton precision-recall curves substantially outperform selected existing methods on both SK-LARGE and SYM-PASCAL.The visual results are reported as thinner, stronger, and more continuous skeletons than three representative approaches.

D. Comparisons of Running Time

DFI is evaluated for both single-task and simultaneous three-task inference, combining competitive speed with joint prediction.

  • D. Comparisons of Running Time: DFI reaches 57 FPS in single-task mode while producing better detection results than compared methods.The speed comparison reports average FPS and corresponding input sizes under the same test environment.
  • D. Comparisons of Running Time: The reported runtime supports using one unified network for three pixel-wise prediction tasks rather than separate simultaneous models.This summarizes the paper’s stated simultaneous-inference capability without extending beyond the reported speed results.
Loading 2004.08595v1…