PaperlayerView exact revision on arXiv

Source-linked AI summary

U$^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection

Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaiane, Martin Jagersand

arXiv:2005.09007v3cs.CV

TL;DR

Salient object detection needs architectures that capture local and global information without depending on classification backbones or excessive computation. U2-Net uses nested U-structures with RSU blocks to train from scratch while combining multi-scale features at different depths. Across six public datasets, both provided model sizes achieve competitive performance, with configurations for distinct resource constraints.

  • Problem

    Existing SOD networks rely on classification backbones and often trade feature-map resolution for depth, complicating efficient multi-scale representation learning.

  • Method

    U2-Net uses a two-level nested U-structure whose stages contain residual U-blocks to extract intra-stage multi-scale and inter-stage multi-level features.

  • Results

    Both U2-Net variants achieve competitive performance against 20 state-of-the-art methods on six public SOD datasets; the full model is 176.3 MB and runs at 30 FPS, while U2-Net† is 4.7 MB and runs at 40 FPS.

  • Takeaways & Limitations

    The RSU-based design supports training from scratch and allows model sizing for different computation and memory constraints.

  • Takeaways & Limitations

    The authors identify a need for faster, smaller models and larger, more diversified salient-object datasets for more accurate and robust models.

Abstract

from arXiv · show

In this paper, we design a simple yet powerful deep network architecture, U$^2$-Net, for salient object detection (SOD). The architecture of our U$^2$-Net is a two-level nested U-structure. The design has the following advantages: (1) it is able to capture more contextual information from different scales thanks to the mixture of receptive fields of different sizes in our proposed ReSidual U-blocks (RSU), (2) it increases the depth of the whole architecture without significantly increasing the computational cost because of the pooling operations used in these RSU blocks. This architecture enables us to train a deep network from scratch without using backbones from image classification tasks. We instantiate two models of the proposed architecture, U$^2$-Net (176.3 MB, 30 FPS on GTX 1080Ti GPU) and U$^2$-Net$^{\dagger}$ (4.7 MB, 40 FPS), to facilitate the usage in different environments. Both models achieve competitive performance on six SOD datasets. The code is available: https://github.com/NathanUA/U-2-Net.

1. Introduction

U2-Net addresses limitations in salient object detection networks by avoiding classification backbones and using a nested architecture to combine depth, high resolution, and multi-scale features at modest cost.

  • Motivation: Existing SOD networks commonly rely on image-classification backbones whose features emphasize semantic meaning over local details and global contrast.These backbones also require ImageNet pre-training, which can be data-inefficient when target data follows a different distribution.
  • Motivation: Deeper SOD architectures often sacrifice feature-map resolution and incur high memory or computation costs, motivating deeper networks that preserve resolution affordably.High-resolution feature maps remain important for segmentation alongside architectural depth.
  • Contribution: U2-Net is a two-level nested U-structure designed for SOD without pre-trained classification backbones, enabling training from scratch with competitive performance.Each top-level stage contains a residual U-block that extracts intra-stage multi-scale features, while the outer structure aggregates features across stages.
  • Results: 176.3 MB U2-Net achieves competitive performance on six public datasets and runs at 30 FPS with 320×320×3 inputs on a 1080Ti GPU.A smaller 4.7 MB version, U2-Net†, is provided for computation- and memory-constrained environments.

2. Related Works

Prior SOD work mainly improves feature aggregation or multi-scale extraction around classification backbones; U2-Net instead directly extracts multi-scale features stage by stage.

  • Related work: Deep SOD networks generally outperform traditional hand-crafted-feature methods, motivating continued architectural development.Traditional approaches use cues such as foreground consistency, hyperspectral information, superpixel similarity, and histograms.
  • Multi-level deep feature integration: Multi-level feature integration methods aggregate representations from different deep layers, often producing saliency maps at multiple resolutions.Examples include patch-based descriptors, multi-resolution aggregation, short connections, iterative refinement, and feature-attention guidance.
  • Multi-scale feature extraction: Multi-scale extraction methods seek local and global context because small filters capture local features but struggle to obtain global information efficiently.Prior methods use modules such as pyramid pooling, global guidance, and multi-scale aggregation, typically on backbone features.
  • U2-Net’s positioning: U2-Net replaces increasingly complicated backbone-based modules with a simple architecture that directly extracts multi-scale features stage by stage.Its RSU blocks provide the paper’s alternative to developing additional strategies for aggregating classification-backbone features.

3. Proposed Method

U2-Net is a two-level nested U-structure built from residual U-blocks (RSUs), combining intra-stage multi-scale extraction with inter-stage feature aggregation. Its encoder-decoder design supports deep, high-resolution processing without significantly increasing memory and computation costs.

  • Residual U-blocks: RSU replaces a single-stream convolution with a U-Net-like encoder-decoder inside a residual block to extract multi-scale features directly.Its depth parameter L controls pooling, receptive-field range, and the richness of local and global features.
  • Residual U-blocks: RSU computation overhead remains small because most operations in its U-structure are applied to downsampled feature maps.This design targets deeper networks while maintaining high-resolution feature maps at affordable memory and computation cost.
  • Architecture of U2-Net: The nested U-structure uses a top-level U-Net with stages filled by RSUs, enabling intra-stage multi-scale extraction and inter-stage multi-level feature aggregation.U2-Net sets the nesting level to 2; its top level has 11 stages, including six encoder and five decoder stages.
  • Architecture of U2-Net: The encoder uses RSU-7, RSU-6, RSU-5, and RSU-4 in En 1 through En 4, while En 5 and En 6 use dilated RSU-4F blocks that preserve input resolution.The decoder has structures corresponding to the encoder stages, and the RSU height is configured according to feature-map resolution.
  • Supervision: Six side-output saliency maps are upsampled and fused through concatenation, a 1×1 convolution, and a sigmoid to produce the final saliency map.Training uses deep supervision with side-output and fusion losses, while testing selects the fusion output.
  • Architecture of U2-Net: The paper provides U2-Net and U2-Net† configurations of 176.3 MB and 4.7 MB, respectively, for different working environments.Both are built from RSU blocks without image-classification backbones, supporting training from scratch and adaptation with insignificant performance loss.

4.1. Datasets

The network is trained on DUTS-TR and evaluated on six commonly used salient-object-detection benchmark datasets with varied object and background complexity.

  • Training dataset: DUTS-TR contains 10553 training images and is augmented by horizontal flipping to produce 21106 training images.DUTS-TR is described as the largest and most frequently used training dataset for salient object detection.
  • Evaluation datasets: Evaluation uses DUT-OMRON, DUTS-TE, HKU-IS, ECSSD, PASCAL-S, and SOD, covering complex objects, multiple foregrounds, cluttered backgrounds, and challenging low-contrast images.The evaluation datasets contain 5168, 5019, 4447, 1000, 850, and 300 images, respectively.

4.2. Evaluation Metrics

The evaluation compares predicted saliency probability maps with binary ground-truth masks using six complementary measures covering threshold performance, error, structure, and boundary quality.

  • Six measures evaluate saliency maps: PR curves, maxFβ, MAE, weighted F-measure, S-measure, and relaxed boundary F-measure.The measures assess thresholded precision-recall performance, pixelwise error, structural similarity, and boundary quality.
  • PR curves average precision-recall pairs across dataset saliency maps while varying thresholds from 0 to 1.Each pair compares a thresholded predicted mask with the ground-truth mask.
  • maxFβ summarizes the maximum F-measure, with β^2 set to 0.3 for each dataset.Fβ combines precision and recall into a single score.
  • MAE measures the average per-pixel difference between a predicted saliency map and its ground-truth mask.The calculation averages absolute differences over image height, width, and pixel coordinates.
  • Weighted F-measure complements maxFβ by addressing interpolation, dependency, and equal-importance flaws.It is included to reduce possible unfair comparisons between saliency predictions.
  • S-measure evaluates structural similarity between non-binary predicted saliency maps and ground truth using region-aware and object-aware components.The measure combines region-aware Sr and object-aware So structural similarity.
  • Relaxed boundary F-measure evaluates predicted boundaries against ground-truth boundaries using precision and recall within a pixel slack range.Predicted maps are thresholded at 0.5, and boundary matches allow a range of ρ pixels.

4.3. Implementation Details

Training uses resized, augmented images and scratch initialization without existing backbones, followed by Adam optimization until convergence.

  • Input images are resized to 320×320, vertically flipped randomly, and cropped to 288×288 during training.
  • The network is trained from scratch without existing backbones, with convolutional layers initialized using Xavier initialization.
  • Adam uses a 1e-3 learning rate, betas of (0.9, 0.999), eps of 1e-8, and zero weight decay.
  • Training runs without a validation set until convergence; after 600k iterations with batch size 12, it takes about 120 hours.

4.4. Ablation Study

Ablation studies compare blocks, architectures, and backbones, showing that RSU-based U2-Net variants outperform alternatives while avoiding significant inference-time increases.

  • Ablation Setup: The ablations examine basic blocks, architectures, and backbones under the same implementation setup.
  • Ablation Setup: Table 2 compares plain, residual, dense, inception, pyramid-pooling, RSU, naive U-Net, and backbone-based variants using performance and inference time.Time is averaged over ECSSD images, and bold values mark the best two performances.
  • Blocks: 3.3% and 1.8% maxFβ improvements, with 12.9% and 21.4% MAE reductions, distinguish full-size U2-Net from the second-best block-ablation model on DUT-OMRON and ECSSD.
  • Blocks: 9.8% and 8.8% maxFβ gains, alongside 34.1% and 27.0% MAE reductions, compare U2-Net and U2-Net† against baseline U-Net on DUT-OMRON.
  • Blocks: 50.0% and 38.0% MAE improvements on ECSSD exceed the corresponding 5.5% and 4.7% maxFβ improvements against baseline U-Net.
  • Blocks: RSU-based U2-Net architectures show no significant increase in time costs while outperforming the alternative blocks.
  • Backbones: Backbone ablations replace the full-size U2-Net encoder with adapted VGG16 and ResNet50 encoders to match the original receptive fields.

4.5. Comparison with State-of-the-arts

U2-Net achieves state-of-the-art or competitive performance across six SOD datasets, while its compact variant offers substantially lower model size with strong qualitative results across varied targets.

  • U2-Net achieves state-of-the-art performance on DUT-OMRON, HKU-IS, and ECSSD, and competitive performance on the other datasets.
  • 4.7 MB U2-Net† is the smallest model in salient object detection and still delivers surprisingly competitive performance.Its smaller size supports applications in computation- and memory-constrained environments.
  • U2-Net and U2-Net† produce accurate results for small and large objects, border-touching targets, thin structures, cluttered backgrounds, and multiple targets.The models also capture fine structures and complicated foreground appearances in qualitative comparisons.
  • Precision-recall curves compare U2-Net and U2-Net† with typical state-of-the-art methods across six SOD datasets.
  • U2-Net and U2-Net† produce more homogeneous prediction maps with fewer gray areas than several competing methods on challenging targets.
  • The models can segment fine structures that are absent from imperfect ground-truth annotations, supported by high-resolution local and low-resolution global information.

5. Conclusions

The paper concludes that U2-Net’s nested U-structure and RSU blocks support salient object detection from scratch with configurable model sizes and competitive results. It also identifies the need for faster, smaller models and more diverse training datasets.

  • U2-Net uses a two-level nested U-structure with RSU blocks to capture richer local and global information across shallow and deep layers.
  • 176.3 MB U2-Net runs at 30 FPS, while 4.7 MB U2-Net† runs at 40 FPS for different target-environment constraints.
  • Both models achieve very competitive qualitative and quantitative performance against 20 state-of-the-art methods on six public SOD datasets.
  • Faster and smaller models are still needed for computation- and memory-limited devices such as mobile phones and robots.
  • Larger and more diversified salient object datasets are needed to train more accurate and robust models.
Loading 2005.09007v3…