Source-linked AI summary
A Simple Pooling-Based Design for Real-Time Salient Object Detection
Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, Jianmin Jiang
TL;DR
Salient object detection needs stronger global location information and better fusion of coarse semantic features with fine details in U-shape networks. PoolNet addresses this with pooling-based global guidance and feature aggregation modules, achieving state-of-the-art benchmark performance and more than 30 FPS on 300 × 400 images. The paper also shows that edge-detection joint training can enhance detected-object details.
Problem
U-shape saliency networks can dilute high-level location information during top-down transmission, while limited receptive fields restrict global-context capture.
Method
PoolNet adds a global guidance module and feature aggregation modules to an FPN, distributing semantic location guidance and fusing features across scales.
Results
PoolNet surpasses previous state-of-the-art approaches on six widely used salient-object-detection benchmarks.
Takeaways & Limitations
The pooling-based modules can be flexibly applied to pyramid-based models, while edge-detection joint training enhances salient-object details.
Takeaways & Limitations
Ground-truth edge maps derived from salient-object boundaries lack much detailed information, especially for low-contrast scenes and overlapping salient objects.
Abstract
from arXiv · showhide
We solve the problem of salient object detection by investigating how to expand the role of pooling in convolutional neural networks. Based on the U-shape architecture, we first build a global guidance module (GGM) upon the bottom-up pathway, aiming at providing layers at different feature levels the location information of potential salient objects. We further design a feature aggregation module (FAM) to make the coarse-level semantic information well fused with the fine-level features from the top-down pathway. By adding FAMs after the fusion operations in the top-down pathway, coarse-level features from the GGM can be seamlessly merged with features at various scales. These two pooling-based modules allow the high-level semantic features to be progressively refined, yielding detail enriched saliency maps. Experiment results show that our proposed approach can more accurately locate the salient objects with sharpened details and hence substantially improve the performance compared to the previous state-of-the-arts. Our approach is fast as well and can run at a speed of more than 30 FPS when processing a $300 \times 400$ image. Code can be found at http://mmcheng.net/poolnet/.
1. Introduction
PoolNet addresses diluted high-level location information in U-shape saliency networks by extending pooling with global guidance and feature aggregation modules. The resulting system improves salient-object localization and detail refinement while remaining fast.
- Motivation: U-shape networks can dilute deep-layer location information as semantic features move toward shallower layers, while limited receptive fields hinder global-context capture.These limitations can cause only parts of salient objects to be detected in complex scenes.
- Approach: PoolNet adds a global guidance module and a feature aggregation module to an FPN-based U-shape architecture.The GGM distributes high-level location information across pyramid levels, while FAMs fuse coarse semantic features with multiscale features.
- Approach: The GGM uses a modified pyramid pooling module and global guiding flows to deliver high-level semantic information throughout the top-down pathway.This design aims to prevent location information from being progressively diluted.
- Results: The proposed approach surpasses previous state-of-the-art methods on multiple salient-object-detection benchmarks and supports edge-joint training for sharper details.The paper also reports ablations examining each component and the effect of edge detection.
- Efficiency: More than 30 FPS is achieved on a single NVIDIA Titan Xp GPU for 300 × 400 inputs.Without the edge branch, training takes less than 6 hours on a 5,000-image training set.
2. Related Work
Related work moves from handcrafted saliency features toward CNN-based, pixel-wise prediction and increasingly multiscale, recurrent, attention-guided, and bidirectional architectures. PoolNet instead emphasizes simple pooling techniques to improve performance and running speed without introducing a new network architecture.
- CNN-based methods: CNN-based saliency methods surpassed many handcrafted-feature approaches by combining local, global, multiscale, or super-pixel information.These methods include CNN saliency estimation, global proposal searching, and multi-context feature extraction.
- Deep architectures: Later approaches predict pixel-wise saliency maps using recurrent refinement, U-shape structures, multilevel context, attention, or bidirectional feature messaging.The related methods target more accurate integration of coarse and fine information.
- PoolNet: PoolNet applies simple pooling techniques to CNNs rather than exploring new network architectures, aiming to improve performance and accelerate running speed.This positions pooling as the central design variable distinguishing the method from the cited approaches.
3. PoolNet
PoolNet augments an FPN with global guidance and feature aggregation to preserve salient-object location information and fuse features across scales. The modules improve localization and detail quality, while edge-detection joint training further sharpens detected objects.
- Overall Pipeline: PoolNet builds on FPNs and places a global guidance module on the bottom-up pathway to signal salient-object locations at different feature levels.The GGM aggregates high-level information and merges it into feature maps across the pyramid.
- Global Guidance Module: FPN top-down transmission can dilute high-level features, limiting global information and causing only parts of salient objects to be detected.The GGM is introduced to address this limitation.
- Global Guidance Module: The GGM combines a four-branch pyramid pooling module with identity-mapping global guiding flows that deliver semantic guidance to multiple pyramid levels.Its pooling branches include identity mapping, global average pooling, and adaptive outputs of 3 × 3 and 5 × 5.
- Experiments: Visual comparisons show that GGM improves salient-object localization, while repeated FAM use better sharpens details and fuses features across scales.Feature maps after FAMs capture salient objects more precisely than maps after two convolution layers.
- Feature Aggregation Module: FAMs reduce upsampling aliasing and expose each spatial location to local context at different scales, enlarging the network’s receptive field.The paper identifies these as the module’s two main advantages.
4. Joint Training with Edge Detection
The authors add an edge-detection branch to address unclear salient-object boundaries, but salient-object boundary supervision alone provides limited detail. Joint training with an external edge-detection task substantially improves detected object details.
- The baseline architecture already surpasses previous state-of-the-art single-model results on multiple salient object detection benchmarks.
- Unclear object boundaries cause many incomplete or over-predicted saliency maps.
- The added branch uses residual blocks after FAMs at three top-down feature levels to estimate salient-object boundaries.
- Joint training with salient-object boundary ground truths alone provides no performance gain and leaves some boundary details unresolved.
- Ground-truth edge maps mainly identify outermost salient-object boundaries, especially when salient objects overlap, and may lack detailed information.
- Joint training alternates salient-object and edge-detection datasets, greatly improving the details of detected salient objects.
5. Experimental Results
Experiments evaluate PoolNet through setup choices, ablations, quantitative benchmarks, precision–recall curves, and qualitative comparisons. The results show complementary module benefits, strong detection and detail refinement, and real-time operation.
- Experiment Setup: Experiments use six salient-object datasets and evaluate methods with precision–recall curves, F-measure, and mean absolute error.The study reports implementation details, datasets, loss functions, and evaluation criteria before ablation and comparison experiments.
- Ablation Studies: GGM improves both F-measure and MAE over the FPN baseline on DUT-O and SOD, helping preserve salient-object integrity and sharpen details.The authors attribute these gains to global guidance information that helps recover details otherwise estimated as background by models with limited receptive fields.
- Ablation Studies: FAMs also improve F-measure and MAE on DUT-O and SOD, with pooling enlarging the receptive field and helping address feature-map aliasing during upsampling.FAMs are evaluated by embedding them into the FPN baseline while keeping other configurations unchanged.
- Ablation Studies: Combining GGM and FAMs further enhances both F-measure and MAE, indicating that the modules complement accurate object discovery and detail refinement.Removing either PPM or GGFs from GGM also decreases performance, supporting the contribution of both components.
- Comparisons to the State-of-the-Arts: Precision–recall curves are especially strong on three datasets, with higher precision as recall approaches 1; qualitative results retain sharp boundaries across varied scenes.The authors interpret the curve behavior as indicating low false positives and report detail preservation for transparent, small, large, textured, and low-contrast objects.
6. Conclusion
PoolNet uses two pooling-based modules within an FPN and surpasses previous state-of-the-art methods on six salient object detection benchmarks. Joint edge-detection training enhances detected details, while the modules remain applicable to pyramid-based architectures.
- PoolNet combines a global guidance module and a feature aggregation module within an FPN architecture.The modules are designed to explore pooling for salient object detection.
- PoolNet surpasses previous state-of-the-art approaches on six widely-used salient object detection benchmarks.
- Joint end-to-end training with standard edge detection greatly enhances the details of detected salient objects.
- The modules are independent of network architectures and can be flexibly applied to pyramid-based models.