PaperlayerView exact revision on arXiv

Source-linked AI summary

Salient Object Detection via Integrity Learning

Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, Ling Shao

arXiv:2101.07663v7cs.CV

TL;DR

SOD has advanced but remains limited in preserving the integrity of predicted salient regions at micro and macro levels. ICON addresses this gap with diverse feature aggregation, integrity channel enhancement, and part-whole verification in a unified framework. It is evaluated on seven challenging datasets and is reported to outperform baseline methods across a wide range of metrics.

  • Problem

    SOD methods remain limited in predicted-region integrity, defined as covering all parts of each salient object and discovering all salient objects in an image.

  • Method

    ICON combines diverse feature aggregation, integrity channel enhancement, and part-whole verification to learn micro- and macro-level integrity features.

  • Results

    ICON is evaluated on seven challenging datasets and outperforms baseline methods across a wide range of metrics.

  • Takeaways & Limitations

    The unified framework addresses both macro- and micro-level integrity without additional supervision about the number of salient objects.

  • Takeaways & Limitations

    The authors report failure cases despite ICON rarely generating completely incorrect prediction results, and note that DFA produces higher MAE scores than other feature enhancement methods.

Abstract

from arXiv · show

Although current salient object detection (SOD) works have achieved significant progress, they are limited when it comes to the integrity of the predicted salient regions. We define the concept of integrity at both a micro and macro level. Specifically, at the micro level, the model should highlight all parts that belong to a certain salient object. Meanwhile, at the macro level, the model needs to discover all salient objects in a given image. To facilitate integrity learning for SOD, we design a novel Integrity Cognition Network (ICON), which explores three important components for learning strong integrity features. 1) Unlike existing models, which focus more on feature discriminability, we introduce a diverse feature aggregation (DFA) component to aggregate features with various receptive fields (i.e., kernel shape and context) and increase feature diversity. Such diversity is the foundation for mining the integral salient objects. 2) Based on the DFA features, we introduce an integrity channel enhancement (ICE) component with the goal of enhancing feature channels that highlight the integral salient objects, while suppressing the other distracting ones. 3) After extracting the enhanced features, the part-whole verification (PWV) method is employed to determine whether the part and whole object features have strong agreement. Such part-whole agreements can further improve the micro-level integrity for each salient object. To demonstrate the effectiveness of our ICON, comprehensive experiments are conducted on seven challenging benchmarks. Our ICON outperforms the baseline methods in terms of a wide range of metrics. Notably, our ICON achieves about 10% relative improvement over the previous best model in terms of average false negative ratio (FNR), on six datasets. Codes and results are available at: https://github.com/mczhuge/ICON.

1 INTRODUCTION

The paper identifies integrity as an under-explored SOD mechanism involving complete coverage within each salient object and discovery of all salient objects in a scene. ICON addresses both levels through diverse feature aggregation, integrity channel enhancement, and part-whole verification.

  • Motivation: SOD integrity has micro- and macro-level requirements: covering all parts of one salient object and identifying all salient objects in an image.The paper reports a strong correlation between integrity and prediction performance.
  • ICON contributions: DFA aggregates features across varied kernel shapes and contexts to increase feature diversity for mining integral salient objects.The authors contrast this with models focused primarily on feature discriminability.
  • ICON contributions: ICE enhances channels highlighting integral salient objects at both integrity levels while suppressing distracting channels.It operates on the diverse features produced by DFA.
  • ICON contributions: PWV checks agreement between part and whole features, further improving micro-level integrity when enhanced channels do not perfectly match salient regions.This verification is applied after extracting the enhanced features.
  • Novelty: ICON tackles macro- and micro-level integrity in one framework without requiring additional supervision about the number of salient objects.Existing auxiliary-task approaches addressing macro integrity require that additional information.
  • Framework: ICON uses five convolutional blocks, diverse feature aggregation, integrity channel enhancement, and part-whole verification within its processing framework.Deep features are aggregated across adjacent feature levels before channel enhancement and part-whole verification.
  • Evaluation: The network incorporates the three components and is evaluated on seven challenging datasets, while also achieving approximately 60fps real-time speed.The supplied passage reports both benchmark evaluation and speed.

2 RELATED WORK

Related SOD work has explored scale, boundary, and integrity learning through feature aggregation, contextual and top-down modeling, edge mechanisms, and specialized integrity approaches. ICON’s integrity modules aggregate varied receptive fields, enhance integral-object channels, and verify part-whole agreement.

  • Overview: SOD methods are broadly categorized into scale learning, boundary learning, and integrity learning approaches.These categories summarize the related-work landscape described by the paper.
  • Scale learning: Scale-learning methods address object-size variation through multi-level or multi-scale feature aggregation and contextual saliency cues.Examples include deep-to-shallow side outputs, multi-level aggregation, Fold-ASPP, and centralized information interaction.
  • Boundary learning: Boundary-learning methods improve contours using preprocessing, postprocessing, auxiliary edge streams, or boundary-aware losses.The paper notes that some preprocessing and postprocessing approaches have slow inference speed, while earlier approaches can blur or lose salient areas.
  • Integrity learning: Integrity learning remains under-explored, with prior methods modeling contrast, holistic attention, part-object relationships, global context, complementary features, or context learning.The paper cites DCL, CPD, TSPOANet, GCPANet, a bi-stream network, and a transformer-based architecture as examples.
  • Proposed approach: ICON’s DFA, ICE, and PWV modules respectively aggregate varied receptive fields, enhance integral-object channels, and judge part-whole feature agreement.These three components are presented as the proposed integrity-learning design.

3 FRAMEWORK

ICON uses an encoder–decoder framework that improves salient-region integrity through diverse feature aggregation, channel enhancement, and part–whole verification. These modules combine multi-level features, emphasize integrity-related channels, and verify agreement between object parts and whole regions.

  • Overview of ICON: ICON uses an encoder–decoder architecture with multi-level backbone features and multi-layer supervision to generate saliency maps.The decoder integrates features from multiple levels before producing the saliency map.
  • Diverse Feature Aggregation: DFA combines asymmetric, atrous, and standard convolutions to enrich feature diversity across receptive fields and contexts.The resulting features fuse crux-shaped, dilated, and normal-kernel knowledge to cover salient regions in varied contexts.
  • Integrity Channel Enhancement: ICE mines cross-channel integrity information and enhances channels that highlight potential integral salient targets.It processes three adjacent feature levels, forms a fusion map, and learns an integrity embedding for channel weighting.
  • Integrity Channel Enhancement: ICE outputs tend to highlight integral salient objects and distinguish foreground from background at both micro and macro levels.The paper reports this behavior in visual heatmap comparisons before and after ICE.
  • Part-Whole Verification: PWV uses capsule-network routing to measure agreement between object parts and whole salient regions, producing part–whole verified features.These features are then fused across adjacent levels through element-wise addition and upsampling.
  • Supervision Strategy: ICON combines cooperative BCE and IoU losses for saliency prediction, with IoU loss defined over predicted and ground-truth saliency maps.The overall loss is LCPR = LBCE + LIoU.

4 EXPERIMENTS

ICON is trained on DUTS-TR and evaluated on seven pixel-level annotated datasets spanning traditional, multiple-object, structurally complex, and complicated-scene SOD settings.

  • Datasets: ICON is trained on 10,553 DUTS-TR images and evaluated on seven popular pixel-level annotated datasets.The evaluation datasets are ECSSD, HKU-IS, OMRON, PASCAL-S, DUTS-TE, SOD, and SOC.
  • Datasets: The evaluation suite includes datasets with meaningful semantics, multiple foreground objects, structurally complex objects, and challenging or complicated scenes.SOC is described as more challenging than the other six datasets.

4.2 Implementation Details

The implementation uses standard deep-learning training infrastructure, fixed image resizing, SGD optimization, learning-rate scheduling, mixed precision, and gradient clipping.

  • Implementation Details: ICON is trained with SGD using lr = 0.05, momentum = 0.9, weight decay = 5e-4, warm-up, and linear learning-rate decay.Batch sizes vary by backbone, and training runs for up to 60 epochs.
  • Implementation Details: Apex, fp16, and gradient clipping are used to accelerate training and prevent gradient explosion.The ResNet-based training time is reported as approximately 2.5 hours.
  • Implementation Details: ResNet-based inference for a 352×352 image takes 0.0164s.This implementation detail corresponds to the reported inference process for the ResNet-based architecture.

4.3 Evaluation Metrics

The evaluation uses five metrics to assess saliency-map quality, structural similarity, alignment, and false negatives across benchmark datasets.

  • Metrics: Five metrics are used to evaluate ICON and existing state-of-the-art algorithms.The supplied passages introduce MAE, weighted F-measure, S-measure, E-measure, and FNR.
  • Metrics: MAE measures the average pixel-wise difference between predicted and ground-truth saliency maps.Both maps are normalized to [0, 1] before computing the average absolute difference.
  • Metrics: Weighted F-measure extends F-measure to non-binary evaluation by assigning location- and neighborhood-dependent error weights.The metric addresses interpolation, dependency, and equal-importance issues associated with MAE and F-measure.
  • Metrics: S-measure evaluates structural similarity using object-aware and region-aware components.The weighting factor m is set to 0.5.
  • Metrics: FNR measures false negative pixels and reflects prediction integrity at both macro and micro levels.The paper states that FNR is sensitive to missed salient regions and missed salient objects.

4.4 Comparison with the SOTAs

ICON is compared with recent state-of-the-art SOD methods on traditional and attribute-based benchmarks. It reports stronger quantitative and qualitative performance across diverse challenging scenes.

  • Quantitative Evaluation: ICON is compared with 14 recent state-of-the-art methods on six traditional benchmark datasets.The comparison uses S-measure, E-measure, weighted F-measure, and MAE.
  • Quantitative Evaluation: ICON achieves the lowest FNR scores across all six traditional datasets.The paper presents FNR comparisons in Fig. 7 and links the result to capturing integral objects.
  • Qualitative Evaluation: Qualitative comparisons report sharper edges and fewer background distractors than baseline methods.The comparisons cover small, large, delicate, low-contrast, and multiple-object scenes.
  • Quantitative Evaluation: ICON performs favorably across nearly all evaluation metrics and datasets.Its precision-recall and F-measure curves are reported as higher than competing curves in Fig. 8.
  • Attribute-Based Analysis: On the SOC test set, ICON achieves clear performance improvement over 16 state-of-the-art methods in attribute-based evaluation.The SOC dataset contains complicated scenes categorized by nine attributes.

4.5 Failure Cases

ICON still has failure cases in visually ambiguous or underrepresented scenes, including confusion over salient regions and missed lamp lights.

  • Failure Cases: ICON rarely produces completely incorrect predictions but still exhibits failure cases.These cases are illustrated in Fig. 10 alongside baseline predictions.
  • Failure Cases: In a tidy-room scene, ICON is confused about whether the pillow or the bed and wall form the salient object.The passage attributes this example partly to strong color contrast and controversial annotations.
  • Failure Cases: ICON misses three salient lamp lights in another failure case.The passage states that other state-of-the-art methods also fail on these samples.
  • Failure Cases: The authors identify strong color contrast, insufficient training samples, and controversial annotations as possible reasons for failures.The stated causes correspond to the tidy-room and lamp-light examples.

4.6 Ablation Study

The ablation study evaluates ICON by progressively adding DFA, ICE, and PWV-related design choices and by comparing DFA with other feature-enhancement methods.

  • Component Ablation: Adding DFA to the encoder-decoder baseline produces an obvious performance improvement.The study progressively extends a UNet-like baseline with DFA, ICE, and PWV modules.
  • DFA Comparison: DFA enhances feature sub-spaces without enlarging the receptive field and provides more diverse representations than competing FEMs.Compared with ASPP, Inception, and PSP, DFA uses fewer convolutional blocks while matching or surpassing performance on Sm and Em.
  • DFA Comparison: DFA can produce higher MAE scores than other feature-enhancement methods, possibly because its diversity also introduces noise.This is identified as a drawback of the module.
  • DFA Comparison: Combining three convolution types gives the best score, while using only 3xAsyConv generally outperforms using only 3xOriConv or 3xAtrConv.These findings come from the reported comparisons among the listed ablation IDs.

4.7 ICE vs. Attention Methods

Ablations show that ICE and the LCP R loss improve ICON, while routing alternatives vary in compatibility with PWV.

  • ICE comparison: CBAM ranks second among the four tested channel-enhancement methods, while SE and GCT cause noticeable performance drops.The comparison evaluates ICE against SE, CBAM, and GCT under the same setting.
  • Routing comparison: SR performs worse than EM routing in PWV, whereas dynamic routing achieves reasonable performance.The authors suggest SR lacks routing-by-agreement, making it incompatible with the PWV scheme.
  • Loss-function comparison: The LCP R loss significantly improves SOD performance across all metrics compared with LBCE.The comparison replaces LCP R with LBCE within the ICON architecture; IoU plus BCE is noted as a common alternative setting.

5 CONCLUSION

ICON integrates diverse feature aggregation, integrity channel enhancement, and part-whole verification to detect salient objects with micro- and macro-level integrity. Experiments on seven benchmark datasets demonstrate contributions from the components and state-of-the-art performance.

  • Conclusion: ICON is designed to detect salient objects by mining integral features at both micro and macro levels.The micro level concerns part-whole agreement within objects, while the macro level concerns detecting salient objects in scenes.
  • Conclusion: DFA captures diverse features, ICE enhances channels highlighting integral objects, and PWV verifies agreement between salient object parts and wholes.The three modules are integrated within ICON to support integrity learning.
  • Conclusion: Experiments on seven benchmark datasets demonstrate each proposed component's contribution and ICON's state-of-the-art performance.
Loading 2101.07663v7…