PaperlayerView exact revision on arXiv

Source-linked AI summary

Re-thinking Co-Salient Object Detection

Deng-Ping Fan, Tengpeng Li, Zheng Lin, Ge-Peng Ji, Dingwen Zhang, Ming-Ming Cheng, Huazhu Fu, Jianbing Shen

arXiv:2007.03380v4cs.CV

TL;DR

Existing CoSOD datasets emphasize visual appearance similarity, despite real-world groups often sharing semantic or conceptual similarity across varied textures, scenes, and backgrounds. The paper introduces a challenging benchmark, a unified trainable framework, and comprehensive evaluation, with CoEG-Net outperforming 18 state-of-the-art models while CoSOD3k broadens dataset diversity and annotations.

  • Problem

    Existing CoSOD datasets focus on appearance similarity, creating data-selection bias because real-world co-salient objects may differ in texture, scene, and background while sharing a category.

  • Method

    The paper builds CoSOD3k, integrates SOD techniques into the trainable CoEG-Net framework, and benchmarks 18 models across three challenging datasets.

  • Results

    CoEG-Net outperforms 18 state-of-the-art models, while CoSOD3k contains 160 groups and 3,316 images with category, bounding-box, object-level, and instance-level annotations.

  • Takeaways & Limitations

    The benchmark and framework provide a unified basis for evaluating CoSOD methods and studying semantically related objects across diverse visual conditions.

  • Takeaways & Limitations

    Existing CoSOD datasets contain only dozens of groups and object-level labels, limiting evaluation of method scalability and richer multi-task modeling.

Abstract

from arXiv · show

In this paper, we conduct a comprehensive study on the co-salient object detection (CoSOD) problem for images. CoSOD is an emerging and rapidly growing extension of salient object detection (SOD), which aims to detect the co-occurring salient objects in a group of images. However, existing CoSOD datasets often have a serious data bias, assuming that each group of images contains salient objects of similar visual appearances. This bias can lead to the ideal settings and effectiveness of models trained on existing datasets, being impaired in real-life situations, where similarities are usually semantic or conceptual. To tackle this issue, we first introduce a new benchmark, called CoSOD3k in the wild, which requires a large amount of semantic context, making it more challenging than existing CoSOD datasets. Our CoSOD3k consists of 3,316 high-quality, elaborately selected images divided into 160 groups with hierarchical annotations. The images span a wide range of categories, shapes, object sizes, and backgrounds. Second, we integrate the existing SOD techniques to build a unified, trainable CoSOD framework, which is long overdue in this field. Specifically, we propose a novel CoEG-Net that augments our prior model EGNet with a co-attention projection strategy to enable fast common information learning. CoEG-Net fully leverages previous large-scale SOD datasets and significantly improves the model scalability and stability. Third, we comprehensively summarize 40 cutting-edge algorithms, benchmarking 18 of them over three challenging CoSOD datasets (iCoSeg, CoSal2015, and our CoSOD3k), and reporting more detailed (i.e., group-level) performance analysis. Finally, we discuss the challenges and future works of CoSOD. We hope that our study will give a strong boost to growth in the CoSOD community. The benchmark toolbox and results are available on our project page at http://dpfan.net/CoSOD3K/.

1 INTRODUCTION

CoSOD detects salient objects shared within or across images, but existing datasets overemphasize visual similarity and provide limited evaluation scope. This paper addresses these gaps with a realistic benchmark, unified framework, and broad model comparison.

  • Existing CoSOD datasets often select image groups whose salient objects have similar visual appearances, whereas real-world groups may share only category-level similarity across texture, scene, or background.This data-selection bias motivates a more challenging in-the-wild setting requiring semantic context.
  • The paper identifies completeness and fairness as evaluation concerns because common metrics have limitations and F-measure varies with binarization strategy.It calls for more accurate evaluation metrics and consistent reporting of thresholding procedures.
  • The paper proposes CoSOD3k, containing 13 super-classes, 160 groups, and 3,316 images with category, bounding-box, object, and instance annotations.The dataset is designed to cover diverse scenes and support related vision tasks beyond CoSOD.
  • The study reviews 40 state-of-the-art CoSOD models and evaluates 18 of them on iCoSeg, CoSal2015, and CoSOD3k using a unified benchmark toolbox.The toolbox integrates publicly available datasets and multiple metrics for performance evaluation.
  • CoEG-Net combines co-attention projection with a basic SOD network to embed appearance and semantic features for CoSOD.The framework is presented as a simple, trainable baseline that outperforms 18 state-of-the-art models.

2 RELATED WORK

Existing CoSOD datasets are limited in scale, variety, and annotation richness, while prior methods use diverse handcrafted and deep representations to model correspondence across images.

  • Existing datasets: WICOS differs from multi-image datasets by treating each single image as one group for co-salient object detection.
  • Dataset limitations: Existing CoSOD datasets contain only dozens of groups, limiting evaluation of method scalability and providing few annotations beyond object-level labels.The paper focuses on iCoSeg, CoSal2015, and CoSOD3k for deeper analysis.
  • Traditional methods: Earlier CoSOD methods model inter-image correspondence using computational units such as superpixels or pixel clusters, combined with heuristic and high-level semantic features.
  • Deep methods: Deep CoSOD models learn joint co-salient representations through domain adaptation, collaborative group-single-image learning, or multi-view features such as SIFT, LBP, and color histograms.

3 COSOD3K DATASET

CoSOD3k is constructed from diverse object-recognition images with hierarchical labels and quality-controlled masks. Its statistics cover varied categories, instance counts, and object sizes, while co-attention projection is designed to retain common objects and suppress interference.

  • Dataset construction: CoSOD3k is built from ILSVRC images selected for diverse categories, realistic scenes, and varied object appearances, then annotated hierarchically from coarse to fine.The hierarchy includes category, bounding-box, object-level, and instance-level information.
  • Hierarchical annotation: The taxonomy contains 160 common sub-classes organized into 13 super-classes, covering categories such as Vegetables, Food, Tool, Traffic, Animal, and Others.
  • Hierarchical annotation: Bounding boxes are redrawn around attention-dominating objects by three viewers and checked by two senior researchers, while twenty annotators produce object- and instance-level masks.
  • Annotation quality: CoSOD3k contains 3,316 object-level and 4,915 instance-level annotations after bounding boxes are refined using instance-level annotations.
  • Co-attention projection: Co-attention projection identifies principal components of common objects and projects them into new features while removing noisy foregrounds and background clutter.
  • Dataset statistics: The dataset includes multi-instance scenes with instance-count ratios of 7:2:1 for images containing 1, 2, and ≥3 instances, respectively.
  • Dataset statistics: Instance sizes range from 0.02% to 86.5% of image pixels, with an average of 13.8%, providing a broad size distribution.

4 PROPOSED METHOD

CoEG-Net extends EGNet with unsupervised co-attention information, combining common-object cues with single-image saliency priors. Its projection estimates informative common activation directions from feature covariance, and the resulting maps are fused into final co-saliency predictions.

  • CoEG-Net extends EGNet by introducing co-attention information in an unsupervised manner for CoSOD.
  • The framework uses separate branches to capture concurrent dependencies and salient foregrounds, producing co-attention and saliency prior maps independently.
  • Element-wise multiplication of each co-attention map A_n and saliency prior S_n produces the final prediction A_n ⊗ S_n.
  • EGNet supplies multi-scale saliency priors from DUTS-trained single-image SOD, while the co-attention branch discovers common information without cross-image supervision.
  • The co-attention projection seeks a linear transformation with maximum activation-map variance by analyzing the covariance of image-feature descriptors.
  • The eigenvector associated with the largest covariance eigenvalue defines the most informative projection for generating co-attention maps.

5.1 Experimental Settings

The experiments evaluate CoSOD methods with four standard metrics across three datasets, comparing representative traditional and deep-learning approaches under available-code or author-result protocols.

  • Four metrics evaluate CoSOD performance: maximum F-measure Fβ, mean absolute error ϵ, S-measure Sα, and maximum E-measure Eφ.
  • Fβ measures the weighted harmonic mean of precision and recall after thresholding saliency maps, whereas MAE measures direct absolute prediction error without binarization.
  • Sα evaluates continuous structural similarity, while Eφ jointly evaluates local and global similarity between predictions and ground truth.
  • The benchmark compares sixteen SOTA CoSOD models, including seven traditional and nine deep-learning methods selected for representativeness and available code or results.
  • Evaluation covers iCoSeg, CoSal2015, and CoSOD3k, using available implementations directly or CoSOD maps supplied by authors.

5.2 Quantitative Comparisons

Quantitative comparisons show dataset-dependent behavior: SOD models are highly competitive on iCoSeg and CoSal2015, while CoEG-Net achieves the best reported performance on the latter and CoSOD3k. The CoSOD3k results also expose substantial difficulty across several super-classes.

  • iCoSeg: On iCoSeg, CoEG-Net achieves the best Eφ, Sα, and Fβ performance, although its results are very close to EGNet.
  • iCoSeg: iCoSeg contains many single-object images, making it easy for SOD models and limiting the importance of co-salient features for evaluation.
  • CoSal2015: On CoSal2015, CoEG-Net obtains the best results and significantly outperforms both SOD and CoSOD baselines.
  • CoSal2015: CSMG reaches Eφ 0.842 and Fβ 0.784 on CoSal2015 but records Sα 0.774 and ϵ 0.130.
  • CoSOD3k: CoEG-Net achieves the best overall performance on CoSOD3k, while lower average Eφ scores occur for Other, Instrument, Necessary, Tool, and Ball super-classes.
  • CoSOD3k: CoSOD3k is challenging and leaves substantial room for further research.

5.3 Qualitative Comparisons

Qualitative comparisons reveal a trade-off between single-image saliency and cross-image commonality. CoEG-Net combines the sharp boundaries of SOD models with the context-aware object selection of CoSOD methods.

  • SOD models detect salient objects with sharp boundaries but ignore cross-image context.
  • In the CoSal2015 banana group, SOD models incorrectly retain irrelevant oranges, pineapples, and apples.
  • In the CoSOD3k dog group, SOD models also detect humans together with the dogs.
  • CoSOD methods remove irrelevant objects such as humans but produce less accurate maps around object boundaries.
  • CoEG-Net preserves the advantages of SOD and CoSOD methods and obtains the best visual results.

5.4 Comparison with Baselines

The baseline study applies the same co-attention projection to three SOD models, using common training and test datasets to compare their CoSOD performance.

  • CoEG-Net consists of a co-attention projection combined with a basic SOD model.
  • The study is designed to explore the efficiency of the co-attention projection across different SOD backbones.
  • Table 6 compares the three baselines using Eφ, Sα, Fβ, and ϵ metrics.Ours-A, Ours-P, and Ours-E denote the co-salient results of Amulet, PiCANet, and EGNet, respectively.
  • The experiment applies one co-attention projection strategy to Amulet, PiCANet, and EGNet under the same DUTS training and CoSOD test datasets.The evaluated datasets are iCoSeg, CoSal2015, and CoSOD3k.

5.5 Running Time

CoEG-Net is evaluated for inference efficiency against traditional and deep CoSOD models on different hardware configurations.

  • CoEG-Net achieves the fastest inference time among the top-three CoSOD models on CoSOD3k when evaluated by Eφ measure.The comparison includes CoEG-Net, CSMG, and UMLF.
  • CoEG-Net obtains a significantly improved Sα measure over CBCS and ESMG despite having a longer test time.CBCS and ESMG are identified as the two fastest comparison models.
  • The authors report that CoEG-Net remains slower than the recently released CPD and EGNet models.They identify running time as an area with substantial room for improvement.
  • The experiments use an RTX 2080Ti for CoEG-Net and specified CPU or GPU configurations for the comparison models.

6 DISCUSSION AND FUTURE DIRECTIONS

The discussion identifies scalability, stability, compatibility, and metrics as unresolved CoSOD issues, while noting applications and weaknesses of CoEG-Net.

  • Scalability: Scalability is limited when methods process numerous related images, causing high computational costs and long runtimes in practice.The paper cites CSHS-102s and UMLF-87s as examples.
  • Stability: Image-pair, subgroup, and RNN-based ordering strategies can make training and test results unstable because no principled grouping or ordering exists.
  • Compatibility: CoEG-Net still follows a two-stage SOD-CoSOD framework, motivating deeper end-to-end integration of CNN-based SOD with CoSOD.The two-stage design spends more inference time than a single SOD model.
  • Metrics: CoSOD metrics based on averaged SOD scores do not fully capture relationships between co-salient objects across images.The discussion notes that current metrics assume target objects have similar sizes across images.
  • Weakness: CoEG-Net produces coarse boundaries and does not finely preserve detailed shape information for co-salient objects.Failed detection cases are shown in Fig. 13.
  • Potential Applications: CoSOD outputs can provide class-agnostic attention cues for object co-localization and support automatic thumbnailing through saliency-based cropping.The paper presents object co-localization and collection-aware crops as potential applications.

7 CONCLUSION

The paper introduces a semantically challenging CoSOD dataset, a unified CoEG-Net framework, and a broad benchmark study of existing methods.

  • CoSOD3k addresses appearance-based dataset bias by grouping objects that are similar semantically or conceptually.It contains 160 groups and 3,316 images with category, bounding-box, object-level, and instance-level annotations.
  • CoEG-Net integrates existing SOD techniques into a unified, trainable framework using co-attention projection for efficient common-information learning.The framework is described as improving scalability and stability.
  • The study summarizes 40 algorithms and benchmarks 18 methods on two classical datasets and CoSOD3k.
  • The evaluation reports that SOD methods are surprisingly better than CoSOD methods in the studied comparisons.The authors present this finding as guidance for further CoSOD investigation.
Loading 2007.03380v4…