Source-linked AI summary
Structure-measure: A New Way to Evaluate Foreground Maps
Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, Ali Borji
TL;DR
Existing foreground-map measures can overlook structural similarity, limiting their ability to reflect the completeness of salient objects. The paper introduces Structure-measure, which combines region-aware and object-aware structural similarity, and reports better performance than AP, AUC, and Fbw across five datasets plus agreement with human preferences.
Problem
Pixel-wise measures such as AP, AUC, and Fbw can fail to capture foreground structure, which matters for selecting models that preserve complete objects.
Method
Structure-measure evaluates similarity between saliency and GT maps using both region-aware and object-aware structural similarity.
Results
Structure-measure performed better than AP, AUC, and Fbw on five datasets, and 45 subjects preferred maps selected by it on average.
Takeaways & Limitations
Structure-measure offers a structural alternative for evaluating salient object detection maps and model comparisons.
Takeaways & Limitations
AP, AUC, and Fbw remain limited because their pixel-wise quantities cannot fully capture predicted-map structure.
Abstract
from arXiv · showhide
Foreground map evaluation is crucial for gauging the progress of object segmentation algorithms, in particular in the filed of salient object detection where the purpose is to accurately detect and segment the most salient object in a scene. Several widely-used measures such as Area Under the Curve (AUC), Average Precision (AP) and the recently proposed Fbw have been utilized to evaluate the similarity between a non-binary saliency map (SM) and a ground-truth (GT) map. These measures are based on pixel-wise errors and often ignore the structural similarities. Behavioral vision studies, however, have shown that the human visual system is highly sensitive to structures in scenes. Here, we propose a novel, efficient, and easy to calculate measure known an structural similarity measure (Structure-measure) to evaluate non-binary foreground maps. Our new measure simultaneously evaluates region-aware and object-aware structural similarity between a SM and a GT map. We demonstrate superiority of our measure over existing ones using 5 meta-measures on 5 benchmark datasets.
1. Introduction
Foreground-map evaluation is important for comparing salient object detection models, but existing measures can miss object structure. Structure-measure addresses this by evaluating region-aware and object-aware similarity, with experiments across five meta-measures and five datasets.
- Non-binary foreground maps are the focus because most salient object detection methods output them.
- Existing measures can rank a structurally accurate dog map below maps with fuzzy or severely degraded structure.
- 60% of ten SalCut-generated binary maps had Structure-measure below 0.95 in the illustrated experiment.
- Structure-measure targets low sensitivity to global object structure by combining region and object perspectives.
- The measure is presented as more effective than other measures using five meta-measures on five public benchmark datasets.
2. Current Evaluation Measures
Current foreground-map evaluation separates binary and non-binary settings. For non-binary maps, AUC and AP threshold predictions into binary maps, while Fbw extends pixel-wise error handling but can still misrank maps.
- Binary-map evaluation commonly uses Fβ-measure and PASCAL VOC segmentation, whereas non-binary evaluation commonly uses AUC, AP, and Fbw.
- AUC: AUC applies multiple thresholds, compares resulting binary maps with GT, plots TPR against FPR, and summarizes the curve area.
- AP: AP averages precision p(r) over evenly spaced recall points from r = 0 to r = 1.
- Fbw: Fbw extends TP, TN, FP, and FN to non-binary values and weights errors using location and neighborhood information.
- Fbw: Despite these extensions, Fbw can still fail to rank foreground maps correctly.
3. Limitations of Current Measures
AP, AUC, and Fbw are based on pixel-wise quantities that cannot fully represent foreground structure. Consequently, they can give identical or incorrect rankings when maps differ in structural completeness, whereas Structure-measure follows application-oriented judgments in the shown examples.
- Pixel-wise TP, TN, FP, and FN calculations cannot fully capture structural information in predicted maps.
- Two maps with different structural errors can receive identical FN, TN, FP, and TP scores.
- Fbw ranks pyramid maps differently from application and user-study judgments because it does not account for structural similarity.
- If an evaluation measure misses object structure, it cannot provide reliable information for application-level model selection.
4. Our measure
Structure-measure evaluates foreground maps through complementary region-aware and object-aware structural similarities. It combines object-part structure with global foreground and background structure to assess similarity between saliency and ground-truth maps.
- Structure-measure combines region-aware and object-aware structural similarities to evaluate foreground-map structure.The region-aware component captures object-part structure, while the object-aware component captures complete foreground-object structure.
- Region-aware structural similarity measure: Region-aware similarity recursively partitions saliency and ground-truth maps into blocks centered on the ground-truth foreground centroid, then computes SSIM for each block.The resulting block similarities are combined to assess object-part structural similarity.
- Region-aware structural similarity measure: Direct image-level or sliding-window SSIM fails to capture region-aware structural similarities, whereas the proposed Sr describes object-part similarity.
- Object-aware structural similarity measure: Object-aware similarity separately compares foreground and background distributions to capture global object-level similarity.It addresses uniform saliency distributions and sharp foreground-background contrast in ground-truth maps.
- Object-aware structural similarity measure: Foreground comparison measures similarity between mean saliency probabilities in the predicted and ground-truth foreground regions.Its value is closer to 1 when the two foreground means are closer; the formulation is symmetric and ranges from 0 to 1.
- Our new structure-measure: The final Structure-measure combines object-aware and region-aware similarities using S = α ∗So + (1 −α) ∗Sr, with α ∈[0, 1].The implementation sets α = 0.5 and correctly ranks the three saliency maps in Fig. 1 according to the application rank.
5. Experiments
The experiments evaluate Structure-measure with five meta-measures, testing application-consistent ranking, preference for state-of-the-art maps, ground-truth switching, annotation changes, and human judgments. Across benchmark datasets and comparisons, Structure-measure generally outperforms existing measures, while the annotation-error meta-measure is considered unreliable.
- Meta-Measure 1: Application Ranking: Structure-measure produced the best ranking consistency under the application-based meta-measure.Ranking accuracy was evaluated with 1-Spearman’s ρ, where lower values indicate better consistency, using SalCut as the application.
- Meta-Measure 2: State-of-the-Art Preference: Over 1000 images, Structure-measure made only 11 errors when generic maps outranked the mean score of five state-of-the-art models.AP and AUC made significantly more mistakes under this meta-measure.
- Meta-Measure 3: Ground-truth Switch: Structure-measure performed about 10 times better than the second-best measure when testing whether maps preferred the correct ground truth.The authors attribute this to capturing object structural similarity, which lowers scores when a random ground truth changes the object structure.
- Meta-Measure 4: Annotation Errors: For slight annotation changes that preserve topology, Structure-measure and Fbw kept the original saliency-map ranking.For major structural changes, Structure-measure correctly changed the ranking when the morphologically altered ground truth made another map more similar, whereas Fbw did not.
- Meta-Measure 4: Annotation Errors: The annotation-error meta-measure was judged unreliable and therefore excluded from further comparisons across datasets.The authors distinguish minor annotation changes from changes that substantially alter ground-truth structure.
- Further Comparison: Across PASCAL-S, ECSSD, SOD, and HKU-IS, Structure-measure won most cases and reduced meta-measure 3 error rates by 67.62%, 44.05%, 17.81%, and 69.23%, respectively, versus the second-ranked measure.It ranked best on meta-measure 1 and generally outperformed existing measures on meta-measure 2, except on ECSSD, where it ranked second.
- Meta-Measure 5: Human Judgments: In human preference trials, viewers preferred maps selected by Structure-measure in 63.69% of trials, versus AP and AUC comparison rates of 72.11% and 73.56%.The authors interpret these results as better correlation with human judgments.
- Saliency Model Comparison: Using Structure-measure to compare 10 saliency models on four datasets, the top three were dhsnet, DCL, and rfcn in that order.The compared datasets were PASCAL-S, ECSSD, HKU-IS, and SOD.
6. Discussion and Conclusion
The paper presents Structure-measure as an efficient evaluation measure that captures structural similarities overlooked by pixel-wise measures. Across benchmark experiments and human judgments, it outperforms existing measures and offers new insights into salient object detection evaluation.
- Structure-measure simultaneously evaluates region-aware and object-aware structural similarities between saliency maps and ground-truth maps.It is based on sharp foreground-background contrast and uniform saliency distribution, while remaining efficient and easy to calculate.
- Experiments on 5 datasets show that Structure-measure performs better than AP, AUC, and Fbw.
- 45 subjects on average preferred saliency maps selected by Structure-measure over those selected by AP, AUC, and Fbw.The behavioral study used a database of 100 saliency maps and 50 ground-truth maps.
- Structure-measure offers new insights for evaluating salient object detection models because current measures can fail to examine their strengths and weaknesses.The authors encourage its use in future model evaluations and comparisons.