Source-linked AI summary
Enhanced-alignment Measure for Binary Foreground Map Evaluation
Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, Ali Borji
TL;DR
Binary foreground-map measures often treat pixel-level and image-level information independently, limiting their evaluation of segmentation quality. The paper proposes E-measure, which combines local pixel values with the image-level mean in one term. Across four datasets and five meta-measures, it substantially outperforms existing measures, including application-ranking gains of 9.08% to 19.65%.
Problem
Existing binary foreground-map measures address pixel-wise or structural errors independently, despite human vision's sensitivity to global information and local details.
Method
E-measure combines image-level statistics and local pixel matching information in a compact term using the foreground map's global mean.
Results
Across four datasets and five meta-measures, E-measure is significantly better than existing measures, improving application ranking by 9.08% to 19.65% over the best prior measure.
Takeaways & Limitations
E-measure provides an effective and efficient binary foreground-map evaluation that jointly considers pixel-level and image-level properties.
Takeaways & Limitations
S-measure is slightly better on PASCAL, whose images contain more structural objects than those in the other three datasets.
Abstract
from arXiv · showhide
The existing binary foreground map (FM) measures to address various types of errors in either pixel-wise or structural ways. These measures consider pixel-level match or image-level information independently, while cognitive vision studies have shown that human vision is highly sensitive to both global information and local details in scenes. In this paper, we take a detailed look at current binary FM evaluation measures and propose a novel and effective E-measure (Enhanced-alignment measure). Our measure combines local pixel values with the image-level mean value in one term, jointly capturing image-level statistics and local pixel matching information. We demonstrate the superiority of our measure over the available measures on 4 popular datasets via 5 meta-measures, including ranking models for applications, demoting generic, random Gaussian noise maps, ground-truth switch, as well as human judgments. We find large improvements in almost all the meta-measures. For instance, in terms of application ranking, we observe improvementrangingfrom9.08% to 19.65% compared with other popular measures.
1 Introduction
Existing binary foreground-map measures can prefer random noise over a segmentation map that is visibly closer to ground truth. The paper proposes E-measure to jointly evaluate pixel-level matching and image-level information, with experiments showing stronger evaluation performance.
- Motivation: Common measures can favor a random Gaussian noise map over a state-of-the-art foreground segmentation map that is closer to ground truth.This issue motivates the proposed evaluation measure.
- Motivation: Binary foreground-map evaluation is used across vision tasks to compare estimated maps with human-labeled ground truth and assess model performance.
- Motivation: Many existing measures assess pixel-wise similarity while discarding structural similarity or image-level information.
- E-measure: E-measure combines image-level statistics and local pixel matching information in a compact term for binary foreground-map evaluation.The method is designed to address the independent treatment of these error types.
- Evaluation: Only E-measure correctly ranks foreground maps from three salient-object-detection models when popular measures including CM, Fbw, and VQ fail.The ranking considers structural information and global shape coverage.
- Evaluation: The evaluation uses five meta-measures on four public datasets and includes a 555-map human-ranked dataset for testing consistency with human judgments.
2 Related Work
This section reviews binary foreground-map measures, emphasizing their pixel-wise foundations and limitations in capturing image-level structure, noise, and shape differences.
- Common measures: Popular binary foreground-map metrics include Fβ, Jaccard Index, and intersection over union.F1 is obtained by setting β = 1 in Fβ, while JI and IOU are equivalent.
- Common measures: Fβ combines recall and precision, with β controlling their trade-off.Recall, precision, and the quantities TP, TN, FP, and FN define the measure.
- Extensions: Weighted Fβ assigns different error weights according to pixel location.This extends the Fβ family beyond treating all pixel positions identically.
- Limitations: Pixel-independent measures can miss image-level information, producing suboptimal judgments for noise, structural errors, and shape differences.Region-based measures may score maps similarly despite spikes, wiggles, or shape discrepancies.
- Limitations: S-measure models region-level structure and object-level properties, but those properties are not well defined for binary maps.It was developed primarily for non-binary foreground-map evaluation.
3 The Proposed Measure
The E-measure is designed for binary foreground-map evaluation by jointly modeling local pixel matching and global image statistics. Its alignment framework uses mean-centered maps and an enhanced mapping to rank foreground estimates more effectively than prior measures.
- The E-measure addresses the limitation that existing measures consider pixel-level and image-level errors independently.The paper motivates simultaneous treatment of local information and global information for binary-map similarity.
- The measure combines local pixel values with each map’s global mean in a compact term.This design jointly captures image-level statistics and local pixel matching information.
- The bias matrix mean-centers each input map before alignment, helping remove errors from intrinsic variations or large numerical differences.For each input I, the bias matrix is defined from pixel values and the global mean; it is computed for both GT and FM.
- The alignment matrix uses the Hadamard product of the ground-truth and foreground bias matrices, with signs indicating aligned or unaligned pixel positions.Its element values also depend on the maps’ global means, incorporating global statistics into local alignment.
- The quadratic mapping function was selected as a simple, effective convex transformation after testing higher-order polynomials and trigonometric alternatives.It is used to suppress decreases in negative alignment regions.
- Using the enhanced alignment matrix, the final E-measure correctly ranks the foreground map above random noise in the Fig. 1 example.The measure aggregates pixel-level matching and image-level statistics to produce this ranking.
4 Experiments
The experiments compare E-measure with existing binary foreground-map measures across four public datasets using application ranking, generic and noise-map discrimination, human ranking, and ground-truth switching. E-measure improves application-ranking performance substantially and performs strongly on the other evaluation criteria, while showing a small ground-truth-switch disadvantage.
- Experimental Setup: Experiments compare E-measure with five popular binary foreground-map measures on four public salient-object-detection datasets using five meta-measures.The evaluation uses four prior meta-measures plus a newly introduced one.
- Meta-Measure 1: Application Ranking: E-measure correctly evaluates foreground maps for application ranking by combining images with their ground-truth or estimated foreground maps before retrieval comparison.The retrieval implementation uses LIRE with CEDD features and compares ordered outputs produced from ground-truth and estimated-map combinations.
- Meta-Measure 1: Application Ranking: 19.65%, 9.08%, 18.42% and 9.64% are E-measure’s improvements over the best prior measure on PASCAL, ECSSD, SOD and HKU, respectively.Application-ranking consistency is measured with θ, where lower values indicate closer ranking agreement.
- Meta-Measure 3: SOTA vs. Random Noise: E-measure achieves the best performance for distinguishing SOTA foreground maps from Gaussian random-noise maps, jointly using local pixel matching and global statistics.Only E-measure and S-measure attain the lowest mis-ranking rate in this experiment.
- Meta-Measures 4–5: E-measure outperforms other measures in correlation with human rankings of 555 foreground maps, while all measures perform well under the ground-truth-switch test.The human-ranking dataset was created from maps sampled across four datasets and ranked by subjects.
- Meta-Measure 5: Ground-truth Switch: 0.0523% is E-measure’s average ground-truth-switch mis-ranking rate, compared with 0.000925% for VQ and 0.001675% for CM.The reported comparison gives E-measure a 0.05% gap relative to other measures.
5 Conclusion and Future Work
The paper concludes that E-measure jointly evaluates pixel-level and image-level errors and performs effectively across five meta-measures on four datasets. It also introduces a dataset for comparing metric rankings with human judgments, while noting that S-measure is slightly better on structurally rich PASCAL images.
- Conclusion: E-measure simultaneously considers pixel-level and image-level errors, addressing a shortcoming of measures that evaluate these error types independently.The authors characterize the proposed measure as simple, effective and efficient.
- Conclusion: Five meta-measures on four popular datasets demonstrate E-measure’s effectiveness compared with existing binary foreground-map measures.The paper evaluates application ranking, generic and noise-map discrimination, human ranking, and ground-truth switching.
- Conclusion: The newly created dataset contains 740 maps, including 185 ground-truth maps and 555 human-ranked maps, for examining agreement between evaluation measures and human judgments.The human-ranked portion supports correlation testing between metric rankings and human preferences.
- Limitation: S-measure is slightly better than E-measure on PASCAL because that dataset contains more structurally rich objects than ECSSD, SOD and HKU-IS.The authors identify one failure case in Fig. 10.
- Future Work: Future work includes developing segmentation models and loss functions based on E-measure, with code and dataset planned for public release.The proposed metric uses simple derivable functions, supporting development of an E-measure-based loss function.