PaperlayerView exact revision on arXiv

Source-linked AI summary

What do different evaluation metrics tell us about saliency models?

Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, Frédo Durand

arXiv:1604.03605v2cs.CV

TL;DR

Choosing metrics for predicting human gaze is difficult because rankings depend on saliency definitions, ground-truth representations, and metric treatment of prediction errors and biases. The paper analyzes eight metrics through systematic experiments and visualizations, then derives recommendations for benchmark and application-specific metric selection. It concludes that metric choice should follow explicit assumptions about probabilistic modeling, viewing behavior, and task requirements.

  • Problem

    The paper addresses the open question of how to evaluate saliency models’ ability to predict human gaze when metrics rank models differently under different assumptions.

  • Method

    The paper quantifies the behavior of eight evaluation metrics through systematic experiments, visualizations, and analyses of varied saliency-map properties.

  • Results

    NSS and CC provide the fairest comparison under MIT Saliency Benchmark assumptions, whereas KL-divergence and IG are recommended when saliency models are evaluated probabilistically.

  • Takeaways & Limitations

    Metric selection should reflect ground-truth representation, viewing-bias assumptions, probabilistic modeling, spatial deviations, and the target application.

  • Takeaways & Limitations

    SIM evaluation is strongly affected by the Gaussian blur parameter, reaching its maximum only when prediction and ground-truth sigmas match.

Abstract

from arXiv · show

How best to evaluate a saliency model's ability to predict where humans look in images is an open research question. The choice of evaluation metric depends on how saliency is defined and how the ground truth is represented. Metrics differ in how they rank saliency models, and this results from how false positives and false negatives are treated, whether viewing biases are accounted for, whether spatial deviations are factored in, and how the saliency maps are pre-processed. In this paper, we provide an analysis of 8 different evaluation metrics and their properties. With the help of systematic experiments and visualizations of metric computations, we add interpretability to saliency scores and more transparency to the evaluation of saliency models. Building off the differences in metric properties and behaviors, we make recommendations for metric selections under specific assumptions and for specific applications.

1 INTRODUCTION

Determining which saliency model best predicts human fixations remains difficult because different metrics rank models inconsistently. The paper analyzes metric behavior and provides benchmark and metric-selection guidance.

  • Predicting human fixation locations is useful for applications including image retargeting, compression, detection, retrieval, and matching.
  • Different evaluation metrics can rank the same saliency models inconsistently, making performance comparisons dependent on metric interpretability.
  • Metric scores differ because metrics treat false positives, false negatives, viewing biases, spatial deviations, and saliency representations differently.
  • The paper analyzes 8 commonly used metrics and studies how input properties affect their scores and evaluation consequences.
  • Systematic experiments and visualizations make metric scores more interpretable and saliency-model evaluation more transparent.
  • The paper offers benchmark-design guidelines and metric-selection advice based on modeling assumptions and target applications.

2 RELATED WORK

Prior work compared saliency models and metrics in varied ways, but often provided limited empirical insight into metric behavior. This paper instead uses systematic experiments and visualizations to explain how metrics differ.

  • RELATED WORK: Earlier studies compared saliency models across datasets and metrics, but typically examined fewer metrics or narrower evaluation criteria.
  • RELATED WORK: This paper analyzes metrics commonly used in prior evaluation efforts, including Information Gain, using standard saliency models for visualization.
  • RELATED WORK: Rather than mainly reporting performance tables or reviewing metrics, the paper studies metric behavior under varied conditions with synthetic and natural data.
  • Evaluation metrics for saliency: Figure 1 illustrates that eight metrics can select different top-three saliency maps from eight models for one image.
  • RELATED WORK: The experiments examine effects of false positives, false negatives, blur, dataset biases, and spatial deviations on metric performance.
  • Qualitative evaluation of saliency: Metric-computation visualizations supplement saliency-map comparisons because visual contrast can strongly affect perceived model quality.

3 EVALUATION SETUP

Evaluation depends on the images, eye-movement collection procedure, and representation of ground truth fixations. This setup uses MIT300 free-viewing data and considers spatial fixation prediction, while smoothing introduces important parameter choices.

  • Evaluation setup: The evaluation setup requires decisions about input images, eye-movement collection, and representation as points, sequences, or distributions.
  • Evaluation setup: The study uses 300 MIT300 natural images viewed freely for 2 seconds, typically producing 4–6 fixations per observer.
  • Evaluation setup: Eye-tracking distance, calibration error, and image size constrain assumptions about the collected fixation data; MIT300 maps one visual-angle degree to approximately 35 pixels.
  • Ground truth representation: Ground-truth fixations can remain discrete locations or be smoothed into continuous fixation maps, with each representation relying on different assumptions.
  • Ground truth representation: Smoothing incorporates measurement and observer uncertainty, improves robustness, and can extrapolate data when few observers are available.
  • Ground truth representation: Smoothing requires parameter selection and post-processing, and its value can significantly affect evaluation scores.
  • Ground truth representation: The analysis considers spatial fixation data rather than temporal fixation sequences because sequences are noisier and harder to evaluate.

4 METRIC COMPUTATION

The paper analyzes eight saliency metrics by varying map representations and examining how each score responds to prediction errors, biases, spatial differences, and preprocessing. These behaviors explain inconsistent model rankings and motivate metric choices matched to evaluation assumptions and applications.

  • Metric computation: Eight popular metrics are analyzed in isolation, distinguishing location-based metrics from distribution-based metrics according to ground-truth representation.Location-based metrics evaluate saliency values at discrete fixation locations, whereas distribution-based metrics compare continuous fixation and saliency maps.
  • Metric computation: The recommended metric depends on the model and benchmark assumptions: KL-divergence or IG suit probabilistic models, while NSS or CC suit intensity maps expected to capture viewing biases.The recommendations are framed for probabilistic saliency models and benchmarks such as the MIT Saliency Benchmark.
  • Metric computation: AUC treats saliency maps as fixation classifiers and is driven primarily by whether high-valued predictions capture fixations in the first few level sets.Low-valued predictions at non-fixated locations have comparatively little effect on the score.
  • Metric computation: NSS penalizes false positives because they lower normalized saliency at every fixation, unlike AUC, which can remain high despite many low-valued false positives.NSS uses absolute saliency values in its normalization and is sensitive to relative saliency differences and monotonic transformations.
  • Metric computation: Information Gain compares fixation-location saliency against a center-prior or another model to quantify image-specific information beyond the baseline.The metric can visualize regions where one model underestimates or exceeds saliency relative to the comparison model.
  • Metric computation: SIM favors partial matches and penalizes false negatives more than false positives, whereas CC treats false positives and negatives symmetrically.SIM is highly sensitive to the blur or Gaussian sigma used to construct the maps, while CC is invariant to linear transformations but cannot identify which error type caused a difference.
  • Metric computation: KL-divergence is asymmetric and can penalize sparse predictions severely when ground-truth density is nonzero where predicted saliency is near zero.Lower KL scores indicate better approximation of the ground truth.
  • Metric computation: EMD penalizes false positives according to their spatial distance from the ground-truth distribution, with larger values indicating greater distributional difference.Spreading density over a larger area generally increases the amount of mass that must be moved to match the ground truth.

5 ANALYSIS OF METRIC BEHAVIOR

Metric rankings diverge because evaluation measures reward and penalize different saliency properties, including false positives and negatives, spatial distance, regularization, and viewing bias. Systematic baseline and perturbation analyses reveal these behaviors and explain when metrics agree or disagree.

  • Scoring baseline models: The center prior and single-observer models receive divergent scores because they capture different viewing properties, while sparsity hurts the single-observer model under KL, IG, and SIM.The center prior captures dataset-specific, image-independent behavior; the single-observer model captures image-specific properties but may miss average viewing behavior.
  • Treatment of false positives and negatives: KL, IG, and SIM heavily penalize false negatives, with KL and IG falling below chance after only 25% false negatives.These penalties can grow arbitrarily large when predictions approach zero at ground-truth fixation locations, making regularization important.
  • Treatment of false positives and negatives: AUC largely ignores low-valued false positives and gives similar scores to maps with different density amounts at correct fixation locations.False positives in the earliest level sets are penalized most, whereas those in the last level set have little effect.
  • Treatment of false positives and negatives: NSS and CC are affected by false positives and negatives, although NSS becomes more sensitive to false positives as they increasingly alter normalization.CC penalizes the two error types equally, while a few NSS false positives can be washed out by other saliency values.
  • Treatment of false positives and negatives: EMD penalizes errors according to spatial distance and can favor predictions between multiple fixation locations rather than predictions covering only some locations.Nearby density can compensate for uniformly distributed false negatives, whereas distant false negatives are highly penalized.
  • Systematic viewing biases: sAUC penalizes explicit center bias, whereas IG measures performance relative to a center-prior baseline and gives no gain when a model explains only systematic viewing bias.Center bias is part of natural viewing behavior in some settings, so metric choice depends on whether that behavior should be included or discounted.
  • Relationship between metrics: NSS, CC, AUC, EMD, and SIM form a similarity cluster with pairwise ranking correlations from 0.76 to 0.98, while KL, IG, and sAUC differ more.EMD still ranks models similarly to the cluster despite being the only metric incorporating spatial distance, likely because it shares center bias and current errors are often grossly incorrect.

6 RECOMMENDATIONS FOR DESIGNING A SALIENCY BENCHMARK

Benchmark design should make saliency assumptions explicit, especially how dataset biases, map formats, and application goals are handled. Metric selection should then match those assumptions and the intended use.

  • Benchmark assumptions: Define whether submitted saliency maps are probabilistic, because KL, IG, and SIM penalize zero-valued predictions.Non-probabilistic models may require regularization and preprocessing before evaluation with these metrics.
  • Benchmark assumptions: Explicitly state whether models must incorporate systematic dataset biases such as center bias, blur, and scale.Benchmarks should clarify whether these biases are modeled by submissions or accounted for during evaluation.
  • Handling dataset bias: Account for center bias consistently at the dataset, model, or evaluation level rather than mixing incompatible treatments.sAUC penalizes strong center bias, while some models and benchmarks explicitly incorporate it.
  • Application-specific metrics: Choose metrics according to the application: AUC, KL, and IG suit target detection, whereas NSS or SIM suit relative regional-importance tasks.Detection metrics penalize missed targets; compression, retargeting, and transmission require meaningful saliency comparisons between regions.
  • Metric trade-offs: Avoid relying on AUC alone when absolute saliency differences matter, because AUC is insensitive to monotonic transformations.EMD can capture spatial differences but is computationally expensive and difficult to optimize.

7 CONCLUSION

The paper shows that metric rankings depend on how saliency, ground truth, bias, probability, and spatial deviation are represented. It recommends metrics that align with benchmark assumptions and application requirements.

  • Conclusion: Eight metrics respond differently to ground-truth representation, dataset bias, probabilistic inputs, and spatial deviations.These input properties help explain why saliency models receive different rankings across metrics.
  • Conclusion: Under MIT Saliency Benchmark assumptions, NSS and Pearson’s CC provide the fairest comparison and produce highly correlated rankings.Because the metrics are mathematically closely related, reporting one is sufficient.
  • Conclusion: For probabilistic saliency models, KL-divergence and Information Gain are recommended.Different metric choices may be more appropriate under alternative definitions of saliency.
  • Conclusion: Benchmark assumptions should be explicit to make evaluation more transparent and reduce confusion.The paper also provides code for evaluating and visualizing metric computations.

A.1 Evaluation setup: data collection

The MIT300 dataset used controlled eye-tracking data collected from Flickr and personal photographs, with standardized presentation and calibration procedures.

  • Data collection: MIT300 images came from Flickr Creative Commons and personal photo collections.Observers viewed each image for 2 seconds while gaze paths were recorded with a table-mounted video-based eye tracker.
  • Data collection: The eye tracker recorded observers’ gaze paths at 240Hz with average calibration error below one degree of visual angle.Images had a maximum dimension of 1024 pixels, with the other dimension ranging from 457 to 1024 pixels.

A.2 Metric computation

Metric implementations differ in input representation, thresholding, sampling, symmetry, and computational cost. These implementation choices can change how saliency predictions are compared with fixation data.

  • Metric representations: Location-based and distribution-based metrics require different ground-truth representations, and some implementations threshold continuous fixation distributions.The parameter-free implementations analyzed here use location-based inputs where applicable.
  • AUC computation: ROC-based AUC metrics estimate curves by varying saliency thresholds, making threshold sampling an important implementation choice.AUC-Judd uses each distinct saliency value, whereas AUC-Borji samples thresholds from 0 to 1 in increments of 0.1.
  • AUC computation: AUC-Borji samples negatives repeatedly from the current image, while sAUC samples negatives from ten other images.Both implementations average results over 100 trials.
  • AUC computation: AUC is non-symmetric, so swapping the reference and comparison maps can produce different scores.A symmetric variant averages the two calculations obtained by exchanging the maps.
  • Correlation metrics: Pearson’s CC uses absolute map values, whereas Spearman’s CC compares ranks and is robust to monotonic transformations.NSS can be viewed as a discrete approximation to CC because both normalize by variance.
  • KL and IG: The adopted KL implementation is asymmetric and measures how well a saliency prediction approximates the ground-truth distribution.Alternative KL variants differ in symmetry, computation, and whether they compare maps directly or use fixation-based histograms.
  • KL and IG: Information Gain is baseline-adjusted and measured in bits per fixation, while its computational form is closer to fixation-based KL than image-based KL.The paper’s visualization assigns per-pixel gain values and modulates them by the human fixation distribution.
  • EMD computation: EMD is more computationally expensive because it requires joint optimization across all image pixels.The implementation resizes both maps and normalizes them to sum to one before computing the distance.

A.3 Normalization of saliency maps

Normalization and histogram matching substantially shape metric behavior by changing saliency ranges, outliers, sparsity, and density. Converting maps to densities also introduces probabilistic assumptions that can make regularization decisive for KL and IG.

  • Histogram matching: Histogram matching leaves AUC calculations unchanged but affects other metrics by making maps more peaked or more uniform.EMD favors sparse predictions near targets, whereas other metrics benefit when predictions assign non-zero values at target locations.
  • Outliers and value ranges: Different normalization schemes alter how very high and very low saliency values affect NSS.A fixation-location outlier can produce a correspondingly high NSS score, while large positive values elsewhere can make fixation values arbitrarily negative after normalization.
  • Normalization strategies: Normalization strategies differ across metrics, and normalization can change how saliency ranges and outliers affect performance.The metrics in the paper apply different preprocessing strategies before scoring.
  • Normalization for conversion to a density: Metrics commonly normalize saliency maps by their sum before computing SIM, KL, and IG, but this does not make a non-probabilistic map probabilistic.In a probabilistic map, zero means fixations are impossible in that region.
  • Normalization for conversion to a density: Adding a small epsilon to predictions can drastically improve KL or IG scores, making regularization an important factor for probabilistic maps.Compressed storage may fail to preserve small regularization values, so file format can facilitate or hinder evaluation.

Empirical limits of metrics

Metric ceilings depend on both theoretical properties and dataset-specific human consistency. Distribution-based metrics are more robust to observer variation, while empirical limits provide realistic upper bounds for comparing models across metrics and datasets.

  • Empirical limits: Location-based metrics such as AUC, sAUC, NSS, and IG do not reach their theoretical limits on a given dataset.Their fixated and non-fixated location sets overlap, so no classifier can reach the theoretical maximum.
  • Empirical limits: Distribution metrics are more robust because continuous fixation maps converge toward the same underlying distribution as observer counts increase.Different observer groups fixate similar but not identical locations, whereas their continuous maps can converge.
  • Empirical limits: Empirical limits are dataset-specific, depend on consistency between humans, and represent realistic upper bounds for model performance.They can make scores comparable across metrics and datasets.
  • Estimating human consistency: Under AUC-Judd, extrapolated human performance reaches a limit of 0.92.The estimate comes from predicting one group of n observers with another group of n observers as n increases.
  • Estimating human consistency: Human consistency is estimated by using one group of n observers to predict another group, measuring n = 1 to n = 19, and extrapolating a fitted power function.The fit constrains the exponent to be negative and the asymptote to remain within the metric’s theoretical range.
  • Model choices: Chance-map sparsity can lower KL, IG, and SIM scores, while single-observer predictions may improve under different blur or regularization choices.The effect follows from sparse predictions and the sensitivity of these distribution-based metrics to zero or low-valued regions.

A.5 Recommendations for designing a saliency benchmark

Benchmark design should make saliency representations and preprocessing choices explicit because histogram matching, probabilistic interpretation, and file format can change reported performance. The paper therefore favors transparent evaluation and representations that preserve the information metrics use.

  • Preprocessing transparency: The MIT Saliency Benchmark stopped histogram matching in September 2014 because it could inflate or deflate scores depending on the model.Evaluating maps as-is makes website scores correspond directly to submitted maps and leaves preprocessing to model submitters.
  • Probabilistic representations: A probabilistic saliency definition makes map values interpretable as expected fixation probabilities or expected fixation counts.Relative values also become meaningful, such as a region having twice the saliency of another.
  • Submission formats: Benchmark submission formats affect performance numbers, so non-compressed formats are preferable to JPG-encoded maps.JPG stores 8 bits per pixel and artifacts can strongly affect regions with low saliency values.
  • Submission formats: Log probability maps can represent a larger range of values than ordinary probability maps.This recommendation addresses the representational limits of submitted saliency maps.
Loading 1604.03605v2…