Source-linked AI summary
A Formal Evaluation of PSNR as Quality Measurement Parameter for Image Segmentation Algorithms
Fernando A. Fardo, Victor H. Conforto, Francisco C. de Oliveira, Paulo S. Rodrigues
TL;DR
Segmentation quality lacks an objective parameter, while PSNR has been used to evaluate threshold-based algorithms. The paper tests PSNR using ground-truth-derived masks and finds that bad masks receive significantly higher PSNR values than good masks, so PSNR is not adequate for segmentation evaluation.
Problem
Segmentation quality evaluation lacks an objective parameter and cannot be determined automatically, despite proposals for generic metrics.
Source: §1 · pp. 2–3Method
The paper compares PSNR values for ground-truth-derived segmentation masks and deliberately corrupted masks, using statistical tests to assess the difference.
Source: §4 · p. 4, §4.1 · p. 6, §5 · p. 7Results
4.735 · 10−14 was the Welch’s T test p value, and PSNR values for bad segmentation masks were significantly higher than those for human-obtained masks.
Source: §5.2 · p. 8Takeaways & Limitations
PSNR should not be considered an adequate method for evaluating segmentation algorithms, although it remains useful for image discrepancies and may evaluate edge detection against ground truth.
Source: §6 · p. 9The mask-generation setup may not represent threshold algorithms’ label assignments, and arbitrary or random-color labels can affect PSNR and impede verification.
Source: §4.1 · p. 6, §6 · p. 9
Abstract
from arXiv · showhide
Quality evaluation of image segmentation algorithms are still subject of debate and research. Currently, there is no generic metric that could be applied to any algorithm reliably. This article contains an evaluation for the PSRN (Peak Signal-To-Noise Ratio) as a metric which has been used to evaluate threshold level selection as well as the number of thresholds in the case of multi-level segmentation. The results obtained in this study suggest that the PSNR is not an adequate quality measurement for segmentation algorithms.
1. Introduction
Threshold segmentation groups pixels by brightness, but its quality is difficult to evaluate objectively. The introduction reviews evaluation approaches and notes that PSNR has been used to assess threshold-based methods, motivating a test of its reliability.
1. Introduction
- Threshold segmentation groups pixels above or below selected brightness levels; the example sets pixels below t to 0 and above t to 255.The example uses t = 118, with 0 and 255 representing background and foreground, respectively.Source: §1 · pp. 1–2
- Threshold algorithms seek levels that separate image regions of interest, but their quality lacks an objective parameter and cannot be determined automatically.Source: §1 · pp. 2–3
- Evaluation methods are analytic, using properties of segmented images, or empirical, comparing results with human-defined references or goodness criteria.Empirical methods include goodness measures and discrepancy measures based on ground-truth images.Source: §1 · p. 3
- Despite limitations, PSNR has been used to evaluate threshold-based algorithms, prompting experiments to test its reliability as a segmentation metric.Source: §1 · p. 3
2. PSNR
PSNR compares a processed signal with its original source, using mean squared error; despite its common use for transmission and compression, it has also been applied to segmentation and threshold selection.
2. PSNR
- PSNR compares a received or processed signal with its original source to quantify fidelity and possible noise or distortion.Source: §2 · p. 3
- PSNR is inversely proportional to mean squared error, and its final value is expressed in decibels.For an 8-bit grayscale image, the highest possible signal value MAX is 255.Source: §2 · p. 4
- PSNR is generally used to evaluate image or video transmission and compression, but has also been used to assess segmentation and select multi-threshold counts and values.The passage cites prior uses for segmentation, threshold counts, and threshold values [13].Source: §2 · p. 4
3. Objective
The paper evaluates whether PSNR is a reliable analytic method for assessing image segmentation algorithms.
3. Objective
- The paper's objective is to evaluate PSNR itself as a reliable analytic method for image segmentation algorithms.Source: §3 · p. 4
4. Methodology
The study evaluates PSNR itself by comparing scores for human-ground-truth-derived masks with scores for artificially degraded masks. It uses the difference between these score sets as the test of whether PSNR ranks segmentation quality appropriately.
4. Methodology
- PSNR was computed between each source image and both its ground-truth-derived binary mask and its salt-and-pepper-degraded counterpart.The masks were generated from the Berkeley BSR300 images and their corresponding ground truths.Source: §4 · pp. 4–6
4.1. Proof
- The main hypothesis was that average PSNR for good masks should significantly exceed average PSNR for bad masks if PSNR is inadequate for segmentation evaluation.The score sets were P for binary masks and P′ for degraded masks.Source: §4.1 · p. 6
5. Results and discussion
The tests found that bad segmentation masks had higher PSNR values than human-derived masks. Because the score variances were not homogeneous, the authors used Welch’s T test rather than relying on Student’s T test.
5. Results and discussion
- The analysis used a 95%-significance Student’s T test plan, checked variance homogeneity with Fisher’s F test, and switched to Welch’s T test when needed.Figures 7 and 8 show the probability densities for P and P′, respectively.Source: §5 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7
5.1. Fisher’s F test for variance
- The F test indicated that variances for P and P′ were not homogeneous, so Student’s T test could not be used reliably.The authors proceeded with Welch’s T test to assess the difference between the score sets.Source: §5.1 · pp. 7–8, Table 1 · p. 8
5.2. Welch’s T test
- 4.735 · 10−14 was the Welch’s T test p value; bad-mask PSNR values were significantly higher than those for human-derived masks.The result rejects equal means in favor of the alternative that P′ exceeds P.Source: Table 2 · p. 8, §5.2 · p. 8
6. Final considerations
The authors conclude that PSNR is inadequate for evaluating segmentation algorithms because artificially inferior masks received higher scores than human-derived masks. They retain PSNR as a measure of image discrepancies and suggest possible use for edge-detection evaluation.
6. Final considerations
- Bad masks had significantly higher PSNR values than good masks, leading the authors to conclude PSNR is inadequate for evaluating segmentation algorithms.The comparison used human-created masks as ideal references and artificially inferior masks for contrast.Source: §6 · p. 9
- Future work could test multi-threshold algorithms, threshold-count selection, and the impact of label values.Source: §6 · p. 9