PaperlayerView exact revision on arXiv

Source-linked AI summary

Consolidated Dataset and Metrics for High-Dynamic-Range Image Quality

Aliaksei Mikhailiuk, Maria Perez-Ortiz, Dingcheng Yue, Wilson Suen, Rafal K. Mantiuk

arXiv:2012.10758v2eess.IVcs.CV

TL;DR

Existing HDR quality datasets are small, while image-quality metrics need to reflect display brightness and dynamic range. The paper merges SDR and HDR datasets and realigns their scores through additional comparisons and psychometric scaling. UPIQ contains 4,159 images, and its scaling compares favorably with a prior rescaling method; the dataset is also used for metric training and brightness-adaptive coding.

  • Problem

    Existing HDR datasets are limited in size, and existing datasets differ in protocol and quality scale, complicating unified training and cross-dataset comparison.

    Source: §I · p. 1

  • Method

    UPIQ merges existing SDR and HDR datasets, collects additional cross-dataset comparisons, and uses psychometric scaling to align scores on a unified scale.

    Source: §I · p. 1, §IV · p. 5

  • Results

    Mean SROCC was 0.71 for the proposed scaling method versus 0.56 for the method from.

    Source: 1) Comparison to previous re-scaling… · p. 7

  • Takeaways & Limitations

    UPIQ supports training and benchmarking HDR metrics, including a deep-learning metric, and demonstrating brightness-adaptive image coding.

    Source: §VI · p. 12, D. Brightness-Adaptive Coding · p. 12

Abstract

from arXiv · show

Increasing popularity of high-dynamic-range (HDR) image and video content brings the need for metrics that could predict the severity of image impairments as seen on displays of different brightness levels and dynamic range. Such metrics should be trained and validated on a sufficiently large subjective image quality dataset to ensure robust performance. As the existing HDR quality datasets are limited in size, we created a Unified Photometric Image Quality dataset (UPIQ) with over 4,000 images by realigning and merging existing HDR and standard-dynamic-range (SDR) datasets. The realigned quality scores share the same unified quality scale across all datasets. Such realignment was achieved by collecting additional cross-dataset quality comparisons and re-scaling data with a psychometric scaling method. Images in the proposed dataset are represented in absolute photometric and colorimetric units, corresponding to light emitted from a display. We use the new dataset to retrain existing HDR metrics and show that the dataset is sufficiently large for training deep architectures. We show the utility of the dataset on brightness aware image compression.

I. INTRODUCTION

The paper addresses the need for image-quality assessment that reflects display brightness and dynamic range, and for a larger, diverse dataset with comparable scores. It reviews existing dataset limitations, assessment criteria, and approaches before introducing UPIQ and its applications.

II. RELATED WORK

  • Prior SDR metrics such as PSNR and SSIM are widely used, but most IQA metrics do not account for display brightness or dynamic range, which can affect perceived artifact severity.For example, compression artifacts may be more visible on a bright HDR display than on a dimmed mobile phone.Source: §I · p. 1

A. Existing IQA Datasets

  • Existing IQA datasets are limited in image count, distortion diversity, or image size, and differing protocols and relative scales prevent straightforward merging and cross-dataset score comparison.The authors argue for consolidating datasets rather than collecting a new one from scratch.Source: §I · p. 1, A. Existing IQA Datasets · p. 2

I. INTRODUCTION

  • UPIQ combines existing SDR and HDR datasets on a unified quality scale and supports retraining and benchmarking full-reference metrics, including deep architectures.The authors report 4,159 images and use the dataset in brightness-adaptive image coding.Source: §I · p. 1

A. Existing IQA Datasets

  • Earlier dataset-alignment work used objective metric predictions, whereas this paper collects subjective cross-dataset comparisons and applies psychometric scaling to unify the scores.The authors note that few metrics reliably predict absolute quality across both SDR and HDR images.Source: A. Existing IQA Datasets · p. 2

B. Quality Assessment Criteria

  • The paper focuses on impairment metrics for distorted images and full-reference assessment, where the original undistorted image is available.It distinguishes impairment assessment from aesthetics and visibility metrics, which respectively judge photographic qualities and detect visibility without measuring distortion magnitude.Source: B. Quality Assessment Criteria · p. 2

C. Existing Approaches to IQA

The paper describes limits of existing IQA approaches and motivates photometric assessment for variable displays. It adapts SDR metrics to HDR and SDR luminance using a display model where needed and a perceptually uniform transform.

C. Existing Approaches to IQA

  • Deep-learning IQA metrics learn features and quality mappings but are susceptible to training-data quality and quantity, while scarce human judgments can impair generalization.Other approaches include parameter-free statistics, visual-system models with few trainable parameters, and hand-crafted features combined with machine learning.Source: C. Existing Approaches to IQA · p. 2

D. Training Robust Deep Learning models

  • Transfer learning and pairwise-ranking training offer ways to improve generalization, but do not remove the need for a larger, diverse IQA dataset.Pairwise training can use comparison data directly, but reduces quality scores to binary labels and discards information.Source: D. Training Robust Deep Learning models · pp. 2–3

E. Influence of a Display on the Perceived Image Quality

  • Representing images in absolute colorimetric units makes quality assessment device-specific and requires knowledge of the target display.Peak luminance can range from 5 cd/m2 on a dimmed phone to 6,000 cd/m2 on a bright HDR display.Source: E. Influence of a Display on the… · p. 3

1) Display model:

  • A gain-gamma-offset display model converts gamma-encoded color values into absolute linear color values.The model uses γ = 2.2 and the display's peak and black luminance levels.Source: 1) Display model · p. 3

2) Photometric image quality metrics:

  • Photometric quality metrics operate on physical luminance values, allowing them to account for the display dynamic range that affects distortion visibility.HDR-VDP and HDR-VQM are cited as photometric metrics that account for a large range of HDR-display luminance.Source: 2) Photometric image quality metrics · p. 3

3) Extending SRD metrics to HDR images:

  • The adaptation converts luminance to perceptually uniform values before applying an SDR metric; SDR inputs first use a display model, while HDR inputs omit it.The PU-transform is designed so the smallest perceivable luminance change maps to a constant PU change, and its SDR range maps to 0–255 for comparable predictions.Source: 3) Extending SRD metrics to HDR images · p. 3

III. PSYCHOMETRIC SCALING

The psychometric scaling method combines measurements from different datasets and ranking and rating protocols into a common quality scale. Its observer model and likelihood-based scaling fundamentals specify how judgments are represented and aggregated.

III. PSYCHOMETRIC SCALING

  • The method extends a probabilistic model to combine measurements across datasets and ranking and rating protocols, aligning them to a unified scale.The authors describe the scaling method and experimental protocol as generalisable to other datasets using rating and ranking preference aggregation.Source: §III · pp. 3–4

A. Observer Model

  • The observer model treats each condition as having an underlying true quality and accounts for variation in observers' notions of quality and repeated judgments.Using Thurstone Case V, measured quality is modeled as normally distributed with a common standard deviation across conditions.Source: A. Observer Model · p. 4

B. Scaling Fundamentals

  • Scaling estimates quality scores from pairwise-choice probabilities, with maximum likelihood combining the observed comparisons across condition pairs.The model relates quality differences to a cumulative normal distribution; a common scale sets a 0.75 choice probability to one unit and 0.91 to two units.Source: B. Scaling Fundamentals · p. 4

C. Mixing Pairwise Comparisons and Rating

The model treats ratings and pairwise comparisons as different measurements of the same latent quality, with MOS related monotonically to psychometric scores. It estimates latent scores and protocol-specific parameters jointly, fixing reference-image scores to zero and expressing quality differences in JOD units.

C. Mixing Pairwise Comparisons and Rating

  • Rating MOS values are mapped to psychometric scores with a linear relationship, using separate parameters a, b, and c for each dataset to accommodate scale and observer-variation differences.The model assumes π_i ∼ N(q_i, c · σ), with c compensating for varying measurement accuracy and observer models.Source: C. Mixing Pairwise Comparisons and… · pp. 4–5
  • The posterior combines pairwise-comparison and rating likelihoods to infer latent quality scores q and parameters a, b, and c by maximum likelihood, while fixing all reference-image scores to zero.The model assumes conditional independence of comparisons and ratings given q; a Gaussian prior improves scaling precision when few comparisons are measured per condition pair.Source: C. Mixing Pairwise Comparisons and… · p. 5
  • A quality difference of 1 JOD corresponds to 75% of observers being able to see a difference between two conditions.The scale fixes σ = 1.048 so that distances can be interpreted in these units.Source: C. Mixing Pairwise Comparisons and… · p. 5

IV. UNIFIED PHOTOMETRIC IQA DATASET (UPIQ)

UPIQ merges existing SDR and HDR image-quality datasets onto a unified JOD scale, with images represented in absolute photometric units. The authors collect additional comparisons to align datasets and describe the selected data and alignment experiments.

IV. UNIFIED PHOTOMETRIC IQA DATASET (UPIQ)

  • UPIQ contains 4,159 images from existing SDR and HDR datasets, with images in absolute cd/m2 units and quality scores in interpretable JOD units.The paper describes it as the largest and most diverse HDR image-quality dataset to date; the prior largest HDR dataset contained 240 conditions.Source: §IV · p. 5

A. Selected Datasets

  • The authors selected TID2013, LIVE, Korshunov, and Narwaria, excluding datasets that lacked reference images, distortion variation, adequate image size, or sufficiently consistent scores.The selected datasets comprise two SDR and two HDR datasets.Source: A. Selected Datasets · p. 5

B. Dataset Alignment Experiments

  • Alignment combines within-dataset comparisons, including reference and cross-content pairs, with cross-dataset comparisons to make quality values comparable across datasets.The authors note that cross-content comparisons improve scale accuracy,; Fig. 4 depicts within-content, reference, cross-content, and cross-dataset pairs.Source: B. Dataset Alignment Experiments · p. 5, Figure 4 · p. 6
  • The alignment experiments collected 6,000 new pairwise comparisons from 20 participants, then iteratively selected cross-dataset pairs to cover the quality scale and updated the scale between batches.Each selected pair was compared by six participants; the authors describe this as moderate effort compared with collecting data from scratch.Source: B. Dataset Alignment Experiments · p. 6

C. UPIQ Dataset Scaling

The combined measurements are rescaled together, and the resulting scores differ substantially from the original dataset scores. Validation reports that images at the same JOD level have comparable perceived distortion, while cross-validation favors the proposed scaling over metric-based alignment.

C. UPIQ Dataset Scaling

  • Substantial differences between original and unified scores indicate that cross-dataset scaling and additional measurements changed the quality estimates.The combined dataset includes 571,215 pairwise comparisons and 27,676 rating measurements.Source: C. UPIQ Dataset Scaling · p. 7

D. UPIQ Dataset Validation

  • Images at the same JOD level show comparable perceived magnitudes of distortion, according to the qualitative comparison reported in the supplementary material.The paper then compares the scaling with metric-based alignment and evaluates pairwise accuracy.Source: D. UPIQ Dataset Validation · p. 7

1) Comparison to previous re-scaling work:

  • 0.71 versus 0.56 mean SROCC: the proposed scaling correlated better with subjective judgments than the method from in five-fold cross-validation.The authors also report that its scaled probabilities fall within the confidence interval of measured probabilities.Source: Table II · p. 7, 1) Comparison to previous re-scaling… · p. 7

2) Measuring pairwise accuracy:

The section evaluates the unified scale’s pairwise accuracy, tests whether multi-task learning can align datasets, and uses UPIQ to train and benchmark HDR quality metrics. It reports that the scale ranks cross-dataset conditions accurately and that a retrained PU-PieAPP outperforms existing metrics in cross-content validation.

3) Cross-dataset scaling versus multi-task learning:

A. Training Data-driven HDR Metric

2) Measuring pairwise accuracy:

B. Benchmark of HDR Quality Metrics

C. Maximum differentiation competition

The gMAD competition exposes distinct failure patterns in the two best-performing metrics: PU-PieAPP tends to underestimate JPEG-artifact image quality, while PU-FSIM has color, contrast, and structural-distortion weaknesses.

C. Maximum differentiation competition

  • PU-PieAPP tends to capture image rankings but underestimate JPEG-artifact quality; PU-FSIM misses color changes, overreacts to contrast changes, and underreacts to structural distortions.Source: C. Maximum differentiation competition · p. 11

D. Brightness-Adaptive Coding

The brightness-adaptive coding study finds that the JPEG bitrate needed for a target perceived quality differs by display brightness. UPIQ is presented as a large-scale photometric dataset for training and testing HDR metrics and evaluating such applications.

D. Brightness-Adaptive Coding

  • JPEG bitrate can be substantially reduced for dimmed mobile displays but should increase for HDR TVs, with larger brightness differences at higher encoded quality.The analysis uses PU-PieAPP to control JPEG compression for target JOD levels across 200 pristine test images.Source: Figure 12 · p. 12, D. Brightness-Adaptive Coding · p. 12
  • Above 100 cd/m2, PU-FSIM predicts falling quality as display luminance rises, while PU-PieAPP predicts improving quality, consistent with reported contrast-threshold measurements.Both metrics predict that JPEG distortions are less noticeable on darker displays.Source: D. Brightness-Adaptive Coding · pp. 11–12, Figure 12 · p. 12

VI. CONCLUSIONS

  • The authors report that merging datasets with additional cross-dataset comparisons is less effort than collecting from scratch and more accurate than alternative procedures.They note that the procedure could also apply to high-frame-rate, omni-directional, or foveated video quality.Source: §VI · p. 12
  • UPIQ is the first large-scale dataset for training and testing HDR image quality metrics, with images represented in absolute photometric and colorimetric units and quality scores in interpretable JOD units.Source: §VI · p. 12

I. INTRODUCTION

The section describes how pairwise comparisons connect dataset quality scales and how JOD-to-MOS relationships were assessed. For TID2013 and LIVE, higher-order polynomial fits add little adjusted R2, while monotonicity constrains suitable mappings.

II. EXPERIMENTAL PROTOCOL FOR DATASET MERGING

  • Aligning datasets requires within-dataset comparisons to link ratings with pairwise judgments and cross-dataset comparisons to connect otherwise disjoint scales.The procedure also relies on comparisons to references, within-content comparisons, and cross-content comparisons to establish a unified scale.Source: §II · p. 15

A. Displays and stimuli

  • Images shown on the HDR display were converted to absolute linear colorimetric units in Rec. 709, with peak luminance matched to the original displays and angular resolution preserved.Observers could pan across images larger than the display.Source: A. Displays and stimuli · p. 15

B. Experimental Procedure and Participants

  • The experiments used within-dataset comparisons before iteratively sampling cross-dataset pairs; each selected pair received six judgments, yielding 6,000 new comparisons from 20 participants.Pairs were selected to have similar quality, and the cross-dataset batches were updated after rescaling.Source: B. Experimental Procedure and… · p. 15

AND MEAN OPINION SCORES

  • Higher-order polynomial fits provide only slight adjusted R2 increases for TID2013 and LIVE, making non-linear fits hard to justify under the monotonicity requirement.A third-order fit for LIVE violates monotonicity; adjusted R2 accounts for the number of model parameters.Source: AND MEAN OPINION SCORES · p. 16, Figure 12 · p. 16

IV. UPIQ DATASET SELECTION

Candidate datasets were screened through within-dataset comparisons, and the authors used additional tests to examine disagreements between collected judgments and existing scores.

IV. UPIQ DATASET SELECTION

  • Empirical pairwise probabilities and DMOS differences from have little correlation, with SROCC 0.27.The comparisons helped assess whether candidate datasets were consistent with the new subjective judgments.Source: Figure 13 · p. 16, §IV · p. 16
  • Datasets with little or no correlation between original scores and new within-dataset judgments were excluded from UPIQ.Pairs in the pilot comparisons were assessed repeatedly, between six and ten times.Source: §IV · p. 16
  • A Maximum Differentiation experiment examined the most inconsistent image pairs; observers were particularly sensitive to JPEG blocking in smooth sky regions.The examples included lake and sunset images.Source: §IV · p. 16

V. PIEAPP DETAILED ARCHITECTURE

PU-PieAPP combines patch-level feature extraction with learned patch errors and weights, while a multitask variant attempts to learn a unified quality scale across datasets. The multitask network does not capture the tested cross-dataset relationships well.

V. PIEAPP DETAILED ARCHITECTURE

  • PU-PieAPP extracts whole-network and multi-depth patch features, predicts patch-wise errors and weights, then averages patch scores by their learned weights.The feature extraction network uses skip connections; the resulting image score is sA.Source: §V · p. 16, Figure 15 · p. 17
  • Replacing the PU-transform with a logarithmic function raises RMSE from 0.47 to 0.68, supporting the PU-transform for SDR/HDR quality predictions.The logarithmic alternative was scaled to the 0–1 range before network training.Source: §V · p. 17

VI. MULTITASK NETWORK

  • The multitask network shares a score-producing network across datasets and linearly maps its unified score to each dataset's scale, fixing TID2013 parameters at aT = 1 and bT = 0.This approach was designed to learn cross-dataset relationships without collecting additional data.Source: §VI · p. 17, Figure 16 · p. 17
  • Neither cross-dataset relationship is well captured by the multitask network when correlating score differences with empirical pairwise probabilities.The comparison is reported for differences between images drawn from different datasets.Source: §VI · p. 17, Table II · p. 17

VII. MAXIMUM DIFFERENTIATION COMPETITION

The competition ranks metrics by how well they distinguish quality-different pairs, with retrained PU-Pie-APP strongest. Dataset examples visually check the unified JOD scale while warning that SDR display conversion can mask or amplify distortions.

VII. MAXIMUM DIFFERENTIATION COMPETITION

  • The competition selects pairs that differ under the tested metric but are similar under the benchmark, then reports precision across 100 pairs.Pairs less than 1 JOD apart are treated as similar; others are different, and Figure 17's matrix gives test metrics by row and benchmark metrics by column.Source: §VII · p. 18
  • PU-Pie-APP (re-trained) leads both matrix axes; PU-FSIM ranks second, resembling FSIM overall but performing better when paired with dynamic-range-accounting metrics.PU-Pie-APP performs almost on par with HDRVDP2-2 when paired with that metric.Source: §VII · p. 18

VIII. EXAMPLES OF THE DATASET

  • Figures 18–19 show examples at JOD −1 and −2 across four datasets, but gamma-corrected SDR display can mask or amplify distortions.Each dataset section has distorted and reference rows; the figures are approximate demonstrations of the final scale.Source: §VIII · p. 18
  • Images from different datasets at the same JOD level have similar distortion severity, despite HDR datasets' local distortions and SDR datasets' uniform distortions.Narwaria's panorama images make local distortions less noticeable because of their size.Source: §VIII · p. 18
Loading 2012.10758v2…