PaperlayerView exact revision on arXiv

Source-linked AI summary

DISRQAD: Diffusion Image Super-Resolution Quality Assessment Dataset and Benchmark

Nikita Kukuzei, Artem Borisov, Evgeney Bogatyrev, Khaled Abud, Egor Chistov, Dmitriy Vatolin

arXiv:2610.09077v1cs.CV

TL;DR

Diffusion SR can generate plausible detail unsupported by the input, and it remains unclear how consistently existing IQA metrics align with human judgments across generators and input conditions. DISRQAD rates outputs from diffusion and non-diffusion methods and benchmarks standard and adapted metrics. SRCC is lower on diffusion outputs for 60 of 62 evaluated variants; a pruned and distilled student reaches 0.496 SRCC on diffusion SR.

  • Problem

    It is unclear whether IQA metrics agree with human judgments across a broader range of SR generators and input conditions, particularly when plausible generated detail does not match the underlying scene.

    Source: §1 · p. 2

  • Method

    DISRQAD collects MOS for 14,000 outputs from diffusion and non-diffusion SR methods and evaluates 51 standard IQA configurations and 11 adapted variants.

    Source: §1 · p. 3, §4.1 · p. 5, §7 · p. 9

  • Results

    SRCC was lower on diffusion outputs for 60 of 62 evaluated variants; the Q-ReAlign-mini student reached 0.496 SRCC on diffusion SR.

    Source: §7 · pp. 9–10

  • Takeaways & Limitations

    The benchmark and analyses provide a basis for developing and testing quality predictors for generative SR.

    Source: §7 · p. 10

    DISRQAD measures perceived output quality, not faithfulness to the LR input; its detailed, high-quality sources and controlled single-degradation inputs mean compound real-world degradations need separate evaluation.

    Source: D LIMITATIONS · p. 21

Abstract

from arXiv · show

Diffusion-based image super-resolution (SR) can create visually plausible detail that is not supported by the low-resolution input. We introduce DISRQAD, a subjective-quality dataset and diagnostic benchmark for this setting. It contains mean opinion scores (MOS) for 14,000 SR outputs from ten diffusion and four non-diffusion methods, spanning four low-resolution degradation conditions and x2/x4 upscaling. We evaluate 51 standard full-reference and no-reference metric configurations and 11 adapted variants. Agreement with MOS is substantially weaker on diffusion outputs: the strongest standard no-reference baseline reaches 0.431 SRCC on diffusion SR versus 0.813 on non-diffusion SR. As a case study in benchmark use, a pruned and distilled Q-ReAlign-mini student reaches 0.496 SRCC on diffusion SR. DISRQAD measures perceived output quality, not faithfulness to the input; it enables analysis of metric behavior across generator families and input conditions. Our findings reveal a substantial gap in the assessment of diffusion-based SR and provide a basis for developing quality models sensitive to diffusion-specific artifacts.

1 INTRODUCTION

Diffusion-based super-resolution can produce plausible details that do not match the underlying scene, making quality assessment ambiguous and metric agreement with human judgments uncertain. DISRQAD provides human ratings across generator families and evaluates existing metrics and SR-specific adaptation.

1 INTRODUCTION

  • Human ratings are needed because plausible but unsupported detail makes both reference fidelity and appearance ambiguous as quality targets.The study measures output quality without showing raters the LR input or HR reference; fidelity is a separate assessment target.Source: §1 · pp. 1–2
  • DISRQAD contains 10,000 diffusion-based and 4,000 classic SR images, with 205,578 judgments.The source images are described as semantically diverse and structurally complex.Source: §1 · p. 3
  • The study compares 51 FR and NR IQA metrics spanning traditional, CNN-based, and VLM-based families, including SR-specific methods.The evaluation is intended to identify metrics suited to diffusion-based SR and examine the effect of fine-tuning on diffusion-generated images.Source: §1 · pp. 2–3
  • The authors fine-tune representative metrics on diffusion-SR-specific data to test whether SR-specific training better aligns metric scores with human perception.The evaluation also compares 10 diffusion and four non-diffusion methods using MOS annotations for 14,000 outputs and more than 100,000 quality judgments.Source: §1 · pp. 2–3

2 RELATED WORK

Prior SR quality datasets mainly represent non-diffusion methods, while newer generative-SR datasets cover fewer methods or use different rating protocols. DISRQAD places diffusion and non-diffusion SR within one direct-rating benchmark and evaluates FR and NR metrics against MOS.

2.1 SUBJECTIVE DATASETS FOR SUPER-RESOLUTION

  • Earlier SR-IQA datasets chiefly represent non-diffusion SR, while newer datasets include diffusion methods but cover five methods each.SRIQA-Bench includes cases where an enhanced image is preferred to its reference; ISRGen-QA provides direct MOS.Source: §2.1 · p. 3

2.2 IQA BENCHMARKS

  • DISRQAD scores each SR output directly on a five-category scale and averages ratings into MOS, whereas some restoration benchmarks also record pairwise preferences.The benchmark holds the task to SR while varying generator family.Source: §2.2 · p. 3
  • The benchmark reports signed SRCC and PLCC against direct MOS separately for diffusion and non-diffusion SR.This tests whether metrics that follow human ratings on non-diffusion SR also follow them on diffusion SR.Source: §2.2 · p. 4

2.3 IMAGE QUALITY METRICS

  • FR metrics compare outputs with an HR reference using differing notions of correspondence, while NR metrics assess appearance without an HR image.FR approaches include pixel or structural similarity, feature comparisons, edge restoration, and handling imperfect references; NR methods vary in supervision.Source: §2.3 · p. 4

3 DISRQAD BENCHMARK

DISRQAD builds a rated SR benchmark from selected source images and multiple degradation and scaling conditions. Raters judge each output’s perceived quality directly, and the protocol uses at least 13 ratings per output.

3 DISRQAD BENCHMARK

  • The source pipeline clusters more than 468,000 candidate images and selects 10,036 sources using spatial-information and Q-Align thresholds.The selection favors detailed, visually clean sources and excludes low-SI or low-Q-Align images from the rated benchmark.Source: §3 · p. 4
  • For each source and scale, the benchmark creates four LR conditions—resized, blurred, noisy, or JPEG-compressed—at ×2 and ×4, yielding 80,288 LR images.Ten diffusion and four non-diffusion SR methods produce outputs, from which 10,000 and 4,000 are selected, respectively.Source: §3 · p. 5

3.1 SUBJECTIVE TARGET AND PROTOCOL

  • The target is perceived output quality: raters see each SR output without its LR input or HR reference and rate it on a five-category scale.Ratings are averaged into MOS, so visually attractive but scene-inconsistent images may receive high scores.Source: §3.1 · p. 5
  • The final annotation collects at least 13 ratings per output, with more than 205,000 judgments from over 80 observers.In the pilot, 13-rating MOS had SRCC 0.983 with the 20-rating baseline (95% CI [0.970, 0.990]); this pilot agreement does not establish precision for each final MOS.Source: §3.1 · p. 5, Table 2 · p. 5

4 EXPERIMENTS

The experiments compare standard IQA configurations with human MOS ratings across diffusion and non-diffusion SR. Agreement is substantially weaker on diffusion outputs, and adaptation effects vary by metric and SR family.

4.1 METRIC SELECTION AND EVALUATION

  • The evaluation covers 51 standard FR and NR configurations and 11 adapted variants, with correlations computed separately on diffusion and non-diffusion outputs.SRCC measures ranking agreement and PLCC linear agreement with MOS, without nonlinear fitting.Source: §4.1 · p. 5

4.2 BENCHMARK RESULTS

  • Across 51 configurations, median SRCC falls from 0.65 on non-diffusion outputs to 0.27 on diffusion outputs, and median PLCC falls from 0.60 to 0.31.The decrease spans FR and NR methods with diverse designs and supervision.Source: §4.2 · p. 5, Table 3 · p. 6
  • Q-Align's SRCC/PLCC falls from 0.81/0.76 on non-diffusion SR to 0.42/0.49 on diffusion SR; TOPIQ(FR) falls from 0.81/0.79 to 0.40/0.42.FSIM reaches 0.39/0.40 on diffusion SR; neither HR-reference access nor general-purpose learned quality representations alone ensures agreement with human judgments.Source: §4.2 · p. 6, Table 3 · p. 6
  • Fine-tuning changes diffusion SRCC from 0.28 to 0.29 for LPIPS-AlexNet, 0.30 to 0.27 for LPIPS-VGG, and 0.27 to 0.31 for MUSIQ; MUSIQ's non-diffusion SRCC falls from 0.60 to 0.50.Adaptation gains on one SR family should not be taken as evidence of transfer to the other.Source: §4.2 · pp. 6–7

5 ANALYSIS BY CONDITION AND SR ADAPTATION

Metric performance varies with SR family, generator, degradation, and scale. Adaptation can improve diffusion SR agreement, but gains may come with lower performance on classic SR and other distortions.

5.1 PERFORMANCE BY CONTENT AND INPUT CONDITION

  • Q-Align reaches 0.813 overall SRCC on classic SR versus 0.415 on diffusion SR, and TOPIQ-FR reaches 0.811 versus 0.397; 60 of 62 variants score higher on classic SR.Strong classic-SR performance provides limited evidence that a metric will rank diffusion outputs according to human judgments.Source: §5.1 · p. 7
  • For diffusion outputs, 52 of 62 metric variants have higher SRCC at ×4 than ×2; the blur subset yields the highest SRCC for almost all metrics.On classic SR, only 25 variants score higher at ×4 than at ×2.Source: §5.1 · p. 7
  • PassionSR and ResShift yield the highest within-method SRCC for 36 and 20 variants, while DiT4SR is lowest for 38; classic-method rankings depend on the metric.RealSR has the highest median classic within-method SRCC, but Q-Align and TOPIQ-FR lead on different classic methods; CODSR and PASD have median SRCCs of 0.135 and 0.158.Source: §5.1 · p. 8
  • Distilled Q-ReAlign-mini achieves the highest overall diffusion SRCC (0.496) and a classic SRCC of 0.729.Fine-tuned TOPIQ-FR rises from 0.397 to 0.436 diffusion SRCC while its classic SRCC falls from 0.811 to 0.694.Source: §5.1 · p. 8, Table 5 · p. 8

5.2 PERFORMANCE GAIN AFTER FINE-TUNING

  • Ten FR and ten NR metrics were fine-tuned on SRIQA-Bench using MOS regression; Q-ReAlign-mini was limited to two epochs because of model size and rapid convergence.The described settings include 30 epochs for other methods, a batch size of 8, and checkpoint selection by SRCC.Source: §5.2 · p. 8
  • Most fine-tunes improve at least one SR track, with three improving both, but quality on non-SR distortions drops significantly, limiting cross-domain universality.The authors call for approaches that retain quality on classic SR and other distortion types.Source: §5.2 · pp. 8–9

6 EFFICIENT SR-ADAPTED METRIC

The proposed efficient metric is a pruned Q-ReAlign-mini student trained through general-IQA distillation followed by SR adaptation. Its compact design combines token-based quality scoring with regression.

6 EFFICIENT SR-ADAPTED METRIC

  • The student is pruned from the 0.8B Q-ReAlign-mini checkpoint to 340M parameters, compared with the 8.2B Q-Align baseline.It retains the visual encoder while reducing 24 language modules to 12 and restricting the vocabulary to 30 tokens.Source: §6 · p. 9
  • Training uses hierarchical distillation on general-IQA datasets before SR adaptation on 990 SRIQA-Bench images with 248 general-IQA replay images.The remaining 110 SRIQA-Bench images are used for checkpoint selection.Source: §6 · p. 9
  • A five-token quality head and a regression head produce scores that are combined through a learned convex combination.One head scores five ordered quality tokens; the other regresses a continuous score from the last hidden state.Source: §6 · p. 9

7 CONCLUSION

DISRQAD shows that IQA metrics generally agree less with human ratings on diffusion SR than on non-diffusion SR, while its benchmark targets perceived output quality rather than fidelity to the input.

7 CONCLUSION

  • For 60 of 62 evaluated variants, SRCC was lower on diffusion than non-diffusion outputs, across FR and NR metrics, with the gap varying by generator, degradation, and scale.Performance on conventional SR alone therefore provides an incomplete picture of suitability for generative SR.Source: §7 · p. 9
  • The Q-ReAlign-mini case-study student reaches SRCC 0.496 on diffusion SR and 0.729 on non-diffusion SR with 340M parameters.Fine-tuning can improve agreement with human ratings, but its effects differ across metrics and SR families.Source: §7 · p. 10
  • DISRQAD measures perceived output quality; assessing whether synthesized details remain faithful to the LR input is left as a complementary direction for future work.The dataset and analyses provide a basis for developing and testing quality predictors for generative SR.Source: §7 · p. 10

REPRODUCIBILITY STATEMENT

The statement maps the paper’s dataset, evaluation, and implementation details to their sections and appendices.

REPRODUCIBILITY STATEMENT

  • Section 3 covers dataset construction, the subjective quality target, and released-data access; Section 4 specifies IQA configurations and correlation measures.Appendices A–C document source selection and data generation, annotation and pilot procedures, and the student architecture and training settings.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASET CONSTRUCTION

The dataset construction selects diverse, detailed source images, generates LR inputs and SR outputs across methods and conditions, then samples outputs for subjective assessment.

A.1 SOURCE SELECTION

  • The source-selection features combine blockiness, detected-object counts, ARNIQA quality and distortion embeddings, and CLIP semantic embeddings with Q-Align and Spatial Information.Q-Align reflects image quality and aesthetics, while Spatial Information measures spatial detail using Sobel-filtered intensities.Source: §A.1 · pp. 16–17
  • K-means clustering and medoid selection, followed by Q-Align and Spatial Information thresholds, yield 10,036 sources from a pool exceeding 468,000 images.The selected images are intended to span semantic and structural diversity, distortions, and visual and aesthetic quality.Source: §A.1 · pp. 16–17

A.2 SR DATASET CONSTRUCTION

  • The 10,036 sources produce 80,288 LR images across four degradation conditions and x2/x4 scaling, and the benchmark selects ten diffusion SR methods spanning different formulations and designs.The stated degradations are Gaussian blur, random noise, and JPEG compression, alongside cubic-interpolation downscaling.Source: §A.1 · p. 17, §A.2 · p. 17
  • K-means clustering on output features selects 10,000 diffusion and 4,000 non-diffusion SR outputs for evaluation; semantic-cluster coverage after this selection is not established.The output pool includes diffusion methods and the non-diffusion methods SwinIR, RealSR, HAT, and Real-ESRGAN.Source: §A.2 · p. 18

B SUBJECTIVE PROTOCOL AND PILOT

The authors compare pairwise preferences with direct full-reference and no-reference ratings, select a no-reference MOS protocol from a pilot, and use at least 13 ratings per image for final annotation.

B SUBJECTIVE PROTOCOL AND PILOT

  • Pairwise assessment aggregates choices between two outputs with a Bradley–Terry model, but exhaustive comparisons grow quadratically and scores across reference images are not directly comparable.Direct MOS ratings instead provide a common scale with annotation cost growing linearly in the number of images.Source: B SUBJECTIVE PROTOCOL AND PILOT · p. 18
  • Assessors saw either the GT reference and SR image for full-reference ratings or only the SR image for no-reference ratings; control items screened assessors after training.The pilot compared three protocols on 50 SR images spanning distortions and quality levels.Source: B SUBJECTIVE PROTOCOL AND PILOT · p. 18
  • In the 50-output pilot, no-reference MOS showed closer fitted agreement with pairwise scores than full-reference MOS, motivating the selected protocol, though the pilot does not establish general superiority.The nonlinear mapping was fitted separately for each reference image and MOS protocol, with agreement assessed by average fitting error.Source: B SUBJECTIVE PROTOCOL AND PILOT · p. 19
  • With 13 ratings per image, the pilot reports SROCC 0.983084 against scores based on at least 20 ratings, and the authors selected a minimum of 13 for final annotation.The pilot confidence interval does not establish the precision of each final MOS.Source: B SUBJECTIVE PROTOCOL AND PILOT · p. 19
  • The final dataset uses no-reference MOS on a five-category scale, with at least 13 ratings per image.The protocol retains training examples and control items used in the pilot study.Source: B SUBJECTIVE PROTOCOL AND PILOT · p. 19

C STUDENT ARCHITECTURE AND TRAINING

To make adaptation more practical, the authors construct a smaller Q-ReAlign-mini student and train it through hierarchical distillation followed by SR adaptation. On held-out DISRQAD, its reported correlations are lower for diffusion than non-diffusion SR.

C.1 STUDENT ARCHITECTURE

  • The 340M-parameter student retains the visual encoder, prunes the language modules, and combines a five-token quality score with a continuous regression estimate.The final prediction is a learned convex combination of the token and regression heads.Source: §C.1 · p. 20

C.2 HIERARCHICAL DISTILLATION AND SR ADAPTATION

  • Hierarchical distillation matches teacher visual and language features, attention maps, scores, and token distributions, while the student also learns from MOS labels and ranking and ordinal objectives.SR adaptation then removes the teacher, trains on 990 SRIQA-Bench images plus 248 general-IQA images, and regularizes toward the first-stage checkpoint.Source: §C.2 · p. 21

C.3 DIFFUSION SR ADAPTATION

  • 0.761 SRCC and 0.773 PLCC on diffusion SR, versus 0.890 SRCC and 0.879 PLCC on non-diffusion SR, show the student predicts perceived quality in-domain while a gap remains.Evaluation uses the held-out private split after fine-tuning on the open part of DISRQAD.Source: §C.3 · p. 21

D LIMITATIONS

  • DISRQAD assesses perceived output quality without reference to the LR input; its detailed sources and controlled single-degradation setting leave compound real-world degradations for separate evaluation.Faithfulness to the LR input and semantic preservation are complementary assessment targets.Source: D LIMITATIONS · p. 21
Loading 2610.09077v1…