PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unanticipated Asymmetry Between Perceptual Optimization and Assessment

Jiabei Zhang, Qi Wang, Siyu Wu, Du Chen, Tianhe Wu

arXiv:2509.20878v1cs.CV

TL;DR

Whether fidelity metrics that perform well in IQA also guide perceptual optimization effectively, and whether discriminator features transfer to IQA, remains underexplored. The paper studies these questions alongside discriminator design in super-resolution. It finds that optimization and assessment effectiveness can diverge, while patch-level convolutional discriminators yield more stable and detailed results than vanilla or Transformer-based alternatives.

  • Problem

    The relationship between fidelity metrics’ IQA capability and their effectiveness as optimization objectives, and the transferability of discriminator features to IQA, remains underexplored.

    Source: §1 · p. 2

  • Method

    The study tests varied DISTS-style fidelity metrics and discriminator designs with SwinIR super-resolution, and evaluates discriminator-feature initialization for full-reference and no-reference IQA.

    Source: §1 · p. 3

  • Results

    Higher IQA capability does not guarantee stronger optimization guidance, discriminator features transfer poorly to IQA, and patch-level convolutional discriminators yield more stable and detailed results than vanilla or Transformer-based alternatives.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Optimization utility and IQA assessment capability are distinct roles, and discriminator design shapes detail reconstruction and training stability.

    Source: §5 · p. 9

    Experiments use only SwinIR, and discriminator-feature transfer findings may partly reflect the relatively small, narrow super-resolution training data.

    Source: §5 · p. 9

Abstract

from arXiv · show

Perceptual optimization is primarily driven by the fidelity objective, which enforces both semantic consistency and overall visual realism, while the adversarial objective provides complementary refinement by enhancing perceptual sharpness and fine-grained detail. Despite their central role, the correlation between their effectiveness as optimization objectives and their capability as image quality assessment (IQA) metrics remains underexplored. In this work, we conduct a systematic analysis and reveal an unanticipated asymmetry between perceptual optimization and assessment: fidelity metrics that excel in IQA are not necessarily effective for perceptual optimization, with this misalignment emerging more distinctly under adversarial training. In addition, while discriminators effectively suppress artifacts during optimization, their learned representations offer only limited benefits when reused as backbone initializations for IQA models. Beyond this asymmetry, our findings further demonstrate that discriminator design plays a decisive role in shaping optimization, with patch-level and convolutional architectures providing more faithful detail reconstruction than vanilla or Transformer-based alternatives. These insights advance the understanding of loss function design and its connection to IQA transferability, paving the way for more principled approaches to perceptual optimization.

1 INTRODUCTION

The paper tests whether fidelity metrics’ IQA strength predicts their optimization utility and whether discriminator features transfer to IQA, finding mismatches between the roles. It also examines how discriminator design affects optimization.

1 INTRODUCTION

  • Perceptual optimization combines reconstruction- and perceptual-oriented fidelity terms with adversarial supervision, which complements fidelity objectives to address artifacts from perceptual losses.Pixel-wise reconstruction losses can over-smooth images, while perceptual objectives can introduce artifacts such as checkerboard patterns.Source: §1 · p. 2
  • Using SwinIR for single-image super-resolution, the study tests DISTS-style metrics with varied backbones and four objective settings, then compares discriminator architectures and IQA transfer.IQA transfer is assessed under full-reference and no-reference settings, against ImageNet-pretrained and randomly initialized backbones.Source: §1 · pp. 2–3
  • Higher IQA capability does not necessarily make a fidelity metric more effective for optimization, particularly when adversarial objectives are involved.The mismatch is more distinct under adversarial training.Source: §1 · p. 3
  • Discriminator features contribute little when reused to initialize IQA models, despite discriminators’ effectiveness in suppressing artifacts during optimization.This highlights an asymmetry between optimization and assessment roles.Source: §1 · p. 3
  • Patch-level discriminators enable more faithful detail reconstruction than vanilla designs, while convolutional discriminators are more stable than Transformer-based alternatives.Source: §1 · p. 3

2 FORMULATION OF PERCEPTUAL OPTIMIZATION OBJECTIVES

The formulation combines reconstruction, perceptual, and adversarial losses, with the analysis focused on perceptual and adversarial components. It also specifies a DISTS-style metric and a relativistic adversarial objective.

2.1 COMPOSITE OBJECTIVE FOR PERCEPTUAL OPTIMIZATION

  • The reconstruction loss supports low-frequency structure and color consistency, while perceptual loss compares deep features to encourage structural alignment and fine-grained texture preservation.Source: §2.1 · p. 3
  • The composite objective combines reconstruction, perceptual, and adversarial losses; the analysis focuses on the perceptual and adversarial components.The authors identify these latter two terms as decisive in shaping perceptual realism and examine their contributions and interactions.Source: §2.1 · p. 3, §2 · p. 3

2.2 COMPUTATION OF DISTS-STYLE PERCEPTUAL METRICS

  • The DISTS-style family applies injective visual feature transformations and assesses global similarity across texture and structure dimensions.Its perceptual metric combines texture and structure similarities across feature layers and channels.Source: §2.2 · pp. 3–4
  • The metric operates on feature channels from generated and reference images, with learnable weights for layer- and channel-level texture and structure similarities.The parameters are subject to a normalization constraint.Source: §2.2 · pp. 3–4

2.3 ADVERSARIAL LOSS IN PERCEPTUAL OPTIMIZATION

  • Vanilla discriminators output one global image score, whereas patch-level discriminators predict real or fake across local patches.Source: §2.2 · p. 3
  • Adversarial loss frames generation as a two-player game: the generator seeks realistic images while the discriminator distinguishes generated from real samples.This interplay steers outputs toward structural faithfulness and perceptual convincingness.Source: §2.3 · p. 4
  • Following ESRGAN, the study adopts a relativistic discriminator formulation for adversarial optimization.The generator and discriminator use paired adversarial losses based on the asymmetric discrepancy D(x, y).Source: §2.3 · p. 4
  • The relativistic discrepancy applies a sigmoid to a sample’s discriminator score relative to the mini-batch average score of real images.The discriminator function and mini-batch averaging operation define the discrepancy used in the losses.Source: §2.3 · p. 4

3 EXPERIMENTS

The experiments test how perceptual metrics and discriminator designs affect super-resolution optimization and IQA transfer. They find that IQA strength does not reliably predict optimization utility, while discriminator architecture shapes detail reconstruction and training stability.

3.1 EXPERIMENTAL SETUPS

  • The study constructs DISTS-style perceptual objectives with varied backbones and evaluates SwinIR super-resolution under four second-stage loss settings.The setup also compares discriminator-trained backbone initializations with random and ImageNet initializations for FR- and NR-IQA.Source: §3.1 · pp. 4–5, §3 · p. 4, §3.3 · p. 7

3.2 ANALYSIS OF PERCEPTUAL METRICS

  • Higher IQA performance does not reliably predict better SR optimization, and the mismatch becomes more pronounced with adversarial supervision.Randomly initialized ℓVGG-16-R ranks last on FR-IQA benchmarks yet surpasses ℓResNet-50 and ℓCLIP-ViT in perceptual-only and reconstruction-plus-perceptual settings; adversarial supervision reduces the score deviation from 5.863 to 1.555.Source: §3.2 · pp. 6–7, Figure 3 · p. 6
  • Adversarial loss generally improves optimization and reduces checkerboard artifacts, while adding a reconstruction term yields only marginal gains.The adversarial benefit does not hold for ℓVGG-16-R; with adversarial loss, outputs optimized by different perceptual metrics show only minor visual differences.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 GAN-BASED INITIALIZATION FOR IQA

  • ImageNet pretraining is the strongest IQA initialization across backbones and datasets, whereas GAN-trained discriminator features yield only small gains over random initialization.The transfer analysis compares FR-IQA benchmarks and NR-IQA on KADID; discriminator features lack the semantic coverage and robustness required for generalizable quality assessment.Source: §3.3 · pp. 7–8, Table 2 · p. 7, Figure 5 · p. 7

3.4 FURTHER ANALYSIS OF DISCRIMINATORS

  • Patch-level discriminators improve perceptual optimization with VGG-16 and ResNet-50, while DINOv2 shows only a marginal gain.Patch-level designs raise the average score by +0.52 for VGG-16 and +0.38 for ResNet-50, versus +0.15 for DINOv2; they also yield sharper textures, clearer local structures, and fewer artifacts.Source: §3.4 · p. 8, Table 3 · p. 8, Figure 6 · p. 8
  • ResNet-50 patch-level discriminators maintain high SR scores across a broad range of adversarial weights, while VGG-16 is moderately sensitive and DINOv2 can collapse.The analysis varies adversarial weight λ3; DINOv2's instability limits its suitability for stable adversarial optimization.Source: §3.4 · pp. 8–9, Figure 7 · p. 8

4 RELATED WORK

Prior work developed perceptual losses and IQA metrics to address the limits of pixel-wise fidelity, while adversarial training adds sharpness and realistic local structure. This work examines how optimization and evaluation relate, including the limits of reference-based assessment.

Perceptual Optimization

  • Pixel-wise fidelity objectives such as ℓ1 or PSNR may fail to reflect human perceptual judgments, motivating deep-feature perceptual losses and task-specific metrics that better align with human opinion scores.Adversarial training emerged as a complementary objective, with discriminators encouraging sharper textures and more perceptually realistic local structures.Source: §4 · p. 9
  • The work addresses a gap in prior comparisons of perceptual metrics by systematically examining their interplay with evaluation, especially the role of adversarial discriminators (Blau & Michaeli, 2018; Ding et al., 2020; 2021b).Earlier perceptual losses used pretrained deep features, while later metrics targeted human opinion scores.Source: §4 · p. 9

Perceptual Metrics

  • FR-IQA assumes a perfect-quality reference, whereas asymmetric metrics such as VIF, CKDN, and A-FINE allow test images to surpass it (Sheikh & Bovik, 2006; Zheng et al., 2021; Chen et al., 2025).The passage traces metric development from error-based measures through structural and feature-domain approaches.Source: §4 · p. 9
  • Most perceptual optimization metrics still guide models toward the reference, implicitly treating it as the perceptual upper bound.The cited progression includes SSIM and its extensions, as well as deep-feature methods such as DISTS (Wang et al., 2004; Wang et al., 2003; Zhang et al., 2011; Ding et al., 2020; Ding et al., 2021a).Source: §4 · p. 9

5 CONCLUSION AND DISCUSSION

The study finds asymmetries between perceptual optimization and IQA, including weak transfer of discriminator features to IQA and benefits from patch-level convolutional discriminators. Its findings are limited by the SwinIR-only experiments and motivate broader testing and co-design of optimization and evaluation.

5 CONCLUSION AND DISCUSSION

  • Stronger fidelity metrics do not guarantee better optimization, discriminator features transfer poorly to IQA, and patch-level convolutional discriminators yield more stable, detailed results than vanilla or Transformer-based alternatives.The study examines perceptual metrics and discriminator architectures in both perceptual optimization and IQA.Source: §5 · p. 9
  • Experiments were conducted only on SwinIR, leaving robustness across weaker models, stronger models, and visual-tokenizer training for future study.The passage names SRResNet, HAT, and visual tokenizers as possible extensions.Source: §5 · p. 9
  • The limited transfer of discriminator features to IQA may partly reflect the small, narrow SR training data compared with large, diverse datasets such as ImageNet.The authors say larger, more diverse training data would allow a fairer and more conclusive assessment.Source: §5 · p. 9
  • The authors question relying solely on perceptual metrics for optimization and judging their capability by resulting outcomes, and call for co-design of metrics, adversarial objectives, and evaluation protocols.They also identify discriminator-driven designs in reinforcement-learning post-training for image generation as a future direction.Source: §5 · p. 9

APPENDIX

The appendix provides supplementary training, metric-rescaling, results, and visualization details.

APPENDIX

  • The appendix collects training configurations, metric-rescaling methods, additional results, and qualitative visualizations.Source: APPENDIX · p. 14

A TRAINING CONFIGURATIONS FOR DISTS-STYLE PERCEPTUAL METRICS

DISTS-style perceptual-metric training fixes the backbone and uses projected zeroth-stage weights for stability, with optimization on KADID-10K.

A TRAINING CONFIGURATIONS FOR DISTS-STYLE PERCEPTUAL METRICS

  • The metric-training backbone is fixed to preserve generalizability, and zeroth-stage weights are projected onto [0.02, 1] after each update for stability.Training follows Ding et al. (2020).Source: A TRAINING CONFIGURATIONS FOR… · p. 14
  • Models are trained for 5,000 iterations on KADID-10K with batch size 32 using Adam and an initial learning rate of 1 × 10^-4, halved every 1,000 iterations.Source: A TRAINING CONFIGURATIONS FOR… · p. 14

B TRAINING CONFIGURATIONS IN GAN-BASED INITIALIZATION

The NR-IQA training experiments use two datasets and evaluate test-set performance with SRCC and PLCC.

B TRAINING CONFIGURATIONS IN GAN-BASED INITIALIZATION

  • NR-IQA models are trained on KADID-10K and KonIQ-10k with a 6:2:2 train-validation-test split, and test performance is reported using SRCC and PLCC.FR-IQA models use VGG-16, ResNet-50, and DINOv2 with the strategies described in Sec. A.Source: B TRAINING CONFIGURATIONS IN GAN-BASED… · p. 14

C RESCALING VISUAL QUALITY METRICS

NR-IQA outputs are mapped to a common 1–100 scale to support comparison of perceptual quality across models.

C RESCALING VISUAL QUALITY METRICS

  • A four-parameter monotonic logistic mapping rescales NR-IQA outputs to a unified perceptual scale from 1 to 100.Larger normalized values correspond to higher perceived image quality.Source: C RESCALING VISUAL QUALITY METRICS · p. 14
  • The raw model score N(x) is mapped with η1 and η2 fixed at 100 and 1, while η3 and η4 are estimated during fitting.The fixed parameters define the upper and lower bounds of the normalized scale.Source: C RESCALING VISUAL QUALITY METRICS · p. 14

D ADDITIONAL RESULTS

The additional results show that convolutional perceptual metrics perform strongly on synthetic FR-IQA datasets, while ImageNet pretraining yields the best NR-IQA initialization results.

D ADDITIONAL RESULTS

  • Convolutional backbones substantially outperform Transformer-based ones for perceptual quality assessment under synthetic distortions.ℓConvNeXt and ℓSwin-T achieve intermediate results, while ℓCLIP-ViT and ℓVGG-16-R have the lowest overall rank.Source: D ADDITIONAL RESULTS · p. 14
  • ℓVGG-16 averages SRCC 0.853 and PLCC 0.867 across four synthetic FR-IQA datasets, while ℓResNet-50 ranks second at 0.810/0.843.ℓResNet-50 achieves the best PLCC on TID2013 at 0.896.Source: D ADDITIONAL RESULTS · p. 14, Table 4 · p. 15
  • ImageNet pretraining yields the best NR-IQA performance across all backbones; GAN initialization improves over random initialization but remains behind ImageNet.DINOv2 scores highest under pretraining but performs poorly with random or GAN initialization; convolutional networks are moderately robust under weaker initialization.Source: D ADDITIONAL RESULTS · p. 14, Figure 8 · p. 15

E ADDITIONAL VISUALIZATIONS

The visualizations show distinct artifact and texture patterns across optimization objectives, with the combined reconstruction, perceptual, and adversarial setting giving the best balance of fidelity and realism.

E ADDITIONAL VISUALIZATIONS

  • Pure reconstruction produces smooth but over-smoothed results, while VGG-16-R and VGG-16 show zippering or checkerboard artifacts.Perceptual loss improves edge continuity and global structure, although ViT backbones can hallucinate elongated fine-texture streaks.Source: E ADDITIONAL VISUALIZATIONS · p. 15
  • Convolutional discriminators paired with CNN backbones yield the most coherent local textures and consistent contrast, while Transformer backbones remain more prone to banding and unstable micro-patterns.Source: E ADDITIONAL VISUALIZATIONS · p. 15
  • The combined reconstruction, perceptual, and GAN objective provides the best balance between fidelity and realism.Across the visualizations, adversarial training recovers stochastic detail and suppresses grid artifacts.Source: E ADDITIONAL VISUALIZATIONS · p. 15, Figure 9 · p. 16, Figure 10 · p. 17

Related papers

Loading 2509.20878v1…