PaperlayerView exact revision on arXiv

Source-linked AI summary

TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment

Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, Weisi Lin

arXiv:2308.03060v1cs.CV

TL;DR

Existing IQA methods do not fully model interactions between high-level semantic regions and low-level distortion features. TOPIQ addresses this with CFANet, a top-down multi-scale network using cross-scale attention, achieving competitive results across FR and NR benchmarks with substantially lower computation.

  • Problem

    Existing IQA approaches do not fully exploit multi-scale features because they lack interaction between high-level semantic features and low-level distortion features.

  • Method

    TOPIQ uses CFANet to progressively propagate high-level semantic information to low-level distortion features through cross-scale attention, with gated local pooling reducing redundant computation.

  • Results

    CFANet achieves competitive performance on public FR and NR benchmarks, including outperforming existing CNN methods and MUSIQ on three authentic-distortion NR datasets.

  • Takeaways & Limitations

    A simple ResNet50-backed top-down design can support effective IQA for both reference-based and no-reference settings while requiring approximately 13% of the FLOPS of the best existing FR method.

Abstract

from arXiv · show

Image Quality Assessment (IQA) is a fundamental task in computer vision that has witnessed remarkable progress with deep neural networks. Inspired by the characteristics of the human visual system, existing methods typically use a combination of global and local representations (\ie, multi-scale features) to achieve superior performance. However, most of them adopt simple linear fusion of multi-scale features, and neglect their possibly complex relationship and interaction. In contrast, humans typically first form a global impression to locate important regions and then focus on local details in those regions. We therefore propose a top-down approach that uses high-level semantics to guide the IQA network to focus on semantically important local distortion regions, named as \emph{TOPIQ}. Our approach to IQA involves the design of a heuristic coarse-to-fine network (CFANet) that leverages multi-scale features and progressively propagates multi-level semantic information to low-level representations in a top-down manner. A key component of our approach is the proposed cross-scale attention mechanism, which calculates attention maps for lower level features guided by higher level features. This mechanism emphasizes active semantic regions for low-level distortions, thereby improving performance. CFANet can be used for both Full-Reference (FR) and No-Reference (NR) IQA. We use ResNet50 as its backbone and demonstrate that CFANet achieves better or competitive performance on most public FR and NR benchmarks compared with state-of-the-art methods based on vision transformers, while being much more efficient (with only ${\sim}13\%$ FLOPS of the current best FR method). Codes are released at \url{https://github.com/chaofengc/IQA-PyTorch}.

I. INTRODUCTION

IQA seeks perceptual quality estimates aligned with human vision, but existing multi-scale methods often overlook interactions between semantic importance and local distortions. TOPIQ addresses this with a top-down CFANet that propagates high-level semantics toward low-level features and reports competitive performance with lower computation.

  • IQA estimates perceptual image quality for applications including acquisition, compression, restoration, editing, and generation.
  • Humans may prefer an image with clearer semantically important regions despite greater background noise, whereas simple multi-scale methods can make inconsistent judgments.
  • Existing IQA methods use parallel, bottom-up, or top-down strategies to extract and employ multi-scale features.
  • TOPIQ introduces CFANet, a top-down network that progressively propagates high-level semantic information from coarse to fine scales.
  • Cross-scale attention uses high-level features to select important low-level distortion features, while gated local pooling filters redundant information to reduce computational cost.
  • Approximately 13% of the FLOPS of the best existing FR method is required by CFANet with a ResNet50 backbone while achieving competitive performance.

II. RELATED WORKS

Prior IQA methods include hand-crafted and learned FR or NR approaches, but multi-scale representations can misalign with human judgments when semantic and distortion information are not jointly modeled. The paper motivates semantic reweighting and a top-down framework to address this issue.

  • FR-IQA compares reference and distorted images, whereas NR-IQA estimates quality without a pristine reference and is more challenging.
  • Vision-transformer methods such as IQT and AHIQ perform strongly, but transformer computation grows quadratically with larger image sizes.
  • A. Observations and Motivation: MS-SSIM and LPIPS can be distracted by large background regions, preferring Image B while humans prefer Image A because of clearer birds.
  • A. Observations and Motivation: MS-SSIM has difficulty extracting semantic features, while LPIPS can lose distortion differences, making high-level feature differences alone insufficient.
  • A. Observations and Motivation: All semantic-weight-map layers improve LPIPS+, with the third layer selected empirically for the strongest improvement.
  • A. Observations and Motivation: LPIPS+ reweights spatial quality maps using reference-image feature activations as rough semantic weights without requiring additional training.
  • A. Observations and Motivation: The resulting top-down framework avoids tedious, non-generalizable manual selection of multi-scale features across datasets.

B. Architecture of Coarse-to-Fine Attention Network

CFANet is a top-down coarse-to-fine IQA network that progressively propagates semantic information from high-level to low-level features. Its gated local pooling selects distortion-related features, aligns scales, and reduces computational cost before attention processing.

  • Architecture of Coarse-to-Fine Attention Network: CFANet applies a top-down architecture to multi-scale features for both FR and NR IQA tasks.The FR pipeline extracts multi-scale features, pools them to a common size, applies attention, and regresses a quality score.
  • Architecture of Coarse-to-Fine Attention Network: The backbone extracts five feature scales with ResNet50, whose lower-level maps have larger spatial dimensions than higher-level maps.For ResNet50, the feature hierarchy uses n = 5, with feature-map height and width decreasing by scale.
  • 1) Gated Local Pooling: Gated local pooling first selects distortion-related responses with a gated convolution, then average-pools and linearly reduces them to common-size features Gi.Naive average pooling can fuse local features and reduce distortion distinguishability; the gated operation is intended to filter redundant features.
  • 1) Gated Local Pooling: For FR inputs, the gating mask is computed from distortion-reference feature differences, while the NR task uses a separate gated-convolution formulation.The FR mask uses paired distortion and reference features, and the mask is constrained to a single channel for efficiency.
  • 1) Gated Local Pooling: The resulting features Gi share the highest-level spatial size and feed the subsequent attention blocks.The reduced feature dimension is denoted D, and the learned masks are reported to filter redundant features.

2) Attention Modules:

The attention modules enhance features within and across scales. Self-attention aggregates information across positions, while cross-scale attention uses higher-level features to guide lower-level distortion selection.

  • Attention Modules: Scaled dot-product attention computes similarities between query and key vectors and returns a weighted sum of value vectors.The formulation supports feature aggregation using query, key, and value representations.
  • Attention Modules: The architecture uses five-scale features in CFANet-FR, consistent with the scale count used in methods such as MS-SSIM and LPIPS.The five-scale design is shown in the architecture overview.
  • Self-attention: Self-attention enhances each scale by aggregating information from other spatial positions after gated local pooling.This compensates for the limited receptive field of low-level features.
  • Cross-scale Attention: The proposed cross-scale attention addresses the missing interaction between high-level semantic features and low-level distortion features.Unlike simple concatenation followed by transformer layers, CSA explicitly models relationships between semantic and distortion representations.
  • Cross-scale Attention: CSA uses features from different scales to generate attention queries, keys, and values, selecting semantically relevant distortions in lower-level features.The query acts as a guide when computing the output, and the final cross-scale features are formed progressively with residual fusion.

3) Unified position encoding:

CFANet adds a shared learnable position encoding to aligned multi-scale features so CSA can match features across scales using spatial information.

  • 3) Unified position encoding: After GLP gives all multi-scale features the same shape, CFANet adds the same learnable position encoding to every Gi.The unified encoding provides position information as an additional clue for cross-scale feature queries.

4) Score Regression:

CFANet pools final semantic-aware distortion features and regresses quality according to each dataset’s label format. The training objective therefore varies across MOS, score-distribution, and preference-labeled datasets.

  • 4) Score Regression: Final quality scores are obtained from the final features through score prediction after semantic-aware feature processing.The final prediction pipeline includes feature pooling and score regression.
  • 4) Score Regression: MOS-labeled datasets normalize scores to [0, 1] and use mean squared error loss.The loss is selected for datasets with normalized MOS targets.
  • 4) Score Regression: Score-distribution datasets such as AVA use predicted distributions trained with Earth Mover’s Distance loss.The model predicts a distribution rather than a single MOS value for these datasets.
  • 4) Score Regression: Preference-labeled 2AFC datasets compare perceptual error scores for two distorted images using a Bradley-Terry sigmoid model.The resulting preference probability is trained with mean squared error loss.

IV. EXPERIMENTS

Experiments evaluate CFANet across public FR and NR benchmarks using standard correlation metrics and visualize its coarse-to-fine attention behavior. GLP filters distortion-related information, while CSA progressively transfers semantic focus toward relevant local regions.

  • Experimental setup: Experiments cover public FR and NR benchmarks, using official splits when available or repeated random splits otherwise.FR splits are based on reference images to avoid content overlap.
  • Experimental setup: PLCC measures linear correlation with ground-truth scores, while SRCC measures rank correlation after fitting a four-parameter logistic function for PLCC.The logistic parameters are fitted by least squares before calculating PLCC.
  • Experimental setup: CFANet uses an ImageNet-pretrained ResNet50 backbone with fixed batch-normalization layers, quality-preserving augmentation, and AdamW optimization.Learning rates are set differently for FR and NR datasets.
  • Attention visualization: GLP masks filter redundant information and reduce feature-map spatial size, while CSA maps propagate semantics from coarse to fine scales.Figure 7 visualizes GLP masks and CSA attention weights for multiple distortion types.
  • Attention visualization: CSA attention progressively focuses on semantically active regions, such as birds rather than large backgrounds, across different distortion types.The visualization supports CFANet’s human-consistent judgment in the illustrated bird example and robustness across shown distortions.

C. Comparison with FR Methods

FR evaluations compare CFANet with traditional and deep learning methods on both conventional and large-scale benchmarks. CFANet matches strong methods on easier datasets, surpasses prior methods on larger datasets, and generalizes robustly with a simple backbone.

  • Benchmark comparison: FR comparisons include intra-dataset and cross-dataset evaluations on LIVE, CSIQ, TID2013, PieAPP, and PIPAL.The experiments also compare against LPIPS under the same setup.
  • Efficiency comparison: Figure 8 reports the best FR performance for CFANet at approximately 13% of AHIQ’s FLOPS.The figure compares computational cost and PLCC, with parameter counts represented by circle radius.
  • Intra-dataset results: CFANet performs as well as AHIQ on LIVE, CSIQ, and TID2013, although results on these datasets can vary substantially across splits.These conventional benchmarks contain relatively few synthetic distortion types.
  • Intra-dataset results: CFANet outperforms previous methods, including the transformer-based AHIQ, on the larger PieAPP and PIPAL datasets.This result is obtained with a simple ResNet50 backbone.
  • Cross-dataset results: CFANet shows stronger cross-dataset robustness with fewer parameters, while AHIQ’s performance gaps are larger across datasets.The authors relate this comparison to overfitting concerns on the 29k-pair PIPAL dataset.

3) Comparison of computation complexity:

The computational comparison positions CFANet as an efficient FR method, while additional BAPPS and NR results assess preference consistency and reference-free quality prediction. Across these evaluations, the reported results emphasize performance, efficiency, and generalization.

  • Computational complexity: CFANet achieves the best reported FR performance with approximately 13% of AHIQ’s FLOPS and around one-seventh of its parameters.It is also reported as more efficient than LPIPS, while simpler methods remain less accurate.
  • BAPPS evaluation: BAPPS uses binary preference labels, so its validation protocol reports consistency with human judgments rather than PLCC or SRCC.The score measures binary preference judgments rather than exact probability values.
  • BAPPS evaluation: CFANet achieves the best performance on both synthetic and real algorithmic distortions in the BAPPS 2AFC test set.Its results are especially close to human judgments on synthetic distortions.
  • NR results: On authentic-distortion NR datasets, CFANet with ResNet50 outperforms existing CNN methods and MUSIQ in both PLCC and SRCC.The reported datasets are CLIVE, KonIQ-10k, and SPAQ.
  • NR results: On the official KonIQ-10k split, CFANet surpasses KonCepth512 and MUSIQ despite using a simple ResNet50 backbone.The authors present this as evidence of effectiveness and efficiency.

3) Results for aesthetic quality estimation:

Aesthetic-quality experiments show that global semantics matter more than local textures, while CFANet remains competitive and its Swin variant surpasses prior state-of-the-art methods. Ablations identify CSA as the strongest component in cross-dataset FR evaluation.

  • Aesthetic quality estimation: Aesthetic assessment emphasizes global feeling, making global semantics more important than local textures.AVA is the primary benchmark for aesthetic evaluation, and FLIVE overlaps with AVA by approximately 23%.
  • Aesthetic quality estimation: ThemeAware improves results through extra theme labels, while KD benefits from distilling semantic knowledge from multiple classification backbones.These methods provide stronger global-semantic modeling than CFANet’s local-distortion-oriented design.
  • Aesthetic quality estimation: CFANet with ResNet50 achieves competitive results on technical and aesthetic quality benchmarks despite being designed primarily to extract local distortions.The authors interpret this as evidence that it preserves global semantic information.
  • Aesthetic quality estimation: CFANet-Swin outperforms previous state-of-the-art methods on both FLIVE and AVA.The Swin backbone is described as relatively cheaper than the original alternatives considered.
  • Ablation of the proposed components: In cross-dataset ablations, GLP, SA, CSA, and positional embedding each improve the baseline, with CSA producing the largest improvement.The full CFANet yields significant gains over the multiscale linear-regression baseline trained on KADID-10k and tested on CSIQ and TID2013.

2) Ablation with different variants:

Ablations show that CFANet performs best when its gated local pooling, cross-scale attention, and multi-scale semantic guidance are retained. Backbone comparisons further indicate that stronger backbones generally improve FR and NR performance, supporting TOPIQ’s top-down design.

  • Ablation with different variants:: The complete CFANet variant outperforms replacing GLP, replacing CSA, or directly using the top-layer feature to guide the lowest layer.The reported ordering is a⃝ > b⃝ > c⃝, where the variants replace GLP, replace CSA, and bypass multi-scale guidance, respectively.
  • Ablation with different variants:: GLP is slightly superior to resize because it more accurately and selectively captures local distortion information.
  • Ablation with different variants:: CSA outperforms convolution fusion, likely because attention more effectively aggregates features from the entire image.
  • Overall conclusion:: The ablation findings support the importance of combining semantics with distortions through the proposed top-down framework for IQA.
  • Backbone comparison:: Stronger backbones generally provide better performance on both FR and NR benchmarks across VGG19, ResNet50, and Swin Transformer.The comparison is reported in Fig. 9 for FR benchmarks and Fig. 10 for NR benchmarks.
Loading 2308.03060v1…