Source-linked AI summary
MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment
Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, Yujiu Yang
TL;DR
Existing NR-IQA methods have limited accuracy on GAN-based distortions, where restored images can contain lifelike textures that challenge learned quality models. MANIQA uses ViT features, channel- and spatial-dimension attention, and patch-weighted prediction, achieving strong results across standard datasets and first place in the NTIRE 2022 NR-IQA challenge. The paper notes that GAN-based distortion types still require more precise definition.
Problem
Existing NR-IQA methods are limited when predicting perceptual quality for GAN-based distorted images, especially because lifelike unreal textures can diverge from machine-learned image priors.
Method
MANIQA extracts ViT features, applies transposed attention across channels and scale Swin Transformer processing across local spatial information, then predicts patch-weighted quality scores.
Results
MANIQA outperforms state-of-the-art methods by a large margin on four standard datasets and ranks first in the NTIRE 2022 NR-IQA challenge.
Takeaways & Limitations
Multi-dimension interaction across channel and spatial dimensions is presented as vital for perceptual quality assessment, including GAN-based distortion.
Takeaways & Limitations
Further progress requires defining particular GAN-based distortion types, including pleasing and unpleasing unreal textures.
Abstract
from arXiv · showhide
No-Reference Image Quality Assessment (NR-IQA) aims to assess the perceptual quality of images in accordance with human subjective perception. Unfortunately, existing NR-IQA methods are far from meeting the needs of predicting accurate quality scores on GAN-based distortion images. To this end, we propose Multi-dimension Attention Network for no-reference Image Quality Assessment (MANIQA) to improve the performance on GAN-based distortion. We firstly extract features via ViT, then to strengthen global and local interactions, we propose the Transposed Attention Block (TAB) and the Scale Swin Transformer Block (SSTB). These two modules apply attention mechanisms across the channel and spatial dimension, respectively. In this multi-dimensional manner, the modules cooperatively increase the interaction among different regions of images globally and locally. Finally, a dual branch structure for patch-weighted quality prediction is applied to predict the final score depending on the weight of each patch's score. Experimental results demonstrate that MANIQA outperforms state-of-the-art methods on four standard datasets (LIVE, TID2013, CSIQ, and KADID-10K) by a large margin. Besides, our method ranked first place in the final testing phase of the NTIRE 2022 Perceptual Image Quality Assessment Challenge Track 2: No-Reference. Codes and models are available at https://github.com/IIGROUP/MANIQA.
1. Introduction
NR-IQA methods show limitations on GAN-based restored images, motivating MANIQA, which combines multi-dimension attention with patch-weighted prediction. Experiments report large-margin gains across four datasets and first place in the NTIRE 2022 NR-IQA challenge.
- Motivation: GAN-based image restoration exposes limitations in general-purpose NR-IQA methods despite their effectiveness on synthetic and authentic distortions.The mismatch arises because restored images can contain lifelike textures that humans accept but machine IQA models may treat as inconsistent with learned image priors.
- Proposed approach: MANIQA combines ViT feature extraction, transposed attention, scale Swin Transformer processing, and dual-branch patch-weighted quality prediction.Its attention modules target channel-wise and local interactions, while the prediction structure estimates each patch’s score and weight.
- Results: MANIQA outperforms state-of-the-art methods by a large margin on PIPAL, TID2013, LIVE, CSIQ, and KADID-10K.The experiments are presented as evidence of generalization across the GAN-focused PIPAL dataset and four standard datasets.
- Results: MANIQA’s ensemble method ranked first in the NTIRE 2022 Perceptual Image Quality Assessment Challenge Track 2: No-Reference.The paper connects this ranking to the model’s effectiveness for GAN-based distortion.
2. Related Work
Prior NR-IQA work includes NSS-based and learning-based methods, but GAN restoration introduces lifelike unreal textures that challenge quality assessment. MANIQA addresses this setting with transformer-based feature interaction across image channels and spatial regions.
- No-Reference Image Quality Assessment: NR-IQA methods estimate quality from low-quality images without directly using reference images.Existing approaches are mainly divided into natural scene statistics-based and learning-based metrics.
- IQA for GAN-based IR Algorithm: GAN-based restoration can produce lifelike unreal textures that are difficult for humans to distinguish but easier for machines to perceive.This property motivates dedicated IQA methods for GAN-based image restoration.
- Transformer for IQA: CNN locality bias and limited non-local feature capture motivate transformer architectures for exploiting information across image regions.The related-work discussion also notes that shared convolutional weights restrict handling of complex feature combinations.
- Attention Mechanism: Attention mechanisms have been applied in computer vision and NR-IQA, including spatial and transposed attention variants.Restormer interchanges spatial and channel dimensions to reduce computation and implicitly fuse global features.
- Attention Mechanism: MANIQA uses transposed attention to capture feature dependencies and obtain weighted channel features for image-quality prediction.Its design is presented as an extension of attention-based approaches for combining channel and spatial information.
3. Proposed Method
MANIQA combines ViT features with channel- and spatial-interaction modules, then predicts image quality from weighted patch scores. Its pipeline uses TAB for channel interactions, SSTB for local interactions, and dual-branch aggregation.
- Overall Pipeline: MANIQA extracts and concatenates features from four ViT layers before applying its attention and prediction modules.The selected layers are 7, 8, 9, and 10.
- Transposed Attention Block: TAB applies self-attention across channels to encode global cross-channel context and reweight extracted features.It forms a transposed attention map from query and key projections, while the value projection carries the attended features.
- Scale Swin Transformer Block: SSTB strengthens local patch interactions using Swin Transformer layers, a convolutional layer, and a residual connection.The convolutional layer is applied before the residual connection, and the scale factor controls the transformed residual.
- Patch-weighted Quality Prediction: The dual-branch module predicts a score and weight for each patch, then aggregates weighted patch scores into the image-level quality score.The score is computed as a normalized weighted sum: sum of ω_i × s_i divided by sum of ω_i.
4. Experiments
MANIQA is evaluated on PIPAL and four standard IQA datasets using PLCC and SROCC, with comparisons, cross-dataset tests, challenge results, and ablations of its components. The experiments examine both overall performance and the contributions of channel attention, local patch interactions, and patch-weighted prediction.
- Datasets: PIPAL contains 29k images with restoration, enhancement, and traditional distortions, including more than 1.13 million human judgments.
- Evaluation Criteria: PLCC and SROCC evaluate agreement between predicted and ground-truth quality scores, with higher values indicating better performance.
- Results: MANIQA outperforms existing methods by a significant margin across the evaluated datasets, including PIPAL and the standard IQA benchmarks.
- Challenge Evaluation: MANIQA’s ensemble ranks first in the NTIRE 2022 No-Reference Perceptual Image Quality Assessment Challenge.
- Ablation Study: The Transposed Attention Block adds 0.05 improvements in both SROCC and PLCC when combined with the other modules.
- Ablation Study: The Scale Swin Transformer Block improves local patch interactions, while the dual branch predicts patch-weighted scores and contributes additional performance gains.
- Visualization: Patch weight maps emphasize salient subjects, while final weighted maps combine patch importance with visual quality because salient regions are not always the highest-quality regions.
5. Conclusion
MANIQA uses multi-dimension attention to improve perceptual image quality assessment, with experiments showing strong performance on standard databases and GAN-based distortions. The authors identify distinguishing different types of GAN-based distortion as an area for further improvement.
- Multi-dimension interaction across channel and spatial dimensions is vital for perceptual quality assessment.The modules cooperatively increase interaction among image regions globally and locally.
- MANIQA demonstrates outstanding performance compared with existing methods on four standard databases.
- MANIQA achieves the best performance for GAN-based distortion on PIPAL in the NTIRE 2022 NR track.
- Further progress requires distinguishing GAN-based distortions such as pleasing and unpleasing unreal textures.