Source-linked AI summary
Perceptual Image Quality Assessment with Transformers
Manri Cheon, Sung-Jun Yoon, Byungyeon Kang, Junwoo Lee
TL;DR
Perceptual IQA needs accurate automatic comparison despite limitations of subjective assessment and conventional metrics. IQT uses CNN-derived representations with a transformer encoder-decoder to compare reference and distorted images. It performed strongly across standard and generative-model IQA datasets and ranked first among 13 NTIRE 2021 participants.
Problem
Subjective assessment is costly, while conventional metrics can inadequately predict perceptual quality, especially for images with generative restoration artifacts.
Method
IQT extracts perceptual representations from reference and distorted images with a CNN and compares them using a transformer encoder-decoder with difference information.
Results
IQT showed outstanding performance on three standard IQA databases, best performance on PIPAL, and first place among 13 NTIRE 2021 participants.
Takeaways & Limitations
Transformer-based IQA can achieve high performance for perceptual full-reference quality assessment across standard and deep-learning-based distortions.
Takeaways & Limitations
The approach needs evaluation across more diverse resolutions and distortion types, and a no-reference perceptual metric remains desirable for real-world scenarios.
Abstract
from arXiv · showhide
In this paper, we propose an image quality transformer (IQT) that successfully applies a transformer architecture to a perceptual full-reference image quality assessment (IQA) task. Perceptual representation becomes more important in image quality assessment. In this context, we extract the perceptual feature representations from each of input images using a convolutional neural network (CNN) backbone. The extracted feature maps are fed into the transformer encoder and decoder in order to compare a reference and distorted images. Following an approach of the transformer-based vision models, we use extra learnable quality embedding and position embedding. The output of the transformer is passed to a prediction head in order to predict a final quality score. The experimental results show that our proposed model has an outstanding performance for the standard IQA datasets. For a large-scale IQA dataset containing output images of generative model, our model also shows the promising results. The proposed IQT was ranked first among 13 participants in the NTIRE 2021 perceptual image quality assessment challenge. Our work will be an opportunity to further expand the approach for the perceptual IQA task.
1. Introduction
Perceptual IQA is important but difficult because subjective assessment is costly and existing metrics can miss unrealistic artifacts from generative restoration. The paper proposes IQT, a transformer-based full-reference approach for perceptual quality assessment.
- Subjective quality assessment is accurate but time-consuming and expensive, motivating objective metrics for automatic perceived-quality prediction.
- Generative restoration models can produce unrealistic artifacts that PSNR, SSIM, and conventional metrics insufficiently predict.
- Transformer-based IQA has received limited study, especially for full-reference perceptual similarity between reference and distorted images.
- IQT uses a CNN backbone and transformer encoder-decoder to predict perceptual quality from reference and distorted images.
- IQT ranked first among 13 participants in the NTIRE 2021 perceptual image quality assessment challenge.
2. Related Work
Related work spans full-reference, reduced-reference, and no-reference IQA, with conventional and CNN-based metrics addressing different aspects of perceived quality. Transformer-based IQA has shown promise, but IQT extends it to perceptual full-reference comparison using reference, distorted, and difference information.
- Image Quality Assessment: IQA methods are categorized as full-reference, reduced-reference, or no-reference according to available reference information.
- Image Quality Assessment: PSNR is convenient for optimization but often predicts perceived visual quality poorly, while SSIM and learned deep features address structural and perceptual information.
- Image Quality Assessment: PLCC, SRCC, and KRCC evaluate accuracy, monotonicity, and consistency of quality predictions.
- Vision Transformer: Transformers aggregate global sequence information through self-attention, supporting whole-image perceptual quality measurement.
- Vision Transformer based IQA: TRIQ applies a transformer encoder with adaptive positional embedding to blind IQA across images with different resolutions.
- Vision Transformer based IQA: IQT adapts transformer IQA to full-reference assessment by using reference and difference information, with Siamese CNN feature extraction from both images.
3. Proposed Method
IQT combines CNN feature extraction with a transformer encoder-decoder to compare reference and distorted images for perceptual quality prediction. The method uses difference features, learnable quality and position embeddings, and an MLP prediction head.
- Architecture: IQT comprises a CNN feature-extraction backbone, transformer encoder-decoder, and prediction head for perceived quality prediction.CNN feature maps are projected to fixed-size vectors, flattened, and augmented with a trainable quality embedding.
- Feature Extraction: Inception-ResNet-V2 extracts and concatenates feature maps from six intermediate layers into a 6 × 320-channel representation.The ImageNet-pretrained backbone is frozen, and the six feature maps share spatial dimensions.
- Transformer Encoder: The transformer receives reference, distorted, and difference feature maps, with the difference defined as fdiff = fref − fdist.The difference map is channel-reduced with a 1 × 1 convolution and flattened into N = H × W patches.
- Transformer Encoder: The encoder adds trainable position embeddings and a quality embedding before processing the difference-feature sequence.The encoder output preserves the input embedding size and is passed to the decoder as encoder output.
- Transformer Decoder: The decoder processes reference feature embeddings while using encoder outputs as key-values in its second multi-head-attention layer.Reference embeddings also receive quality and position embeddings before decoder processing.
- Prediction Head: The first decoder output vector is passed through two fully connected layers to predict a single quality score.The first layer uses ReLU, while the second has one output channel.
4. Experiments
Experiments evaluate IQT across standard and generative-model image-quality datasets, analyze attention behavior, and test architectural choices. IQT performs competitively across benchmarks, while feature-level difference information improves prediction and PIPAL training exposes robustness limits.
- Datasets: IQT is trained on KADID-10k and evaluated on LIVE, CSIQ, TID2013, and PIPAL using image-quality datasets with traditional and generative-model distortions.The standard datasets provide full-reference IQA baselines, while PIPAL contains output images from generative models.
- Standard IQA results: For LIVE and TID2013, IQT achieves the best SRCC, ranks in the top three across all benchmarks for SRCC and KRCC, and outperforms recent deep-learning methods in most cases.The predicted scores are compared with ground-truth MOS values in scatter plots for all four datasets.
- PIPAL results: On PIPAL, IQT achieves the best performance among the compared metrics and shows promise for assessing generative-model output images.The model’s predicted quality ordering is similar to MOS for example validation images, although some images remain difficult to assess accurately.
- Attention visualization: Attention maps are spatially localized or spread across the whole image depending on the image and distortion type.The maps average attention weights from encoder and decoder multi-head attention layers, indicating both global and localized focus.
- Ablation studies: Feature-level difference information improves performance over image-level differences and over directly using only reference and distorted feature embeddings.The selected design uses difference features in the encoder and reference features in the decoder.
- Limitations: PIPAL-trained IQT performs strongly on PIPAL but shows lower performance on the three standard IQA datasets, indicating overfitting risk and limited robustness to other distortion types.The paper identifies improving robustness when training on PIPAL as future work.
5. Conclusion
IQT applies a transformer encoder-decoder to perceptual full-reference image quality assessment and performs strongly across standard and deep-learning-based distortion datasets. Its scope still leaves diverse resolutions, distortion types, and no-reference assessment for future work.
- IQT uses a transformer encoder-decoder architecture for perceptual image quality assessment.
- The model demonstrates outstanding performance on three standard IQA databases against existing methods.
- IQT achieves the best performance on an IQA dataset containing deep learning-based distorted images.
- Further work should examine more diverse resolutions and distortion types and develop a no-reference perceptual quality metric for real-world scenarios.