Source-linked AI summary
Transformer for Image Quality Assessment
Junyong You, Jari Korhonen
TL;DR
The paper asks how Transformer architectures can be applied to image quality assessment without overlooking resolution effects. It proposes a CNN–Transformer hybrid with adaptive positional embeddings and reports outstanding performance across several image-quality databases, while cross-resolution and cross-database settings remain challenging.
Problem
Image quality assessment is mainly CNN-based, but image resizing can change perceived quality, creating a need for models that handle varying resolutions.
Method
TRIQ places a shallow Transformer encoder on CNN-derived feature maps, using adaptive positional embeddings and resolution-dependent max-pooling.
Results
TRIQ significantly outperforms other IQA models on the combined database and shows promising performance across databases, contents, and resolutions.
Takeaways & Limitations
TRIQ demonstrates a promising way to combine CNN inductive capability with Transformer attention for image quality assessment.
Takeaways & Limitations
Cross-resolution and cross-database evaluation negatively affects IQA performance, motivating more generic models for future work.
Abstract
from arXiv · showhide
Transformer has become the new standard method in natural language processing (NLP), and it also attracts research interests in computer vision area. In this paper we investigate the application of Transformer in Image Quality (TRIQ) assessment. Following the original Transformer encoder employed in Vision Transformer (ViT), we propose an architecture of using a shallow Transformer encoder on the top of a feature map extracted by convolution neural networks (CNN). Adaptive positional embedding is employed in the Transformer encoder to handle images with arbitrary resolutions. Different settings of Transformer architectures have been investigated on publicly available image quality databases. We have found that the proposed TRIQ architecture achieves outstanding performance. The implementation of TRIQ is published on Github (https://github.com/junyongyou/triq).
1. INTRODUCTION
The paper examines Transformer-based image quality assessment against CNN-driven approaches. It motivates a Transformer encoder for IQA because image quality depends on spatial content and resolution, which resizing can alter.
- Transformer has expanded from NLP into computer vision through architectures such as DETR and ViT.
- Existing deep-learning IQA models mainly use CNN feature extraction followed by an MLP quality predictor.Earlier systems also combined quality predictions from small image patches into one image-level indicator.
- Image quality can depend strongly on spatial saliency distribution, motivating methods that jointly or perceptually model saliency and quality.
- Image resizing should be avoided in IQA because resolution changes can significantly affect perceived quality.The paper therefore considers adapting Transformer encoders to images with different resolutions.
2. TRIQ: TRANSFORMER FOR IMAGE QUALITY ASSESSMENT
TRIQ combines CNN-derived features with a shallow Transformer encoder for image quality prediction. Adaptive positional embeddings and resolution-dependent pooling let the architecture process images with varying resolutions.
- A Conv2D-projection layer maps patches or CNN features to the Transformer model dimension D.With the hybrid approach, the final CNN activation map is projected using kernel size and stride 1.
- TRIQ follows ViT’s hybrid approach by applying a Transformer encoder to a feature map produced by a CNN.The proposed implementation uses ResNet50 as the base network, although other CNN architectures can also be used.
- Learnable positional embeddings are truncated from a length covering the datasets’ maximal resolution to accommodate smaller images.
- Adaptive max-pooling reduces resolution-dependent feature counts before or after projection, producing features of shape [H/(32×P), W/(32×P), D].The pooling size P is determined from the input resolution to limit memory use and support long-range modeling.
- The encoder outputs shape (1+N)×D, and its first vector is passed to an MLP head for quality prediction.The head predicts a five-grade quality distribution with Softmax; cross entropy compares it with the ground-truth distribution.
3. EXPERIMENTS
The experiments evaluate TRIQ against existing IQA models across combined, cross-database, and cross-resolution settings using correlation and error metrics. TRIQ shows promising or outstanding performance, although cross-resolution and cross-database evaluation negatively affects all models.
- Datasets: The combined database includes KonIQ-10k, KonIQ-half-sized, and LIVE-wild, with splits inherited from the original databases.The combined data were formed after calibration suggested that randomly selected image pairs from the databases had similar quality levels.
- Combined evaluation: TRIQ significantly outperforms the other IQA models on the combined testing set, including patching-based and hybrid ViT variants.The authors suggest that the relatively small IQA databases limit the benefit of large-scale pretraining, while CNN-derived features are better suited to image-quality features.
- Cross-database evaluation: TRIQ shows promising performance when trained on the combined data and evaluated on the entire SPAQ database.This tests transfer across databases rather than performance only on the combined test split.
- Separate-database evaluation: TRIQ shows promising performance as a generic IQA model for diverse image contents and resolutions on separate database test sets.The evaluation uses models trained on the combined training set and tests them on KonIQ-10k, KonIQ-half-sized, and LIVE-wild splits.
- Limitations: Cross-resolution and cross-database evaluation negatively affects IQA performance, although TRIQ retains promising potential in this setting.The authors identify developing a more generic IQA model for future work.
4. CONCLUSIONS
The paper proposes TRIQ, an image quality assessment model combining CNN-derived quality features with Transformer-based attention aggregation. It reports outstanding performance on two publicly available large-scale image quality databases and identifies broader adaptability as future work.
- TRIQ combines CNN inductive capability for quality feature derivation with a Transformer encoder for attention-based aggregated representation.
- TRIQ demonstrates outstanding performance on two publicly available large-scale image quality databases compared with other deep learning-driven IQA models.
- Future work targets IQA models adaptable to diverse image contents, resolutions, and distortion types.