Source-linked AI summary
No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency
S. Alireza Golestaneh, Saba Dadsetan, Kris M. Kitani
TL;DR
NR-IQA must predict perceptual quality without a pristine reference, while existing models can underuse relative rankings and become sensitive to equivariant transformations. The paper proposes a hybrid CNN-Transformer model with relative ranking and self-consistency losses, and reports strong performance across seven synthetic and authentic IQA datasets. Its failure analysis points to the limited information in image-level quality scores and the need for richer descriptions of subjects’ reasoning.
Problem
NR-IQA seeks accurate perceptual quality prediction without a reference image, but existing models mainly use image-level MOS/DMOS and can produce inconsistent predictions under transformations.
Method
TReS combines CNN local features, Transformer self-attention for non-local features, relative ranking within batches, and self-consistency between original and transformed images.
Results
The method performs well across seven benchmark IQA datasets covering both synthetically and authentically distorted images.
Takeaways & Limitations
Combining local and non-local representations with ranking and transformation-consistency objectives provides a unified approach for robust NR-IQA evaluation.
Takeaways & Limitations
Failure cases arise when images have close ground-truth scores but the model predicts different scores; richer subject reasoning descriptions are identified as a future need.
Abstract
from arXiv · showhide
The goal of No-Reference Image Quality Assessment (NR-IQA) is to estimate the perceptual image quality in accordance with subjective evaluations, it is a complex and unsolved problem due to the absence of the pristine reference image. In this paper, we propose a novel model to address the NR-IQA task by leveraging a hybrid approach that benefits from Convolutional Neural Networks (CNNs) and self-attention mechanism in Transformers to extract both local and non-local features from the input image. We capture local structure information of the image via CNNs, then to circumvent the locality bias among the extracted CNNs features and obtain a non-local representation of the image, we utilize Transformers on the extracted features where we model them as a sequential input to the Transformer model. Furthermore, to improve the monotonicity correlation between the subjective and objective scores, we utilize the relative distance information among the images within each batch and enforce the relative ranking among them. Last but not least, we observe that the performance of NR-IQA models degrades when we apply equivariant transformations (e.g. horizontal flipping) to the inputs. Therefore, we propose a method that leverages self-consistency as a source of self-supervision to improve the robustness of NRIQA models. Specifically, we enforce self-consistency between the outputs of our quality assessment model for each image and its transformation (horizontally flipped) to utilize the rich self-supervisory information and reduce the uncertainty of the model. To demonstrate the effectiveness of our work, we evaluate it on seven standard IQA datasets (both synthetic and authentic) and show that our model achieves state-of-the-art results on various datasets.
1. Introduction
NR-IQA seeks to predict perceptual quality without a pristine reference, a difficult problem spanning synthetic and authentic distortions. The paper proposes TReS, combining CNNs, Transformers, relative ranking, and self-consistency, and evaluates it across seven benchmark datasets.
- Motivation: NR-IQA estimates perceptual image quality when a reference image is unavailable, supporting applications where robust quality prediction matters.The paper motivates NR-IQA for computer vision, social media, streaming, and safety-relevant systems.
- Motivation: Synthetic distortion datasets have limited content and distortion diversity, while authentic distortions involve complex real-world degradations and unclear quality cues.In-the-wild datasets address conditions such as poor lighting, sensor limitations, lens imperfections, and amateur manipulation.
- Motivation: Existing deep learning IQA models mainly use MOS/DMOS in regression or classification, leaving relative ranking between images underused.The proposed ranking objective uses relative distance information within each batch.
- Proposed approach: CNNs and Transformer self-attention jointly model local and non-local image information, fusing multi-scale features to predict quality.CNN features from different layers are modeled as a sequence by the Transformer before local and non-local representations are fused.
- Proposed approach: Self-consistency training enforces similar outputs for an image and its equivariantly transformed version, targeting sensitivity to horizontal flipping.The method uses transformed inputs as self-supervision to improve robustness and reduce prediction sensitivity.
- Evaluation: Extensive experiments on seven benchmark datasets covering authentic and synthetic distortions show that the proposed method performs well across datasets.The supplied introduction identifies this multi-dataset evaluation as evidence for the method’s effectiveness.
2. Related Work
Prior NR-IQA research includes handcrafted NSS methods, learning-based and CNN-based deep models, ranking approaches, and emerging Transformer architectures. The paper positions its method against limitations in locality, ranking supervision, and robustness to simple image transformations.
- NR-IQA methods: Early general-purpose NR-IQA methods used natural scene statistics or learned mappings from handcrafted features to subjective quality scores.NSS methods assume regularities in natural images that distortions disrupt, while learning-based methods map extracted features to ratings.
- Deep learning for NR-IQA: Deep learning NR-IQA methods commonly leveraged CNN features, with some incorporating distortion type and subjective quality in multi-task training.The related work describes CNN features pretrained on ImageNet and multi-task approaches using quality scores alongside distortion information.
- Transformers for NR-IQA: CNNs capture local image structure but may miss non-local information because of locality bias, motivating Transformer-based modeling of long-range feature dependencies.The proposed related-work framing applies self-attention to dependencies among multi-scale CNN features.
- Transformers for NR-IQA: Transformer applications were expanding in computer vision, but their application to NR-IQA was described as unexplored in the cited related work.The paper connects this gap to attention mechanisms that aggregate information across an input sequence.
- Learning to rank for NR-IQA: Learning-to-rank NR-IQA methods use discrete, continuous, or binary ranking information, with some binary approaches requiring reference images.The related work distinguishes ranking supervision based on distortion levels, MOS statistics, or full-reference metrics.
- Generalization and robustness: Deep IQA models can generalize poorly across datasets and still degrade under horizontal flipping or rotation despite augmentation.The paper notes domain, content, and subjective-score shifts, while common augmentation may not ensure transformation-robust predictions.
3. Proposed Method
TReS combines CNN-derived local features with Transformer self-attention to model non-local relationships, then fuses both representations for quality prediction. It additionally uses batchwise relative ranking and self-consistency objectives to improve quality-score learning and robustness to image transformations.
- Feature extraction and attention: TReS extracts multi-scale CNN features and processes them with Transformer encoders to obtain local and non-local image representations.CNN features from different blocks are modeled as a sequence; self-attention captures relationships beyond local neighbors.
- Feature extraction and attention: Positional encoding supplies feature-position information while Transformer self-attention models dependencies across features from different CNN layers.The encoder uses multi-head self-attention, residual connections, layer normalization, and feed-forward layers.
- Feature fusion and quality prediction: Fully connected fusion layers combine convolutional and self-attention features to predict perceptual image quality, alongside a regression loss against subjective scores.The predicted score q_i is compared with its corresponding ground-truth subjective score s_i.
- Relative ranking: The relative-ranking loss enforces ordering among extreme-quality images within each batch using triplet constraints and adaptive margins derived from subjective scores.It encourages nearest scores among the highest- and lowest-quality pairs while separating opposite-quality extremes.
- Self-consistency: The self-consistency loss matches CNN and Transformer outputs for an image and its horizontal flip, while also aligning relative-ranking losses across transformed batches.This supplies self-supervision without additional labels or external supervision and targets more robust predictions.
4. Experiments
Experiments evaluate the proposed model on seven synthetic and authentic IQA datasets using PLCC and SROCC, with repeated random splits and patch-based training. The model shows strong overall and cross-dataset performance, interpretable feature behavior, competitive smaller-backbone results, and identifiable failure cases.
- Datasets and evaluation: PLCC and SROCC measure performance, with higher values indicating better agreement with subjective quality scores.For PLCC, logistic regression is applied first to remove nonlinear rating effects from human visual observation.
- Overall performance: The model outperforms existing methods on weighted average PLCC and SROCC across all seven datasets and achieves significant gains on LIVE-FB and KADID.Weighted averages use dataset sizes as performance weights, while smaller datasets show competitive results.
- Generalization: Cross-dataset testing without fine-tuning shows that the proposed method outperforms competing algorithms on four of six evaluated datasets.Synthetic evaluations use four distortion types common to LIVE, CSIQ, and TID2013.
- Feature analysis: Nearest-neighbor retrieval groups images with similar perceptual artifacts and subjective scores, while spatial maps highlight poor-quality regions.Examples retrieve motion-blurred or low-light images, and bright map regions indicate poor quality.
- Ablation and failure analysis: Smaller backbones remain competitive on large datasets, but failure cases occur when images have close subjective scores and the model predicts different qualities.The authors discuss limited score-only supervision and the absence of subject reasoning descriptions as possible explanations.
5. Conclusion
The paper presents an NR-IQA method combining CNNs, Transformers, relative ranking, and self-consistency, and reports strong performance across synthetic and authentic IQA datasets.
- The method combines CNN and Transformer features to represent both local and non-local information in input images.
- A relative ranking loss incorporates ranking information among images alongside quality assessment.
- A self-consistency loss improves the robustness of predictions under transformed inputs.
- Experiments evaluate the method on several IQA datasets containing synthetic and authentic distortions.
FQAs
The FAQs distinguish self-consistency from ensembling and augmentation, explain why ranking focuses on extreme samples, and show that CNN and Transformer features capture different information.
- Self-consistency trains one network to produce similar outputs for an image and its transformed version, unlike ensembling multiple models.
- Self-consistency differs from augmentation by explicitly enforcing matching outputs for transformed versions while retaining augmentation during training.
- Applying ranking loss to all batch samples did not significantly improve results over using only the highest- and lowest-quality samples.
- Using more samples for ranking increases gradient back-propagation and slows training.
- CNN and Transformer features capture different information for individual images and across distortion types.