Source-linked AI summary
Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild
Avinab Saha, Sandeep Mishra, Alan C. Bovik
TL;DR
Perceptual quality assessment for images in the wild is difficult because distortions interact with image content and affects large-scale image processing and recommendation systems. Re-IQA uses unsupervised Mixture-of-Experts training to learn complementary content and quality representations, then maps them to quality scores with a linear regressor. The method is reported to achieve state-of-the-art performance across multiple IQA databases with real and synthetic distortions.
Problem
NR-IQA for images in the wild must handle complex distortions whose perceived severity depends on image content, with predictions affecting image processing and recommendations.
Method
Re-IQA independently trains two unsupervised encoders for complementary high-level content and low-level quality representations, then maps their frozen features to scores with a linear regressor.
Results
Re-IQA achieves state-of-the-art performance across eight IQA datasets containing real and synthetic distortions.
Takeaways & Limitations
Complementary content and quality-aware features can be learned and combined for competitive NR-IQA performance across evaluated databases.
Takeaways & Limitations
The quality-aware representation learning relies on stated hypotheses about perceptual quality-aware features, including their variation within images and differences across randomly selected images.
Abstract
from arXiv · showhide
Automatic Perceptual Image Quality Assessment is a challenging problem that impacts billions of internet, and social media users daily. To advance research in this field, we propose a Mixture of Experts approach to train two separate encoders to learn high-level content and low-level image quality features in an unsupervised setting. The unique novelty of our approach is its ability to generate low-level representations of image quality that are complementary to high-level features representing image content. We refer to the framework used to train the two encoders as Re-IQA. For Image Quality Assessment in the Wild, we deploy the complementary low and high-level image representations obtained from the Re-IQA framework to train a linear regression model, which is used to map the image representations to the ground truth quality scores, refer Figure 1. Our method achieves state-of-the-art performance on multiple large-scale image quality assessment databases containing both real and synthetic distortions, demonstrating how deep neural networks can be trained in an unsupervised setting to produce perceptually relevant representations. We conclude from our experiments that the low and high-level features obtained are indeed complementary and positively impact the performance of the linear regressor. A public release of all the codes associated with this work will be made available on GitHub.
1. Introduction
NR-IQA for images in the wild must account for complex distortions and content-dependent quality perception while supporting large-scale image-processing and recommendation systems. Re-IQA addresses this by independently learning complementary content and quality representations with unsupervised encoders.
- Motivation: IQA supports visual-quality decisions for billions of shared images, including post-upload resizing, compression, enhancement, and recommendation.These predictions influence how images are stored, processed, displayed, and selected for user feeds or search responses.
- Challenge: NR-IQA in the wild is difficult because distortion perception depends on both distortion combinations and image content.Images with identical JPEG encoding can exhibit visibly different artifacts, illustrating content-dependent distortion perception.
- Approach: Re-IQA uses a Mixture of Experts to independently learn high-level content and low-level quality-aware representations without supervision.The framework is inspired by momentum contrastive learning and targets authentically distorted images.
- Contribution: The learned representations are complementary, and their mixture produces competitive NR-IQA predictions across traditional, CNN, and Transformer-based methods and several databases.The framework combines the two representation types for perceptual quality prediction.
- Contribution: Unsupervised pre-training of a ResNet-50 content encoder produces stronger NR-IQA content representations than supervised ImageNet pre-training.This comparison concerns high-level image-content representations used for the NR-IQA task.
- Contribution: Image Augmentation and Intra-Pair Image Swapping learn quality-aware features by applying dynamic distortions to pristine and authentically distorted images.The dynamic scheme is designed to prevent learning discrete distortion classes and instead enforce perceptually relevant image-quality features.
2. Related Work
Prior NR-IQA methods use handcrafted statistics, pretrained CNNs, or specialized supervised architectures, but some rely on unavailable references or potentially inaccurate distortion labels. Re-IQA instead learns content and quality features independently without distortion-class labels.
- NR-IQA Background: Full-reference IQA methods require undistorted references, limiting their use for images in the wild.No-reference methods avoid requiring either a reference image or prior knowledge of the distortion type.
- Traditional Methods: Traditional NR-IQA models commonly extract statistical distortion traces from transformed image domains and map them to quality scores with low-complexity regressors.Examples include normalized bandpass, steerable-pyramid, and DCT-based representations.
- Deep Methods: Deep NR-IQA approaches often use pretrained CNN backbones because existing perceptual-quality databases are too small for end-to-end supervised training.Architectures may combine distortion-related features with high-level content information.
- Recent Methods: PaQ-2-PiQ uses image-level and patch-level subjective quality data, while other recent methods include CONTRIQUE and MUSIQ.The related work describes PaQ-2-PiQ as achieving state-of-the-art performance with a dataset large enough for supervised deep-model training.
- Re-IQA Positioning: Re-IQA avoids distortion-class labels that may be inaccurate for in-the-wild data by independently learning semantic content and low-level quality features.The resulting representations are mapped to subjective scores with a low-complexity regression model without fine-tuning the deep networks.
3. Rethinking-IQA
Re-IQA learns separate high-level content and low-level quality representations through contrastive encoders, then combines them for perceptual quality prediction. Its quality-aware training constructs image pairs using distortions, overlap-based crops, and intra-pair swapping to reduce reliance on content alone.
- 3. Rethinking-IQA: Re-IQA trains two ResNet-50 encoders independently to learn high-level content and low-level image quality representations using contrastive learning.The encoders are later used as frozen representation backbones for downstream quality prediction.
- 3.2.1 Quality-Aware Image Augmentation Scheme: The quality-aware framework uses a distortion augmentation bank and intra-pair image swapping to model perceptual quality differences.The augmentation bank contains 25 distortion methods, each applied at 5 severity levels.
- 3.2.1 Quality-Aware Image Augmentation Scheme: OLA-based cropping selects overlapping neighboring patches as similar-quality pairs while pairing different distortions or different images as different-quality samples.Crop locations remain consistent across images within a chunk but differ between chunks.
- 3.2.2 Intra-pair Image Swapping: Half Swap labels identically positioned crops with different distortions as different-quality despite shared visual content, encouraging features beyond content-dependent cues.This procedure operationalizes the hypothesis that different distorted versions of the same image have different perceptual quality-aware features.
- 3.2.3 Quality-Aware Training: InfoNCE brings similar-quality representations together and separates different-quality representations while updating the online encoder and momentum encoder.The momentum encoder uses a weighted update from its previous weights and the online encoder weights.
- 3.2.4 Quality Prediction: A single-layer perceptron regressor concatenates content and quality-aware representations to map them to perceptual image quality scores without fine-tuning the encoders.Only the low-complexity regressor head is trained across the evaluated databases.
4. Experimental Results
Experiments evaluate Re-IQA across authentic and synthetic IQA databases using a linear regressor over learned representations. Results support complementary roles for content-aware and quality-aware features, with their combination achieving competitive performance across tested databases.
- Training Configurations: The Re-IQA framework trains separate ResNet-50 encoders for high-level content and low-level quality features, using unsupervised learning and a Mixture of Experts design.The content encoder follows an unsupervised ImageNet configuration, while the quality-aware branch uses contrastive training with OLA-based crops and a projection head.
- Training Datasets: The training data combine pristine images from KADIS with authentically distorted images from AVA, COCO, CERTH-Blur, and VOC.The augmentation scheme uses images with matching semantic content to learn distortion-sensitive representations for in-the-wild images.
- Evaluation Protocol: The evaluation includes legacy synthetic-distortion datasets alongside in-the-wild databases, addressing differences in content and distortion diversity.Legacy datasets contain relatively few pristine images with synthetically applied distortions, whereas newer in-the-wild datasets target more realistic variation.
- Evaluation Protocol: Re-IQA is evaluated on IQA databases containing both authentic and synthetic distortions, using SRCC and PLCC against subjective MOS scores.A single-layer regressor maps pretrained encoder outputs to quality scores using L2 loss.
- Results: Re-IQA achieves competitive performance across all tested databases, and combining content-aware with quality-aware features benefits the final NR-IQA task.Content-aware features dominate on authentic distortions, whereas quality-aware features dominate on synthetic distortions; the combined regressor can adapt to both settings.
- Results: Quality-aware features alone do not surpass other methods on in-the-wild images, but they outperform most competitors on synthetic-distortion datasets.The authors associate this pattern with the stronger impact of image content in authentic distortions and conclude that the learned quality representations align with image distortions.
5. Concluding Remarks
The paper presents Re-IQA as a holistic IQA approach that separately learns content and distortion representations and combines them for competitive performance. The framework achieves state-of-the-art performance across eight evaluated datasets and is described as extensible to other architectures and video quality assessment.
- Concluding Remarks: Re-IQA separately targets the effects of image content and distortion on overall quality, then combines complementary representations for IQA.The framework re-engineers MoCo-v2 with novel augmentation, OLA-based smart cropping, and a Half-Swap scheme.
- Concluding Remarks: Re-IQA consistently achieves state-of-the-art performance across the eight evaluated IQA datasets.
- Concluding Remarks: The framework can extend to other CNN and Transformer architectures and may serve as a spatial feature extraction module for video quality assessment.The paper specifically mentions MUSIQ and supervised ResNet-50-based video quality assessment systems as possible extensions.
7. Appendix
The appendix details distortion augmentations, cross-database evaluations, Full-Reference IQA extensions, and visualizations of learned quality representations.
- Distortion Bank: 25 distortion augmentations, each with five levels, train the quality-aware sub-module.The distortions include contrast, masking, noise, compression, color changes, resizing, pixelation, and multiple blur types.
- Cross-Database Performance: Cross-database evaluation uses CLIVE, KonIQ, LIVE-IQA, and CSIQ-IQA with SRCC as the performance measure.The comparison includes Re-IQA and three state-of-the-art NR-IQA models across synthetic and authentic distortions.
- Cross-Database Performance: Re-IQA attains comparable performance to NR-IQA models across synthetic and authentic distortions in the cross-database evaluation.The results are reported in Table 3.
- Full Reference Image Quality Assessment: Re-IQA-FR predicts differential mean opinion scores from reference and distorted image representations while keeping the pretrained sub-module weights unchanged.Only the score regressor is modified and trained for the Full-Reference setting.
- Full Reference Image Quality Assessment: Re-IQA-FR achieves state-of-the-art performance across all evaluated synthetic-distortion databases and slightly outperforms CONTRIQUE-FR in cross-database comparison.The evaluation compares Re-IQA-FR with traditional and deep-learning-based FR-IQA models.
- Visualization: In a t-SNE visualization of 1338 KonIQ images, lower- and higher-quality points cluster toward opposite corners, with overlap indicating likely score disagreement.Orange points represent scores below 50 and blue points represent scores above 50 on a 0–100 scale.
- Discussion on Hypotheses: Similar-quality samples are generated from overlapping patches, based on the hypothesis that quality varies with content and changes only slightly between neighboring patches.The authors evaluate the effect of patch overlap through an ablation study and do not explicitly sample distant patches.
8. Change Log
The change log records the paper’s initial arXiv upload and a later minor typo correction.
- April 02, 2023 marks the first version uploaded to arXiv.
- May 27, 2023 records a minor typo fix.