Source-linked AI summary
RankIQA: Learning from Rankings for No-reference Image Quality Assessment
Xialei Liu, Joost van de Weijer, Andrew D. Bagdanov
TL;DR
Limited IQA datasets make it difficult to train deep CNNs for no-reference quality assessment, because collecting human annotations is labor-intensive and costly. RankIQA learns quality rankings from automatically distorted images with a Siamese Network, transfers that knowledge through fine-tuning, and efficiently uses all image pairs in a batch. It improves TID2013 results by about 5% over HOSA and outperforms existing NR-IQA methods on LIVE and TID2013, including FR-IQA methods on LIVE.
Problem
IQA datasets are too small for training increasingly deep CNNs because collecting multiple human annotations per image is labor-intensive and costly.
Method
RankIQA trains a Siamese Network on automatically generated image-quality rankings, transfers its representation to a CNN regressor, and backpropagates from all batch pairs.
Results
RankIQA+FT achieves about 5% higher SROCC than HOSA on TID2013 and outperforms existing NR-IQA and FR-IQA methods across the reported benchmarks.
Takeaways & Limitations
Automatically generated rankings provide abundant training data for deeper and wider NR-IQA networks, while efficient Siamese backpropagation converges faster and to lower loss than comparison protocols.
Takeaways & Limitations
The expected convergence speedup is lower than the mini-batch-size bound because all pairs in a mini-batch are highly correlated.
Abstract
from arXiv · showhide
We propose a no-reference image quality assessment (NR-IQA) approach that learns from rankings (RankIQA). To address the problem of limited IQA dataset size, we train a Siamese Network to rank images in terms of image quality by using synthetically generated distortions for which relative image quality is known. These ranked image sets can be automatically generated without laborious human labeling. We then use fine-tuning to transfer the knowledge represented in the trained Siamese Network to a traditional CNN that estimates absolute image quality from single images. We demonstrate how our approach can be made significantly more efficient than traditional Siamese Networks by forward propagating a batch of images through a single network and backpropagating gradients derived from all pairs of images in the batch. Experiments on the TID2013 benchmark show that we improve the state-of-the-art by over 5%. Furthermore, on the LIVE benchmark we show that our approach is superior to existing NR-IQA techniques and that we even outperform the state-of-the-art in full-reference IQA (FR-IQA) methods without having to resort to high-quality reference images to infer IQA.
1. Introduction
NR-IQA is important but difficult to scale because deep CNNs need large datasets, while IQA annotations are costly and existing datasets are small. RankIQA addresses this gap by learning from automatically generated quality rankings, then transferring the learned representation to a CNN regressor.
- IQA predicts perceptual image quality, commonly against human Mean Opinion Scores, and supports applications including restoration, super-resolution, and retrieval.
- NR-IQA estimates quality without a reference image, but distortion-specific methods limit applicability.
- Deep CNNs require massive training datasets, whereas IQA datasets are small because collecting multiple human annotations per image is labor-intensive and costly.
- RankIQA generates automatically ranked image sets by applying distortions of varying intensities, avoiding absolute quality labels for each generated image.
- The method trains a Siamese Network to rank image quality, then transfers its learned knowledge to a traditional CNN fine-tuned for absolute IQA.
- An efficient Siamese training method forwards a batch once and backpropagates gradients from all image pairs, improving training efficiency over alternative protocols.
2. Related work
Prior NR-IQA work uses hand-crafted statistics, small-patch augmentation, multi-task learning, or pre-trained features to cope with limited labeled data. RankIQA instead uses abundant automatically generated rankings as data augmentation and introduces efficient all-pair Siamese backpropagation.
- Traditional NR-IQA approaches: Traditional NR-IQA methods include Natural Scene Statistics approaches and learning-based methods that infer quality from image features.
- Deep learning for NR-IQA: Deep NR-IQA methods are constrained by the lack of large labeled datasets, motivating patches, multi-task learning, and pre-trained networks as mitigation strategies.
- Deep learning for NR-IQA: RankIQA uses many automatically generated image-quality rankings to train deeper and wider networks than methods trained directly on absolute IQA data.
- Learning to rank: Unlike conventional ranking applications, RankIQA uses rankings primarily as data augmentation before fine-tuning for NR-IQA.
- Hard-negative mining for Siamese network training: Hard-negative mining improves Siamese pair selection but can substantially increase computational cost.
- Hard-negative mining for Siamese network training: RankIQA considers all pairs in each mini-batch without hard-negative selection, exploiting forward propagation more efficiently.
3. Learning from rankings for NR-IQA
RankIQA learns image-quality rankings from synthetically distorted images, then transfers the learned representation to absolute NR-IQA. Its efficient Siamese training forwards each batch image once while using gradients from image pairs.
- 3.1. Overview of our approach: RankIQA generates ranked image sets from arbitrary reference images by applying distortions whose relative quality is known without absolute IQA scores.Within each deformation type, image pairs can be ordered by quality.
- 3.1. Overview of our approach: After ranking pretraining, a single Siamese branch is fine-tuned on available IQA data to calibrate its representation into absolute IQA measurements.The fine-tuning stage addresses the final single-image quality-estimation task.
- 3.2. Siamese networks for ranking: A Siamese network uses two weight-sharing branches and a loss module to learn a scalar output that ranks paired images by quality.The pairwise ranking hinge loss is zero when the predicted ordering satisfies the margin and otherwise updates the branches using the difference between their gradients.
- 3.3. Efficient Siamese backpropagation: Efficient Siamese backpropagation forwards each mini-batch image once and derives gradients from all possible image pairs, avoiding redundant branch computation.The method uses shared parameters and aggregates pairwise gradients, rather than selecting hard negatives.
- 3.3. Efficient Siamese backpropagation: The best-case computational speed-up is on the order of the mini-batch size, although correlated pairs can reduce the final convergence speed-up.For n images, the stated speed-up is n − 1; the best case occurs when n equals the mini-batch size M.
4. Experimental results
The experiments evaluate RankIQA across datasets, architectures, ranking strategies, and standard IQA metrics. Results show that ranking-based training supports deep networks, improves convergence, and achieves strong performance against IQA baselines and state-of-the-art methods.
- Datasets: Experiments use generic non-IQA datasets to generate ranked pairs, then IQA datasets for fine-tuning and evaluation.The study evaluates LIVE and TID2013, while Waterloo and Places2 provide images for generating ranked distortions.
- Evaluation protocols: The evaluation measures image-quality prediction with LCC and SROCC, using randomly sampled sub-images whose outputs are averaged at test time.LCC measures linear correlation, while SROCC measures the monotonic relationship between ground-truth and estimated IQA.
- Siamese networks and IQA discrimination: The Siamese network separates six JPEG distortion levels, demonstrating that ranking training discriminates image quality across distortion severity.The network is trained on one dataset and tested on synthetically distorted images from another dataset using the same distortion levels.
- Efficient Siamese backpropagation: Efficient Siamese backpropagation converges much faster and to a considerably lower loss than standard random pair sampling and hard-negative mining at equal computational cost.The method forwards 72 images per mini-batch and derives gradients from all image pairs in the batch.
- Network performance analysis: VGG-16 achieves the best results among the Shallow, AlexNet, and VGG-16 architectures, indicating that ranking training supports deeper networks without overfitting.ImageNet initialization further improves results, so later experiments use pretrained VGG-16.
- Evaluation on TID2013 and LIVE: RankIQA+FT achieves about 5% higher performance than HOSA on all TID2013 distortions, while on LIVE it is about 1% better than the best reported result for all distortions.On LIVE, the approach also surpasses existing NR-IQA methods and the state-of-the-art FR-IQA method DCNN using LCC and SROCC.
5. Conclusions
RankIQA addresses scarce IQA data by learning from abundant ranked image datasets and introduces efficient Siamese-network backpropagation. On LIVE and TID2013, the approach outperforms existing NR-IQA and FR-IQA methods.
- RankIQA learns from ranked image datasets to address the scarcity of IQA data.Because ranked data can be generated abundantly, the approach supports training deeper and wider networks.
- The proposed Siamese-network backpropagation method avoids hard-negative mining and converges faster to a lower loss.Comparisons included standard pair sampling and hard-negative sampling.
- Experiments on LIVE and TID2013 show that the NR-IQA approach outperforms existing NR-IQA techniques and even FR-IQA methods.