Source-linked AI summary
From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality
Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, Alan Bovik
TL;DR
No-reference perceptual picture quality prediction remains difficult, especially for real-world distorted pictures, despite its importance to high-volume media applications. The paper introduces a large real-world picture-and-patch database with extensive human labels and trains region-based models linking local and global quality. These models achieve state-of-the-art picture-quality performance and transfer well to other in-the-wild databases, while the task remains constrained by content–distortion interactions and varied picture sizes.
Problem
No-reference picture quality prediction is largely unsolved because quality assessment lacks a pristine reference and real-world distortion perception depends on content.
Method
The paper builds a large real-world picture-and-patch database and trains deep region-based architectures using picture and patch quality labels for global, local, and feedback-based prediction.
Results
The models obtain state-of-the-art picture-quality performance on the new database and transfer well to CLIVE and KonIQ-10k without fine-tuning.
Takeaways & Limitations
The database enables global-to-local and local-to-global quality inferences, including local quality maps and whole-picture predictions.
Takeaways & Limitations
Training on thousands of picture sizes and aspect ratios caused unstable losses with fixed-size pooling and fully connected layers.
Abstract
from arXiv · showhide
Blind or no-reference (NR) perceptual picture quality prediction is a difficult, unsolved problem of great consequence to the social and streaming media industries that impacts billions of viewers daily. Unfortunately, popular NR prediction models perform poorly on real-world distorted pictures. To advance progress on this problem, we introduce the largest (by far) subjective picture quality database, containing about 40000 real-world distorted pictures and 120000 patches, on which we collected about 4M human judgments of picture quality. Using these picture and patch quality labels, we built deep region-based architectures that learn to produce state-of-the-art global picture quality predictions as well as useful local picture quality maps. Our innovations include picture quality prediction architectures that produce global-to-local inferences as well as local-to-global inferences (via feedback).
1. Introduction
Perceptual picture quality depends strongly on image content and differs from aesthetics, making no-reference prediction difficult despite its practical importance. The paper addresses this gap with a large real-world database, extensive human judgments, and models that connect global and local quality.
- Motivation: Hundreds of billions of photos are shared annually, while streaming services account for 60% of downstream internet traffic, making perceptual quality prediction high-stakes.The problem affects social media, streaming, surveillance, medical visualization, and scientific imaging.
- Perceptual challenges: Identical JPEG compression can appear relatively unimpaired in one picture but unacceptable in another, while uniform noise varies in visibility across image content.Content-dependent masking and artifact processing influence distortion visibility and annoyance.
- Quality versus aesthetics: Picture quality concerns perceptual distortion, whereas aesthetics also includes subject placement, mood, and artistic value.A blurrier image may be more aesthetically pleasing, so the two concepts should not be conflated.
- Research gap: No-reference picture quality prediction is largely unsolved because assessment lacks a pristine reference, including at camera output and user-content ingestion.This limits quality assessment for platforms accepting user-generated content.
- Contributions: The database contains about 40,000 real-world pictures, 120,000 patches, and about 4M human perceptual quality judgments.The study collected both picture- and patch-level labels to relate local and global quality.
2. Background
Earlier IQA datasets use few pristine images with single synthetic distortions, whereas newer in-the-wild datasets contain more realistic but difficult distortion mixtures. Existing shallow and deep models perform well on legacy data but struggle on in-the-wild databases.
- Legacy datasets: Legacy IQA datasets contain small numbers of pristine images with diverse types and amounts of single synthetic distortions.Their limited content and distortion diversity do not capture complex real-world mixtures.
- In-the-wild datasets: In-the-wild databases contain pictures impaired by complex mixtures of highly diverse distortions that vary across images.Each distortion mixture is described as unique as the picture it affects.
- Existing models: Shallow learners predict quality accurately on legacy datasets but struggle on recent in-the-wild databases.The contrast reflects the greater realism and diversity of newer data.
- Existing models: Deep no-reference models achieve state-of-the-art performance on legacy synthetic-distortion databases but also struggle on CLIVE.CLIVE is characterized as difficult and too small to span the perceptual quality space sufficiently for very deep models.
3. Large-Scale Dataset and Human Study
The study constructs a large, UGC-like database of diverse pictures and patches, then collects and validates millions of human quality judgments using separate AMT tasks.
- Dataset and human study: The database contains about 40,000 pictures, 120,000 patches, 4M human judgments, and nearly 8,000 unique subjects.The collected labels include 3,931,710 judgments after subject rejection.
- UGC-like picture sampling: Pictures were sampled from public databases to match feature histograms of 15M randomly selected social-media pictures.The matching features included brightness, colorfulness, contrast, spatial information, pixel count, and detected faces.
- Patch cropping: Each picture produced three same-aspect-ratio patches at 40%, 30%, and 20% of its linear dimensions, with limited overlap.Every patch remained within the source picture, and overlap between patches from one image was capped at 25%.
- Crowdsourced subjective study: Picture and patch quality were rated in separate AMT tasks using instructions, training, repeated contents, and gold images to assess worker consistency.Most workers participated in only one task, and the study increased each HIT from 60 to 210 contents without reducing performance.
- Processing subjective scores: Subject labels were normalized into Z scores, averaged by content, and scaled to [0, 100] as Mean Opinion Scores.Inter-subject consistency was evaluated by splitting subjects into two disjoint sets and comparing their MOS values.
- Picture–patch quality relationships: Picture MOS correlated with largest-patch MOS at LCC 0.43, falling to 0.36 and 0.28 for 30% and 20% patches.The largest patch represents only 16% of the picture area; the realistic MOS distribution is narrower and makes small quality differences important to predict.
4. Learning Blind Picture Quality Predictors
The paper develops progressively richer blind picture-quality predictors, beginning with a picture-only baseline and extending to models that learn from picture and patch labels. The resulting architectures support global predictions, local quality maps, and local-to-global feedback, while transferring to other in-the-wild databases.
- Model progression: The learning study evolves from a picture-only network into models that exploit both picture and patch human quality labels.The baseline uses a modified, fine-tuned ResNet-18; later models incorporate region-based processing and feedback.
- Model progression: Thousands of picture sizes and aspect ratios destabilized raw training with fixed pooling and fully connected layers.The authors white-padded training pictures to 640 × 640, while pictures exceeding 640 in either dimension were moved to the test set.
- Baseline evaluation: The Baseline Model outperformed most compared methods and competed well with NIMA on full-picture quality prediction.The evaluation also included NIQE, BRISQUE, CNNIQA, RoIPool, and Feedback models.
- Patch evaluation: The Baseline Model maintained or slightly improved performance across patches, whereas NIMA deteriorated as patch sizes decreased.Patch evaluations used associated patches at 40%, 30%, and 20% of linear image dimensions.
- Local quality maps: The RoIPool Model predicts patch-wise quality maps from picture and patch labels, producing spatially localized quality estimates.It aggregates information at patch- and picture-sized scales and can process specified patch grids.
- Feedback model: The Feedback Model improved picture and patch performance further and consistently outperformed shallow and deep quality models.Its local-to-global feedback architecture was intended to improve whole-picture predictions by integrating localized quality information.
- Cross-database evaluation: The three proposed models transferred well to CLIVE and KonIQ-10k without fine-tuning, outperforming shallow and other deep models on both datasets.The authors present this transfer as evidence of the proposed dataset's representativeness and model efficacy.
5. Concluding Remarks
The paper presents a large, difficult database and models supporting global-to-local and local-to-global picture-quality inference. The authors identify persistent challenges from subjective judgments, distortion-content interactions, and masking.
- The database and models are designed to support global-to-local and local-to-global quality inferences.The model produces local quality inferences, picture quality maps, and global image quality.
- No-reference picture-quality assessment remains difficult because judgments are subjective and distortions interact complexly with picture content.The authors also highlight masking as a perceptual factor and suggest more complex architectures or semantic side-information as possible future directions.
A. Performance Summary
The proposed models are evaluated on full-size pictures, multiple patch scales, and cross-database tests. The reported comparisons show competitive or superior performance across diverse distortion databases.
- The proposed models compete well with or outperform other models on images and patches in the new database and on CLIVE and KonIQ-10K.Cross-database evaluations apply models trained on the new database without fine-tuning.
- Table 5 reports picture-quality prediction performance on full-size validation and test pictures, with higher values indicating superior performance.NIQE is included as an untrained comparison.
- Table 6 reports patch-quality prediction results for largest, middle-size, and smallest patches using the same protocol as Table 5.The patch sizes are 40%, 30%, and 20% of the linear dimensions.
- Table 7 compares models trained on the new database when applied to CLIVE and KonIQ-10K without fine-tuning.
B. Information on Model Parameters
The parameter comparison shows that model size alone does not determine performance. A RoIPool model improves over a parameter-matched baseline, while MobileNet-v2 NIMA matches ResNet-18 NIMA with fewer parameters.
- Table 8 summarizes the number of learnable parameters for each compared model.
- NIMA (MobileNet-v2) performs consistently at par with NIMA (ResNet-18) using only 20% of the total parameters.
- CNNIQA uses fewer than 1M parameters and performs poorly, indicating limited ability to model the complex problem.
- RoIPool Model achieves significantly better performance than the parameter-matched Baseline Model.The comparison suggests that accurate local quality predictions are important for global quality.
C. Picture MOS vs Patch MOS scatter plots
Figure 12 compares picture MOS with patch MOS for patches cropped from the same pictures at two smaller scales.
- The left scatter plot compares picture MOS with MOS for the second-largest patches, cropped at 30% of the picture’s linear dimension.
- The right scatter plot compares picture MOS with MOS for the smallest patches, cropped at 20% of the picture’s linear dimension.
D. Amazon Mechanical Turk Interface
The AMT interface let workers review instructions before participation, then rate picture and patch quality using a five-level Likert scale.
- Workers could preview the instructions page before accepting participation in the study.
- Picture quality was rated on a five-level Likert scale ranging from “Bad” to “Excellent.”The training and testing interfaces demonstrated this rating task.
- A similar user interface was used for the patch quality rating task.