Exploring Rich Subjective Quality Information for Image Quality Assessment in the Wild

I. INTRODUCTION

• In-the-wild distortions are complex and diverse, and traditional NR IQA methods usually do not perform best on these images.Such distortions can arise from sensor noise, transmission errors, and compression, often in combination.Source: §I · p. 1
• RichIQA predicts MOS, DOS, and SOS, addressing subjective rating information that a single MOS does not capture.MOS represents average quality, SOS rating dispersion, and DOS the distribution across quality labels.Source: §I · p. 2, Figure 1 · p. 2
• RichIQA’s three-stage network uses multi-stage CvT features and short- and long-term memory mechanisms to predict image quality.Its feature module captures multi-level image features, its SLM module models memory mechanisms, and its prediction module outputs a quality distribution and mean.Source: §I · p. 2
• The multi-label strategy trains the network simultaneously with MOS and DOS labels, using the subjective rating information beyond MOS.The strategy is designed to learn the mapping from in-the-wild image features to the quality-score distribution.Source: §I · p. 2
• The authors report improved prediction performance and generalizability from multi-label training and state that RichIQA outperforms state-of-the-art methods on five in-the-wild databases.They also characterize the method as having low computational cost.Source: §I · p. 2

II. PROPOSED METHOD

• The proposed method uses the three-module RichIQA framework to extract features, model memory, and predict image quality.The stated sequence is multi-level feature extraction, short- and long-term memory modeling, then quality prediction.Source: §II · pp. 2–3, Figure 2 · p. 3

A. Multi-level Feature Extraction

• Three-stage CvT features capture multiple levels of image information relevant to in-the-wild quality modeling.The paper notes that perceived quality is affected by both low-level visual features such as distortions and high-level semantic information such as content.Source: A. Multi-level Feature Extraction · p. 3

B. Short-Term and Long-Term Memory Modeling

• The model uses a CNN and a GCN to simulate short- and long-term memory, respectively.The paper relates in-the-wild quality decisions to both short-term information and long-term experience and knowledge.Source: B. Short-Term and Long-Term Memory… · pp. 3–4

1) Short-term Memory Modeling:

• The short-term memory block uses a feature mask to filter multi-level image features, then combines attention features with global semantics.Its output is formed by concatenating and convolving the attention features and global semantics.Source: 1) Short-term Memory Modeling · p. 4

2) Long-term Memory Modeling:

• A gated GCN represents relationships among quality labels as long-term memory, with graph adjacency and weights encoding those relationships.The graph’s vertices represent the C image-quality labels.Source: 2) Long-term Memory Modeling · p. 4
• Attention features and short-term memory modulate the excitation threshold used to update long-term memory.The authors describe this as a simulation of gating detection and propose changing the input to modulate the threshold.Source: 2) Long-term Memory Modeling · p. 4
• The update uses the prior adjacency and weight matrices together with current attention features and short-term memory.The paper defines the prior matrices and the current inputs, and describes nonlinear mappings and learned linear-layer weights in the modulation.Source: 2) Long-term Memory Modeling · p. 4

C. Quality Prediction

• A direct and an indirect pathway exchange information between long-term memory and attention features for quality decision-making.The indirect pathway updates long-term memory using direct-pathway information, then feeds semantic knowledge back to the direct pathway.Source: C. Quality Prediction · p. 4
• The algorithmic decision pathway predicts a DOS from features without the influence of subjective memory.The paper contrasts this with the memory DOS, which is influenced by the SLM mechanism.Source: C. Quality Prediction · pp. 4–5
• The final DOS combines the predicted memory DOS and algorithmic DOS using a decision-weight hyperparameter.The parameter λ controls the weights between the two predicted distributions.Source: C. Quality Prediction · p. 5
• The model calculates the predicted MOS and SOS from the final predicted DOS.The quality labels’ assigned scores are used in deriving SOS.Source: C. Quality Prediction · p. 5

• RichIQA uses MOS, DOS, and SOS as training labels, selecting labels according to the subjective quality information each database provides.Source: D. Multi-label Training Strategy · p. 5

1) DOS as Training Label:

• When ground-truth DOS is available, RichIQA uses earth mover’s distance to measure the difference between predicted and ground-truth distributions.Source: 1) DOS as Training Label · p. 5
• When a database provides MOS and SOS but not DOS, the method assumes a Gaussian DOS parameterized by those ground-truth scores and applies EMD against it.Source: 1) DOS as Training Label · p. 5
• When only MOS is available, the method estimates expected SOS using an empirical parameter and the database score range, assumes a Gaussian DOS, and applies EMD.Source: 1) DOS as Training Label · p. 5

2) MOS as Training Label:

• The MOS prediction is trained with an L1 loss measuring the distance between predicted and ground-truth MOS.Source: 2) MOS as Training Label · p. 5
• The method uses an expected standard deviation loss to train predicted SOS against expected SOS, reflecting greater rating diversity for middle-quality images than for relatively high- or low-quality images.Source: 2) MOS as Training Label · pp. 5–6

3) Total Loss:

• Joint training combines EMD, L1, and ESD losses using balance factors α, β, and γ.Source: 3) Total Loss · p. 6

III. EXPERIMENTS / A. In the Wild IQA Databases

• The experiments compare RichIQA with state-of-the-art methods and analyze its effectiveness across five in-the-wild IQA databases.Source: §III · p. 6, A. In the Wild IQA Databases · p. 6

1) KonIQ-10K:

• KonIQ-10K contains 10,073 diverse internet images with authentic distortions and provides MOS and DOS ratings.Source: 1) KonIQ-10K · p. 6

2) BID:

• BID contains 6,000 images spanning scenes and blur types and levels, with observer scores supporting MOS and DOS.Source: 2) BID · p. 6

3) LIVE Challenge:

• LIVE Challenge contains 1,162 natural images with authentic distortions and provides MOS and SOS from crowdsourced ratings.Source: 3) LIVE Challenge · p. 6

4) SPAQ:

• SPAQ contains 11,125 smartphone images rated for perceptual quality and releases MOS only.Source: 4) SPAQ · p. 6

5) FLIVE:

• FLIVE contains 39,810 authentically distorted images and releases MOS only, extending local-patch quality assessment to whole images.Source: 5) FLIVE · p. 6

1) Implementation Detail:

• Experiments use 80/20 train-test splits repeated ten times; Table II compares MOS prediction on KonIQ-10K and BID, while implementation uses Adam at a 0.00001 initial learning rate.Source: 1) Implementation Detail · pp. 6–7, Table II · p. 7

2) Competing Methods and Evaluation Criteria:

• MOS prediction compares RichIQA with 14 listed methods using SRCC, PLCC, and RMSE; higher SRCC and PLCC and lower RMSE indicate better performance.PLCC is calculated after mapping predicted MOS with a four-parameter logistic function.Source: 2) Competing Methods and Evaluation… · p. 7
• DOS prediction compares RichIQA with NIMA, Liu, KonCept512, and Gao using JSD, EMD, RMSE, intersection, and cosine.Higher intersection and cosine and lower JSD, EMD, and RMSE indicate better performance.Source: 2) Competing Methods and Evaluation… · p. 7

C. Comparison Results

• KonIQ-10K and BID provide DOS; LIVE Challenge provides MOS and SOS; SPAQ and FLIVE provide only MOS, so missing DOS is supplemented for joint training.The paper describes the DOS supplements in Section II-D.Source: C. Comparison Results · p. 8

1) MOS Prediction:

• RichIQA gains 3.1% and 3.0% SRCC over second-best on KonIQ-10K and BID, 5.0% PLCC on LIVE Challenge, and 1.5% and 4.8% SRCC on SPAQ and FLIVE.The gains use database-provided or estimated DOS alongside MOS, depending on available labels.Source: 1) MOS Prediction · p. 8

D. Cross-Database Validation

• When trained on KonIQ-10K, RichIQA gains 7.5%, 4.9%, 7.2%, and 4.3% SRCC on BID, LIVE Challenge, SPAQ, and FLIVE; BID training yields gains of 12.9%, 4.3%, 0.1%, and 18.2% on those databases.These cross-database tests evaluate prediction on databases not used for training.Source: Table VI · p. 9, D. Cross-Database Validation · p. 9

2) DOS Prediction:

• RichIQA's predicted DOS is closer to human perceptions than competitors', with 2.6% and 1.2% intersection gains over the second-best method on KonIQ-10K and BID.DOS prediction is a byproduct of RichIQA and a capability rarely found in mainstream IQA measures.Source: 2) DOS Prediction · p. 9

1) Ablation with Different Stages of CvT:

• Using all three CvT stages improves prediction performance; removing any stage reduces it, while activation maps show a progression from low-level to high-level features.The first stage highlights edges and contours, while the third focuses on semantic features such as objects.Source: Table VII · p. 10, 1) Ablation with Different Stages of CvT · pp. 10–11, Figure 4 · p. 10

2) Ablation with the SLM Module:

• The SLM module sends attention features directly to quality prediction and sends long-term-memory-processed image features through an indirect pathway.The paper evaluates each pathway by removing it, as well as removing both pathways.Source: 2) Ablation with the SLM Module · p. 11
• Removing either pathway decreases MOS prediction performance on all databases; removing the SLM module significantly reduces performance on most databases.The reported significant drops include KonIQ-10K, BID, LIVE Challenge, and FLIVE.Source: 2) Ablation with the SLM Module · p. 11
• Both direct and indirect SLM pathways are essential to RichIQA's performance on most in-the-wild databases.The comparison includes variants that remove either pathway or both.Source: 2) Ablation with the SLM Module · p. 11

3) Ablation with Loss Functions:

• Using EMDLoss, L1Loss, and ESDLoss together gives the best performance across all databases, although the second-best loss combination varies by database.The ablation reports that single losses or any two-loss combination also achieve good prediction results.Source: 3) Ablation with Loss Functions · p. 11

4) Ablation with Balance Factors of Loss Functions:

• On KonIQ-10K and SPAQ, doubling or halving the loss-balance factors has little effect, whereas any tested change decreases performance on BID and LIVE Challenge.The authors therefore use α:β:γ = 200:10:1.Source: 4) Ablation with Balance Factors of… · p. 11

F. Computational Complexity

• RichIQA achieves the best prediction performance among the compared methods while balancing computational complexity; FLOPS and parameter comparisons use 224 × 224 inputs.The figure encodes network parameter count as circle radius.Source: Figure 6 · p. 12, F. Computational Complexity · p. 12

IV. CONCLUSION

• RichIQA consistently outperforms state-of-the-art NR IQA models on five mainstream in-the-wild databases, with large cross-database leads and the best performance without much added complexity.The conclusion also reports good generalizability in cross-database validation.Source: §IV · p. 12
• The authors suggest that exploring subjective quality information beyond MOS could generalize to other perceptual quality assessment scenarios in the future.The conclusion frames this as a future possibility, not a demonstrated result in those scenarios.Source: §IV · p. 12