PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Wei Sun, Xiongkuo Min, Danyang Tu, Guangtao Zhai, Siwei Ma

arXiv:2105.14550v3cs.MM

TL;DR

In-the-wild BIQA is challenged by authentic distortions, content dependence, and insufficiently diverse labeled databases. The paper combines a staircase feature-fusion structure with iterative mixed-database training. It reports the best performance on six in-the-wild databases and strong cross-database results, including SRCC values above 0.8 on four databases.

  • Problem

    Existing BIQA models mainly target synthetic distortions, while in-the-wild assessment requires representations that handle diverse distortions, image content, and limited database scale.

  • Method

    The model hierarchically fuses intermediate CNN features and uses IMDT to train feature extraction across multiple databases with database-specific quality regressors.

  • Results

    The model achieves the best performance on all six in-the-wild IQA databases and cross-database SRCC values above 0.8 on CLIVE, BID, KonIQ10k, and SPAQ.

  • Takeaways & Limitations

    The reported results indicate that the learned quality-aware representation is effective across in-the-wild databases and robust to varied image distributions.

Abstract

from arXiv · show

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

I. INTRODUCTION

In-the-wild BIQA is needed because authentic distortions and image content differ substantially from synthetic settings, while existing databases and CNN architectures limit effective learning. The paper addresses these challenges with hierarchical feature fusion and iterative mixed-database training.

  • Motivation: In-the-wild images commonly contain authentic distortions such as exposure problems, low visibility, motion blur, and ghosting, motivating effective quality assessment.Quality assessment supports viewer QoE, computer vision algorithms, service providers, and camera-based filtering systems.
  • Challenges: BIQA models trained mainly on synthetic distortions face a mismatch because authentic distortions arise from varied shooting factors and may combine uneven, non-uniform degradations.Synthetic distortions are more regular, whereas authentic distortions can depend on environment, equipment, and photographing techniques.
  • Challenges: Deep BIQA must preserve both low-level distortion cues and high-level semantic information, but classification-oriented CNN backbones can lose low-level features.The paper identifies this mismatch as a challenge for directly using popular CNN architectures in BIQA.
  • Challenges: Existing IQA databases provide too few diverse annotated samples for deep CNN training, while patch-based augmentation cannot increase content diversity and is unsuitable for non-uniform authentic distortions.Subjective IQA labeling is time-consuming, and assigning an image score to every patch is problematic when distortion varies spatially.
  • Proposed approach: The staircase structure hierarchically fuses intermediate CNN features so the representation uses visual information from low-level to high-level.Intermediate CNN stages encode different information, including edges and corners at lower levels and semantics at higher levels.
  • Proposed approach: IMDT trains feature extraction across multiple databases while using corresponding regressors for database-specific quality scales, increasing sample and distortion diversity.The strategy treats relative quality as consistent within each database even when absolute scores are not directly comparable across databases.

A. Blind Image Quality Assessment

BIQA research spans handcrafted and learning-based features, but in-the-wild assessment requires preserving both low-level distortion cues and high-level semantic information. The proposed staircase network hierarchically fuses intermediate CNN features to form quality-aware representations.

  • Prior BIQA methods: Handcrafted BIQA methods use NSS, free-energy, texture, corner, and edge features, while learning-based methods automatically derive representations from IQA data.Examples include DIIVINE, BLIINDS-II, BRISQUE, NFERM, shallow and deep CNNs, multitask learning, bilinear pooling, and hypernetworks.
  • Prior BIQA methods: Mixed database training combines multiple IQA databases, but database-specific quality spaces require regressors trained for corresponding target databases.Training the whole network jointly across databases can ignore the distinct mapping functions required by each quality scale.
  • Motivation: Directly using classification-oriented CNN backbones for BIQA loses low-level information because their features are primarily semantic.Perceptual quality depends on visual information ranging from low-level distortions to high-level content.
  • Staircase network: The staircase structure hierarchically incorporates intermediate-layer features so the final representation uses low-level through high-level visual information.The architecture assumes Ns backbone stages and constructs progressively fused feature maps from stage outputs Fi.
  • Staircase network: Feature maps from different stages cannot be added directly because their resolutions and channel counts differ.The method uses convolutional transformations to align feature-map dimensions and channels before addition.
  • Staircase network: Hierarchical merging avoids the training difficulty caused by directly connecting low-level features to the final stage through a short connection.The authors state that direct addition can route gradients through the shortcut while bypassing the backbone, hindering deep semantic feature learning.

B. Image Quality Regressor

The image quality regressor converts staircase features into quality scores. It uses global average pooling followed by two fully connected layers and is trained end-to-end with the feature extractor.

  • Regressor architecture: Global average pooling converts the extracted feature maps into a P × 1 feature vector before regression.P denotes the number of final feature maps.
  • Regressor architecture: Two fully connected layers with 128 and 1 neurons predict the image quality score.The final scalar output is the predicted quality score.
  • Training: The staircase network and image quality regressor are trained jointly in an end-to-end manner.The loss is based on the Euclidean distance between predicted and ground-truth quality scores.
  • Training: The framework includes image-quality regressors associated with the IQA databases trained simultaneously.Algorithm 1 describes iterative updates and validation for database-specific regressors.

C. Iterative Mixed Database Training Strategy

Existing IQA databases are small and lack sufficient content and distortion diversity for deep-model training. IMDT trains shared feature extraction across multiple databases while preserving database-specific quality regressors.

  • Motivation: Subjective IQA labeling is expensive and time-consuming, leaving existing databases too small for effective deep CNN training.Cropping and rotation increase sample count but do not increase content or distortion diversity.
  • IMDT overview: IMDT trains the model on multiple databases simultaneously to exploit their diverse image content and distortions.The strategy is designed to increase the diversity available to the learned feature representation.
  • IMDT overview: IMDT trains the shared feature extraction module across databases while training each image quality regressor on its corresponding database.This separates shared representation learning from database-specific quality-scale mapping.
  • Optimization: The method alternates optimization of database-specific subproblems while updating shared feature-extraction parameters across all subproblems.A loop consists of solving the subproblems for the participating databases.
  • Optimization: For unequal database sizes, the per-loop epoch count is max([Nmax/Ni], E) to reduce imbalance in parameter updates.Nmax is the largest database size, Ni is the size of database i, and E is the upper epoch bound.

IV. EXPERIMENTAL RESULTS

The experimental protocol compares the proposed model with state-of-the-art BIQA methods on in-the-wild and synthetic IQA databases and includes ablation studies.

  • Experimental protocol: Experiments compare the proposed model with state-of-the-art BIQA models on six in-the-wild and four synthetic IQA databases.The protocol also includes cross-database evaluation and ablation studies according to the paper context and section description.
  • Experimental protocol: Ablation studies evaluate the effectiveness of each module in the proposed model.

1) Test Databases:

The evaluation uses six in-the-wild and four synthetic IQA databases spanning diverse authentic and synthetic distortions, image contents, and collection settings.

  • In-the-wild databases: The proposed method is mainly validated on six authentically distorted IQA databases.These include CLIVE, BID, KonIQ10K, SPAQ, FLIVE, and FLIVE Patch.
  • In-the-wild databases: FLIVE contains about 40,000 real-world distorted images and 120,000 randomly cropped patches with diverse contents, sizes, and aspect ratios.The paper evaluates both FLIVE and FLIVE Patch.
  • Synthetic databases: The evaluation also includes four synthetic databases: Kadid10K, LIVE, CSIQ, and LIVEMD.Kadid10K contains 10,125 images, LIVE 770, CSIQ 886, and LIVEMD 450 distorted images.
  • Evaluation settings: The databases use either online crowdsourcing or controlled laboratory subjective experiments, and their image contents do not overlap.This separation avoids overlap between training images in one database and testing images in another.

2) Implementation Details:

The experiments use a ResNet50-based staircase network, compare against handcrafted and deep BIQA methods, and evaluate predictions with correlation-based metrics under repeated splits.

  • Model setup: ResNet50 is used as the staircase network backbone, with six image quality regressors added for the six databases.Backbone weights are initialized on ImageNet, while other weights are randomly initialized.
  • Compared methods: The comparison includes five handcrafted-feature BIQA models and six deep-learning BIQA models, including UNIQUE trained on multiple databases.The compared handcrafted models are QAC, NIQE, ILNIQE, BRISQUE, and BMPRI; deep models include CNNIQA, WaDIQaM-NR, SFA, DB-CNN, HyperIQA, and UNIQUE.
  • Evaluation metrics: PLCC measures prediction linearity and SRCC measures prediction monotonicity, with values closer to 1 indicating better IQA performance.Both criteria are used to evaluate the quality models.
  • Evaluation protocol: Each database uses an 80% training and 20% testing split, repeated 10 times, with median SRCC and PLCC reported.Images derived from the same reference are kept in the same split for synthetic databases.

1) In-the-wild Databases:

The proposed model performs best across six in-the-wild databases, remains competitive on synthetic databases, and reveals dataset-distribution and training-scale limitations.

  • In-the-wild database results: The proposed model achieves the best performance on all six in-the-wild IQA databases by a significant margin.The authors attribute the result to more powerful quality-feature representation than competing deep and handcrafted methods.
  • In-the-wild database results: IMDT particularly improves CLIVE and BID, whose smaller image counts make individually learning strong feature representations difficult.Training across databases brings their performance to the level of larger databases such as KonIQ10K and SPAQ.
  • In-the-wild database results: FLIVE performance is lower for all models because its MOS distribution is concentrated near a high-quality score of about 75 and is narrower and peakier.The other in-the-wild databases cover more evenly distributed quality ranges.
  • Limitations and future work: The authors identify fine-grained quality evaluation as future work because more images shared on social media are high-quality.This observation is linked to advances in camera devices and computational photography.
  • Synthetic database results: The proposed model achieves competitive performance on four synthetic IQA databases, similar to DB-CNN and HyperIQA and better than other deep methods.It also outperforms all handcrafted-feature methods on these databases.
  • Synthetic database results: IMDT does not significantly improve small-scale synthetic databases such as LIVE, CSIQ, and LIVEMD.The paper relates this to their limited distortion types and the stronger relevance of distortion type and degree than image content.

3) Visualization of the Evaluation Effect of the Proposed Model:

The proposed model’s predictions are mostly consistent with subjective quality scores, while ablation results show that both staircase fusion and IMDT improve performance. Statistical testing further reports superiority over other deep-learning BIQA methods on six authentic-distortion databases.

  • Visualization: Predicted quality scores are mostly consistent with subjective MOS values, although failing examples include low-visibility and blurred-content images.The model tends to assign lower scores to images with low visibility and blurred content.
  • Ablation: Both removing the staircase structure and removing IMDT lowers performance relative to the complete model, while remaining above the baseline.This ablation pattern indicates contributions from both modules.
  • Ablation: The staircase structure and IMDT contribute differently across databases, with IMDT especially improving results on the smaller CLIVE and BID databases.The reported explanation is that IMDT transfers useful representations from databases with more abundant and diverse content and distortions.
  • Statistical comparison: The proposed method is statistically superior to other deep-learning IQA methods on six authentically distorted IQA databases.Figure 4 encodes pairwise outcomes as worse, better, or statistically indistinguishable comparisons.
  • Ablation: On larger databases such as KonIQ10k and SPAQ, staircase fusion achieves competitive performance with IMDT because their content and distortion diversity is already sufficient.The staircase structure still improves representation by incorporating intermediate-layer features.

2) The IMDT vs. Transfer Learning Method:

IMDT substantially outperforms transfer learning from a single database in cross-database evaluation. Its multi-database training supports more general representations, while regressors trained on content-poor FLIVE patches are less effective when perceptual quality depends on image content.

  • IMDT versus transfer learning: Models pretrained on other large-scale IQA databases improve performance, but remain far below models trained with IMDT.IMDT trains on multiple databases simultaneously rather than relying on weights learned from one database.
  • Evaluation protocol: Cross-database testing trains on five databases and evaluates on the held-out database without fine-tuning, using five corresponding regressors and their ensemble.This protocol evaluates robustness to database shifts.
  • Cross-database results: Except for the FLIVE patch regressor, SRCC exceeds 0.8 on CLIVE, BID, KonIQ10k, and SPAQ in cross-database evaluation.The reported values surpass most BIQA models trained on the corresponding databases.
  • Cross-database results: IMDT improves cross-database performance over training without IMDT, supporting its goal of learning more general quality-aware features from multiple databases.The ensemble is normally stronger than a regressor trained on one database because it uses all training samples and is more robust to unseen images.
  • Cross-database results: HyperIQA trained on KonIQ10k obtains SRCC values of 0.785 on CLIVE and 0.819 on BID, versus 0.846 and 0.868 for the proposed model.These are the paper’s explicit comparison values.
  • Regressor behavior: The FLIVE patch regressor performs worse because small patches often omit semantic content, causing quality regression to focus mainly on distortions.Regressors trained on the other databases better account for both distortion and content.

E. The Effects of Different Backbones

The staircase structure works with multiple CNN backbones, allowing a trade-off between accuracy and efficiency. Stronger backbones improve performance, while MobileNetV2 remains competitive despite using far fewer parameters.

  • Backbone comparison: ResNext50 outperforms ResNet50 despite having a similar number of parameters, and ResNext101 further improves performance as network depth increases.These comparisons are reported on KonIQ10k with the staircase structure.
  • Backbone comparison: MobileNetV2 uses ten times fewer parameters than ResNet50 while achieving an SRCC value only 0.0123 lower.This supports using the lightweight backbone when faster running time is preferred.
  • Practical trade-offs: The staircase structure can be integrated with popular CNN models, enabling selection of a backbone according to accuracy or runtime requirements.The paper presents this flexibility as a practical advantage of the module.
  • Staircase structure: The staircase structure consists of four convolutional paths that fuse features from different CNN stages into the final representation.The path experiments examine how intermediate-stage features contribute to quality evaluation.
  • Staircase structure: The model addresses BIQA’s need to represent both low-level distortions and high-level content by hierarchically fusing intermediate CNN features.The conclusion links this feature design with the model’s broader BIQA objective.
Loading 2105.14550v3…