Source-linked AI summary
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, Kede Ma
TL;DR
BIQA lacks pristine references and faces realistic distortions that limit common training assumptions. LIQE addresses this with automated vision-language multitask learning over BIQA, scene classification, and distortion identification, and the resulting model outperforms state-of-the-art methods while improving generalizability and MOS realignment.
Problem
Realistic camera distortions invalidate assumptions underlying training strategies that rely on pristine images, simulated distortions, or full-reference quality models.
Method
LIQE represents combinations of quality, scene, and distortion labels as textual templates, derives joint probabilities from visual-textual cosine similarities, and automatically weights multitask losses.
Results
LIQE benefits from scene classification and distortion type identification, outperforms state-of-the-art BIQA methods on multiple IQA datasets, and improves gMAD generalizability and MOS realignment.
Takeaways & Limitations
The proposed multitask learning perspective yields a BIQA model that performs competitively across datasets and realigns quality annotations on a more perceptual common scale.
Abstract
from arXiv · showhide
We aim at advancing blind image quality assessment (BIQA), which predicts the human perception of image quality without any reference information. We develop a general and automated multitask learning scheme for BIQA to exploit auxiliary knowledge from other tasks, in a way that the model parameter sharing and the loss weighting are determined automatically. Specifically, we first describe all candidate label combinations (from multiple tasks) using a textual template, and compute the joint probability from the cosine similarities of the visual-textual embeddings. Predictions of each task can be inferred from the joint distribution, and optimized by carefully designed loss functions. Through comprehensive experiments on learning three tasks - BIQA, scene classification, and distortion type identification, we verify that the proposed BIQA method 1) benefits from the scene classification and distortion type identification tasks and outperforms the state-of-the-art on multiple IQA datasets, 2) is more robust in the group maximum differentiation competition, and 3) realigns the quality annotations from different IQA datasets more effectively. The source code is available at https://github.com/zwx8981/LIQE.
1. Introduction
BIQA must estimate image quality without a pristine reference, but realistic camera distortions undermine common training assumptions. The paper proposes automated multitask learning that transfers auxiliary knowledge from scene classification and distortion identification to BIQA.
- Motivation: BIQA predicts perceptual quality without access to an underlying pristine-quality counterpart.
- Motivation: Realistic camera distortions invalidate assumptions that pristine images, efficiently simulated distortions, and adequate full-reference models are available.These include sensor noise, motion blurring, or combinations of distortions.
- Motivation: Can BIQA benefit from auxiliary knowledge provided by other tasks in a multitask learning setting?
- Motivation: Scene classification favors distortion-insensitive features, whereas BIQA preferentially uses distortion-sensitive features to quantify perceptual quality.Gaussian-blurred parrots can remain recognizable, illustrating the tension between the tasks.
- Approach: LIQE automatically determines parameter sharing and task loss weighting while jointly using BIQA, scene classification, and distortion type identification.It represents combinations of scene, distortion, and quality labels with textual templates and uses vision-language correspondence.
- Results: LIQE outperforms state-of-the-art BIQA methods on multiple IQA datasets and improves gMAD generalizability and cross-dataset MOS realignment.
2. Related Work
Related BIQA work spans deep feature learning and newer learning paradigms, while CLIP has enabled broad vision transfer. LIQE differs by fine-tuning CLIP for automated multitask auxiliary knowledge transfer rather than relying on prompt engineering alone.
- BIQA: Deep learning advanced BIQA through end-to-end optimization of feature extraction and quality regression.
- BIQA: Recent BIQA paradigms include patch-to-picture learning, active learning, unified cross-distortion optimization, meta-learning, continual learning, and perceptual attacks.
- CLIP: CLIP was pretrained on 400 million image-text pairs and demonstrated zero-shot transfer across downstream vision tasks.
- CLIP: CLIP applications expanded to semantic segmentation, object detection, and object sketching.
- Positioning: LIQE uses CLIP for multitask auxiliary knowledge transfer and fine-tunes the model instead of using prompt tuning.The paper reports better quality prediction performance without sacrificing language interpretability.
- Multitask Learning: Multitask learning primarily requires choices about model parameter sharing and loss weighting, which LIQE automates.
3. LIQE from Multitask Learning
LIQE formulates BIQA, scene classification, and distortion identification through a shared vision-language joint distribution. It derives task predictions by marginalization and trains the tasks with fidelity losses whose weights adapt to training dynamics.
- 3.1. Preliminaries: BIQA maps an image to a perceptual quality estimate and also represents quality with five Likert levels: bad, poor, fair, good, and perfect.The predicted quality is the probability-weighted sum of the five levels.
- 3.2. Vision-Language Correspondence: LIQE jointly models scene category, dominant distortion type, and quality level using candidate textual descriptions.The setup contains nine scene categories, eleven distortion types, and 495 candidate descriptions.
- 3.2. Vision-Language Correspondence: A pretrained CLIP image encoder processes cropped sub-images, while its language encoder embeds the 495 candidate textual descriptions.Cropping avoids quality changes that naive resizing may introduce.
- 3.2. Vision-Language Correspondence: LIQE averages visual-textual cosine similarities across cropped sub-images and applies a temperature-scaled softmax to obtain the joint probability over quality, scene, and distortion labels.
- 3.2. Vision-Language Correspondence: Task predictions are obtained by marginalizing the joint distribution, with separate fidelity losses for BIQA, multilabel scene classification, and dominant distortion identification.Scene classification averages binary fidelity losses, while distortion identification uses a multiclass fidelity loss.
- 3.2. Vision-Language Correspondence: The BIQA fidelity target compares image pairs according to their ground-truth quality ordering, while Thurstone’s model estimates the probability that one image is perceived better than another.The fidelity loss measures statistical distance between predicted and target pairwise probabilities.
- 3.2. Vision-Language Correspondence: The final multitask objective linearly combines the three losses, with weights automatically computed from their relative descending rates.The weighting scheme can approach equal weighting when its temperature is sufficiently large.
4. Experiments
The experiments evaluate LIQE against BIQA baselines, cross-dataset generalization, perceptual-scale realignment, and design choices across multitask variants. Results support LIQE’s effectiveness, especially for realistic and challenging distortions, annotation realignment, and joint task learning.
- 4.1. Experimental Setup: Experiments compare LIQE with opinion-unaware and data-driven BIQA methods across six IQA datasets.The evaluation includes separately trained methods and jointly trained models using shared parameters.
- 4.2. Main Results: LIQE outperforms UNIQUE with clear margins on the three datasets containing realistic distortions and on KADID-10k.KADID-10k is described as a challenging synthetic-distortion BIQA benchmark.
- 4.3. Cross-Dataset Evaluation: LIQE and UNIQUE are the only tested models reported to handle both synthetic and realistic distortions well with single parameter sets in cross-dataset evaluation.None of the tested methods performs promisingly on PIPAL, whose algorithm-dependent distortions create significant distributional shifts.
- 4.4. Realignment of Quality Annotations: LIQE improves perceptual-scale realignment across quality annotations from different IQA datasets beyond pairwise learning-to-rank training.The realignment evaluation uses sampled images, monotonic nonlinear mappings, and ten sessions.
- 4.5. Ablation Studies: Dynamic loss weighting improves performance over equal task weightings while avoiding laborious hyperparameter tuning.A five-level Likert scale also performs better than a two-level scale, and the full method outperforms the separate-template variant.
- 4.6. Task Relationship Analysis: Distortion type identification benefits BIQA, while all three tasks achieve the best mSRCC when jointly trained with distortion identification acting as an intermediate bridge.Adding only scene classification neither improves nor impairs BIQA, and BIQA training does not improve the other two tasks.
5. Conclusion
The paper formulates BIQA through vision-language correspondence and multitask learning across BIQA, scene classification, and distortion type identification. LIQE uses designed fidelity losses with dynamic weighting and produces effective BIQA and perceptual-scale realignment results.
- LIQE jointly optimizes image and language encoders for BIQA, scene classification, and distortion type identification across multiple IQA datasets.The method uses three fidelity losses and dynamically weights their sum.
- The learned model realigns mean opinion scores from different datasets in a more perceptually meaningful way.