PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient Image Quality Assessment with Attention-Panel Decoder

Guanyi Qin, Runze Hu, Yutao Liu, Xiawu Zheng, Haotian Liu, Xiu Li, Yan Zhang

arXiv:2304.04952v1cs.CVcs.LGeess.IV

TL;DR

BIQA must assess image quality without reference images, while small labeled datasets make adapting pretrained representations to quality assessment difficult. DEIQT refines the encoder’s CLS token with a Transformer decoder and adds an attention panel to assess quality from multiple perspectives. It outperforms competing methods across eight benchmark datasets and is reported to improve prediction accuracy, training efficiency, and generalization.

  • Problem

    BIQA lacks pristine reference images in real-world settings, and adapting pretrained semantic features to local quality details through fine-tuning demands substantial data despite labor-intensive image annotation.

    Source: Introduction · pp. 1–2

  • Method

    DEIQT uses a Transformer decoder to adapt the encoder’s CLS token into quality-aware features and an attention panel to evaluate quality from different perspectives.

    Source: Introduction · p. 2

  • Results

    DEIQT outperforms other methods across eight benchmark IQA datasets covering varied image contents and distortion types.

    Source: Introduction · p. 2, Overall Prediction Performance… · p. 6

  • Takeaways & Limitations

    Experiments on eight standard datasets demonstrate DEIQT’s superiority in prediction accuracy, training efficiency, and generalization capability.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Blind Image Quality Assessment (BIQA) is a fundamental task in computer vision, which however remains unresolved due to the complex distortion conditions and diversified image contents. To confront this challenge, we in this paper propose a novel BIQA pipeline based on the Transformer architecture, which achieves an efficient quality-aware feature representation with much fewer data. More specifically, we consider the traditional fine-tuning in BIQA as an interpretation of the pre-trained model. In this way, we further introduce a Transformer decoder to refine the perceptual information of the CLS token from different perspectives. This enables our model to establish the quality-aware feature manifold efficiently while attaining a strong generalization capability. Meanwhile, inspired by the subjective evaluation behaviors of human, we introduce a novel attention panel mechanism, which improves the model performance and reduces the prediction uncertainty simultaneously. The proposed BIQA method maintains a lightweight design with only one layer of the decoder, yet extensive experiments on eight standard BIQA datasets (both synthetic and authentic) demonstrate its superior performance to the state-of-the-art BIQA methods, i.e., achieving the SRCC values of 0.875 (vs. 0.859 in LIVEC) and 0.980 (vs. 0.969 in LIVE).

Introduction

BIQA is challenging because it must assess image quality without pristine references, while fine-tuning on small labeled datasets must adapt semantic features to local quality details. DEIQT addresses this with a decoder that refines the encoder’s CLS token and an attention-panel mechanism for evaluating quality from multiple perspectives.

Introduction

  • DEIQT treats fine-tuning as interpreting abstract pretrained features and uses a Transformer decoder to adapt the encoder’s CLS token for BIQA.The paper describes the CLS token as an abstract characterization of quality-aware features.Source: Introduction · p. 2
  • Decoder self-attention downweights features less relevant to image quality, while cross-attention relates decoder queries to encoder features.The resulting decoder embeddings are intended to be more sensitive to quality-aware image features.Source: Introduction · p. 2
  • A single decoder layer is sufficient for satisfactory performance, keeping DEIQT lightweight.The paper reports that increasing decoder depth leaves performance nearly unchanged.Source: Introduction · p. 2
  • The attention-panel mechanism evaluates image quality from different perspectives, improving model stability while introducing almost no parameters.It mimics subjective evaluation by learning differing “human” opinions and combining their evaluations.Source: Introduction · p. 2
  • DEIQT outperforms competitors across eight benchmark IQA datasets spanning varied image contents, distortion types, and dataset sizes.The paper presents this as evidence of the method’s effectiveness across the benchmark set.Source: Introduction · p. 2

Related Work

Prior BIQA work uses CNNs and Transformers to represent image quality, but both approaches leave opportunities for improved efficiency and quality-aware feature modeling. The paper identifies abstract encoder features as a particular limitation of Transformer-based BIQA.

Related Work

  • CNN-based BIQA uses patches or features from multiple layers to capture local detail and high-level semantics, but these strategies add computational burden.CNN locality bias can also constrain characterization of non-local image features, which BIQA uses alongside local information.Source: Related Work · pp. 2–3
  • Transformer BIQA methods use hybrid CNN-Transformer or pure ViT architectures, with Transformers modeling non-local dependencies among image features.The passage distinguishes hybrid methods, which feed CNN features to the encoder, from pure ViT methods, which feed image patches.Source: Related Work · p. 3
  • Existing Transformer-based BIQA methods use only an encoder, whose abstract features make relations between image features and quality scores difficult to model.The paper argues that additional efforts are needed to derive an optimal image-quality representation.Source: Related Work · p. 3

Data-Efficient Image Quality Transformer

DEIQT uses a Transformer encoder-decoder to turn the encoder’s CLS representation into quality-aware features, with attention-panel embeddings enabling predictions from multiple perspectives. The decoder is lightweight, and the paper reports faster gradient stabilization with it.

Overall Architecture

Quality-Aware Decoder

  • The quality-aware decoder interprets the encoder’s CLS token, using self-attention to form queries and cross-attention to refine image features into quality-aware representations.The refined features are sent to an MLP head for quality prediction.Source: Quality-Aware Decoder · p. 4
  • With the decoder, CLS-token gradient variations decrease faster during training; a one-layer decoder is reported to achieve satisfactory performance while keeping the model small.The decoder is described as compatible with ViT encoder variants and able to use pretrained encoder weights.Source: Figure 3 · p. 4, Quality-Aware Decoder · p. 4

Attention-Panel Mechanism

  • Attention-panel embeddings give decoder members distinct perspectives on image quality; their scores are averaged, and their quality embeddings show extremely low pairwise cosine similarity.The mechanism is modeled on multiple human raters whose scores differ, with the mean opinion score used as the image label.Source: Attention-Panel Mechanism · p. 5

Experiments

DEIQT outperforms other BIQA methods across eight datasets and shows data-efficient learning, faster convergence, and strong cross-dataset generalization. Ablations find contributions from both the decoder and attention panel, with one decoder layer preserving a lightweight design.

Experiments

  • Evaluation covers eight BIQA datasets—four synthetic and four authentic—with median SRCC and PLCC reported across repeated 80/20 train-test splits.The training and testing split is repeated ten times.Source: Benchmark Datasets and Evaluation… · p. 5, Implementation Details · p. 6
  • Low cosine similarity between panel members indicates that they characterize perceptual features from distinct quality-evaluation perspectives.Figure 4 compares models with 6, 12, and 18 panel members.Source: Figure 4 · p. 5
  • Across eight datasets, DEIQT outperforms all 13 state-of-the-art BIQA methods in the overall comparison.Table 1 reports median SRCC and PLCC.Source: Table 1 · p. 6, Overall Prediction Performance… · p. 6
  • In cross-dataset validation, DEIQT performs best on four of five datasets and achieves competing performance on KonIQ.Models are tested on other datasets without fine-tuning or parameter adaptation.Source: Table 2 · p. 6, Generalization Capability Validation · p. 6
  • At 40% training data, DEIQT outperforms all methods on synthetic datasets, while 60% yields competing performance on authentic datasets.With 20% training data, it already matches the second-best method on synthetic datasets.Source: Table 3 · p. 6, Data-Efficient Learning Validation · pp. 6–7
  • On LIVEC and KonIQ, DEIQT reaches satisfactory performance in two epochs and converges significantly faster than ViT-BIQA.Figure 5 plots median SRCC against epochs on the two testing datasets.Source: Figure 5 · p. 7, Data-Efficient Learning Validation · p. 7
  • Both the quality-aware decoder and attention panel contribute to performance, with the decoder contributing more and the panel adding negligible expense.The ablation comparison covers LIVEC and KonIQ.Source: Table 4 · p. 7, Ablation Study · p. 7
  • Increasing decoder depth leaves performance almost unchanged on LIVEC and KonIQ, so the model uses one decoder layer.The authors choose one layer to maintain a lightweight design.Source: Table 5 · p. 7, Ablation Study · p. 7

Conclusion

The conclusion presents DEIQT as a data-efficient Transformer for image quality assessment, using a decoder to extract quality-aware features and an attention panel to mitigate prediction uncertainty. Across eight standard datasets, the authors report gains in prediction accuracy, training efficiency, and generalization capability.

  • DEIQT adapts the CLS token, treated as abstract quality-aware features, into decoder queries and uses cross-attention to decouple quality-aware features from encoder outputs.Source: Conclusion · p. 7
  • The attention-panel mechanism is designed to mitigate prediction uncertainty while adding almost no parameters.Source: Conclusion · p. 7
  • Experiments on eight standard datasets demonstrate DEIQT's superiority in prediction accuracy, training efficiency, and generalization capability.Source: Conclusion · p. 7
Loading 2304.04952v1…