PaperlayerView exact revision on arXiv

Source-linked AI summary

MUSIQ: Multi-scale Image Quality Transformer

Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, Feng Yang

arXiv:2108.05997v1cs.CV

TL;DR

IQA models commonly rely on CNNs whose fixed-size batch inputs can distort images through resizing or cropping. MUSIQ instead processes native-resolution, variable-aspect-ratio images through a multi-scale patch Transformer with hash-based spatial and scale embeddings. It reports state-of-the-art performance on three technical-quality datasets and parity with the state of the art on an aesthetic-quality dataset.

  • Problem

    CNN-based IQA models require fixed-size batch inputs, so resizing or cropping images with varying aspect ratios and resolutions can alter composition or introduce distortions.

  • Method

    MUSIQ uses a patch-based multi-scale Transformer with native-resolution and resized inputs, hash-based 2D spatial embedding, and scale embedding.

  • Results

    MUSIQ achieves state-of-the-art performance on PaQ-2-PiQ, KonIQ-10k, and SPAQ, and performs on par with the state of the art on AVA.

  • Takeaways & Limitations

    MUSIQ supports full-size images with varying resolutions and aspect ratios while capturing image quality at multiple granularities.

  • Takeaways & Limitations

    The hash grid trades off expressiveness against trainability: small grids cause collisions, whereas large grids waste memory and may require more diverse resolutions.

Abstract

from arXiv · show

Image quality assessment (IQA) is an important research topic for understanding and improving visual experience. The current state-of-the-art IQA methods are based on convolutional neural networks (CNNs). The performance of CNN-based models is often compromised by the fixed shape constraint in batch training. To accommodate this, the input images are usually resized and cropped to a fixed shape, causing image quality degradation. To address this, we design a multi-scale image quality Transformer (MUSIQ) to process native resolution images with varying sizes and aspect ratios. With a multi-scale image representation, our proposed method can capture image quality at different granularities. Furthermore, a novel hash-based 2D spatial embedding and a scale embedding is proposed to support the positional embedding in the multi-scale representation. Experimental results verify that our method can achieve state-of-the-art performance on multiple large scale IQA datasets such as PaQ-2-PiQ, SPAQ and KonIQ-10k.

1. Introduction

MUSIQ addresses fixed-size CNN preprocessing in IQA by processing native-resolution images with varying aspect ratios and multi-scale representations. It introduces positional mechanisms for multi-scale inputs and reports state-of-the-art or comparable performance across four IQA datasets.

  • CNN-based IQA models require fixed-size inputs, so resizing or cropping can alter image composition, aspect ratio, or distortions and change perceived quality.
  • MUSIQ uses a patch-based Transformer to process full-size images with varying resolutions and aspect ratios without fixed-resolution preprocessing.
  • Its multi-scale representation combines fine-grained detailed patches with coarse-grained global patches to capture image quality at different granularities.
  • Hash-based 2D spatial embedding maps patch positions to a fixed grid, while scale embedding distinguishes patches from different scales and aligns spatially close patches across scales.
  • MUSIQ achieves state-of-the-art performance on PaQ-2-PiQ, KonIQ-10k, and SPAQ, while performing on par with the state of the art on AVA.

2. Related Work

Related work establishes CNN-based IQA as a strong baseline and highlights positional-embedding challenges for variable-resolution vision inputs. Transformer-based representations and multi-scale processing motivate MUSIQ’s design.

  • CNN-based IQA methods have achieved state-of-the-art performance, but typically require fixed-size input processing.
  • MUSIQ’s overview uses native and resized multi-scale images, patch embeddings, spatial and scale embeddings, and Transformer self-attention.
  • Fixed-length positional embeddings are generated for fixed-length sequences and become unsuitable when input resolution changes.

3. Multi-scale Image Quality Transformer

MUSIQ processes native-resolution images with arbitrary aspect ratios and resolutions through a multi-scale patch representation. It combines spatial and scale embeddings so Transformer tokens retain location and scale information.

  • Overall Architecture: MUSIQ handles input images with arbitrary aspect ratios and resolutions while preserving native-resolution processing.The model is designed to bypass fixed-size CNN input constraints.
  • Multi-scale Patch Embedding: The multi-scale representation combines the full-size image with resized variants, enabling aggregation of local details and global composition.Patches from different scales provide information at fine-grained and coarse-grained levels.
  • Multi-scale Patch Embedding: Patches are extracted at fixed size, encoded into D-dimensional embeddings, concatenated across scales, and padded with masks for fixed-length training.Masked self-attention ignores padding tokens, while single-input evaluation can use arbitrary sequence lengths.
  • Hash-based 2D Spatial Embedding: The hash-based 2D spatial embedding maps patch coordinates to a shared G × G learnable grid, preserving spatial information across aspect ratios and scales.Spatial embeddings are added element-wise to patch embeddings without modifying Transformer attention.
  • Hash-based 2D Spatial Embedding: The hash grid trades expressiveness against trainability; the experiments set G = 10 because larger grids produced only small performance differences once sufficiently large.Small grids increase collisions, whereas large grids use more memory and may require more diverse resolutions.
  • Scale Embedding: A learnable scale embedding marks whether each patch comes from the native image or a particular resized variant.This distinguishes information from different scales because the spatial hash is shared across them.

4. Experimental Results

MUSIQ is evaluated on four large-scale IQA datasets and consistently performs strongly across technical and aesthetic quality assessment. Ablations show benefits from native-resolution input, multi-scale representations, aspect-ratio-preserving resizing, and the proposed spatial and scale embeddings.

  • Datasets and setup: MUSIQ is evaluated on PaQ-2-PiQ, SPAQ, KonIQ-10k, and AVA, covering technical and aesthetic image quality.The experiments use full-size images, patches, or dataset-specific splits and preprocessing protocols.
  • Comparison with state of the art: MUSIQ outperforms previous methods on PaQ-2-PiQ, KonIQ-10k, and SPAQ, while achieving the best MSE on AVA.Its PaQ-2-PiQ advantage is especially large on the full-size test set, whose images have at least one dimension exceeding 640.
  • Aspect-ratio-preserving resizing: Aspect-ratio-preserving resizing performs better than square resizing, and MUSIQ detects quality degradation caused by unnatural aspect-ratio changes.Models trained with square resizing are not sensitive to these changes in the same evaluation.
  • Full-size and multi-scale input: MUSIQ-single already achieves promising results, and adding multi-scale inputs further improves performance, especially on PaQ-2-PiQ and AVA.The gains are larger on datasets with more diverse image resolutions.
  • Full-size and multi-scale input: Multi-scale input consistently improves single-scale models and outperforms averaging predictions from individual scales.The Transformer aggregates quality information across scales through its full receptive field over the multi-scale input sequence.
  • Spatial and scale embeddings: Ablations show that spatial embeddings are important, HSE better handles varying aspect ratios and cross-scale locations, and SCE improves scale-information modeling.Learned HSE similarities also correspond to spatially close locations and the image’s 2D structure.

5. Conclusion

MUSIQ processes full-size images with varying resolutions and aspect ratios through a multi-scale representation, using spatial and scale embeddings to encode positional information. Experiments on four large-scale IQA datasets show consistently state-of-the-art performance.

  • MUSIQ handles full-size image inputs with varying resolutions and aspect ratios.
  • The model combines global and local views to capture image quality at different granularities.
  • Hash-based 2D spatial embedding and scale embedding encode positional information in the multi-scale representation.
  • Experiments on four large-scale IQA datasets show that MUSIQ consistently achieves state-of-the-art performance.

A.1. Transformer Encoder Structure

MUSIQ encodes multi-scale patches and positional information as a token sequence for a Transformer encoder. The encoder applies standard Transformer processing to produce successive layer outputs.

  • Input representation: MUSIQ encodes patches from multiple scales, with k = 0 representing the full-size image.
  • Input representation: HSE and SCE are added to patch embeddings to form the multi-scale Transformer input.
  • Input representation: A learnable [class] token embedding is prepended to the sequence of embedded tokens.
  • Encoder computation: The Transformer encoder is described as operating through successive layers indexed from q = 1 to L.
  • Encoder computation: Each encoder-layer output zq is formed from multi-head self-attention applied to the normalized preceding representation, with a residual connection.
  • Encoder computation: Figure 6 illustrates the Transformer encoder structure.

A.2. Multi-head Self-Attention (MSA)

MUSIQ uses multi-head self-attention to project token sequences into query, key, and value representations, compute similarity-based weighted values, and combine parallel heads. Masking suppresses invalid attention positions before softmax.

  • Self-attention: Self-attention projects the input sequence into query, key, and value representations using learnable matrices.
  • Self-attention: Attention weights represent pairwise similarities between queries and keys and determine a weighted sum over values.
  • Multi-head attention: Multi-head self-attention runs s self-attention operations in parallel, concatenates their outputs, and projects the result.
  • Self-attention: Figure 7 illustrates single-head self-attention.
  • Masked attention: An input mask marks effective inputs and ignores padding tokens during batch training.
  • Masked attention: Invalid attention positions receive −inf before softmax, making their attention weights close to zero.

A.4. Different Transformer Encoder Settings

The appendix evaluates MUSIQ variants using different Transformer encoder settings and reports their performance on KonIQ-10k.

  • Encoder settings: The main experiments use a lightweight Transformer encoder setting to make model size comparable to ResNet-50.
  • Encoder settings: The paper reports results from different Transformer encoder settings in addition to the lightweight main configuration.
  • Encoder settings: Table 10 lists MUSIQ variants with different Transformer encoder settings.
  • Performance comparison: Table 11 reports the performance of different MUSIQ variants on the KonIQ-10k dataset.

B.1. Grid Size G in HSE

The HSE grid size balances spatial collisions against sparsity and data requirements. Experiments indicate that rough spatial encoding is sufficient once the grid is large enough, while learnable HSE performs slightly better than fixed sinusoidal encoding.

  • B.1. Grid Size G in HSE: Small G can cause spatial collisions, whereas large G produces sparse hashing that requires more diverse training resolutions.The authors note that larger grids may leave some positions undertrained unless enough resolutions are available.
  • B.1. Grid Size G in HSE: G × G × P × P = H × W relates the grid size, patch size, and image resolution; the experiments use G around 10 to 15 for typical images.The average resolution across four datasets is around 450 × 500, with patch size 32.
  • B.1. Grid Size G in HSE: Different G values change performance little once G is sufficiently large, indicating that rough spatial encoding is adequate for IQA.This conclusion is based on the HSE grid-size ablation on the AVA dataset.
  • B.1. Grid Size G in HSE: Learnable HSE gives slightly better performance than fixed sinusoidal HSE across the compared grid sizes.The comparison is reported for the AVA dataset.
  • B.1. Grid Size G in HSE: Even with G = 5, the learned HSE similarity matrix corresponds well to patch positions, showing that HSE captures spatial location.Figures visualize learned HSE for G = 5 and G = 15.
  • B.1. Grid Size G in HSE: Patch-size ablation finds P = 32 performs well across datasets, while larger fine-tuning l improves performance by reducing image cutoff.Images with higher resolution contain more patches, so too-small l can truncate them.

E. KonIQ-10k More Results

Additional KonIQ-10k experiments evaluate MUSIQ under the same fixed split used by KonCept512, complementing the main evaluation based on averaged random splits. The comparison addresses differences in split protocol and fixed-size requirements.

  • E. KonIQ-10k More Results: MUSIQ is evaluated on KonIQ-10k using the same fixed split as KonCept512, alongside the main results averaged over ten random 80/20 splits.The averaged splits are used to reduce bias, whereas KonCept512 reports a fixed split.
  • E. KonIQ-10k More Results: KonIQ-10k images share the same resolution, while CNN models such as KonCept512 typically require a cherry-picked fixed size.MUSIQ is described as avoiding the fixed-size constraint of CNN models.

F. SPAQ Full-size Results

The SPAQ experiments compare training and evaluation on full-size images with processing images resized so their shorter side is 512. The reported performance differs very little between these settings, while attention visualizations examine how outputs integrate multi-scale inputs.

  • F. SPAQ Full-size Results: MUSIQ trained and evaluated on full-size SPAQ images shows very little performance difference from training and evaluating on images resized to a 512-pixel shorter side.The resized setting follows prior work for a fair comparison, while MUSIQ can operate directly on native-resolution images.
  • F. SPAQ Full-size Results: The default MUSIQ model has about 27M parameters, and the authors leave model speedup for future work.For a 224x224 image, the model requires 8.86 × 10^9 FLOPS and is compatible with efficient Transformer backbones.
  • F. SPAQ Full-size Results: Attention maps are computed by averaging across heads and recursively multiplying attention weights through the layers.This procedure accounts for attention mixing across tokens throughout the network.
  • F. SPAQ Full-size Results: Figure 10 compares output-token attention across the full-size image and ARP-resized images at L = 384 and L = 224.The displayed images are resized for the grid, but the model inputs use three different resolutions.
Loading 2108.05997v1…