PaperlayerView exact revision on arXiv

Source-linked AI summary

MUSIQ: Multi-scale Image Quality Transformer

Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, Feng Yang

arXiv:2108.05997v1cs.CV

TL;DR

Fixed-shape input requirements can distort images used for IQA, while variable-resolution inputs pose positional-encoding challenges. MUSIQ processes native-resolution images at multiple scales and achieves state-of-the-art performance on three technical-quality datasets and performance on par with the state of the art on AVA.

  • Problem

    CNN-based IQA often requires resizing or cropping images to fixed shapes, which can alter composition or introduce distortions in images with varying aspect ratios and resolutions.

    Source: §1 · p. 1

  • Method

    MUSIQ uses a patch-based Transformer on native-resolution images with a multi-scale representation, hash-based 2D spatial embedding, and scale embedding.

    Source: §1 · pp. 1–2

  • Results

    MUSIQ consistently achieves state-of-the-art performance on PaQ-2-PiQ [43], KonIQ-10k [17], and SPAQ, and is on par with the state of the art on AVA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MUSIQ can also be applied to tasks beyond IQA whose labels are sensitive to image resolution and aspect ratio.

    Source: §5 · p. 8

    Hash-grid size trades off expressiveness against trainability; the experiments use G = 10, with rough positional information considered sufficient for IQA.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Image quality assessment (IQA) is an important research topic for understanding and improving visual experience. The current state-of-the-art IQA methods are based on convolutional neural networks (CNNs). The performance of CNN-based models is often compromised by the fixed shape constraint in batch training. To accommodate this, the input images are usually resized and cropped to a fixed shape, causing image quality degradation. To address this, we design a multi-scale image quality Transformer (MUSIQ) to process native resolution images with varying sizes and aspect ratios. With a multi-scale image representation, our proposed method can capture image quality at different granularities. Furthermore, a novel hash-based 2D spatial embedding and a scale embedding is proposed to support the positional embedding in the multi-scale representation. Experimental results verify that our method can achieve state-of-the-art performance on multiple large scale IQA datasets such as PaQ-2-PiQ, SPAQ and KonIQ-10k.

1. Introduction

MUSIQ uses a patch-based Transformer to assess image quality on native-resolution images with varying sizes and aspect ratios, avoiding resizing or cropping that can alter composition or introduce distortions. It combines multi-scale image representation with hash-based 2D spatial and scale embeddings, and the paper reports state-of-the-art performance on three technical-quality datasets and performance on par with the state of the art on AVA.

Introduction

  • Earlier CNN-based full-image strategies used adaptive pooling or resizing, single-image batches, offline feature storage, or aspect-ratio-preserving convolutions that still required resizing and smart grouping.The cited approaches include MNA-CNN, Hosu et al. [16], and Chen et al. [7].Source: §1 · p. 1
  • MUSIQ is a patch-based multi-scale image quality Transformer designed to process native-resolution images with varying aspect ratios and resolutions.Unlike CNN-based batch pipelines that resize or crop images to fixed shapes, this approach processes full-size inputs; Figure 1 contrasts the approaches.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Patch-based Transformers can accept variable-length patch sequences, motivating their use on full-size IQA images without fixed-resolution preprocessing.The paper notes this flexibility is subject to memory constraints; ViT [11] encodes images as fixed-size patch tokens.Source: §1 · p. 2
  • Its multi-scale representation lets self-attention capture fine-grained and coarse-grained image information across different granularities.The authors draw inspiration from the human visual system’s multi-scale processing and prior work using CNN features at different depths [16] [22] [48].Source: §1 · p. 2
  • Hash-based 2D spatial embeddings map patch locations to a fixed grid, while separate scale embeddings identify each patch’s scale in the representation.The shared spatial grid supports alignment of spatially close patches across scales despite varying image aspect ratios and resolutions.Source: §1 · p. 2
  • MUSIQ is reported to achieve state-of-the-art performance on PaQ-2-PiQ [43], KonIQ-10k [17], and SPAQ, and on-par performance on AVA.The paper applies MUSIQ to four large-scale IQA datasets, with AVA representing aesthetic quality.Source: §1 · p. 2

2. Related Work

CNN-based IQA methods achieve state-of-the-art performance but often resize or crop images, while ViT likewise assumes fixed-size inputs. MUSIQ is positioned to process full-size images, and its multi-scale setting motivates positional embeddings beyond fixed-sequence and relative-position approaches.

Image Quality Assessment

  • CNN-based IQA methods achieve state-of-the-art performance, but fixed-size cropping or resizing can affect image quality.Multi-crop ensembles [7] [16] [24] add inference cost; MNA-CNN accepts only one full-resolution image per training batch, Hosu et al. [16] incur extra storage, and Chen et al. [7] cannot handle full-size inputs without resizing and require smart batch grouping.Source: §2 · p. 2

Transformers in Vision

  • ViT [11] resizes images to a fixed square size and uses a fixed number of patches and fixed-length positional embeddings, limiting its use for IQA.The passage presents MUSIQ as a Transformer-based architecture that accepts full-size images for IQA.Source: §2 · p. 2

Positional Embeddings

  • Relative positional embeddings [2] [32] support variable-length inputs but require substantial attention modifications and cannot capture multi-scale positions in MUSIQ’s use case.ViT [11] reports that deterministic and learnable positional embeddings [13] work equally well, but these embeddings are generated for fixed-length sequences and become meaningless when input resolution changes.Source: §2 · p. 3

3. Multi-scale Image Quality Transformer

MUSIQ processes images at native resolution and across multiple scales, using spatial and scale embeddings to represent variable-sized patches before Transformer-based quality prediction.

3.1. Overall Architecture

  • MUSIQ handles arbitrary image resolutions and aspect ratios by encoding multi-scale inputs as tokens with pixel, spatial, and scale information.The encoded sequence is processed by a Transformer; its classification-token output serves as the image representation for quality prediction.Source: §3.1 · p. 3

3.2. Multi-scale Patch Embedding

  • MUSIQ combines native-resolution and ARP-resized images, divides them into fixed-size patches, and uses masked padding to batch variable-length patch sequences.Each patch is encoded by a shared five-layer ResNet and a fully connected layer, rather than linear projection.Source: §3.2 · p. 4

3.3. Hash-based 2D Spatial Embedding

  • HSE maps patch locations to embeddings on a shared G × G grid, aligning nearby patches across scales without modifying Transformer attention.The method uses G = 10; the paper reports that once the grid is large enough, changing its size causes only small performance differences.Source: §3.3 · pp. 4–5

3.4. Scale Embedding

  • SCE adds a learnable embedding identifying whether each patch comes from the native-resolution image or a particular resized scale.A distinct embedding is added element-wise to every patch embedding from each scale.Source: §3.4 · p. 5

3.5. Pre-training and Fine-tuning

  • Pre-training skips square resizing, while IQA fine-tuning preserves image composition and aspect ratio without resizing or cropping, and evaluation uses the original image directly.Pre-training retains random cropping; fine-tuning uses only random horizontal flipping for augmentation.Source: §3.5 · p. 5

4. Experimental Results

MUSIQ outperforms prior methods across four image-quality datasets, while ablations support its use of aspect-ratio-preserving, multi-scale inputs and spatial and scale embeddings. The experiments also examine the datasets, training setup, and patch-encoding choices.

4.1. Datasets

  • Experiments cover PaQ-2-PiQ, SPAQ, KonIQ-10k, and AVA, using the prior-literature splits except for KonIQ-10k’s repeated 80/20 sampling.PaQ-2-PiQ training uses 30k full-size images and 90k patches; SPAQ uses images and MOS without tags, and AVA provides 250k images with 10-scale score distributions.Source: §4.1 · p. 5

4.2. Implementation Details

  • MUSIQ’s default input uses the native-resolution image and two ARP-resized variants, with 32-pixel patches and a 27-million-parameter Transformer pretrained on ImageNet.Fine-tuning uses dataset-specific epochs and learning rates; AVA uses EMD loss with r = 2, while other datasets use L1 loss, and results average 10 runs.Source: §4.2 · pp. 5–6

4.3. Comparing with the State-of-the-art (SOTA)

  • MUSIQ outperforms other methods on PaQ-2-PiQ validation and test sets, KonIQ-10k, and SPAQ, and achieves the best AVA MSE with top SRCC and PLCC.On PaQ-2-PiQ, the full-size test set contains pictures with at least one dimension exceeding 640; on KonIQ-10k, MUSIQ uses a single full-size-image evaluation rather than BIQA’s 25-crop ensemble.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.4. Ablation Studies

  • ARP resizing performs better than square resizing on AVA, and MUSIQ detects quality degradation from unnatural aspect-ratio changes unlike models trained with square resizing.Aggressive resizing lowers perceived quality; Figure 3 shows MUSIQ’s predicted scores responding to the changed aspect ratios while the other square-trained models are not sensitive.Source: §4.4 · pp. 6–7, Table 5 · p. 7, Figure 3 · p. 7
  • Multi-scale input improves on MUSIQ-single, with larger gains on PaQ-2-PiQ and AVA, and outperforms averaging individual scales; attention focuses on fine detail at high resolution and global areas at lower resolution.The gain is larger on PaQ-2-PiQ and AVA, whose images have more diverse resolutions than KonIQ-10k and SPAQ.Source: §4.4 · pp. 7–8, Table 6 · p. 7, Figure 4 · p. 7
  • Spatial embeddings improve performance, HSE outperforms fixed-length positional embeddings for varying aspect ratios and cross-scale localization, and SCE further improves performance by encoding scale independently.HSE similarity reflects 2D image structure, with spatially close locations more similar.Source: §4.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8
  • Simple convolution-based patch encoding boosts performance over linear projection, while adding more convolutional layers has diminishing returns and a 5-layer ResNet provides satisfactory patch representations.The comparison uses PaQ-2-PiQ full-size test results.Source: §4.4 · p. 8, Table 9 · p. 8

5. Conclusion · Appendix for “MUSIQ: Multi-scale Image Quality Transformer” · A. Transformer Encoder

MUSIQ processes full-size images with varying resolutions and aspect ratios by using multi-scale global and local views, with hash-based 2D spatial and scale embeddings for positional information. Experiments on four large-scale IQA datasets show consistent state-of-the-art performance.

5. Conclusion

  • MUSIQ consistently achieves state-of-the-art performance across four large-scale IQA datasets.Source: §5 · p. 8
  • MUSIQ handles full-size image inputs with varying resolutions and aspect ratios.Source: §5 · p. 8
  • Its multi-scale representation combines global and local views to capture image quality at different granularities.Source: §5 · p. 8
  • MUSIQ encodes positional information in its multi-scale representation using hash-based 2D spatial and scale embeddings.Source: §5 · p. 8
  • MUSIQ can apply to tasks whose labels are sensitive to image resolution and aspect ratio.Source: §5 · p. 8
  • MUSIQ is compatible with any Transformer that accepts a sequence of tokens as input.Source: §5 · p. 8

A.1. Transformer Encoder Structure

MUSIQ forms a multi-scale token sequence by adding spatial and scale embeddings to patch embeddings and prepending a learnable class token. It processes this sequence through a classic Transformer encoder with residual self-attention layers.

A.1. Transformer Encoder Structure

  • The encoder uses multi-head self-attention, Layernorm, and MLP layers, with residual connections between layers.The described block structure is illustrated in Figure 6.Source: §A.1 · p. 11
  • MUSIQ adds HSE and SCE to multi-scale patch embeddings, with k = 0 denoting full-size patches, then prepends a learnable [class] token.Here, k is the scale index and n is the patch index within a scale.Source: §A.1 · p. 11
  • At each layer q, MUSIQ applies normalized self-attention to the previous representation and adds a residual connection; L is the total number of layers.The input representation z0 comprises patch embeddings and their spatial and scale embeddings, while zq is the output of layer q.Source: §A.1 · p. 11

A.2. Multi-head Self-Attention (MSA) · A.3. Masked Self-Attention

The appendix defines self-attention through learned query, key, and value projections and extends it to parallel heads; masked self-attention suppresses invalid positions before softmax.

A.2. Multi-head Self-Attention (MSA)

  • Q, K, and V are produced by projecting input sequence z ∈ R^(N×D) with learnable matrices, and attention weights represent pairwise Q–K similarities used to weight V.The projections have inner dimension Dh.Source: §A.2 · p. 11
  • MSA runs s self-attention heads in parallel, concatenates their outputs, and projects the concatenation to the final output.The final projection uses learnable matrix Um ∈ R^(s·Dh)×D.Source: §A.2 · p. 11
  • Dh is typically set to D/s so computation and parameter count remain constant as the number of heads s changes.Source: §A.2 · p. 11

A.3. Masked Self-Attention

  • Masking can ignore padding or restrict attention positions to prevent leakage in causal or temporal prediction.In batch training, the input mask marks effective inputs and ignores padding tokens.Source: §A.3 · p. 11
  • The mask is added to attention weights before softmax, assigning 0 to valid positions and −inf to invalid positions.Source: §A.3 · p. 11
  • Setting invalid-position weights to −inf before softmax makes their resulting attention weights close to zero.Source: §A.3 · p. 11

A.4. Different Transformer Encoder Settings · B. Additional Studies for HSE

The appendix compares MUSIQ Transformer encoder settings, noting that the main experiments use a lightweight encoder to keep model size comparable to ResNet-50. The tested variants perform similarly with ImageNet pretraining, while larger backbones may need more pretraining data to improve.

A.4. Different Transformer Encoder Settings

  • The tested Transformer encoder variants have similar performance when pretrained on ImageNet.The passage identifies MUSIQ-Small as the model used in the main experiments.Source: §A.4 · p. 12
  • The main experiments use a lightweight Transformer encoder setting to keep MUSIQ's model size comparable to ResNet-50.Table 10 presents MUSIQ variants with different Transformer encoder settings.Source: §A.4 · p. 12, Table 10 · p. 12
  • Larger Transformer backbones may need more pretraining data to improve performance.The passage says this may be necessary for larger backbones, but provides no quantitative comparison in the supplied text.Source: §A.4 · p. 12

B.1. Grid Size G in HSE … D. The Maximum Number of Patches (l) from Full-size Image

The ablations examine HSE grid size and encoding, patch size, and the maximum number of full-image patches. They find rough spatial encoding sufficient, learnable HSE slightly better than sinusoidal HSE, and larger fine-tuning patch limits beneficial.

B.1. Grid Size G in HSE

  • Small HSE grids risk spatial collisions, while large grids require more diverse training resolutions; the suggested rule is G × G × P × P ≈ H × W.With average resolution around 450 × 500 and patch size 32, the authors use grid sizes around 10–15.Source: §B.1 · p. 12, G SRCC PLCC SRCC PLCC · p. 12

G SRCC PLCC SRCC PLCC

  • Different HSE grid sizes do not substantially change performance once G is large enough, indicating rough spatial encoding suffices for IQA.The grid-size ablation is on AVA; the authors suggest using G around 10–15 for their average resolution and patch size.Source: Table 12 · p. 12, G SRCC PLCC SRCC PLCC · p. 12

B.2. Sinusoidal HSE v.s. Learnable HSE

  • Learnable HSE gives slightly better performance than fixed sinusoidal HSE [38] across the tested grid sizes.The comparison is reported on AVA.Source: Table 13 · p. 12, §B.2 · p. 12

B.3. Visualization of HSE with Different G

  • Even at G = 5, the learned HSE similarity matrix corresponds well to patch positions, showing that HSE captures image location.Figures 8 and 9 visualize the HSE grids for G = 5 and G = 15, respectively.Source: §B.3 · p. 12, Figure 8 · p. 13, Figure 9 · p. 13

C. Effect of Patch Size

D. The Maximum Number of Patches (l) from Full-size Image

E. KonIQ-10k More Results

On KonIQ-10k, MUSIQ is evaluated using KonCept512’s fixed split and does not require input-size tuning, unlike CNN models constrained by fixed input sizes.

E. KonIQ-10k More Results

  • Unlike CNN models that need a cherry-picked fixed input size, MUSIQ requires no input-size tuning and generalizes across diverse resolutions.KonIQ-10k images share the same resolution, and CNN models such as KonCept512 typically need a fixed size to work well.Source: E. KonIQ-10k More Results · p. 13
  • MUSIQ’s KonIQ-10k results are reported on the same fixed split as KonCept512 [17].The main experiment instead averages results across 10 random 80/20 train-test splits, following BIQA and MetaIQA [50], to avoid bias.Source: Table 16 · p. 13, E. KonIQ-10k More Results · p. 13

F. SPAQ Full-size Results · G. Computation Complexity · H. Multi-scale Attention Visualization

On SPAQ, full-size training and testing yields very little performance difference from the 512-shorter-side setup; the paper also reports MUSIQ’s compute cost and visualizes attention across scales.

F. SPAQ Full-size Results

G. Computation Complexity

  • MUSIQ has around 27M parameters and 8.86 × 10^9 FLOPS for a 224x224 image; ResNet50 has 23M parameters and 3.8 × 10^9 FLOPS.The paper describes these FLOPS levels as comparable.Source: G. Computation Complexity · p. 13
  • Training IQA with MUSIQ takes 0.8 TPUv3-core-days on average.Source: G. Computation Complexity · p. 13
  • MUSIQ is compatible with efficient Transformer backbones such as Linformer and Performer, while model speedup is left for future work.These backbones greatly reduce the original Transformer’s complexity.Source: G. Computation Complexity · p. 13

H. Multi-scale Attention Visualization

  • To visualize multi-scale information integration, attention weights are averaged across heads and recursively multiplied to account for token mixing through all layers.The visualization traces output-token attention to each image in the multi-scale representation, following [11].Source: H. Multi-scale Attention Visualization · pp. 13–14
  • Figure 10 shows attention on the full-size image and ARP-resized images at L=384 and L=224, with model inputs at three resolutions.The displayed images are resized to fit the grid.Source: Figure 10 · p. 15

Related papers

Loading 2108.05997v1…