PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring CLIP for Assessing the Look and Feel of Images

Jianyi Wang, Kelvin C. K. Chan, Chen Change Loy

arXiv:2207.12396v2cs.CV

TL;DR

Assessing image quality and abstract feel remains difficult because conventional measures are loosely tied to language and abstract judgments often require labeled data. This paper uses CLIP with antonym-paired prompts for zero-shot image assessment and reports that it can assess both quality and abstract perceptions to a satisfactory extent.

  • Problem

    Assessing image look and feel remains challenging because abstract attributes are not easily quantified, and a universal understanding could reduce manual labeling and facilitate restoration tasks.

    Source: §1 · p. 1

  • Method

    The paper adapts CLIP to image perception without task-specific finetuning, using paired antonym prompts to reduce linguistic ambiguity.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    CLIP-IQA assesses both image quality and abstract perception to a satisfactory extent, with high correlation to human perception on common IQA datasets and demonstrated fine-grained quality and abstract assessment.

    Source: §1 · p. 2, §2.3 · p. 6

  • Takeaways & Limitations

    These findings support CLIP-based assessment as a basis for further visual-perception research, including work on sophisticated prompts and generalizability.

    Source: §4 · p. 7

    CLIP-IQA is sensitive to prompt choices, and systematic prompt selection remains unresolved.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been developed to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness levels, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional paradigms by exploring the rich visual language prior encapsulated in Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images in a zero-shot manner. In particular, we discuss effective prompt designs and show an effective prompt pairing strategy to harness the prior. We also provide extensive experiments on controlled datasets and Image Quality Assessment (IQA) benchmarks. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments. Code is avaliable at https://github.com/IceClear/CLIP-IQA.

1 Introduction

The paper explores CLIP as a zero-shot way to assess both the quantifiable “look” and abstract “feel” of images, using prompt design to connect visual perception with language.

1 Introduction

  • Image “look” concerns quantifiable attributes such as exposure and noise, whereas “feel” concerns abstract qualities such as emotion and aesthetics.The paper frames universal understanding of both as potentially useful for vision tasks such as restoration.Source: §1 · p. 1
  • Hand-crafted quality features may correlate poorly with human perception, while supervised methods require laborious labels and generalize poorly to unseen attributes.Prior approaches also tend to address either quality or abstract perception, limiting versatility.Source: §1 · pp. 1–2
  • CLIP performance can vary sharply with prompt choice because perception lacks standardized adjectives and linguistic ambiguity can change prompt meaning.For example, “a clean image” may refer to low noise or to the action of cleaning.Source: §1 · p. 2
  • CLIP-IQA pairs antonym prompts to reduce linguistic vagueness and enables visual-perception assessment without task-specific fine-tuning.The paper motivates CLIP by its ability to relate text and visual entities after training on massive image-text pairs.Source: §1 · p. 2
  • CLIP-IQA achieves high human-perception correlation on common IQA datasets and distinguishes feelings in aesthetic benchmarks and user studies.The paper reports that it can assess both overall and fine-grained image quality as well as abstract perception.Source: Figure 1 · p. 1, §1 · p. 2
  • CLIP determines fine-grained quality attributes such as brightness and noisiness on common restoration benchmarks and synthetic data.The reported results indicate it can assess these attributes, as illustrated in Figure 1(a).Source: §1 · p. 2

2 CLIP for Visual Perception

The section adapts CLIP for image perception with antonym prompt pairing and removal of positional embeddings, then evaluates CLIP-IQA on overall and fine-grained quality and abstract attributes.

2.1 Extending CLIP for Visual Perception

  • Antonym prompt pairs cast ambiguous assessment as binary relative similarity, and removing fixed-size positional embeddings further improves human-score correlations, yielding CLIP-IQA.The fixed input-size constraint can cause resizing or cropping to alter image quality; the authors use a ResNet variant to complement removal of positional information.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 Quality Perception

  • CLIP-IQA is comparable to BRISQUE and surpasses other non-learning methods on all three benchmarks, while also outperforming annotation-trained CNNIQA without task-specific training.The comparisons use LIVE-itW, KonIQ-10k, and SPAQ; CLIP-IQA requires no hand-crafted features.Source: Table 1 · p. 4, §2.2 · p. 4
  • CLIP-IQA+ reaches SROCC/PLCC 0.895/0.909 on KonIQ-10k, 0.805/0.832 on LIVE-itW, and 0.864/0.866 on SPAQ after CoOp prompt fine-tuning.CoOp updates the two prompts while keeping network weights fixed; CLIP-IQA+ has subtler cross-dataset performance drops and stores only two prompts per domain.Source: §2.2 · p. 5
  • Without quality annotations, CLIP-IQA scores user-selected attributes, tracking brightness, noisiness, colorfulness, and sharpness changes and separating high- from low-quality restoration results.Synthetic tests used 200 KonIQ-10k images; real-world evaluations covered low-light enhancement, denoising, retouching, and deblurring.Source: Figure 4 · p. 4, §2.2 · pp. 5–6, Figure 5 · p. 5

2.3 Abstract Perception

  • About 80% user-study accuracy across five attributes shows CLIP-IQA's abstract perception ability on AVA, covering complex/simple, natural/synthetic, happy/sad, scary/peaceful, and new/old.The study used 15 image pairs per attribute, each rated by 25 subjects; ranked examples also showed that images could be assessed across multiple abstract dimensions.Source: Figure 7 · p. 6, §2.3 · p. 6, Figure 6 · p. 6

3 Discussion

The discussion examines how prompt and image-encoder choices affect CLIP-IQA, then outlines limitations involving prompt sensitivity, uncommon professional terms, and gaps from task-specific methods.

3.1 Prompt Designs

  • Prompt templates and adjectives affect CLIP-IQA: the authors adopt “[text] photo.”, while “Good/Bad” correlates better with human quality judgments than the other tested adjective pairs.They attribute prompt sensitivity partly to uncommon adjectives and synonyms, and leave exploration of more sophisticated templates for future work.Source: §3.1 · p. 7

3.2 Backbone of Image Encoder

  • With positional embeddings, the Transformer variant outperforms ResNet, but removing them causes a significant Transformer performance drop, supporting their greater importance for Transformers.For ResNet, removing positional embeddings outperforms interpolation; the authors use the removal variant to accept arbitrary-sized inputs.Source: §3.2 · p. 7

3.3 Limitations

  • CLIP-IQA remains sensitive to prompt choices, struggles with uncommon professional terms, and trails task-specific methods because it lacks task-specific architecture designs.The authors propose systematic prompt selection, pretraining with professional terms, and combining task-specific designs with CLIP’s vision-language prior as future directions.Source: §3.3 · p. 7

4 Conclusion

With suitable modifications, CLIP can assess both image quality and abstract perceptions, providing a foundation for further development of CLIP-based perception assessment.

4 Conclusion

  • With suitable modifications, CLIP can understand both image quality and abstract perceptions.The paper investigates CLIP’s effectiveness for perceiving subjective image attributes.Source: §4 · p. 7
  • The study provides a foundation for CLIP-based perception assessment across varied domains.Source: §4 · p. 7
  • Future development could explore sophisticated prompts, better generalizability, and effective adoption of CLIP’s prior.Source: §4 · p. 7

Appendix · A Related Work

Prior image-quality and abstract-perception methods often depend on reference images or human-labeled training data. The paper positions CLIP-IQA as a way to assess overall and fine-grained quality as well as abstract image attributes, while modifying vanilla CLIP to address perception-specific limitations.

A Related Work

  • FR-IQA compares two images for coding and restoration evaluation; PSNR and SSIM are common but weakly correlate with human perception, while LPIPS improves correlation using feature-space distance.FR-IQA is limited to settings where references exist.Source: A Related Work · p. 8
  • NR-IQA assesses images without references; conventional methods model natural-scene statistics with handcrafted degradation-aware features and regression or multivariate Gaussian scoring.Conventional NSS-based approaches are described across both passages.Source: A Related Work · p. 8
  • Learning-based NR-IQA methods outperform conventional approaches, but training on human-labeled benchmarks limits their generalizability and versatility.This limitation motivates using CLIP, trained on large-scale image-text pairs.Source: A Related Work · p. 8
  • CLIP-IQA targets overall image quality, fine-grained quality attributes, and abstract attributes, achieving performance comparable to existing IQA methods and extending assessment to varied image aspects.The paper describes this as a more general task of perception assessment.Source: A Related Work · p. 8
  • Aesthetic assessment and image-emotion analysis commonly require laborious human annotations, whereas CLIP-IQA assesses abstract image aspects without annotation training.The approach explores CLIP vision-language priors to bypass quality labeling.Source: A Related Work · p. 8
  • Prior CLIP applications link text with visual entities, but linguistic ambiguity and CLIP’s constant-shape constraint hinder perception assessment; two modifications make CLIP-IQA outperform vanilla CLIP.The passage does not specify the modifications here.Source: A Related Work · p. 8

B Experimental Settings

Experiments use ResNet-50-based CLIP-IQA, with separate settings for overall quality, fine-grained quality, and abstract perception. Overall quality is evaluated across three NR-IQA datasets using specified test protocols and standard correlation metrics.

Settings for overall quality

  • All experiments adopt ResNet-50-based CLIP-IQA; overall quality is compared with existing NR-IQA methods on KonIQ-10k, LIVE-itW, and SPAQ.KonIQ-10k uses the Hosu et al. (2020) train/test split, LIVE-itW is tested in full following Hosu et al. (2020), and SPAQ uses a randomly selected 20% test set resized to a 512-pixel shorter side; the SPAQ selection differs from Ke et al. (2021) because test-image names are unavailable.Source: B Experimental Settings · pp. 8–9

Comparison methods

  • Non-learning baselines use Hosu et al. (2020) results on LIVE-itW and KonIQ-10k, but are run with official code on SPAQ.Learning-based methods except MUSIQ are trained on KonIQ-10k using their official-code defaults and tested on all three datasets; MUSIQ uses its official KonIQ-10k-trained model because training code is unavailable.Source: B Experimental Settings · p. 9

Prompt settings

Metrics and implementation

  • Overall quality is evaluated with SROCC and PLCC, where higher values indicate greater consistency with human-labeled MOS scores.CLIP-IQA is implemented in PyTorch and run on one NVIDIA Tesla V100 GPU.Source: B Experimental Settings · p. 9

C Overall Quality Assessment on Synthetic Data

On TID2013, CLIP-IQA outperforms all non-learning methods in SROCC and PLCC, while fine-tuned CLIP-IQA+ surpasses learning-based methods.

C Overall Quality Assessment on Synthetic Data

D Synthetic Samples for Fine-grained Quality

CLIP-IQA identifies fine-grained qualities in synthetic data, illustrated by additional image samples in Fig. 8.

D Synthetic Samples for Fine-grained Quality

  • CLIP-IQA identifies fine-grained qualities on synthetic data, as illustrated by the additional samples in Fig. 8.The samples are provided because space constraints prevented their inclusion in Fig. 4 of the main paper.Source: D Synthetic Samples for Fine-grained… · p. 9

E More Quantitative Results for Abstract Perception

CLIP-IQA is reported to perceive five additional abstract attributes on the AVA benchmark, with prompt settings and examples provided for these evaluations.

E More Quantitative Results for Abstract Perception

Related papers

Loading 2207.12396v2…