PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring CLIP for Assessing the Look and Feel of Images

Jianyi Wang, Kelvin C. K. Chan, Chen Change Loy

arXiv:2207.12396v2cs.CV

TL;DR

The paper addresses the difficulty of measuring both image quality and abstract visual feel without laborious task-specific labeling. It adapts CLIP with antonym prompt pairing for zero-shot perception assessment, finding meaningful performance across quality and abstract-perception tasks while identifying remaining performance gaps.

  • Problem

    Image quality is loosely coupled with human language, while abstract visual feel generally requires supervised models trained on laboriously collected labels.

  • Method

    The paper adapts CLIP for zero-shot visual perception assessment using antonym prompt pairing, without task-specific fine-tuning.

  • Results

    CLIP assesses both quality and abstract perception, achieving human-aligned results across IQA, aesthetic, and user-study evaluations.

  • Takeaways & Limitations

    CLIP provides a basis for visual perception assessment across quality and abstract attributes, supporting further work on prompts, generalizability, and CLIP-prior adoption.

  • Takeaways & Limitations

    Performance gaps remain between CLIP-IQA and task-specific methods because CLIP-IQA lacks task-specific architectural designs.

Abstract

from arXiv · show

Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been developed to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness levels, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional paradigms by exploring the rich visual language prior encapsulated in Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images in a zero-shot manner. In particular, we discuss effective prompt designs and show an effective prompt pairing strategy to harness the prior. We also provide extensive experiments on controlled datasets and Image Quality Assessment (IQA) benchmarks. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments. Code is avaliable at https://github.com/IceClear/CLIP-IQA.

1 Introduction

The paper frames image perception as a long-standing challenge spanning quantifiable visual quality and abstract visual feel. It explores CLIP as a task-specific-training-free approach, using antonym prompts to assess both dimensions and reporting broad evidence of human-aligned performance.

  • Image perception combines quantifiable look attributes, such as exposure and noise, with abstract feel attributes, such as emotion and aesthetics.
  • Hand-crafted quality features have limited general correlation with human perception, while learned methods require laborious labels and remain task-specific.
  • CLIP offers a potential alternative because image-text pretraining builds semantic relationships between text and visual entities without explicit task training.
  • Prompt selection is challenging because CLIP is prompt-sensitive, perception lacks standardized adjectives, and linguistic ambiguity can make assessment volatile.
  • The proposed antonym prompt pairing enables direct CLIP-based assessment without task-specific fine-tuning, covering overall and fine-grained quality as well as abstract perception.
  • Experiments report high correlation with human perception on common IQA datasets and successful discrimination of differing feelings on aesthetic benchmarks and user studies.

2 CLIP for Visual Perception

The paper adapts CLIP for visual perception by pairing antonym prompts and removing positional embeddings, enabling quality and abstract-perception assessment. CLIP-IQA correlates with human judgments across IQA benchmarks and identifies fine-grained quality attributes.

  • Overall quality: CLIP-IQA with the proposed modifications obtains higher SROCC and PLCC correlations with human MOS scores on KonIQ-10k and LIVE-itW than the single-prompt approach.Figure 2 reports SROCC/PLCC, with higher values indicating better correlation.
  • Prompt design: Antonym prompt pairing reduces linguistic ambiguity by casting perception assessment as binary classification with a relative similarity score.A larger score indicates a closer match to the first prompt's attribute.
  • Prompt design: Removing positional embeddings relaxes the fixed image-size constraint because perception assessment primarily captures relationships between images and descriptions.The authors conjecture that positional embeddings have minimal effect for this task.
  • Overall quality: Without hand-crafted features or task-specific training, CLIP-IQA is comparable to BRISQUE and surpasses other non-learning methods on three NR-IQA benchmarks.The benchmarks are LIVE-itW, KonIQ-10k, and SPAQ; it also outperforms CNNIQA, which requires annotated training data.
  • Abstract perception: CLIP-IQA also assesses abstract attributes such as complex/simple, natural/synthetic, happy/sad, scary/peaceful, and new/old using antonym prompt pairs.The selected images illustrate different abstract-attribute scores, including overlapping attributes such as complexity, synthetic appearance, and age.

3 Discussion

The discussion examines how prompt design and image-encoder backbones affect CLIP-IQA, then identifies unresolved limitations involving prompt sensitivity, uncommon terminology, and task-specific performance gaps.

  • 3.1 Prompt Designs: Prompt templates and adjective choices significantly affect CLIP-IQA performance, with “Good/Bad” outperforming “High quality/Low quality” and “High definition/Low definition” for overall quality.The adopted template is “[text] photo.”, and the authors conjecture that uncommon adjectives perform worse.
  • 3.2 Backbone of Image Encoder: The Transformer variant suffers a significant performance drop when positional embeddings are removed, whereas positional information is less crucial for the ResNet variant.The result supports the authors’ hypothesis that positional embeddings are more important in Transformers.
  • 3.2 Backbone of Image Encoder: For ResNet variants, removing positional embeddings performs better than interpolating them, motivating removal to support arbitrary-sized inputs.The authors conjecture that interpolation introduces inaccurate positional embeddings.
  • 3.3 Limitations: CLIP-IQA remains sensitive to prompt choices, leaving systematic prompt selection as an unresolved limitation and future research direction.The authors specifically identify improved prompt design as important for performance.
  • 3.3 Limitations: CLIP recognizes common conversational language more readily than uncommon professional terms such as “Long exposure,” “Rule of thirds,” and “Shallow DOF.”The authors suggest pretraining with additional vision-language pairs could attenuate this problem.
  • 3.3 Limitations: CLIP-IQA retains performance gaps relative to task-specific methods because it lacks task-specific architectural designs.The authors propose combining task-specific designs with vision-language priors as a future direction.

4 Conclusion

The conclusion finds that suitably modified CLIP can perceive both image quality and abstract attributes. It positions the study as groundwork for future improvements in prompts, generalizability, and CLIP-prior adoption.

  • 4 Conclusion: With suitable modifications, CLIP can understand both quality and abstract perceptions of an image.The study presents this capability as its central finding.
  • 4 Conclusion: The study aims to support future development of sophisticated prompts, better generalizability, and effective adoption of CLIP priors.These directions are stated as areas that the study could motivate.

A Related Work

Related work spans hand-crafted, learning-based, and CLIP-oriented approaches to image perception assessment. Existing methods face limitations in reference requirements, labeling burden, task specificity, and generalizability.

  • Quality perception: Image Quality Assessment methods address image quality using full-reference and no-reference settings, with no-reference methods evaluating arbitrary images without references.Conventional no-reference methods mainly rely on natural scene statistics.
  • Quality and abstract perception: The paper broadens assessment beyond overall quality to fine-grained and abstract attributes using CLIP trained on large-scale image-text pairs.CLIP-IQA is reported to achieve comparable performance to existing IQA methods while extending to various image aspects.

B Experimental Settings

The experiments evaluate CLIP-IQA variants across standard quality datasets, fine-grained attribute prompts, and established IQA metrics, with supplementary comparisons using official implementations and models.

  • General settings: All supplementary experiments use the ResNet-50-based CLIP-IQA.This backbone is adopted throughout the reported settings.
  • Overall quality: Overall-quality evaluation compares CLIP-IQA with existing no-reference IQA methods on KonIQ-10k, LIVE-itW, and SPAQ.The datasets cover realistic camera distortions and smartphone photography settings.
  • Overall quality: Non-learning-based baselines use published results where available or official code on SPAQ, while most learning-based methods train on KonIQ-10k and test across three datasets.MUSIQ is evaluated using its official model trained on KonIQ-10k.
  • Fine-grained quality: Fine-grained quality experiments use five paired prompts for the evaluated attributes.The supplementary material refers to prompt settings for the fine-grained attributes in the main paper.
  • Evaluation metrics: Overall-quality performance is evaluated with SROCC and PLCC, where higher values indicate greater consistency with human-labeled MOS scores.Both metrics are standard measures used for IQA evaluation.

C Overall Quality Assessment on Synthetic Data

On the synthetic TID2013 dataset, CLIP-IQA outperforms non-learning methods, while CLIP-IQA+ surpasses learning-based methods.

  • CLIP-IQA outperforms all other non-learning methods on TID2013 in SROCC and PLCC.
  • CLIP-IQA+ surpasses learning-based methods on TID2013 when using the proposed fine-tuning scheme.

D Synthetic Samples for Fine-grained Quality

Additional synthetic examples show that CLIP-IQA can identify fine-grained image qualities, supporting its use beyond overall quality assessment.

  • CLIP-IQA identifies fine-grained qualities on synthetic data in the additional image samples.

E More Quantitative Results for Abstract Perception

Additional AVA experiments show that CLIP-IQA can perceive further abstract attributes beyond those evaluated in the main paper, while synthetic examples illustrate fine-grained attribute assessment.

  • CLIP-IQA perceives five additional abstract attributes on the AVA benchmark.These results extend the five attributes reported in the main paper: complex/simple, natural/synthetic, happy/sad, scary/peaceful, and new/old.
  • CLIP-IQA shows positive correlations with changes in fine-grained attributes on synthetic data at different input scales.The displayed scores correspond to the attribute extents for each image.
  • Examples on AVA show CLIP-IQA identifying multiple abstract attributes, including the five attributes reported in the supplementary results.
Loading 2207.12396v2…