Source-linked AI summary
Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, Miles Brundage
TL;DR
The paper examines how CLIP’s broad, user-defined zero-shot capabilities complicate the characterization of generalizable vision models and their deployment. Through exploratory bias probes, it finds that class design can shift biases and that CLIP inherits gender biases, supporting broader evaluations that include deployment-critical features. The authors conclude that models should not be judged as better solely by task accuracy.
Problem
CLIP’s flexible zero-shot capabilities create a wide and unpredictable range of uses, making capability-only evaluation insufficient for understanding deployment suitability.
Method
The paper uses exploratory bias probes and experiments on class design, demographic denigration, label thresholds, and deployment-critical evaluation concerns.
Results
CLIP’s biases can shift with class design, and the model inherits gender biases that become especially apparent among lower-probability labels.
Takeaways & Limitations
Evaluating generalizable models requires holistic qualitative and quantitative characterization of capabilities, biases, misuse potential, and other deployment-critical features.
Takeaways & Limitations
The experiments are not comprehensive and are intended to illustrate potential issues and spark further inquiry.
Abstract
from arXiv · showhide
Recently, there have been breakthroughs in computer vision ("CV") models that are more generalizable with the advent of models such as CLIP and ALIGN. In this paper, we analyze CLIP and highlight some of the challenges such models pose. CLIP reduces the need for task specific training data, potentially opening up many niche tasks to automation. CLIP also allows its users to flexibly specify image classification classes in natural language, which we find can shift how biases manifest. Additionally, through some preliminary probes we find that CLIP can inherit biases found in prior computer vision systems. Given the wide and unpredictable domain of uses for such models, this raises questions regarding what sufficiently safe behaviour for such systems may look like. These results add evidence to the growing body of work calling for a change in the notion of a 'better' model--to move beyond simply looking at higher accuracy at task-oriented capability evaluations, and towards a broader 'better' that takes into account deployment-critical features such as different use contexts, and people who interact with the model when thinking about model deployment.
1. Introduction
CLIP and similar generalizable vision models transfer across many tasks without task-specific training data, while allowing users to define classification classes in natural language. Their broad and unpredictable uses create challenges that capability metrics alone do not capture.
- CLIP and ALIGN transfer non-trivially to tasks including OCR, object classification, and geolocalization, often competing with fully supervised baselines without dataset-specific training.Their generalization reduces the need for task-specific training data.
- Users can create their own natural-language classes without task-specific training data, exposing a broad and potentially difficult-to-characterize range of capabilities.This capability resembles challenges associated with other models showing non-trivial zero-shot or few-shot generalization.
- Exploratory probes identify class design as a performance-sensitive challenge and show that CLIP can make niche tasks easier without task-specific training data.The authors highlight these as two challenges posed by CLIP.
- Evaluating such models only with task-oriented capability metrics is insufficient because deployment involves different contexts and people who interact with the model.The paper argues that capability metrics help describe model abilities but do not determine deployment suitability or provide a holistic account of performance.
2. Understanding CLIP’s Usage
CLIP learns visual concepts from image–text pairs and supports flexible zero-shot classification, but its outputs and apparent biases depend strongly on class design, thresholds, and deployment context. Exploratory probes found inherited and demographic biases, while class changes could substantially alter harmful classifications.
- CLIP learns visual concepts from natural-language supervision using image and text encoders trained with contrastive loss.
- Its flexible class design enables zero-shot classification without task-specific training data, but makes output behavior sensitive to chosen categories and wording.
- Bias probes: 4.9% of FairFace images were misclassified into non-human categories, with approximately 14% of Black images and people aged 0–20 receiving such classifications.Confidence intervals for the overall rate were 4.6%–5.4%; other racial groups had rates under 8%.
- Bias probes: 16.5% of male images versus 9.8% of female images were assigned crime-related classes, while people aged 0–20 had an approximately 18% rate.The corresponding rate was approximately 12% for ages 20–60 and 0% for people over 70.
- Bias probes: Adding the class “child” drastically reduced crime-related and non-human classifications for people under 20, showing that class design can change unwanted behavior.
- Bias probes: With combined commercial-system labels, CLIP associated appearance terms more often with women and high-status occupations more often with men, while lower thresholds shifted men’s labels toward “suit” and “tie.”Women were more often assigned labels such as “brown hair” and “blonde,” whereas men were more often assigned “executive” and “doctor.”
- These preliminary experiments show that capability metrics alone miss deployment-critical features, motivating broader evaluation of capabilities, biases, misuse potential, and downstream impacts.