PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Algorithmic Bias from Archival Artifacts: A Controlled Audit of Vision-Language Model Valuation in Metropolitan Museum Archives

Manpreet Singh, Rhythm Bhatia, Rahul Joshi

arXiv:2609.17572v1cs.LG

TL;DR

This study asks whether apparent gender differences in VLM artwork valuation reflect algorithmic bias or archival confounding. It audits CLIP models with zero-shot prompt differentials and multivariate controls, finding statistical gender equivalence but also measurement and survival-bias limits.

  • Problem

    VLM fairness audits in cultural archives must distinguish direct demographic valuation effects from confounders embedded in archival metadata.

  • Method

    The study audits OpenAI CLIP and OpenCLIP on 1,500 Metropolitan Museum records using three prompt pairs, hypothesis tests, equivalence testing, and multivariate OLS controls.

  • Results

    OpenAI CLIP and OpenCLIP showed no statistically significant composite gender disparity, while TOST confirmed equivalence across Cohen’s d ≥0.25 bounds.

  • Takeaways & Limitations

    Fairness assessments in cultural heritage collections require multivariate confound controls, equivalence testing, and archival provenance auditing.

  • Takeaways & Limitations

    Macro-level score equivalence may reflect zero-shot metric insensitivity, while excluding 41.2% of unattributed holdings creates institutional survival bias.

Abstract

from arXiv · show

Auditing vision-language models (VLMs) for societal bias requires distinguishing direct algorithmic valuation disparities from confounders embedded within archival metadata. In this study, we audit Contrastive Language-Image Pretraining (CLIP) models using historical artwork metadata from the Metropolitan Museum of Art Open Access collection (N = 1,500 total objects; N = 743 attributed works: Male n = 534, Female n = 209; n = 618 anonymous). We establish a quantitative audit framework evaluating zero-shot CLIP logit differential scores across three semantic prompt pairs (masterpiece, quality, and influence). Unadjusted evaluations demonstrate high score convergence without a statistically significant main gender effect under OpenAI CLIP (mu_F = -0.0067 vs mu_M = -0.0035, p = 0.1829) or OpenCLIP (mu_F = 0.0171 vs mu_M = 0.0237, p = 0.1224). Two One-Sided Tests (TOST) confirm statistical equivalence across Cohen's d >= 0.25 bounds (pTOST < 0.005). Multivariate OLS regression controlling for artwork medium, creation era, and aspect ratio (R^2 < 0.02) confirms that artist gender has no statistically significant conditional effect (p > 0.20). High residual embedding variance (R^2 < 2%) indicates that global zero-shot valuation metrics operate near an embedding noise floor, showing that broad zero-shot prompt logit differentials are a coarse measurement instrument rather than proving absolute model fairness. We highlight two key caveats: (i) macro-level score equivalence reflects metric insensitivity to fine-grained visual-semantic features and does not preclude localized micro-level visual biases, and (ii) excluding 41.2% unattributed holdings reflects institutional survival bias. These results demonstrate the necessity of multivariate confound control, equivalence testing, and archival provenance auditing when assessing AI fairness in cultural heritage collections.

1 Introduction

The paper asks whether apparent gender disparities in VLM artwork valuation reflect direct algorithmic bias or confounding archival structures. It audits two CLIP regimes with prompt-sensitive scoring, equivalence tests, and multivariate controls, finding convergence while emphasizing confound control.

  • 1 Introduction: The study frames VLM fairness auditing as a problem of separating demographic model effects from archival metadata confounders.Museum classifications and metadata can embed historical power relations, institutional boundaries, and gendered exclusions.
  • 1.2 Research Questions and Core Contributions: The audit evaluates three semantic prompt pairs across two CLIP pretraining regimes using non-parametric tests, confidence intervals, and robust OLS regression.The corpus contains 1,500 records, including 743 attributed works and 618 anonymous or unattributed objects.
  • 1.2 Research Questions and Core Contributions: Unadjusted composite scores showed no significant gender valuation gap under OpenAI CLIP or OpenCLIP.OpenAI CLIP reported μF = −0.0067 versus μM = −0.0035 (p = 0.1829); OpenCLIP reported μF = 0.0171 versus μM = 0.0237 (p = 0.1224).
  • 1.2 Research Questions and Core Contributions: TOST established statistical equivalence across Cohen’s d ≥ 0.25 bounds for both OpenAI CLIP and OpenCLIP.Reported pTOST values were 0.0042 at d = 0.30 for OpenAI CLIP and 0.0024 for OpenCLIP.
  • 1.2 Research Questions and Core Contributions: Controlling for medium, creation century, and aspect ratio left artist gender statistically nonsignificant, while total model fit remained below 2%.The paper interprets the low R^2 as high residual embedding variance and a coarse global valuation instrument.
  • 1 Introduction: The authors conclude that responsible cultural-heritage audits require multivariate confound control alongside raw model-output analysis.The paper connects apparent algorithmic disparities to possible archival curation artifacts rather than standalone model bias.

2 Related Work

Related work presents museum metadata as a biased sociotechnical record and motivates controlled auditing of VLM outputs. The study operationalizes this approach through reproducible Met collection harvesting, demographic enrichment, and representation analysis.

  • 2 Related Work: Museum archives and digitized APIs preserve historical gender, geographic, and institutional representation imbalances rather than neutral collection coverage.Prior surveys report that more than 85% of cataloged artists in major U.S. art museums are male and predominantly Euro-American.
  • 2 Related Work: CLIP aligns visual and textual representations from web-scraped image-text pairs, which can encode stereotypes, demographic biases, and valuation skews.Value-laden prompts such as masterpiece, high quality, and influential operationalize learned cultural associations with subjective judgments.
  • 2 Related Work: Unadjusted VLM comparisons can confuse medium-specific visual scoring with direct demographic bias because artwork attributes correlate with artist gender.Relevant covariates include medium, physical dimensions, historical era, and digitized image resolution.
  • Data Acquisition: The harvesting pipeline collected 1,500 public-domain artwork records from nine Met curatorial divisions under rate limiting, caching, and metadata inclusion criteria.Required fields included objectID, title, medium, objectBeginDate, and a high-resolution visual asset URL.
  • Metadata Enrichment: Conservative demographic enrichment routed ambiguous, unisex, non-Western, and historically difficult names to Unknown, producing 618 unattributed objects.This reduces false-positive demographic assignments but limits catalog coverage.
  • Empirical Representation Audit: Among 882 named attributed works, male artists comprised 70.86% and female artists 29.14%, while female attributions remained a minority across every historical period.Female shares ranged from 24.0% in the 15th century to 33.7% in the 18th century.
  • Evaluation Cohort: Sequential filtering concentrated the final audited cohort in high-status canonical Western oil paintings, constraining how broadly the evaluation can generalize.The final cohort contained 743 works, with 74.43% paintings, after excluding unattributed objects and unindexed image assets.

5. Valuation Scoring

The valuation-scoring framework compares two CLIP model regimes within an expanded audit pipeline. It combines dual-model inference, differential scoring, statistical testing, and medium-aware regression over the attributed evaluation cohort.

  • Dual VLM Inference: The audit compares OpenAI CLIP ViT-B/32 with OpenCLIP trained on LAION-2B.Both models are evaluated as distinct pretraining regimes within the same audit framework.
  • Statistical Analysis: The scoring framework uses non-parametric hypothesis tests and multivariate OLS with heteroskedasticity-robust inference.The listed methods include Mann-Whitney U, bootstrap confidence intervals, and multivariate OLS regression.
  • Audit Framework: The pipeline proceeds from Met data harvesting and demographic enrichment to zero-shot logit differentials, non-parametric testing, and multivariate OLS regression.Figure 3 presents this end-to-end sequence.
  • Medium Stratification: Table 3 cross-tabulates the complete attributed cohort by inferred artist gender and physical artwork medium.The cohort contains 743 attributed works and supports medium-aware modeling of the valuation scores.

4 Methodology: Audit Framework and Metrics

The audit framework compares zero-shot valuation scores across prompt-defined value judgments, model architectures, and artist-gender groups while controlling for archival confounders.

  • 4.1 Evaluated Model Architectures: The study audits two ViT-B/32 dual encoders that differ primarily in pretraining-data curation: OpenAI CLIP uses WIT, whereas OpenCLIP uses LAION-2B.Both architectures use 86M visual and 63M text parameters.
  • 4.1 Evaluated Model Architectures: Prompt probabilities are computed by softmax scaling over visual-text cosine similarities, with τ serving as the learned logit-scale parameter.The models optimize a symmetric contrastive loss over L2-normalized visual and text embeddings.
  • 4.2 Value Prompt Operationalization and Logit Scoring: Zero-shot valuation scores combine prompt-pair probabilities into a composite relative-value metric, where positive scores indicate canonical masterwork alignment and negative scores indicate minor or amateur association.The framework uses three value prompt pairs with neutral baseline prompts.
  • 4.3 Statistical Testing: Gender-group differences are assessed with Mann-Whitney tests, rank-biserial effects, bootstrap confidence intervals, and TOST equivalence bounds rather than null-hypothesis non-significance alone.The equivalence procedure tests bounds defined by ∆E.
  • 4.4 Multivariate Confound Control Regression: Multivariate OLS models estimate gender effects while controlling for artwork medium, creation era, and visual framing, using HC3 robust errors and Huber RLM validation.These controls target structural archival confounders that could otherwise be mixed with demographic attribution.

5 Experimental Results

Across two CLIP models and multiple prompt formulations, attributed artworks show no statistically significant gender valuation disparity; adjusted regressions likewise find no significant gender effect and explain less than 2% of score variance.

  • 5.1 Unadjusted Gender Disparity Analysis: OpenAI CLIP and OpenCLIP show high score convergence without statistically significant gender differences, with TOST confirming equivalence across Cohen’s d ≥0.25 bounds.OpenAI CLIP means are −0.0067 versus −0.0035 (p = 0.1829), while OpenCLIP means are 0.0171 versus 0.0237 (p = 0.1224).
  • 5.2 Prompt Sensitivity and Robustness: Prompt-set analyses find no statistically significant gender disparities across architectures, with p > 0.092 for all raw comparisons and adjusted p-values of 1.0000 except 0.5532.The six prompt-architecture comparisons were corrected using αadj = 0.00833.
  • 5.3 Multivariate Confound Analysis: The full regression models explain less than 1.8% of score variance and are not statistically significant overall, indicating that the measured valuation scores capture little variation from the included covariates.Model fit is R2 = 0.018 for OpenAI CLIP and R2 = 0.017 for OpenCLIP.
  • 5.3 Multivariate Confound Analysis: Multivariate OLS models controlling for medium, creation century, and aspect ratio find no significant conditional gender effect: B = 0.0037, p = 0.202 for CLIP and B = 0.0065, p = 0.356 for OpenCLIP.Robustness checks pooling sparse sculpture observations preserve the primary demographic conclusion.

6 Discussion

The discussion argues that cultural-heritage VLM audits must separate demographic model effects from archival and visual confounders. Results indicate broad zero-shot metrics have low explanatory power and can miss localized bias, while dataset filtering and model scope constrain generalization.

  • Unconditioned demographic comparisons can confound creator gender with medium, era, and image characteristics, making multivariate controls necessary.The discussion identifies Simpson’s paradox as a risk when covariates are unevenly distributed across demographic cohorts.
  • R^2 = 0.018 for CLIP and R^2 = 0.017 for OpenCLIP, with structural covariates explaining less than 2% of zero-shot score variance.Both full multivariate models were non-significant overall.
  • 98.2% residual variance places broad prompt differentials near an embedding noise floor, so score equivalence indicates instrument insensitivity rather than definitive neutrality.Scalar prestige prompts compress subtle features such as gaze, lighting, composition, and texture.
  • Filtering out 41.2% of records for unattributed creators yields a survival cohort concentrated in canonical Western oil paintings and excludes archival strata where creators were often anonymized.The remaining named works therefore cannot represent the full institutional collection or upstream archival erasure.
  • OpenAI CLIP and OpenCLIP remained stable across masterpiece, quality, and influence prompts despite different pretraining regimes, although OpenCLIP showed higher variance.Reported standard deviations were σ = 0.0886 for OpenCLIP versus σ = 0.0361 for OpenAI CLIP.
  • Museum AI governance should formalize controls for automated indexing and discovery because uncalibrated rankings can reinforce historical representational imbalances.The proposed governance agenda includes provenance standards, multivariate confound controls, and pretraining-curation audits.
  • The audit’s primary empirical scope is limited to ViT-B/32, while larger encoders may alter residual noise and sensitivity to subtle attributes.The paper recommends benchmarking parameter scales from 86M to 632M and expanding to other architectures and repositories.

7 Conclusion

The study finds no statistically significant gender disparity in broad zero-shot CLIP valuations after auditing Metropolitan Museum records and controlling for archival covariates. It cautions that this macro-level parity does not establish absolute fairness because metric insensitivity and archival survival bias remain.

  • OpenAI CLIP showed no significant composite gender disparity, while OpenCLIP produced consistent baseline stability; TOST confirmed equivalence within Cohen’s d ≥ 0.25 bounds.OpenAI CLIP reported p = 0.1829 and pTOST = 0.0042; OpenCLIP reported p = 0.1224 and pTOST = 0.0024.
  • Artist gender had non-significant conditional effects in multivariate OLS, while overall model R^2 remained below 2%, placing global metrics near an embedding noise floor.The reported coefficients were B = 0.0037, p = 0.202 for CLIP and B = 0.0065, p = 0.356 for OpenCLIP.
  • Macro-level score equivalence does not preclude localized micro-level visual biases, and excluding 41.2% unattributed holdings introduces institutional survival bias.Named creators represent a subset already shaped by historical acquisition and cataloging filters.

Appendix A Qualitative Case Audits of Archival and Prompt Artifacts

The appendix uses qualitative case audits to show how digitization artifacts and prompt taxonomies can shape apparent valuation signals. Conditioning on physical attributes and testing prompt robustness clarify whether these effects produce systematic demographic shifts.

  • Appendix A Qualitative Case Audits of Archival and Prompt Artifacts: The appendix frames these comparisons as qualitative context for the quantitative regression findings, rather than as a separate population-wide fairness estimate.
  • A.1 Case Audit 1: Photographic Framing and Surface Texture Artifacts: Photographed 3D sculptures and 2D paintings differ in lighting, backgrounds, and surface presentation, creating medium-related visual artifacts.
  • A.1 Case Audit 1: Photographic Framing and Surface Texture Artifacts: Medium and aspect-ratio conditioning links photographic presentation artifacts to the broader confound-control strategy.
  • A.1 Case Audit 1: Photographic Framing and Surface Texture Artifacts: B = +0.0002, p = 0.716 after conditioning on physical medium and aspect-ratio framing, indicating that the audited photographic artifact was eliminated in multivariate OLS.
  • A.2 Case Audit 2: Fine Art vs. Decorative Craft Taxonomy Disparities: Prompt Set 3 operationalizes artistic status as the probability difference between “a groundbreaking artwork” and “a decorative craft object.”
  • A.2 Case Audit 2: Fine Art vs. Decorative Craft Taxonomy Disparities: The fine-art versus decorative-craft comparison tests whether taxonomy-sensitive prompts alter valuation patterns across the attributed sample.
  • A.2 Case Audit 2: Fine Art vs. Decorative Craft Taxonomy Disparities: p > 0.10 across prompt-pair robustness checks indicates no systematic demographic valuation shift from prompt phrasing among 743 attributed works.
Loading 2609.17572v1…