PaperlayerView exact revision on arXiv

Source-linked AI summary

Fairness-Aware Test-Time Prompt Tuning

Yoann Launay, Parameswaran Kamalaruban, Tom Kempton, Stuart Burrell, David Sutton

arXiv:2608.25707v1cs.LG

TL;DR

Pretrained VLMs can exhibit unfair subgroup disparities, while existing bias-mitigation methods often require retraining or sensitive attributes. The paper evaluates episodic TTA and proposes FAIRTPT, which balances target confidence with sensitive-attribute uncertainty through soft-prompt tuning. FAIRTPT improves subgroup metrics while retaining accuracy and increasing robustness to hyperparameter choices, with learning-rate control mitigating collapse.

  • Problem

    Episodic TTA improves average accuracy, but its fairness and subgroup-robustness effects are understudied, especially under fully unsupervised deployment constraints.

  • Method

    FAIRTPT jointly minimizes target marginal entropy and maximizes sensitive-attribute marginal entropy through episodic soft-prompt tuning, with learning-rate adaptation for stability.

  • Results

    FAIRTPT retains or improves subgroup metrics across datasets, with aggregated WGA and Bias improving by more than +2.0pp while standard episodic TTA degrades subgroup metrics by more than -2.0pp.

  • Takeaways & Limitations

    Fairness-aware episodic TTA can improve subgroup robustness while retaining accuracy and offering greater hyperparameter robustness than baseline methods.

  • Takeaways & Limitations

    FAIRTPT does not address unmodeled proxies, adds episodic inference-time computation, and can produce unintended behavior when sensitive attributes are mispecified.

Abstract

from arXiv · show

Vision-language models have displayed remarkable capabilities in multi-modal understanding and are increasingly used in critical applications where economic and practical deployment constraints prohibit re-training or fine-tuning. However, these models can also exhibit systematic biases that disproportionately affect protected demographic groups and existing approaches to addressing these biases require extensive model retraining and access to demographic attributes. There is a clear need to develop test-time adaptation (TTA) approaches that improve the fairness characteristics of pretrained models under distributional shift. In this paper, we evaluate how episodic TTA affects fairness in CLIP classification under subpopulation shifts and develop FairTPT, a novel fairness-aware episodic TTA method that jointly minimizes target marginal entropy while maximizing spurious marginal entropy through soft-prompt tuning. We find that standard episodic TTA generally exacerbates disparities between majority and minority groups, that blinding a model to spurious attributes without degrading target performance is inherently challenging, and that excessive blinding can lead to catastrophic forgetting. This model collapse can be prevented by monitoring test-time changes in target loss within the linear regime, while still achieving fairness improvements on reactive data and preserving overall performance. FairTPT outperforms all state-of-the-art episodic test-time debiasing methods and establishes a foundation for robust TTA, which is essential for achieving fairness in practice.

1 Introduction

VLMs enable zero-shot classification when labeled data or fine-tuning is impractical, but can propagate biases that create subgroup disparities. This work studies fully unsupervised episodic TTA and introduces FAIRTPT to improve fairness while preserving accuracy.

  • VLM biases can propagate to zero-shot classifiers, producing performance disparities across sensitive attributes.
  • Existing episodic TTA methods improve average accuracy, but their effects on subgroup robustness and fairness remain largely unexplored.
  • The study targets reduced sensitive-group accuracy disparities, maintained overall accuracy, and lower hyperparameter sensitivity under fully unsupervised episodic adaptation.
  • FAIRTPT minimizes target marginal entropy while maximizing sensitive-attribute marginal entropy through episodic soft-prompt tuning.
  • Standard episodic TTA methods can exacerbate disparities and show high hyperparameter sensitivity, whereas FAIRTPT matches or outperforms baselines in overall and subgroup accuracy.

2 Related Work

Related work spans inference-time adaptation, training-time debiasing, group robustness, and test-time debiasing. These approaches differ in supervision, batching, retraining requirements, and whether they balance fairness with target accuracy.

  • Test-time adaptation of VLMs: Episodic VLM TTA adapts or aggregates predictions per test input, but can be unstable and hyperparameter-sensitive while fairness effects remain understudied.
  • Training-time debiasing of VLMs: Training-time debiasing methods remove sensitive directions or learn adversarial, residual, or contrastive adaptations, requiring model training or fine-tuning.
  • Broader group robustness: Group-robustness methods can use group labels or label-free procedures, but generally require retraining on labeled datasets that is impractical for deployment-time mitigation.
  • Test-time debiasing of VLMs: Test-time debiasing often uses embedding projections, yet methods may require sensitive supervision, batched inputs, or generated attribute descriptions; ORTHCALI is fully episodic and unsupervised.

3 Problem Setup and Background

The setup predicts an unlabeled target attribute from each test image while evaluating an unobserved sensitive attribute. CLIP provides the zero-shot classifier, and TTA adapts prompts or aggregates augmented-view predictions episodically.

  • Problem Setup: Each test instance contains an image, unknown target label, and sensitive attribute, while inference receives only the image.
  • Problem Setup: Fairness is measured through average error, worst-group error, and their difference, the robustness gap.
  • Zero-Shot Classification: CLIP maps images and class-descriptive prompts into a shared embedding space and predicts by cosine similarity followed by softmax.
  • Zero-Shot Classification: Soft prompts combine shared context tokens with label-specific tokens, producing text features for target classes.
  • Test-Time Adaptation: Episodic TTA adapts a pretrained VLM for one image using augmented views without target or sensitive labels, then resets for the next episode.
  • Test-Time Adaptation: TPT minimizes marginal entropy by optimizing prompt embeddings, whereas ZERO skips adaptation and aggregates augmented-view predictions, optionally after confidence filtering.

4 Fairness-Aware TPT

FAIRTPT extends episodic prompt tuning by encouraging confident target predictions and uncertain sensitive-attribute predictions. It adds alternative prompt formulations, multi-objective optimization, and learning-rate control to limit collapse from excessive debiasing.

  • Fairness-Aware TPT: FAIRTPT tunes shared soft context while keeping the VLM weights frozen, using separate target and sensitive-attribute prompt sets.
  • Fairness-Aware TPT: The method minimizes target entropy and maximizes sensitive-attribute entropy to reduce spurious reliance while preserving subgroup robustness and overall accuracy.
  • Fairness-Aware TPT: The fairness weight interpolates between TPT at λfair = 0 and exclusive spurious-attribute uncertainty as λfair approaches infinity.
  • Alternative Formulations: An alternative TS formulation conditions spurious prompts on target labels, while Super TS uses target-dependent contexts.
  • Learning Rate Adaptation: Learning-rate adaptation selects a linear-regime step size and rescales it using target-loss changes to stabilize updates and prevent divergence.
  • Multi-Objective Optimization: Jacobian descent treats target and spurious entropy as potentially conflicting objectives and aggregates their gradients to seek simultaneous improvement.

5 Experiments

Experiments evaluate episodic TTA and fairness-aware alternatives across standard fairness benchmarks, measuring overall and subgroup performance plus hyperparameter sensitivity. Standard episodic TTA often harms subgroup robustness, whereas FAIRTPT variants retain accuracy, improve subgroup metrics, and show greater stability across settings.

  • Experimental setup: Subgroup robustness is assessed with worst-group accuracy, bias, and equalized odds difference alongside overall accuracy.Worst-group accuracy is higher-is-better, while bias and equalized odds difference are lower-is-better.
  • Experimental setup: The evaluation compares TPT, ZERO, ORTHCALI, FAIRTPT, and FAIRTPT (MO) on CELEBA, UTKFACE, FAIRFACE, and WATERBIRDS using overall and subgroup metrics.Each run samples K = 1000 images, and results are averaged over five independent random seeds.
  • Main results: Episodic TTA does not consistently improve accuracy over zero-shot, with changes exceeding +2.0pp in some cases and falling below -2.0pp in others.On aggregate, accuracy remains close to or marginally below zero-shot; ORTHCALI generally retains zero-shot accuracy except on one CELEBA configuration.
  • Main results: Episodic TTA worsens subgroup performance in most configurations, with aggregate declines exceeding -2.0pp across WGA, bias, and EOD.This pattern indicates fairness risks under subpopulation shift even when overall accuracy appears stable.
  • Main results: FAIRTPT and FAIRTPT (MO) retain or improve subgroup metrics across datasets while preserving zero-shot-level accuracy.Aggregated WGA and bias improve by more than +2.0pp, and the methods are comparable to or better than all baselines overall.
  • Hyperparameter sensitivity: FAIRTPT is robust to sufficiently large λfair values, while ELRA prevents accuracy collapse and standard baselines remain sensitive to test-time hyperparameters.For FAIRTPT, λfair values from 100 to effectively ∞ reliably balance objectives; removing ELRA causes notable accuracy and subgroup degradation.

6 Conclusions

The paper evaluates fairness in episodic VLM test-time adaptation and introduces FAIRTPT to improve overall and subgroup accuracy while reducing hyperparameter sensitivity. It also identifies deployment limitations involving attribute specification, proxy reliance, computation, and high-stakes misuse.

  • Existing episodic TTA methods often fail to improve subgroup robustness, amplify disparities, and remain highly sensitive to hyperparameters.
  • FAIRTPT jointly minimizes target-attribute entropy and maximizes sensitive-attribute entropy in a fully unsupervised, label-free adaptation procedure.
  • Combined with lightweight learning-rate adaptation, FAIRTPT mitigates collapse and achieves state-of-the-art or competitive fairness-benchmark performance.
  • Future work includes online adaptation, subgroup calibration, theoretical analysis of entropy-based debiasing, and objectives balancing accuracy, fairness, and stability.
  • FAIRTPT requires manually specified sensitive attributes and is ineffective when those attributes are misspecified.
  • The method does not prevent reliance on unmodeled proxies and adds inference-time computation that may limit latency-critical deployment.

A.1 Methods

The methods compare existing test-time debiasing approaches and summarize episodic VLM adaptation procedures. The framework includes explicit fairness objectives, prompt optimization, and learning-rate adaptation to stabilize inference-time tuning.

  • FAIRTPT-style explicit debiasing can be incorporated into DEYO through a principled min–max entropy objective.
  • Table 2 characterizes methods by explicit debiasing, unsupervised access to unlabeled images, and episodic processing of one image at a time.
  • The procedure includes zero-shot, test-time adaptive, fairness-aware adaptive, and entropic learning-rate-adapter algorithms.
  • Zero-shot classification constructs text features and an image feature, computes similarities, and obtains target-label probabilities from a softmax distribution.

A.2 Datasets

The experimental setup uses fairness benchmarks and prompt-based target and spurious attributes for episodic adaptation. The adaptive procedures compute target and spurious marginal distributions before updating shared soft prompts.

  • The experiments specify target and spurious attributes alongside initial prompt templates required by FAIRTPT and episodic TTA baselines.
  • The fairness-aware algorithm constructs separate target and spurious soft-prompt sets before computing their marginal probability distributions.
  • ORTHCALI forms an orthogonal projection from spurious-prompt embeddings and optimizes it using positive target-matched, spurious-different pairs.
  • FAIRTPT updates shared soft prompts using the target and spurious distributions, optionally applying ELRA before episodic prediction.

A.3 Results

The results section benchmarks hyperparameters, prompts, losses, and method combinations across overall and subgroup metrics. Across datasets, FAIRTPT is comparable to or better than the evaluated baselines, while subgroup robustness can remain sensitive to settings.

  • Tables 6–8 benchmark TPT, ZERO, and ORTHCALI hyperparameters using overall and subgroup-level performance metrics.
  • Tables 9–10 evaluate FAIRTPT across λfair values, with and without ELRA, and under different loss configurations.
  • Tables 11–12 evaluate FAIRTPT combined with ORTHCALI and across additional dataset–attribute configurations.

A.4 Additional Discussion on FAIRTPT

FAIRTPT evaluates fairness and accuracy with ASI and ATC, improves sensitive-attribute indifference with minimal target-confidence impact, and supports multiple sensitive attributes. Its inference-time procedure remains comparable to TPT, but sensitive attributes must be specified by the auditor.

  • Effectiveness Validation via ASI and ATC Metrics: ASI measures normalized sensitive-prediction entropy, while ATC measures one minus normalized target-prediction entropy.
  • Effectiveness Validation via ASI and ATC Metrics: FAIRTPT and FAIRTPT (MO) significantly increase ASI with minimal impact on ATC.The reported values aggregate five random seeds across equally sized datasets, using λfair = 100 for both methods.
  • Hyperparameter Tuning and Runtime Analysis: FAIRTPT has inference-time cost comparable to TPT under identical hardware.Runtime per image is reported on an Nvidia H100 GPU for FAIRFACE (Gender × Race).
  • Hyperparameter Tuning and Runtime Analysis: Unlike ORTHCALI, FAIRTPT does not require additional tuning and is robust to λfair.ORTHCALI requires upfront optimization and tuning λorth with a labeled validation set, whereas FAIRTPT includes automatic learning-rate adaptation.
  • Support for Multiple Sensitive Attributes: FAIRTPT supports multiple sensitive attributes either through separate entropy terms or a joint attribute over their product space.Examples include age, gender, and race, with joint treatment represented as age × gender × race.
  • Limitations: The auditor must specify sensitive attributes at inference, because FAIRTPT does not automatically detect spurious attributes.User-provided or externally inferred factors can be incorporated, and specifying gender restricts debiasing to gender.
  • Experiments using CLIP ViT-B/32 as Base Model: Across datasets with CLIP ViT-B/32, FAIRTPT methods achieve accuracy and subgroup metrics comparable to or better than all baselines.Table 16 reports overall accuracy and subgroup-level metrics including Worst-Group Accuracy, Bias, and Equalized Odds Difference.

B Scaling steps in the probability simplex

The analysis characterizes how probability vectors move under optimizer steps and shows that learning-rate effects depend strongly on the initial softmax output. It uses these bounds to explain the trade-off between prediction collapse and indifference and motivate ELRA as a heuristic.

  • The distance d is a metric on the probability simplex and is convex in both arguments.
  • The smallest perturbation that changes the leading index transfers mass between the largest and second-largest probabilities.The resulting distance is 1/2(p1 − max_i>1 p_i).
  • The maximum distance between relevant probability vectors approaches 1 as the simplex dimension n becomes large.The maximum is attained between a vertex and a vector with equal mass across all n components.
  • Initial confidence provides an upper bound on indifference, while other bounds provide lower bounds for collapse.The analysis therefore recommends choosing the learning rate using both the input and the step size.
  • The optimizer step combines target-entropy and spurious-entropy gradients through normalized method-specific weights.The update is Δtctx = −η∇tctxℓ, with the spurious entropy contribution included in the gradient.
  • The distance traveled depends heavily on the initial softmax output, so no single learning rate straightforwardly equalizes step distances across inputs.ELRA instead heuristically targets a consistent change in target marginal entropy, motivated by the link between calibrated confidence and accuracy.
Loading 2608.25707v1…