Rethinking Post-hoc Debiasing of Large Pretrained Models

1. Introduction

• Spurious correlations can make classifiers rely on shortcuts, leading to test failures when the correlation breaks.For example, a classifier may rely on color when most white dogs and black cats occur in training data.Source: §1 · p. 1
• Post-hoc methods keep the pretrained backbone frozen but still require the spurious attribute to be identified, by practitioners or language models.The paper characterizes LLM-based attribute proposals as a change of oracle rather than removal of the external specification requirement.Source: §1 · p. 1
• On UrbanCars, LLM-specified methods reach 3.23% worst-group accuracy versus 38.71% for an undebiased linear classifier.Supplying ground-truth attribute descriptions improves accuracy but leaves the methods below the undebiased baseline, as suppressing multiple text-guided subspaces can damage class-discriminative semantics.Source: §1 · pp. 1–2
• The proposed method detects bias-sharing cross-class pairs from cosine similarity and learns a linear map to separate them while maintaining intra-class cohesion.The approach needs no text prompts or attribute supervision and is agnostic to the number or nature of shortcuts.Source: §1 · p. 2
• 81.18% worst-joint accuracy on UrbanCars versus 61.29% for the best competing method; the approach also improves overall accuracy.It achieves state-of-the-art worst-group accuracy across BAR bias-conflicting ratios and scales dynamically on BFFHQ as conflicting samples increase.Source: §1 · p. 2
• The paper evaluates its annotation-free approach across multimodal and vision-only backbones, reporting competitive single-shortcut results and state-of-the-art worst-group performance on the multi-shortcut benchmark.The evaluated backbones include CLIP, SigLIP, and DINOv3.Source: §1 · p. 2

2. Related Work

• Robust-training methods can infer atypical samples from a deliberately biased model’s errors and reweight them, but assume a trainable encoder.Distributionally robust optimization and group-balanced last-layer retraining [6] use group annotations; error-based approaches infer atypical samples.Source: §2 · p. 2
• Most post-hoc methods for frozen vision–language embeddings still need a practitioner or language model to identify the spurious attribute.FairerCLIP requires a named attribute and value prompts; RoboShot and PRISM use LLM-proposed attributes to form embedding directions or projections.Source: §2 · p. 2
• SEMi avoids specifying a spurious attribute, instead suppressing embedding components deemed broadly responsive rather than selectively associated with a target class.It compares responses to neutral scene captions and LLM-generated class-name paraphrases.Source: §2 · p. 3
• Frozen-backbone methods had been evaluated almost exclusively on single-shortcut benchmarks, while UrbanCars exposes failures when two shortcuts coexist.Li et al. term the amplification of reliance on another shortcut after mitigating one the “whac-a-mole” effect; ground-truth bias specifications do not resolve it here.Source: §2 · p. 3
• The proposed post-hoc approach requires neither practitioner nor language-model bias specifications and is reported to outperform existing methods, including on UrbanCars.Table 1 describes method requirements by bias information and text-encoder or prompt dependence.Source: §2 · p. 3, Table 1 · p. 3

3. Problem Setting

• The problem setting treats the bias attribute as latent and unrelated causally to class, and evaluates frozen-encoder features by worst-group and overall accuracy.Bias-sharing cross-class pairs share an attribute value and comprise one bias-aligned and one bias-conflicting sample; training seeks performance when the spurious correlation is absent.Source: §3 · p. 3

4. Motivation

• Post-hoc methods adapt representations while keeping the pretrained backbone frozen, but require the spurious attribute to be specified beforehand.Earlier methods rely on practitioner-written prompts; recent methods use language models to propose attributes and controlled descriptions.Source: §4 · pp. 3–4, Table 2 · p. 4

4.1. Debiasing Performs Below the Baseline

• 3.23% worst-group accuracy for PRISM, 8.06% for RoboShot, and 11.29% for PRISM-mini all trail the undebiased linear classifier’s 38.71%.Overall accuracy also declines, so the result is not a robustness–accuracy trade-off.Source: Figure 1 · p. 3, §4.1 · p. 4

4.2. Expert Bias Specification Is Not Enough

• With oracle prompts, PRISM’s worst-joint accuracy rises from 1.61% to 30.65% with a cosine classifier and from 3.23% to 17.74% with a trained linear head, still below 38.71%.The two shortcut directions have cosine similarity 0.58, and removing one can leave the other partly intact.Source: §4.2 · p. 4
• Overlapping shortcut directions can leave residual reliance, so fixing one bias may worsen another—the whac-a-mole failure described in.The text-derived directions for the two UrbanCars attributes have cosine similarity 0.58.Source: §4.2 · pp. 4–5
• LLM proposals can miss UrbanCars’ image-composition shortcut because it does not correspond to a predominant natural-world observation.The benchmark also includes a more readily identifiable natural association, such as vehicle type and background.Source: §4.2 · p. 5

4.3. Intrinsic Trace of Bias in the Feature Space Geometry

• Cosine similarity is 0.630 for same-class, same-background images, 0.589 when only background changes, and 0.584 when only class changes.Thus, changing the spurious attribute reduces similarity nearly as much as changing class, placing cross-class bias-sharing pairs near same-class pairs.Source: §4.3 · p. 5
• Among selected high-similarity cross-class pairs, 94.8% share at least one shortcut, versus 17.2% of random pairs; only 5.2% share neither, versus 81.9% at random.Pairs sharing both shortcuts rise from 0.9% at random to 47.6% among selected pairs; the bias-sharing proportion increases with the similarity threshold.Source: Table 3 · p. 4, Figure 2 · p. 4, §4.3 · p. 5

5.1. Bias-sharing pair selection

• Cross-class pairs with high frozen-embedding similarity identify bias-sharing pairs; on UrbanCars, 94.8% of selected pairs were bias-sharing versus 18.1% of unconstrained cross-class pairs.The map is trained using this geometric signature rather than a specified bias attribute.Source: §5.1 · p. 6, §5 · p. 6
• Distant same-class pairs anchor within-class geometry, while bias-sharing pairs are separated; both pair sets are computed once from frozen embeddings.The same-class set captures natural within-class variation and some aligned–conflicting pairs, supporting class cohesion and invariance to the spurious attribute.Source: §5.1 · p. 6

5.2. Managing bias-sharing pairs and low similarity same class pairs

• The objective uses one-sided hinge losses to update the map until pair similarities reach their margins; removing the margins costs 4.84 points of UrbanCars Worst Joint accuracy.Thresholds select which pairs move, while margins set how far they move.Source: §5.2 · p. 6

5.3. Overview and Inference

• An identity-initialized linear map W is the only component receiving gradients, and the approach uses image-to-image similarity and class labels without bias annotations or text embeddings.After projection learning, a linear classifier is fit with W fixed; the objective reshapes pairwise similarities in the frozen feature space.Source: §5.3 · p. 6, §5.2 · p. 6

6.1. Setup

• The evaluation covers UrbanCars with two shortcuts, BAR and BFFHQ at multiple bias ratios, and Waterbirds with a background shortcut; metrics include overall and worst-group accuracy.UrbanCars reports Worst BG, Worst CoObj, and Worst Joint; results average three random seeds after validation-based hyperparameter selection.Source: §6.1 · pp. 6–7

6.2. Main results

• 81.18% Worst Joint accuracy on UrbanCars exceeds the best competing method by 19.89 points and CLIP-linear by 42.47 points; overall accuracy rises from 81.18% to 90.59%.Every LLM-attribute method falls below CLIP-linear on all four UrbanCars metrics.Source: §6.2 · p. 7, Table 4 · p. 7
• On BAR, the linear-head method reaches 83.10% worst-group accuracy at 1% bias ratio and 84.51% at 5%; it also leads on BFFHQ and Waterbirds under the reported comparisons.On BFFHQ, linear-head WGA rises from 84.00% at 0.5% to 95.87% at 5%; on Waterbirds, it reaches 80.01%.Source: §6.2 · p. 7, Table 5 · p. 8

6.3. Ablations

• Removing either loss term drops UrbanCars Worst Joint accuracy by at least 22.58 points, while margins add a further 4.84 points over the unbounded objective.The two terms separate bias-sharing cross-class pairs and maintain same-class cohesion.Source: §6.3 · pp. 7–8, Table 6 · p. 8

7. Conclusion

• The bias signal is already encoded in the frozen feature geometry, so explicit spurious-attribute identification is not required.Source: §7 · p. 8
• Figure 4 reports threshold stability on UrbanCars, with each cell showing the mean and spread across four margin settings.The red border marks the selected configuration.Source: Figure 4 · p. 8
• Highly similar cross-class pairs are overwhelmingly bias-sharing, detectable from pairwise similarity and class labels alone.The method uses this signal to separate bias-sharing pairs while maintaining same-class cohesion.Source: §7 · p. 8

1. Implementation Details

• Pair selection runs once over cached frozen embeddings and is the only stage whose cost grows with training-split size.Learning W then operates only on selected pairs.Source: §1 · p. 10
• Training updates only W, initialized at identity, while the frozen encoder receives no gradient.W has about 0.59M parameters versus roughly 300M in the backbone; training uses Adam at 10^-4 learning rate and weight decay.Source: §1 · p. 10
• Under cosine classification, projected image embeddings are compared with frozen class-wise prompt embeddings without fitting a classifier.The linear-head alternative fits logistic regression on projected training features with balanced class weights and holds W fixed.Source: §1 · p. 10
• The push margin is below its threshold and the pull margin above its threshold, so selected pairs are pushed or pulled beyond their respective thresholds.Thresholds and margins are selected per dataset.Source: §1 · p. 10

2. Baselines

• Every method uses the same frozen features and evaluation split; SEMi returns unchanged image embeddings, so it has no linear-head variant.For methods that transform embeddings, the head variant fits logistic regression on rectified features.Source: §2 · p. 10
• The algorithm selects cross-class pairs whose similarity exceeds τpush and forms a capped same-class pull set from pairs below τpull.It initializes W to identity, trains it on selected pairs, then fits a linear head on projected features.Source: §2 · p. 10

2.1. Oracle specifications supplied to PRISM

• The oracle experiment substitutes an expert-written specification of UrbanCars’ background and co-occurring-object shortcuts for language-model proposals.PRISM consumes generated scene descriptions, whereas PRISM-mini uses attribute values to build its orthogonal projection.Source: §2.1 · p. 10
• PRISM descriptions span all eight class-by-background-by-object combinations, varying incidental details while holding the specified attributes fixed.This setup lets the objective attribute contrasts to the specified attributes rather than correlated details.Source: §2.1 · pp. 10–11
• Figures 1 and 2 show threshold stability on BAR at 1% and 5% conflicting ratios, respectively.Each cell reports the mean and spread across four margin settings; the red border marks the selected configuration.Source: Figure 1 · p. 11, Figure 2 · p. 11

3.1. Seed sweep

• Table 2 reports results on BAR, BFFHQ, and Waterbirds over three seeds under both classification rules.The supplied passage describes the table scope but gives no cell values.Source: §3.1 · p. 11, Table 2 · p. 12

3.2. Sensitivity to the selection thresholds

• For every dataset and ratio, a broad region around the selected threshold configuration performs within a few points of the best configuration.Variance concentrates where a threshold leaves very few pairs to work with.Source: §3.2 · p. 11, Figure 3 · p. 11, Figure 4 · p. 11

3.3. Other backbones

• The method’s objective uses image-embedding cosine similarities, so it applies to frozen encoders beyond CLIP.DINOv3 is vision-only and has no text embeddings, so prompt-based comparison methods cannot be applied to it; this method does not use the text side.Source: §3.3 · pp. 11–12, Table 3 · p. 13

3.4. Class imbalance

• Under both 70:30 and 90:10 class ratios, the method performs significantly better than the undebiased reference of 38.71.The shortcut correlations are unchanged, and the result holds under both classification rules.Source: §3.4 · pp. 12–13

3.3. Other backbones

• Accuracy improves on all four metrics for every tested backbone; Worst Joint accuracy rises by 45.16 points on CLIP, 24.20 on SigLIP, and 3.22 on DINOv3.The improvement narrows as the frozen representation becomes less entangled; these are single runs at a fixed seed.Source: §3.3 · p. 12, Table 3 · p. 13

• The frozen encoder is CLIP ViT-L/14@336px, and W transforms its embeddings before the linear head is trained.Source: §4 · p. 13
• Baselines other than CLIP-linear and the proposed method are reproduced using the implementation from.Source: §4 · p. 13
• The evaluation follows, and the proposed method uses the hyperparameter search procedure described in Sec. 1.Source: §4 · p. 13

5.1. Class structure in the projected space

• Across the stated class-imbalance ratios, worst Joint accuracy never falls below twice the undebiased reference of 38.71; rows are single runs at a fixed seed.Source: Table 4 · p. 13
• 0.980 versus 0.754: the probability that a same-class pair is more similar than a cross-class pair rises after projection on UrbanCars training pairs.The same-class and cross-class mean similarities shift from 0.626 and 0.551 to 0.792 and 0.421, respectively; the fraction of cross-class pairs exceeding average same-class similarity falls from 12.7% to 0.45%. These are training-split measurements, not evidence of generalization.Source: Figure 8 · p. 13, §5.1 · p. 13

5.2. Where conflicting samples move

• After projection, conflicting samples sit 11% of the way toward the opposing class along the class direction, down from 47% in frozen space.The visualization retains two of 768 dimensions; neither direction enters the objective.Source: Figure 9 · p. 14, §5.2 · p. 14