PaperlayerView exact revision on arXiv

Source-linked AI summary

When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders

Seyedarmin Azizi, Parsa Razmara, Sajjad Ghiasvand, Ardalan Aryashad, Tina Khezresmaeilzadeh

arXiv:2610.35017v1cs.CV

TL;DR

Text-steerable visual encoders let users direct image attention with language, raising the question of whether prompts can induce demographic routing or require the model to refuse. This paper audits SteerViT, traces these failures to steering, and adds SAFESTEER’s abstention output and counterfactual-invariance training. SAFESTEER separates should-point from should-abstain prompts, reduces race routing, and preserves localization and retrieval close to the released model.

  • Problem

    Text-steered pointing can route attention by demographics when prompts provide no visual basis, and the model may need to abstain when a query should not be answered.

    Source: §1 · p. 1

  • Method

    The authors audit SteerViT with counterfactual prompts and introduce SAFESTEER, adding a learned abstention output and counterfactual-invariance training on the same steering parameters.

    Source: §1 · pp. 1–2

  • Results

    SAFESTEER separates should-point from should-abstain prompts at AUROC 0.997, generalizes to held-out words at 0.998, and removes most race routing while keeping localization and retrieval within 1 point.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this benchmark, an explicit abstention output and counterfactual-invariance training address refusal and most race routing without over-refusing visible people or roles.

    Source: §1 · p. 2, §8 · p. 8

    On fully cluttered natural scenes, the fix reduces crime→Black routing only partly (2.6 →1.8 pp), and it never refuses unseen nationality and migration words on real photos.

    Source: §7 · p. 8

Abstract

from arXiv · show

Text-steerable visual encoders condition a frozen Vision Transformer on a natural-language prompt and point at the matching image region, producing prompt-conditioned heatmaps and features. We audit the released SteerViT model and find that its steering interface turns language-driven pointing into demographic profiling. On counterfactual face mosaics and real multi-person photographs, "the person" selects a man more often than the base rate, crime words select Black people more often than occupation words do, "the man" among four women selects the Black woman 75.6% of the time on SocialCounterfactuals, and "the monkey" selects Black people. None of them is in the backbone's unsteered saliency, and the crime association is not in the frozen text encoder; the race effects grow with steering strength and enter through the last two cross-attention layers: giving the prompt to those layers alone reproduces them. Four other grounding models, CLIPSeg, Grounding-DINO, SAM 3, and Qwen2.5-VL, show no crime routing and weaker race routing. The model cannot refuse. Its patch softmax must place its mass somewhere, and its heatmap separates a groundable request ("the police officer") from one that is absent ("the pizza") or that would require a stereotyped demographic inference ("the criminal") only at chance (AUROC 0.52). We introduce SafeSteer, which adds a learned abstention output and counterfactual-invariance training on the same 23M steering parameters. SafeSteer separates should-point from should-abstain prompts at AUROC 0.997, generalizes to held-out sensitive and absent words (0.998; at a 0.5 threshold it refuses 99.8% of trained and 62% of held-out sensitive words) with no over-refusal of visible people on the benchmark, and removes most race routing while keeping localization and retrieval within 1 point of the released model. Code, benchmarks, and the curated mosaic sets will be released.

1. Introduction

This paper audits whether text-steered pointing routes attention by perceived demographics when prompts provide no visual basis, and whether the model can abstain. It traces these failures to steering and introduces SAFESTEER, which adds abstention and counterfactual-invariance training while preserving localization and retrieval.

1. Introduction

  • “The person” selects a man 84.6% of the time, crime prompts favor Black faces over occupation prompts, and “the man” among four women selects the Black woman 75.6% of the time.The audit also finds that “the monkey” selects Black faces and confirms male-default, crime, primate, and masculine readings of Black women on real multi-person photographs.Source: §1 · p. 1
  • The demographic routing arises in steering, not unsteered backbone saliency or the frozen text encoder; the last two cross-attention layers alone reproduce 93% of crime routing and 98% of the masculine reading.Crime terms are absent from training captions, and four other grounding models show no crime routing and weaker race routing.Source: §1 · pp. 1–2
  • The released model cannot abstain: its patch softmax always places mass, and its heatmap distinguishes answerable prompts from absent or stereotype-eliciting prompts at AUROC 0.52.Confidence signals handle object absence better than fairness refusals; SAM 3 and Qwen2.5-VL refuse sensitive prompts only by also refusing visible roles.Source: §1 · p. 2
  • SAFESTEER adds a learned abstention output and counterfactual-invariance training on the same 23M steering parameters, separating should-point from should-abstain prompts at AUROC 0.997 and held-out words at 0.998.It does not over-refuse visible people or roles on the benchmark.Source: §1 · p. 2
  • SAFESTEER removes most race routing while keeping localization and retrieval within 1 point of the released model.It trains the same steering parameters rather than changing the frozen backbone.Source: §1 · p. 2
  • The paper contributes a counterfactual audit protocol and locates demographic routing in the steering adapters’ last two cross-attention layers.Its positioning distinguishes active stereotype injection in steered pointing from inherited disparity and extends abstention to should-not-answer queries about present people.Source: §1 · pp. 2–3, Table 1 · p. 3

2. Related Work

Related work studies bias in vision-language outputs, demographic accuracy disparities in detection and grounding, and abstention for absent referents. This paper distinguishes its focus on demographic routing in text-conditioned pointing and refusal when people are present but a query is underdetermined or stereotype-eliciting.

Bias in vision-language models

  • Prior vision-language audits examine labels or text outputs, while this work studies where a steerable encoder points and attributes the harm to steering rather than a joint embedding space.The paper adopts counterfactual construction from SocialCounterfactuals [16] and PAIRS [12], and intersectional framing from Gender Shades, for localization.Source: §2 · p. 3
  • Earlier bias mitigations project biased directions from text embeddings [10] or learn debiasing prompt tokens [5], unlike this paper’s steering-mechanism focus.Related work also describes gender bias in captioning, text cues overriding visual evidence in VL-BERT [30], and stereotypes in training data [6].Source: §2 · p. 3

Fairness in detection, grounding, and segmentation

  • FACET [14] and segmentation self-audits measure accuracy disparities, whereas this work measures active stereotype injection in prompt-conditioned pointing.The authors state that, to their knowledge, prior work had not audited text-conditioned grounding models for demographic routing.Source: §2 · p. 3

Abstention in grounding

  • Prior grounding abstention handles physically absent objects, while this paper targets should-not-answer prompts about people who are present.Confidence signals handle referential absence but fall short on fairness refusal.Source: §2 · p. 3

Counterfactual invariance

  • Matched counterfactuals keep the scene fixed while changing one attribute, addressing concerns about ill-defined counterfactuals [26].The paper uses these counterfactuals both to measure bias and to supervise its fix.Source: §2 · p. 3

3. Audit Protocol

The audit probes SteerViT with prompts on counterfactual mosaics and real multi-person photographs, measuring heatmap mass and top-1 selections while controlling for position. It separately benchmarks whether abstention scores distinguish prompts that should be answered from those that should be refused.

Model and steering setup

  • SteerViT uses six gated cross-attention layers to steer a frozen DINOv2-B/14 ViT with a frozen RoBERTa-Large prompt encoder; the steering adapters, connector, and head are trained.Scaling each learned gate by steering strength ω interpolates from the frozen backbone at ω=0 to full steering at ω=1.Source: §3 · p. 3

Counterfactual mosaics

  • The counterfactual mosaic audit places four people in a 2×2 grid and uses prompts whose referent is not depicted, with equal heatmap mass as the fair reference.Each set is shown in all four cyclic rotations to remove position priors; the audit also uses uncurated PHASE and Open Images MIAP photographs with per-person boxes.Source: §3 · pp. 3–4

Model and steering setup

  • Because the patch softmax always sums to one, the released model must point somewhere; Figure 2 shows SAFESTEER appending a learned null logit to the patch logits.Demographic routing enters through the last two gated layers, blocks 9 and 11.Source: Figure 2 · p. 4

Counterfactual mosaics

  • The audit measures each person’s heatmap-mass share and whether their cell has the maximum mass, averaging results across rotations.People are drawn from FairFace, SocialCounterfactuals [16], and PAIRS [12], alongside real photographs from PHASE and Open Images MIAP.Source: §3 · p. 4

Prompts and metrics

  • Prompt effects are measured relative to “the person,” with 95% bootstrap confidence intervals and Wilcoxon signed-rank tests corrected across headline prompt families.Bias-only CIS and SAFESTEER use three seeds, reported as mean ± standard deviation.Source: §3 · p. 4

Safety benchmark

  • The 72-prompt safety benchmark separates should-point prompts from absent-object and non-visual sensitive prompts that should be refused.It distinguishes object absence, which a no-object class [20] already covers, from fairness refusal when people are present; abstention is evaluated by AUROC.Source: §3 · p. 4

4. Steering Injects Stereotypes

On counterfactual mosaics and real photographs, SteerViT’s prompts route attention by gender and race, including when the prompt does not visually justify that choice. The effects arise in late steering layers, while other grounding models show weaker race routing.

4. Steering Injects Stereotypes

  • “The person” selects women only 15.4% of the time in mosaics with two women and two men, versus 50% parity; with steering off, it selects women 54.5% on FairFace.Training captions refer to men with gender-neutral words 63.0% of the time; steering amplifies this to 84.6%.Source: §4 · p. 4
  • Crime prompts route attention to Black people more than occupation prompts by 9.2 pp on SocialCounterfactuals, 4.7 pp on FairFace, and 5.0 pp on PAIRS.The crime words do not occur in training captions, and frozen RoBERTa embeddings show no greater association with “Black” than occupation words do (d=−0.04, p=0.55).Source: §4 · p. 5
  • Among four women of four races, “the man” selects the Black woman 75.6% of the time, against 25% chance; “the person” selects her 38.4%.The 7.6 odds ratio remains after controlling for masculinity score, hair length, and background brightness; it is 2.0 on real FairFace faces.Source: §4 · p. 5
  • “The monkey” points at the Black person 12.8 pp more often than “the dog” on SocialCounterfactuals.The passage describes this as reproducing a known denigration association through a localization interface [1].Source: §4 · p. 5
  • The race effects grow with steering strength, and zeroing the ninth cross-attention block reduces crime–occupation routing from 0.092 to 0.005.Giving the prompt only to blocks 9 and 11 reproduces 93% of the crime contrast and 98% of excess “the man” routing; the effects appear across all seven released checkpoints, with DINOv2-B most biased.Source: §4 · pp. 5–6, Table 2 · p. 6
  • Four comparison models show no crime routing and weaker race routing than SteerViT, while SAFESTEER removes most race routing in the released-model comparison.CLIPSeg, SAM 3, and Qwen2.5-VL show stronger occupational gender stereotypes; Table 3 compares released SteerViT, CIS, and SAFESTEER across race-routing measures.Source: Figure 3 · p. 5, §4 · p. 6, Table 3 · p. 6
  • On PHASE, “the person” selects women in 24.5% of 3,096 mixed-gender photos, versus a 48.8% base rate; MIAP shows 36.7% versus 44.4%.With steering off, the heatmap selects women at or above base rates: 58.7% on PHASE and 45.3% on MIAP.Source: §4 · p. 6

5. The Model Cannot Refuse

SteerViT’s patch softmax must assign attention somewhere, so its heatmap provides no reliable refusal signal. Confidence measures can help with absent objects but fall short when refusal is warranted because a prompt requests a sensitive inference.

5. The Model Cannot Refuse

  • The released heatmap separates should-point from should-abstain prompts at AUROC 0.517, near chance.It is more peaked on absent objects than on present people (AUROC 0.31), because the softmax must place its mass somewhere.Source: §5 · p. 6
  • Figure 4 locates most bias in the last two cross-attention layers and shows both race effects rising monotonically with steering strength.At ω=0, the frozen backbone is used without steering.Source: Figure 4 · p. 7
  • On PHASE and MIAP, “the person” selects women in mixed-gender photographs far below their base rate, while SAFESTEER moves selection toward that rate.Source: Figure 5 · p. 7
  • The released model’s pre-softmax logits separate absent objects from present people at AUROC 0.880; with people present, they score 0.757 on the fairness case, while CLIPSeg is at chance (0.480).Source: §5 · pp. 6–7
  • SAM 3 scores 0.852 on the fairness case, but its refusal score falls below 0.5 on 43% of visible-role and 15% of benign-present prompts.Qwen2.5-VL refuses 82% of sensitive prompts but also 51% of visible roles (0.655).Source: §5 · p. 7

6. SAFESTEER

SAFESTEER adds abstention and counterfactual-invariance training to the steering adapters, addressing refusal and demographic-routing failures while preserving localization and retrieval. It generalizes refusal to held-out words and removes most race routing.

  • SAFESTEER adds a learned abstention output and counterfactual-invariance objective to the steering adapters, leaving the frozen backbone untouched.It trains absent-object and non-visual-attribute prompts to abstain, underdetermined prompts to spread mass evenly, and visible roles to receive answers.Source: §6 · pp. 7–8
  • AUROC 0.997 ± 0.001 separates should-point from should-abstain prompts, and held-out words reach AUROC 0.998 ± 0.001.At a 0.5 threshold, it refuses 99.8% of trained and 62% of held-out sensitive words without abstaining on benign-present prompts or visible roles.Source: Figure 6 · p. 7, Table 4 · p. 8, §6 · p. 8
  • SAFESTEER reduces crime−occupation routing from 0.092 to 0.014 and “the man”→Black woman routing from 0.756 to 0.294.On prompts never used in fine-tuning, it removes at least 80% of each race contrast.Source: §6 · p. 8
  • On real photos, the male default moves toward the base rate: 24.5 → 37.6% on PHASE and 36.7 → 42.0% on MIAP.Training-free alternatives either leave the male default and cannot refuse, or break grounding when demographic directions are projected out of text embeddings.Source: §6 · p. 8
  • Referring pointing accuracy is 0.819 ± 0.001 versus 0.826 for the released model, while retrieval precision@1 is 0.962 ± 0.001 versus 0.964.An equal-budget fine-tune on ground-truth masks lowers pointing accuracy to 0.793.Source: §6 · p. 8

7. Limitations

The reported limitations concern natural-scene crime routing, shared adapter capacity, and explicit race-word grounding under the bias-only variant.

  • On cluttered natural PHASE scenes, the fix only partly reduces crime→Black routing, from 2.6 to 1.8 percentage points.The authors suggest box-level supervision on natural images such as FACET may be needed to close the gap.Source: §7 · p. 8
  • Because refusal and invariance share the steering adapters, SAFESTEER leaves more crime routing than bias-only CIS and does not refuse unseen nationality or migration words on real photos.Source: §7 · p. 8
  • Bias-only CIS weakens explicit race-word grounding, including the query “the Black person.”Source: §7 · p. 8

8. Conclusion

SteerViT’s language steering injects social stereotypes into pointing, and it cannot refuse; SAFESTEER removes most race routing and preserves benchmark utility without over-refusing visible people. The conclusion argues that refusal should be a first-class property of steering interfaces.

  • SteerViT’s language steering injects social stereotypes into where the model points, and the model cannot refuse.Source: §8 · p. 8
  • SAFESTEER removes most race routing with no over-refusal of visible people on the benchmark and utility within 1 point of the released model.Source: §8 · p. 8
  • The paper argues that refusing stereotype-eliciting queries should be a first-class property of steering interfaces.Source: §8 · p. 8

A. Audit Protocol and Datasets

The audit uses counterfactual mosaics and real multi-person photos to measure demographic routing, with rotation, within-image comparisons, and statistical tests to assess effects and prompt robustness.

A. Audit Protocol and Datasets

  • Race and gender labels are assigned from appearance or generation prompts, are not self-identifications, and are used only to measure behavioral differences across perceived groups.The datasets are public and used under their terms; synthetic images are released as indices and a build script rather than redistributed face images.Source: A. Audit Protocol and Datasets · p. 10
  • Counterfactual mosaics place four people differing in one protected attribute in a 2×2 grid and average all four rotations to marginalize position.A fair model places 25% of heatmap mass on each person; contrasts compare prompt families’ top-1 selection rates on the same mosaics.Source: A. Audit Protocol and Datasets · p. 10
  • The audit uses balanced FairFace faces, synthetic SocialCounterfactuals siblings, PAIRS counterfactual pairs, PHASE photos, and MIAP photos.Natural-photo contrasts are within-image: each person’s heatmap mass is summed inside their box, with the prompt changing while the photo and people stay fixed.Source: A. Audit Protocol and Datasets · p. 10
  • The benchmark groups bias prompts into crime/suspicion, occupation, wealth, trait, protected-attribute, and non-human families, and tests benign-present, visible-role, absent-object, and non-visual sensitive prompts.The safety benchmark uses 72 prompts; the listed crime words do not occur in the released model’s 2,274,513 training referring expressions.Source: A. Audit Protocol and Datasets · p. 10

B. Statistical Significance of All Findings

C. Prompt-Template Robustness

D. Per-Layer and Steering-Strength Attribution

The attribution analyses localize prompt-driven demographic routing to the final cross-attention layers, rather than the frozen text encoder. Ablations distinguish layers carrying bias from those needed for general grounding.

D. Per-Layer and Steering-Strength Attribution

  • Crime and “the man” race contrasts grow with steering strength, while the “person” male default is non-monotonic.At ω=0, prompt contrasts are zero; the male default reverses at ω=1.0.Source: D. Per-Layer and Steering-Strength… · p. 11
  • Prompt features in blocks 9 and 11 are both sufficient and necessary for almost all demographic routing.Giving those blocks the prompt while earlier layers receive “the person” reproduces the routing; swapping those blocks to “the person” removes it.Source: Table 7 · p. 11, D. Per-Layer and Steering-Strength… · p. 11
  • Zeroing block 9 removes 95% of the crime contrast and 71% of excess “the man” routing while gender grounding stays at least 0.96.General referring pointing drops to 0.664; zeroing block 11 also lowers bias but breaks grounding more broadly.Source: Table 8 · p. 11, D. Per-Layer and Steering-Strength… · p. 11
  • The frozen text encoder does not carry the crime association: crime and occupation prompts have embedding effect size d=−0.04 (p=0.55).The steering adapters map never-trained crime words into the visual stream, consistent with localization to the final two layers.Source: D. Per-Layer and Steering-Strength… · p. 11

E. All Released Checkpoints

The race-routing effects recur across released SteerViT checkpoints and differ from other grounding models’ results. Steering redirects demographic information without erasing it, while training-free mitigation can impair grounding.

F. Steering Redirects and Leaks but Does Not Erase

G. Why Naive Mitigations Fail

  • Lowering steering strength halves some race effects but leaves the male default, while text-embedding projection collapses “the woman” accuracy to 0.046 on SC and 0.061 on FF.The projection makes every prompt point at men, so its near-zero stereotype contrast is an artifact rather than a fix.Source: G. Why Naive Mitigations Fail · p. 12

E. All Released Checkpoints

  • All seven released SteerViT checkpoints show both headline race effects, with DINOv2-B the most biased; CLIPSeg is at the fair value on both.Grounding-DINO and CLIPSeg show no crime routing and stay near chance on the Black-woman selection measure; SAM 3 and Qwen2.5-VL show no crime routing.Source: E. All Released Checkpoints · pp. 11–12, Table 11 · p. 13, Table 10 · p. 13

H. Generalization of the Fix

Counterfactual-invariance training reduces demographic routing on prompts not used as training targets. The reported results extend to held-out words and preserve retrieval performance within the tested range.

H. Generalization of the Fix

  • CIS reduces held-out “the girl”→Asian routing on SC from 0.549 to 0.14±0.01 across three seeds, below chance 0.25.The bias-only variant drives held-out cross-gender prompts near or below chance on both datasets.Source: H. Generalization of the Fix · p. 13, Table 14 · p. 13
  • Across invariance-loss weights λ∈{1, 5, 20}, CORE P@1 remains 0.954–0.955 versus 0.964 for the released model.The reported sweep reduces race spread while leaving retrieval essentially unchanged.Source: H. Generalization of the Fix · p. 13
  • SAFESTEER removes at least 80% of each held-out race contrast, with held-out neutral prompt “the face” an exception.For “the face,” the female share falls further below parity.Source: H. Generalization of the Fix · p. 14, Table 15 · p. 14

I. Full Refusal and Utility Tables

The full evaluations test refusal, utility, and generalization across real photographs, unseen categories, and other model architectures. SAFESTEER’s refusal performance coexists with low over-refusal and pointing and retrieval close to the released model.

I. Full Refusal and Utility Tables

  • SAFESTEER separates should-point from should-abstain prompts, with prompt-bootstrap 95% CIs above the released model’s max-patch-logit proxy across tested categories.It has zero over-refusal on benign-present prompts and visible roles; its pointing and retrieval stay within 1 point of the released model.Source: §I · p. 14, Table 16 · p. 14, Table 17 · p. 14

J. Refusal on Real Photographs

  • On 4,317 PHASE photographs never used in training, refusal transfers to real scenes for trained and held-out words; among never-seen words, only the disabled-person prompt is refused, not seven nationality and migration words.Groundable prompts are refused on 1.2% of photo–prompt pairs, almost all from the bald-person prompt.Source: J. Refusal on Real Photographs · p. 15, Table 18 · p. 15

K. Refusal of Unseen Categories

  • For whole unseen categories, fold A ranks held-out categories at AUROC 0.977 versus 0.976 for trained categories.The fold-A model reaches 0.926±0.010 on never-seen PHASE identity words and retains RefCOCO-family pointing 0.820 versus 0.819 and CORE P@1 0.962 for both.Source: K. Refusal of Unseen Categories · p. 15, Table 19 · p. 15

L. SAFESTEER on Other Backbones; M. SAFESTEER on a Second Model Family: CLIPSeg

N. The Refusal Is Not a Prompt Classifier

O. Gender-Presentation Control

P. Additional Bias Axes

  • Steering also increases stigma toward tattooed people and worsens misgendering for Black women relative to unsteered features.“Ignore the woman” still selects a woman 82.4% on FairFace and 91.5% on SC; Black women’s misgendering error rises from 0.083 to 0.100.Source: P. Additional Bias Axes · p. 18

Q. Qualitative Examples on Real Photographs

  • In each of three illustrative PHASE photographs, “the person” selects a man and “the criminal” selects a different, Black person.The images are examples selected from seven candidate photos with this pattern; rates over all photos are reported separately.Source: Figure 8 · p. 17, Q. Qualitative Examples on Real… · p. 18

Related papers

Loading 2610.35017v1…