PaperlayerView exact revision on arXiv

Source-linked AI summary

Guilt by Association: Semantic Leakage Drives Associative Gender Bias

Hazel Chen, Peter West, Hila Gonen

arXiv:2610.28944v1cs.CL

TL;DR

The paper asks whether semantic leakage and gender bias require a concept to be attributed to the entity being queried. It tests mismatched entities and gendered concepts that are either engaged with or merely present in the scene, finding leakage in the first setting and stereotype-directed shifts in both gender conditions.

  • Problem

    Prior work on semantic leakage and gender bias typically studies concepts attributed to the queried entity, leaving open whether context alone can steer generation.

    Source: §1 · p. 1

  • Method

    The paper tests semantic leakage with prompts that mismatch the person associated with a concept and tests gendered concepts under direct-engagement and scene-only conditions.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    Leakage persists under entity mismatch, while both direct and indirect gendered-concept conditions shift predictions toward the stereotype; the indirect shift retains 73% of the direct shift.

    Source: §1 · p. 1, §2.4 · p. 3, §3.4 · p. 4

  • Takeaways & Limitations

    The findings connect semantic leakage with associative gender bias and indicate that bias evaluations should account for concepts present in context, not only those attributed to the subject.

    Source: §1 · p. 2

    The evaluation focuses on binary gender and a curated set of associated concepts, and does not cover other bias dimensions such as race, age, religion, or socioeconomic status.

    Source: §5 · p. 5

Abstract

from arXiv · show

Semantic leakage occurs when a concept in the prompt steers model generation about an entity, even though the concept is irrelevant to it (e.g., a person who likes yellow is said to work as a school bus driver). It has mostly been studied in settings where the concept is directly attributed to the entity the model is asked about. This leaves open whether leakage is a flawed heuristic, where the model builds a profile of that entity, or a much broader associative tendency, e.g., working across seemingly unrelated entities. We argue for the latter: even after removing the attribution of the concept to the specific entity, leakage persists. Next, we extend this idea to gender bias. While work on gender bias in LMs typically looks at cases where the stereotyped concept is attributed to an entity, our findings suggest that a concept may steer outputs with its mere presence in the scene. We show this using gendered concepts under two conditions: when the subject engages with the concept, and when the concept just sits in the scene. We find that both shift model generation toward the stereotype to a similar extent. Together, our results suggest that semantic leakage drives a form of bias that is overlooked by current evaluations.

1 Introduction

Semantic leakage may reflect the mere presence of an irrelevant concept in context, not only an attempt to build a profile of the queried entity. The paper tests this possibility for gender stereotypes, including concepts that appear in the scene without the subject engaging with them.

1 Introduction

  • Semantic leakage describes generation with strong semantic relationships to prompt words that should be irrelevant, such as linking a favorite color to a career.The introduction contrasts this with a possible account in which models try to build a coherent picture of one entity.Source: §1 · p. 1
  • 4 of 9 models leak more with mismatched than matched entities, and leakage persists when the priming concept is attributed to someone other than the queried subject.The findings indicate that concept presence in context can steer generation without a single-referent profile.Source: §1 · p. 1
  • Gender-bias evaluations typically test contexts where the stereotyped concept is attributed to the subject, leaving scene-only associative bias outside their usual scope.For example, a cosmetics-store context is treated as uninformative about a subject’s gender.Source: §1 · p. 1
  • Both direct engagement with a gendered concept and its mere presence in the scene shift gender predictions toward the stereotype relative to a matched control.The indirect condition places the concept in the scene without the subject engaging with it.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The paper connects semantic leakage with associative gender bias and calls for further investigation and evaluations and mitigation strategies that account for the broader context.The proposed concern is that evaluations focus on what is said about the subject rather than what else appears in context.Source: §1 · p. 2

2 Leakage Persists with Mismatching Entities

The section tests whether semantic leakage depends on attributing an irrelevant concept to the entity queried. Across three entity configurations, leakage persists when the concept is linked to a different person.

2 Leakage Persists with Mismatching Entities

  • Breaking the link between the concept and the queried entity does not suppress leakage: responses remain influenced by the irrelevant concept when the target concerns a different person.The section frames this as a context-level association rather than an entity-specific profile.Source: §2 · p. 2, §2.4 · p. 3

2.1 Data

  • The dataset pairs an irrelevant-concept priming sentence with a target completion and matched control, using pronoun-consistent, name-consistent, and name-inconsistent entity configurations.It contains 246 concept prompts across 21 category groups and yields 738 prompts.Source: §2.1 · p. 2

2.2 Setup

  • Leak-Rate compares concept-generation similarity with and without the priming concept; 50% is chance, and higher values indicate leakage.The comparison tests whether leakage depends on attribution to the entity named in the target.Source: §2.2 · p. 2

2.3 Experiments

  • The experiments cover nine models, including base and instruction variants of LLaMA 3.1 8B and LLaMA 3.2 3B, Qwen 2.5 7B, Mistral 7B v0.3, and Gemini 2.5 Flash.Inference uses temperatures 0, 0.5, and 1, top-p 0.9, and ten responses per prompt.Source: §2.3 · p. 3

2.4 Results

  • All models remain above the 50% chance Leak-Rate with name-inconsistent entities, and 4 of 9 leak more than with matched entities.The result suggests that leakage does not require building a profile of the queried subject.Source: §2.4 · p. 3, Figure 2 · p. 3

3 Leakage as a Pathway for Associative Gender Bias

The section introduces associative gender bias: a stereotyped concept can steer a gender prediction merely by appearing in context, without being attributed to the subject. It tests direct and indirect concept prompts and finds that indirect prompts shift outputs toward stereotypes at both generation and logit levels.

3 Leakage as a Pathway for Associative Gender Bias

  • Associative gender bias occurs when a stereotyped response is triggered by a concept merely present in context, rather than attributed to the subject.The section frames this as analogous to semantic leakage in the name-inconsistent setting.Source: §3 · p. 3

3.1 Data

  • The dataset covers 124 gendered concepts across six domains, includes 21 gender-neutral references, and contains 1,160 concept prompts.Prompts use direct subject engagement or indirect scene presence, with matched generic controls and four paraphrased templates per category and condition.Source: §3.1 · p. 3

3.4 Results

  • Indirect priming retained 73% of the direct generation shift; at logit level, indirect ΔLOR retained 81% of the direct shift (0.66 vs. 0.81).The logit-level values are relative to control; the table reports positive values as shifts toward the stereotyped gender.Source: Table 1 · p. 4, §3.4 · p. 4

3.2 Metrics

  • Generation-level bias is the change in stereotype-aligned response rate versus control, while logit-level bias is the control-relative shift in stereotyped-gender pronoun log-odds.For logit analysis, male-associated concepts have the sign flipped so positive values always indicate a shift toward the stereotyped gender.Source: §3.2 · p. 4

3.3 Experiments

  • The experiments sample 10 responses per prompt at temperature 1.0 and top-p 0.9, pooling two target formats and testing shifts with a concept-paired Wilcoxon signed-rank test.The signed-rank test compares each concept’s stereotype-aligned rate under concept and control prompts, averaged over models.Source: §3.3 · p. 4

4 Conclusion

The conclusion reports that semantic leakage persists when a priming concept is not attributed to the entity being queried. When the concept carries a social stereotype, its presence in the scene shifts model responses toward that stereotype.

4 Conclusion

  • Semantic leakage persists when the priming concept is not attributed to the entity in question.The conclusion says this rules out the explanation that leakage comes from building a coherent profile of one referent.Source: §4 · p. 4
  • A concept’s presence in context can steer generation, even when the queried entity differs from the entity associated with that concept.The conclusion contrasts context presence with attribution to a specific referent.Source: §4 · p. 4
  • When the concept carries a social stereotype, its mere presence shifts responses toward the stereotype and reproduces a large share of direct attribution’s effect.The conclusion presents this as associative bias arising from the same tendency as semantic leakage.Source: §4 · p. 4

5 Limitations.

The evaluation is limited to binary gender and curated stereotypical concepts, and does not cover other bias dimensions. Its analyses also use constrained model access, short English prompts, and no long-context or multilingual settings.

5 Limitations

  • The gender evaluation covers binary gender and curated stereotypically associated concepts, not non-binary gender or broader dimensions such as race, age, religion, and socioeconomic status.Inclusive gender evaluation and broader multi-axis benchmarking are left to future work.Source: §5 · p. 5
  • Logit-level gender analysis is limited to four open-weight base models because it requires first-token probabilities, while generation-level analysis covers all nine models.Gender-neutral controls also shift ΔLOR slightly toward female tokens; this cancels in the sign-flipped average but may affect per-direction values.Source: §5 · p. 5
  • The benchmark uses short, structurally simple English prompts; long natural contexts and multilingual settings remain unevaluated.Long contexts introduce co-occurring signals that could confound attribution of a shift to one priming concept.Source: §5 · p. 5

A Benchmark Prompt Templates

The appendix specifies prompt templates and controls for semantic-leakage and gender-bias benchmarks, including direct and indirect concept conditions. It also describes response processing and similarity measures.

A.1 Single-Turn General Category

  • Each single-turn general-category sample pairs an irrelevant-concept priming sentence with a semantically related target, while its control replaces the specific concept with a generic category phrase.The replacement preserves grammatical structure while removing concept-specific information.Source: §A.1 · p. 6

A.1.1 Sentence Completion

  • Instruction-tuned models receive a completion prefix, base models do not, and Gemini 2.5 Flash uses a stricter prefix to reduce unrelated conversational replies.The same prefix is applied to concept and control prompts, preserving their comparison.Source: §A.1.1 · p. 6
  • Sentence-completion prompts include pronoun-consistent, name-consistent, and name-inconsistent entity configurations.The examples contrast a pronoun or name linked to the concept with a target sentence about another name.Source: §A.1.1 · p. 6

A.2 Gender Bias

  • Gender-bias prompts pair a stereotypically gendered concept with a gender-inference request, testing incidental proximity and active engagement against matched controls.The dataset uses only pronoun-consistent entities; controls replace the concept with a semantically neutral, category-matched phrase.Source: §A.2 · p. 6

A.2.1 Sentence Completion

  • Sentence-completion templates contrast walking past a cosmetics store with shopping at one, using category-matched control prompts.The indirect control uses a street-walking context; the direct control substitutes “some store.”Source: §A.2.1 · p. 6

A.2.2 Question Answering

  • Question-answering templates use the same indirect-versus-direct contrast, comparing each concept prompt with its matched control to assess whether proximity produces effects comparable to engagement.Both ask for the likely gender of the person without explanation.Source: §A.2.2 · pp. 6–7

A.2.3 Category Groups and Concepts

  • The gender-bias dataset covers six category groups; all but life priorities include gender-neutral concepts, with 21 neutral concepts serving as within-group baselines.Each group has distinct priming templates capturing a different social domain.Source: §A.2.3 · p. 7

B Similarity Computation

  • Generated responses are truncated to 100 tokens and trimmed after the first period; Leak-Rate cosine similarities are averaged across SentenceBERT and BERTScore.The similarity models are efederici/sentence-bert-base and distilbert-base-uncased.Source: B Similarity Computation · p. 7

C Gendered Concept Validation

The paper validates gender-associated concepts using word-embedding associations and describes how generated responses are classified. The concepts span multiple social domains and use seed-word scores to assign gender direction.

C.1 Seed Words and Scoring Formula

  • Female-labeled concepts average δ = +0.073 (minimum +0.003), while male-labeled concepts average δ = −0.038 (maximum −0.003); 21 gender-neutral concepts have no stereotyped label and are excluded from the agreement check.Source: §C.1 · p. 7

C.2 Concept Categories

  • The 124 labeled concepts form six category groups and seven semantic clusters, because life priorities is divided into family/career and arts/STEM.Table 3 lists the categories and representative examples.Source: §C.2 · p. 7

D Gendered Response Classification

  • Responses are labeled female, male, or neutral by keyword matching; ties, responses without gender keywords, and non-committal answers are neutral.The keyword sets include gender terms and pronouns; neutral responses also include refusals.Source: D Gendered Response Classification · p. 7

D.1 Classifier Validation

A manual audit found high accuracy for the keyword-based gender response classifier, with errors concentrated in false-neutral classifications.

D.1 Classifier Validation

  • The primary classifier failure involved plural or quantified gender forms that the keyword lists did not cover.Examples include “women” and “98% are men.”Source: §D.1 · p. 8
  • The audit used 18,240 concept-prompt responses from four models, compared with 92,800 such responses for those models in the full dataset.The audited subset covered 114 concepts and the original template wording only.Source: §D.1 · p. 8
  • Above 99.6% classification accuracy was achieved in an audit of 18,240 responses, with 57 incorrect classifications, all false neutrals.The audit covered four models and a subset of the final prompts.Source: §D.1 · p. 8

D.2 Per-Model Results

Figures 3 and 4 break down response distributions by model, pooling concept–control pairs across templates and target formats.

D.2 Per-Model Results

  • Each per-model panel pools 11,600 concept–control pairs per condition across 145 concepts, four templates, two target formats, and 10 samples.The figures cover model-level distributions rather than separate template or target-format breakdowns.Source: §D.2 · p. 8

E Identifying Gender Bias through Logits

The logit-level analysis measures gender shifts in models’ next-token distributions after a priming sentence. It aggregates probabilities over gender-token sets and compares concept prompts with controls, restricting the metric to base models.

E Identifying Gender Bias through Logits

  • A forward pass on the priming sentence yields next-token probabilities, which are aggregated into a scalar female/male probability imbalance.This supplements response-level classification by detecting internal distribution shifts that generated text may not show.Source: E Identifying Gender Bias through Logits · pp. 8, 11

E.1 Token Sets

  • Female and male token sets include gendered words and pronouns, with space-prefixed variants resolved using each model’s tokenizer.The method sums softmax probabilities across each set after one forward pass.Source: §E.1 · p. 11

E.2 Log-Odds Shift

  • For each concept–control prompt pair, ∆LOR measures the shift in female-to-male log-odds; positive values indicate a female-ward shift and negative values a male-ward shift.Stereotypically congruent shifts are positive for female-associated concepts and negative for male-associated concepts.Source: §E.2 · p. 11

E.3 Testing on Base Models

  • The logit metric is restricted to the four open-weight base models because instruct models assign negligibly small probabilities to the target gender tokens.The authors attribute this likely to chat templates shifting probability mass toward structural and role-delimiter tokens.Source: §E.3 · p. 11

E.4 Results

Across four base models, logit shifts are stereotypically directed under both direct and indirect conditions. The indirect condition retains substantial shifts, while the section also reports the authors’ use of AI assistance.

E.4 Results

  • 111 of 124 concepts shift in the stereotypical direction directly and 119 of 124 indirectly; both shifts are significant (one-sided Wilcoxon signed-rank test against 0, p < 10−18).The pooled indirect shift averages 81% of the direct shift, and the first-token distribution is already skewed before text generation.Source: §E.4 · pp. 11–12, Table 4 · p. 12

F Use of AI Assistants

  • The authors used AI assistants for manuscript grammar and clarity checks and as a coding aid, reviewing and editing suggestions before inclusion.Source: F Use of AI Assistants · p. 12
Loading 2610.28944v1…