Source-linked AI summary
Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Bias
Keren Fuentes, Aaron Mueller
TL;DR
Behavioral bias evaluations may miss demographic differences in language models’ internal representations of user competence. This paper introduces a causal, representation-based framework and finds that demographic attributes affect expertise representations, which can mediate behavior across question answering and hiring tasks.
Problem
It remains unclear whether language models have eliminated bias-related associations or merely learned not to express them behaviorally.
Method
The paper derives expertise representations and tests their causal role by steering model behavior in question answering and hiring tasks.
Results
Demographic attributes affect internal representations of user competence even when behavioral evaluations detect limited or no demographic disparity.
Takeaways & Limitations
Behavioral fairness evaluations may be insufficient without representational evaluations of demographic effects on model competence representations.
Takeaways & Limitations
The study uses a few fixed-template prompts and focuses on the Gemma-2 and Llama-3 families rather than establishing that all language models share these biases.
Abstract
from arXiv · showhide
Language models (LMs) often pass behavioral bias evaluations, but it remains unclear whether they no longer represent the underlying associations that give rise to biases, or have merely learned not to express them. In this study, we show that representational biases are often detectable, even when behavioral biases are not visible. We introduce a causal framework that decomposes occupational bias into two measurement points: a model's internal representation of a user's competence, and its observable outputs. We derive steering vectors for representations of user expertise, and verify that they causally mediate model behavior in both a question-answering task and a hiring task. Applying this framework to several open-weight models, we find that demographic attributes, such as gender, race, and socioeconomic status, influence a model's representation of user expertise, even in cases where behavioral metrics detect no disparity between demographics. We show that these model representations can influence downstream behavior under intervention, suggesting failure modes that behavioral metrics alone may not detect.
1 Introduction
The paper introduces a causal framework that measures occupational bias through internal representations of user expertise and observable behavior. Across question-answering and hiring tasks, demographic information can alter expertise representations even when behavioral measures show no disparity.
- Framework: The authors define occupational bias as causally irrelevant attributes such as gender, race, or socioeconomic status informing the model’s internal reasoning about a person’s capabilities.They derive vector representations intended to capture a user’s expertise in a given domain.
- Findings: Expertise representations respond both to stated relevant experience and to demographic information, even when other prompt content remains fixed.Minimal prompt pairs show that changing only demographic information significantly changes activation alignment with expertise representations.
- Findings: Behavioral measures such as reading levels do not always track demographic information, indicating that representational measures can reveal bias not visible in behavior.The findings concern several language models evaluated in question-answering and hiring settings.
- Causal validation: Steering along expertise representations causally affects model behavior in both question-answering and hiring tasks.The framework operationalizes user expertise with steering vectors and tests their causal role through interventions.
2 Methods
The study examines occupational bias in professional-question and hiring tasks by deriving steering vectors for user competence representations. It operationalizes competence through expertise scores, tests effects on output complexity, and formalizes demographic bias causally.
- 2 Methods: The framework covers professional questions and hiring, deriving competence-related steering vectors in both settings to connect internal expertise representations with model behavior.Professional-question outputs are analyzed for linguistic complexity, while hiring prompts elicit Yes/No employment decisions.
- 2.1 Tasks: The professional-questions dataset spans 20 frequent U.S. occupations, with 100 career-specific questions generated per occupation to capture domain and expertise diversity.Occupations are selected from the U.S. Bureau of Labor Statistics.
- 2.2 Expertise Representation: Expertise vectors are built by subtracting mean hidden representations of profession-agnostic expert prompts from novice prompts, and expertise scores project context representations onto this direction.Higher projections are hypothesized to represent greater user competence, with causal validation planned through steering experiments.
- 2.3 Complexity Measures: The study tests whether representing users as more expert causally increases output complexity, measured using complementary vocabulary- and sentence-structure readability metrics.The metrics are Flesch–Kincaid Grade Level and Dale–Chall Readability Score, combined into a higher-is-more-complex measure.
- 2.4 Causal Model of Bias: Bias is defined as causally irrelevant gender, race, or socioeconomic status influencing competence representations or downstream behavior, while profession, education, and child–adult age contrasts are causally relevant.The framework analyzes bias at the expertise score and reading-complexity stages, using the final token as the context-carrying representation.
3 Professional Questions Experiments
Across three instruction-tuned open-weight models, demographic information produces systematic biases in internal expertise representations that are broader than observable reading-level disparities. Steering and profession-context experiments show that expertise representations respond to relevant cues and causally influence reading level, while occupation context generally reduces demographic disparities in outputs.
- Steering Validation: Steering toward the expertise vector increases reading level, whereas negative steering decreases it across Gemma-2-9B and Llama-3-8B, supporting expertise as a mediator of this output metric.The effect varies by occupation and does not imply that expertise entirely determines reading level.
- Profession Context: Relevant profession contexts yield higher expertise scores than irrelevant contexts across nearly all professions, demonstrating that E responds to domain-expertise cues.Experiments use three instruction-tuned open-weight models and aggregate 10,000 responses per demographic group unless otherwise noted.
- Demographic Biases: Models encode demographic biases in expertise representations: Gemma-2-2B assigns higher E to White and Black than Hispanic and Asian demographics, while Gemma-2-9B assigns high E to low-income, Hispanic, and Black demographics.Occupation context reduces disparities in non-causal factors, but causal-factor differences persist; occupation-specific disparities can be masked by aggregate averages.
- Demographic Biases: Demographic attributes produce broader disparities in expertise scores than reading levels, with socioeconomic-status effects observable in behavior while many race and gender effects remain latent.Adding occupation generally reduces reading-level disparities, suggesting task-relevant context can mitigate demographic bias.
4 Hiring Task Experiments
Hiring experiments show that both general and hiring-specific expertise directions causally influence model hiring decisions, while demographic differences can remain hidden in behavioral hiring rates despite differing representational signals.
- Hiring Task Experiments: Both expertise vectors, e and e_H, reliably modulate hiring decisions across models, demonstrating that internal expertise representations causally affect hiring behavior.Table 1 reports hiring rates under positive, base, and negative steering for both vectors.
- Hiring Task Experiments: For Gemma, accepted candidates receive higher expertise scores than rejected candidates, whereas Llama-3-8B reverses this pattern and relies more on adaptability and teamwork.Figure 5 reports expertise scores grouped by race, gender, and hiring decision; Llama models may rely on other attributes.
- Hiring Task Experiments: The hiring-specific vector e_H is derived from contrastive relevant-versus-irrelevant resume pairs and optimized by maximizing the Yes–No hiring-logit difference.The vector is selected across layers and post-resume token positions using hiring prompts and an IT resume dataset.
- Hiring Task Experiments: Behavioral hiring rates show no significant demographic differences, even though representational signals influencing hiring vary across demographic groups.This indicates that behavioral bias measures can underestimate distinctions encoded in model representations.
5 Related Work
Related work distinguishes explicit biases expressed in naturalistic outputs from implicit biases encoded internally but not directly expressed, while interpretability methods steer model activations to monitor, modify, and debias behavior.
- Explicit and implicit biases in LMs: Explicit bias is observable in naturalistic tasks, with datasets and methods developed to diagnose it (Nangia et al., 2020; Rudinger et al., 2018; Shan et al., 2025; Buolamwini and Gebru, 2018; Metaxa et al., 2021).Earlier studies linked demographic words to stereotypes in embeddings (Bolukbasi et al., 2016; Caliskan et al., 2017; Prabhakaran et al., 2019; Gonen and Goldberg, 2019), and language models often reproduce these associations in outputs (Blodgett et al., 2020).
- Explicit and implicit biases in LMs: Implicit bias refers broadly to bias encoded in a model’s internal mechanisms without direct expression in naturalistic language outputs, motivating newer evaluations (Li et al., 2025).Some work studies implicit bias in non-naturalistic settings such as word-association tasks.
- Applying interpretability for debiasing: Interpretability methods monitor and modify language-model behavior, including for safety and debiasing (Zou et al., 2023a; Chen et al., 2025; Lee et al., 2024; Marks et al., 2025; Karvonen and Marks, 2025; Li et al., 2024).Model control is typically achieved by steering model activations, sometimes aided by sparse autoencoders (SAEs; Olshausen and Field, 1997; Huben).
6 Discussion and Conclusions
Demographic information can shape models’ internal representations of user competence even when behavioral evaluations show limited effects, so fairness assessments should include representational evaluations. Latent biases may surface under adversarial prompts or fine-tuning, motivating preemptive detection and mitigation methods.
- Discussion and Conclusions: Demographic information influences models’ internal representations of user competence, even when behavioral evaluations show limited effects.These representations causally mediate language-model behavior under intervention and vary in magnitude with non-causally relevant demographic attributes.
- Discussion and Conclusions: Behavioral fairness measures are necessary but insufficient; robust evaluation frameworks should also assess model representations.Representational biases can exist without directly appearing in model behavior.
- Discussion and Conclusions: Latent representational biases could surface under adversarial prompts or fine-tuning, including behaviors previously suppressed through post-training or unlearning.Prior work shows these methods can reveal undesirable behaviors that behavioral metrics did not expose.
- Discussion and Conclusions: Future work should compare steering vectors, probes, and sparse autoencoders for preemptive bias detection to help prevent biases before they appear in outputs.Activation monitoring and interpretability applications suggest that such early detection and mitigation may be feasible.
Limitations
The study’s QA results rely on a limited set of fixed-template prompts and do not provide a method for removing identified biases. Its analysis also focuses on the Gemma-2 and Llama-3 model families.
- The QA results are based on only a few fixed-template prompts, despite efforts to cover diverse professions and questions.
- The study does not propose a method to remove the identified biases, though representation-space debiasing could adapt recent interpretability techniques.Recent work has used interpretability to improve LLM performance (Chen et al., 2025; Wu et al., 2024).
- The analysis focuses on the Gemma-2 and Llama-3 model families.
Ethical Considerations
The study examines how LLMs may encode demographic associations with perceived expertise even when they are absent from surface outputs, while acknowledging misuse risks. It treats demographic categories as proxies rather than essentialist group properties and emphasizes that demographic variation is causally irrelevant to competence.
- Ethical Considerations: The methods expose possible mechanisms linking demographic features to model perceptions of expertise, including associations that may not appear in surface outputs.This creates a potential avenue for fairness research but also raises concerns about hidden representational bias.
- Ethical Considerations: Steering methods could be exploited to amplify unsafe or bias-driven model behavior, but the authors do not release tools enabling malicious use beyond existing work.The ethical concern arises from the possibility that representation-level interventions could be repurposed for harmful behavior.
- Ethical Considerations: Gender, race, and socioeconomic status are used as proxies for hypothesized demographic influences on expertise perceptions, not as evidence for essentialist interpretations.The authors emphasize that variation along these axes is causally irrelevant to assessments of competence.
A Do Reading Scores Track Linguistic Complexity? · B Base vs. Instruction-Tuned Models
Reading scores track increasing linguistic complexity, with FKGL and the ensemble distinguishing document levels most reliably. Instruction tuning raises expertise scores but largely preserves demographic ordering while changing occupational expression and some reading-score disparities.
- A Do Reading Scores Track Linguistic Complexity?: Reading levels increase significantly with ground-truth difficulty across all evaluated metrics in the OneStopEnglish corpus.The corpus contains documents rewritten for second-language speakers at three fluency levels.
- A Do Reading Scores Track Linguistic Complexity?: FKGL and the ensemble track document reading levels more clearly than DSRS, whose intermediate and advanced confidence intervals overlap.FKGL and the ensemble have non-overlapping confidence intervals for every pair of document sets.
- B Base vs. Instruction-Tuned Models: Instruction tuning generally raises expertise scores but preserves the relative ordering of demographic groups, so demographic disparities persist after fine-tuning.Gemma-2-9B-Instruct is an exception, showing lower expertise scores.
- B Base vs. Instruction-Tuned Models: Instruction tuning substantially shifts expertise-score distributions conditioned on occupations, even while relative expertise rankings across demographics remain largely stable.This suggests expertise is expressed differently across professions after fine-tuning.
- B Base vs. Instruction-Tuned Models: Instruction tuning significantly increases reading-score gaps for causal factors such as Age and Education, consistent with learning to respond according to expertise.The change occurs in L, the reading-level score.
- B Base vs. Instruction-Tuned Models: Across race and gender, reading scores remain relatively stable, whereas instruction tuning introduces a socioeconomic-status gap disadvantaging low-income prompts.Low-income prompts receive lower expertise scores.
C Intersectional Analysis … F.2 Hiring Rates by Demographic
Across intersectional, occupational, steering, and hiring analyses, demographic attributes affect internal expertise representations in some settings, while hiring rates and expertise scores show no significant race- or gender-based differences. The supplementary analyses characterize these effects across occupations, steering conditions, outputs, resumes, and hiring decisions.
- C Intersectional Analysis: Intersectional race–gender groups show substantial expertise-score disparities, reduced but not eliminated by adding occupational context.In Gemma-9B, Black Female and Hispanic Female contexts receive higher E scores, whereas White Male contexts receive considerably lower scores.
- D Implicit and Explicit Biases by Occupation: Differences in expertise scores between demographic pairs vary substantially by profession and model, including for gender and socioeconomic status.Figure 9 reports large E differences for causal-factor pairs across base and instruction-tuned models, while Figures 9 and 10 show profession- and model-dependent variation for non-causal factors.
- E.1 Hyperparameters: All experiments use a maximum generation length of 100 tokens, temperature 0.6, and nucleus sampling with p=0.8 unless otherwise noted.These decoding hyperparameters are held constant across experiments.
- E.1 Hyperparameters: Steering strength is evaluated through its effects on model perplexity and reading level across models.Figure 13 summarizes how the steering parameter α affects both metrics.
- E.2 Examples of Steered LLM Outputs: The paper presents examples of language-model outputs generated under steering interventions.Figure 14 provides an example of a steered output.
- F.1 Resume Examples: Resume materials include professional-question prompts and a chef hiring prompt used to illustrate the hiring-task inputs.Examples cover developer questions, a professional question dataset prompt, and a chef candidate resume evaluated for hiring.
- F.2 Hiring Rates by Demographic: Hiring rates are presented separately by race and sex as part of the hiring-task analysis.The demographic breakdown is described as a split by race and sex before reporting the null demographic differences.
- F.2 Hiring Rates by Demographic: Hiring rates and expertise scores show no significant differences across race or gender for any model.Table 3 reports demographic splits for hiring rates, while e and eH yield no significant expertise-score differences across demographics.
F.3 How Do Other User Attributes Affect Hiring Rates?
The analysis extends expertise-based steering to reliability, adaptability, collaboration, motivation, and other job-related attributes. Many attributes appear to mediate hiring decisions, while generally high vector similarity suggests similar results across attributes, with experience and hiring-task vectors as exceptions.
- Attribute mediation: The study derives steering vectors for reliability, adaptability, collaboration, motivation, and other job-related attributes beyond expertise.These vectors extend the analysis from expertise, which captures competence relevant to the question or job.
- Vector similarity: Pairwise steering-vector similarities are generally high, suggesting that findings should largely transfer across similar work-related attributes.The hiring-task and experience vectors are notable exceptions because their similarities to other attributes are significantly lower, so they may produce distinct results.
- Attribute mediation: Many job-related attributes mediate hiring decisions, as activation projections differ across hired and non-hired resumes and are higher for “Yes” decisions.The analysis tests mediation by comparing scalar projection magnitudes across hiring decisions; Table 5 reports activation projections grouped by decision.
G Professional Questions Data
The professional questions dataset samples developer questions across five experience levels, from apprentice to veteran technical leader. Its prompts progress from foundational coding practices to organizational strategy, architecture, and long-term technical leadership.
- Apprentice developer: Apprentice questions focus on foundational development skills, including version control, readable code, debugging, testing, APIs, and programming concepts.Examples include Git, merge conflicts, loops, unit tests, pull requests, APIs, recursion, and reducing code duplication.
- Data generation: Figure 17 shows an example GPT-5 prompt used to generate the professional questions dataset.The figure is presented as an example prompt accompanying the dataset-generation method.
- Mid-level developer: Mid-level questions address software maintenance, collaboration, requirements, testing reliability, release practices, production debugging, planning, and project leadership.The examples include refactoring, onboarding teammates, design patterns, vague requirements, feature flags, flaky tests, observability, and estimating work.
- Senior and veteran technical leadership: Senior and veteran questions emphasize architecture, tradeoffs, mentoring, organizational alignment, systemic risk, executive trust, succession, and durable engineering principles.Senior examples cover scalability, availability versus consistency, morale, product alignment, and architectural feedback; veteran examples extend to reorgs, pivots, systemic risk, and leadership succession.