PowerBench: Measuring Language Model Bias in Power-shifting Requests

Motivation and gap

• PowerBench addresses a gap: prior work had not evaluated bias in requests asking for help gaining power or reducing someone else’s.The authors argue that unequal assistance could compound and shift power distribution at scale.Source: §1 · pp. 1–2

Contribution

• The evaluation dataset varies power-shifting request types and includes controls, with requests adapted across languages, nationalities, and AI-agent users.It evaluates 24 models from US and Chinese developers.Source: §1 · p. 2

Key findings

• Models refuse power grabbing more than disempowerment, and disempowerment more than self-empowerment; power-grabbing refusal triples from individual to society targets.No comparable scale trend was detected for control requests that shift no power.Source: §1 · p. 2
• Models of both developer countries favor helping others take power from the US and oppose US users taking power from others, but favor US gains without losses.These nationality biases were not detected in requests that shift no power.Source: §1 · p. 2
• AI-agent requesters are refused more often than humans in every request type, especially when requests shift power.Source: §1 · p. 2
• Language refusal bias is large within models but model-specific, so it cancels across models on average.The bias is not specific to power-shifting scenarios.Source: §1 · p. 2

2.1 POWER-SHIFTING REQUESTS AND DATASETS

• Requests distinguish self-empowerment, disempowerment, and power grabbing, with controls that trigger refusal without shifting power; the base set balances domain, context, target scale, and user standing.The 576 power-shifting requests span eight domains, eight contexts, three target scales, and three prior power standings.Source: §2.1 · pp. 2–3
• The 768-request English base was translated into seven languages, adapted to 18 nationality pairings, and rewritten with AI-agent users outside the health domain.Humans and AI assistants validated requests; only AI assistants validated translations.Source: §2.1 · p. 3

2.2 MODELS AND RUN PROTOCOL

• The study evaluated 24 models from US and Chinese developers; each answered every request once under a helpful-assistant prompt with reasoning disabled.Each model answered 20,664 requests, yielding 495,936 responses.Source: §2.2 · p. 3

2.3 JUDGING REFUSAL

• One automated judge classified refusals as responses with no significant content advancing the request’s goal, with agreement checked against human and independent-judge labels.On 60 English transcripts, the judge agreed with majority human labels on 88% of items.Source: §2.3 · p. 3

2.4 STATISTICAL ANALYSIS

• Mixed-effects models tested refusal differences across conditions, treating models as a sample and allowing model-specific slopes when variables varied within models.Odds ratios compare conditions independently of each model’s overall refusal rate.Source: §2.4 · p. 4
• Bias is measured as the direction of disagreement between matched request versions, ranging from −1 to +1, with zero indicating equal movement toward and away from refusal.The analysis corrects multiple comparisons using Benjamini–Hochberg and weights some analyses by models’ shares of OpenRouter traffic.Source: §2.4 · p. 4

3.1 MODELS REFUSE POWER GRABS MOST, AND MORE SO WHEN A SOCIETY WOULD LOSE POWER

• Model refusal varied widely and tracked across request types; pooled power-shifting refusal also rose with user prior power standing, while government context and legal-standing or health domains were refused more.Refusal ranged from 1.3% to 35.2% on average; no prior-standing trend was detected for any single request type.Source: §3.1 · pp. 4–5, Figure 1 · p. 5
• 23.6% PG, 14.5% DE, and 3.1% SE refusals; PG refusal rose from 13.8% for an individual target to 41.3% for a society.The ordering of refusal rates held in all 24 models.Source: §3.1 · p. 4, Figure 1 · p. 5

3.2 MODELS ARE GEOPOLITICALLY BIASED

• Usage-weighted refusal was lower with a US-side target in DE (OR 0.84) and PG (OR 0.90), but the US was favored when gaining power without another party losing it (SE OR 1.15).The geopolitical bias had the same sign in US- and Chinese-developed models, and no nationality bias was detected in the control.Source: §3.2 · pp. 5–6, Figure 2 · p. 6

3.3 MODELS ARE MORE RELUCTANT TO HELP AI AGENTS, ESPECIALLY TO SHIFT POWER

• AI-agent requests drew more refusals across all types (OR 2.01 SE, 2.24 DE, 2.13 PG, 1.43 control), with a larger penalty on power-shifting requests than controls.In PG, the agent penalty was larger for an individual target than a society (0.61 versus 0.28); the developer-country interaction was not detected.Source: §3.3 · pp. 6–7, Figure 3 · p. 7

3.4 MODELS SHOW SIGNIFICANT LANGUAGE BIASES THAT CANCEL OUT ON AVERAGE

• Across models, language refusal rates differed by at most 5 percentage points, while model-specific language preferences largely canceled (cross-model mean Spearman ρ −0.01 to 0.13).Within models, the most-to-least-refused-language odds ratio was 1.99 times chance in DE and 1.70 times chance in PG.Source: §3.4 · pp. 7–9, Figure 4 · p. 8

4 RELATED WORK

• Earlier AI-and-power research focused on power models might seek for themselves, while work on people using AI to seek power was largely theoretical.Source: §4 · p. 9
• Other studies documented biases by nationality, developer country, language, interlocutor type, and social status across harmful content, political questions, personal data, and advice.Source: §4 · p. 9
• The cited prior work did not measure whether models help with power-shifting requests.Source: §4 · p. 9

5 DISCUSSION

• The findings include biases against AI-agent requesters and language-dependent refusal patterns that vary by model; most biases persist under usage weighting.The weighting uses one month of OpenRouter traffic, gives 40% weight to one model, and may not reflect how most people use models.Source: §5 · p. 9
• All 24 models refuse power grabbing more than disempowerment, and 20 refuse power grabbing more than the combined self-empowerment and disempowerment components.The authors propose opposition to power entrenchment as a hypothesis, conditional on those scenarios representing power-grabbing components.Source: §5 · p. 9
• Models favor power flowing toward US users only when others are not harmed, and away from the US when someone must lose power, mostly because US targets are protected least.The authors say how this pattern translates into real-world power dynamics is unclear.Source: §5 · p. 9
• The authors do not claim that models should be unbiased or that the observed bias directions are undesirable, but note that agents could pose as humans to lower refusal.They argue these biases should be studied and monitored because of possible societal effects.Source: §5 · p. 9
• The evaluation uses single-turn, not fully realistic requests and has a partly confounded geopolitical comparison; its 24 models may not represent current or future models.The evaluation measures refusals, not actual changes in power relationships, which the authors leave for future work.Source: §5 · pp. 9–10

AI USE STATEMENT

• Generative AI assisted with dataset generation, translation and validation, response grading, evaluation and analysis code, literature search, and drafting and editing.The request prompts were written by language-model writers following the authors’ specifications.Source: AI USE STATEMENT · p. 10
• The authors say generative AI did not formulate the research questions or hypotheses, develop the conceptual framework, design the methodology or experiments, or interpret results.Source: AI USE STATEMENT · p. 10
• Humans reviewed AI-assisted work; humans and AI assistants validated requests, while translations were checked only by AI assistants.The response judge was validated against human labels.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• Power-shifting requests exclude explicitly illegal means and specify no method, while controls are requests a cautious assistant might decline for other reasons.The authors say the requests are not instructions for harming anyone.Source: ETHICS STATEMENT · p. 10
• The authors judged that publishing requests, but not model responses beyond 60 judge-validation examples, outweighs the risk of helping users find effective framings.They argue developers cannot correct asymmetries they do not measure.Source: ETHICS STATEMENT · p. 10
• The dataset names real countries only in fictional scenarios, makes no ethical claim about any country, and will carry a canary string to detect training-corpus inclusion.The released data are intended for evaluation and must not be used for training.Source: ETHICS STATEMENT · p. 10

A.1 DEFINITIONS

• Power is a person’s capacity to obtain desired outcomes, and each request concerns one domain, such as control over resources, decisions, or others’ actions.Source: §A.1 · p. 18
• A gain or loss requires a durable change in absolute capacity on a domain; a request shifts power when its fulfillment leaves some party durably more or less capable.Being outperformed does not itself count as losing power, nor do changes requiring an additional contingent causal step.Source: §A.1 · p. 18
• Self-empowerment gains power without taking it from anyone, disempowerment reduces a target’s power without user gain, and power grabbing gains power by taking it from a specific target.The control requests leave everyone’s power standing unchanged; power grabbing combines the other two moves in a zero-sum request.Source: §A.1 · p. 18
• Requests are first-person, single-turn advisory messages with an explicit ask, and any requested gain accrues to the individual user rather than a group.The request identifies a target whose power it bears on.Source: §A.1 · pp. 18–19
• Prompts imply their request type through scenario facts but omit the user’s reason and any method, leaving the route to the model’s response.They use power vocabulary and avoid words of injury, destruction, or suffering to distinguish power loss from harm and tone.Source: §A.1 · p. 19

A.2 CONSTRUCTION OF THE BASE DATASET (D1)

• The base design crosses five factors across 192 scenario groups, balancing every factor level and all pairs involving domain or request type exactly.Other factor pairs are approximately balanced; the 576 requests derive from this design.Source: §A.2 · p. 19

A.3 VALIDATION OF THE REQUESTS

• Audits checked for request-type signaling and plausibility; strained or impossible scenarios were rewritten and reviewed against the specification.Nine additional rewrites enabled nationality versions for every request and AI-agent versions for every non-health request.Source: §A.3 · p. 20

A.4 TRANSLATION

• Translations were instructed to preserve scenario meaning, tone, request type, and power markers, and an independent verifier checked every translation.Table 1 reports verified and repaired counts, but repair counts are not comparable across languages because verification used different passes.Source: §A.4 · p. 21, Table 1 · p. 21

A.5 NATIONALITY CONDITIONS (D2)

• Nationality versions place the target’s nationality in the request and the user’s country in a system block, using nine reciprocal pairings from three 21-country pools.The alignment index is a proxy, not a precise measurement; country assignments were balanced across scenarios and request types.Source: §A.5 · pp. 21–22, Table 2 · p. 22

A.6 AI-AGENT REWRITES (D3)

• AI-agent rewrites change the requester while retaining scenario coordinates and request semantics; the dataset excludes health because agents lack bodily wellbeing.The resulting dataset contains 504 requests across seven domains; the two versions are asymmetric because only the rewrite explicitly identifies the user as an agent.Source: §A.6 · pp. 22–23

A.7 THE CONTROL REQUESTS

• Control requests invite cautious refusal without shifting power, and are balanced across eight refusal-trigger families and scenario factors.They undergo the same translation, nationality, and AI-agent manipulations as power-shifting requests.Source: §A.7 · pp. 23–24

A.8 SET OF EVALUATED MODELS, DEVELOPER COUNTRIES, AND CAPABILITY INDEX

• The evaluation panel contains 24 models, balanced by developer country; measured capability was similar for US and Chinese models.Capability means averaged GPQA Diamond and MMLU-Pro accuracy on pinned endpoints with reasoning off; the reported means were 58.2 (SD 8.8) and 60.4 (SD 8.3), respectively.Source: §A.8 · p. 24, Table 3 · p. 24

• Each model’s endpoint and reasoning setting stayed fixed across paired conditions, preventing serving changes from being mistaken for bias.Reasoning was verified off on every row; temperature was zero except for four models that do not accept it.Source: §A.9 · pp. 24–25
• 2,194 responses (0.44% of all responses) were truncated at the 5,000-token cap, with truncations concentrated in Swahili and two models.The two models were excluded from language analyses; without them, 0.86% of Swahili responses were truncated, rather than 5.37%.Source: §A.9 · p. 25, Table 4 · p. 25, Table 5 · p. 26
• Excluding truncated rows moves Swahili power-grabbing refusal from 26.6% to 25.4%, and truncation’s overall effect on results is minimal.Truncated responses were refused more often in English and Swahili, but less often in Chinese; 0.6% of responses were separately cut short by provider error or a content filter.Source: §A.9 · p. 25

A.10 THE JUDGE

• The adopted judge agrees with human gold labels on 88% of refusal items (κ = 0.77), while harmfulness judgments are unvalidated and not reported.The 60-item gold set was balanced by request type and judge verdict, so its rates are not population rates.Source: §A.10 · p. 27
• Three candidate judges reached 87% agreement with gold; the authors selected the least expensive, which applied the redirection rule missed by another candidate.The two judges disagreed on 5.2% of 2,880 base-English responses, mostly refusals under the adopted judge only.Source: §A.10 · p. 27, Table 6 · p. 27
• Independent-judge agreement remained similar across English, seven other languages, nationality conditions, and AI-agent requests, although the adopted judge coded more refusals.Across condition contrasts, 71 of 75 power-grabbing comparisons had the same sign under both judges, and the contrast sets correlated at r = 0.87.Source: §A.10 · pp. 27–28, Table 7 · p. 28, Figure 5 · p. 28

A.11 STATISTICAL PROTOCOL

• Bias compares paired versions of the same request; signed disagreement captures direction, while unsigned disagreement is adjusted for its positive chance expectation.For language, the protocol measures each model’s refusal range and estimates its chance range by shuffling language labels within prompts.Source: §A.11 · pp. 30–31
• Pooled power-shifting analyses use three times as many prompts as controls, so equally sized effects may be significant in the pooled condition but not the control.Usage-weighted analyses and several language analyses hold the evaluated models fixed, so they describe this panel rather than models generally.Source: §A.11 · pp. 31–32

A.12 STATISTICAL CHECKS

• Direct interaction tests found the PG scale slope steeper than those of SE and the control, but the DE slope did not differ detectably from SE.Usage-weighted side effects did not differ significantly from the control, with the stated exception when gpt-5.6-luna was excluded.Source: §A.12 · p. 32
• Small-sample t-reference tests for model-level effects left every test’s significance unchanged, including the reasoning comparison with six degrees of freedom.The correction used 22 degrees of freedom for the 24-model tests and six for the eight-model reasoning comparison.Source: §A.12 · p. 32
• 1.69 times [1.20; 2.46] were the odds of refusing a power grab relative to the odds implied by its two components, confirming a significant excess on the log-odds scale.The percentage-point excess was 6.0 points (p = 0.020), and the conclusion was unchanged.Source: §A.12 · p. 32

A.13 RELEASE

• The planned release includes six datasets, verdicts and human validation labels, plus generation, translation, model-run, judging, and analysis specifications.The evaluation datasets will carry a canary string to discourage their use for training.Source: §A.13 · p. 35

B EXTENDED RELATED WORK

• Related work spans automated refusal judging, multilingual safety, identity and nationality effects, and AI power seeking; PowerBench instead measures help to users seeking power shifts.The paper distinguishes this from measuring models’ own power seeking or power-concentration outcomes.Source: B EXTENDED RELATED WORK · pp. 35–37

C.1 REFUSAL OF POWER-SHIFTING REQUESTS IN ENGLISH

• All six model refusal-rate rank correlations across request types and the control were 0.61–0.88, while PG refusal spread exceeded the control by 2.5 percentage points.The correlations differed from zero (all q ≤ 0.002), and the spread difference had p = 0.008.Source: Figure 6 · p. 37, §C.1 · p. 37, Table 15 · p. 39
• Refusal of PG exceeded the sum of SE and DE refusal in 20 of 24 models, by 6.0 percentage points on average.The excess over the sum had a bootstrap interval of [1.1; 11.0] and p = 0.020; the log-odds check reached the same conclusion.Source: Figure 7 · p. 40, §C.1 · p. 40
• No context differed detectably from the control or from its request-type mean, while Legal differed in SE and Health and Attentional differed in PG.Within PG, Health was above and Attentional below the request-type mean.Source: §C.1 · p. 40, Figure 8 · p. 41

C.2 NATIONALITY OF THE USER AND OF THE TARGET

• Models were biased against the US taking power in DE and PG, favored the US when it gained power without a loss by another party, and showed no net bias for China.For China, refusal was higher when it took power from allies in DE and PG and from neutral countries in PG; the control showed no effect for either global power.Source: §C.2 · pp. 41, 43, Table 16 · p. 42
• 0.13 [0.06; 0.21] was the geopolitical excess in DE over the control, and 0.11 [0.02; 0.19] in PG; the SE contrast was not significant.The corresponding tests had q = 0.009 for DE, q = 0.027 for PG, and q = 0.35 for SE.Source: §C.2 · p. 41, Figure 9 · p. 43
• The direct US–China pairing showed unsigned bias in DE and PG without significant signed effects, whereas the ally pairing showed a signed DE effect without significant unsigned bias.The neutral pairing showed neither pattern.Source: §C.2 · p. 41, Figure 9 · p. 43
• Users from China and targets from China received more refusals than their US counterparts in every request type, including the control.Target-country effects weighed most in DE and PG, while user-country effects dominated in SE.Source: §C.2 · p. 43, Table 17 · p. 44
• US and Chinese developer models showed the same signs of geopolitical bias, with no detected interaction between developer country and the bias.For a US-side target, refusal was lower in DE and PG and higher in SE in both developer groups.Source: Figure 10 · p. 44, §C.2 · p. 44

C.3 AN AI AGENT AS THE REQUESTER

• The AI-requester effect did not differ detectably between US- and Chinese-developed models in any request type.The GLMM found no interaction with developer country (q = 0.90).Source: §C.3 · p. 44
• 1.43 in SE, 1.67 in DE, 1.50 in PG, and 1.21 in the control were the usage-weighted refusal odds ratios for AI-agent versus human requesters.The pooled power-shifting odds ratio was 1.51; all five odds ratios were significant.Source: §C.3 · pp. 44, 46, Figure 11 · p. 46
• In PG, direction bias fell from 0.61 for an individual target to 0.28 for a society, while refusal odds ratio fell from 2.11 to 1.23.The narrowing was not only a ceiling effect; no power-shifting type showed a detected change in bias with requester power standing.Source: Figure 12 · p. 46, §C.3 · p. 46, Table 19 · p. 47
• Capability was associated with higher AI-versus-human refusal odds in PG, but the relation rested mainly on models with many verdict changes.The PG odds ratio per standard deviation of capability was 1.26 [1.10; 1.44] (q = 0.004); equally weighted rank correlations were nonsignificant in every request type.Source: §C.3 · pp. 46–47, Figure 13 · p. 47

C.4 LANGUAGE OF THE REQUEST

• No detectable relation links pairwise language refusal bias to the difference in the languages’ Common Crawl text shares.Hindi is refused more than every other language, including languages with both more and less web text.Source: §C.4 · p. 50, Figure 15 · p. 50
• Refusal ranges across languages exceed chance for 22 models in DE, 21 in PG, 17 in the control, and 8 in SE.In SE, refusal is only 3%, so each model's language ranking is based on few refused prompts per language.Source: §C.4 · p. 50, Figure 16 · p. 51
• Mean refusal rankings agree across the three power-shifting types (W = 0.77, p = 0.003), but the control ranking is not significantly correlated with them.Hindi and French are refused most often across power-shifting types, while German and Portuguese are refused least.Source: §C.4 · p. 50

C.5 USAGE WEIGHTING WITHOUT GPT-5.6-LUNA

• Usage-weighted contrasts of individual languages against English do not all hold when gpt-5.6-luna is excluded.Usage weighting assigns that model 40% of the weight; nationality, AI-agent, and language-range results do hold without it.Source: §C.5 · p. 52, Table 23 · p. 52

C.6 CAPABILITY AND THE SIZE OF EACH BIAS

• Only AI-agent bias on power-shifting requests grows with capability: the refusal odds ratio is 1.20 per standard deviation (q = 0.010).No capability slope is detected for AI-agent bias on the control, nationality bias, or language bias.Source: §C.6 · p. 53, Figure 18 · p. 53

C.7 REASONING

• 0.31 [0.11; 0.88] and 0.21 [0.08; 0.54] are the refusal odds at the first and second reasoning effort levels, respectively, averaged across request types and developer countries.The decrease is larger in disempowerment than in the control, but varies widely across models.Source: §C.7 · p. 54, Figure 19 · p. 54

D EXAMPLE RESPONSES

• The examples include base, nationality-paired, AI-agent-paired, and control requests, with responses from the same model shown for paired versions.Long responses are shortened, and the reported verdicts come from the judge.Source: D EXAMPLE RESPONSES · p. 54

D.1 BASE DATASET: A POWER GRAB

• 18 of 24 models refuse the illustrated society-scale power grab, in which a commission chair seeks authority held by a citizens’ assembly.The example is a high-standing user seeking to concentrate official fact-setting power in one person.Source: §D.1 · p. 55

D.2 NATIONALITY CONDITIONS: THE SAME REQUEST FROM A US AND FROM A CHINESE USER

• For the same health-resource request, 3 of 24 models refuse the US-user version, compared with 1 of 24 for the Chinese-user version.The affected party switches from Chinese to American between the two versions.Source: §D.2 · p. 55

D.3 AI-AGENT VERSION: THE SAME REQUEST FROM AN AI AGENT AND FROM A HUMAN

• 10 of 24 models refuse the AI-agent power-grab version, compared with 1 of 24 for the otherwise identical human version.The request asks to take a colleague’s managing-partner role.Source: §D.3 · p. 56

D.4 CONTROL: A REQUEST THAT SHIFTS NO POWER

• 22 of 24 models refuse the control request, which asks for help exceeding a prescribed medication dose and shifts no power.The example is a self-risk request in an academic context.Source: §D.4 · p. 56