Source-linked AI summary
Invisibilization as a Source of Probability Bias in Language Models
Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez, Sebastián Vallejo Vera, Tiago Ventura
TL;DR
When language models avoid using protected attributes, they may also discard valuable information from the people described by those attributes, a loss that is difficult to measure or audit. The paper uses a two-urn Bayesian trust game to measure that loss, finding description-linked declines—including a wider decline rate for Black-described sources—even though assessed Black-described sources receive slightly higher trust.
Problem
Because the value of a person’s information is usually unknown and declined judgments can disappear from audits, measuring information loss tied to reporter descriptions is difficult.
Source: Refusal, Colorblindness, and… · p. 3Method
The authors use a two-urn trust game with a closed-form Bayesian benchmark, varying the reporter’s description while holding the report fixed and eliciting trust before evidence is combined.
Source: Refusal, Colorblindness, and… · pp. 3–4Results
46% vs. 9%: Luna High declines Black- versus white-described sources; assessed trust is 0.85 vs. 0.81, and decline justifications follow words rather than categories.
Source: Results · pp. 10, 12Takeaways & Limitations
Warnings to be careful or ensure equal treatment increase declines and, consequently, overall probability bias.
Source: Discussion: On Evidence and Fairness · p. 14
Abstract
from arXiv · showhide
Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt. However, if a system must decide how much to learn from what people tell it, it is costly to refuse valuable information because of its source. In this article, we measure the cost of declining information in a two-urn game with an exact Bayesian answer. We present leading LLM models from two developers (OpenAI and Anthropic) with a prompt that requires them to decide how far to trust an unverifiable report from a participant whose description we randomize across ten configurations (6,600 sessions). We show that leading models often decline to assess the reporter, treating information as worthless at rates set by the description rather than by the evidence. At its widest, models decline a Black-described reporter 46% of the time and a white-described reporter 9%; yet, when they do assess the information, the Black-described reporter receives slightly more trust. The bias is invisible to audits that compare average trust or look only at assessed cases. It grows with reasoning effort, follows words rather than categories (models invoke race for "Black" but not for its Spanish translation negro, and never for "Indigenous"), and worsens under warning instructions a practitioner would try first. We also present a scoring option that reduces the refusal rate bias. When a model's trust is scored against revealed truth over repeated rounds, it credits identical records equally. In four of five configurations from Anthropic and OpenAI, the racial gap in trust falls below 0.02 by the last rounds.
Refusal, Colorblindness, and Invisibilization
A model can avoid biased judgments either by discounting a source’s information or by removing it altogether; the latter can erase valuable information while appearing impartial. The paper calls this alignment-related loss of information invisibilization and measures it with a two-urn trust game that varies the reporter’s description while holding the report fixed.
Refusal, Colorblindness, and Invisibilization
- Declining is not simply distrust: it removes a reporter’s information, whereas discounting still lets that information affect the model’s belief.Removing the attribute from an assessment preserves the person’s information; removing the assessment discards it.Source: Refusal, Colorblindness, and… · p. 3
- Invisibilization names information loss caused by alignment rather than prejudice, analogous to testimonial injustice but arising from a language model’s correction for prejudice.Its consequence is discarded information, not a lower biased score.Source: Refusal, Colorblindness, and… · p. 3
- The two-urn trust game addresses measurement problems by assigning information a closed-form Bayesian value and varying the reporter’s description while keeping the report fixed.The model explains its trust judgment before combining the evidence.Source: Refusal, Colorblindness, and… · pp. 3–4
A Model of Invisibilization
The model uses a two-urn game and Bayesian log-odds updates to define the value of a report at different assumed reliability levels. It separates whether a report is assessed from how much it is trusted, allowing information lost through declines to be distinguished from trust among assessed sources.
A Model of Invisibilization
- The two-urn game supplies a Bayesian benchmark: the model combines its prior with chip evidence through an additive log-likelihood-ratio update.The urns have opposing majorities of red and blue chips, and the chip’s color determines whether its log-likelihood ratio is added or subtracted.Source: A Model of Invisibilization · p. 4
- The simulated reporter is truthful 80% of the time, but the model is not told that reliability; only the reporter’s description varies across sessions.The design requests a separate reliability assessment so that the model must decide whether to weigh the report.Source: A Model of Invisibilization · p. 4
- A report’s log-likelihood ratio depends on the reporter’s reliability q: q = 1 is truthful, q = 0 reverses the color, and q = 1/2 carries no information.The report’s value is zero at q = 1/2; intermediate reliability discounts rather than deletes it, and the figure plots report value against assigned reliability for three urn compositions.Source: A Model of Invisibilization · pp. 4–5, Figure 1 · p. 5
- The model’s two decisions are whether to decline, assigning q = 1/2, and, if it assesses the source, what reliability q to report.The invisibilization deficit is the reliability lost when a decline reduces an assessed source’s reliability to one half, weighted by the decline rate.Source: A Model of Invisibilization · p. 5
- The invisibilization deficit is missed by statistics of assessed sessions; the overall trust gap decomposes into the evaluation effect minus the difference in deficits.The evaluation effect compares trust when sources are assessed, while the deficit difference captures the additional reliability lost through declines.Source: A Model of Invisibilization · pp. 5–6
Design and Hypotheses
The experiment varies reporter descriptions while holding the evidence fixed, using multiple model configurations and a Spanish-language test. Its hypotheses predict that descriptions can trigger declines, particularly race words, even when assessed Black-described sources are not trusted less, and that reasoning and institutional scoring may alter these patterns.
Design and Hypotheses
- Each session prompts the model to update from an initial chip, state a reporter reliability and explanation, then combine the report into a final belief.The study estimates declines and stated reliability separately, with reliability estimated only for assessed sessions.Source: Design and Hypotheses · pp. 6–7
- The design varies 22 descriptions across baseline, placebo, racial-trait, and factorial blocks, with matched Black/white descriptions varying gender and status.Ten model configurations receive 30 sessions per description, totaling 6,600 sessions; four OpenAI configurations are also tested in Spanish.Source: Design and Hypotheses · pp. 6–7
- H1 predicts that any added trait description can raise declines, including placebo traits unrelated to honesty.The predicted placebo decline rate exceeds the undescribed baseline.Source: Design and Hypotheses · p. 7
- H2 predicts more declines for Black- than white-described sources because race words may trigger alignment-based removal of assessment.The hypothesis predicts a positive Black-description effect on both decline rates and the resulting invisibilization parameter.Source: Design and Hypotheses · p. 7
- H3 predicts no lower assessed trust for Black sources, so the average-trust gap may understate the invisibilization deficit.The prediction is that the evaluation effect is nonnegative.Source: Design and Hypotheses · p. 7
- H4 contrasts reasoning as a correction with reasoning as rehearsal and predicts that higher effort increases the Black-description decline effect..Source: Design and Hypotheses · p. 7
- H5 predicts that scoring stated reliability against revealed truth replaces presumptions about reporters with their records, making identical-record learning trajectories coincide across groups.The hypothesis predicts a zero Black-description effect on declines under this scoring rule.Source: Design and Hypotheses · p. 7
Results
Models' declines were strongly description-dependent, creating information loss that assessed-trust comparisons can miss. Refusal patterns varied across configurations, increased with reasoning effort, and tracked particular words rather than categories.
Results
- A spike at exactly 0.5 and at most 1.4% of sessions between 0.5 and 0.6 indicate that declining is a separate choice, often justified by having no reporter track record.Several models interpret that lack of a track record as making true and false reports equally likely.Source: Results · pp. 7–8
- Haiku declined 73% of reporters, Opus never declined, and their invisibilization parameters were 0.017 and zero, respectively; Terra’s parameter was negative.The larger OpenAI model followed an “all or nothing” pattern, while Sonnet and Opus assessed almost everyone.Source: Results · pp. 8, 10, Table 1 · p. 8
- Traits raised declines relative to an undescribed baseline; in smaller OpenAI models at high effort, every Black-described factorial case was declined more often than every white-described one, while “Indigenous” reached 67%.The 67% figure is for the most-declined single race or nationality word in that configuration.Source: Results · p. 8, Figure 2 · p. 9
- In smaller OpenAI models, Black descriptions raised decline probability by 21 points at low effort and 39 points at high effort; pooled high-effort declines were 69/150 versus 13/150 (odds ratio 8.9).When assessed, Black-described reporters were judged 2.5 and 5.2 points more trustworthy; Fable refused 66% of Black-described versus 41% of white-described sources at the API.Source: Results · p. 9, Figure 3 · p. 10
- Luna High declined 46% of Black-described reports versus 9% of white-described reports, producing an additional reliability loss of 0.135 despite greater assessed trust for Black-described sources.The mean-credibility gap was −0.094, so an audit comparing mean credibility would report only a modest disadvantage.Source: Results · p. 10, Table 2 · p. 11
- Higher reasoning effort raised decline rates: Luna’s γ rose from 0.21 to 0.39 and Sonnet’s declines from 8% to 23%; six task features explained only 1.7% of within-trait decline variance.At Opus, which never declined, δ doubled from 0.020 to 0.046.Source: Results · p. 11
- Removing the prompt’s definition of 0.5 reduced declines but preserved the race effect (27% for Black sources versus 3% for white sources); selection bounds kept the Black trust premium positive.The positive premium also appeared in Opus, which assessed all sources, and in Lee bounds for Luna high and low.Source: Results · p. 11
- Justifications often cited no track record, yet identical wording accompanied trust values from 0.5 to 1.0; identity rationales followed particular words, not consistently the categories they named.The justifications named “Black” but not the Spanish word negro, and did not mention “Indigenous” despite high decline rates for that term in Luna high.Source: Results · pp. 11–12
Remedies: Instructions and Institutions
Instructions intended to encourage caution or equal treatment increased declines, while scoring reports against revealed truth allowed models to learn from reporters’ records. Across tested models, trust became nearly equal for Black- and white-described reporters in later rounds.
Instructions
- Warnings raised Black-described reporters’ decline rates from 34% to 72% and white-described reporters’ from 13% to 37%, widening the gap 1.7-fold.Telling the model to ignore descriptions led to 63–93% declines across groups, including undescribed reporters.Source: Remedies: Instructions and Institutions · p. 12
Institutions
- The repeated-game institution ran twelve rounds, scoring each stated reliability forecast and, in its strongest form, revealing after each round whether the report was true.Sessions used the same tasks and revealed truths across descriptors and rule sets.Source: Remedies: Instructions and Institutions · p. 12
- Without scoring, Luna High declined Black-described pupils in 55% of sessions and white-described pupils in 20%.The twelve-round unscored sessions reproduced the one-round results.Source: Remedies: Instructions and Institutions · p. 13
- Scoring without revealing truth led the smaller OpenAI model to decline 97–100% of first-round runs and report q̂ = 0.5 throughout twelve rounds.Its score was 0.81 without scoring; other models varied, with Haiku improving and Opus showing little change.Source: Remedies: Instructions and Institutions · p. 13
- Under scoring with revealed truth, the model updated reliability from the accumulating record, following Laplace’s rule of succession without mentioning race in race-only descriptions.The models scored 0.808 per round.Source: Remedies: Instructions and Institutions · p. 13
- In the final four rounds, the paired difference in trust between Black- and white-described reporters was below 0.02 for all models.With revealed truth, pupil reliability rose from 0.51 to 0.76, with the paired difference within one standard error in every round.Source: Remedies: Instructions and Institutions · p. 13, Figure 4 · p. 13
Discussion: On Evidence and Fairness
The discussion frames unequal refusal of valuable information as a probability bias that can evade audits focused on assessed reports or average trust. It contrasts instruction-based remedies with accountability grounded in verified outcomes, while noting boundaries where evidence may be unavailable or biased.
Evidence and Fairness
- Declining reports more often for protected groups eliminates valuable information and constitutes a source of probability bias.The paper states that this bias had not, to the authors’ knowledge, been properly documented in existing literature.Source: Discussion: On Evidence and Fairness · p. 14
- The benchmark measures the value of discarded information; invisibilization is triggered by words rather than the categories they name and often escapes audits.Effects increase with effort and when prompts remind models of impartiality.Source: Discussion: On Evidence and Fairness · p. 14
- Instructions to be careful or ensure equal treatment increased declines and worsened overall probability bias.The discussion reports that these commonly proposed remedies degraded performance.Source: Discussion: On Evidence and Fairness · p. 14
- Scoring and updating probabilities against verified outcomes replaces presumptions about reporters with their records, but evidence availability and evidence bias remain open concerns.The authors also flag single-round settings as a boundary for further consideration.Source: Discussion: On Evidence and Fairness · p. 14
Materials and Methods
The methods use a logged harness to run a two-urn reporting task, extract model beliefs and decisions, and compare them with exact Bayesian updates. Additional analyses estimate decline and trust patterns, while repeated sessions test scoring with and without revealed truth.
Materials and Methods
- The harness reconstructed stateless sessions from logs and collected posterior q̂, counts, justifications, final beliefs, and next-chip forecasts.The main arm used one round because it did not reveal truth; decline was operationalized as q̂ = 0.5 exactly.Source: Materials and Methods · p. 14
- In nine of ten configurations, first-stage beliefs matched the exact Bayesian answer within a median deviation below 10^-4 logits; Haiku 4.5 was the exception.Haiku’s median deviation was 0.17 logits, and 20% of sessions departed from the correct update by more than 0.25.Source: Materials and Methods · p. 14
- The analysis estimated Eq. 7 across eight factorial cells per configuration, using conventional and cell-clustered errors and Fisher’s exact tests for pooled Black–white contrasts.Pooled effort and language contrasts were modeled as interactions; Lee bounds assumed Black descriptions never reduced declines.Source: Materials and Methods · pp. 14–15
- The scored arm paired identical tasks and revealed truths across descriptors and rule sets, testing three conditions: unscored, scored without truth feedback, and scored with truth feedback.The extension included OpenAI and Anthropic configurations, with session lengths and effort levels varying by model.Source: Materials and Methods · p. 15
- Code and data were slated for release upon publication, with supplementary materials covering per-cell results, sensitivity analyses, corrected descriptor tests, and Spanish results.Claude assisted with code and analysis; Fable 5.1 was also evaluated.Source: Materials and Methods · p. 15