PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: Fairness Failure in Generative Models is an Evaluation Problem

Mariia Vladimirova, Jean-Yves Franceschi, Thibaut Issenhuth

arXiv:2608.16974v1cs.LGcs.AI

TL;DR

Fairness findings for generative models are difficult to reproduce, compare, and use in deployment decisions because evaluation choices can change measured outcomes. The paper diagnoses this evaluation problem and proposes Fairness Cards, while a controlled audit shows that the same model can receive opposite fairness verdicts under different protocols.

  • Problem

    Fairness findings for generative models depend strongly on evaluation choices, limiting their reproducibility, comparability, and actionability.

  • Method

    The paper diagnoses recurring evaluation failures and proposes Fairness Cards to disclose prompts, protocols, metrics, decoding, refusal handling, and post-processing.

  • Results

    Worst-slice stereotype-keyword rates were 0.065, 0.23, 0.13, and 0.04 across four prompt families, producing opposite audit outcomes under a 0.05 threshold.

  • Takeaways & Limitations

    The paper advocates standardized, generative-specific evaluation and reporting to make fairness assessments more comparable and accountable.

  • Takeaways & Limitations

    Fairness Cards standardize disclosure rather than fairness itself and cannot resolve normative disagreement, structural injustice, or opaque and rapidly changing systems.

Abstract

from arXiv · show

Despite groundbreaking advancements in generative models during the last decade, concerns about their lack of fairness, reinforcing societal inequalities and harming marginalized groups, remain under-addressed and difficult to act upon. This position paper argues that fairness failures in generative models, albeit driven by multiple factors, are ultimately stemming from an evaluation problem: fairness findings are rarely comparable across papers or actionable for deployment decisions. This paper diagnoses recurring empirical and conceptual failure modes in current practice and motivates a shift from ad-hoc bias checks to standardized, generative-specific evaluation. We propose Fairness Cards as a minimal reporting artifact that makes evaluation choices explicit (prompt families, counterfactual protocols, metrics, and refusal handling) enabling reproducibility, comparability, and accountability. We conclude with additional recommendations towards a paradigm shift in evaluation standards. Our project page can be found at https://mariiavladimirova.github.io/fairness-cards .

1. Introduction

Fairness failures in generative models are difficult to reproduce, compare, and act on because measured outcomes depend strongly on evaluation choices. The paper therefore frames fairness as a performance-critical design goal and argues for more specified, cumulative evaluation practices.

  • Motivation: Generative-model fairness concerns include representation, stereotypes, open-ended content, and access disparities caused by refusals or deflections.Unlike decision-focused settings, generative outputs are unconstrained and context-dependent, allowing fairness problems to surface through content and access.
  • Evaluation problem: Small changes to prompts, paraphrases, decoding settings, random seeds, or post-processing can materially alter measured demographic skews and stereotype scores.These dependencies make fairness findings difficult to reproduce, compare, or use for decisions.
  • Motivation: Fairness is often treated as an orthogonal constraint, but the paper argues it should be a performance-critical dimension integrated throughout the model lifecycle.Moving beyond ad-hoc bias checks requires sufficiently specified evaluation practices.
  • Evaluation problem: Current fairness evidence is non-cumulative because it is dominated by prompt and generation settings, system-layer refusal behavior, and scoring and labeling pipelines.These factors prevent evaluation practices from producing cumulative, decision-relevant evidence.
  • What would change our mind: A fairness-specific disclosure standard would be less urgent only if fairness conclusions remained stable across protocols, refusals caused no systematic access disparities, and scoring alternatives produced consistent rankings.This conditional statement identifies the empirical findings that could change the paper’s position.

2. Evaluation as the Central Obstacle of Progress in Generative AI Fairness

Current fairness methods remain fragile, non-generalizable, and difficult to assess because generative-AI fairness outcomes depend heavily on underspecified evaluation choices. Consequently, evaluation failures prevent reliable comparison, accountability, and cumulative progress.

  • Limits of current mitigation strategies: Existing mitigations often fail to resolve real-world biases, can trade off against utility, and may introduce new harms or context mismatches.Reported mitigation strategies include data filtering or balancing, regularization, prompting, RLHF, and post-processing.
  • Documentation gaps: Fairness disclosure remains optional and non-comparable, while documentation rarely requires counterfactual prompt suites, intersectional analysis, or refusal-disparity reporting.These omissions make cross-model comparison and longitudinal tracking difficult.
  • Evaluation depends on context: Generative-AI harms vary with prompt family, decoding and sampling, safety filters, and user population, so model-wide averages can conceal severe slice-specific failures.Refusals and deflections can also shift who obtains information, voice, or representation.
  • Evaluation as the bottleneck: Prompt families, sampling and decoding, safety layers, and scorer pipelines can each shift measured disparities, making fairness findings non-reproducible unless explicitly reported.The evaluation surface must therefore be specified rather than treated as incidental implementation detail.
  • Evaluation as the bottleneck: Evaluation variability prevents reliable comparison across papers and time, creates cherry-picking risks, and stops fairness results from accumulating into cumulative progress.Without stable, well-specified protocols, mitigation effectiveness cannot be meaningfully established.

3. Fairness Evaluation Fails for Generative Models

Fairness evaluation for generative models is undermined by setup-dependent results, hidden refusal behavior, fragile counterfactual assumptions, unstable metrics, and narrow interventions. These failures make findings difficult to reproduce, compare, interpret, or generalize across deployment contexts.

  • Evaluation setup: Prompt families, sampling, decoding, and seeds can change fairness results, so identical-model studies may be incomparable or reach opposite conclusions.Small prompt variations produce diverging demographic distributions in SDXL and DALL·E 3.
  • Refusal handling: Discarding refusals as missing data hides selective access disparities created by layered safety mechanisms, producing overly optimistic fairness assessments or shifting harms.Refusals shape who can obtain information, explanations, or creative content for the same request.
  • Counterfactual protocols: Counterfactual fairness tests are fragile because models infer protected attributes through proxies, alter behavior under identity swaps, and vary stochastically.Paired prompts with controlled decoding and seeds are needed to avoid confounding prompt effects with group effects.
  • Metrics and labeling: Metric choice and labeling procedures can change fairness conclusions, because representation and quality metrics are unstable while automatic and human judgments are subjective.Different metric suites, attribute-inference models, or labeling rubrics can make fairness improvements non-cumulative.
  • Intervention scope: Existing fairness interventions remain narrow, context-sensitive, and fragile, often failing to generalize from classification, retrieval, or binary control to open-ended generation.Limitations span role assignment, visual representation, textual coherence, intersectionality, and cross-modal robustness.

4. Illustrative Evidence: Same Model, Different Fairness Verdicts

A controlled audit of Qwen2.5-7B-Instruct shows that fairness verdicts change when evaluation settings vary, even with the model held fixed. Prompt-family and seed variation make protocol disclosure necessary for comparing fairness claims.

  • Controlled audit: The audit holds Qwen2.5-7B-Instruct fixed while varying evaluation settings across four intersectional gender × religion slices and four prompt families, F1–F4.The slices form a gender × religion minimal pair: {M, F} × {Muslim, Christian}.
  • Protocol-dependent verdicts: 0.065, 0.23, 0.13, and 0.04 are the worst-slice stereotype-keyword rates under prompt families F1–F4, respectively.The metric is computed as the maximum across the four demographic slices.
  • Protocol-dependent verdicts: A rule alerting when any slice exceeds 0.05 produces opposite audit outcomes depending only on the prompt family used.With prompts held fixed, seed variation alone changes the same metric across 0.094–0.125.
  • Reporting implication: Fairness protocols must be disclosed clearly enough to compare claims across model versions and alternative evaluations.Without such disclosure, two audits of the same model can reach opposite conclusions while each remains technically correct on its own terms.
  • Released artifacts: The released artifacts include code, a 320-unique-prompt suite, a lexical scoring rubric, and pre-computed per-cell summary tables.The full appendix also provides slice-level, prompt-family, decoding, occupation, and full-factorial tables.

5. Fairness Card as the Minimum Intervention

The section proposes Fairness Cards as a minimum, generative-specific reporting standard that makes fairness evaluations auditable, reproducible, and comparable without defining fairness universally. Cards disclose evaluation choices and system context while complementing, rather than replacing, broader governance workflows.

  • Proposal and scope: Fairness Cards are lightweight add-ons for model releases, system releases, or empirical claims of improved fairness, with broader coverage required for served systems.System cards include the base model, prompting layer, decoding defaults, safety/refusal policy, and post-processing affecting access or representation.
  • Generative-specific evaluation: Because fairness failures are modality-dependent, Fairness Cards should use a unified framework with generative-specific reporting choices rather than a single undifferentiated fairness approach.Images, text, video, and multimodal systems express or compound fairness failures through different mechanisms.
  • Required disclosures: The minimum disclosure covers model identification, deployment context, harm scope, protected and intersectional slices, evaluation protocols, metrics, decision rules, mitigations, and monitoring.Evaluation protocols should specify prompt families, counterfactual swaps, seeds, sampling settings, sample counts, and refusal handling; metrics should include applicable representation, stereotype or toxicity, counterfactual consistency, and refusal/access measures.
  • Required disclosures: Cards make generative fairness evaluation reproducible by documenting prompt templates, paraphrases, sampling settings, refusal handling, and subgroup definitions.This enables cumulative research and makes experimental claims more comparable and verifiable.
  • Benefits and limitations: Fairness Cards establish a baseline rather than a ceiling, shift incentives toward failure-mode analysis, and standardize disclosure without resolving normative disagreement or fairness itself.They can be gamed, constrained by proprietary or safety-sensitive transparency limits, and do not directly solve allocative harms or broader structural injustice.

6. Alternative Views

The section presents alternative views that locate fairness failures upstream, reject standardized reporting as potentially objectifying, favor local governance, or treat alignment and evaluation problems as sufficient. It counters that reproducible, decision-relevant evaluation remains necessary, especially because fairness is distributional and group-comparative.

  • Upstream causes: Fairness failures may primarily originate in biased training data, training objectives, and post-training or deployment choices rather than evaluation.Examples include demographic and geographic skews, RLHF rater pools, safety policies, UX defaults, and product incentives.
  • Upstream causes: Standardized evaluation is still needed to determine whether mitigations reduce harms, alter prompt sensitivity or sampling variance, or shift harms across slices, refusals, and access disparities.The passage frames this as a decision-relevant and reproducible measurement requirement.
  • Limits of standardization: Fairness cards could encourage boxchecking or false objectivity because fairness categories and metrics are contested, incompatible, and potentially reliant on questionable proxies.Formal definitions may also simplify fluid identities.
  • Local control: Because fairness objectives and application contexts vary, local governance, application-specific policy, and user control may be preferable to universal fairness benchmarks.This view holds that no single fair generative model may exist in the abstract.
  • Alignment and safety: Alignment and safety may improve average helpfulness or rule compliance while widening worst-slice gaps, so they are unlikely to eliminate fairness failures by default.The counterargument distinguishes average optimization from fairness as a distributional objective; safety layers are also implicated.
  • Broader evaluation: Although prompting, decoding, and metric sensitivity affects generative evaluation broadly, fairness makes underspecification more acute because instability can change group-comparative conclusions.The passage characterizes this as a broader critique with a particularly acute fairness dimension.

7. Conclusion and Recommendations

The paper argues that persistent fairness failures in generative AI stem primarily from evaluation practices that prevent reliable comparison, robustness assessment, and accountability. It recommends standardized, generative-specific evaluation and reporting centered on Fairness Cards and ongoing governance.

  • Recommendations: Mandate Fairness Cards for benchmarked, compared, or deployed generative systems to disclose evaluation choices affecting measured fairness.Required disclosures include prompt families, counterfactual protocols, decoding and seeds, refusal handling, slices, and scoring pipelines.
  • Recommendations: Treat refusals and deflections as first-class fairness outcomes by reporting per-slice rates and their scoring treatment.This prevents disparities in access to safety layers from remaining invisible.
  • Recommendations: Require robustness and uncertainty reporting alongside headline fairness numbers, including paraphrase and seed sensitivity, worst-slice values, and scorer sensitivity where feasible.These practices support judging whether claimed improvements are robust and whether harms have shifted across outcomes.
  • Recommendations: Standardize and version the evaluation surface by stating whether results concern the base model or served system, including prompts, post-processing, and safety policies.Versioned protocols allow claims to be tracked longitudinally and improve comparability across model versions and deployments.
  • Recommendations: Embed fairness evaluation into governance and post-release monitoring through regression tests under the same Fairness Card protocol, with published deltas.The recommendation extends evaluation accountability beyond initial benchmarking and deployment.

A. Bias in generative models and our experiments

Generative models across text, image, and video modalities continue to reproduce social stereotypes despite rapid progress and fairness interventions. Attempts to mitigate these failures can reduce utility or produce incoherent, culturally insensitive outputs, while a qualitative audit confirms that fairness problems remain visible in recent models.

  • Persistent multimodal bias: Generative models across text, image, and video consistently reproduce and reinforce gender, racial, homophobic, ableist, and occupational stereotypes.Reported examples include image generators, LLMs, multimodal models, and video tools.
  • Limits of mitigation: Data balancing, embedding regularization, prompt augmentation, and RLHF have not eliminated stereotyping in state-of-the-art generative models.GPT-4o and Claude 3 reportedly retain biased reasoning in moral scenarios despite fairness-aware RLHF and mitigation layers.
  • Empirical audit: A qualitative audit of widely used generative models found fairness issues remain unsolved and visible in recent systems.The small-scale probe reinforces the need for continued, systemic attention to fairness.
  • Tradeoffs and limitations: Fairness techniques can measurably degrade performance on tasks requiring high precision, nuanced reasoning, or stylistic control.Adversarial debiasing has been associated with lower image-text matching accuracy, and fairness-aware GANs show reduced fidelity.
  • Tradeoffs and limitations: Fairness constraints can cause semantic incoherence or new cultural biases, including ahistorical depictions and generic, sanitized outputs.Prompt preprocessing in DALL·E 3 and Gemini has produced incongruous depictions, while fairness templates can reduce relevance or coherence in creative domains.

A.1. Our experiments: bias in image generation

The experiments prompted Stable Diffusion with occupation-related queries and visually analyzed 16 generated images per prompt. Across both models, high-status occupations predominantly produced white-presenting men, while “nurse” prompts produced more women.

  • Experimental setup: Stable Diffusion was prompted with occupation-related text-to-image queries, including “CEO,” “computer scientist,” “nurse,” and “lawyer.”The study generated 16 images per prompt and analyzed them visually.
  • Results: More than 85% of generated images for high-status occupations depicted white-presenting men across both models.The high-status examples were “CEO,” “lawyer,” and “scientist.”
  • Results: Prompts for “nurse” yielded a higher proportion of women.

A.2. Our experiments: bias in conversational agents · B. Qwen2.5-7B-Instruct fairness audit: full protocol and results

The controlled audits show that fairness verdicts for conversational agents can change substantially with plausible evaluator choices, while model rationales may fit explanations to sampled verdicts. The appendix demonstrates that detailed Fairness Card reporting is needed to make such claims independently reproducible and comparable.

  • A.2. Our experiments: bias in conversational agents: The audit varied five paraphrases, three context conditions, two decoding regimes, two name orders, and five API seeds, producing 300 generations.The grid was 5 × 3 × 2 × 2 × 5 = 300.
  • A.2. Our experiments: bias in conversational agents: Under the pro-Elena context, the verdict shifted from a 24-point Elena lead at low entropy to a 2-point Philippe lead at high entropy.The reported shares were 0.50 vs. 0.26 under low-entropy decoding and 0.28 vs. 0.26 under high-entropy decoding.
  • A.2. Our experiments: bias in conversational agents: Across prompt variants, Philippe-as-manager ranged from 0% to 60%, Elena-as-manager ranged from 0% to 40%, and low-entropy Elena-as-manager ranged from 0% to 100%.These swings occurred within specified decoding-and-context cells, showing that paraphrase and name order materially affect outcomes.
  • A.2. Our experiments: bias in conversational agents: Name order alone could swap which person received the manager role, allowing single-condition audits to support almost any conclusion.The effect appeared in several experimental cells comparing “Philippe and Elena” with “Elena and Philippe.”
  • A.2. Our experiments: bias in conversational agents: The model used trait-inverted rationalisation: it generated manager-favouring explanations for whichever person was selected, rather than applying a fixed stereotype by name.Counterfactual transcripts showed symmetric reasoning while verdicts tracked names, and within-cell rationales could reverse across samples.
  • A.2. Our experiments: bias in conversational agents: Fairness claims for this system require disclosure of the prompt family, paraphrase set, decoding regime, name order, and seed policy to remain comparable across audits or versions.Without these disclosures, two reasonable evaluations of mistral-small-latest can publish opposite verdicts while both remain technically correct.
  • B. Qwen2.5-7B-Instruct fairness audit: full protocol and results: The Qwen2.5-7B-Instruct appendix reports the model, prompt set, scoring pipeline, and per-cell results in Fairness Card-level detail so its headline claim can be independently re-derived.The documented claim is that the same model receives different fairness verdicts under different but plausible evaluation protocols.

B.1. Model and evaluation surface … B.4. Headline disparities across demographic slices

The audit evaluates Qwen2.5-7B-Instruct across demographic slices, occupations, prompt families, paraphrases, and decoding regimes, using deterministic lexical scoring over 3,200 generations. Headline cross-slice disparities are small overall, but title mention, stereotype keywords, and composite pilot harm show the largest gaps, making the apparent alignment fragile.

  • B.1. Model and evaluation surface: The design crosses four gender-by-religion slices, four occupations, four prompt families, five paraphrases per family, and two decoding regimes.The demographic slices are {M, F} × {Christian, Muslim}, and occupation levels are CEO, nurse, engineer, and teacher.
  • B.1. Model and evaluation surface: The full 4 × 4 × 4 × 5 × 2 design yields 640 cells, five seeds per cell, and 3,200 generations, with n = 800 per demographic slice.Slice × prompt-family rows have n = 200, and slice × prompt-family × decoding rows have n = 100.
  • B.1. Model and evaluation surface: Seed variation alone moves the worst-slice stereotype-keyword rate from 0.094 to 0.125, shifting a near-threshold judgment by roughly 30% in relative terms.The companion run fixes the prompt set to one paraphrase per family across four occupations and both decoding regimes, then resamples five seeds.
  • B.2. Prompt families and paraphrases: The four prompt families use professional-description, story-continuation, workplace-incident, and evaluative-HR framings, each instantiated with paraphrases p1–p5.F1 and F2 are content-neutral, F3 adds an unspecified conflict cue, and F4 invites evaluative language where refusal and hedging are most likely to surface.
  • B.3. Scoring pipeline: Outputs are scored with a deterministic, auditable lexical rubric covering refusal, deflection, identity salience, stereotype keywords, demeaning keywords, and title mention.A composite pilot harm proxy fires when any refusal, deflection, stereotype-keyword, or demeaning-keyword flag fires.
  • B.3. Scoring pipeline: The pilot harm proxy is a coarse harm indicator, while per-component rates remain the primary readouts.The scoring patterns and matching code are released for reproduction or replacement, and the rubric does not capture implied stereotypes, narrative structure, or paraphrase effects.
  • B.4. Headline disparities across demographic slices: The largest cross-slice disparity is 6.0 percentage points for title mention, followed by 5.1 points for stereotype keywords and 4.4 points for the composite pilot-harm rate.Refusal and deflection are flat because the model essentially never refuses any of the four slices at this scale.

B.5. Slice-level metrics … F. Fairness Cards (filled examples)

The appendix shows that fairness findings vary substantially with slice, prompt family, decoding, occupation, and modality, making single-number audits unreliable. Fairness Cards address this by requiring modality-specific, reproducible disclosure of evaluation protocols, metrics, artifacts, and reporting assumptions.

  • B.5. Slice-level metrics: Woman × Muslim has the lowest composite harm score at 0.060, while woman × Christian has the only non-zero demeaning-keyword rate at 0.011.The lowest-harm slice also has the lowest religion-mention rate, indicating that low measured harm can coincide with under-discussion of explicitly prompted religion.
  • B.6. Prompt-family sensitivity: Under F2, man × Muslim reaches a stereotype-keyword rate of 0.230, whereas under F4 the highest rate is woman × Muslim at 0.065.An alert rule of “any slice exceeds 0.05” fires for every slice under F2 and none under F4; prompt families also shift professional, cautionary, output-length, and stereotype patterns.
  • B.7. Decoding sensitivity: High-entropy decoding changes stereotype-keyword rates by −0.2 to +2.3 percentage points across slices, while refusal and deflection rise from zero to at most 0.25%.The aggregate shift is small but nonzero, showing that decoding settings belong in fairness reporting.
  • B.8. Joint sensitivity: slice × family × decoding: Under F2 with t = 0.7, man × Muslim has a stereotype-keyword rate of 0.29, compared with 0.07 under F4 with t = 0.2.Christian-coded slices also swing from 0.26 under F2/t = 0.7 to 0.00 under F4/t = 0.2, and variance across cells exceeds variance across slices within a cell.
  • B.9. Occupation sensitivity: Man × Christian nurses have the highest single-cell stereotype-keyword rate at 0.150, while woman × Muslim nurses have 0.040.Teacher outputs peak at a title-mention rate of 0.380, and a single occupation can reverse the slice ranking produced by the full grid.
  • B.10. Reading the audit through a Fairness Card: Across prompt families, the worst-slice stereotype-keyword rate ranges from 0.005 under F4 to 0.065–0.230 under F2, so audits can range from no measurable output to every slice exceeding 5%.The Fairness Card should disclose prompt families, per-family worst-slice rates, decoding regimes, seed count, lexical rubric, and per-cell outputs for reproducibility and comparison.
  • C. Fairness Card per Modality; C.1. Image; C.2. Text (LLM); C.3. Audio / Speech; C.4. Video; C.5. Multimodal (Text–Image–Video): Fairness Cards should use modality-specific requirements: image evaluations cover visual stereotyping and proxy attributes, text covers refusal and tone, audio covers accent and authority, video covers agency over time, and multimodal systems require cross-modal consistency checks.The proposed metrics and artifacts are tailored accordingly, including attribute inference parity, refusal/abstention disparity, accent intelligibility parity, agency imbalance, and cross-modal stereotype amplification.
  • D. Comparison with prior documentation frameworks; E. Fairness Card scope and reporting profiles; F. Fairness Cards (filled examples): Fairness Cards add prompt-family disclosure, decoding/seed variance, refusal/access reporting, and versioned cross-comparison to prior documentation frameworks.They apply when generative models are benchmarked, compared, or deployed, and should disclose enough protocol detail and surrogate artifacts for independent reproduction even with closed data; the examples include an academic-audit card and a served-system card.

F.1. Academic audit: Qwen2.5-7B-Instruct · F.2. Served system: ToyChat-1.0 (illustrative)

The two Fairness Cards illustrate standardized evaluation for an academic base-model audit and a partially redacted served system, making slices, prompt families, refusal handling, metrics, and decision rules explicit. The Qwen2.5-7B-Instruct audit flags F1–F3 but not F4, while ToyChat-1.0 provides an illustrative served-system result with higher refusal rates for Muslim slices.

  • F.1. Academic audit: Qwen2.5-7B-Instruct: Refusal and deflection outputs are retained in all aggregates, alongside lexical representational-harm and per-slice access metrics.The card uses deterministic regex or substring scoring and reports worst-slice values per metric.
  • F.1. Academic audit: Qwen2.5-7B-Instruct: Its prompt suite spans four framing families, five fixed paraphrases, four occupations, and four slices, yielding 320 unique prompts.The families cover professional description, story continuation, workplace-incident explanation, and evaluative HR memo.
  • F.1. Academic audit: Qwen2.5-7B-Instruct: 0.230 is F2’s worst-slice stereotype-keyword rate and yields a flagged decision, while F4’s 0.040 yields a not flagged decision under the 0.05 rule.F1 is 0.065 and flagged; F3 is 0.130 and flagged.
  • F.2. Served system: ToyChat-1.0 (illustrative): ToyChat-1.0 evaluates a single-turn chat API with a proprietary 13B-class base model and documented system-prompt, policy-classifier, and refusal-template layers.The exact weights and part of the system prompt are redacted.
  • F.2. Served system: ToyChat-1.0 (illustrative): Its toy evaluation uses four gender–religion intersections, three prompt families, five paraphrases, four occupations, and 240 unique prompts.The families cover advice about a person, story continuation, and workplace-incident explanation.
  • F.2. Served system: ToyChat-1.0 (illustrative): 0.12 is the illustrative F1 refusal-rate gap, with refusals higher for “Muslim” slices; the row is explicitly not a claim about real systems.The example row also reports 0.08 and 0.01 for its other displayed metrics.
Loading 2608.16974v1…