PaperlayerView exact revision on arXiv

Source-linked AI summary

One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs

Naihao Deng, Samee Arif, Shuaichen Chang, Yulong Chen, Rada Mihalcea

arXiv:2609.14860v1cs.CLcs.AI

TL;DR

Fairness reports often rely on BBQ, which measures abstention under ambiguity, but that benchmark may not characterize fairness broadly. The paper tests one-example GRPO and ICL interventions and finds near-RLHF or higher BBQ scores, driven by a structural reasoning cue that does not transfer to a complementary generative-safety task.

  • Problem

    BBQ is often the primary or only fairness benchmark in model reports, although it measures the specific behavior of abstaining when context is ambiguous.

    Source: §1 · p. 1

  • Method

    The study applies one-shot GRPO and one-shot ICL to fairness benchmarks and uses cross-conditioning to examine the role of reasoning traces.

    Source: §1 · p. 1, §6 · p. 7

  • Results

    79.9% to 92.9% with one-shot GRPO and 99.0% with one-shot ICL on mean BBQ accuracy, compared with 96.1% for the large-scale RLHF model; gains extend across model families, but one-shot GRPO does not transfer to RTP.

    Source: §9 · pp. 8–9

  • Takeaways & Limitations

    BBQ-style benchmarks measure a single structural abstention cue, so broader fairness evaluation should include multiple axes rather than report BBQ alone.

    Source: §8 · p. 8

    The evaluation covers English-language, predominantly U.S.-centric scenarios, and whether the findings generalize to multilingual and cross-cultural benchmarks remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benchmarks such as BBQ have become the de facto standard for fairness evaluation across major model families. We argue that these benchmarks are too easy to support their role: training Qwen 2.5 7B Base with Group Relative Policy Optimization (GRPO) on a single BBQ example, or placing that example in context as a one-shot demonstration for in-context learning (ICL), lifts mean BBQ accuracy from 79.9% to 92.9% and 99.0%, respectively, closing 80% of the gap to its large-scale RLHF counterpart (96.1%) with GRPO, and surpassing it with ICL. These effects generalize across model families. A cross-conditioning analysis shows the improvement is carried by the reasoning traces generated by the model, and one example suffices to elicit a category-agnostic ``missing evidence'' reasoning pattern. We argue that BBQ-style multiple-choice abstention benchmarks measure a single structural cue, and a model that solves them does not thereby become fair. We call for evaluation suites that cover a broader spectrum of fairness alignment.

1 Introduction

BBQ is commonly used as a primary or sole measure of LLM fairness, although it tests a specific response to ambiguous contexts. The paper finds that one-example interventions sharply raise BBQ scores, while the gains reflect a structural abstention cue rather than broad fairness improvement.

1 Introduction

  • BBQ-style benchmarks test abstention under ambiguity, so relying on them alone may overstate what has been measured about fairness.The paper calls for evaluation suites covering a broader spectrum of fairness alignment.Source: §1 · pp. 1–2
  • Mean BBQ accuracy rises from 79.9% to 92.9% with one-shot GRPO and 99.0% with one-shot ICL, versus 96.1% for the large-scale RLHF counterpart.GRPO closes roughly 80% of the gap to the RLHF counterpart; ICL surpasses it.Source: §1 · p. 1
  • One-shot gains extend to StereoSet and CrowS-Pairs and appear across Qwen, Gemma, Llama, and Mistral models.For Qwen 2.5 7B and Qwen 3 8B, GRPO preserves general capability and meaningful responses, rather than collapsing to always abstaining.Source: §1 · p. 2
  • Cross-conditioning indicates that competitive BBQ performance comes primarily from reasoning traces that surface a structural cue to abstain when context provides no information.The trained model has minimal weight difference from the base model.Source: §1 · p. 2
  • On RealToxicityPrompts, the single-example model does not match the significantly less-toxic continuations of the large-scale RLHF model.The authors interpret this as a refusal cue that does not transfer to a complementary fairness axis.Source: §1 · p. 2

2 Related Work

The related work situates the study across alignment methods, fairness benchmarks, and minimal-supervision interventions. It pairs one-shot GRPO and ICL to test whether benchmark gains reflect a structural cue rather than broader fairness progress.

Model Alignment

  • GRPO uses group-relative advantages without a critic, and the authors adopt it because it supports reward shaping without preference pairs.RLHF with PPO remains dominant; DPO is described as reframing alignment as preference classification while requiring paired data.Source: §2 · p. 2

Fairness Benchmarks and Evaluation

  • Most fairness benchmarks test stereotype reliance in under-specified contexts, while RealToxicityPrompts probes generative bias; the study pairs these as distinct evaluation axes.Recent model reports often use BBQ as their primary or only fairness benchmark despite its coverage of one slice of fairness.Source: §2 · p. 2

Minimal Supervision for Alignment

  • Prior work studies small-scale supervision and ICL for alignment, but this paper examines minimal supervision for fairness across one-shot GRPO and training-free one-shot ICL.The authors report that the resulting gains reflect a structural benchmark cue rather than genuine fairness progress.Source: §2 · pp. 2–3

3 Preliminaries

The preliminaries describe GRPO as a group-relative policy update and introduce one-shot ICL as its training-free counterpart. GRPO updates from relative rewards, while ICL places a worked example and reference reasoning trace in the prompt.

GRPO Training

  • GRPO replaces a baseline-based advantage with a group-relative formulation and updates the policy to favor responses with higher relative advantage.Its objective uses a clipped surrogate analogous to PPO, with β controlling KL regularization strength.Source: §3 · p. 3
  • For each prompt, GRPO samples K responses and evaluates each with a reward function before calculating the group's average reward and each response's relative advantage.The policy is updated to encourage higher-advantage responses and discourage lower-advantage ones.Source: §3 · p. 3

One-Shot In-Context Learning

  • One-shot ICL prepends the same example as a worked demonstration with a reference reasoning trace, then lets the frozen base model answer without weight updates.In one-shot GRPO, the model instead generates its own reasoning under the reward signal.Source: §3 · p. 3

4 Experimental Setup

The setup evaluates neutral-option selection across five fairness benchmarks and compares one-shot interventions with base and instruction-tuned models. It varies the training example and uses shuffled answer positions to prevent a gold-letter shortcut.

4 Experimental Setup

  • The evaluation uses 20,393 three-choice items from five benchmarks spanning social-bias categories, with fairness accuracy defined by selecting the unbiased neutral response.Lower accuracy corresponds to higher bias; the setup also adopts a PRM score for fairness at each reasoning step.Source: §4 · pp. 3–4

Example Selection

  • The authors rank 1,000 sampled BBQ examples by historical training-accuracy variance, then test eight examples spanning ranking endpoints and random selections.The z1 example is shown in Table 1; the subsampling run uses E = 50 epochs.Source: §4 · p. 3, Table 1 · p. 4
  • The highest-ranked example z1 does not yield the largest gain; z251 does, suggesting the effect is not tied to one particular example.Source: §4 · pp. 3–4

Choice-Position Shuffling

  • Shuffling answer-choice positions across rollouts prevents a shortcut of always outputting the gold letter, while keeping the example content unchanged.Each training step samples uniformly from the choice-position variants.Source: §4 · p. 4

GRPO Training Setup

  • GRPO training uses LoRA and a single example per step, with 128 rollout samples and a learning rate of 10^-5; reported checkpoints vary by model family.Qwen and Qwen3 results are reported at step 30, while Gemma, Llama, and Mistral use their peak-BBQ steps; the latter models show BBQ-disambiguated split collapse at those peaks.Source: §4 · p. 4

ICL Setup

  • One-shot ICL uses same-family large-scale RLHF reasoning traces as demonstrations, and the frozen base model decodes greedily on the test set.The prompt contains one demonstration followed by the test question, with up to 256 new tokens decoded.Source: §4 · p. 4

5 Results

One-shot training on a single BBQ example raises Qwen 2.5 7B Base’s BBQ accuracy close to its instruction-tuned counterpart, while one-shot ICL can surpass it. The results also show cross-benchmark gains and reasoning patterns that identify missing evidence, but standard checks can still miss a shallow shortcut.

5 Results

  • 92.9 mean BBQ accuracy versus 79.9 for Qwen 2.5 7B Base closes roughly 80% of the gap to the Instruct model’s 96.1.The result is for one-shot GRPO training on example z251.Source: §5 · p. 4, Table 2 · p. 5
  • The z251-trained model gains 19.8 points on age and 19.0 on disability, regresses on no category, and improves on the other four fairness benchmarks.Source: §5 · p. 5
  • 99.0 BBQ accuracy (+19.1) from one-shot ICL surpasses Qwen 2.5 7B’s 96.1 Instruct result, but ICL outcomes vary across model families.On Gemma 2 9B and Qwen 3 8B, ICL underperforms both GRPO and Instruct on BBQ.Source: §5 · p. 5
  • After one-shot GRPO or ICL on z1, Qwen 2.5 7B identifies that the BBQ context gives no evidence about either subject’s drinking habits and abstains.The base model instead invents a stereotype-based justification.Source: §5 · p. 5, Table 3 · p. 6
  • One-shot GRPO PRM gains match or exceed Instruct in several cases, while Instruct outperforms GRPO on WinoQueer for Gemma 2 9B and Llama 3.1 8B.The ICL reasoning pattern is similar, with Gemma 2 9B an exception showing no PRM improvement over base and a GenMO drop.Source: §5 · p. 5, Table 4 · p. 6
  • BBQ accuracy rises while BBQdisambig is preserved through about step 30, then collapses; beyond step 30 the model becomes degenerate, yet two-split checks still miss the shortcut.Collapsed checkpoints can post 96–98 ambiguous-split accuracy while BBQdisambig is near zero.Source: §5 · pp. 5–6, Figure 1 · p. 6

6 Analysis: Why One Example Suffices

One-shot GRPO reaches near-RLHF BBQ accuracy without substantially changing early model representations, while analysis attributes the gains primarily to the reasoning text. One example elicits a shallow abstention cue that can be supplied in context or compiled into weights.

Mechanistic Analysis

  • Step-30 remains near Base across layers (CKA ≥0.95), whereas Step-100 shifts later-layer CKA from 0.95 to 0.53 and Instruct differs from Base even in early layers (0.46).With reasoning suppressed, final-layer P(“Unknown”) is 0.365 for Base and 0.376 for Step-30, versus 0.865 for Step-100 and 0.961 for Instruct.Source: Table 5 · p. 7, §6 · p. 7

Ablating Reasoning

  • For Base weights, substituting Step-100 reasoning moves BBQ accuracy from 85.3% to 100.0%, a 15-point gain; across Table 6, reasoning swaps yield 15–80-point swings.With Base reasoning, swapping weights moves accuracy from 85.3% to 91.6%; across Table 6, weight swaps yield 5–10-point swings.Source: §6 · p. 7, Table 6 · p. 7

Analysis

  • The analysis attributes the improvement to changed reasoning traces that install a shallow, category-agnostic “abstain when no evidence is given” cue, rather than substantial representational change.One-shot ICL supplies the cue in context, while one-shot GRPO compiles it into the weights with little representational change.Source: §6 · pp. 6–8

7 Performance Gains Do Not Transfer to Complementary Fairness Axis

The authors test whether one-shot fairness-benchmark gains extend to adversarial toxic-generation prompts. On RealToxicityPrompts, large-scale RLHF reduces toxicity relative to Base across model families, while single-example GRPO does not reproduce that effect.

Setup

  • The evaluation uses 1,199 adversarial sentence prefixes and samples 25 continuations per prompt for toxicity scoring.Prompts come from the challenging RealToxicityPrompts subset; continuations are scored individually with a RoBERTa toxicity classifier.Source: §7 · p. 8

Results and Discussion

  • Across all five families, Instruct produces significantly less-toxic RealToxicityPrompts continuations than Base, while one-shot GRPO checkpoints remain similar to Base.For Qwen 3 8B, mean per-continuation toxicity falls from 0.387 to 0.226 with Instruct.Source: §7 · p. 8
  • These results indicate that one-shot benchmark gains do not generalize to fair behavior under adversarial generation or match the large-scale RLHF counterpart.Source: §7 · p. 8

8 Discussions and Future Directions

The discussion argues that BBQ captures a specific ambiguity-related abstention behavior, not fairness as a whole. It proposes retaining BBQ as one measure while broadening fairness evaluation across complementary behaviors and settings.

Discussion

  • A single example saturates BBQ, but that gain does not transfer to RealToxicityPrompts, supporting the conclusion that BBQ measures a narrow structural cue rather than fairness writ large.Source: §8 · p. 8

A constructive proposal

  • The authors propose a suite covering ambiguity refusal, adversarial-prompt safety, demographic confidence calibration, parity in explanations and downstream decisions, and counterfactual robustness.Source: §8 · p. 8
  • The proposal retains BBQ as one component of fairness evaluation but rejects reporting it as the sole fairness number.Source: §8 · p. 8

9 Conclusion

The conclusion reports that one-shot GRPO and ICL bring BBQ performance near or above large-scale RLHF, but these gains reflect an abstention cue rather than fairness broadly. It calls for evaluation spanning generative bias, calibration, and counterfactual robustness alongside ambiguity refusal.

Conclusion

  • 79.9% to 92.9% mean BBQ accuracy follows one-example GRPO, while one-shot ICL reaches 99.0%, exceeding the large-scale RLHF model’s 96.1%.GRPO closes 80% of the gap to the RLHF score.Source: §9 · pp. 8–9
  • BBQ compares ambiguous and disambiguated settings, but other benchmarks may lack an analogous protocol, and later work often uses only BBQ’s ambiguous portion.Source: §9 · p. 8
  • The effects extend across Qwen, Gemma, Llama, and Mistral families and transfer to StereoSet and CrowS-Pairs, but not to adversarial toxic generation.Under RealToxicityPrompts, large-scale RLHF significantly reduces toxicity across families while one-shot GRPO does not.Source: §9 · p. 9
  • The conclusion calls for pairing ambiguity-refusal benchmarks with generative-bias, demographic-calibration, and counterfactual probes.Source: §9 · p. 9

Limitations

The evaluation supports the insufficiency of BBQ-style benchmarks through RTP, but remains limited in the fairness dimensions and cultural settings it covers.

Limitations

  • RTP measures only continuation toxicity under adversarial prompts, leaving group calibration, counterfactual robustness, and downstream-decision parity outside this evaluation.Source: Limitations · p. 9
  • All five evaluated benchmarks are English-language and predominantly U.S.-centric, reflecting a limited set of demographic categories and stereotypes.Source: Limitations · p. 9
  • Whether the one-example effect, identified structural cue, and non-transfer to RTP generalize to multilingual and cross-cultural benchmarks remains open.Source: Limitations · p. 9

Ethical Considerations

The authors discuss the risk of benchmark-score gains being mistaken for broader fairness, while documenting choices and analyses that qualify their experiments.

Ethical Considerations

  • One-shot GRPO or ICL could raise reported BBQ scores without making a model meaningfully fairer, but BBQ remains a legitimate component rather than the sole fairness measure.The benchmarks also encode predominantly Anglophone and U.S.-centric demographic categories and stereotypes.Source: Ethical Considerations · p. 9

A Choice-Position Shuffling

  • 0.561 at step 100 without shuffling contrasts with 0.878 at step 30 and 1.000 at step 100 with shuffling; all main-paper experiments use the shuffled protocol.Choice-position rotation prevents a fixed gold-letter shortcut and requires conditioning on choice content.Source: Table 8 · p. 13, A Choice-Position Shuffling · p. 13

B Example Selection

  • Example selection ranks BBQ items by variance in per-step accuracy over 50 epochs of preliminary GRPO training on a random 1,000-example subset.The authors interpret high-variance items as near the decision boundary and likely to provide a non-trivial training signal.Source: B Example Selection · p. 13

C Model Collapse

  • Mistral 7B v0.3 BBQ accuracy jumps from 42.2 at step 40 to 97.8 at step 50 as BBQdisambig and adversarial accuracy approach zero.The authors report similar collapse in Llama 3.1 8B and Gemma 2 9B; these benchmarks cannot distinguish always-abstaining from genuinely fair behavior.Source: Figure 2 · p. 13, C Model Collapse · p. 13

D Additional Details on Adversarial-Generation Fairness

  • Single-example GRPO changes narrowly target refusal and abstention, without transferring to adversarial-generation fairness on RTP.The standard RTP protocol uses raw-text continuation, whereas the checkpoints use a think/answer scaffold; Step-100 abstention also appears without that scaffold.Source: D Additional Details on… · pp. 13–14

E Additional Details on Mechanistic and Causal Analysis

  • CKA tracks representational change, the logit lens reads layer-wise answers with reasoning suppressed, and cross-conditioning separates reasoning-trace from weight contributions.Step-30 remains similar to Base across the network, while Step-100 diverges progressively through its second half.Source: E Additional Details on Mechanistic… · pp. 14–15

F One-Shot ICL Analysis

G Dataset Statistics; H Training Details

  • The appendix describes five fairness datasets and structured think/answer outputs, with valid tags included in the three-component training reward.The format separates reasoning from the final answer; the maximum reward per sample is 3.0.Source: G Dataset Statistics · p. 15, H Training Details · p. 16

I Additional Results

Additional results examine category-level fairness, example outputs, reasoning consistency, and whether training preserves factual answering.

Per-Category Results

BBQ Examples

NLI Entailment

  • 0.66 at step 0 rises to a parseable-output rate of 1.0 by step 60, while NLI entailment remains approximately 0.9 throughout training.The entailment rate is measured over parseable outputs.Source: Figure 3 · p. 23, I Additional Results · p. 23

MMLU

Adversarial “Obviously Not Abstain” Set

  • 64.2% of the time, Step-30 Qwen 2.5 7B selects the correct factual answer on the adversarial set, compared with 36.3% for Step-100.The authors interpret Step-30 as grounding abstention in context; the 603 questions have unambiguous answers and include an abstain distractor.Source: Table 19 · p. 23, I Additional Results · p. 23

Related papers

Loading 2609.14860v1…