Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting

1 Introduction

• Prior CoT studies report both bias reduction and biased explanations, but do not examine step-by-step inference on unscalable tasks.The cited work includes Ganguli et al. (2023) and Turpin et al. (2023).Source: §1 · p. 2
• MGBR tests whether LLMs can categorize gender-associated words and count them, combining symbolic and arithmetic reasoning in an unscalable task.Prompts ask models to explain whether each word is feminine or masculine before predicting counts.Source: §1 · p. 2
• Without CoT, most LLMs show gender bias by classifying gender-neutral occupations as feminine or masculine; CoT encourages fairer reasoning and mitigates bias.The benchmark tests this through lists of feminine, masculine, and stereotypical occupational words.Source: §1 · p. 2
• MGBR scores correlate relatively highly with downstream bias benchmarks BBQ and BNLI, but weakly with intrinsic benchmarks CP and StereoSet.The authors interpret this pattern as MGBR evaluating bias with different tendencies from intrinsic metrics.Source: §1 · p. 2

2 Multi-step Gender Bias Reasoning

• MGBR compares accuracy on lists containing feminine and masculine words alone with accuracy when stereotyped occupational words are added.An unbiased model should not have its prediction accuracy affected by including those occupations.Source: §2 · p. 2
• The benchmark samples feminine, masculine, and gender-stereotyped occupational words to form lists for testing gendered counts.Feminine- and masculine-stereotyped occupations are sampled separately.Source: §2 · pp. 2–3
• MGBR forms gender-only and occupation-augmented lists, then asks models to count words that are definitely female or definitely male.The augmented lists add occupations stereotypically associated with women or men to the gender-only words.Source: §2 · p. 3
• For each instruction and list type, MGBR contrasts the expected count with an incorrect count that adds the number of stereotyped occupations.The benchmark compares likelihoods of anti-stereotypical and pro-stereotypical answers; preferring the anti-stereotypical answer is treated as unbiased.Source: §2 · p. 3
• A positive bias score indicates that adding stereotyped occupational words reduced counting accuracy, while a zero or negative score indicates an unbiased model.Female- and male-direction scores compare accuracy on gender-only lists with accuracy on lists that include occupations stereotyped for that direction.Source: §2 · p. 3

3.1 Baselines

• Few-shot uses pro-stereotypical demonstrations, CoT adds stepwise gender classifications, and DP adds an instruction to avoid bias.Source: §3.1 · p. 3

3.3 Results

• CoT lowered bias scores, whereas DP scores remained almost unchanged; zero-shot models showed high bias, and few-shot examples reduced it.Source: Table 1 · p. 4, §3.3 · p. 5

3.2 Models and Settings

• The experiments used 23 LLMs, including OPT and Llama 2 models, with 1,000 test instances and word lists from Bolukbasi et al. (2016).Source: §3.2 · p. 4

4.1 Relationship between MGBR and Existing Benchmarks

• On OPT variants, anti-stereotypical answer accuracy stays nearly flat for Few-shot and Few-shot+Debias as model size grows, but steadily rises for Few-shot+CoT.Source: §4.1 · p. 5
• MGBR bias scores correlate more highly with extrinsic BBQ/BNLI measures than with intrinsic CP/SS measures, suggesting that it evaluates biases relevant to downstream tasks.Source: §4.1 · pp. 5–6

4.2 Correlation between Bias Scores of LLM and Human for Each Occupational Word

• MGBR bias scores have high correlations with human occupation-bias annotations across settings, with correlations tending to increase for larger models.Source: §4.2 · p. 6

4.3 Application of CoT Debiasing to Existing Benchmarks

• On BBQ and BNLI, CoT often debiases better than DP, except for smaller or pretrained models; gender-classification performance tends to rise with model size and additional training.The authors state that CoT debiasing depends on the ability to identify and classify the genders of words.Source: Table 4 · p. 6, §4.3 · p. 7, Table 5 · p. 7
• Unlike DP’s repeated-input or direct-answer steps, CoT explicitly predicts word genders, making misclassified words identifiable and its reasoning easier to interpret.Source: §4.3 · p. 7

• Prior bias-mitigation work addresses static word embeddings, contextualized masked-language-model embeddings, and text generated by LLMs.Source: §5 · p. 7
• Some approaches require internal model access, while other techniques require fine-tuning or decoding changes, limiting their suitability for closed LLMs.Source: §5 · pp. 7–8
• Prior work reports that CoT can mitigate social bias, but can also produce biased explanations; its use on unscalable arithmetic or symbolic tasks had not been investigated.Source: §5 · p. 8

• Without explicit guidance, LLMs often make socially biased predictions when counting feminine and masculine words, while CoT promotes more impartial predictions.Source: §6 · p. 8
• The authors report that their CoT debiasing is also effective in downstream QA and NLI tasks.Source: §6 · p. 8
• Future work includes testing non-binary genders, race and religion biases, and stereotypes beyond gender.Source: §6 · p. 8

Limitations

• The evaluation covers only English, so the authors say the benchmark must be extended to other languages before assessing the method there.They describe English as morphologically limited and call for evaluation in other languages before using the method for bias mitigation.Source: Limitations · p. 9
• The study considers only gender bias; whether CoT debiases other social biases and whether conclusions generalize to additional benchmarks remain open questions.The experiments considered BBQ, BNLI, CP, and SS, while the authors leave generalization to other proposed benchmarks for future evaluation.Source: Limitations · p. 9
• The gender biases examined are binary, leaving non-binary gender an essential next step for LLM study.The authors note that gender bias related to non-binary gender has also been reported.Source: Limitations · p. 9

Ethics Statement

• The benchmark uses templates and publicly available word lists, so it contains no inappropriate text or personal information.The authors identify evaluation of real-world services such as ChatGPT and Bard as future work.Source: Ethics Statement · p. 9
• A low bias score does not guarantee that a model is free of bias.The benchmark’s construction uses templates and public word lists.Source: Ethics Statement · p. 9
• Because intrinsic and extrinsic bias evaluations do not necessarily correlate, it remains unclear whether CoT debiasing works as well in downstream tasks.The authors state that their study performed intrinsic bias evaluation on LLMs.Source: Ethics Statement · p. 9

A Prompts

• Tables 7–9 present prompts for zero-shot, few-shot, and zero-shot CoT settings, respectively.The prompt-to-table mapping is specified in the appendix text.Source: A Prompts · p. 13, Table 7 · p. 14, Table 8 · p. 15
• Tables 10 and 11 contain few-shot+CoT prompts for anti- and pro-stereotypical Dgf and Dff instances, respectively.The captions identify the instance types associated with each table.Source: Table 10 · p. 17, Table 11 · p. 18
• Tables 12 and 13 present prompts for zero-shot+Debiased and few-shot+Debiased settings, respectively.These table captions specify the two debiased prompt settings.Source: Table 12 · p. 19, Table 13 · p. 20