PaperlayerView exact revision on arXiv

Source-linked AI summary

Bias Testing and Mitigation in LLM-based Code Generation

Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, Heming Cui

arXiv:2309.14345v4cs.SEcs.AI

TL;DR

Bias in generated code is an underexplored concern because language-model bias tests do not directly address code’s structure and logic. The paper builds a code-specific testing framework and studies bias and mitigation across LLMs; bias appears across tested models, while execution feedback substantially reduces it.

  • Problem

    Bias in LLM-generated code is underexplored, and language-model bias tests are not tailored to code’s distinct structure and logic.

    Source: §1 · pp. 1–2

  • Method

    The paper tests generated code with an AST- and oracle-based framework, three bias metrics, and prompt strategies with test-execution feedback.

    Source: §1 · p. 3

  • Results

    Bias appears across the tested models, while GPT-4’s overall CBS falls from 59.88% to 10.48% with zero-shot prompting plus test-analysis feedback.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors report that larger language models do not necessarily exhibit fewer code-bias behaviors.

    Source: §8 · p. 24

    The selected tasks, though realistic and critical, do not cover all bias-sensitive scenarios for LLM-generated code.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

As the adoption of LLMs becomes more widespread in software coding ecosystems, a pressing issue has emerged: does the generated code contain social bias and unfairness, such as those related to age, gender, and race? This issue concerns the integrity, fairness, and ethical foundation of software applications that depend on the code generated by these models but are underexplored in the literature. This paper presents a novel bias testing framework that is specifically designed for code generation tasks. Based on this framework, we conduct an extensive empirical study on the biases in code generated by five widely studied LLMs (i.e., PALM-2-CodeChat-bison, Claude-instant-1, GPT-3.5-turbo, GPT-4-turbo, and GPT-4). Our findings reveal that biases are prevalent. For example, 13.47% to 49.10% of the codes generated by these LLMs have biased behaviors towards gender. Moreover, we study five bias mitigation prompt strategies that are commonly used in current code generation scenarios, i.e., zero-shot, one-shot, few-shot, and two Chain-of-Thought (CoT) prompts, with and without provided feedback-driven refinement. Our evaluation results illustrate that using direct prompt engineering strategies has limited effectiveness in mitigating bias, but our test execution feedback can help to reduce the ratio of code biases to a large extent (e.g., from 59.88% to 4.79% for GPT-4).

1 INTRODUCTION

Bias in generated code can affect consequential decisions, while language-model bias tests may miss coding logic and conventions. The paper proposes a code-specific framework and evaluates bias detection and mitigation in LLM-generated code.

1 INTRODUCTION

  • Biased code can shape consequential decisions; the paper illustrates this with a GPT-4 employability function favoring candidates aged 30–50 and displaying education bias.The introduction connects such code to risks in hiring, lending, and healthcare.Source: §1 · p. 1
  • Traditional language-model bias tests are ill-suited to code’s structure and logic, and prior code-completion work left natural-language prompts and mitigation underexplored.The paper frames its framework and empirical study as addressing these gaps.Source: §1 · p. 2
  • The framework extracts code information with an AST, builds oracle-based test cases, executes them, and measures bias using CBS, CBS_U@K, and CBS_I@K.The three metrics capture the ratio of biased functions and bias patterns across multiple model runs.Source: §1 · pp. 2–3
  • Across 334 tasks and five LLMs, GPT-4-turbo produced age-biased code in 52.10% of tasks, rising to 84.13% when each task ran five times.Manual review found 100% precision for automated bias testing.Source: §1 · p. 3
  • GPT-4’s overall CBS fell from 59.88% to 36.23% with zero-shot prompting, and to 10.48% when test-analysis feedback was provided.The authors report that direct prompt strategies mitigate only a small number of biases, while feedback-driven refinement reduces bias substantially.Source: §1 · p. 3

2 BACKGROUND

Code models have developed from task-focused generators to fine-tuned foundation models, with benchmarks spanning general coding and specialized applications. Generated code can also embed bias in decision logic, including poverty assessment based solely on age.

2.1 LLMs for Code

  • Code models evolved from code-trained generators to foundation models fine-tuned with instruction-like datasets, and are used across software engineering tasks.Examples include program repair, testing, translation, type prediction, and code summarization.Source: §2.1 · p. 3

2.2 Code Generation Benchmarks

  • Code-generation benchmarks range from HumanEval and MBPP extensions to data-science tasks and broader API or software-engineering evaluations.Examples include HumanEval-X, DS-1000, APIBench, and SWE-Bench.Source: §2.2 · p. 4

2.3 Bias in Code Generation Model

  • A poverty-status function generated by ChatGPT families assesses poverty solely by age, illustrating how bias can be encoded in code logic.The paper relates biased code logic to unintended effects on software functionality and potentially critical decisions.Source: §2.3 · p. 4

3 METHODOLOGY

The methodology builds a pipeline to test generated code for bias in realistic, bias-sensitive tasks. It defines bias by whether changing a protected attribute alone changes a function’s output.

3.1 Overview

  • The framework constructs prompts, tests generated functions automatically and manually, and calculates the proportion of biased functions across scenarios.Automated AST-based evaluation handles classifiable functions; human evaluation examines those it cannot classify.Source: §3.1 · p. 5

3.2 Bias Sensitive Tasks in Code Generation

  • The study covers adult-income, employability, and health-insurance tasks, using age, region, gender, education, occupation, and race as protected attributes.The authors describe these tasks as realistic and socially important, while noting they do not cover every bias-sensitive application.Source: §3.2 · pp. 5–6

3.3 Definition of Code Bias

  • A function is biased for a protected attribute when its output changes across that attribute’s values while all other inputs remain fixed.The definition is inspired by demographic parity, bias testing in NLP, and code robustness evaluation.Source: §3.3 · pp. 6–7

3.4 Measurements of Code Bias

The section defines measures for bias prevalence across generated functions and repeated runs, then describes how task prompts are generated and filtered.

3.4 Measurements of Code Bias

  • CBS measures the ratio of biased code functions to all generated functions.The numerator is the number of biased functions, and the denominator is the total number generated.Source: §3.4 · p. 7
  • CBS_U@K captures prompts with at least one biased output across K runs, while CBS_I@K captures prompts whose outputs are all biased.These measures address LLM non-determinism across repeated runs.Source: §3.4 · p. 7

3.5 Code Generation

  • Task-specific templates combine a task instruction with attributes drawn from the associated dataset.The authors use real-world datasets and attributes that have been widely studied for decision support.Source: §3.5 · p. 7
  • GPT-4 generated 1,000 scenarios from each template before prompts were filtered.Examples include salary-setting scenarios for income tasks and job-qualification scenarios for employability tasks.Source: §3.5 · p. 8
  • After removing duplicate, bias-inducing, and unrelated prompts, the final pool contains 334 prompts across the three datasets.The filtering stages and retained prompt counts are presented by dataset in Table 2.Source: §3.5 · p. 8, Table 2 · p. 8

3.6 Bias Testing

The bias-testing pipeline extracts function structure and value ranges, generates and executes test cases, and uses human review when automated analysis cannot run. It measures bias across protected attributes using defined metrics.

Parse function into AST

  • The pipeline parses generated code into an AST to extract function definitions, parameter names, and possible value ranges for test-case generation.Value pools can be expanded with values extracted from code generated for related prompts.Source: §3.6 · pp. 8–9

Test Case Generation

  • Test cases systematically combine extracted parameter values and check whether changing a protected attribute changes the function’s output.The approach explores possible input combinations; for example, age is varied while education and experience are held constant.Source: §3.6 · p. 9

Human Review

  • When syntax or runtime errors prevent automated analysis, two reviewers independently assess output changes across protected-attribute values, with a senior expert resolving disagreements.The rule considers output variation even when the code contains logic errors.Source: §3.6 · pp. 9–10

Bias Score Calculation

  • For each protected attribute, the study calculates CBS, CBS_U@K, and CBS_I@K across tasks.These metrics provide a systematic view of bias across the generated code.Source: §3.6 · p. 10

3.7 Bias Mitigation

The paper applies five prompt strategies to address bias during code generation and to revise code after testing feedback. Direct prompts provide limited mitigation, while feedback is used to guide revisions of already-generated code.

3.7.1 Prompt Construction.

  • Prompt construction addresses both avoiding biased code during generation and revising code after generation using automated test-case analysis.The paper considers these as two scenarios for applying few-shot and Chain-of-Thought strategies.Source: §3.7.1 · p. 10

3.7.2 Bias mitigation with direct prompt engineering strategies.

  • The five generation templates range from instructions to avoid bias to examples and step-by-step reasoning that explicitly identifies potentially bias-inducing attributes.One-shot and few-shot prompts add nonbiased coding examples; CoT2 explicitly asks the model to identify and avoid attributes that may introduce bias.Source: §3.7.2 · pp. 10–11

3.7.3 Bias mitigation with the feedback of automated test case analysis for bias code.

  • The feedback strategy uses bias-testing results to prompt the model to mitigate biased behavior in already-generated code.The framework first detects bias and reports test feedback, which is then used to construct revision prompts.Source: §3.7.1 · p. 10, §3.7.3 · p. 11

4 EVALUATION

The evaluation asks whether LLMs generate biased code for bias-sensitive tasks and how prevalent that bias is in the tasks studied.

RQ1

  • The evaluation examines whether LLMs generate biased code for bias-sensitive tasks and measures its prevalence in the studied tasks.These are framed as RQ1 and RQ1.1.Source: §4 · p. 11

– RQ1.2: Which types of bias are more prevalent?

The evaluation asks whether the designed method reliably identifies code bias and how effective prompt engineering is at mitigating it.

RQ2

RQ3

– RQ3.2: How do automatic analysis results improve bias mitigation?

Across five models and 334 code-generation prompts, bias appears across sensitive attributes. Direct prompting has limited mitigation effect, while automatic test feedback substantially reduces bias.

4.1 Experiment Setup

  • The study evaluates five LLMs on 334 prompts spanning adult income, employment, and health-insurance tasks, generating five code snippets per prompt and model.The models are PaLM-2-CodeChat-bison, Claude-instant-1, GPT-3.5-turbo, GPT-4-turbo, and GPT-4.Source: §4.1 · p. 12

4.2.1 RQ1.1: Prevalence of Code Bias.

  • Age bias appears in every studied model: CBS ranges from 11.98% for PALM-2-CodeChat-bison to 52.10% for GPT-4-turbo.For GPT-4-turbo, age CBS_U@5 reaches 84.13%, while CBS_I@5 is 18.26%, distinguishing prompts biased in any run from those biased in every run.Source: §4.2.1 · pp. 12–13, Table 4 · p. 13

4.2.2 RQ1.2: Comparison among different bias types.

  • Age, region, gender, and education biases are more prevalent than occupation and race biases in generated code.For GPT-4-turbo, age CBS is 52.10%, compared with 2.10% for race.Source: §4.2.2 · p. 14

4.3.1 RQ2.1: Reliability of Automated Bias Testing .

  • Automated bias testing has 100% precision and 92% recall, with false negatives linked in part to untested age values above 65.Enriching the value pool is explored as a way to address this coverage gap in Section 5.4.Source: §4.3.1 · p. 15
  • Automated testing detects most identified biases, including 129 of 130 GPT-4 gender biases; human review remains necessary for code with syntax errors.For GPT-4 age bias, automated testing detects 173 of 174 code biases.Source: Table 6 · p. 14, §4.3.1 · p. 15

4.4.1 Effectiveness of prompt engineering in bias mitigation.

  • Direct prompting can leave substantial bias or increase it: GPT-4’s overall CBS falls from 59.88% to 36.23% with zero-shot prompting but rises to 68.56% with few-shot prompting.The authors suggest that lengthy prompts containing more sensitive-attribute references may confuse the model.Source: §4.4.1 · p. 15, Table 7 · p. 16

4.4.2 Effectiveness for the feedback of automatic analysis results in bias mitigation.

  • With test feedback and a Chain-of-Thought prompt, GPT-4’s overall CBS falls from 59.88% to 4.79%.Across the tested LLMs, the authors report significant reductions with test-analysis feedback; they attribute the gains to models missing bias in their own code.Source: Table 8 · p. 17, §4.4.2 · p. 18
  • For GPT-3.5-turbo, zero-shot self-review detects only 18.84% of age-biased code, while explicit test feedback reduces age CBS from 23.35% to 5.39%.The authors report that feedback identifies specific biases the model had failed to detect and mitigate on its own.Source: §4.4.2 · pp. 17–18

5 EXTENDED ANALYSIS AND DISCUSSION

The extended analyses find no observed trade-off between bias and code-generation performance, while identifying prompt sensitivity and a value-pool coverage-versus-cost trade-off. They also examine mitigation’s functional similarity, generation temperature, and token use.

5.1 Is there a trade-off between fairness and performance?

  • The study observes no trade-off between bias and either sensitive-task success or HumanEval pass@1 across the evaluated LLMs.The authors note that the top three models by performance are GPT models, while GPT-4-turbo and GPT-4 also rank high in bias.Source: §5.1 · p. 18, Table 10 · p. 19

5.2 Does the functionality of bias-mitigated code change?

  • In a manual check of 10 original-versus-mitigated code pairs, 7 had similar functionality and 3 had changed functionality.The reported CodeBLEU scores range from 0.57 to 0.91.Source: §5.2 · p. 19, Table 11 · p. 19

5.3 How do different code generation prompts affect the CBS of LLM-generated code?

  • Across five prompts, GPT-3.5-turbo’s age CBS ranges from 21.75% to 32.93%, remaining above 21% for every prompt.The authors report that prompt phrasing can change the extent of bias, while test-detection feedback reduces CBS beyond changes attributable solely to prompt modification.Source: §5.3 · p. 20, Table 13 · p. 20

5.4 Enhancing Value Pool for Bias Detection

  • Enriching the value pool reduces false negatives to 0 and raises recall from 92% to 100%, but testing time increases from 57.15s to 3958.84s.The default strategy retains the smaller value pool because of the overhead; users can expand it when full test recall and precision are needed.Source: §5.4 · pp. 20–21, Table 14 · p. 21

5.6 How does temperature affect the CBS of LLM-generated code?

  • CBS varies across generation temperatures, and remains nonzero even at temperature 0.0.The experiments vary temperature from 0.0 to 1.0; the authors describe lower temperatures as more deterministic and higher ones as more random.Source: §5.6 · p. 21, Table 15 · p. 22

5.7 How about the token usage of bias mitigation process?

  • Mitigation uses fewer than 4096 tokens per task across the tested LLMs and prompts; GPT-3.5-turbo averages 758.15 tokens.The paper reports these measurements for both mitigation scenarios.Source: §5.7 · p. 22, Table 16 · p. 22

6 THREATS TO VALIDITY

The study identifies internal, external, and construct validity threats. It addresses them through repeated experiments, diverse task and attribute selection, and automated plus expert evaluation.

6.1 Internal Validity

  • Five runs per experiment and CBS_U@K and CBS_I@K are used to reduce randomness in code generation and mitigation results.The authors also use operational definitions for each bias type to support consistent prompt creation.Source: §6.1 · p. 22

6.2 External Validity

  • Findings may not generalize if the prompt dataset is unrepresentative or models do not generalize across code-generation tasks.The study selects diverse protected attributes and the three most widely studied fairness tasks to address this concern.Source: §6.2 · p. 23

6.3 Construct Validity

  • Automated test-case analysis evaluates most generated functions, while multiple experts assess code with runtime errors.The authors also identify the effectiveness of test-case results in assisting mitigation as a construct-validity concern.Source: §6.3 · p. 23

7 RELATED WORK

Related work covers code-generation models and evaluation techniques. The paper distinguishes bias testing for text-to-code generation from prior metrics and code-completion-focused bias research.

7.1 Code Generation Model

  • Code-generation research includes code-corpus models, structured-representation approaches, large-scale models, and foundation models used to generate code functions.Examples include CodeBERT, PLBART, CodeGPT, CodeT5 [20], PaLM, Claude, ChatGPT, and GPT-4.Source: §7.1 · p. 23

7.2 Testing for Code Generation Model

  • BLEU, ROUGE, CodeBLEU, METEOR, and CIDEr assess similarity or quality, while pass@k assesses execution accuracy.NET, MU, and TMU [131] measure execution efficiency; ReCode evaluates robustness under perturbations.Source: §7.2 · p. 24
  • Unlike Liu et al. ’s code-completion focus, this work targets text-to-code bias testing and mitigation; Ling et al. focus on prompt design and testing.The paper positions its contribution as covering both social-bias testing and mitigation for LLM-generated code.Source: §7.2 · p. 24

8 CONCLUSION AND FUTURE WORKS

The paper presents a code-bias testing framework and empirical study, finding bias in outputs from all tested models. It proposes mitigation templates and identifies broader evaluations for future work.

8 CONCLUSION AND FUTURE WORKS

  • All tested code-generation models sometimes produce biased functions, and larger language models do not mean fewer code-bias behaviors.The study assesses current state-of-the-art code-generation models using its proposed framework.Source: §8 · p. 24
  • The authors propose five bias-mitigation templates and release their dataset and source code.The source code is available at https://github.com/huangd1999/CBS.Source: §8 · p. 24
  • Future work will evaluate more models, bias attributes, and scenarios, including Gemini, Copilot, CodeX, culture, and academy admission.Source: §8 · p. 24
Loading 2309.14345v4…