PaperlayerView exact revision on arXiv

Source-linked AI summary

A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Code

Manaal Basha, Aimee M. Ribeiro, Gema Rodriguez-Perez

arXiv:2609.30642v1cs.SEcs.AI

TL;DR

Research on AI-generated code has documented bias but has limited systematic identification, categorization, and explanation of it. This study builds an annotated Python-code dataset and evaluates LLMs using ICL for post-hoc detection and explanation. The authors report a broader range of bias types and moderate-to-high alignment between generated explanations and expert justifications.

  • Problem

    Research has limited systematic identification and categorization of bias in AI-generated code, and little evaluation of whether post-hoc explanations align with expert judgment.

    Source: §1 · p. 2, §2.4 · p. 5

  • Method

    The study extends a dataset of biased AI-generated Python code with manual bias-category labels and human justifications, then evaluates proprietary and open-source LLMs using ICL.

    Source: §1 · p. 3

  • Results

    The authors report a broader range of biases than prior work primarily explored and moderate-to-high explanation alignment with expert justifications, with variation in scope, detail, and granularity.

    Source: §7 · p. 23

  • Takeaways & Limitations

    The findings move analysis beyond whether bias exists toward how it manifests, can be identified, and can be explained in AI-assisted software engineering.

    Source: §7 · p. 23

    The experimental pipeline is restricted to isolated Python functions, so the findings may not fully generalize to other languages or larger software systems.

    Source: §6 · p. 22

Abstract

from arXiv · show

As Large Language Models (LLMs) become integrated into software development workflows, concerns regarding unintentional biases in AI-generated code. Although evidence suggests these biases exist, limited research has systematically identified, categorized, and explained them. This study investigates bias in AI-generated code and evaluates whether LLMs can reliably identify and explain it through a taxonomy-driven framework. We extended an existing dataset of biased AI-generated Python code and manually annotated snippets with bias categories and human-authored justifications to establish a ground-truth dataset. Using this dataset, we evaluated proprietary and open-source LLMs as automated bias detection and justification systems through ICL. Finally, we analyzed similarity between LLM-generated explanations and human-authored justifications using structured justification and code identification metrics. Our findings demonstrate that LLMs can effectively support code bias identification and explanation. Gemini achieved 80.14% classification accuracy, with 84.0% precision and 95.7% recall, while the best open-source alternative, Qwen3-coder, achieved 82.45% accuracy, 68.64% precision, and 80.22% recall. Additionally, the models achieved justification similarity scores of 80.4% and 80.14%, respectively, relative to human-authored reasoning, and code identification similarity scores of 86.0% and 87.82%. These results suggest that LLMs can detect biased logic in generated Python code and produce explanations that substantially align with expert interpretations.

1 Introduction

As AI-generated code becomes part of routine development, developers need ways to identify and understand potentially biased logic. This study uses in-context learning to examine post-hoc bias detection and explanation against human-authored annotations.

1 Introduction

  • Prior code-bias research largely measures prevalence or mitigation during generation, leaving post-generation auditing and explanation reliability underexplored.The authors also identify limited exploration of fine-grained bias types and alignment with expert annotations.Source: §1 · p. 2
  • Post-hoc bias analysis must identify biased logic and explain why it is biased to support developers’ evaluation of generated code.The paper argues that detection without explanation is insufficient for appropriate human judgment.Source: §1 · p. 3
  • The study extends a biased AI-generated Python-code dataset, manually annotates bias categories and justifications, and evaluates proprietary and open-source LLMs using ICL.Explanation similarity is assessed against human justifications with structured justification and code identification metrics.Source: §1 · p. 3
  • The paper characterizes semantic and group-based biases as most prevalent and evaluates ICL detection and explanation alignment, while contributing a fully labelled dataset.Its findings are presented as a foundation for research on auditing, explainability, and mitigation.Source: §1 · p. 3

2 Related Work

Prior work documents bias in generated code and develops bias evaluation approaches, but much of it focuses on generation or binary presence rather than structured post-hoc analysis and explanation. The related work also draws on fairness concepts and taxonomies spanning software and adjacent domains.

2 Related Work

  • Bias can be harmful or an unavoidable reflection of perspective and context, so the paper distinguishes avoidable bias from forms that must be accounted for.The discussion presents bias awareness as a basis for classifying bias and justifying why code should be considered biased.Source: §2 · pp. 3–4

2.1 Bias in AI-Generated Code

  • Prior AI-generated-code studies examine demographic and provider bias, but generally focus on bias emerging during generation rather than post-hoc auditing.The paper identifies binary presence-focused work as leaving fine-grained bias types and bias-typed datasets underexplored.Source: §2.1 · p. 4

2.2 Bias Detection and Explanation Beyond Text

  • Text-focused tools such as Dbias and Bias-Aware Agent demonstrate modular bias analysis, but executable code can encode bias in control flow, logic, or API use.The cited approaches focus on unstructured natural language rather than executable artifacts.Source: §2.2 · pp. 4–5

2.3 Protected and Non-Protected Attributes in Bias Definitions

  • The paper’s bias definition considers protected attributes and non-protected proxy variables, which can encode bias even when protected attributes are absent.The taxonomy and annotation guidelines account for both attribute types.Source: §2.3 · p. 5

2.4 Evaluation Methodologies Across Domains

  • Code-bias evaluation uses scenario-based measures such as CBS and distributional measures such as the Gini Index, while post-hoc explanation reliability remains underexamined.The paper identifies comparison with experienced human judgment as a gap in evaluation.Source: §2.4 · p. 5

2.5 Relevant Biases

  • The authors synthesize literature on algorithmic fairness, software engineering, and AI-generated code into a unified taxonomy, and identify documentation/naming and proxy bias as underexplored.The taxonomy is intended as the conceptual foundation for the annotation framework.Source: §2.5 · p. 5

2.7 In-Context Learning for Post-Hoc Bias Classification

Prior work motivates in-context learning as a lightweight alternative to fine-tuning for bias analysis. It uses prompting without changing model parameters, though this paper notes potential trade-offs and concerns reported for fine-tuning.

2.7 In-Context Learning for Post-Hoc Bias Classification

  • Fine-tuning may suppress useful sociolinguistic information, worsen some social biases, or cause catastrophic forgetting in fairness-sensitive settings.These concerns motivate moving away from supervised fine-tuning for downstream bias classification tasks.Source: §2.7 · p. 7
  • ICL uses contextual prompts without modifying model parameters, making it a lightweight post-hoc option when access to model internals or retraining is limited.Prior work cited in the paper reports few-shot performance and reduced false-positive and false-negative disparities in clinical documentation and electronic health records.Source: §2.7 · p. 7
  • Prior work reports that ICL can mitigate some structural biases without parameter updates or expensive retraining, supporting its use to prompt analysis of bias in generated code.The paper notes that ICL avoids some performance trade-offs and bias-amplification concerns associated with fine-tuning.Source: §2.7 · p. 7

2.8 Gaps and Opportunities

The authors identify gaps in fine-grained bias categorization, post-generation analysis, and evaluation of explanation alignment with expert judgment. They address these gaps through post-hoc ICL analysis and an expanded annotated dataset.

2.8 Gaps and Opportunities

  • Prior work often treats code bias as binary, leaving fine-grained bias types and bias-typed datasets insufficiently explored.The paper notes uncertainty about what structured forms bias takes in practice and what previously unlabelled categories may exist.Source: §2.8 · p. 7
  • Few studies treat detection as post-hoc analysis, despite its relevance to auditing arbitrary code outputs.Existing work has more often studied in-generation mitigation.Source: §2.8 · p. 7
  • The study addresses these gaps by analyzing code after generation with ICL, evaluating explanations against expert annotations, and providing an expanded annotated dataset.The dataset supports future research on bias detection, explanation, and mitigation.Source: §2.8 · p. 7

3 Experimental Design

The study builds an annotation framework and ground-truth dataset for examining bias in AI-generated code. It operationalizes bias categories and structures justifications to support consistent annotation and comparison with LLM explanations.

3.1 Data Source and Preparation

  • The manually labelled test subset of 784 snippets from a 7,840-snippet Python dataset supported taxonomy annotation and expansion to all 7,840 data points.The source dataset was introduced by Huang et al..Source: §3.1 · p. 8

3.2 Bias Taxonomy and Labelling Schema

  • The taxonomy groups bias by protected attribute, association type, and bias type, refining literature-derived categories into labels observable in individual code snippets.Rule-based dependencies constrain valid label combinations; for example, Threshold/Cutoff requires a threshold comparison, and Representational bias applies only to positive associations.Source: §3.2 · pp. 8, 10
  • The schema excludes categories needing broader system or deployment context and retains categories supported by program logic, naming conventions, or decision rules.Protected Attribute and Association Type specify whether a protected attribute is referenced and whether the outcome is favourable or unfavourable.Source: §3.2 · pp. 8–9

3.3 Standardized Justification Framework

  • Each justification separates the code trigger, the mechanism by which it creates bias, and the broader impact into distinct reasoning components.The framework is an annotation schema, not a restriction on bias detection; it supports comparison of human and model explanations.Source: §3.3 · pp. 10–11

1. Trigger

The framework treats a trigger as the code element that introduces or enables bias.

1. Trigger

  • A trigger is a specific code element that introduces or enables bias, such as conditional logic, thresholds, names, defaults, scoring rules, or ranking criteria.Source: §1 · p. 11

2. Mechanism

The framework describes mechanism as the way a bias trigger affects system behavior.

2. Mechanism

  • A mechanism explains how a trigger changes system behavior through allocation, exclusion, scoring, ranking, role assignment, representation, or interpretation.Source: §2 · p. 11

3. Impact

The paper defines impact as the broader implication of biased behavior and evaluates explanations by comparing their reasoning components with human annotations. It describes the annotation and model-selection processes and refines prompts using observed errors.

3. Impact

  • Impact reasoning specifies the affected group, the form of differential treatment, and how representation or downstream decisions may be influenced.Source: §3 · p. 11

3.4 Automated Justification Evaluation Framework

  • Justification evaluation compares Trigger, Mechanism, and Impact components separately, with open-ended mechanism text and standardized taxonomy-specific impact templates.The component-wise comparison uses sequence-based syntactic similarity to assess alignment with manually annotated components.Source: §3.4 · p. 11

3.5 Annotation Process

  • Two authors iteratively calibrated annotations through five rounds, reaching consensus on about 98% of annotations; this agreement is not a formal inter-rater reliability statistic.Source: §3.5 · p. 12

3.6 LLM Model Selection

  • The model set includes proprietary Gemini 2.5-pro and three open-source models, selected for coverage across architectures, scales, optimization paradigms, and deployment environments.Models used their default temperature settings.Source: §3.6 · p. 12

3.7 ICL Prompt Refinement

  • Five ICL prompt configurations were refined on Gemini using error patterns, then validated before full-dataset inference with four LLMs.Refinement continued while agreement or performance criteria were unmet; a random sample of 50 snippets was manually reviewed after full-dataset labelling.Source: §3.7 · p. 13

4 Results

The results characterize generated-code bias as diverse and uneven, and evaluate ICL for identifying bias and aligning explanations with expert reasoning.

4.1 RQ1: Bias Types Found in the Dataset

  • Naming/documentation bias dominated (100% of the manual subset; 98.89% of the full dataset), while stereotype and demographic biases were common and proxy and representational biases less frequent.Over 20% of snippets originally labeled unbiased contained subtle bias; naming-related prevalence may also reflect greater observability in code text.Source: §4.1 · p. 14

4.2 RQ2: LLMs for Bias Type Identification

  • Gemini was used to select an ICL configuration, then ICL5 was applied to open-source models for controlled comparisons across model families.The authors note that each architecture may have a different optimal configuration.Source: §4.2 · p. 14

4.2.1 Bias Type Identification.

  • Gemini's type classification was stable across prompt configurations, with less than 0.15% average inter-trial variation; proxy performance was weakest with only five labeled instances.Naming/documentation achieved optimal performance, while stereotype, threshold/cutoff, and demographic classification also performed strongly.Source: §4.2.1 · p. 16

4.2.2 Bias Code Identification.

  • Code-level identification was more challenging than type classification; prompt structure mattered more, and open-source models struggled with precision and structural consistency.Gemini's average inter-trial performance decrease was less than 1%.Source: §4.2.2 · p. 16

4.3 RQ3: LLMs for Bias Code Justification

  • Gemini's recall was 95.69%–98.81% across ICL configurations, while precision ranged from 66.32% to 85.69% and justification similarity remained approximately 80%–81%.Mechanism explanations varied most; trigger localization could omit context or include irrelevant code, while impact explanations were most consistent.Source: §4.3 · p. 17

4.4 Key Findings

  • Across questions, ICL supported post-hoc detection without fine-tuning, while generated justifications showed moderate-to-high, multidimensional alignment with expert reasoning.Alignment differences involved granularity, scope, and framing.Source: §4.4 · p. 17

5 Discussion

The discussion presents ICL as a feasible post-hoc approach to identifying bias in generated code, while emphasizing limitations in subtle-bias detection, explanation alignment, and generalizability. It frames auditing as human-centered decision support and recommends extending evaluation beyond code correctness.

5.1 Feasibility of Post-Hoc Bias Detection in AI-Generated Code

  • Across all tested Gemini configurations (ICL1-ICL5), bias-type identification accuracy remained near 90%, and recall exceeded 95% in nearly all settings, suggesting post-hoc detection is feasible without fine-tuning or model-internal access.The framework presents LLMs as scalable, post-generation auditing mechanisms.Source: §5.1 · pp. 17–18
  • Higher recall than precision means audits may rarely miss known biases but can over-attribute bias to ambiguous or context-dependent code.The authors recommend interpreting the system as decision support, with human validation retained for fairness judgments.Source: §5.1 · p. 18, §5.3 · p. 20
  • Manual review found subtle naming, documentation, and exclusionary-assumption biases among snippets labeled unbiased, suggesting existing datasets may miss implicit bias.The manually labeled subset contained 784 snippets; the authors argue for finer taxonomies and contextual reasoning.Source: §5.1 · pp. 18–19
  • Proxy bias was hardest to detect: precision was 2.6%, recall 20%, and F1-score 4.7%, based on only five labeled examples.The authors attribute the challenge to needing contextual knowledge about whether a neutral attribute proxies for a protected characteristic.Source: §5.1 · p. 18

5.2 Increasing ICL Complexity Improves Precision More Than Recall

  • Bias in generated code is predominantly semantic, with naming, stereotype, and demographic biases more frequent than proxy or structural biases.The authors describe bias as appearing in developer-facing semantics such as identifiers and documentation, rather than primarily in complex program logic.Source: §5.2 · p. 19
  • Gemini ICL5 reached 85.69% precision and 90.97% code recognition accuracy, while recall remained relatively stable across prompting configurations.The authors associate richer examples and structured prompts with better discrimination between biased logic and benign implementation details.Source: §5.2 · p. 19

5.3 Explanation Alignment Suggests Interpretative Reliability

  • Explanation similarity remained near 80% across Gemini configurations, but differences in scope, granularity, and framing complicate interpreting alignment scores.Some explanations with similarity differences above 20% remained semantically consistent with the intended bias category while using more generalized language.Source: §5.3 · pp. 19–20

5.4 Implications for Responsible AI and Software Engineering

  • The discussion recommends integrating post-hoc audits into review workflows while treating explanations as auxiliary and expanding evaluation beyond code correctness.It also calls for broader benchmarks and domains, human oversight in high-impact settings, and evaluation that distinguishes reasoning disagreement from differences in explanatory style.Source: §5.4 · pp. 20–21, §5.5.1 · p. 21, §5.5.2 · p. 22

6 Threats to Validity

The authors identify threats involving subjective annotation, possible training-data contamination, limited assessment of downstream harms, and restricted external validity.

Construct Validity

  • Bias judgments are context-dependent and socially situated, so expert annotations and justifications involve subjective interpretation.Structural templates and inter-rater calibration were used, but shared template wording may increase justification similarity scores.Source: §6 · p. 22

Internal Validity

  • Closed-source models or updated open-weight models may have encountered the baseline corpus during pretraining, making high performance partly reflect memorization.The evaluation builds on an existing corpus.Source: §6 · p. 22

Conclusion Validity

  • The evaluation focused on textual and logical reasoning, not downstream behavior; identifying biased logic may not predict real-world harms after deployment.Reported metrics were averaged over three independent executions.Source: §6 · p. 22

External Validity

  • The study evaluates isolated Python functions, so findings may not generalize to other languages, larger repositories, complex systems, or unrepresented bias types.The selected ICL configuration may also not be optimal for every model architecture.Source: §6 · p. 22

7 Conclusion

The conclusion characterizes a wider range of biases in AI-generated code and evaluates ICL for post-hoc detection and explanation. It finds detection feasible and explanation alignment substantial, while noting variation in explanatory scope and granularity.

7 Conclusion

  • AI-generated code contains proxy, representational, and documentation or naming biases alongside demographic and stereotype bias.The paper presents this as a broader characterization than prior work primarily explored.Source: §7 · p. 23
  • ICL enables post-hoc bias detection without fine-tuning or access to proprietary model internals, with strong identification and code-recognition performance across prompting strategies.The conclusion also reports moderate-to-high alignment between generated explanations and expert-authored justifications, alongside variation in scope, detail, and granularity.Source: §7 · p. 23
  • The study contributes a characterization of code bias, an empirical evaluation of post-hoc auditing, and an analysis of explanation reliability.It shifts attention toward how bias manifests, can be identified, and can be explained in AI-assisted software workflows.Source: §7 · p. 23

Related papers

Loading 2609.30642v1…