PaperlayerView exact revision on arXiv

Source-linked AI summary

BiasAsker: Measuring the Bias in Conversational AI System

Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, Michael Lyu

arXiv:2305.12434v1cs.CLcs.AI

TL;DR

Social bias in conversational AI can harm groups, while its evaluation seeks to measure bias and reveal group-property associations. BiasAsker combines a social-bias dataset with automated question generation and bias measurement. Its generated queries triggered biased behavior at a maximum rate of 32.83% across tested conversational systems.

  • Problem

    The paper focuses on evaluating social bias in conversational AI, whose biased content may discomfort groups and aggravate social conflicts.

    Source: §1 · p. 1

  • Method

    BiasAsker combines social groups and biased properties in a dataset with automated query generation and similarity-based detection to measure absolute and relative bias.

    Source: §1 · p. 2

  • Results

    32.83% maximum and 20% average bias finding rates were reported across eight commercial systems and two research models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors released the dataset, code, and experimental results to facilitate real-world fairness testing and further research.

    Source: §1 · p. 2

    The evaluation covers selected commercial systems and research models, not other conversational AI systems.

    Source: §5 · p. 10

Abstract

from arXiv · show

Powered by advanced Artificial Intelligence (AI) techniques, conversational AI systems, such as ChatGPT and digital assistants like Siri, have been widely deployed in daily life. However, such systems may still produce content containing biases and stereotypes, causing potential social problems. Due to the data-driven, black-box nature of modern AI techniques, comprehensively identifying and measuring biases in conversational systems remains a challenging task. Particularly, it is hard to generate inputs that can comprehensively trigger potential bias due to the lack of data containing both social groups as well as biased properties. In addition, modern conversational systems can produce diverse responses (e.g., chatting and explanation), which makes existing bias detection methods simply based on the sentiment and the toxicity hardly being adopted. In this paper, we propose BiasAsker, an automated framework to identify and measure social bias in conversational AI systems. To obtain social groups and biased properties, we construct a comprehensive social bias dataset, containing a total of 841 groups and 8,110 biased properties. Given the dataset, BiasAsker automatically generates questions and adopts a novel method based on existence measurement to identify two types of biases (i.e., absolute bias and related bias) in conversational systems. Extensive experiments on 8 commercial systems and 2 famous research models, such as ChatGPT and GPT-3, show that 32.83% of the questions generated by BiasAsker can trigger biased behaviors in these widely deployed conversational systems. All the code, data, and experimental results have been released to facilitate future research.

1 INTRODUCTION

Conversational AI systems can produce socially harmful bias, yet their black-box behavior, diverse responses, and limited evaluation data make comprehensive assessment difficult. BiasAsker combines social groups with biased properties to generate tests and measure bias; experiments find biased behavior in up to 32.83% of generated queries.

1 INTRODUCTION

  • Conversational systems can produce biased content that discomforts some groups and may worsen social atmosphere or intensify conflicts.The introduction cites Tay's racist comments and BlenderBot's offensive output to females as examples.Source: §1 · p. 1
  • Black-box, data-driven systems are difficult to assess comprehensively, while diverse responses, limited biased data, and manual annotation constrain existing approaches.Sentiment and toxicity checks can miss biased content, and existing studies may omit group-property relationships.Source: §1 · pp. 1–2
  • The paper frames comprehensive evaluation as measuring bias extent and revealing associations between social groups and biased properties.Examples include associations between men or women and properties such as financial status or competence.Source: §1 · pp. 1–2
  • BiasAsker combines 841 social groups and 8,110 properties with automated question generation and similarity-based bias detection.The dataset organizes groups under 11 attributes and properties under 12 categories.Source: §1 · p. 2
  • 32.83% maximum and 20% average bias finding rates were reported across evaluations of eight commercial systems and two research models.The authors report that the generated queries triggered biased behavior in these systems.Source: §1 · p. 2
  • The authors released the dataset, code, and experimental results for reproduction and further research.They also describe the release as facilitating real-world fairness testing tasks.Source: §1 · p. 2

2 BACKGROUND

The background treats conversational AI as black-box software spanning task-oriented and open-domain systems. It frames social bias as consequential and distinguishes direct group comparisons from differences in responses across groups.

2.1 Conversational AI System

  • Conversational AI includes task-oriented systems for specific tasks and open-domain systems for broad conversation, and the paper tests both as black-box software.Examples of task-oriented uses include shopping and reservations; open-domain uses include replying to tweets and entertainment.Source: §2.1 · p. 2

2.2 Social Bias in Conversational AI Systems

  • Social bias is unfair or prejudicial discrimination, and biased outputs may discomfort groups and aggravate social conflicts.The text notes prior findings that dialog models' performance improvements have made little improvement to bias safety.Source: §2.2 · pp. 2–3
  • Absolute bias directly states one group's superiority, whereas relative bias appears when responses differ across groups and may evade direct “wh”-questions.Commercial protection mechanisms and varied responses to semantically similar questions can make bias difficult to expose.Source: §2.2 · p. 3

3 APPROACH AND IMPLEMENTATION

BiasAsker uses a social-bias dataset to generate test questions and then measure and visualize absolute and relative bias in conversational systems.

3 APPROACH AND IMPLEMENTATION

  • BiasAsker's workflow consists of question generation followed by bias detection.The approach section places the framework after its description of social-bias dataset construction.Source: §3 · p. 3
  • The question-generation stage forms absolute- and relative-bias tuples from the dataset and creates Yes-No, Choice, and Wh questions with rules and templates.The questions serve as inputs for bias testing.Source: §3 · p. 3
  • The detection stage collects suspicious responses through affirmation, choice, and explanation measurements, then quantifies and visualizes bias using absolute bias rate and relative bias score.The measurements correspond to the three generated question types.Source: §3 · p. 3

3.1 Social Bias Dataset Construction

The dataset construction combines social groups and biased properties, then cleans, categorizes, and translates its contents for testing.

3.1.1 Collecting Social Groups.

  • The group collection merges social-group resources from StereoSet, SBIC, and HolisticBias, then removes duplicates and manually filters or consolidates entries.The sources cover demographic groups and axes; the final group statistics and examples are listed in Table 1.Source: §3.1.1 · pp. 3–4, Table 1 · p. 4

3.1.2 Collecting Biased Properties.

  • 8,110 biased properties were obtained from SBIC implied statements by removing subjects and filtering nonstandard or nonbiased statements.Subject identification used spaCy noun chunks and dependency analysis, followed by manual filtering.Source: §3.1.2 · p. 4

3.1.3 Annotating Biased Properties.

  • Three annotators iteratively organized properties into a multi-label taxonomy and manually annotated antonyms to help evade chatbot defenses.The taxonomy was refined over three iterations through merging similar labels, removing inadequate categories, and clarifying definitions.Source: §3.1.3 · p. 4

3.1.4 Translation.

  • The dataset was translated into Chinese using Google Translate and DeepL, with manual review when translator outputs had similarity below 0.7.Items with similarity at least 0.7 used DeepL's translation directly.Source: §3.1.4 · p. 4

3.2 Question Generation

BiasAsker builds biased tuples from social groups and properties, then turns them into questions to probe absolute and relative bias.

3.2.1 Constructing Biased Tuples.

  • Absolute-bias tuples pair two groups sharing an attribute with one biased property, while relative-bias tuples pair one group with one property.The framework forms these combinations systematically, enabling comparisons across groups and properties rather than relying only on biases already present in source data.Source: §3.2.1 · p. 5
  • Systematically varying groups and properties lets BiasAsker compare chatbot behavior across both dimensions and examine which properties are associated with which groups.The method is not limited to the original social-media biases in the SBIC dataset.Source: §3.2.1 · p. 5

3.2.2 Generating Questions.

  • BiasAsker generates yes-no, choice, and wh-questions for absolute-bias tuples using comparative forms and predefined templates.For example, a tuple with “are smart” can be converted to “are smarter than” for a comparative question.Source: §3.2.2 · p. 5
  • For relative-bias tuples, BiasAsker generates yes-no and wh-questions through rule-based and template-based methods.Rule-based yes-no generation moves an auxiliary to the front or inserts an appropriate form of “do”; templates also produce tag questions.Source: §3.2.2 · p. 5

3.3 Biased Answer Collection

BiasAsker filters generated responses by checking whether they contain expected answer types, using sentence similarity as an existence measurement.

3.3 Biased Answer Collection

  • BiasAsker uses sentence similarity to check whether a response contains the expected affirmation, choice, or explanation before collecting it for bias measurement.The expected answer type depends on whether the prompt is a yes-no, choice, or wh-question.Source: §3.3 · p. 6
  • For existence measurement, BiasAsker implements multiple sentence-similarity approaches: n-gram matching by exact-match ratio, cosine similarity, n-gram sentence similarity, and cosine similarity with a position penalty.Source: §3.3 · p. 6
  • A yes-no response counts as affirmation when it contains an affirmation expression and no negation expression; a choice response counts when it names one group but not both.For wh-questions, BiasAsker collects responses containing an explanation expression such as “because” or “due to.”Source: §3.3 · p. 6

3.4 Bias Measurement

BiasAsker quantifies directly expressed absolute bias and differences in how the system favors groups. It defines rates for each form to measure bias by category.

3.4 Bias Measurement

  • BiasAsker measures absolute bias through direct affirmations, choices, or explanations, and relative bias through differences in favor across groups.The framework defines an absolute bias rate and a relative bias rate for quantifying these patterns.Source: §3.4 · p. 6, §3.4.1 · pp. 6–7, §3.4.2 · p. 7

3.4.1 Absolute Bias Measurement.

  • Absolute bias rate is the percentage of questions triggering absolute bias among queries sharing a bias-property category or social group.For a group pair and bias category, BiasAsker also counts which group gets an advantage and compares the groups’ advantage values; equal advantage or no selections means no preference.Source: §3.4.1 · p. 7

3.4.2 Relative Bias Measurement.

  • Preference rate is the share of a group’s queries that express favor toward it, while relative bias rate is the variance of preference rates across groups in a category.A higher relative bias rate indicates that the system treats groups more differently.Source: §3.4.2 · p. 7

4 EVALUATION

The evaluation tests BiasAsker across commercial and research systems, assesses the validity of its identifications, and examines the group–bias associations it reveals. It reports substantial absolute and relative bias, 0.93 identification accuracy, and system-specific patterns in the discovered associations.

4.1 Research Questions.

  • The study frames RQ1 around whether BiasAsker can systematically trigger and identify bias and measure its degree across conversational systems.RQ2 checks validity through manual inspection, while RQ3 examines whether results provide an intuitive and constructive impression of bias.Source: §4.1 · p. 7

4.2 Experimental Setup.

  • The evaluation covers eight commercial systems and two research models; four systems without API access are tested manually.API-access systems are tested across seven attributes and twelve bias categories; manual tests use seven attributes and three categories.Source: §4.2 · p. 7, Table 5 · p. 8, Table 6 · p. 8

4.3.1 RQ1 - The overall effectiveness of BiasAsker.

  • GPT-3 and Jovi reach absolute bias rates of 25.03% and 32.82%, respectively, while all tested systems exhibit relative bias.Relative bias is especially severe on race, gender, and ability; DialoGPT has the highest relative bias among systems with API access.Source: §4.3.1 · pp. 7–8, Table 7 · p. 9, Table 8 · p. 9

4.3.2 RQ2 - Validity of identified biases.

  • BiasAsker achieves 0.93 accuracy on manually annotated data, supporting the reliability of its bias identification results.The evaluation samples 3,000 question-response pairs for annotation, then tests identification on 2,600 labeled pairs.Source: §4.3.2 · p. 8

4.3.3 RQ3 - Insight of the discovered bias.

  • DialoGPT favors men and associates gay people negatively with culture, while Jovi favors women and negatively associates transgender people with health, mistreatment, and morality.Jovi also associates men negatively with morality.Source: §4.3.3 · pp. 8–9, Figure 4 · p. 9
  • For positive descriptions of social status, Jovi favors young people over older groups, while DialoGPT most disadvantages middle-aged people.Jovi’s most disadvantaged age group is old people.Source: §4.3.3 · p. 9, Figure 5 · p. 9
  • ChatGPT exhibits almost no absolute bias but retains relative bias, with transgender and older people receiving its highest preference rates on gender and age.The authors note that groups most preferred by ChatGPT tend to receive consistently less preference from other tested systems.Source: Figure 6 · p. 9, Table 7 · p. 9, §4.3.3 · p. 10
  • The evaluation concludes that BiasAsker can visualize latent associations between social groups and bias categories.Its case studies compare systems selected for high relative or absolute gender bias and examine ChatGPT.Source: §4.3.3 · pp. 8, 10

5 THREATS TO VALIDITY

The authors identify three threats to validity: imperfect NLP-based bias identification, limits in the source data, and limits in the evaluated systems. They describe steps taken to mitigate each threat.

5 THREATS TO VALIDITY

  • NLP techniques may produce false positives or miss biased responses, although human annotation measured 0.93 accuracy in detecting bias.The authors also compared widely used similarity methods and selected the one with the best performance.Source: §5 · p. 10
  • Because the test inputs draw on existing social-bias datasets, they may limit comprehensiveness or include biases that are unrealistic and rare in the real world.The authors combined datasets collected from real-world media posts and manually annotated by researchers.Source: §5 · p. 10
  • The evaluation covers selected commercial systems and state-of-the-art academic models, leaving other conversational AI systems unevaluated.The authors propose testing more commercial software and research models in future work.Source: §5 · p. 10

6 RELATED WORK

Related work spans AI software testing and bias testing in conversational AI. The cited approaches address software correctness or narrower bias targets, while BiasAsker focuses on measuring conversational-system bias.

6.1 Testing of AI Software

  • Conversational AI testing includes reference-based benchmarks and newer automatic techniques, but the cited work focuses on software correctness rather than bias.This work is situated within broader efforts to test AI and NLP software robustness.Source: §6.1 · p. 10

6.2 Testing the Bias of Conversational AI Software

  • The authors reviewed conversational-AI bias testing across software engineering, natural language processing, and security.Source: §6.2 · p. 10
  • Prior work targeted particular biases, including gender, race, social class, and profession; BiasAsker measures biases across different groups and properties.Earlier detection approaches used classifiers, moderation APIs, or human annotation; BiasAsker instead uses both questions and generated responses.Source: §6.2 · p. 10

7 CONCLUSION

BiasAsker is presented as an automated framework for comprehensively measuring social bias in conversational AI systems. Experiments on commercial systems and research models show it can trigger a massive amount of biased behavior.

7 CONCLUSION

  • Experiments on eight commercial systems and two research models demonstrate that BiasAsker can effectively trigger a massive amount of biased behavior.Source: §7 · p. 10
  • BiasAsker is presented as the first automated framework for comprehensively measuring social biases in conversational AI systems.Source: §7 · p. 10
  • The framework evaluates both the degree of system bias and the association between social groups and biased properties.Source: §7 · p. 10

Related papers

Loading 2305.12434v1…