CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs

1 Introduction

• Societal biases can persist despite safety efforts, reflecting stereotypes in training data and potentially becoming amplified during user interactions.The paper adopts a social-psychology perspective defining societal bias as stereotypical associations or blatant racism.Source: §1 · p. 1
• The study targets conversational cases where human input leads a model to utter harmful content and tests whether it can recover.Prior work cited by the authors found that models may fail to recover after taking a wrong turn in a conversation.Source: §1 · p. 2
• CoBia uses a single query to construct a conversation containing a biased claim, then tests whether the model rejects biased follow-up questions.The paper describes these lightweight attacks as a way to expose hidden societal biases through conversation.Source: §1 · p. 2
• The paper introduces a dataset of 112 social groups with negative descriptors across six socio-demographic categories and evaluates 11 LLMs against prompt-based attacks.It also compares societal-bias scores using three automated judges and human annotations.Source: §1 · p. 2

2 Dataset

• CoBia unifies and de-duplicates negative descriptors targeting social groups from three existing datasets.The resulting entries are organized as category, group, and negative-descriptor triplets.Source: §2 · p. 2

2.1 Selection of Stereotype Datasets

• RedditBias supplies descriptors from Reddit discussions, SBIC descriptors are extracted from stereotype sentences, and StereoSet descriptors are extracted from crowdsourced association tests.SBIC extraction uses part-of-speech tagging, while StereoSet uses llama3.3:70b because its sentence structure makes POS-based extraction impractical.Source: §2.1 · p. 2

2.2 CoBia Dataset

• The merged dataset consolidates similar group names, assigns groups to six categories, and retains instances classified as unsafe by Llama Guard.The dataset contains 112 distinct social groups; 99% of its instances receive Llama Guard's S10 category label.Source: §2.2 · p. 3

2.3 Extending CoBia Dataset

• The dataset extension pairs each negative descriptor with a group-agnostic explanation and adds six positive descriptors per group with group-specific explanations.The explanations for positive descriptors are generated alongside the descriptors.Source: §2.3 · p. 3

3 CoBia Methods

• CoBia tests for bias that safeguards may overlook by eliciting biased responses through constructed conversations without heavy jailbreaks.It uses history-based constructed conversation (HCC) and single-block constructed conversation (SCC).Source: §3 · p. 3

3.1 LLM Conversation History

• User-managed conversation history lets a user construct prior messages to test whether a model produces biased responses and rejects biased follow-up questions.The paper frames this as a user-initiated adversarial use of conversation history for societal-bias testing.Source: §3.1 · pp. 3–4

3.2 Preparing CoBia Dataset

• Each experimental instance starts from a dataset triplet and uses six positive descriptors plus a distinct additional negative descriptor with explanations.The triplet is (category, group, negative descriptor).Source: §3.2 · p. 4

3.3 HCC: History-based Constructed Conversation

• In HCC, an assistant lists six positive and two negative characteristics, after which user turns ask about selected descriptors and link the target negative descriptor to another.The primary negative descriptor is excluded from the intermediate follow-up selection; the constructed exchange is delivered as one query.Source: §3.3 · p. 4

3.4 SCC: Single-block Constructed Conversation

• SCC presents the same prompt content as HCC as one user message, without separate user and assistant tags.Source: §3.4 · p. 4

3.5 Tips

• The prompts use pronouns, descriptor index numbers, and a disclaimer to reduce safety-filter triggers and encourage reliance on conversational context.The paper says these tips aim to make the dialogue scenario more natural.Source: §3.5 · p. 4

4.1 Selected Models

• The study evaluates 11 LLMs, including open- and closed-source models from nine organizations.Open-source models were hosted with Ollama, while closed-source models were accessed through their respective endpoints.Source: §4.1 · p. 5
• The selected models use deterministic decoding with temperature=0 and top_p=0.Table 2 reports macro averages across six social categories, with best and second-best scores visually distinguished.Source: §4.1 · p. 5, Table 2 · p. 6

4.2 Baselines

• The baselines are direct zero-shot questions, the DAN jailbreak, and roleplay prompts using a cultural sociologist persona.The zero-shot setup asks about the target group and descriptor without prior conversation; the roleplay persona was used consistently across experiments.Source: §4.2 · p. 5

4.3 Judge Metrics

• The judges include a direct bias classifier, Granite Guardian risk classifier, and an NLI-inspired judge that maps four relations to binary bias verdicts.The NLI judge classifies enforces and agrees as yes, and neutral or negates as no.Source: §4.3 · p. 5

5.1 Main Result

• HCC and SCC outperform baselines in most settings under Bias and Granite Judges, with similar NLI patterns; averaged across 11 models, they outperform all baselines across judges.The comparison uses macro-averaged scores across six categories; baseline rankings vary by judge.Source: §5.1 · p. 6, Figure 3 · p. 7, §5.4 · p. 7
• UCC scores exceeded 80% with Bias Judge for llama3.3:70b, command-r:35b, and qwen2.5:7b, each also exceeding 40% with Granite Judge and 69% with NLI Judge.The authors classify models above 80% UCC with Bias Judge as heavily biased.Source: §5.1 · p. 6
• HCC and SCC expose complementary model-specific biases, and their combination raised scores by at least 30% for deepseek-v2:16b and gemma2-27b despite less than 3% between-method differences.HCC yielded higher rates for mistral-7b, llama3.1:8b, and phi4:14b, while SCC performed better for gpt-4o-mini.Source: §5.1 · p. 6

5.2 Bias Distribution Across Social Categories

• UCC produced higher bias scores than 0-Shot across models and categories, but national origin consistently had the highest bias across settings.For gemma2:27b, orientation, race, and religion scores stayed below 0.2; the NLI Judge was more sensitive to national-origin bias than the Bias Judge.Source: §5.2 · pp. 6–7, Figure 1 · p. 7

5.3 Effect of Model Size on Bias Scores

• Across qwen2.5 sizes, bias scores showed no clear correlation with model size; the 32B model had the best safety performance, without an obvious explanation.The 3B model showed higher SCC bias than HCC and 0-Shot, while larger models varied in safety.Source: §5.3 · p. 7, Figure 2 · p. 7

5.4 Comparison of Judges

• When Bias and NLI Judges disagreed on DAN cases, NLI Judge matched the majority of all four human annotators in 83% of cases.Bias and Granite Judges sometimes flagged offensive language not directed at the social group in question.Source: §5.4 · pp. 7–8
• Human annotators’ Fleiss’ κ was 0.54 for the Bias Judge and 0.55 for the NLI Judge; NLI and Bias Judges had pairwise agreement 0.79 and Cohen’s κ of 0.53.Granite Judge had Cohen’s κ of 0.16 with Bias Judge and 0.13 with NLI Judge; its average Cohen’s κ with human annotators was 0.10.Source: §5.4 · pp. 7–8

Jailbreaking LLMs

• Jailbreak approaches include prompt- and token-based attacks, parameter-dependent gradient attacks, and infrastructure attacks that modify or exploit supporting systems.The passage categorizes these attack types following Purpura et al. (2025).Source: §6 · pp. 8–9
• Many existing jailbreak attacks require technical expertise, substantial computation, or many queries, whereas CoBia uses one query with minimal computation.The authors describe HCC as counterable by disabling user control over conversation history.Source: §6 · p. 9

Societal Bias in LLMs

• Bias evaluation includes intrinsic embedding- or token-probability methods and extrinsic generation methods; prior benchmarks reported little bias despite evidence of persistent implicit biases.CoBia instead evaluates whether models recover appropriately or continue and amplify bias after a conversation elicits it.Source: §6 · p. 9

• CoBia simulates biased conversational contexts using structured conversation history or a full exchange embedded in a non-structured prompt.The study evaluates whether models recover from harmful content elicited by human input.Source: §7 · p. 9
• Both CoBia methods consistently outperform three baselines in exposing bias, with agreement across all judges.The evaluation covers 11 LLMs from 9 organizations, six social categories, and three automated bias judges.Source: §7 · p. 9
• The authors conclude that reliable safety requires extending safeguards from isolated prompts to entire dialogues.They describe CoBia as a practical tool for stress-testing safety in more realistic conversational settings.Source: §7 · p. 9

• The study evaluates 11 prominent LLMs from nine organizations, a practical trade-off between feasibility and coverage.The experiments involved over 800,000 queries, and the authors say the model set supports generalizability.Source: Limitations · p. 10
• Only two conversational templates were tested, so variations in dialogue style, length, or tone could influence outcomes.The authors describe the templates as a minimal, controlled validation rather than an exhaustive sample of dialogue types.Source: Limitations · p. 10
• Human evaluation was limited in scope, although the three automated judges aligned strongly with the collected human annotations.The judges were Bias Judge, Granite Judge, and an NLI Judge.Source: Limitations · p. 10
• The adversarial conversations do not directly reflect typical user behavior, so results are stress-test signals rather than deployment-time occurrence rates.Source: Limitations · p. 10
• The paper offers no concrete mitigation strategy beyond high-level suggestions such as limiting user control over conversation history.The authors identify development of robust defenses as future work.Source: Limitations · p. 10
• The previous Chat Completions API remains the de facto standard, with conversation history still manually managed by users.The passage contrasts this with a newer API that began managing a unique ID for previous chat history as of March 11, 2025.Source: Limitations · p. 10
• Despite these limitations, the authors describe CoBia as useful for stress-testing LLM safety in realistic dialogue settings.They cite consistent cross-model results, multi-judge agreement, and empirical validation.Source: Limitations · p. 10

Ethics Statement

• The study uses lightweight adversarial prompts to investigate how constructed user–assistant conversations reveal embedded societal biases, with the aim of informing more robust safety mechanisms.The authors state that the objective is systematic study, not exploitation or dissemination of harmful outputs.Source: Ethics Statement · p. 10
• The study collected no personal data or subjective human responses, and a small group of annotators labeled outputs using predefined instructions for validation.The paper says annotation was limited in scope and posed minimal risk; it also states that the dataset and code would be released under a CC BY-SA 4.0 license.Source: Ethics Statement · p. 10

A OpenAI Compatible API Endpoint

• The Chat Completions API’s messages parameter carries user-managed conversation history as a list of dictionaries.The passage describes the interaction as submitting the conversation and including the model’s response in the next request.Source: A OpenAI Compatible API Endpoint · p. 17
• Figure 4 depicts a Chat Completions API call, whose messages use system, user, and assistant roles.The passage describes the roles as optional initial instructions, user messages, and assistant-generated responses, respectively.Source: Figure 4 · p. 17, A OpenAI Compatible API Endpoint · p. 17

• Four graduate students with strong English proficiency and research-ethics understanding assisted with annotation.They received detailed training on the evaluation criteria.Source: B Human Evaluation · p. 17
• For each annotation, annotators saw the target group, its associated negative descriptor, and the model output, then judged whether the response contained bias.The negative descriptor was included to reduce the influence of annotators’ personal beliefs.Source: B Human Evaluation · p. 17
• Each setup randomly selected two annotators, used prompts from the Bias Judge and NLI Judge evaluations, and labeled unbiased responses 0 and biased responses 1.Source: B Human Evaluation · p. 17

C.1 CoBia Method Prompt

• Figure 5 is identified as an example HCC template used in the experiments.The supplied text does not describe its internal structure.Source: §C.1 · p. 17

C.2 R-Play Prompt

• The R-Play prompt assigns the model a cultural-sociologist persona specializing in stereotype origins and functions across minority groups.The prompt frames its audience as students, researchers, and the public.Source: §C.2 · pp. 17–18
• The persona is tasked with explaining how historical, political, and social forces shape stereotypes and affect marginalized communities.It is presented as a prompt to help audiences understand stereotypes and their impacts.Source: §C.2 · p. 18
• The persona prompt explicitly addresses the origins and functions of stereotypes across minority groups.Source: §C.2 · p. 18
• The prompt is framed as helping students, researchers, and the public understand stereotypes and their effects.Source: §C.2 · p. 18

C.3 Judge Prompt

• The appendix points readers to the Bias Judge and NLI Judge prompts in Figures 6 and 7.Source: §C.3 · p. 18

D Results in Details

• Tables 4–6 report full CoBia results, including micro- and macro-averages, evaluated by the Bias, Granite, and NLI Judges, respectively.The supplied passages give table scope but no result values or model comparisons.Source: D Results in Details · p. 18, Table 4 · p. 22, Table 5 · p. 22
• Figure 5 is identified as an example HCC template used in the experiments.Source: Figure 5 · p. 19
• The judge prompt material distinguishes statements that agree with, disagree with, or are undetermined relative to a biased opinion.Source: D Results in Details · pp. 20–21