Effects of Model Scale and Quantization on Social Bias and Abstention in Question Answering

1 Introduction

• In BBQ, “Unknown” can be appropriate for ambiguous contexts but inappropriate when context supplies the answer, so fewer stereotypical responses may reflect abstention rather than fairer behavior.The introduction argues that this distinction matters for interpreting bias scores.Source: §1 · p. 1
• Prompt wording can change responses and accuracy, making fairness trends from a single prompt potentially prompt-specific.The introduction calls for checking trends across semantically equivalent prompts.Source: §1 · p. 2
• The study examines model sizes in Qwen3, Gemma-3, and SmolLM2, analyzing accuracy, social bias, and unknown/abstention behavior.It also considers low-precision quantization.Source: §1 · p. 2
• The study evaluates five prompt formulations and adds a forced-choice condition that removes the “Unknown” option to examine whether observed bias patterns persist.It asks whether lower biased-response rates reflect substantive decisions or increased abstention.Source: §1 · p. 2

1.1 Main Contributions

• The contributions are characterizing scale and quantization effects across three model families, checking prompt robustness, and using forced choice to examine abstention’s role in observed bias.Source: §1.1 · p. 2

2.1 Social Bias and Fairness in QA tasks

• BBQ evaluates bias across demographic categories in ambiguous and disambiguated QA contexts, where prior work found stereotype reliance under ambiguity and sometimes with sufficient evidence [Parrish et al., 2022b].Earlier QA work also reported strong stereotyping across several attributes [Li et al., 2020].Source: §2.1 · p. 2

2.2 Fairness in Model Size and Quantization

• Findings on model scale are mixed: some studies found little fairness variation, while others associated larger models with stronger stereotypes [Baldini et al., 2022; Tal et al., 2022; Gonçalves and Strubell, 2023].Source: §2.2 · p. 3
• Quantization studies report bias reductions under some conditions, increased stereotyping under aggressive quantization, and prediction shifts despite stable aggregate scores [Gonçalves and Strubell, 2023; Marcuzzi et al., 2025; Hua et al., 2026].Reported effects depend on architecture, precision, and evaluation setting.Source: §2.2 · p. 3
• Prior work also links quantization with changes in “Unknown” responses, but has paid comparatively limited attention to its interaction with scale, prompt robustness, and explicitly removing abstention.The study examines these factors jointly.Source: §2.2 · p. 3

3.1 Benchmark and Task Setting

• BBQ contains 58,492 QA examples across 11 social-bias categories and uses ambiguous and disambiguated contexts to assess stereotype reliance when information is insufficient.The evaluation varies prompt phrasing, model architecture, and quantization settings.Source: §3.1 · p. 3
• Five prompts preserve the same QA task and answer space while varying wording and instructional detail; “Unknown” is appropriate in ambiguous contexts and over-abstention in disambiguated ones.Two-choice questions use prompts 1–2, while three-choice questions use prompts 1–5.Source: §3.1 · p. 3, Figure 1 · p. 4
• The evaluated models span Qwen3, Gemma-3, and SmolLM2 at multiple parameter scales, with FP16 checkpoints as Dense baselines and weight-only or dynamic quantization configurations.Quantization settings include FP8, INT8, and INT4 for weight-only quantization, and FP8 and INT8 for dynamic quantization.Source: §3.1 · pp. 3–4

3.2 Evaluation Measures

• The four primary measures are QA accuracy, biased-response rate, appropriate-abstention rate, and over-abstention rate.Response rates are proportions of evaluated instances assigned to each behavioral category.Source: §3.2 · p. 4
• The main analysis aggregates results across 11 BBQ categories for each prompt, then averages across five prompts; prompt- and category-level results are retained for stability checks.Quantized-versus-Dense changes are reported in percentage points, with positive values indicating increases and negative values decreases.Source: §3.2 · p. 4

4.1 Effect of Model Scale and Quantization on Accuracy and Response Behavior

• Accuracy increased with model size, while biased-response changes varied in direction and magnitude across model families, sizes, and quantization configurations.INT4 reduced accuracy in every evaluated configuration, while FP8 generally produced smaller changes.Source: §4.1 · pp. 4–5, Table 1 · p. 5, Figure 2 · p. 5

4.2 Bias–Abstention Relationship

• 20 of 26 configurations with fewer biased responses also showed increased over-abstention, with the bias–abstention relationship varying across families and quantization methods.For SmolLM2 1.7B under INT8 Dynamic, biased responses fell 15.07 pp while over-abstention rose 15.95 pp and appropriate abstention rose 11.31 pp.Source: §4.2 · pp. 5–6, Figure 3 · p. 5, Figure 4 · p. 6

4.3 Robustness Across Prompt Formulations

• The SmolLM2 1.7B INT8 Dynamic pattern—fewer biased responses and greater over-abstention—held across all five prompts, while many configurations varied by prompt.Qwen3 1.7B under INT4 showed increased biased responses and decreased over-abstention across all five prompts.Source: §4.3 · pp. 6–7, Table 2 · p. 6

4.4 Two-Choice Format Question

• Removing “Unknown” produced category-dependent response patterns, with composite attributes showing consistently high response rates across the two prompt formulations.Prompt 1 and Prompt 2 category results correlated 0.956 under the Negative condition and 0.889 under the Non-negative condition.Source: §4.4 · p. 7, Figure 7 · p. 8

• Some configurations showed substantial changes in biased responses and abstention despite relatively small accuracy changes.Source: §5 · p. 8
• Many configurations changed direction across prompts, while SmolLM2 1.7B under INT8 Dynamic and Qwen3 1.7B under INT4 retained the same directional pattern across all five.Source: §5 · p. 8
• Removing “Unknown” redistributed those responses into biased or anti-biased choices, showing that answer-option availability can affect observed bias.Source: §5 · p. 8
• Fairness evaluation should consider accuracy, biased responses, abstention, and prompt formulation jointly.Source: §5 · p. 8

• Accuracy increased with model size, whereas bias showed no simple scaling trend and quantization effects varied across models.Source: §6 · p. 8
• Lower biased-response rates were often accompanied by increased abstention.Source: §6 · p. 8
• Under forced choice, category differences were more pronounced than model differences and remained largely stable across prompts and after INT4 quantization.Source: §6 · p. 8
• Fairness evaluation should consider abstention, prompt formulation, and answer-option design.Source: §6 · p. 8

• The evaluation uses only BBQ and relatively lightweight models from three families, which may limit generalizability to other bias benchmarks.Source: Limitations · p. 9

• BBQ’s sensitive-attribute and stereotype examples are used to measure model behavior, not to endorse or validate the stereotypes.Source: Ethical Considerations · p. 9
• The analysis uses an existing public benchmark and involves neither human participants nor personal-data collection.Source: Ethical Considerations · p. 9

A.1 Computing Resources

• The study's execution conditions are reported in Table 3, titled “Execution Environment.”Source: §A.1 · p. 10, Table 3 · p. 10

A.2 Selected Models

• Qwen3, Gemma-3, and SmolLM2 were selected for comparisons of scale and quantization, with free checkpoints and relevance to prior lightweight-model research.Source: §A.2 · p. 10

B Additional Results of Forced-Questioning Analysis

• Table 4 reports category-level response-pattern stability across prompt formulations and INT4 quantization using Pearson correlation and median absolute difference.Source: Table 4 · p. 11