PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Receives Respect? A Counterfactual Benchmark for Honorific Allocation Bias in Bengali

Ajwad Abrar, Md Kamrul Hasan, Hasan Mahmud

arXiv:2610.14769v1cs.CL

TL;DR

English–Bengali translation requires models to choose a respectful or ordinary register that the English source does not specify. The paper evaluates that allocation across gender and occupation, finding much greater occupation-associated variation and substantial differences in overall HBS across models.

  • Problem

    English does not specify the level of deference that Bengali translation requires, leaving the model to choose between ordinary and respectful registers.

    Source: §1 · p. 2

  • Method

    The paper evaluates nine models using a 2,016-sentence set spanning 72 occupations and four pronoun conditions, and reports HBS as a combined 0–100 measure of gender- and occupation-related effects.

    Source: §1 · pp. 2–3, §3.4 · p. 6

  • Results

    Occupation was associated with 3.2%–51.5% of register-choice variation versus 0.00%–1.18% for gender, while HBS ranged from 19.9 to 70.1.

    Source: §5 · p. 9

  • Takeaways & Limitations

    HBS is a descriptive summary rather than a significance test and should be interpreted alongside HAI, GSR, occupation spread, and η2.

    Source: §3.4 · p. 6

    The findings are limited to controlled template-based translation and do not establish how occupational patterns relate to real-world social or gender distributions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Bengali has no grammatical gender, but honorificity is obligatory: every third-person reference must choose ordinary or respectful register. English has no such signal, so the system must choose a level of deference that the source never specified. We use this property to measure how large language models (LLMs) allocate respect in English--Bengali translation. Our evaluation set contains 2{,}016 sentences across 72 occupations. Each appears in seven sentence frames under four pronoun conditions. Masculine and feminine versions differ by one word, forming counterfactual pairs. We report a single Honorific Bias Score (HBS) for each model on a 0--100 scale, covering both the gender and occupation axes. A lower score means that respect is allocated more evenly. For the occupation analysis, we compute each occupation's honorific rate across the seven no-pronoun sentence frames. The standard deviation of these 72 occupation-level rates ranges from 17.2 to 35.5 percentage points across models, showing substantial variation in how honorifics are allocated across occupations. Occupation is associated with 3.2\% to 51.5\% of the variance in register choice, compared with at most 1.2\% for gender. Gender effects are much smaller overall: only two of the nine models show significant directional gender gaps, and they favor opposite genders. HBS ranges from 19.9 to 70.1 across the nine models, showing substantial variation in how evenly the models allocate respect.

1 Introduction

Bengali translation requires a register choice that English leaves unspecified, while the language’s lack of grammatical gender lets honorific allocation be examined separately. The paper introduces an evaluation set and metrics for assessing how gender and occupation relate to that choice.

  • Bengali requires a choice between ordinary and respectful registers, although English sources may specify no level of deference.Source: §1 · pp. 1–2
  • Bengali’s lack of grammatical gender makes it possible to measure honorific allocation without that channel being confounded by grammatical gender.The paper contrasts this with Korean and Japanese, where honorificity and gender-related cues are closely connected.Source: §1 · p. 2
  • The evaluation set contains 2,016 English sentences across 72 occupations, with each occupation appearing in seven sentence frames and four pronoun conditions.Source: §1 · p. 2
  • The paper introduces HAI for register-shift direction, GSR for register-change frequency, and HBS for combining gender- and occupation-related effects on a 0–100 scale.Source: §1 · pp. 2–3
  • Across nine models, occupation was associated with substantially greater variation in honorific allocation than gender.Source: §1 · p. 3

2 Related Work

Prior machine-translation bias benchmarks examine gender resolution and occupational stereotypes, while Bengali bias research has studied several other phenomena. The cited studies do not examine honorificity allocation.

  • WinoMT and Winogender test coreference under occupational stereotypes, while TGBI measures gender-neutral Korean pronoun resolution in English and was adapted for Nepali.Source: §2 · p. 3
  • Other work considers religious bias and its interactions with demographic factors such as gender, but few studies address honorific-register allocation.Source: §2 · p. 3
  • Bengali bias studies cover gendered emotion attribution, embedding-based measures, anti-women text, and question answering, but none examines honorificity.Source: §2 · p. 3

3 Method

The method pairs counterfactual English–Bengali prompts across occupations and pronoun conditions, then classifies Bengali register choices and quantifies gender and occupation effects. It evaluates nine models using paired metrics, a combined score, and statistical tests.

3.1 Dataset

  • The 2,016-sentence set covers 72 occupations in seven frames and four pronoun conditions, including 504 matched masculine–feminine pairs and a no-pronoun condition.Masculine and feminine sentences differ by one word; the no-pronoun versions keep the occupation as subject.Source: §3.1 · pp. 3–4

3.2 Models

  • Nine models from six developers were tested with the same zero-shot prompt at temperature 0, with each sentence submitted independently.Models were accessed through Vertex AI in August 2026.Source: §3.2 · pp. 4–5

3.3 Register classification

  • Register classification uses respectful pronouns and clause-final verb endings; mixed-register cases are excluded, and the rule agreed with human labels on 178 of 180 translations (κ = 0.98).Two native Bengali speakers independently labelled the validation sample before resolving disagreements through discussion.Source: §3.3 · pp. 5–6

3.4 Metrics

  • HAI measures the direction of gender differences, while GSR measures how often pronoun changes alter register, capturing sensitivity that can cancel out in HAI.The matched-pair metrics use counts of changes favoring men or women and the number of usable pairs.Source: §3.4 · p. 6
  • HBS averages gender and occupation components on a 0–100 scale, with higher scores indicating less even allocation; it is not a significance test.The gender component can be unstable when few matched pairs change register, so HBS should be read with HAI, GSR, and significance tests.Source: §3.4 · p. 6

3.5 Statistical analysis

  • The analysis compares occupation and pronoun associations with register variation using η2, and tests directional gender gaps with Holm-corrected McNemar tests.A sensitivity analysis estimates detectable directional gaps, with nominal thresholds treated as lower bounds because tests are corrected across nine models.Source: §3.5 · p. 7

4 Results

HBS varies substantially across models, and occupation is associated with much more variation in register choice than gender. Gender effects are usually not significant, while the two significant directional effects favor opposite genders.

Occupation effects

  • Occupation is associated with 3.2%–51.5% of register-choice variation, compared with 0.00%–1.18% for gender; occupation-rate standard deviations range from 17.2 to 35.5 percentage points.Occupation-level honorific rates are computed across seven no-pronoun frames.Source: §4 · p. 8
  • The nine models rank occupations similarly, with mean pairwise Spearman correlation 0.77 (range 0.55–0.91), and leave-one-out checks find no dependence on a single occupation or frame.Source: §4 · p. 8

Gender effects

  • Seven of nine models show no significant directional gender difference after Holm correction; Llama 3.3 70B favors men by 7.77 points and Llama 4 Maverick favors women by 7.19 points.Both directional effects remain significant after Holm correction.Source: §4 · p. 8
  • Kimi K2 Thinking and DeepSeek-V3.1 change register relatively often, yet their HAI values remain near zero because changes do not consistently favor one gender.Source: §4 · p. 8

Overall Honorific Bias Score

  • HBS ranges from 19.9 for gpt-oss-120b to 70.1 for Llama 4 Maverick, and the two models with significant directional gender effects have the highest scores.HBS is descriptive rather than a significance test and is interpreted alongside other measures.Source: §4 · p. 8

Singular they

  • Every model uses respectful register less often for singular they than for the feminine condition, while most pluralize they in 45.8%–98.8% of outputs.Among outputs preserving singular reference, honorific rates rise in seven models and by more than 25 points in four; Llama 3.3 70B and Llama 4 Maverick are exceptions.Source: §4 · p. 8

5 Conclusion

The study evaluates honorific allocation in Bengali translation using 2,016 sentences across 72 occupations. It finds occupation-associated variation substantially exceeds gender-associated variation, while results remain limited to controlled templates.

5 Conclusion

  • Occupation is associated with 3.2%–51.5% of register-choice variation, compared with 0.00%–1.18% for gender, across nine evaluated models.Source: §5 · p. 9
  • HBS ranges from 19.9 to 70.1, and only two models show significant directional gender gaps, favoring opposite genders.The models rank occupations in broadly similar ways.Source: §5 · p. 9
  • The findings are limited to controlled template-based translation and do not establish how occupational patterns relate to real-world social or gender distributions.Source: §5 · p. 9

Dataset Availability

The evaluation set and occupation list with Bengali terms are publicly available.

Dataset Availability

Loading 2610.14769v1…