PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Language Gap in Scholarly Data II: Enhancing Gender Inference for Chinese Names

Lisette Espín-Noboa, Mingrong She, Ana Maria Jaramillo, Liuhuaying Yang

arXiv:2610.23734v1cs.CLcs.CYcs.LGcs.SIphysics.soc-ph

TL;DR

Chinese names in scholarly records are often written in Pinyin, where multiple Hanzi forms can carry different gender associations. This paper trains Hanzi models and aggregates their predictions across Pinyin-to-Hanzi mappings; the models outperform off-the-shelf tools in the reported comparisons, while the authors identify limits tied to their annotation population and changing tool outputs.

  • Problem

    Chinese names are poorly supported by existing gender-inference tools, while Pinyin can represent multiple Hanzi characters with different gender associations.

    Source: §1 · pp. 1–2

  • Method

    The paper trains Hanzi regression and classification models from human-rated names and aggregates predictions across possible Hanzi forms for Pinyin, reporting ambiguity.

    Source: §1 · p. 2, §7 · p. 13

  • Results

    The models performed best in all four comparisons against human annotations for Hanzi and Pinyin scores and labels.

    Source: §6 · p. 13, §7 · p. 13

  • Takeaways & Limitations

    The authors conclude that Hanzi-level information is key to reliable gender inference for Chinese names and that Pinyin inference should be accompanied by ambiguity.

    Source: §7 · p. 13

    The gold-standard annotations come from Chinese physics-publication names, so model performance may differ for other disciplines or diaspora communities.

    Source: §5 · p. 11

Abstract

from arXiv · show

Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars. We address this gap for Chinese names, which appear in two forms, Hanzi and Pinyin, its Romanized form (e.g., Zhang Wei, Li Na). In international bibliographic records, names are often available only in Pinyin. However, a single pinyin name can correspond to multiple distinct hanzi characters with different gender associations, so gender inference from Pinyin alone is intrinsically ambiguous. To address this, we first develop two hanzi models, a regression model that produces a continuous gender score from 0 (male) to 1 (female), and a classification model that outputs one of three labels (\em{male}, \em{neutral}, \em{female}). We train them on a sample of hanzi names from Chinese-language physics publications, rated by 11 native Chinese speakers. On this sample, the models achieve an RMSE of $0.09$ and an R$^2$ of $0.85$ (regression), and macro-AUC of $0.97$ and a Cohen's $\kappa$ of $0.81$ (classification), demonstrating high performance that is robust to class imbalance. For pinyin names, we propose a pipeline that aggregates hanzi predictions via transliteration and frequency weighting, together with an ambiguity score that flags names whose gender cannot be resolved from Pinyin alone. Benchmarked against three off-the-shelf tools (Namsor, GPT, and Claude), our models achieve the highest agreement with human annotations for both gender scores and gender labels, in both scripts. Applied to an independent sample of Chinese-named authors in APS journals, our models assign a gender to every name, whereas existing Pinyin-based methods leave between 20\% and 100\% of names unclassified. These findings demonstrate the importance of hanzi-level information for reliable gender inference of Chinese names. Code and data are publicly available.

1 Introduction

Chinese names are often available in Pinyin in international scholarly records, but Pinyin syllables can map to many Hanzi characters with different, probabilistic gender associations. The paper develops Hanzi-based gender models and extends them to Pinyin by aggregating predictions across possible Hanzi forms.

1 Introduction

  • Existing gender-inference tools are poorly suited to many non-Western names, and Chinese names have been reported to receive substantially lower accuracy.The authors connect this limitation to the exclusion of Chinese names and systematic bias in studies of collaboration, citation, and diversity.Source: §1 · p. 1
  • Pinyin syllables can map to many Hanzi characters with different gender associations, making gender inference from Pinyin alone inherently ambiguous.In the authors’ corpus, 8,094 distinct single characters collapse into 408 syllables; yi, ji, and yu each map to more than 120 characters.Source: §1 · p. 2
  • The framework trains Hanzi regression and three-class classification models, then reuses their predictions across plausible Hanzi mappings for Pinyin names.Pinyin scores use uniform or frequency-weighted aggregation; label predictions use majority voting, with neutral assigned when labels tie.Source: §1 · p. 2, Figure 1 · p. 2
  • The models are evaluated using names from Chinese-language physics publications and human annotations, while Pinyin strategies are assessed by transliterating annotated Hanzi names.The authors also benchmark against Namsor, GPT, and Claude on both Hanzi and Pinyin names.Source: §1 · pp. 2–3

2 Related work

Prior work has studied gender inference for Chinese names, but the reviewed methods do not develop and evaluate a method operating directly on Hanzi characters. This paper reuses Hanzi predictions across possible Pinyin mappings and evaluates them against Chinese-speaking annotators and off-the-shelf tools.

2 Related work

  • A review of 14 papers found that none developed or evaluated gender inference operating directly on Hanzi characters.The review also identifies studies of authorship, survey experiments, and tool benchmarks; one Chinese-name benchmark found evaluated tools inaccurate on Pinyin given names.Source: §2 · p. 3
  • Earlier combined-script methods use Hanzi and Pinyin embeddings, character graphs, or auxiliary pinyin-to-character learning with binary classification.The latest approach uses Hanzi during training but requires Pinyin alone at inference, and its authors report more frequent misclassification of female names in male-skewed training data.Source: §2 · p. 4
  • This work instead reuses Hanzi-based predictions across Pinyin candidates, offers continuous scores or three-class labels with ambiguity scores, and benchmarks against annotators and off-the-shelf tools.The cited comparison includes recent LLMs GPT and Claude.Source: §2 · p. 4

3 Methodology

The methodology builds Hanzi gender models from tool-derived name features and human ratings, then aggregates predictions across possible Hanzi forms for Pinyin names. It evaluates regression against held-out annotations and compares its predictions with off-the-shelf methods.

3.1.1 Hanzi given names (CNKI corpus)

  • The corpus contains 46,144 unique Hanzi given names, including 6,432 added single-character names to improve coverage of that less common form.The original CNKI set contains 39,712 unique given names, 4% of them single-character.Source: §3.1.1 · p. 4

3.1.2 Transliteration (Hanzi–Pinyin mapping)

  • A dedicated Hanzi–Pinyin table supports lookup in both directions, with tone-stripped Pinyin used for names as commonly romanized internationally.The table prioritizes the pronunciation most commonly used in given names.Source: §3.1.2 · pp. 4–5

3.1.3 Hanzi features (X)

  • Three name-gender tools provide ten Hanzi features, combining numeric scores, labels, votes, and name characteristics; the R–N average is the best-performing numeric feature.The features rely on name strings rather than bibliometric context, so the model can apply to names outside publication settings.Source: §3.1.3 · pp. 5–6

3.1.4 Gold-standard annotations

  • Eleven native Chinese speakers rated 166 sampled names on a five-point scale, with mean, median, and mode aggregates used as alternative regression targets.Scores were rescaled to [0, 1]; the same sample was discretized under eight threshold configurations for classification.Source: §3.1.4 · p. 6

3.2 Regression models

  • The regression model maps each Hanzi given name to a continuous gender score rather than forcing a binary decision.The methodology describes regression and classification models, with the classification details deferred to Section D.Source: §3 · p. 4, §3.2 · p. 6

3.2.1 Hanzi-based regression (gender score ˆyh)

  • The Hanzi regressor is selected by held-out RMSE after stratified splitting, cross-validation, hyperparameter search, and repeated feature and target-aggregation comparisons.The training sample comprises 166 gold-standard names; three random seeds are used.Source: §3.2.1 · pp. 6–7

3.2.2 Pinyin-based regression (gender score ˆyp)

  • Pinyin scores reuse Hanzi predictions across candidate mappings and aggregate them with four strategies varying candidate set and uniform versus frequency weighting.The national uniqueness index supplies population-proxy weights; ambiguity is the 90th-minus-10th percentile spread across candidate Hanzi predictions.Source: §3.2.2 · p. 7

3.2.3 Comparison with off-the-shelf tools.

  • Off-the-shelf comparisons evaluate Namsor, Claude Opus 4.8, and GPT 5.4 Nano separately on Hanzi and Pinyin inputs using score metrics.For Namsor, the authors re-query the API for comparison; Claude and GPT use prompts described in the appendix.Source: §3.2.3 · p. 8

4 Results

Hanzi-based inference closely reproduces human-rated gender scores, while Pinyin aggregation is less accurate and varies with transliteration ambiguity. Against off-the-shelf tools, the proposed model performs best in both scripts.

4 Results

  • Averaged hanzi ratings are highly reliable (ICC(2,k) = 0.95, 95% CI [0.94, 0.96]), despite moderate single-annotator agreement (ICC(2,1) = 0.62, 95% CI [0.57, 0.68]).Most variance reflects differences between names rather than disagreement among annotators.Source: §4 · pp. 8–9

4.1 Hanzi-based inference (gender score ˆyh)

  • On the held-out hanzi test set, the model achieves R2 = 0.85, RMSE = 0.09, and Pearson ρ = 0.93.The model uses seven of ten derived features; numeric gender signals from R and N contribute most, while single_char contributes less than 1%.Source: §4.1 · p. 9
  • The best hanzi model uses the aggregated R and N numeric score as its most important feature, at approximately 40%.The binary single_char feature contributes less than 1%, and SHAP values corroborate the feature-importance ranking.Source: §4.1 · p. 9

4.2 Pinyin-based inference (gender score ˆyp)

  • Across four Pinyin aggregation strategies, single-character names are hardest: their scores cluster near 0.4 although corresponding hanzi scores span 0.2 to 0.7.Mean CNKI performs best on the full ground-truth set (R2 = 0.36, RMSE = 0.19); Weighted Pop. performs best on the held-out test set (R2 = 0.30, RMSE = 0.20).Source: Figure 4 · p. 9, §4.2 · p. 10
  • Ambiguity reflects the spread and concentration of gender associations across hanzi variants, not simply the number of variants.Ji has 122 variants but low ambiguity, whereas lili has 19 variants and considerably higher ambiguity.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Comparison with off-the-shelf methods

  • R2 = 0.85 and RMSE = 0.09 for hanzi inference, while Weighted pop. leads pinyin inference with R2 = 0.30 and RMSE = 0.20; our model outperforms the compared tools in both scripts.Figure 6 compares 34 held-out names; on hanzi, Claude follows with R2 = 0.75 and RMSE = 0.12. Pinyin scores drop sharply across methods.Source: §4.3 · pp. 10–11, Figure 6 · p. 11

5 Limitations and future work

The authors identify limits in the population and corpus used to build and evaluate their models, as well as reproducibility concerns with commercial tools. They outline ways to extend and recalibrate the work.

5 Limitations and future work

  • Names from Chinese physics publications may not represent other disciplines, diaspora communities, or birth cohorts, where name–gender associations can differ over time.The annotations were stratified to cover a range of gender scores; population weights partly account for differences in Pinyin aggregation but not Hanzi predictions.Source: §5 · pp. 11–12
  • Pinyin evaluation compares an aggregate prediction over plausible Hanzi variants with the annotation for a single variant, so some error reflects unresolved ambiguity rather than model failure.The ambiguity score flags names whose Pinyin predictions warrant caution and can be recalibrated using variants observed in another population.Source: §5 · p. 12
  • Pinyin aggregation depends on corpus-specific candidate variants and weights; population-level weights performed best on the test set, though strategy differences were small.Candidate sets and weights can be re-estimated for a target corpus without retraining the Hanzi models.Source: §5 · p. 12
  • Commercial-tool outputs can change without notice, limiting reproducibility; in a Namsor re-query, 63 of 156 shared labels changed.The 63 changes represented 40% of the shared labels, and 55 changed from male to female; Claude versions used in prototyping were unavailable at scale-up.Source: §5 · p. 12

6 Discussion

The discussion reports complete name coverage in an APS sample where existing Pinyin methods leave many names unclassified, and explains why perceived gender and uncertainty matter. It distinguishes continuous scores from labels and compares the models with LLMs.

Perceived gender and ambiguity

  • Pinyin inference relies on possible Hanzi forms because Pinyin carries little gender information, and the ambiguity and disagreement scores quantify how consistent those candidates’ gender associations are.The authors argue that Pinyin-based inference should report an uncertainty index alongside the most likely gender.Source: §6 · p. 12

Choosing between scores and labels

  • Continuous scores suit aggregate-trend analyses, while three-class labels suit studies requiring discrete groups; neutral perception matters because collapsing neutral names can skew aggregate gender ratios.The authors compared eight ways to derive labels and recommend reporting the thresholds used for discrete classes.Source: §6 · pp. 12–13

Unknowns and downstream consequences

  • Existing Pinyin methods leave 20% to 100% of fully spelled-out Chinese-named APS authors unclassified, while the authors’ models assign a score and label to every name in both scripts.Discarding unclassified names may bias estimates when excluded researchers differ in seniority, subfield, or productivity.Source: §6 · p. 13

The role of LLMs

  • The models performed best in all four comparisons with human annotations, while Claude ranked second; Claude’s performance was weaker on Pinyin names than on Hanzi names.The discussion also notes that LLM outputs are difficult to reproduce over time and that the pipeline could extend to Japanese kanji with suitable character-level resources.Source: §6 · p. 13

7 Conclusion

The paper presents a framework for perceived-gender inference from Chinese names in Hanzi and Pinyin. It reports closer agreement with human perception than off-the-shelf tools and emphasizes Hanzi information and uncertainty for Pinyin inference.

7 Conclusion

  • The framework infers perceived gender from Chinese names in Hanzi or Pinyin using Hanzi models and a Pinyin pipeline that reports ambiguity.The Hanzi models produce either continuous scores or three-class labels; the Pinyin pipeline reuses Hanzi predictions through transliteration.Source: §7 · p. 13
  • The Hanzi models agree more closely with human perception than off-the-shelf tools, including commercial LLMs, and the paper identifies Hanzi-level information as key to reliable Chinese-name inference.The authors say Pinyin inference should be accompanied by a measure of uncertainty.Source: §7 · p. 13
  • The authors release code, predictions, annotated data, and an interactive tool for reproduction, use on other corpora, and benchmarking future tools.They hope these resources help science-of-science research include Chinese scholars in gender analyses.Source: §7 · p. 13

Funding

The acknowledgments identify funding and institutional support for the authors’ work.

Funding

  • The authors acknowledge support from WWTF, FFG, the China Scholarship Council, Horizon 2020, and the Austrian Ministry for Innovation, Mobility and Infrastructure.The passage lists project and grant identifiers for these awards.Source: Funding · p. 14

Supplementary information

The section outlines the CNKI corpus, disambiguation algorithm, and its validation so the reliability of the author-count weighting scheme can be assessed.

Supplementary information

  • The section summarizes the corpus, disambiguation algorithm, and validation underlying the author-count weighting scheme.The full treatment, including threshold search, annotation interface, and per-method confusion matrices, appears in the companion study.Source: A CNKI corpus and author name… · p. 19

A.1 Data source

The supplementary materials describe the CNKI corpus and author disambiguation, validate the resulting identities across records and scripts, and explain how these identities support name weights and model features.

A.1 Data source

  • The corpus contains 65,241 parseable Chinese-name publications from 20 Chinese Physical Society journals, spanning coverage up to 2024-02-14.The corpus preserves names as printed and lacks author identifiers, motivating disambiguation.Source: §A.1 · p. 19, Table A.1 · p. 19

A.2 Why disambiguation is required

  • Counting people rather than name occurrences prevents prolific authors from dominating frequency weights, making disambiguation necessary.The algorithm tests shared-name paper pairs against co-authorship, affiliation, content, and citation criteria, merging a pair if any criterion is met.Source: §A.2 · p. 19, §A.3 · p. 20

A.4 Resolved identities

  • 14,467 pinyin names resolve into multiple identities, versus 13,580 hanzi names; transliteration collisions still leave fewer identities overall in the pinyin run.The pipeline yields 99,222 pinyin identities and 106,411 hanzi identities.Source: §A.4 · p. 20

A.5 Validation

  • F1 reaches 0.89 for hanzi names and 0.88 for pinyin names, with precision of 1.00 and 0.95, respectively, and lower recall.The algorithm outperforms two established baselines, chiefly through higher recall.Source: §A.5 · p. 21

A.6 Consistency across scripts

  • Mean Jaccard similarity across 106,411 hanzi identities is 0.92, and 87.3% receive exactly the same papers under both scripts.Disagreements concentrate where distinct hanzi names collide under a shared pinyin spelling.Source: §A.6 · p. 21

A.7 From identities to name weights

  • Weights count disambiguated people bearing each hanzi given name once, regardless of publication volume.The 6,432 added single-character names have zero CNKI weight, so Weighted CNKI uses the corpus hanzi set while population weighting can extend to the larger candidate set.Source: §A.7 · p. 21

B Chinese name-to-gender literature search

  • The literature search retained 10 of 14 retrieved papers; none addressed gender inference from Hanzi, and only one used computational methods for Chinese names, focusing on Pinyin.Six other computational studies used proprietary databases, one used its own data, and two examined name meanings or stereotypes.Source: B Chinese name-to-gender literature… · pp. 21–22

C.1 Hanzi features’ predictive power

  • The average numeric scores from R and N were the dominant predictor across regression and classification, and their combination yielded the best regression fit (R2 = 0.85; RMSE = 0.09).The RN combination also had the lowest RMSE and highest correlation with annotator ratings among tool combinations.Source: §C.1 · pp. 22–23

C.2 Pinyin-based gender score

Pinyin scores aggregate predictions across matching hanzi, but the aggregation can obscure variation among characters. The supplementary classification results report strong performance for the best hanzi model.

C.2 Pinyin-based gender score

  • Single-character pinyin names with many hanzi variants can pull strong gender signals toward the center; for meng, MCNKI = 0.47 and MCNKI+ = 0.40 versus gold score 0.79.Weighting raises the prediction to WCNKI = 0.55 and WPop. = 0.56, but does not fully resolve high ambiguity.Source: §C.2 · p. 24, Table A.3 · p. 24
  • The ambiguity score uses the central 80% of hanzi predictions (q90 − q10), discounting outliers; high ambiguity indicates disagreement no aggregation strategy can fully resolve.Low ambiguity indicates greater agreement across variants and a more trustworthy aggregated score.Source: §C.2 · p. 25
  • Mean CNKI+ aligns most closely with hanzi-based predictions (R2 = 0.69, RMSE = 0.116, ρ = 0.83), marginally outperforming Mean CNKI.Weighted CNKI and Weighted population have R2 values of 0.63 and 0.59, respectively.Source: §C.2 · p. 25, Figure A.3 · p. 25

D.1 Gender score discretization

  • The eight discretization schemes vary class boundaries and proportions; retaining all eight allows robustness to be assessed across choices rather than prespecifying one.A neutral class preserves genuinely neutral names and avoids forcing ambiguous cases into male or female.Source: §D.1 · p. 26, Figure A.4 · p. 26

D.2.1 Performance

  • The best classifier uses Cut 1 and five features, with mean_ngender_rn accounting for roughly 60% of predictive power.The remaining features contribute smaller complementary shares, and the SHAP summary shows class-resolved feature effects aligned with their expected semantics.Source: §D.2.1 · p. 27
  • Only two of 32 held-out samples are misclassified; the classifier reaches macro-averaged AUC of 0.971 and Cohen’s κ of 0.807.The test-set per-class AUC values range from 0.945 to 1.000.Source: §D.2.1 · p. 27, Figure A.5 · p. 27

D.2.2 Comparison with off-the-shelf tools

The hanzi classifier agrees with human labels more than the compared off-the-shelf tools. Pinyin labels are assigned by a supermajority of associated hanzi labels, with separate measures describing ambiguity.

D.2.2 Comparison with off-the-shelf tools

  • Our method has the highest agreement with ground truth (κ = 0.81, accuracy = 0.88), ahead of Claude (κ = 0.64, accuracy = 0.75), GPT, and Namsor.GPT and Namsor each have accuracy 0.44, with κ values of 0.26 and 0.22, respectively.Source: §D.2.2 · p. 28, Figure A.6 · p. 28
  • Our method leads on male precision (0.90), female precision (1.00), and male and neutral recall (0.82 and 0.92); Claude leads on neutral precision (0.82).Namsor’s female recall of 1.00 is offset by poor precision across other classes, and it cannot predict neutral labels.Source: §D.2.2 · p. 28, Figure A.7 · p. 29

D.3.1 Aggregation

  • A pinyin label is male or female only when that class exceeds a two-thirds share of its mapped hanzi labels; otherwise it is neutral.The label proportions are computed over the hanzi associated with each pinyin.Source: §D.3.1 · p. 28

D.3.2 Ambiguity

  • Normalized entropy measures heterogeneity among hanzi labels for a pinyin, with larger values indicating greater heterogeneity.Source: §D.3.2 · p. 29
  • Disagreement rate measures how often a pinyin-level label differs from its underlying hanzi labels, with larger values indicating greater disagreement.Source: §D.3.2 · p. 29

D.3.3 Performance

Pinyin-based gender classification performs best among the compared tools but remains less reliable than Hanzi-based inference. Its aggregate labels also conceal variation among the Hanzi mappings behind each Pinyin name.

D.3.3 Performance

  • Macro-AUC is 0.890 on the full set and 0.864 on the test set, while Cohen’s κ is 0.328 and 0.259, respectively.The high AUCs coexist with low agreement when the supermajority rule is used as a hard classifier.Source: §D.3.3 · p. 29, Figure A.8 · p. 30
  • A Pinyin supermajority label is a lossy summary because one Pinyin name can map to Hanzi names spanning multiple gender categories.This ambiguity is absent in Hanzi-based classification.Source: §D.3.3 · p. 30
  • Entropy and disagreement alongside the supermajority label capture variation that the label alone misses, including cases with no dominant gender category.The held-out-set table defines entropy as Hanzi-label mixture and disagreement as the share of labels differing from the Pinyin label.Source: §D.3.3 · pp. 30–31, Table A.4 · p. 31

D.3.4 Comparison with off-the-shelf tools

  • Female precision is 1.00 and male recall is 0.73 for our method, its strongest reported class-wise results.The method also has the highest neutral recall (0.46), while GPT has higher neutral precision (1.00).Source: §D.3.4 · p. 31, Figure A.10 · p. 32
  • Accuracy is 0.53 and Cohen’s κ is 0.27 for our Pinyin method, the highest agreement with ground truth among the compared tools; all methods trail their Hanzi counterparts.Claude follows with accuracy 0.47 and κ 0.20; GPT and Namsor have lower values.Source: §D.3.4 · p. 32

D.4 Implications for Scientometrics and Science-of-Science research

In the APS comparison, open-source Pinyin baselines often leave names unclassified, while Claude’s Hanzi predictions agree most with the paper’s Hanzi model. The paper also documents an iterative, modular prompt design for comparing scripts and inference modes.

D.4 Implications for Scientometrics and Science-of-Science research

  • The APS evaluation uses Hanzi names as the reference and excludes authors whose Pinyin given names appear only as initials.Those initials account for 46% of Chinese-named APS authors; the authors describe the resulting coverage comparison as conservative.Source: §D.4 · p. 32
  • 100% of names receive no label from Saeta, versus ≈81% from van Buskirk et al. and 20% from Karimi et al.; our models always return labels.The comparison covers the 54% of Chinese-named APS authors whose Pinyin given names are fully spelled out.Source: §D.4 · pp. 32–33
  • Raw agreement with Ours (HZ) is 80% for Claude (HZ), 73% for GPT (HZ), and 72% for the Pinyin model; chance-corrected κ values are modest overall.Claude (HZ) has the highest κ among the compared methods.Source: §D.4 · p. 33, Figure A.12 · p. 34

E LLM prompt design

  • After long single-request name lists produced worse intermediate results, the authors revised prompts and scoring definitions and moved Claude inference to API batches of 100.Claude’s chat interface blocked full Pinyin requests; the API provided programmatic access for the batch workflow.Source: E LLM prompt design · pp. 34–35
  • The final prompt has four modular blocks, allowing script or inference mode to change by swapping one block while keeping the others constant.The Pinyin preamble reasons over plausible character mappings, while the Hanzi preamble uses character-level gender signals; scoring can be continuous or categorical.Source: E LLM prompt design · p. 35
  • The prompt distinguishes intrinsic name ambiguity from model confidence and uses a flat JSON schema for lossless input-output matching across batches of up to 100 names.The uncertainty definition is shared across both inference modes.Source: E LLM prompt design · p. 35, Figure A.16 · p. 36
Loading 2610.23734v1…