PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

Sahil Pardasani, Madhusudan Singh

arXiv:2609.26942v1cs.CLcs.AI

TL;DR

The paper addresses whether multilingual models can generate culturally specific kinship terms in complete messages, not only select them in multiple-choice tests. It benchmarks five open-weight models on Hindi, Korean, and Tamil prompts and compares generation with option-supported selection. Explicit L3 generation varies widely across models, while selection is substantially higher for two models than their generation performance.

  • Problem

    Multilingual kinship understanding is commonly evaluated through multiple-choice recognition, leaving open whether models can generate culturally specific terms in complete messages.

    Source: §1 · p. 1

  • Method

    The authors test five open-weight models on Hindi, Korean, and Tamil kinship prompts across two communicative tasks, pairing repeated generation with matched four-option selection.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Explicit L3 generation ranges from 72.29% for GLM-5.1 to 24.24% for Llama-3.3-70B, while option-supported selection is substantially higher for GPT-OSS-120B and Llama-3.3-70B.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Culturally specific kinship generation remains difficult even when relationships are explicit, supporting generation-based evaluation alongside multiple-choice testing.

    Source: §5 · p. 6

    The selection comparison covers two of five models and does not isolate retrieval because options display the answer and remove spelling, script production, and message composition demands.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a recognition problem. We instead prompt five open weight LLMs to generate kinship terms in three non Western languages (Hindi, Tamil, and Korean) across two communicative tasks and pair this with a matched option-supported selection baseline. On identical relation language cells, GPT OSS120B selects the correct term in 90.67% of 75 valid cells but produces an accepted term in 36.00% of the corresponding attempts; Llama 3.370B shows the same pattern (77.92% versus 24.24%). Since the four-option condition displays the candidate terms and does not require script production, the difference is interpreted as an evaluation format gap rather than direct proof that lexical knowledge is intact. On explicitly specified L3 prompts, accuracy varies sharply, from GLM-5.1 at 72.29% to Llama-3.370B at 24.24%. The paternal-lineage advantage is language specific; it is large in Hindi but weak or reversed in Korean, while Tamil shared-term pairs provide a control for measurement variation. These results show that culturally specific kinship generation remains difficult even when the relationship is explicitly stated and motivate generation-based evaluation alongside multiple-choice testing.

1 Introduction

The paper asks whether models that can select culturally specific kinship terms can also generate them reliably in natural messages. It introduces a generation benchmark paired with option-supported selection on matched relation–language cells.

1 Introduction

  • Culturally specific kinship systems encode distinctions such as lineage, age, marriage, and speaker position that English terms like “uncle” or “grandmother” collapse.For example, Hindi distinguishes a father’s elder brother as Tau/Tauji from a younger brother as Chacha/Chachaji.Source: §1 · p. 1
  • The study tests whether models can reliably generate culturally specific kinship terms in birthday greetings and wedding invitations when the relationship is specified.It evaluates five open-weight models in Hindi, Korean, and Tamil.Source: §1 · p. 1
  • The benchmark pairs generation with four-option selection over the same relation–language cells, treating selection as option-supported rather than direct proof of recognition.Displayed terms remove spelling and native-script production demands.Source: §1 · p. 1

2 Related Work

Prior work documents kinship lexical gaps, multilingual representation challenges, and cultural-alignment issues, but does not directly test culturally specific kinship generation in complete role-based messages. This paper addresses that narrower gap with matched generation and selection tasks.

2 Related Work

  • Prior work identifies kinship lexical gaps and shows that English-centered resources can collapse gender-, age-, or lineage-specific concepts into broader categories.Li et al. (2024b) study lexical-gap detection and generation; Khishigsuren et al. (2022) document multilingual kinship diversity and translation errors.Source: §2 · p. 2
  • Research on multilingual representation links generation errors to morphology and non-Roman scripts, while cultural-alignment studies find multilingual fluency alone does not ensure culturally appropriate behavior.Nzeyimana (2024) examines morphology-aware translation; Husain et al. (2024) propose ROMANSETU, while Li et al. (2024a) and AlKhamissi et al. (2024) study cultural alignment.Source: §2 · p. 2
  • Work separating generation from verification reports that models may verify factual statements more reliably than they generate them; multilingual instruction-tuning gains also vary by language, task, and translation direction.Davidson et al. (2026) argue for independent evaluation of generation and verification; Ranaldi et al. (2024) report cross-lingual instruction-tuning results.Source: §2 · p. 2
  • Culture-aware translation studies evaluate cultural adaptation and functional equivalence or use multilingual knowledge-graph retrieval for culturally nuanced entities.Yuan et al. (2026) introduce CanMT; Conia et al. (2024) focus on culturally nuanced entities.Source: §2 · p. 2
  • Prior work does not directly test whether recent open-weight models use culturally specific kinship terms in complete role-based messages when relationships are explicit.The paper compares repeated generation with option-supported selection on matched cells and examines variation across languages and family relations.Source: §2 · p. 2

3 Methodology and Experimentation

The benchmark combines reviewed kinship mappings, prompts with different relationship specificity, rule-based scoring, and a matched option-supported selection condition. Its design spans multiple relations, languages, tasks, and repeated generations.

3 Methodology and Experimentation

  • 81 language-specific term entries were independently reviewed by a second native speaker per language, with no changes requested.The 25 tested relations are lexically distinct in at least one of Hindi, Tamil, and Korean.Source: §3 · p. 2

3.1 Prompt and Benchmark Construction

  • L1 prompts use underspecified English hypernyms, whereas L3 prompts specify the relationship; birthday greetings and wedding invitations test term use in socially meaningful messages.Elder brother appears only at L3, including Korean speaker-gender variants.Source: §3.1 · p. 3
  • Each model receives 300 unique prompts and three runs per prompt, yielding 4,500 responses overall; cell accuracy averages all six binary outcomes across the two tasks and three runs.The design has 146 L1 and 154 L3 prompts per model.Source: §3.1 · p. 3

3.2 Evaluation Protocol

  • The frozen matcher counts accepted native-script terms found as literal substrings, so its score measures accepted-term production rather than full grammar or message quality.Failures can include omitted terms, wrong scripts or languages, wrong relations, or unlisted valid variants.Source: §3.2 · p. 3
  • Four of five mirrored paternal–maternal pairs share the same gold token in Tamil, providing a control for measurement variation in the lineage analysis.For example, both paternal and maternal grandfathers map to thaatha.Source: §3.2 · p. 3

3.3 Option-Supported Selection Baseline and Decoding Setup

  • Option-supported selection uses 77 fully specified relation–language–perspective items for GPT-OSS-120B and Llama-3.3-70B, with native-script options and a Latin-letter response.The four options include the gold answer once; distractors are broad inventory terms rather than deliberately close alternatives.Source: §3.3 · p. 3
  • Generation uses temperature 0.7 with three runs, while selection uses temperature 0 and one run; two blank GPT outputs leave 75 valid selection results.Table 1 reports L3 generation accuracy over 77 cells per model, with intervals resampling whole relation–language–perspective cells.Source: §3.3 · p. 4, Table 1 · p. 4

4 Findings

Option-supported selection substantially outperforms open generation on matched cells, while L3 generation varies widely across models. Failure patterns include script-production problems, inventory mismatches, and relation-specific difficulty; paternal–maternal differences vary by language and model.

4.1 Overall generation accuracy

  • 72.29% for GLM-5.1 and 24.24% for Llama-3.3-70B mark the high and low ends of L3 accepted-term accuracy.Kimi-K2.6 scored 48.70%, Sarvam-M 40.48%, and GPT-OSS-120B 35.06%.Source: §4.1 · p. 4
  • 44.07% for birthday prompts and 44.24% for wedding invitations were the accepted-term accuracies across all five models.Task-level differences were small relative to variation between models; direction also differed by model.Source: §4.1 · p. 4, Table 2 · p. 4
  • 68 of 75 valid cells for GPT-OSS-120B and 60 of 77 for Llama-3.3-70B were selected correctly, while matched-cell generation accuracy was 36.00% and 24.24%, respectively.The matched-cell comparison reflects different evaluation demands: selection displays candidate terms and avoids spelling, native-script production, and full-message generation.Source: Table 3 · p. 4, §4.1 · p. 4
  • Hindi’s pooled paternal–maternal mean difference was +43.3 points, versus +4.0 in Korean, where the direction reversed for two relation pairs.Across models, signed means ranged from +3.3 to +90.0 points in Hindi, −6.7 to +16.7 in Korean, and −16.7 to +30.0 in Tamil.Source: §4.1 · p. 5, Table 4 · p. 5, Table 5 · p. 5

4.2 Failure modes

  • Llama-3.3-70B produced no Hangul in 162 of 162 Korean L3 responses, while script-absence rates varied across other model–language pairs.The L3 breakdown distinguishes missing-script outputs from target-script outputs that fail the accepted-term inventory, but manual semantic error analysis remains necessary.Source: §4.2 · p. 5
  • 1,742 of 4,500 generations passed the accepted-term matcher; among failures, 57.90% contained another target-language inventory token and 17.01% lacked requested-script characters.A further 24.62% contained target-script characters but no inventory token; 0.47% had target-script text and expected Romanization but no accepted native-script form.Source: §4.2 · p. 5
  • Among L3 failures, 52.09% contained another inventory token, 27.21% had target-script text but no inventory token, and 19.69% lacked the target script.These are surface categories, not semantic judgments; no-inventory cases can include omitted terms, valid unlisted variants, or scorer false negatives.Source: §4.2 · p. 5
  • L3 accuracy ranged from 4.44% for husband’s sister’s husband to 58.33% for elder brother’s wife among the listed affinal relations.Younger sister’s husband scored 6.67%, elder sister’s husband 15.56%, husband’s sister 18.89%, and wife’s brother 20.00%.Source: §4.2 · p. 5

5 Conclusion

The benchmark evaluates culturally specific kinship-term generation across 25 relations, three languages, two communicative tasks, and five open-weight models. Results vary across models, tasks, languages, and relations, supporting generation evaluation alongside option-supported selection.

5 Conclusion

  • 72.29% for GLM-5.1 and 24.24% for Llama-3.3-70B mark the range in explicitly specified L3 generation.Source: §5 · p. 6
  • A matched option-supported selection baseline is substantially higher for GPT-OSS-120B and Llama-3.3-70B than complete-message generation.The comparison is on matched relation–language cells.Source: §5 · p. 6
  • The benchmark covers 25 tested relations, three languages, two communicative tasks, and five open-weight models across 4,500 generations.Source: §5 · p. 6
  • The paternal–maternal difference is large in aggregate for Hindi but varies by model, while Korean is weak or reversed.Tamil shared-term pairs expose measurement variation even when the expected token is identical.Source: §5 · p. 6

Limitations

The evaluation is limited by its language and relation coverage, token-presence scoring, and design choices that constrain interpretation of results. These boundaries include unvalidated scorer error rates, restricted selection comparisons, and uncontrolled cross-model differences.

Limitations

  • Three languages, 25 tested relations, and a single standard register do not support claims about kinship generation generally.Regional dialects and evolving contemporary usage may license terms the lists reject.Source: Limitations · p. 6
  • Token-presence scoring can miss legitimate inflections or credit a token in an explanation despite an incorrect address term.Without a stratified human audit, matcher false-positive and false-negative rates cannot be reported.Source: Limitations · p. 6
  • The inventory review confirmed 81 entries without changes, but was table-level rather than blinded item-level annotation and reported no agreement coefficient.Slash-marked alternatives that may represent different linguistic conditions were not tested as separate randomized items.Source: Limitations · p. 6
  • L1 prompts allow several valid answers but are scored against one gold term, so L1 is excluded from headline and selection–generation tables.Source: Limitations · p. 6
  • Option-supported selection covers two models, always includes the answer, and avoids spelling, script production, and message composition.It uses one temperature-0 run rather than three at 0.7, so the comparison is a task-format difference, not a direct measure of recognition.Source: Limitations · p. 6
  • The model panel lacks smaller-model and language-specialist baselines, and equivalent proficiency across the three languages was not established.Cross-model differences are descriptive rather than controlled evidence about scale or cultural knowledge; training corpora are unobserved, precluding causal claims about the patterns.Source: Limitations · p. 6
  • The original LaTeX setup did not reliably render Devanagari and Hangul, so the appendix uses transliteration.Source: Limitations · p. 6

Future Work

Future work should make selection comparisons more controlled, distinguish ambiguity from defaulting, and broaden validation of both the gold inventory and output scoring. A paired retrieval-support study is also planned, following an exploratory pilot that is not benchmark evidence.

Future Work

  • Future selection tests should use close relation- and lineage-level distractors, balanced answer positions, answer-absent items, free recall, and script controls.The proposed design also supplies the correct term for use in a complete message and includes all evaluated models plus smaller and language-specialist baselines with independently assessed proficiency.Source: Future Work · p. 6
  • Future L1 evaluation should distinguish clarification, multiple valid alternatives, and unjustified defaulting, while composite gold entries become condition-specific items.The proposed validation includes regional varieties, reviewer backgrounds, adjudication, term provenance, and a blinded stratified output audit to estimate scorer error rates.Source: Future Work · p. 6
  • The authors plan a controlled paired study of retrieval support after an exploratory pilot exposed a kinship-mapping graph to a model as an MCP tool.The pilot examples came from a locally served 20B model outside the evaluation set and were illustrative, not benchmark statistics or evidence that the tool solves the task.Source: Future Work · pp. 6–7

Data and Code Availability

The paper provides its data, code, and documented offline reproduction workflow online, but not human output-audit records.

Data and Code Availability

  • Data, code, and a documented offline reproduction workflow are available, while human output-audit records are absent because stratified annotation has not been conducted.Source: Data and Code Availability · p. 7

Ethics Statement

The ethics statement describes native-speaker review and synthetic prompts without personal data; Table 6 is identified as reconciling the paper’s main denominators.

Ethics Statement

  • Two native speakers per language reviewed the gold terms and accepted variants, and the second reviewers confirmed all 81 entries without changes.The study collected no private or personally identifiable information and used synthetic role-based prompts.Source: Ethics Statement · p. 7

A Count Reconciliation

  • Table 6 reconciles the main denominators used in the paper.Source: Table 6 · p. 8

A.1 Model and run documentation

The experiment documentation records configured model identifiers and response counts, including which models contributed to selection and where per-call audit data can be found.

A.1 Model and run documentation

  • All five models used the Section 3.1 generation templates and contributed 900 generation responses.Source: §A.1 · p. 8
  • Only GPT-OSS-120B and Llama-3.3-70B were included in option-supported selection.Source: §A.1 · p. 8
  • Per-call timestamps, provider labels, error fields, and analysis membership are available in model_run_metadata.csv and row-level audit files.Source: §A.1 · p. 8

A.2 Recognition examples

The examples show English relationship questions paired with target-language options and Latin-letter answers, while the taxonomy distinguishes tested expected-term keys from reference-only mappings and conditional variants.

A.2 Recognition examples

  • The Tamil father’s-younger-brother example lists chithappa as option C, the gold answer.Source: §A.2 · p. 8
  • The Hindi father’s-younger-brother example has gold answer C; GPT-OSS-120B answered incorrectly, while Llama-3.3-70B answered correctly.Source: §A.2 · p. 8

B Kinship Taxonomy

  • Table 8 retains 27 taxonomy mappings but uses 25 tested relationships; Sister and Wife’s sister are reference-only.Slash alternatives represent different kinds of variation, including address alternatives, speaker-gender-conditioned Korean forms, and relative-age-conditioned Tamil forms; they were not separate randomized conditions.Source: Table 8 · p. 9, B Kinship Taxonomy · p. 9
Loading 2609.26942v1…