PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border Communities

Minglu Sun, Lan Chen, Hang Jiang

arXiv:2610.12610v1cs.CYcs.CL

TL;DR

Evaluations that focus on factual accuracy or culture within a single country may miss how LLMs frame identities that cross borders. This paper introduces TIODF to compare direct factual probes with open-ended narratives across nine communities; ossification varies by model and query language, and factual retrieval does not reliably predict non-ossified framing.

  • Problem

    Evaluations that treat the nation-state as the natural unit for culture and identity may miss how models frame communities whose identities cross borders.

    Source: §1 · pp. 1–2

  • Method

    TIODF uses community-specific Knowledge Cards to pair direct binary probes with open-ended narratives and compare factual retrieval with narrative framing.

    Source: §1 · p. 2, §3.2.1 · p. 4

  • Results

    Ossification ranges from 10.1% of GPT-5.1’s English responses to 62.6% of DeepSeek-V3.2’s Chinese responses, while successful factual retrieval does not reliably translate into non-ossified framing.

    Source: §5.1 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    Cultural evaluation should examine not only what models retrieve, but how they organize that knowledge into narratives.

    Source: §6 · p. 8

    Five models across two corpora remain a small sample of the broader LLM landscape, especially for claims about general versus model-specific mechanisms.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Ethnic and cultural identities rarely align with political borders, a tension made sharper by migration and long-standing cross-border ties. Communities such as the Sami, the Rohingya, and the Kurds have long lived across more than one state. This tension is especially clear in mainland Southeast Asia, where many communities share history, language, and culture across state lines. We study nine such transborder ethnic communities along China's southwestern border with mainland Southeast Asia, looking at how large language models (LLMs) represent them. We define and study identity ossification: a pattern where LLMs turn a fluid, cross-border ethnic identity into a single, fixed nation-state identity. We introduce the Transborder Identity Ossification Diagnostic Framework (TIODF) that compares direct factual questions with open-ended narrative questions to test whether ossification comes from missing knowledge or from how models frame their answers. We test GPT-5.1 and DeepSeek-V3.2 in English and Chinese. Ossification varies substantially across models and query languages. It appears in only 10.1% of GPT-5.1's English answers, but in 62.6% of DeepSeek-V3.2's Chinese answers. Models often still tell a single-nation story even when they answer the factual question correctly. This decoupling suggests that ossification reflects narrative framing rather than simple knowledge absence, which matters for how we test cultural fairness in multilingual AI systems.

1 Introduction

The paper identifies a gap in evaluations that treat the nation-state as the natural unit for culture and identity: they can miss how models frame cross-border communities. It defines identity ossification and introduces TIODF to distinguish missing factual knowledge from nation-state-centered narrative framing.

1 Introduction

  • Existing evaluations often test factual correctness or value-laden outputs while taking the nation-state as the natural unit, missing failures specific to cross-border identities.A model may describe a community accurately within one country yet frame its fluid heritage as a rigid state-centered category.Source: §1 · pp. 1–2
  • TIODF compares direct factual probes with independently generated open-ended narratives to test whether nation-state framing reflects knowledge–framing dissociation.It is grounded in community-specific Knowledge Cards and is presented as applicable beyond the nine communities studied.Source: Figure 2 · p. 2, §1 · p. 2
  • Identity ossification is the default compression of a fluid, multidimensional, cross-border ethnic identity into a single nation-state frame.It is an output-level framing pattern, not necessarily a factual error or claim about model-internal mechanisms.Source: §1 · p. 2
  • Across nine communities, the study finds recurrent ossification and reports that successful factual retrieval does not reliably yield non-ossified narrative framing.It also identifies systematic variation by query language and model family and replicates the core dissociation on three additional models.Source: §1 · p. 2

2 Related Work

Prior work examines bias and representation in LLMs, but this paper extends that work to cross-border identities and bilingual evaluation. It connects administrative national categories and narrative framing to a form of identity reduction distinct from factual error.

2.1 Bias and Representation in LLMs

  • The paper extends bias research from factual correctness to narrative framing, from nationally bounded populations to trans-border communities, and from monolingual to bilingual testing.Prior work also documents representational harms that can occur even when individual outputs are not factually wrong.Source: §2.1 · pp. 2–3

2.2 Trans-border Communities and Methodological Nationalism

  • Cross-border ties in the Zomia region predate current borders, while national classifications can assign different names and boundaries to the same community.The paper connects those classifications to possible nation-state defaults in model narratives, even when a model can retrieve a cross-border relation when asked directly.Source: §2.2 · p. 3

2.3 Framing Analysis in NLP

  • Framing analysis distinguishes omission from the ideological reduction of a present community into a nation-state category, even when responses share facts.The paper applies these ideas to uncontested facts about communities that span borders, a setting existing NLP framing analysis had not addressed.Source: §2.3 · p. 3

3 Framework

TIODF pairs community-specific reference material and independent factual probes with open-ended narratives, then compares narrative framing with probe outcomes. The framework also uses descriptive pattern coding and supplementary multilingual embeddings to examine responses.

3 Framework

  • TIODF combines community-specific Knowledge Cards, independent binary probes and open narratives about matched facts, and framing analysis paired with probe outcomes.A successful probe alongside a narrowly nation-state-centered narrative is evidence of knowledge–framing dissociation.Source: §3 · pp. 3–4

3.1 Knowledge Card Construction

  • The study examines nine Zomia communities with documented populations across at least one international border and constructs reviewed Knowledge Cards from ethnographic, historical, and linguistic sources.Cards provide the reference basis for prompt construction and evaluation; Table 1 identifies five dimensions, three grounded in theory and two serving as descriptive baselines.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Response Elicitation

  • Each narrative prompt has an independently administered binary probe about the same relation, preventing the probe from supplying facts or framing to the narrative.The probe tests constrained retrieval, while the narrative tests how the model selects and organizes information under open generation.Source: §3.2 · p. 4

3.2.1 Binary Knowledge Probes

  • Binary probes are closed-form yes-or-no questions about Knowledge Card-documented relations such as shared origins, naming, migration, cultural practice, or religious continuity.The researchers also test probe stability across repeated runs and reversed response-label order.Source: §3.2.1 · p. 4

3.2.2 Open Narrative Prompt Battery

  • The eleven bilingual narrative prompts cover basic community knowledge, cross-border continuity, fluid or dual belonging, and broader historical and cultural narratives.Prompts do not state the expected cross-border relation, allowing the model to select its own subject, scope, and narrative structure.Source: §3.2.2 · p. 4

3.3 Analysis

  • The analysis codes narratives, compares scores with probe outcomes, describes low-scoring patterns, and uses multilingual embeddings as supplementary validation.Three authors identify four framing dimensions scored from 1 to 3, yielding totals from 4 to 12; factual errors are recorded separately. In 144 community-annotated responses, LLM-judge agreement on the total score is ICC(2,1) = 0.75.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.2 Knowledge–Framing Diagnosis

  • Low narrative scores paired with correct probes provide case-level evidence of knowledge–framing dissociation; the aggregate test asks whether probe-passed responses score higher.Low-scoring responses are also coded for recurring patterns, which are descriptive rather than an independent test of ossification.Source: §3.3.2 · p. 5, §3.3.3 · pp. 5–6

3.3.4 Cross-Language Embedding Validation

  • Matched Chinese and English responses are compared using multilingual sentence embeddings, providing a judge-independent supplementary signal of language-conditioned divergence.Embedding distance and rubric coding measure different properties, so embeddings supplement rather than replace the framing rubric.Source: §3.3.4 · p. 6

4 Experimental Setup

The setup compares GPT-5.1 and DeepSeek-V3.2 across English and Chinese, with model–language conditions abbreviated for reporting. Narrative responses use providers’ default sampling settings, unlike factual probes, which use T = 0.

4 Experimental Setup

  • The primary study compares GPT-5.1 and DeepSeek-V3.2; a robustness check adds Gemini-3.1-Pro, Qwen3.6-Max, and Claude Sonnet 4.6.Subject-model queries use OpenRouter; the primary narrative judge is accessed through the Claude web interface.Source: §4 · p. 6
  • EN and ZH denote English- and Chinese-language queries, while DS-EN, DS-ZH, GPT-EN, and GPT-ZH name the corresponding model–language conditions.DS refers to DeepSeek-V3.2 and GPT to GPT-5.1.Source: §4 · p. 6
  • Narrative generation uses provider-default sampling settings rather than a fixed temperature, introducing sampling variance that cannot be fully characterized post hoc.Binary knowledge probes use T = 0 to reduce sampling variation.Source: §4 · p. 6

5 Results

Across nine communities, the results document recurring identity ossification and a disconnect between factual retrieval and narrative framing. The section also examines variation across model families and languages, checks cross-language semantic divergence, and tests whether the patterns generalize to additional models.

5 Results

  • Binary pattern outcomes are compared with chi-square or Fisher’s exact tests, and continuous narrative scores with reported group-mean tests; p-values are approximate, directional evidence.The p-values are approximate because responses to the same prompt across conditions are not fully independent.Source: §5 · p. 6

5.1 RQ1: Presence

  • 32.8% of primary-corpus responses (130/396) show at least one coded manifestation; rates range from 10.1% for GPT-5.1 English to 62.6% for DeepSeek-V3.2 Chinese.Rates vary by community from 14% for Jingpo/Kachin to 48% for Dulong/Rawang; the taxonomy-based 32.8% is a lower bound, not a definitive measure.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 RQ2: Mechanism

  • Probe-passed responses score 8.50 versus 8.35 for probe-failed responses (p = 0.696), so factual access does not predict narrative framing quality.Probe outcomes are imperfect, particularly for DeepSeek-V3.2, but 99.1% of stable probes were unchanged when Yes/No labels were reversed. The authors treat pattern-level analyses as descriptive and the rubric comparison as primary evidence.Source: §5.2 · p. 7

5.3 RQ3: Conditions

  • Model family shapes which failure modes appear, while Chinese queries elicit more ossification than English in both models; Political Substitution appears only in DeepSeek-V3.2.Political Substitution occurs 14 times across eight communities in DeepSeek-V3.2 and zero times in GPT-5.1; the model-family-by-language interaction is not significant (p = 0.812).Source: §5.3 · pp. 7–8

5.4 Cross-Language Embedding Validation

  • All four community-level correlations between cross-language embedding divergence and narrative scores are negative; English-condition correlations are significant for both models.Each correlation uses nine communities; the embedding measure captures any semantic difference, not framing specifically, so this is supporting rather than establishing evidence.Source: §5.4 · p. 8

5.5 Robustness Check

  • Across three additional models, the knowledge–framing dissociation and the primary model-family and language patterns recur, with P2 again confined to the China-origin model.Probe outcome again has no significant relationship with narrative score (8.45 vs. 8.79, p = 0.466); an independent, version-pinned GPT-4.1 judge scores the added-model responses.Source: §5.5 · p. 8

6 Conclusion

The paper defines identity ossification as compressing trans-border communities into nation-state categories and finds it varies across models and query languages. Factual accessibility does not positively predict narrative framing quality, motivating cultural evaluations that examine how models organize retrieved knowledge.

6 Conclusion

  • Identity ossification is the representational failure in which LLM narratives compress trans-border communities into nation-state categories.The framework is intended to distinguish what models can retrieve from how they organize that knowledge into narratives.Source: §6 · p. 8
  • Ossification ranged from 10.1% of GPT-5.1’s English responses to 62.6% of DeepSeek-V3.2’s Chinese responses across nine communities and five models.The conclusion also reports that factual accessibility does not positively predict narrative framing quality, a dissociation replicated across three additional models.Source: §6 · p. 8
  • Applying TIODF to the Sami, Rohingya, Kurds, or other trans-border populations could test whether the documented pattern extends beyond China’s southwestern border.The conclusion leaves open whether the pattern reflects that region’s administrative history or a more general property of LLM identity narratives.Source: §6 · p. 8

Limitations

The study identifies output-level ossification but cannot isolate its mechanisms, and several design and measurement constraints qualify how broadly or precisely to interpret its findings. These include the sampled communities’ administrative status, annotation and judging caveats, and the small model sample.

Mechanistic Explanations

  • API-based evaluation identifies output-level ossification but cannot distinguish whether corpus imbalance, administrative sources, post-training preferences, or alignment procedures contribute to it.The authors suggest comparing base and instruction-tuned models, retrieval-augmented settings, or documentary sources in future work.Source: Limitations · p. 9

Corpus Selection and Frame Availability

  • All nine communities are officially recognized among China’s 56 minzu, so Chinese-language prompts may offer a ready-made national frame rather than isolate a purely linguistic effect.Recognition status is a necessary design feature for comparing conditions, but it bounds interpretation of the language effect.Source: Limitations · p. 9

Measurement Caveats

  • The primary narrative judge’s unpinned version and temperature limit exact reproducibility, although community validation and a version-pinned robustness judge yielded consistent conclusions.Community annotation validation had ICC = 0.75; the robustness corpus used GPT-4.1 at temperature 0.Source: Limitations · p. 9
  • Responses across conditions were not strictly independent, multiple testing was uncorrected, and temperature-zero probes were not perfectly deterministic, so significance tests are indicative rather than exact guarantees.The probe instability was particularly notable for DeepSeek-V3.2.Source: Limitations · p. 9
  • Five models are enough to show ossification is not confined to one model, but remain a small sample for claims about the broader LLM landscape or general mechanisms.Three empty generations also left the robustness corpus with 591 of 594 planned responses and slightly unbalanced model-language conditions.Source: Limitations · p. 9

Community Membership and Knowledge Heterogeneity

  • Community membership alone did not ensure annotators knew cross-border facts, and qualified annotators may still miss omitted ties; Knowledge Cards limit but do not eliminate that risk.Human and LLM raters may also share an interpretive lens, so ICC = 0.75 indicates reliable tracking of human judgment, not freedom from shared bias.Source: Limitations · p. 9

Ethical Considerations

The section describes the study’s ethical safeguards and supplementary materials for its community sample, prompts, and evaluation pipeline. It directs readers to the tables that document community scope, prompt design, model settings, and narrative-framing criteria.

Ethical Considerations

  • The lead author’s regional experience informed community selection, Knowledge Cards, and prompts; external review of each card by community-tied or regional experts is described as a partial check.Annotators participated voluntarily with consent, pseudonyms replaced identifiers, and only aggregate results were reported; the protocol was IRB-exempt.Source: Ethical Considerations · p. 10

A Community and Prompt Materials

  • Supplementary materials introduce the nine communities, four prompt categories, and a Dai/Thai worked example, with complete cards and batteries released with the code and data.The section uses the Dai/Thai case to illustrate translating a Knowledge Card into an eleven-prompt bilingual battery.Source: A Community and Prompt Materials · p. 12

A.1 Community Overview

  • Table 4 summarizes where the nine communities are distributed and the linguistic, historical, or cultural ties motivating their treatment as trans-border communities.Its caption identifies the table as an overview of the included communities.Source: §A.1 · p. 12, Table 4 · p. 13

A.2 Prompt Categories

  • The eleven-prompt narrative battery is divided into four categories whose purposes and representative examples are summarized in Table 5.The categories move from basic knowledge to cross-border continuity, identity fluidity, and narrative framing.Source: §A.2 · p. 12, Table 5 · p. 13

A.3 Dai/Thai Worked Example

  • The Dai/Thai worked example links Knowledge Card dimensions to bilingual prompts, distinguishing national labels from a broader trans-border Tai continuum and shared historical, linguistic, and religious anchors.Table 6 condenses the reference dimensions; Table 7 presents the eleven prompts, which address the same topics but are naturalized rather than strict word-for-word translations.Source: §A.3 · p. 12, §A.3.1 · p. 12, Table 6 · p. 13

A.3.2 Bilingual Prompt Battery

  • The Dai/Thai prompt pairs are written naturally in English and Chinese rather than as strict word-for-word translations.The paired binary probes are included in the released study materials.Source: §A.3.2 · p. 14

B Experimental Pipeline and Model Specifications

  • The supplementary pipeline section covers model configurations, the narrative-framing rubric, and binary-probe robustness analyses.It directs readers to Table 8 for versions, temperatures, and access methods, distinguishing subject from judge models and fixed settings from defaults or unavailable metadata.Source: B Experimental Pipeline and Model… · p. 14, §B.1 · p. 14

B.2 Narrative Framing Rubric

  • Table 9 presents criteria for the four narrative-framing dimensions, with three score levels per dimension representing increasingly trans-border and community-centered framing.Source: §B.2 · p. 14

B.3 Binary-Probe Stability and Label-Order Robustness

Repeated binary probes were mostly stable, and reversing response-label order rarely changed unanimous answers. The section also points readers to pattern definitions and examples for interpreting recurring forms of ossification.

B.3 Binary-Probe Stability and Label-Order Robustness

  • The stability check repeated 396 model-language-probe combinations twice beyond the initial run, keeping endpoints, prompts, temperature, and token limits unchanged.A probe counted as stable only when all three normalized Yes/No answers matched; Unknown responses were retained but not counted as stable Yes/No answers.Source: §B.3 · p. 14
  • 331 of 396 probe combinations were stable across three runs (83.6%); GPT-EN was most stable at 98.0%, while DS-ZH was least stable at 71.7%.Table 10 reports stability and label-order agreement by model-language condition.Source: §B.3 · p. 14
  • Among 331 unanimous probes, 328 (99.1%) retained the same answer under reversed label order, with no statistically significant directional shift in any model-language condition.The three disagreements were two Yes-to-No changes in DS-EN and one No-to-Yes change in DS-ZH.Source: §B.3 · p. 14

C Pattern Definitions and Illustrative Examples

  • The following section defines five recurring ossification patterns and pairs their scope and defining features with observed response examples.Table 11 compares the patterns before the examples illustrate them.Source: C Pattern Definitions and Illustrative… · p. 14

C.1 Illustrative Examples

The illustrative examples show several ways responses frame cross-border identities, with quoted answers interpreted against their framing operation and scored out of 12. The accompanying materials describe the prompts, evaluation setup, pattern definitions, and pattern severity, while the later robustness section covers additional analyses.

C.1 Illustrative Examples

  • The Wa response describes the community within China and southwestern Yunnan but omits Myanmar’s Wa State.The De’ang example contrasts this by also locating De’ang communities in northern Myanmar.Source: §C.1 · pp. 14, 16
  • The accompanying materials include a Dai/Thai bilingual prompt battery, model and evaluation specifications, a four-dimension three-point coding rubric, and definitions and examples for five patterns.Source: Table 7 · p. 15, Table 8 · p. 15, Table 9 · p. 15
  • Table 10 reports probe stability across three original-order Yes/No runs and order agreement for stable probes under reversed order.Source: Table 10 · p. 16
  • The Jingpo response invokes China’s national family and ethnic classification but does not address whether someone can identify as both Chinese Jingpo and Wunpawng.The response also refers to the Jingpo as one of China’s 56 ethnic groups.Source: §C.1 · pp. 14, 16
  • A Dulong example frames identity through China’s ethnic identification and classifies the person as Dulong by ethnic component.Source: §C.1 · p. 16
  • The model describes the Kawaqa festival as a Dulong tradition rather than a festival shared with Rawang communities.Source: §C.1 · p. 16

C.2 Pattern Severity

  • Pattern severity compares narrative scores when each pattern is present versus absent; P1 and P3 co-occur frequently, so their differences are not independent or additive.Readers are directed to the score-difference and significance columns.Source: §C.2 · p. 16, Table 12 · p. 17

D Robustness Corpus and Embedding Results

D.1 Robustness Corpus Design

In the robustness corpus, probe success was not significantly related to narrative scores, while language and model-family patterns broadly echoed the primary analysis. The corpus applied the full TIODF pipeline to three additional models, with a few excluded empty outputs.

D.1 Robustness Corpus Design

  • The robustness corpus applied the full TIODF pipeline to Gemini-3.1-Pro, Qwen3.6-Max, and Claude Sonnet 4.6; after excluding three empty outputs, it contained 591 of 594 planned responses.Narrative responses were scored by pinned-version GPT-4.1 at temperature 0, and no model scored its own output.Source: §D.1 · p. 17

D.2 Knowledge–Framing Dissociation

  • Probe-passed responses averaged 8.45 and probe-failed responses 8.79 (p = 0.466), showing no significant relationship with narrative score.The null also held within each model and model–language condition, though subgroup tests had lower statistical power.Source: §D.2 · p. 17

D.3 Model-Family and Language Effects

  • Chinese-language queries elicited more ossification than English in all three robustness models; P2 appeared only in Qwen3.6-Max, with 9 instances and none in Gemini-3.1-Pro or Claude Sonnet 4.6.This replicates the language effect and the association of P2 with China-origin models across a second, disjoint system set.Source: §D.3 · p. 17

D.4 Probe Outcomes in the Robustness Corpus

  • Only Lahu B3 failed in all six robustness conditions as well as all four primary conditions, making it the sole candidate for a gap general to LLM training data.Jingpo/Kachin D1 and De’ang D2 failed only sporadically across the added models, suggesting primary-model limitations rather than documentary-record absences.Source: §D.4 · p. 17

D.5 Embedding-Based Results

The embedding-based validation presents cross-language semantic divergence by community and model, alongside plots relating divergence to narrative scores. The supplied descriptions identify the comparisons and plot encodings but do not state the outcomes.

D.5 Embedding-Based Results

  • Figure 6 plots cross-language semantic divergence against narrative scores across nine communities and four model–language conditions, with 95% confidence intervals; inverted score axes place stronger ossification higher.Compare whether greater divergence accompanies stronger narrative ossification.Source: §D.5 · p. 17, Figure 6 · p. 18
  • Figure 5 presents cross-language semantic divergence by community and model, allowing comparison across those groups.Source: §D.5 · p. 17, Figure 5 · p. 18
Loading 2610.12610v1…