Source-linked AI summary
Convergence in Science, Divergence in Religion: Calibrated Framing Differences Across Wikipedia's Language Editions
Hung-Hsuan Chen
TL;DR
The paper asks how matched Wikipedia articles frame the same concepts across independently written language editions, beyond prior measures of coverage gaps. It introduces a per-language-pair calibrated distance to remove encoder-alignment baselines and finds robust domain and concept-level divergence patterns across three multilingual encoders. Religion ranks highest and science/technology lowest, while political divergence concentrates in sensitive concepts rather than politics as a whole.
Problem
Prior cross-lingual Wikipedia work mainly measures coverage gaps, leaving framing differences between matched concepts less directly measured.
Method
The study compares multilingual article embeddings for 150 Wikidata-anchored concepts across 20 languages and subtracts each pair’s mean distance on stable-denotation calibration concepts.
Results
Religion ranks first and science/technology last across LaBSE, multilingual MPNet, and CMLM; concept rankings correlate at Spearman ρ = 0.75–0.79 relative to LaBSE.
Takeaways & Limitations
Calibrated distance supports cross-domain, cross-concept, and cross-language-pair comparisons without treating encoder artifacts as cultural difference.
Takeaways & Limitations
The metric is not validated against human ratings, and language editions should not be interpreted as direct national or cultural perspectives.
Abstract
from arXiv · showhide
When Wikipedia's language editions describe the same concept, how differently do they frame it? Prior work measures coverage gaps between editions; we measure framing distance for matched concepts. We analyze 2,799 valid articles from 3,000 possible concept-language observations, spanning 150 Wikidata-anchored concepts, 20 language editions, 4 domains, and a calibration set. Raw embedding distances reflect both content differences and how well the encoder aligns each language pair. Even among calibration concepts with stable cross-cultural denotations (e.g., chemical elements, numbers, colors), the largest language-pair mean distance is 3.6 times the smallest, and distances are typically smaller within language families. We define a baseline-adjusted distance (calibrated distance): the distance between two language versions of a concept minus the mean distance for calibration concepts in the same language pair. This adjustment substantially reduces pair-specific alignment differences and the language-family pattern. Across three multilingual encoders (LaBSE, multilingual MPNet, and CMLM), scientific articles align more closely than calibration articles, and all three rank religion first and science/technology last. Concept-level rankings are highly consistent across encoders (Spearman rho=0.75-0.79 for MPNet and CMLM relative to LaBSE). Religion lies significantly above the calibration baseline under LaBSE. Within politics, divergence concentrates on concepts such as censorship and refugee, while democracy and human rights are among the most aligned. Code, data, and per-language-pair calibration baselines are released.\footnote{https://github.com/hhchen1105/cross-linqual-concept}
1 Introduction
The paper measures how matched Wikipedia articles frame shared concepts across language editions, rather than merely comparing coverage. It develops calibration to separate content divergence from language-pair encoder effects and finds the greatest divergence in religion and politically sensitive concepts.
- Motivation: Wikipedia’s independently written language editions provide a corpus for comparing how communities frame the same concepts.The study asks whether framing differs more for culturally loaded topics than for settled scientific knowledge.
- Motivation: 3.6× variation in calibration-article distances across language pairs shows that raw embedding distances include encoder-alignment and edition-writing effects.Calibration means range from 0.107 for Persian–Indonesian to 0.389 for Hindi–Chinese, and raw distances are typically smaller within language families.
- Method: Per-pair calibration subtracts each language pair’s mean distance on stable-denotation concepts, substantially reducing language-family and alignment effects.The resulting calibrated distance is designed to compare framing differences after accounting for pair-specific baselines.
- Findings: 2,799 articles across 20 languages reveal a domain ordering in which religion diverges most and science/technology aligns most closely.Under LaBSE, religion is above the calibration baseline while science is below it; two additional encoders reproduce the ordering.
- Findings: Political divergence concentrates in specific concepts such as censorship, whereas democracy is described similarly across editions.The results distinguish sensitive concepts from politics as a whole domain.
2 Related Work
The paper distinguishes framing divergence from prior work on cross-lingual Wikipedia coverage and from research on culturally varying language-model outputs. It also uses an encoder-agnostic calibration approach to address language-pair alignment differences.
- Coverage versus framing: Prior Wikipedia comparisons mainly measure which facts or articles editions include, while this study measures how matched article content is framed.Coverage and framing are complementary: editions can emphasize different facts while maintaining similar framing, or share facts with different framing.
- Coverage versus framing: InfoGap uses LaBSE retrieval followed by LLM verification, whereas this study’s continuous distance measure directly inherits encoder alignment quality.That distinction motivates calibration in the present method.
- Model alignment: Research on multilingual-model cultural alignment studies model outputs, while this paper measures divergence in human-written Wikipedia content.The authors state that the claims do not overlap because they address a data-side question.
- Multilingual encoders: LaBSE aligns sentence embeddings across 109 languages, but training-pair availability produces uneven alignment quality across language pairs.The calibration method is encoder-agnostic and can be recomputed for other multilingual encoders.
3 Data
The dataset covers 150 Wikidata-anchored concepts across 20 language editions and four domains, with a separate stable-denotation calibration set. After filtering unavailable, disambiguation, and stub pages, 2,799 of 3,000 observations remain valid.
- Concepts: 150 concepts are divided across religion, politics, science/tech, and pop culture, with 30 minimally culture-loaded calibration concepts.Calibration concepts include chemical elements, small numbers, colors, common animals, and universal natural kinds.
- Concepts: Each concept is anchored by a Wikidata QID that maps it to article titles across language editions.Manual overrides corrected five disambiguation or redirect errors, and all 150 concepts resolved successfully.
- Languages: 20 languages span 15 families and nine scripts, selected to maximize diversity while keeping the 3,000-article pipeline tractable.Every selected language is supported by LaBSE, yielding 190 language pairs.
- Article collection: Lead sections are retrieved as plain text because their standardized length and structure reduce comparability confounds.Observations are excluded when no sitelink exists, the page is a disambiguation page, or the article is a stub.
- Article collection: 2,799 of 3,000 concept–language observations are valid, including 154 without sitelinks and 47 stubs, with zero fetch errors.Swahili has 71% valid observations, while every other edition reaches at least 89%.
4 Method
The method embeds comparable article leads, computes raw cosine distances between language versions, and subtracts a language-pair calibration floor. It then tests domain effects and ranking stability while accounting for dependence among concepts and language pairs.
- 4.1 Embeddings: Each valid lead is sentence-embedded, L2-normalized, mean-pooled, and L2-normalized again to produce one 768-dimensional concept–language vector.The procedure gives each sentence equal contribution regardless of vector magnitude.
- 4.2 Calibrated distance: Raw distance is d(c, i, j) = 1 − cos(v_c,i, v_c,j) between two language versions of concept c.Cosine similarity is computed from the pooled vectors for the two editions.
- 4.2 Calibrated distance: Calibrated distance subtracts the mean calibration-concept distance for the same language pair from each raw distance.The calibration floor uses minimally culture-loaded concepts available in both languages, averaging 27.5 of 30 concepts per pair.
- 4.2 Calibrated distance: Positive calibrated distance means greater difference than the calibration concepts, while negative values mean greater similarity.The baseline mainly captures encoder alignment and edition-level writing differences, though some content variation may remain.
- 4.3 Statistical validation: Five checks assess domain effects and ranking sensitivity, including concept and language bootstraps, permutation tests, and Mantel tests.The permutation design recomputes each language-pair baseline and avoids treating overlapping language pairs as independent replicates.
- 4.3 Statistical validation: Concept bootstraps resample the 30 concepts per group, so their intervals measure sensitivity to purposive concept composition rather than population sampling uncertainty.Language-node and concept-node bootstraps separately assess the stability of concept and language rankings.
- 4.4 Divergence-profile geometry: Because calibrated distances can be negative, each language is represented by its mean-distance profile rather than directly using the pairwise matrix as an MDS dissimilarity matrix.The profiles are 20-dimensional, and bootstrap projections assess geometry stability.
5 Results
After calibration, religion shows the strongest domain-level divergence while science/technology aligns more closely than calibration articles; language-pair structure remains partly shared across domains, but low-dimensional religious geometry is unstable.
- Domain-level divergence: +0.034: Religion lies significantly above the calibration baseline under LaBSE, while science/technology lies below it at −0.024.Politics and pop culture do not differ significantly from the baseline.
- Language structure: 79.9%, 74.4%, 81.5%, and 59.2%: Korean, Polish, Persian, and Turkish remain in religion’s top four across bootstrap draws, while Japanese and Chinese remain in the bottom four in 100.0% and 96.3%.The aligned end is more stable than the exact order of the divergent cluster.
- Language structure: 68.6% variance and stress 0.269: two-dimensional MDS provides only a moderate-to-poor fit for religion’s language profiles.The analysis therefore treats the projection as diagnostic rather than as a reliable map of language groupings.
- Concept-level structure: Sacrifice and censorship remain in the top ten in 99.0% and 99.3% of bootstrap draws, whereas pilgrimage and scripture enter it in only 45.5% and 37.7%.The authors interpret broad concept tiers rather than exact rankings; democracy and human rights are among the most aligned political concepts.
- Cross-domain structure: All six cross-domain Mantel correlations are positive, ranging from Spearman ρ = 0.36–0.74, with four significant after Holm correction.These correlations indicate shared relative language-pair structure, not necessarily similar absolute distances.
6 Robustness
Robustness checks show that the paper’s domain ordering persists across encoders and calibration choices, while the religion effect depends on the encoder. Audits also indicate that flagged bot- or translation-mediated articles do not drive the main pattern.
- Encoder robustness: All three encoders rank religion first and science/technology last, with concept rankings correlating with LaBSE at Spearman ρ = 0.75 for MPNet and 0.79 for CMLM.The relative ordering replicates, although effect magnitudes and religion’s significance are encoder-dependent.
- Encoder robustness: Religion’s mean calibrated distance falls from +0.034 with LaBSE to +0.010 with MPNet and +0.004 with CMLM, and neither replication encoder yields a significant domain-level test.The adjusted p-values are .37 and .76 for the replication encoders.
- Encoder robustness: Science/technology remains below the calibration baseline under all three encoders, with adjusted p-values of .030, .006, and < .001.This contrasts with the encoder-dependent significance of the positive religion effect.
- Bot-created and translation-generated articles: 13.3% of audited articles are flagged, but excluding them changes mean religion calibrated distance only from +0.034 to +0.032, and no domain changes sign.The audit is exhaustive for five editions and samples 15 concepts elsewhere, so it is a subset robustness check rather than a corpus-wide exclusion analysis.
- Calibration-set composition: Leaving out each calibration subdomain changes mean baselines by at most 0.011 and preserves the domain ordering.Religion remains above the baseline in every omission, though the leave-one-out test cannot detect bias shared by all five subdomains.
- Calibration-set composition: Wikipedia and parallel-text floors have similar pairwise structure, with Spearman ρ = 0.79–0.82, supporting the calibration baselines’ alignment interpretation.Their average levels differ—0.215 for Wikipedia articles versus 0.062 for pooled parallel text—because independently authored calibration articles also vary in wording, emphasis, style, and possible cultural content.
7 Discussion and Limitations
The paper argues that calibrated distance helps compare framing across language pairs while preserving important limitations about interpretation, scope, and measurement reliability. It identifies implications for multilingual models and proposes several directions for future validation and comparison.
- Discussion and Limitations: Calibrated distance measures relative excess divergence over same-pair calibration articles, not absolute cultural difference or ground truth.It also cannot distinguish disagreement about meaning from different emphasis on facets of the same concept.
- Implications: The results can help separate disagreement already present in encyclopedic data from disagreement introduced by multilingual models.The paper locates divergence especially in ritual, institutional religious, and specific politically sensitive concepts rather than politics or science as whole domains.
- Discussion and Limitations: Wikipedia editions are not equivalent to nations or cultures, and the Chinese-language result reflects a transnational editor community rather than a PRC-based perspective.The paper also notes that only lead sections are analyzed, so full articles may differ.
- Discussion and Limitations: Religion’s significant positive effect appears only under LaBSE, and calibrated-distance magnitudes should not be transferred across encoders.Calibration floors also absorb edition style, residual cultural content in neutral concepts, and encoder noise.
- Future Work: Future work could validate calibrated distance against human ratings, compare same-script editions and differently governed platforms, and track framing across revision histories.These directions target metric validation, editorial-setting comparisons, and temporal change.
8 Conclusion
The paper introduces calibrated distance to measure framing differences between matched Wikipedia concepts while controlling for language-pair encoder variation. Across the study, religion is most divergent and science/technology least divergent, with political divergence concentrated in particular sensitive concepts; released resources support further measurement.
- Conclusion: Calibrated distance subtracts a language-pair calibration baseline from cross-lingual concept distance.The study covers 150 concepts, 20 languages, four domains, and a calibration set.
- Conclusion: Across encoders, religion ranks highest and science/technology lowest, while politics shows concentrated divergence in specific sensitive concepts rather than across the whole domain.The conclusion reports this ordering under LaBSE and its preservation by two additional encoders.
- Conclusion: Two additional encoders preserve the domain ordering but not the significant positive religion effect.This separates robust ranking structure from encoder-specific significance.
- Conclusion: Released code, data, and calibration baselines enable future framing comparisons without mistaking encoder artifacts for cultural differences.Coverage and framing are presented as complementary views of cross-lingual knowledge inequality.
A Appendix: Concept Inventory
The appendix documents the study’s 150-concept inventory and explains how ranked concepts are evaluated for stability across bootstrap draws. It cautions readers to interpret broad divergence tiers rather than fixed adjacent ranks.
- Concept Inventory: The inventory contains 150 concepts organized by domain using English Wikipedia titles and anchored by Wikidata QIDs in the released data.The calibration set is separated into five subdomains for leave-one-out analysis.
- Rank Stability: Table 6 evaluates the ten highest- and ten lowest-ranked concepts across 5,000 language-node bootstrap draws.Rank 1 is most divergent and rank 120 most aligned; tail inclusion records how often concepts remain in the corresponding top or bottom ten.
- Rank Stability: Wide bootstrap intervals mean adjacent point-estimate ranks should not be interpreted as fixed.The appendix therefore emphasizes broad tiers rather than exact ordering.