Source-linked AI summary
Quantitative Measurement of Language Distance among Closely Related Indo-European Languages Using Pretrained Language Models: A Case Study on the North Germanic Branch
Yiping Bai
TL;DR
Traditional qualitative approaches make cross-branch language-distance comparisons difficult, and written distance among closely related North Germanic languages remains insufficiently measured. The paper applies three pretrained-model metrics to parallel Danish, Norwegian Bokmål, and Swedish texts; the measures support the conclusion that Danish–Norwegian written distance is smallest. The authors present the framework as a reproducible foundation for future study, while noting its limited sample and language-branch scope.
Problem
Traditional qualitative methods use differing comparison criteria, and systematic pretrained-model measurement of written distance among closely related languages is limited.
Source: §1 · p. 1Method
The paper applies semantic sentence-vector distance, orthographic fragmentation, and MLM predictability to the same 150 parallel triplets, with two models used for cross-validation.
Source: §1 · p. 2, §4 · p. 4, §4.1 · p. 4Results
The three metrics consistently identify Danish–Norwegian written distance as smallest, and LaBSE and mBERT yield consistent distance rankings.
Source: §6.1 · p. 6, §7 · p. 8Takeaways & Limitations
The historically established North Germanic branch serves as a calibration target for a reproducible framework intended for future extension to less-studied language groups.
Source: §7 · pp. 8–9The study uses only 150 triplets from one language branch, and MLM confidence is influenced by training-corpus distribution.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Among closely related North Germanic languages, the quantification of language distance has traditionally relied on qualitative methods, lacking a unified multi-dimensional computational framework. Multilingual pretrained models based on the Transformer architecture can map texts from different languages into a shared vector space, enabling quantitative measurement of language distance. This paper focuses on the three North Germanic languages---Danish, Norwegian (Bokm\aa{}l), and Swedish---and proposes a three-metric quantitative framework based on pretrained language models: (1)~sentence-level semantic distance, computed as cosine similarity between LaBSE and mBERT encodings of parallel sentences; (2)~orthographic fragmentation rate, measuring subword tokenization efficiency when cross-applying monolingual BERT vocabularies to parallel texts; (3)~MLM predictability, comparing prediction confidence and entropy in masked language modeling using mBERT across languages. Using 150 trilingual parallel sentence triplets from the Tatoeba corpus as controlled samples, we obtain consistent distance rankings on two independent models: LaBSE: da--no $0.012 < $ no--sv $0.016 < $ da--sv $0.020$; mBERT: da--no $0.016 < $ no--sv $0.045 \approx $ da--sv $0.046$. This ranking is consistent with the historical linguistic conclusion that ``400 years of Danish rule over Norway (1380--1814) led to highly cognate written languages.'' The three metrics---semantic, orthographic, and predictability---converge on the same conclusion, providing a reproducible computational framework for the quantitative study of distance among closely related languages, extensible in principle to more branches of the Indo-European language family, pending validation on additional language groups.
1 Introduction
Quantitative comparison of language distance is difficult because traditional studies rely on qualitative methods and historical accounts often lack measurements. This paper examines written distance among Danish, Norwegian Bokmål, and Swedish using three pretrained-model metrics.
1 Introduction
- Traditional qualitative approaches use differing word lists and criteria, making cross-branch comparison difficult, while historical narratives often lack quantitative measurements that make conclusions falsifiable.The paper presents multilingual pretrained models and parallel corpora as making controlled cross-lingual distance measurement possible [4] [17].Source: §1 · p. 1
- The three mainland Scandinavian languages descend from Old Norse and have long exhibited “semicommunication”.Danish rule over Norway from 1380 to 1814 made Danish the written language there; Bokmål later developed on a Danish substrate.Source: §1 · pp. 1–2
- Whether written language distance among the three languages follows the patterns found in spoken mutual-intelligibility research remains systematically unverified.Prior spoken-language work found phonetic distance explained 64% of intelligibility variance, whereas lexical distance did not reach significance.Source: §1 · p. 2
- The paper proposes semantic, orthographic, and predictability metrics for written distance, cross-validating results on two models and comparing them with historical conclusions and spoken-intelligibility data.Source: §1 · p. 2
2 Related Work
Prior work examined Scandinavian mutual intelligibility mainly in speech, while fine-grained written-distance ranking with pretrained models remains underexplored. The paper asks whether a three-metric framework can quantify written relationships and complement spoken-language evidence.
2.1 Mutual Intelligibility among North Germanic Languages
- Spoken studies found asymmetric intelligibility among the Scandinavian languages, but systematic pretrained-model quantification of written distance alongside spoken evidence remains rare.Earlier work linked intelligibility to lexical similarity, phonological distance, and speech rate.Source: §2.1 · p. 2
2.2 Pretrained Language Models and Language Distance
- Multilingual representations recover broad language-family structure, but fine-grained ranking within a branch and cross-applying monolingual BERT vocabularies to measure orthographic distance remain underexplored.MLM confidence is also affected by corpus distribution, vocabulary coverage, and morphological complexity, so it requires caution as a distance measure.Source: §2.2 · p. 2
2.3 Research Questions
- The research questions test model-ranking consistency, orthographic corroboration, and comparison with spoken mutual intelligibility, using North Germanic as a calibration target for a reproducible framework.Source: §2.3 · p. 3
3 Data
The study uses a random sample of 150 Danish–Norwegian Bokmål–Swedish parallel sentence triplets from Tatoeba. Data preparation was local, while model inference used Colab resources.
3.1 Parallel Sentence Triplet Construction
- Filtering Danish-anchored Tatoeba links for non-empty sentences yielded 456 trilingual triplets, from which 150 were sampled with random seed 42 [17].The corpus contains text only; Danish, Bokmål, and Swedish contributed 66,455, 18,858, and 56,768 sentences, respectively.Source: §3.1 · p. 3
- Table 1 presents five sample parallel sentence triplets.Source: §3.1 · p. 3, Table 1 · p. 3
- One sample phenomenon is identical written forms in Danish and Norwegian; another is Norwegian sharing Swedish “Helsingfors” where Danish uses “Helsinki.”The text attributes the latter to historical Swedish influence on Norwegian toponymic conventions.Source: §3.1 · p. 3
3.2 Experimental Environment
- Data preparation ran on a local Windows 11 machine, while model inference used Google Colab resources.Table 2 is identified as covering computing resources and model configuration.Source: §3.2 · p. 3, Table 2 · p. 4
4 Methods
The methods measure written language distance along semantic, orthographic, and predictability dimensions using the same 150 parallel triplets. Sentence-vector and tokenizer comparisons yield pairwise distances or corroboration, while MLM supplies predictability measures.
4 Methods
- All three experiments use the same 150 trilingual parallel sentences to hold semantic content constant while measuring linguistic differences.Table 3 summarizes the experimental design.Source: §4 · p. 4, Table 3 · p. 4
4.1 Metric 1: Sentence Vector Semantic Distance
- Sentence-vector distance uses cosine similarity between encodings of parallel versions, with LaBSE distance defined as one minus the median cosine similarity.LaBSE produces 768-dimensional normalized vectors; mBERT provides an independent comparison using normalized [CLS] vectors, and the models differ in architecture and training objective.Source: §4.1 · p. 4
4.2 Metric 2: Orthographic Fragmentation Rate
- Orthographic fragmentation measures average subword tokens per whitespace-delimited word when a monolingual BERT vocabulary segments another language’s text.The experiment computes a median fragmentation rate for each language using all three vocabularies, producing a 3 × 3 matrix.Source: §4.2 · p. 4
4.3 Metric 3: MLM Predictability
- The MLM metric masks alphabetic tokens of length at least three and records prediction confidence, entropy, top-1 accuracy, and top-10 recall.These predictability differences are an exploratory proxy for mutual intelligibility and may reflect corpus coverage and contextual transparency.Source: §4.3 · p. 5
5 Results
Sentence-vector distances rank Danish–Norwegian as closest and Danish–Swedish as most distant across both models. Orthographic fragmentation and MLM predictability add comparisons across vocabularies and languages, with vocabulary size limiting interpretation of absolute fragmentation rates.
5.1 Sentence Vector Semantic Distance
- Both models rank da–no closest, no–sv next, and da–sv most distant; LaBSE separates three levels, while mBERT separates two tiers.LaBSE similarities are 0.988 > 0.984 > 0.980, while mBERT gives 0.984 ≫ 0.955 ≈ 0.954.Source: Table 4 · p. 5, §5.1 · p. 5
5.2 Orthographic Fragmentation Rate
- Danish text segments more efficiently with the Norwegian vocabulary (1.5000) than with the Swedish vocabulary (1.7071), while Swedish text has similar rates with Norwegian (1.5505) and Danish (1.5000) vocabularies.Vocabulary sizes differ substantially (da 253K, no 996K, sv 399K), confounding absolute rates; comparisons should focus on relative increases for the same text across vocabularies.Source: Table 6 · p. 5, §5.2 · p. 6
5.3 MLM Predictability
- Danish has the highest mBERT top-1 confidence (0.2581) and lowest entropy (6.0705 bit), while top-10 recall varies little across languages (55.25%–56.15%).The reported difference lies mainly in the concentration of the prediction distribution.Source: Table 7 · p. 6, §5.3 · p. 6
6 Discussion
Across the three metrics, Danish–Norwegian written distance is smallest, consistent with historical accounts of their shared written tradition. The discussion compares these measurements with spoken-language findings, examines model differences, and considers calibration and applications.
6.1 Consistency across Three Metrics
- Danish–Norwegian written distance is smallest across semantic, orthographic, and predictability measures, though MLM gives monolingual rankings rather than pairwise distances.The three metrics converge on the same conclusion, but MLM confidence may partly reflect differences in training-corpus coverage.Source: §6.1 · pp. 6–7, Table 8 · p. 7
6.2 Comparison with Historical Linguistics
- LaBSE 0.012 and mBERT 0.016 place Danish–Norwegian closest; fragmentation is 1.500 for Danish text under Norwegian vocabulary versus 1.707.The paper relates this pattern to Denmark’s rule over Norway from 1380–1814 and the development of Bokmål on a Danish substrate.Source: §6.2 · p. 7
6.3 Comparison with Spoken Mutual Intelligibility
- Phonetic and sentence-vector distances share rankings, but spoken intelligibility is asymmetric (53% versus 24%) and reflects different modality factors.The sentence-vector measure is symmetric and therefore cannot represent the directional difference in spoken understanding.Source: §6.3 · pp. 7–8, Table 9 · p. 7
6.4 Model Sensitivity
- LaBSE’s 0.980–0.988 similarities distinguish three levels; mBERT’s 0.984 versus approximately 0.955 yields two tiers, with Norwegian–Swedish and Danish–Swedish nearly tied.LaBSE uses translation pairs to optimize cross-lingual alignment, while mBERT’s alignment is a byproduct of MLM training; the paper favors LaBSE for fine-grained discrimination and mBERT for MLM predictability.Source: §6.4 · p. 8
6.5 Significance and Applications
- The historically well-studied North Germanic branch serves as a calibration target before the framework is applied to less-studied language groups.The paper frames pretrained-model measurements as a reproducible, scalable complement to manual word-list comparisons and human perception experiments.Source: §6.5 · p. 8
- Measured distances suggest possible uses in cross-lingual NLP migration, translation-memory cost estimation, and language-learning comparisons.The paper gives Danish-to-Norwegian versus Danish-to-Swedish transfer and Swedish learners’ transition to Norwegian versus Danish as examples.Source: §6.5 · p. 8
7 Conclusion
The three-metric framework consistently ranks Danish–Norwegian written distance as smallest, in line with historical accounts, and its results complement spoken-intelligibility data. The North Germanic case provides a calibration point, while the authors identify sample size, language coverage, and MLM corpus effects as limitations.
7 Conclusion
- Sentence-vector distance, fragmentation rate, and MLM predictability all rank Danish–Norwegian written distance as smallest, consistent with historical accounts; LaBSE and mBERT rankings also agree.The written-distance results complement Gooskens’s spoken mutual-intelligibility data.Source: §7 · p. 8
- The North Germanic case validates the framework as a calibration target and supports a reproducible foundation for future work on less-studied closely related language groups.The authors describe this paper as a starting point for further computational studies.Source: §7 · pp. 8–9
- Current scope is limited to 150 triplets and one language branch, and MLM confidence may be influenced by training-corpus distribution.Future work proposes broader language coverage, additional models and metrics, and speech data.Source: §7 · p. 9