There is No Theoretical Curse of Multilinguality For Embedding Space Structure

Niyati Bafna; Neha Verma; Vilém Zouhar; Philipp Koehn; David Yarowsky

Source-linked AI summary

There is No Theoretical Curse of Multilinguality For Embedding Space Structure

Niyati Bafna, Neha Verma, Vilém Zouhar, Philipp Koehn, David Yarowsky

arXiv:2608.17088v1Aug 17, 2026cs.CL

TL;DR

Multilingual models often degrade as language coverage expands, raising the question of whether embedding spaces face an inherent capacity limit. The paper formalizes perfect multilinguality and proves that the required dimensionality grows only logarithmically with the number of languages, while empirical degradation depends on data and training conditions.

Problem
The paper asks whether fixed-capacity embedding spaces inherently prevent perfect multilinguality as language coverage increases.
Method
The paper formalizes perfect multilinguality through shared monolingual semantics and cross-lingual alignment, then analyzes the dimensionality required for these conditions.
Results
The minimum dimensionality of a perfect multilingual space encoding L languages grows as Θ(log L), with sufficient dimensionality D_XL = D_Z + O(log L).
Takeaways & Limitations
Empirical cursed behavior depends on data, compute, sampling conditions, and evaluated languages, and is mitigated under favorable conditions.
Takeaways & Limitations
The theory assumes a global concept set with shared semantics, whereas language-specific concepts and relationships may require additional dimensionality.

Abstract

from arXiv · show

A central goal of multilingual NLP is to achieve high monolingual performance per language and cross-lingual alignment for large-scale language coverage with a multilingual model. The curse of multilinguality describes the phenomenon of degradation in multilingual model performance as we increase language coverage, posing a threat to the above goal. This paper asks whether multilingual embedding spaces are inherently incapable of achieving perfect multilinguality without a prohibitive increase in required capacity. We first formalize the goal of "perfect multilinguality", embodied in two multilinguality conditions. We then prove that the minimum dimensionality required for perfect multilinguality grows only logarithmically in the number of languages. That is, we show that there is no theoretical curse of multilinguality for embedding space structure. This suggests that the empirical curse of multilinguality is a result of real world data and training conditions. We back this understanding with a small-scale empirical study. Our paper provides the first theoretical and intrinsic perspective on the curse of multilinguality, with implications for the scientific understanding of this phenomenon.

1 INTRODUCTION

The paper formalizes perfect multilinguality through shared monolingual semantics and cross-lingual alignment, then shows its required embedding dimensionality grows logarithmically with language count. It argues that empirical degradation instead reflects real-world training conditions and evaluates this through controlled Transformer-based experiments.

Formalization: Perfect multilinguality requires high-quality shared monolingual semantics for each language and cross-lingual alignment of translation equivalents.These are formulated as two intrinsic multilinguality conditions.
Theoretical result: Θ(log L) dimensionality suffices and is necessary for a perfect multilingual space encoding L languages.The minimum sufficient extra dimensionality is O(log L), while any such space requires at least Ω(log L).
Theoretical result: The logarithmic capacity cost shows multilinguality is not entirely free, despite remaining theoretically achievable at arbitrary language scale.The result assumes a base D-dimensional concept space and adds capacity depending on L.
Empirical study: Empirical degradation in multilingual embedding structure is attributed to real-world conditions such as data and compute constraints.The study characterizes empirical COM-EMB by measuring degradation of the multilinguality conditions as language coverage increases.
Empirical study: The controlled study uses Transformer-based models across eight configurations varying compute constraints, sampling conditions, and language sets.It examines empirical degradation of the multilinguality conditions with increasing language coverage.

2 BACKGROUND

The curse of multilinguality refers to observed performance degradation as training languages increase for a fixed-capacity model, while prior work has primarily studied it empirically. This paper introduces a theoretical lens and emphasizes intrinsic analysis of multilingual embedding-space structure alongside prior extrinsic evaluations.

Curse of multilinguality: The curse of multilinguality is observed degradation in multilingual model performance as training languages increase for a fixed-capacity model.It is commonly described as capacity dilution across languages, although multilinguality can benefit low-resource languages through cross-lingual transfer.
Curse of multilinguality: Mitigation strategies include improved data sampling, mixture-of-experts architectures, and optimization techniques such as Gradient Vaccine.These approaches target the phenomenon through data, architecture, and optimization changes.
Curse of multilinguality: Prior studies explore the curse of multilinguality from a purely empirical lens, whereas this paper provides a theoretical lens.The paper presents this theoretical perspective as novel to the authors’ knowledge.
Intrinsic characterization of embedding spaces: Earlier work assessed embedding-space quality indirectly through extrinsic tasks such as XNLI and machine translation.Broader research also investigates multilingual-space structure, including cross-lingual overlap and alignment.

3 MULTILINGUALITY CONDITIONS FOR PERFECT MULTILINGUAL SPACES

This section defines perfect multilinguality relative to a high-quality reference concept space through monolingual structure and cross-lingual alignment conditions. It also motivates a non-degeneracy requirement to exclude trivially coincident translation embeddings.

Preliminaries: Perfect multilinguality is formulated relative to a reference high-quality monolingual concept space.The reference space supplies the semantic relationships that multilingual embeddings must preserve.
Monolingual structure: The monolingual structure condition requires every language subspace to have identical similarity rankings to the reference concept space.This preserves high-quality and shared semantics across languages, using relative similarity judgments between concept pairs.
Cross-lingual alignment: The cross-lingual alignment condition requires each concept encoding to be closer to its translation equivalent than to any different concept in any language.Formally, for distinct concepts c and c′, cos(x_c,ℓ, x_c,m) > cos(x_c,ℓ, x_c′,n).
Reference space: In practice, a high-resource language such as English can serve as the reference space, making monolingual structure require parity in monolingual quality and shared semantics.The paper notes that perfect concept relationships are generally unavailable.
Non-degeneracy: Non-degeneracy rules out placing all translation equivalents at one coincident point, which would trivially satisfy both multilinguality conditions.Such collapse conflicts with the practical assumption that embeddings distinguish points at a fixed resolution scale.

4 THEORETICAL CURSE OF MULTILINGUALITY FOR EMBEDDING SPACE STRUCTURE

The minimum dimensionality of a perfect multilingual embedding space encoding L languages grows as Θ(log L), not linearly or exponentially. A constructive space achieves the multilinguality conditions with DZ + O(log L) dimensions, while any such space requires Ω(log L) dimensions.

Theoretical result: Θ(log L) is the minimum dimensionality growth for a perfect multilingual space encoding L languages.This rules out a theoretical curse of multilinguality for embedding-space structure under the paper’s formulation.
Upper bound: DZ + O(log L) dimensions suffice to construct a perfect multilingual space XL encoding L languages.The construction combines a fixed concept space with a language-offset space whose dimension is O(log L).
Lower bound: Ω(log L) dimensions are necessary for any perfect multilingual space XL encoding L languages.Because DZ is independent of L, the upper and lower bounds establish the Θ(log L) result.
Construction properties: The constructed space satisfies monolingual structure, cross-lingual alignment, and non-degeneracy conditions.Its cosine scale remains fixed independently of the number of encoded languages.
Practical implication: 7000 language offset unit vectors can theoretically fit in 30 dimensions while satisfying the construction’s similarity constraints.The result uses a realistic choice of γ and is established through a random construction.
Multilinguality tax: Ω(log L) additional dimensions constitute a multilinguality tax when the base concept space fully requires DZ dimensions.This interpretation assumes the original monolingual concept space is minimal for representing concept semantics.

5 EMPIRICAL CURSE OF MULTILINGUALITY FOR EMBEDDING SPACES

A controlled study finds an empirical curse of multilinguality for embedding-space structure in four of eight configurations, while the other configurations remain stable or improve when target-language token counts stay high. The section evaluates monolingual structure, cross-lingual alignment, and non-degeneracy under systematically varied compute, sampling, and aggregation choices.

Configurations: The study covers eight configurations combining fixed or increasing compute, uniform or realistic sampling, and aggregate-all or aggregate-fixed evaluation.These configurations span all combinations of the three experimental dimensions.
Metrics: Monolingual structure is measured with MS-NNO, reported as percentage overlap at k = 20, alongside language-mean token-normalized masked-language-modeling loss.MS-NNO compares each target-language neighborhood with a strong monolingual reference space.
Metrics: Cross-lingual alignment is measured by CLA-WV, the percentage of concepts whose equivalent in one language is closer than other concepts in that language, averaged over language pairs.The reported weak-view metric has similar trends to the strong view shown in Appendix B.3.
Results and discussion: Four of eight configurations are cursed, showing consistent degradation in both monolingual structure and cross-lingual alignment as training-language coverage increases.A configuration is termed cursed when all metrics deteriorate.
Results and discussion: The other four configurations are non-cursed, with stability or improvements in one or more metrics when compute and sampling choices maintain high token counts for target languages.In one example, increasing-compute, realistic-sampling, and aggregate-fixed keep token counts for 10 high-resource target languages high and unchanged, while metrics remain roughly stable.
Recommendation regarding empirical studies: The authors recommend that future empirical studies state their configuration explicitly because differing compute, sampling, and evaluation choices can produce different characterizations of the multilinguality curse.This recommendation addresses the use of “curse of multilinguality” as an overloaded catchall term.

6 CONCLUSION

The paper provides a theoretical and intrinsic grounding of the curse of multilinguality by formalizing perfect multilinguality for embedding spaces. It shows that maintaining representation quality across languages requires capacity growing only logarithmically with language coverage.

The curse of multilinguality is performance degradation in fixed-capacity multilingual models as language scale increases.
The paper formalizes perfect multilinguality for embedding spaces through two multilinguality conditions.
Logarithmic capacity growth in the number of languages is necessary and sufficient to scale representation spaces while maintaining their quality.

LIMITATIONS … A.2 SPHERICAL CAP BOUND FOR NECESSARY DIMENSIONALITY

The paper limits its perfect-multilinguality formulation to simplified, embedding-retrieval-relevant settings and leaves model scaling and architectures for future work. Its appendices establish matching logarithmic upper and lower dimensionality bounds for spherical point packings.

LIMITATIONS: Perfect multilinguality is formulated for embedding-based retrieval, but other representation spaces may require downstream adaptation and the formulation assumes globally shared concepts and semantics.The conception uses a simplified scenario with a global concept set and global semantics across languages.
LIMITATIONS: The empirical studies use small encoder models under controlled language-scale conditions, leaving model scaling and architectural effects for future work.The paper explicitly defers studying how scaling and different architectures affect the curse of multilinguality for embedding space structure.
AI USE STATEMENT: The authors used generative-AI tools for mathematical brainstorming, proof checking, and Appendix A.3 proof generation, while writing and verifying all mathematical arguments themselves.They also combined AI-assisted literature search with manual relevance review and used coding assistants for experiments.
A.1 SPHERICAL CAP BOUND FOR MINIMUM SUFFICIENT DIMENSIONALITY: A.1 proves that d dimensions can pack exp(Ω(d)) points on the unit sphere with pairwise cosine similarities between γ and ρ under the stated angular constraint.The construction uses a spherical cap and separates points by a minimum angular distance.
A.1 SPHERICAL CAP BOUND FOR MINIMUM SUFFICIENT DIMENSIONALITY: The resulting minimum sufficient dimension for L points with pairwise cosine similarities between γ and ρ is d = O(log L).This is the upper bound needed to show logarithmic growth in sufficient dimensionality.
A.2 SPHERICAL CAP BOUND FOR NECESSARY DIMENSIONALITY: A.2 proves that any set of M sphere points separated by a fixed angle θ has M ≤ exp(Aθd), so such a packing requires d = Ω(log M).The proof places disjoint spherical caps of angular radius θ/2 around the points and bounds their total measure.
A.2 SPHERICAL CAP BOUND FOR NECESSARY DIMENSIONALITY: Any set of L points on Sd−1 ⊂Rd with fixed pairwise cosine separation γ therefore requires d = Ω(log L).This lower bound follows from the exponential smallness of fixed-radius spherical-cap measures in d.

A.3 FITTING 7000 LANGUAGES IN 30 EXTRA DIMENSIONS · A.4 FIXED VERSUS VARIABLE NORM ASSUMPTIONS · B ADDITIONAL DETAILS AND RESULTS FOR THE EMPIRICAL CURSE

The paper constructs 7000 language offsets in 30 additional dimensions while preserving pairwise cosine similarities in [0.9, 0.99]. Its dimensionality results extend to bounded variable-norm spaces, retaining logarithmic dependence on the number of languages.

A.3 FITTING 7000 LANGUAGES IN 30 EXTRA DIMENSIONS: O(log L) dimensions suffice to fit L points on a unit sphere with minimum cosine similarity γ and maximum cosine similarity ρ.The construction defines a language offset space U with dimensionality D_L = O(log L).
A.3 FITTING 7000 LANGUAGES IN 30 EXTRA DIMENSIONS: 7000 points fit on a 30-dimensional sphere with minimum pairwise cosine similarity γ = 0.9 and maximum cosine similarity ρ = 0.99.This establishes the concrete constant hidden by the asymptotic dimensionality bound.
A.3 FITTING 7000 LANGUAGES IN 30 EXTRA DIMENSIONS: The construction uses 1 shared dimension to guarantee minimum similarity and 29 remaining dimensions to limit pairwise similarity.The remaining dimensions are chosen so that random points avoid crossing the maximum-similarity threshold.
A.3 FITTING 7000 LANGUAGES IN 30 EXTRA DIMENSIONS: 30 additional dimensions suffice for 7000 language offsets whose pairwise cosine similarities lie in [0.9, 0.99], preserving the multilinguality conditions.The proof obtains the required realization through an existence argument over points sampled on a 29-dimensional sphere.
A.4 FIXED VERSUS VARIABLE NORM ASSUMPTIONS: Theorem 1 extends from fixed-norm concept and multilingual spaces to variable-norm multilingual spaces using fixed-norm language offsets U and bounded norms.The construction sets x_c,ℓ = z_c ⊕ ∥z_c∥u_ℓ, with ∥u_ℓ∥ = r.
A.4 FIXED VERSUS VARIABLE NORM ASSUMPTIONS: The multilinguality and non-degeneracy proofs continue to hold, and the sufficient-dimensionality upper bound remains valid because U is fixed-norm.The similarity decomposition also remains valid under the variable-norm construction.
A.4 FIXED VERSUS VARIABLE NORM ASSUMPTIONS: Logarithmic dimensionality remains sufficient when variable-norm multilingual spaces are bounded by a maximum norm.The proof places L points inside a sphere rather than on its surface; exponential volume growth in dimensionality yields the logarithmic bound.

B.1 METRICS FOR MULTILINGUALITY CONDITIONS

The section defines metrics for evaluating monolingual structure and cross-lingual alignment using a theoretical ideal concept space and multiparallel data. It reports MS-NNO at k = 20, supplements it with MS-MLM, and evaluates alignment under weak and strong views while reporting the weak view.

Monolingual structure: MS-NNO measures monolingual neighbourhood quality by comparing each language’s k-nearest concept overlap against a reference concept space.The metric evaluates neighbourhoods in each language subspace relative to a theoretical ideal.
Monolingual structure: The English concept space from a strong reference model serves as the theoretical ideal, with multiparallel mappings checking concept-set membership.This substitutes for required-scale multiparallel retrieval relevance judgments by treating reference-model neighbourhoods as gold target sets.
Monolingual structure: MS-MLM measures language-averaged normalized masked language-modeling loss on monolingual test sets as an indirect monolingual-structure measure.It complements the neighbourhood-overlap metric with a loss-based proxy.
Cross-lingual alignment: Cross-lingual alignment tests whether a concept’s equivalent in another language is closer than competing concepts under weak and strong views; the paper reports the weak view.The weak view compares against other concepts in the target language, whereas the strong view compares against concepts across all languages; both show similar trends, and the evaluation is equivalent to parallel text mining.

B.2 FURTHER EXPERIMENTAL DETAILS

The experiments use e5-large-v210 as the reported reference model and evaluate multilingual structure across datasets covering 55–90 languages. Training languages prioritize MADLAD-400 token availability and evaluation coverage.

Models: The reported reference model is e5-large-v210, with consistent trends also observed using Llama-3.1-8B-Instruct with last-token pooling.Results use e5-large-v210.
Languages: Evaluation covers FLORES+ and BOUQuET at 90 languages each, plus WMT24++ at 55 languages.Training languages were prioritized by MADLAD-400 token availability and evaluation-dataset coverage.

B.3 ADDITIONAL RESULTS

Additional results reproduce the main trends across multiparallel datasets and show stable, generally high pairwise similarity indicative of anisotropy. Performance stability depends on preserving target-language token counts, while realistic sampling and other factors complicate interpretation.

Other datasets: Figures 4 and 5 show trends similar to Figure 3 on FLORES+ and WMT24++, while CLA-SV is too low on FLORES+ and BOUQuET for meaningful trends.The figures cover encoder models across eight configurations, distinguishing cursed and non-cursed settings, with error bars over five seeds.
Non-degeneracy: Stable, generally high mean pairwise cosine similarity across BOUQuET models provides evidence that the embedding space is highly anisotropic.The paper reports similar behavior for other datasets.
Maintaining token count: The four settings where metrics remain stable or improve generally maintain token counts for the target languages of interest.The section elaborates on this relationship between token allocation and performance stability.
Maintaining token count: With fixed-compute, adding languages reduces each language’s token count and deteriorates mean conditions under aggregate-all for both sampling types.Under aggregate-fixed with realistic-sampling, high-resource languages experience a somewhat softened effect because later low-resource languages consume less token budget.
Maintaining token count: Realistic-sampling lowers performance means because low-resource languages drive them down, while COM-EMB trends may also reflect model scale, optimization, and data quality.The impact of these additional factors is left for future investigation.

B.4 ADDITIONAL EXPLORATION OF LANGUAGE SUBSPACE USE AND DIMENSIONALITY

The section examines how trained multilingual models represent language information and whether language and concept subspaces interact. Models use increasing effective language-subspace dimensionality with more languages, while their subspaces substantially overlap, unlike the orthogonal constructive solution.

Language subspace use: Language-subspace efficiency is defined as the effective dimensionality of language offsets or language-expressing directions.The language subspace is spanned by difference vectors between translation equivalents across languages.
Language subspace use: Participation ratio provides a soft dimension count based on the singular-value spectrum of centered language-offset differences.Equal active singular values recover the number of active directions, whereas concentrated spectra yield smaller values.
Language subspace use: 50% of ambient dimensionality is consumed for language information at higher language counts in the non-cursed BOUQuET configuration.The analysis aggregates all training languages and considers the only non-cursed configuration that maintains the multilinguality conditions.
Interaction between language and concept subspace: Language–concept overlap is generally high across settings and training-language counts, including non-cursed settings, rather than zero as in the constructive solution.The overlap metric uses subspaces whose minimal orthonormal bases capture k = 20% of energy and adjusts for random overlap.
Conclusion: Small-scale empirical embedding spaces do not resemble the ideal constructive solution even under non-cursed conditions, leaving underlying mechanisms for future work.The authors call for investigation across architectures, training configurations, conducive settings, and scale.