PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Low-Resource Language Representations

Francois Meyer, Jan Buys

arXiv:2608.23358v1cs.CL

TL;DR

The paper examines which internal factors contribute to low-resource language underperformance by comparing LLM representational geometry across languages. It finds systematic final-layer degeneration and shows that geometric regularisation reduces this degeneration, with marginal performance gains for larger models.

  • Problem

    The relationship between data scarcity and representational geometry in modern decoder-only LLMs, and whether geometric regularisation can address it, remains unexplored.

  • Method

    The paper compares geometry across 30 languages and applies geometric regularisation during continued pretraining for monolingual adaptation to 10 African languages.

  • Results

    Low-resource languages show greater final-layer representational degeneration, while geometric regularisation reduces degeneration and CosReg marginally improves larger-model CPT performance.

  • Takeaways & Limitations

    Representational geometry is a measurable and actionable factor, making targeted geometric intervention a practical option for low-resource language modelling.

  • Takeaways & Limitations

    Performance gains are inconsistent across tasks, mostly small in absolute terms, and CosReg improves average performance only for larger models.

Abstract

from arXiv · show

The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.

1 Introduction

The paper links low-resource language underperformance to differences in LLM representational geometry, especially degeneration in final layers. It then tests geometric regularisation during continued pretraining as a targeted intervention.

  • The study evaluates cosine similarity and isotropy as complementary metrics of representational geometry.Cosine similarity captures local directional distinctness, whereas isotropy measures global space utilisation.
  • Across 30 languages and 9 open models, low-resource languages show higher final-layer cosine similarity and anisotropy, both reflecting representational degeneration.These patterns indicate less geometrically rich representations and reduced use of representational capacity.
  • Geometric regularisation augments continued pretraining by encouraging separable or more uniformly distributed representations.CosReg reduces pairwise cosine similarity, while I-STAR increases isotropy.
  • CosReg marginally improves continued-pretraining performance for larger models, with its largest gains on AfriMGSM.The comparison uses 9 base LLMs adapted monolingually to 10 African languages.
  • The paper presents representational degeneration as a measurable factor in low-resource underperformance and geometrically regularised CPT as a practical countermeasure.The intervention is framed as targeted geometric adjustment rather than a general solution to all performance gaps.

2 Related Work

Prior work has studied representational collapse and proposed regularisation, but the relationship between geometry and downstream performance remains unsettled. The paper addresses the unexamined link between data scarcity and geometry in decoder-only multilingual LLMs.

  • Existing studies identify representational degeneration as embeddings failing to use the full capacity of the representation space.Cosine similarity and isotropy-based measures are common tools for characterising collapse and anisotropy.
  • Proposed remedies include cosine-similarity penalties, isotropic metrics, and contrastive learning, with mixed downstream-performance findings.The literature does not establish a clear consensus connecting geometric properties to task performance.
  • For modern decoder-only LLMs, prior work had not tested whether data scarcity systematically changes multilingual geometry or whether geometric regularisation can address it.Earlier multilingual studies focused on masked language models and sentence-level semantic tasks.

3 Methodology

The methodology combines multilingual geometric analysis with continued pretraining experiments using geometric regularisation. It measures representation spaces across languages, models, and layers, then evaluates interventions on African-language adaptation.

  • Geometric Analysis: The study compares representation geometry across 30 languages spanning five data-availability tiers, using MADLAD corpus size as a proxy for language data scarcity.The proxy approximates relative web-scale availability because the models’ exact pretraining mixtures are undisclosed.
  • Geometric Analysis: Representations are analysed across 9 open models ranging from 1B to 12B parameters.The model families are Llama, Gemma, and Qwen.
  • Geometric Analysis: Cosine similarity and IsoScore quantify local separability and global space utilisation from token embeddings extracted at each model layer.Cosine similarity uses 1,000 sampled token pairs, while IsoScore uses 5,000 sampled embeddings.
  • CPT with Geometric Regularisation: Continued pretraining further trains a pretrained checkpoint on a target-language corpus under the original pretraining objective.The paper applies monolingual CPT to 9 models on 10 African languages using WURA, a filtered mC4 subset.
  • CPT with Geometric Regularisation: Geometrically regularised CPT adds penalties targeting representational degeneration to the vanilla CPT objective.CosReg penalises high pairwise cosine similarity, while I-STAR uses a differentiable isotropy objective; negative I-STAR coefficients showed no CPT benefit.
  • Evaluation: Adapted models are evaluated against vanilla CPT on target-language datasets in IrokoBench.The benchmark includes AfriXNLI and other African-language tasks.
  • Evaluation: Pairwise cosine similarity is averaged across model layers and languages grouped by data-availability tier.The resulting plots support comparisons of geometric profiles across resource levels.

4 Multilingual Geometric Analysis

Across 30 languages and nine open models, representational geometry varies systematically with language data availability, especially in final layers. Low-resource languages generally show greater degeneration, while Amharic demonstrates that script and linguistic typology also shape geometry.

  • Across most layers, lower-resource languages have higher cosine similarity, indicating less locally separable representations.
  • Amharic consistently deviates from resource-based predictions because its Ge’ez script shares no subword tokens with the Latin-script languages.
  • Final-layer degeneration is the most consistent cross-model pattern: low-resource languages show higher cosine similarity and anisotropy than high-resource languages.
  • Final-layer correlations link data scarcity with degeneration across all models and both metrics: cosine-similarity ρ is negative, whereas IsoScore ρ is positive.
  • Geometric differences between lower- and higher-resource languages shrink from Llama to Qwen to Gemma, although model families retain distinct geometric profiles.

5 CPT with Geometric Regularisation

The authors apply final-layer geometric regularisation during continued pretraining to counter degeneration in low-resource language adaptation. Regularisation changes geometry as intended, while CosReg yields the clearest downstream benefit for larger models and challenging tasks.

  • CosReg decreases final-layer pairwise cosine similarity, while I-STAR increases final-layer IsoScore; each intervention has negligible effect on the other metric.
  • The regularisation effects remain localised to the targeted final layer, without materially disrupting geometry in other layers.
  • Vanilla CPT already marginally reduces final-layer degeneration, particularly in less multilingual Llama and Qwen models.
  • For models 3B and below, geometric regularisation adds no gains beyond vanilla CPT, while all CPT variants produce marginal or no improvement over base models.
  • For models 4B and above, CosReg marginally improves average performance over vanilla CPT, with notable gains on AfriXNLI, AfriMMLU, and AfriMGSM.
  • Across models and tasks, CosReg generally outperforms I-STAR, suggesting final-layer local separability is more influential than global space utilisation for this adaptation setting.

6 Conclusion

The paper links low-resource language underperformance to measurable differences in LLM representation geometry and uses geometric analysis to guide targeted interventions during CPT. Geometric regularisation shifts representations toward desirable properties and often improves downstream performance for larger models.

  • Systematic geometric analysis reveals clear differences in how LLMs construct representation spaces for low- and high-resource languages.
  • Lower-resource languages exhibit progressively worse representational degeneration, especially in final layers.
  • Geometric regularisation successfully reduces final-layer degeneration during continued pretraining for target languages.
  • Cosine similarity-based regularisation frequently improves downstream performance for larger models without requiring additional data.
  • Actionable interpretability connects geometric analyses of internal deficiencies with targeted interventions that inform modelling decisions for low-resource languages.

7 Limitations

The study’s conclusions are bounded by its focus on resource level and continued pretraining, while performance gains from geometric regularisation remain modest and task-dependent. The authors present geometry as an actionable factor rather than claim it is the primary determinant of representation quality.

  • Analysis scope: The geometric analysis focuses on resource level and degeneration, leaving linguistic typology and architectural effects for future study.
  • Analysis scope: Data availability is a strong and consistent correlate of representational geometry, but the authors do not claim it is the primary determinant.
  • Training scope: The experiments evaluate geometric regularisation only during continued pretraining, so effectiveness in training from scratch or instruction tuning is not guaranteed.
  • Performance gains: The impact on performance is inconsistent across tasks, and most gains are small in absolute terms.
  • Performance gains: CosReg marginally improves average performance over vanilla CPT only for larger models, although it can produce meaningful gains on challenging tasks such as AfriMGSM.

A.1 Vanilla CPT

Vanilla CPT hyperparameters are selected through pilot-language validation, resulting in a one-epoch training setup with a low learning rate and bounded sequence length.

  • Vanilla CPT settings are selected using Oromo, isiXhosa, and isiZulu pilots with Llama 3.1 8B and Gemma 3 4B.
  • AfriXNLI validation performance determines the final vanilla CPT settings.
  • 1 × 10−5 with cosine decay to 1 × 10−6 is adopted after 1 × 10−4 performs worse.
  • Performance plateaus after 1 epoch, so subsequent models are trained for 1 epoch with a maximum sequence length of 512 tokens.

A.2 Geometric regularisation

Geometric regularisation augments CPT with terms targeting final-layer representational degeneration, using cosine similarity or isotropy objectives. The methods rely on tuned hyperparameters and covariance estimation choices, with effects assessed across layers and CPT task evaluations.

  • Regularisation objectives: CosReg penalises high cosine similarity to encourage more separable representations, while I-STAR increases isotropy to encourage more uniform embedding-space use.Both regularisers target representational degeneration through complementary geometric properties.
  • Layer selection: CosReg regularisation across more layers fails to improve over applying it only to the final layer.The comparison includes the final layer, all layers, and subsets of the final 3 and 5 layers.
  • I-STAR design: I-STAR estimates covariance by combining batch covariance with a stable reference covariance before computing differentiable IsoScore.The shrinkage parameter ζ controls the balance between the batch and global reference estimates.
  • Layer selection: Final-layer regularisation is used because the strongest IsoScore-based degeneration effect occurs in the final layer, while applying I-STAR across all layers is computationally costly.The experiments apply I-STAR to final-layer representations.
  • Hyperparameter tuning: λ = 1, ζ = 0.2, and N = 250,000 perform best and are used across all models and languages.N determines the number of token embeddings used to compute the stable reference covariance C0.
Loading 2608.23358v1…