PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen

arXiv:2608.27115v1cs.CL

TL;DR

The paper asks whether cross-lingual alignment requires joint training. Using strictly monolingual models, it tests representational geometry, post hoc rotations, and functional transfer, finding that alignment and causal content transfer emerge across independently trained models.

  • Problem

    The paper asks whether models trained independently on distinct languages and corpora develop alignable representations without shared parameters or alignment signals.

  • Method

    The study evaluates strictly monolingual models using correlational geometry analyses, Procrustes alignment, and residual patching across languages.

  • Results

    Independent monolingual models show alignable geometry, a sufficient single rotation, and factual-transfer success rates of 66–85% across five target languages.

  • Takeaways & Limitations

    The findings suggest that cross-lingual alignment can emerge from language and world structure without joint training, supporting post hoc model stitching and merging.

  • Takeaways & Limitations

    The evidence is weighted toward higher-resource, predominantly Indo-European languages, and reconstruction and patching remain untested for the low-resource extension.

Abstract

from arXiv · show

Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.

1 Introduction

The paper asks whether independently trained monolingual models can develop cross-lingually alignable representations without joint training. Across correlational, constructional, and causal tests, it finds shared geometry and functional transfer between models.

  • Existing accounts explain cross-lingual alignment through joint-training features, but do not predict alignment between independently trained models.
  • The study tests strictly monolingual Transformers trained on distinct corpora and languages, including five independently developed ∼1B-parameter models.
  • Independent monolingual models develop alignable representational geometry, with alignment increasing alongside data scale, model scale, or linguistic proximity.
  • A single Procrustes rotation fit on parallel sentence representations aligns hidden-state clouds between models.
  • Patching a rotated English residual into a German model flips factual predictions toward the donor’s capital in most cases.

2 Related Work

Prior work established shared or alignable structure in multilingual and independently trained representations. This paper extends that line to strictly monolingual decoder-only language models and tests whether aligned states affect model behavior.

  • Multilingual models show partially shared cross-lingual geometry, with overlapping linear subspaces and internal components shared across languages.
  • The Platonic Representation Hypothesis proposes that representations converge toward a shared model of reality as models scale.
  • Strictly monolingual Goldfish models provide a controlled setting for testing whether language-agnostic structure emerges without multilingual joint training.
  • Prior embedding work found that independently trained semantic spaces can often be aligned post hoc through approximately orthogonal transformations.
  • The closest prior study aligned monolingual BERT representations with a learned linear map; this paper studies decoder-only causal LMs and adds causal experiments.

3 Representational Alignment

Independent monolingual models show cross-lingual representational alignment beyond shuffled and architectural controls, across layers, languages, data scales, and independently developed model families. Alignment strengthens with linguistic similarity and training-data scale, while low-resource and cross-lab tests extend the correlational evidence.

  • Evaluation setup: Independent monolingual models are evaluated with CKA on parallel sentences, contrasting matched pairs against a fixed shuffled control across layers and four parallel corpora.The models have disjoint vocabularies, parameters, and training corpora; the shuffled control preserves architecture and corpus statistics while disrupting semantic correspondence.
  • Cross-lingual alignment: 0.78 matched last-layer CKA versus 0.17 shuffled under SGPT on FLORES demonstrates alignment above architectural and surface-statistical baselines.Matched CKA exceeds shuffled at every layer and for every language pair, with the gap preserved across datasets.
  • Cross-lingual alignment: Matched CKA remains stable across layers under mean, token-aligned, and SGPT representations, with layer variation smaller than the matched–shuffled gap.Figure 2 reports the layerwise English–French comparison under all three representations.
  • Factors associated with alignment: Linguistic proximity predicts stronger alignment: English–German reaches 0.82 and English–French 0.81, versus 0.64 for Chinese–Arabic and 0.65 for Chinese–Hindi.URIEL syntactic distance is the strongest predictor of matched CKA (ρ=−0.64, p<.001), while mean NLL is also significant (ρ=−0.43, p=.01).
  • Factors associated with alignment: Matched CKA gaps of +0.39 to +0.58 for English paired with Tagalog, Swahili, Northern Uzbek, and Amharic remain comparable to high-resource references at middle and last layers.The low-resource extension tests different language families and writing systems but covers correlational experiments only.
  • Scaling and independent model families: Matched CKA increases monotonically from 5 MB to 1000 MB across all three representations, and exceeds shuffled baselines when models are trained at different data sizes.A five-model ∼1B replication finds matched CKA above shuffled in all 10 pairs, averaging 0.71 versus 0.18.
  • Scaling and independent model families: Cross-lab alignment persists across five ∼1B monolingual models sharing no architecture, tokenizer, training data, or research group.The replication includes Pythia, Zh-Pythia, Tucano, Bielik, and Minerva; a Goldfish–Pythia check also stays above shuffled at every layer.
  • Evidence boundary: CKA establishes similarity only up to an unspecified transformation, so these correlational results do not identify the map aligning the representation spaces.The paper treats construction and causal experiments as stricter subsequent tests.

4 Cross-Lingual Representation Reconstruction

The study reconstructs one independently trained monolingual model’s representations from another using post-hoc mappings, finding that a single orthogonal rotation is highly effective for retrieval across language pairs and layers.

  • Projection methods: A single Procrustes rotation preserves the source space’s pairwise inner products, whereas Affine and MLP allow unconstrained transformations or nonlinear mappings.The rotation is solved in closed form from the SVD of X⊤Y.
  • Method comparison: Across 36 Goldfish language pairs, Procrustes retrieves matching target representations with 88.7% P@1std accuracy, versus 0.2% for the identity baseline.The comparison uses final-layer means across the 36 pairs.
  • Method comparison: Procrustes reduces MSE from 2.258 to 0.689, while Affine and MLP achieve lower MSE values of 0.439 and 0.456.Affine and MLP optimize reconstruction in larger hypothesis classes than the constrained rotation.
  • Method comparison: Procrustes outperforms Affine and MLP on retrieval despite their better MSE and CKA, consistent with retrieval rewarding preserved angles and neighbourhood structure.The paper attributes this to a mismatch between Euclidean reconstruction objectives and rank-based, direction-sensitive retrieval.
  • Projection methods: Figure 5 shows the Procrustes-mapped English and French representation clouds coinciding in a shared 2D PCA basis while retaining the source geometry.The visualization is for the final layer of the English–French pair.
  • Layerwise analysis: Retrieval remains above the identity baseline across all 13 layers and peaks at layer 8 for English–French with P@1=0.981 and layer 6 for English–Russian with 0.987.These results indicate that the rotation is usable throughout network depth, with strongest retrieval in mid layers.

5 Cross-Model Activation Patching

The paper tests whether a rotation learned from sentence representations transfers functional content between monolingual models. Cross-model patching shows that rotated donor residuals frequently shift factual predictions toward the donor answer, approaching within-model performance.

  • Patching procedure: The patch replaces the target model’s concept-position residual with a projected donor residual across layers j→L rather than at one layer.The span patch targets the country word so injected information can propagate through attention to the answer logits.
  • Patching procedure: The experiment compares within-language, Procrustes, unprojected, and shuffled residual sources, with directional success defined by whether the donor answer’s log probability increases.Unprojected residuals provide a basis-mismatch control, while shuffled samples match source activation statistics.
  • Results: Within-model patching produces 76–98% directional success across five targets, establishing the ceiling for cross-lingual transfer.This reproduces the concept-position span-patch design in the study’s setting.
  • Cross-model transfer: Across five target languages, Procrustes reaches 66–85% directional success against a 76–98% within-model ceiling without retraining.The evaluation uses a different task, token position, and readout from the sentence-level mapping fit.
  • Results: Procrustes exceeds shuffled control across seven factual relations, with Wilson intervals separating on four larger sets and directional consistency on three smaller sets.The same per-layer maps are reused without refitting, with a fixed patch start j=2.
  • Results: Procrustes patching flips the target prediction to the donor’s capital in 85% of cross-fact pairs, while shuffled Gaussian samples remain near 50%.The directional success rate is measured over 870 country→capital cross-fact pairs per cell.

6 Conclusion and Discussion

The findings indicate that independently trained monolingual models can develop cross-lingually alignable representations without joint training. The paper connects this convergence to modular possibilities while identifying typologically distant language pairs as an open scaling boundary.

  • Conclusion: Alignment exceeds shuffled controls, orthogonal rotations reconstruct hidden states, and rotated English residuals flip German factual predictions in 85% of cases.These results span the paper’s correlational, constructional, and causal standards of evidence.
  • Conclusion: The results suggest that cross-lingual alignment can emerge from the structure of language and the world it describes without joint training.The conclusion frames this as evidence for representational convergence in strictly monolingual models.
  • Implications: The findings motivate stitching independently trained monolingual specialists into a shared space and exploring model merging across languages.The paper reports preliminary stitching experiments producing a functioning bilingual system.
  • Implications: An explicit alignment objective could train low-resource monolingual models to align with high-resource anchors and support cross-lingual transfer through post-hoc rotation.This is presented as a future direction rather than an established result.
  • Limitations: Scaling anchor-based projection to typologically distant language pairs remains open.This is the paper’s stated scope boundary for the proposed alignment approach.

Limitations

The study’s evidence is bounded by model and language coverage, possible residual contamination, reliance on parallel data, and a narrowly scoped causal evaluation.

  • Language and model coverage: The main analyses emphasize higher-resource, predominantly Indo-European languages and models up to approximately 1B parameters.Low-resource results cover correlation only; larger monolingual checkpoints remain untested.
  • Language contamination: English contamination is bounded at or below 0.1% in the nine main-analysis corpora, but residual contamination remains an unquantified contributor.The concern is greater for noisier low-resource corpora.
  • Parallel-data assumption: Rotations require parallel sentence pairs, and the study does not evaluate unsupervised mapping alternatives.Sentence-aligned data is still not free for low-resource languages.
  • Scope of the causal evaluation: The causal evaluation covers seven factual relations, one concept-bearing span, first-subword scoring, and limited language-task sweeps.Multi-token generation and open-ended use fall outside the design.

Ethical Considerations

The paper uses public research artefacts and factual probes without personal data, while cautioning that modular combinations could inherit component-model weaknesses.

  • Data and artefacts: All models and corpora are publicly released research artefacts, and the authors release no new model or corpus.The factual probes cover encyclopaedic relations over public entities and contain no personal data.
  • Modular systems: Stitching or merging monolingual models could carry each component’s biases and failure modes into the combined system.The paper states that any such system needs its own evaluation.

Use of AI assistants.

The paper reports AI coding-assistant use for implementation support while retaining author responsibility for experimental design, analyses, and claims.

  • Use of AI assistants: AI coding assistants were used for implementation support.The statement concerns coding support rather than experimental design or analysis.
  • Author responsibility: The authors state that all experimental design, analyses, and claims are their own.
  • Reported analyses: Appendix H reports complete pairwise CKA values for 36 language pairs across nine monolingual 1000 MB Goldfish models.The tables distinguish matched and shuffled CKA across three pooling methods and four datasets.
  • Scale analyses: The appendix also provides full pairwise CKA matrices at four Goldfish training-data scales, using shuffled controls throughout.Cross-size comparisons pair 1000 MB with 100 MB models and 10 MB with 5 MB models.

B.1 Does Alignment Grow with Training Data?

The appendix examines how alignment varies with training scale, model scale, architecture, language relationship, and model fit. Across settings, matched CKA remains above shuffled controls, while linguistic distance predicts alignment strength.

  • Does Alignment Grow with Training Data?: Matched off-diagonal CKA increases monotonically from 5 MB to 1000 MB across all three pooling strategies.Shuffled baselines remain near zero, widening the matched−shuffled gap with more training data.
  • Cross-size comparisons: Cross-size comparisons show that same-language diagonal CKA dominates, while off-diagonal matched signal remains above shuffled baselines across scales.The 10 MB versus 5 MB comparison has lower absolute values, but the matched-vs-shuffled gap persists.
  • Cross-architecture comparisons: Matched CKA exceeds shuffled CKA at every layer for both Goldfish–Pythia English and Chinese comparisons.The layerwise profile is U-shaped, with higher alignment at embedding and final layers than in the middle.
  • Independent model coverage: The shared-nothing comparison spans five independently developed monolingual models across three architecture families, five languages, and five research groups.These models differ in hidden dimension, depth, attention heads, and vocabulary.
  • Predictors of alignment: Syntactic distance is the strongest single predictor of alignment: r = −0.60, ρ = −0.64, p < .001.Genetic, mean-NLL, and geographic distances follow, while phonological inventory distance is not significantly correlated.
  • Predictor interpretation: The predictor ranking is descriptive because URIEL distances are intercorrelated, and the study does not dissociate their contributions.Lower model quality, measured by mean NLL, is also associated with weaker pairwise alignment.

E Training-Corpus Language Contamination Audit

The Goldfish training-corpus audit finds very little confidently identified English in the high-resource non-English corpora, limiting contamination as an explanation for the observed alignment. Low-resource corpora contain more foreign material, creating a stated caveat for the corresponding results.

  • English exposure: At most 0.1% of sampled lines in the nine main-analysis languages were confidently identified as English.French contained 0.08% and Hindi 0.10%; all other languages were at or below 0.02%.
  • English exposure: At the 1000 MB tier, the audit bounds incidental English exposure to roughly 1 MB of text.This is below the 5 MB tier, the smallest at which alignment was measured.
  • Interpretation: The audit concludes that these contamination rates cannot supply the cross-lingual competence observed in Sections 3–5.The passage contrasts the measured rates with the meaningful high-resource leakage required by the cited contamination mechanism.
  • Caveat: Low-resource corpora contain up to 2.14% other-language material and up to 1.54% English for Swahili.The passage attributes this to noisier web sources and identifies it as a caveat for Table 2.
  • Caveat: Line-level language identification on web data is imperfect, especially for closely related varieties.The passage notes that this limitation is visible in the Arabic dialect labelling.

F.2 Dimensional Analysis

The dimensional analysis asks whether Procrustes residual error is concentrated in a few correctable directions or dispersed across the representation space. Across language pairs and layers, the residual is high-rank and largely outside the target’s dominant variance directions, despite its substantial raw magnitude.

  • Analysis: The analysis compares residual effective rank and residual energy location relative to the target representation.It uses SVD-based summaries, including effective rank and the fraction of residual energy outside the target’s top-k directions.
  • Spectral structure: Target representations use 43–67 directions out of 768, while residual effective rank is 147–246.The residual’s half-energy and 90%-energy thresholds also require many singular values, showing that the error is diffuse rather than concentrated.
  • Subspace location: 88–93% of residual energy lies outside the target’s top-10 directions, and 68–74% lies outside its top-50 directions.Thus, the rotation places the residual mainly outside the subspace where the target stores its variance.
  • Functional consequence: The residual Frobenius norm is 0.79–0.93 of the target’s at layer 8, yet its location can make it functionally irrelevant.The passage distinguishes large raw magnitude from whether the model reads those directions.
  • Across layers: Residual effective rank exceeds target effective rank by at least twofold across layers, with frac_out10 above 0.79 and frac_out50 above 0.57.The pattern holds at every layer for every headline pair, ruling out a low-dimensional residual collapse.
  • Interpretation: The high-rank, off-subspace residual explains why more flexible Affine and one-layer MLP mappings do not improve retrieval over Procrustes.The supported interpretation is approximate isometry along the target’s dominant directions rather than a concentrated correction.
Loading 2608.27115v1…