PaperlayerView exact revision on arXiv

Source-linked AI summary

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos

arXiv:2608.26576v1cs.CL

TL;DR

The paper asks whether aligned shared-language embeddings justify treating English-only and bilingual models as internally equivalent. It compares tightly controlled paired models across eight languages and finds that token embeddings align while predictive hidden states remain mismatched beyond seed variation. The results caution that embedding alignment alone does not establish contextual-state equivalence.

  • Problem

    Embedding alignment may not reveal whether models share the deeper contextual representations used for prediction, although such equivalence matters for model comparison.

  • Method

    The authors pretrain matched 310M-parameter English-only and bilingual decoder-only models across eight languages, control English exposure, compute, and document overlap, then align shared English vocabulary and test held-out probes.

  • Results

    Across all eight languages and control settings, token embeddings align near seed-level similarity while contextual hidden states differ beyond random-seed variation, with mismatch peaking in middle layers.

  • Takeaways & Limitations

    Lexical alignment should not be treated as evidence of contextual-state equivalence; studies of probing, similarity, and interpretability should validate the representation family analyzed.

  • Takeaways & Limitations

    The study is restricted to one 310M decoder-only architecture and 50 semantic axes with a translation pipeline, so its findings are descriptive of this evaluation setup.

Abstract

from arXiv · show

When researchers compare multilingual models for probing, interpretability, or cross-lingual transfer, they often align embedding spaces and assume that shared-language representations are comparable. We show that this assumption can be premature for decoder-only models. We pretrain paired 310M-parameter models (one English-only, one bilingual) across eight typologically diverse languages, separately controlling for English exposure, total compute, and document overlap. After aligning on shared English vocabulary, we test held-out words and find that token embeddings look similar after alignment, but the deeper hidden states that the model uses for prediction do not. This gap holds for all eight languages and survives controls for document overlap and alternative alignment methods. This hidden-state mismatch grows through middle transformer layers, suggesting that it arises from contextual processing rather than the input representations where alignment is performed. Embedding alignment can mask real differences in how models internally represent a shared language, which matters for any downstream study that treats aligned models as interchangeable.

1 Introduction

The paper tests whether aligning shared English vocabulary makes English representations comparable across English-only and bilingual decoder-only models. Across controlled eight-language comparisons, embeddings realign closely while predictive hidden states diverge beyond seed variation, revealing an alignment fallacy.

  • Motivation: Linear alignment at the input layer can make bilingual and English-only token embeddings appear equivalent without testing contextual hidden states.The paper calls this misleading conclusion the alignment fallacy.
  • Findings: The hidden-state disagreement appears across all eight language pairs, indicating that shared-language representations can differ beyond the input layer.The disagreement is found in contextual hidden states rather than token embeddings.
  • Experimental control: Matched 310M-parameter model pairs separately control English exposure, total compute, and training-document overlap across eight languages.These controls isolate the effect of adding a second language while holding architecture, tokenizer, hyperparameters, and data domain fixed.
  • Approach: 3000 shared English words are used for alignment, while 1000 held-out words test whether bilingual exposure changes shared-language representations.The held-out evaluation is compared against a seed-variation baseline from English-only models.
  • Findings: After alignment, token embeddings return to seed-level similarity, but deeper predictive hidden states diverge beyond random-seed variation.The mismatch persists with non-overlapping English documents, affine alignment, and intermediate training checkpoints.

2 Related Work

Prior research studies multilingual sharing, language-specific organization, alignment, and probing, but does not isolate the effect of adding a second language to an otherwise identical training setup.

  • Multilingual structure: Earlier multilingual studies examine shared or language-specific structure in models trained multilingually from the start.They do not isolate adding a second language to an otherwise identical training run.
  • Research gap: The paper addresses the unresolved comparison by controlling training conditions and testing shared-language representations beyond lexical alignment.This design targets the isolated effect of language mixture.
  • Alignment: Bilingual lexicon induction aligns monolingual lexical spaces and evaluates translation or neighbor agreement, primarily at the embedding level.Later work identifies failures under non-isometry, domain shift, and frequency mismatch.
  • Directional probing: Directional probing measures positions along interpretable semantic axes and can detect directional reversals that aggregate similarity metrics may miss.The paper adapts this approach to pairwise cross-run comparison.

3 Experimental Design

The experimental design compares matched English-only and bilingual models after representation alignment, measuring held-out semantic disagreement relative to random-seed variation under controlled training conditions.

  • Baseline comparison: Positive Δm means bilingual exposure produces more mismatch than random initialization explains, using the average mismatch of six English-only seed pairings as baseline.The baseline is computed at each checkpoint.
  • Data and models: Matched model pairs hold architecture, training recipe, and language budgets constant while varying the presence of a second language across eight languages.All models use a 310M-parameter decoder-only architecture.
  • Confound controls: The controls separately address unequal English exposure, total optimizer updates, and shared English documents between bilingual and English-only models.Control settings must not be compared with one another as treatment effects.
  • Evaluation: The analysis aligns representations using 3000 English anchor words and evaluates 1000 separate held-out probes balanced across semantic domains.Separating anchors from probes prevents the alignment objective from directly fitting the evaluation vocabulary.
  • Metrics: The primary metric, DAxis, measures disagreement in held-out words’ relative positions along matched semantic directions after alignment.Nearest-neighbor and pairwise cosine-similarity disagreement provide complementary local and global geometry checks.

4 Results

Across controlled experiments, bilingual training changes shared-English hidden states beyond seed variation while aligned token embeddings remain near baseline similarity. The mismatch persists with non-overlapping data, appears beyond the input layer, peaks in middle layers, and is present early under alternative alignment methods.

  • Experimental design: The experiments ask whether mismatch survives exposure, compute, document-overlap, layer, checkpoint, and alignment-method controls.Sixteen model pairs were evaluated across eight languages using held-out probes.
  • Exposure and compute controls: Hidden-state differences remain above seed variation across all control settings, while embedding differences stay near the seed baseline.This pattern holds across all eight languages, including matched-English C1 conditions.
  • Document-overlap control: Removing document overlap does not eliminate the mismatch: embedding changes cluster near zero, whereas hidden-state changes vary in sign across languages.Overlap can perturb individual measurements but is not the primary driver of the persistent hidden-state gap.
  • Input layer versus hidden states: For every language, hidden states diverge more than token embeddings in both English-centered and target-language-centered comparisons.The same ordering outside English rules out an English-specific artifact.
  • Layer localization: Mismatch is small at the input layer, rises through middle transformer layers, and partially decreases near the end while remaining above the embedding baseline.Row-L2 normalization and neutral-anchor z-scoring preserve the middle-layer peak, indicating a directional rather than magnitude effect.
  • Training and alignment robustness: The gap is already above embedding mismatch at 500 steps and remains through 3000 steps; contextual DAxis is 0.386 after direct contextual alignment.Different anchor subsets produce the same result, and alternative alignment methods do not eliminate the mismatch.

5 Discussion

The discussion argues that lexical alignment can conceal meaningful differences in contextual representations between matched English-only and bilingual models. It recommends checking hidden states against seed variation before treating aligned models as interchangeable.

  • 5 Discussion: Lexical agreement does not imply contextual-state equivalence in controlled 310M decoder-only models.Substantial held-out contextual geometric mismatch remains under orthogonal and affine alignment, including contextual-anchor alignment.
  • 5 Discussion: Bilingual exposure is associated with measurable changes to shared-language representations beyond input-layer alignment.The finding complements prior work on internal language separation while comparing matched monolingual and bilingual models.
  • 5 Discussion: The experiments do not isolate whether shared-vocabulary superposition or bilingual polysemy contributes to the mismatch.The authors explicitly leave the mechanism unresolved.
  • 5 Discussion: The study establishes mismatch under tested alignment families, not failure of every contextual alignment or downstream behavioral equivalence.Output-level behavioral differences remain untested, and broader architectures and scales are future work.
  • 5 Discussion: Practitioners should compare hidden states alongside embeddings and against random-initialization variation.The proposed protocol evaluates held-out words in embeddings and hidden states after alignment on common vocabulary.

6 Conclusion

The conclusion reports that adding a second language changes shared-language representation geometry beyond seed variation, despite aligned token embeddings. The mismatch appears early, peaks in middle layers, and survives the tested controls and alignment methods.

  • 6 Conclusion: Adding a second language changes shared-language representation geometry beyond seed variation in controlled 310M-parameter experiments.Token embeddings align as expected from different seeds, while contextual states remain mismatched.
  • 6 Conclusion: The mismatch is present at 500 training steps, peaks in middle transformer layers, and survives equal exposure, equal compute, and non-overlapping documents.It also persists under the alignment families tested, including direct contextual-anchor alignment.
  • 6 Conclusion: Lexical alignment should not be treated as evidence of contextual-state equivalence.The conclusion recommends checking hidden states alongside embeddings and comparing against seed variation.
  • 6 Conclusion: Whether the observed geometry produces downstream behavioral differences remains open.The study does not evaluate output-level behavior.

Limitations

The study deliberately fixes one 310M decoder-only architecture and evaluates cross-linguistic geometry with a specific semantic-axis and translation setup. Broader scale, architectures, training recipes, and downstream behavior remain untested.

  • Limitations: The study is restricted to one 310M decoder-only architecture as a control choice, not a claim about scale invariance.Independent controls include English exposure, total compute, document overlap, alignment method, and representation layer.
  • Limitations: The evaluation depends on 50 semantic axes and a translation pipeline for extending them to non-English languages.The authors characterize the results as descriptive of this setup rather than exhaustive measurements of cross-linguistic difference.
  • Limitations: Larger models, broader architectures, training recipes, contextual alignment families, causal mechanisms, and downstream behavioral evaluations remain future work.Scaling without matched controls would make mismatch harder to interpret.

Ethical Considerations

The paper’s broader impact is methodological: it warns that embedding alignment may underestimate differences in predictive layers. It reports no direct application to individuals or production systems and no use of sensitive individual data.

  • Ethical Considerations: The work studies structural properties of bilingual language-model representations rather than individual users or downstream production systems.Its primary broader impact is methodological.
  • Ethical Considerations: No personally identifiable or sensitive individual data were used.Word translations received automatic quality control and aggregate-level manual review.

Appendix

The appendix documents implementation details, evaluation setup, quantitative support, robustness checks, and category- and language-level signal breakdowns.

  • Appendix A.1 records implementation details.
  • Appendix A.2 fixes probe inventories, axis grounding, translation quality control, and split protocols.
  • Appendix A.3–A.5 provide detailed values, robustness tests, and signal breakdowns across categories and languages.

A.1 Complete Implementation Details

All runs use a shared 12-layer Llama-style decoder architecture with approximately 310M parameters and a fixed multilingual tokenizer.

  • All runs share one 12-layer Llama-style pre-norm decoder with RMSNorm, 768 hidden units, 3072-unit FFNs, and 12 attention heads.
  • The models use untied token/output matrices and approximately 310M parameters.
  • Every checkpoint uses the unchanged Llama-3.2-1B BPE tokenizer with its complete 128,256-token multilingual vocabulary.

A.2 Probe Details

The appendix fixes the semantic-axis inventory, documents translation quality controls, and establishes the scale of the directional metric before presenting later results.

  • The semantic-axis tables enumerate the full inventory and its theoretical grounding before analysis.This fixes the directional diagnostics rather than tuning them post hoc.
  • Probe translation quality is documented with per-language scores and quality-tier breakdowns for the overlap ablation.The appendix records the filters applied before probes enter evaluation tables or figures.
  • The axis-magnitude table fixes the scale of the main directional metric before later sections introduce more conditions.

A.3 Data Statistics and Split Protocol

The appendix provides reproducibility details for data, splits, axes, controls, and robustness analyses, while extending the main findings across training settings and languages.

  • Data Statistics and Split Protocol: The corpus combines language-specific conversational, educational, literary, encyclopedic, news, and subtitle sources, with source proportions varying by language.Within each target-language experiment, conditions use the same source pools and preparation pipeline.
  • Data Statistics and Split Protocol: The mixed-language audit flags 12,382 candidates, or 0.385% of 3,213,686 documents, including 2,721 target-plus-English candidates.Per-language target-plus-English rates range from 0.001% to 1.217%.
  • Data Statistics and Split Protocol: The study uses a prespecified training recipe without hyperparameter search and reports exact values behind the main-text claims.
  • Data Statistics and Split Protocol: The fixed 50-axis inventory, framework mappings, and split summaries make probe construction and evaluation auditable.
  • Robustness Beyond the Main Controls: Additional-language summaries confirm hidden-state amplification, while exploratory analyses do not support a strong monotonic typology claim.Aggregate tests keep the hidden-state-over-embedding contrast positive, and negative controls prevent interpreting the result as uniquely cultural.
Loading 2608.26576v1…