PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Language Neutrality of Pre-trained Multilingual Representations

Jindřich Libovický, Rudolf Rosa, Alexander Fraser

arXiv:2004.05160v4cs.CL

TL;DR

Earlier work assessed multilingual representations mainly through indirect zero-shot transfer, leaving open how directly they encode lexical semantics across languages. This paper evaluates representations directly and tests centering and small-parallel-data projections. Contextual embeddings outperform aligned static embeddings on semantic similarity tasks, yet retain a strong language-identity signal that can be reduced by the proposed methods.

  • Problem

    Earlier zero-shot transfer studies did not directly establish whether multilingual representations encode similar lexical-semantic phenomena similarly across languages.

    Source: §1 · p. 1

  • Method

    The paper directly evaluates multilingual representations on semantic tasks and investigates language-specific centering and supervised projection using small parallel data.

    Source: §1 · p. 1, §8 · p. 9

  • Results

    Contextual embeddings outperform aligned static embeddings on cross-lingual semantic similarity, retain a strong language-identity signal, and become more language-neutral through centering or a small-data supervised projection.

    Source: §8 · pp. 8–9

  • Takeaways & Limitations

    Language-specific centering improves neutrality to some extent, but stronger neutrality is attainable with a supervised linear projection fitted on a small set of parallel sentences.

    Source: §8 · p. 9

    Centering requires knowing the language in advance, so it is not a language-independent modification.

    Source: §8 · p. 9

Abstract

from arXiv · show

Multilingual contextual embeddings, such as multilingual BERT and XLM-RoBERTa, have proved useful for many multi-lingual tasks. Previous work probed the cross-linguality of the representations indirectly using zero-shot transfer learning on morphological and syntactic tasks. We instead investigate the language-neutrality of multilingual contextual embeddings directly and with respect to lexical semantics. Our results show that contextual embeddings are more language-neutral and, in general, more informative than aligned static word-type embeddings, which are explicitly trained for language neutrality. Contextual embeddings are still only moderately language-neutral by default, so we propose two simple methods for achieving stronger language neutrality: first, by unsupervised centering of the representation for each language and second, by fitting an explicit projection on small parallel data. Besides, we show how to reach state-of-the-art accuracy on language identification and match the performance of statistical methods for word alignment of parallel sentences without using parallel data.

1 Introduction

The paper directly evaluates multilingual representations on tasks involving lexical semantics, addressing limits of indirect zero-shot evaluation. It finds contextual embeddings outperform aligned static embeddings on semantic comparisons, yet retain language-identity information; centering and projection offer ways to strengthen neutrality.

1 Introduction

  • Direct representation comparisons on lexical-semantic tasks address open questions left by earlier zero-shot studies of morphology and syntax.The earlier work also reported promising initial sentence-retrieval results, but left questions about lexical-semantic representation unresolved.Source: §1 · p. 1
  • Zero-shot transfer can be an uncertain measure of neutrality because a probe may overfit the training language before evaluation stops.The concern is that same-language validation may favor continued training even when earlier stopping would better serve cross-lingual transfer.Source: §1 · p. 1
  • Contextual embeddings consistently outperform aligned static word embeddings on cross-lingual semantic similarity tasks, despite the latter being explicitly trained for language neutrality.The paper also finds contextual embeddings generally more informative than aligned static embeddings.Source: §1 · p. 1
  • A mean-pooled contextual-embedding classifier reaches state-of-the-art language-identification results, showing that the representations still strongly encode language identity.A simple algorithm using contextual embeddings also reaches state-of-the-art word-alignment results.Source: §1 · pp. 1–2
  • The authors propose stronger language neutrality through language-specific centering or explicit projections fitted on small parallel data.They also report that XLM-R outperforms mBERT on sentence retrieval and MT QE, with similar performance on language identification and word alignment.Source: §1 · p. 2

2 Related Work

Prior work commonly used zero-shot transfer to assess multilingual representations, but its results varied across languages and tasks. This paper distinguishes downstream task success from direct evidence that phenomena are represented similarly across languages.

2 Related Work

  • Zero-shot findings were mixed: transfer success depended on language similarity, and some studies struggled on POS tagging, NER, and XNLI.Pires et al. (2019) report typological-similarity dependence; Wu and Dredze (2019) report difficulties, as do Rönnqvist et al. (2019) for language generation.Source: §2 · p. 2
  • Some successful zero-shot dependency parsing results used supervised projection trained on word-aligned parallel data, while other work suggested structural probing may be more suitable.Wang et al. (2019) report the supervised-projection result; Chi et al. (2020) suggest structural probing.Source: §2 · p. 2
  • Pires et al. (2019) found sentence-retrieval accuracy increased after subtracting the average target-language embedding difference, an idea this paper studies systematically.Source: §2 · p. 2
  • XTREME and XGLUE broadened zero-shot evaluation to tasks including natural language inference and question answering, with XLM-R outperforming mBERT.These benchmarks assess a broader range of zero-shot transfer tasks.Source: §2 · p. 2
  • Downstream models can extract useful features without showing whether similar phenomena are represented similarly across languages or whether embeddings retain language identity.The authors therefore use tasks that directly compare representations.Source: §2 · p. 2

3 Centering Representations

The paper explores centering as an unsupervised way to reduce language-specific information in representations, contrasting it with explicit projections that require parallel data. The approach rests on an additive language-specific and language-neutral representation hypothesis.

3 Centering Representations

  • Explicit projections can suppress language identity while retaining sentence meaning, but fitting them requires parallel data; representation centering is explored as an unsupervised alternative.Source: §3 · p. 2
  • The centering hypothesis treats a sentence representation as additive language-specific and language-neutral components, with the language-specific component shared across sentences in that language.The language-neutral component is hypothesized to capture meaning independently of language.Source: §3 · p. 2
  • Centering subtracts a language’s mean sentence representation from its contextual embeddings, placing the language’s average at the vector-space origin.The centroid is estimated from a set of sentences in that language.Source: §3 · p. 2
  • The intuition is that frequent within-language phenomena such as function words contribute to the centroid, while sentence-varying meaning features are averaged out.Subtracting the centroid is hypothesized to remove language-specific features while preserving meaning content.Source: §3 · pp. 2–3
  • The experiments compare original and centered representations across probing tasks, testing all model layers and both [cls] and mean-pooled sentence vectors where applicable.Source: §3 · p. 3

4 Probing Tasks

The probing tasks use representation comparisons to assess language neutrality and include language identification and language similarity analyses. The paper also describes task-specific procedures for retrieval, word alignment, and translation-quality estimation.

4 Probing Tasks

  • The probing setup treats cross-lingual word- and sentence-similarity tasks as measures of neutrality when cosine distances can estimate those similarities.Additional task training can improve probe performance, though the authors say it reveals less about the representations themselves.Source: §4 · p. 3
  • Language identification trains a linear classifier on sentence representations, while language similarity uses V-measure and hierarchical clustering of language centroids.The clustering uses centroid cosine distance and stops at a cluster count matching the number of language families.Source: §4 · p. 3
  • Parallel sentence retrieval selects the candidate sentence with the smallest representation cosine distance; the study also evaluates a linear projection into English space fitted on parallel sentences.The projection minimizes element-wise mean squared error between English representations and projected representations of their translations.Source: §4 · p. 3
  • Word alignment is computed as a minimum-weight edge cover of a bipartite graph whose potential links are weighted by token-representation cosine distance.Tokens split into subwords are represented by the average of their subword embeddings; alignment is evaluated with F1 over sure and possible links in manually aligned data.Source: §4 · pp. 3–4
  • A distortion penalty of 1/d is added to each alignment edge to prefer monotonic links, where d is the tokens’ absolute-position difference.Its weight is estimated on a development set; the alignment algorithm is invariant to representation centering.Source: §4 · p. 3
  • MT quality estimation compares source-sentence and translation-output representations, using cosine distance to estimate translation quality without a reference translation.Evaluation uses Pearson correlation with TER; tested approaches include plain and centered representations, bilingual projection, and supervised regression.Source: §4 · p. 4

5 Probed Models

The probed models include mBERT and variants, XLM-R, and aligned static word embeddings as a baseline. The experiments also compare sentence representations and examine language identification.

5 Probed Models

  • Aligned static word embeddings provide a language-neutrality baseline, but unlike Transformer hidden states they do not capture sentence context.Sentences are represented by averaging their word embeddings.Source: §5 · p. 4
  • mBERT has 12 layers and a shared 120k-wordpiece vocabulary, and the experiments use either its [cls] vector or mean-pooled token embeddings.Its training combines masked language modeling and sentence-adjacency prediction.Source: §5 · p. 4
  • The adversarial lng-free model continues mBERT training with masked language modeling while gradient reversal pits language-ID classifiers against the representation model.The sentence-adjacency objective is omitted.Source: §5 · p. 4
  • Table 1 reports language-identification accuracy using the best-scoring layers.Source: Table 1 · p. 5
  • DistillmBERT is a six-layer model initialized from mBERT and trained to match its output distribution while retaining the masked-language-modeling objective.Its reduced representation space was proposed as a way to encourage language neutrality.Source: §5 · pp. 4–5
  • XLM-R uses the mBERT architecture with a 250k-token SentencePiece vocabulary and is trained with masked language modeling only.It was trained on two terabytes of CommonCrawl text.Source: §5 · p. 5

6 Experimental Setup

The setup evaluates language identification, sentence retrieval, word alignment, machine-translation quality estimation, and language-family clustering. It uses Wikipedia and WMT data, manually annotated alignment datasets, and aligned static embeddings.

6 Experimental Setup

  • Language-ID training uses 110,000 Wikipedia sentences per language across 73 languages, with 5,000 sentences each reserved for validation and testing.Sentences are at least 20 characters long.Source: §6 · p. 5
  • Word alignment is evaluated on English paired with Czech, Swedish, German, French, and Romanian, against FastAlign and Efmaral given one million additional ParaCrawl sentence pairs.The evaluation uses manually annotated alignment datasets.Source: §6 · p. 5
  • Sentence retrieval uses 3,000 WMT14 test sentences in Czech, English, French, German, Hindi, and Russian; projection fitting uses WMT14 development data.The development data contains 500–3,000 sentences per language pair.Source: §6 · p. 5
  • MT quality estimation uses English-German WMT19 data and an MLP trained to estimate HTER from source sentences, automatic translations, and corrected references.The MLP has a hidden layer of size 256 and uses mean-squared-error loss.Source: §6 · pp. 5–6
  • Language-centroid clustering is evaluated against genealogical language families using homogeneity, completeness, and V-measure.Table 2 presents the clustering of languages.Source: §6 · p. 6, Table 2 · p. 6
  • Static word embeddings are pretrained on Wikipedia and aligned using a projection trained on small bilingual dictionaries.The embedding tables cover 350,000 Romanian to 2.5 million English word types.Source: §6 · p. 6

7 Results

Results show that centering and small-data projection substantially improve sentence retrieval, while contextual representations match state-of-the-art word alignment without parallel training data. The experiments also find language identity remains recoverable, language centroids reflect language families, and sentence embeddings provide only limited direct signal for translation quality estimation.

Language Identification

  • Centering considerably decreases mBERT language-ID accuracy, while the best mBERT model slightly exceeds FastText’s 91.8% accuracy.Adversarial finetuning prevents identification only from [cls] and only marginally for mean pooling; aligned word embeddings’ 99.5% accuracy falls to 2.3% after centering, though that comparison is also an oracle experiment.Source: §7 · p. 6
  • Centering representations within languages is an oracle experiment because it requires knowing each input’s language in advance.The authors interpret the resulting decrease in language-ID accuracy as removal of language-identity information.Source: §7 · p. 6

Language Similarity

  • Language-centroid similarity tends to track language similarity, and hierarchical clustering mostly recovers language families.XLM-R performs worse than mBERT at capturing language similarity; the proposed next-sentence-prediction explanation is a hypothesis.Source: §7 · p. 7

Parallel Sentence Retrieval

  • An additional 50% error reduction from projection yields close-to-perfect sentence retrieval after centering improves accuracy dramatically.Across the 30 language pairs, XLM-R significantly outperforms all models; the retrieval comparison also includes static bilingual word embeddings.Source: §7 · p. 7, Table 3 · p. 7
  • The best sentence-retrieval accuracy comes from the 8th layer for both mBERT and XLM-R, and this pattern is consistent across models and tasks.Figure 3 shows retrieval accuracy by BERT layer for mean-pooled contextual embeddings.Source: §7 · p. 7, Figure 3 · p. 7

Word Alignment

  • mBERT and XLM-R word alignment match state-of-the-art aligners trained on large parallel corpora, enabling comparable alignment without parallel training data.Contextual embeddings capture word-level semantics, and explicit projection adds only .005 F1 points.Source: §7 · pp. 7–8, Table 5 · p. 8

MT Quality Estimation

  • Sentence-vector distances do not correlate with MT quality without centering; centering or projection yields only mild correlation, while supervised regression finds useful signal in sentence embeddings.Source-only regression gives respectable correlation, and target-only performance is almost as good as using both source and hypothesis.Source: §7 · p. 8

8 Conclusions

Unsupervised multilingual contextual embeddings capture similar semantic phenomena across languages and outperform aligned static embeddings on cross-lingual semantic similarity. They nevertheless retain a strong language-identity signal; centering and parallel-data projections can improve neutrality, but centering requires knowing the language in advance.

8 Conclusions

  • Contextual embeddings capture similar semantic phenomena across languages and consistently outperform explicitly language-neutral static embeddings on cross-lingual semantic similarity.The contextual embeddings require no tuning or adaptation for this comparison.Source: §8 · p. 8
  • Vanilla contextual embeddings retain a strong language-identity signal, demonstrated by state-of-the-art language-identification performance.The authors hypothesize that the sentence-adjacency training objective contributes to this signal.Source: §8 · p. 8
  • Language-specific centering improves neutrality to some extent, while a supervised linear projection fitted on a small parallel set can achieve stronger neutrality.The passage also reports that centering improves probing-task performance and that language centroids estimate language similarity strongly.Source: §8 · p. 9
  • Centering requires knowing the language in advance, motivating future work on representations neutral by default.Source: §8 · p. 9

A Notes on Reproducibility

The reproducibility notes report computational hardware, model sizes, and average experiment runtimes across the paper’s experiments.

A Notes on Reproducibility

  • Language identification, language-similarity, and adversarial-removal experiments used GPUs; other experiments used CPUs, and all fit within 32 GB RAM.The GPU was a GeForce GTX 1080 Ti with 11 GB memory; the CPU was an Intel Xeon E5-2630 v4 at 2.20 GHz.Source: A Notes on Reproducibility · p. 11
  • The language-ID classifier has 56k parameters, while adversarial language-ID removal uses 1.3M parameters across two classifiers per layer.Each five-seed Table 1 experiment averaged 1.38 hours; validation results are reported in Table 7.Source: A Notes on Reproducibility · p. 11
  • Sentence-retrieval projections have 590k parameters; fitting one layer and language pair averaged 7 minutes, and a retrieval experiment averaged 25 minutes.Source: A Notes on Reproducibility · p. 12
  • The minimum weighted edge-cover implementation uses SciPy’s linear-sum assignment solver, and one experiment averaged 10 minutes.Source: A Notes on Reproducibility · p. 12
  • Cosine-similarity MT QE experiments averaged 2 minutes; supervised-regression models have 197k parameters and experiments averaged 22 minutes.Source: A Notes on Reproducibility · p. 12

Related papers

Loading 2004.05160v4…