PaperlayerView exact revision on arXiv

Source-linked AI summary

Finding Universal Grammatical Relations in Multilingual BERT

Ethan A. Chi, John Hewitt, Christopher D. Manning

arXiv:2005.04511v2cs.CLcs.LG

TL;DR

The paper asks what properties enable mBERT’s cross-lingual effectiveness and whether its representations contain shared syntactic structure. It extends structural probing and analyzes the resulting syntactic subspaces without dependency-label supervision. The results provide evidence for overlapping cross-lingual syntactic representations and dependency clusters that largely agree with Universal Dependencies.

  • Problem

    Although mBERT shows strong cross-lingual performance, the properties of its representations that support this effectiveness remain unclear.

  • Method

    The paper applies structural probes to recover dependency-tree distances across languages and analyzes head-dependent difference vectors in the learned syntactic subspaces.

  • Results

    mBERT’s syntactic subspaces overlap across languages, and unsupervised dependency clusters largely overlap with the Universal Dependencies taxonomy.

  • Takeaways & Limitations

    The findings provide evidence that mBERT induces universal grammatical relations without explicit supervision.

  • Takeaways & Limitations

    The study quantitatively evaluates cross-lingual dependency-distance transfer but only qualitatively evaluates unsupervised dependency-label emergence via t-SNE.

Abstract

from arXiv · show

Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations are shared cross-lingually. To better understand this overlap, we extend recent work on finding syntactic trees in neural networks' internal representations to the multilingual setting. We show that subspaces of mBERT representations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representations of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evidence suggests that even without explicit supervision, multilingual masked language models learn certain linguistic universals.

1 Introduction

The paper investigates why mBERT transfers effectively across languages by probing whether its representations contain shared syntactic structure. It finds cross-lingual syntactic subspaces and unsupervised dependency clusters that largely align with Universal Dependencies.

  • Motivation: Prior work shows multilingual masked language models achieve strong cross-lingual performance, motivating analysis of what makes their representations effective across languages.The cited applications include XNLI and dependency parsing.
  • Approach: The study extends structural probing to examine whether mBERT learns cross-lingual representations of syntactic structure.It evaluates both recovered dependency tree structure and properties not used to train the probe, such as dependency labels.
  • Findings: Structural probes extract considerably more syntax from mBERT than baselines in 10 languages.This extends structural-probe results from a monolingual setting to multilingual mBERT.
  • Findings: Probes trained on one language recover syntax in other languages, indicating that mBERT’s syntactic subspaces overlap across languages.The result is described as zero-shot cross-lingual recovery of syntax.
  • Findings: Unsupervised head-dependent representations form dependency clusters that largely overlap with the Universal Dependencies taxonomy.The clusters are obtained from a syntactic subspace without explicit dependency-label supervision.
  • Conclusion: Together, the experiments provide evidence that mBERT induces universal grammatical relations without explicit supervision.The analysis combines zero-shot structural probing with unsupervised dependency-label discovery.

2 Methodology

The paper uses structural probing to learn linear syntactic subspaces from mBERT representations, then studies the transformed vectors themselves. Experiments evaluate an 11-language, 110M-parameter multilingual BERT model against chain and randomly initialized baselines.

  • Structural probe: Structural probing learns a linear transformation whose squared distances reconstruct dependency-tree distances from mBERT word representations.The transformation defines a lower-dimensional syntactic subspace, with each transformed vector represented in that space.
  • Analysis target: The analysis departs from prior work by treating probe-transformed word vectors as objects of study, not only their pairwise distances.This enables analysis of dependency representations beyond the property used to train the probe.
  • Experimental settings: Experiments probe 11 languages in Universal Dependencies v2 using mBERT’s 110M-parameter BERT-Base, Multilingual Cased model.The language set is constrained by available large UD datasets and mBERT pretraining coverage and is not typologically representative of all human languages.
  • Evaluation: Evaluation reports Spearman correlation for predicted versus true word-pair distances and UUAS for correctly reconstructed undirected tree edges.The baselines include an untrained mBERT-like model with reinitialized contextual attention layers and an exclusively left-to-right chain analysis.

3 Does mBERT Build a Syntactic Subspace for Each Language?

The structural probe recovers syntactic tree structure across the examined languages, with strongest performance in middle-to-late mBERT layers and limited gains from very high probe rank. These results support language-specific syntactic subspaces while motivating tests of cross-lingual overlap.

  • Experimental question: The experiments test whether mBERT forms language-specific syntactic subspaces as a prerequisite for a shared cross-lingual syntactic subspace.Probes are trained across 11 languages, varying both probe rank and mBERT layer.
  • In-language results: 22 UUAS points and 0.175 DSpr. are the average improvements over both baselines across all examined languages.The probe recovers syntactic trees in every language investigated.
  • In-language results: Western European languages achieve 9.3 points better absolute UUAS and 6.7 points better improvement over baseline.The corresponding DSpr. gains are 0.041 absolute and -0.013 relative.
  • Layer and rank effects: The structural probe most effectively recovers tree structure from the 7th or 8th mBERT layer across the examined languages.Figure 4 reports UUAS across layers 1–12 with probe maximum rank 128.

4 Cross-Lingual Probing

The experiments test whether mBERT’s syntactic subspaces transfer across languages and whether their geometric similarity predicts transfer quality. Results support a shared cross-lingual syntactic space, including for less common adjective-order relations.

  • Cross-lingual transfer: Structural probes are evaluated with direct transfer, hold-one-out transfer, and a joint subspace trained on all languages.The evaluation asks whether syntax learned in one language also predicts syntax in another, or across all languages together.
  • Cross-lingual transfer: Syntactic subspaces transfer across all examined languages, with joint training improving UUAS by 19 points and DSpr by 0.156 over baseline.Holdout training improves UUAS by 16 points and DSpr by 0.137, while best-source transfer improves UUAS by 14 points and DSpr by 0.128.
  • Cross-lingual transfer: Joint syntactic spaces account for 88% of recoverable UUAS improvement and 89% of DSpr improvement over the best baseline.These results quantify how much syntactic information is represented cross-lingually.
  • Subspace similarity: Lower average principal angles correlate with better cross-lingual transfer, with an average Spearman correlation of 0.78.The analysis compares language orderings by subspace angle and by probing transfer performance.
  • Extrapolation testing: Accuracy does not differ significantly between primarily-prenominal and primarily-postnominal training languages for either adjective ordering.Postnominal adjectives score approximately 2 points better than prenominal adjectives in both training-language categories.

5 mBERT Dependency Clusters Capture Universal Grammatical Relations

The paper analyzes projected head-dependent vectors to determine whether mBERT’s syntactic subspaces organize dependencies into linguistically meaningful clusters. The clusters are multilingual and largely align with Universal Dependencies while also revealing distinctions based on word order and finer grammatical patterns.

  • 5.1 Methodology: Head-dependent vectors are computed after projection into syntactic subspaces and visualized with t-SNE across monolingual, holdout, and joint cross-lingual settings.For each dependency, the method computes vdiff = B(hhead − hdep) and projects the resulting vectors into two dimensions.
  • 5.2 Experiments: A Spanish holdout subspace captures fine-grained Spanish dependency structure despite never seeing Spanish during probe training.The holdout subspace is trained on languages other than Spanish.
  • 5.3 Results: Projected difference vectors form clusters reflecting linguistic dependency characteristics, and same-label dependencies share clusters across languages.The clusters largely overlap with Universal Dependencies labels but do not exactly reproduce them.
  • 5.3 Results: mBERT separates prenominal from postnominal adjectives, while preserving the UD subject–object distinction and grouping indirect objects with direct objects.It also groups some adjunct arguments with subjects or objects depending on sentence position.
  • 5.3 Results: mBERT groups relative-pronoun and subordinating-marker relations together, separates negative adverbial modifiers, and splits determiners into several clusters.Definite articles are further separated by sentence-initial position, while expletive subjects form a distinct cluster.
  • 5.5–5.6 Discussion: The unsupervised clustering view differs from supervised dependency-label probing because the structural probe receives no label supervision.Its linear transformation may nevertheless miss representation aspects that causally influence the model.

6 Related Work

Related work studies cross-lingual alignment, shared linguistic subspaces, and structural probing in multilingual and contextual representations. This paper differs by focusing specifically on syntactic subspaces and by analyzing dependency structure without direct label supervision.

  • Cross-lingual embedding alignment: Prior studies report geometric alignment and cross-lingual transfer in independently trained or multilingual contextual representations.These studies motivate examining whether mBERT contains shared representational components across languages.
  • Analysis with the structural probe: Earlier structural-probe work visualized syntax in BERT and found that mBERT distances separate clusters of word senses for identical word types.The present work extends structural probing to multilingual syntactic subspaces.
  • Syntactically motivated dimensionality reduction: Related dimensionality-reduction work found UD part-of-speech clusters in ELMo using direct POS supervision, unlike this paper’s unlabeled tree-structure supervision.The distinction concerns the supervision used to shape the representation analysis.
  • Understanding representations: CCA studies analyze overlap across full mBERT representations, whereas this paper evaluates overlap specifically within syntactic subspaces.The two analyses therefore need not yield contradictory conclusions.

7 Discussion

The analysis provides evidence that mBERT shares syntactic representation space across languages and forms cross-linguistic grammatical-relation clusters without dependency-label supervision. However, the methods cannot establish causal implications or fully separate subword overlap from deeper parameter sharing.

  • mBERT shares portions of its representation space between languages at a syntactic level, including fine-grained cross-lingual syntactic distinctions.
  • Cross-linguistic grammatical-relation clusters qualitatively overlap considerably with the Universal Dependencies formalism despite lacking dependency-label supervision.
  • The UUAS metric alone is insufficient for evaluating structural-probe accuracy because it can reward correct edge selection without accurately recreating longer-path distances.
  • The methods cannot disentangle whether transfer reflects subword overlap or more fundamental parameter sharing for all language pairs.
  • The unsupervised dependency-label analysis is qualitative, and interpreting t-SNE plots remains subject to known confounds.
  • The methods cannot support claims about whether the discovered properties have causal implications for the model’s decisions.

A.1 Visualization of All Relations

The appendix visualizes dependency representations across all studied relations and languages using t-SNE, including cross-lingual and model-specific projections. These visualizations compare cluster structure against gold Universal Dependencies labels.

  • The all-relations visualization includes 36 dataset relations, whereas the top-relations visualization excludes punctuation and covers 79.36% of dependencies.
  • The cross-lingual syntactic-subspace visualization colors dependency head-dependent pairs by their 43 gold Universal Dependencies labels.
  • In MBERTRAND, edge clusters are primarily type-based, while parallel syntactic functions across languages show insignificant overlap.
  • The appendix also evaluates simpler dimensionality-reduction strategies by projecting syntactic difference vectors with PCA instead of t-SNE.
  • The MBERTRAND visualization uses head-dependent representations and a syntactic subspace derived from MBERTRAND, with colors denoting gold dependency types.
  • PCA-based two-dimensional visualization shows no significant trends, unlike the t-SNE visualization of the same syntactic difference vectors.

B.2 PCA for Dimensionality Reduction

PCA preserves major and some fine-grained groups in syntactic difference vectors, but it also introduces clusters driven more by semantics or language identity than syntax. The contrast is visible in several relation-specific examples.

  • PCA projections retain major groups and fine-grained distinctions, including a division between common nouns and pronouns.
  • The nsubj and obj clusters overlap under PCA, whereas the syntactically projected visualization separates them more clearly.
  • Postnominal adjectives split into several language-specific clusters under PCA instead of forming one coherent cluster.
  • PCA produces small monolingual clusters without common syntactic meaning, mainly for Chinese, Arabic, Farsi, and Indonesian.
  • Figure 10 visualizes syntactic differences across all studied languages after projection to 32 dimensions using PCA.

C Additional Experiment Settings

The additional experiment settings describe repeated transfer experiments, their UUAS and DSpr. reporting conventions, subspace-angle analyses, and the ten Universal Dependencies corpora used.

  • The syntactic subspace has 32 dimensions, matching the dimensionality used in the PCA comparison.
  • The structural-probe results are reported using Undirected Unlabeled Attachment Score (UUAS).
  • Table 3 reports UUAS and DSpr. for transfer between all language pairs, with every run repeated three times.
  • Runs are marked when performance ranges exceed 2 UUAS points or 0.02 DSpr.
  • Table 4 reports pairwise mean principal angles between language-specific subspaces, while Table 5 relates subspace-angle ordering to transfer accuracy using UUAS and DSpr.
  • Experiments use Arabic-PADT, Chinese-GSD, Czech-PDT, English-EWT, Finnish-TDT, French-GSD, German-GSD, Indonesian-GSD, Latvian-LVTB, Persian-Seraji, and Spanish-Ancora.

E t-SNE reproducibility

The paper uses multiple t-SNE perplexity settings to reduce interpretability concerns in its qualitative visual analysis. The accompanying table compares language orderings based on subspace angles and structural-probe transfer accuracy.

  • Multiple t-SNE plots use perplexities of 5, 10, 30, and 50 to address interpretability concerns.These settings span the range suggested by van der Maaten and Hinton (2008).
  • Table 5 compares language orderings from subspace angles with orderings from structural-probe transfer accuracy.The comparison is repeated for both structural-probe evaluation metrics.
  • Figure 11 visualizes English and French head-dependent pairs projected into a syntactic subspace learned from English syntax trees.Colors represent gold Universal Dependencies labels across the different perplexity settings.
Loading 2005.04511v2…