Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

1 Introduction

• Cross-lingual word embeddings can improve transfer over delexicalized models, but static embeddings do not change with context.The paper uses “static” for embeddings contrasted with contextualized ones.Source: §1 · p. 1
• Prior work found contextualized embeddings superior to traditional static embeddings in monolingual NLP, motivating their extension to cross-lingual settings.The cited work reports gains across various monolingual NLP tasks, including dependency parsing.Source: §1 · p. 1
• CLBT learns a linear transformation from contextual word alignments to project embeddings from different pre-trained BERT models into a shared semantic space for parsing.The transformed BERT encodings for each sentence are fed as input to a parser.Source: §1 · pp. 1–2
• Because CLBT uses semantic equivalence in word alignments, the authors describe it as intended to preserve word sense.Figure 1 illustrates Spanish “canal” contextual embeddings transformed to the corresponding English semantic space.Source: Figure 1 · p. 1, §1 · p. 2
• A 2.91% absolute gain in averaged LAS over previous models using static cross-lingual embeddings is reported, alongside highly competitive results against XLM with fewer training data, computing resources, and training time.Experiments use Universal Dependencies treebanks (v2.2).Source: §1 · p. 2

2 Related Work

• Static cross-lingual embedding methods include online joint-objective approaches and offline methods that retrofit pretrained monolingual embeddings into a shared semantic space.Examples cited include Klementiev et al. (2012), Kočiský et al. (2014), Guo et al. (2016), Xing et al. (2015), Lample et al. (2018), and Chen and Cardie (2018).Source: §2 · p. 2
• Contextual approaches include Mulcaire et al. (2019), who capture character-level information from multilingual data, and XLM, which adds parallel-data supervision to BERT objectives.XLM is reported to have achieved state-of-the-art results on several cross-lingual tasks.Source: §2 · p. 2
• Schuster et al. (2019) align pretrained contextual embeddings offline using averaged embeddings as type-level anchors, whereas CLBT learns directly in contextual space for word-sense preservation.The direct contextual-space transformation is explicitly designed to preserve word sense.Source: §2 · p. 2

3.1 Contextual Word Alignment

• Unsupervised bidirectional alignment of parallel text produces one-to-one contextual word pairs, using the left-most aligned word; this performed better than keeping all alignments.Sentences are tokenized with BERT’s WordPiece vocabulary before alignment.Source: §3.1 · p. 2, §3.2 · p. 2

3.2 Off-Line Transformation

• CLBT maps target-language contextual embeddings into the source-language space used by the parser.The parser is trained on source-language data and applied directly to target languages.Source: §3.2 · p. 3
• An orthogonal transformation has an SVD solution, while an unconstrained gradient-descent approximation addresses memory growth and can train on one GPU.The gradient-descent approach typically converges in several hours.Source: §3.2 · p. 3
• The transformed embeddings replace static embeddings as input to the graph-based parser, using each word’s left-most WordPiece embedding.Alternative WordPiece selection strategies showed no significant difference.Source: §3.2 · p. 3
• Nonlinear transformations brought no improvement, and orthogonality did not help the gradient-descent method.The paper also reports no significant difference among alternative WordPiece selection strategies.Source: §3.2 · p. 3

4.1 Data and Settings

• The experiments train on English UD data and apply the parser directly to 17 target languages, using Europarl alignment data for 18 selected languages.The parser uses the Biaffine Graph-based architecture; 10,000 sentence pairs are used per target language.Source: §4.1 · p. 3

4.2 Baseline Systems

• The baselines include direct mBERT embeddings, FT-SVD static embeddings, and XLM contextualized embeddings learned with cross-lingual data.FT-SVD is presented as the previous state-of-the-art; the reported results use its RNNGraph model with the same architecture as the authors’ parser.Source: §4.2 · p. 3

4.3 Comparison with Off-Line Methods

• On test sets, the approach using either SVD or GD outperforms both FT-SVD and mBERT by 2.91% in averaged LAS across all target languages.Table 1 marks statistically significant differences from mBERT at p-value < 0.05 under McNemar’s test.Source: §4.3 · p. 4, Table 1 · p. 4

4.4 Comparison with On-Line Methods

• CLBT outperforms XLM in three of four overlapping languages, though it is lower in German; CLBT uses 10,000 parallel sentences per language versus XLM’s 0.2–13.1 million.Source: §4.4 · p. 4

4.5.1 Transformation of Cross-lingual BERT Embedding

• After transformation, Spanish naturaleza embeddings are closer to English nature embeddings with similar meanings across the visualized contexts.The t-SNE plots color points by word sense and compare contextual representations before and after transformation.Source: Figure 2 · p. 5, §4.5.1 · p. 5

4.5.2 Effect of Training Data Size

• For most languages, the best parsing performance is achieved with 5,000 parallel sentences; 100 sentences yield comparable results for many Germanic and Romance languages.Figure 3 plots LAS against training-data amount for different languages.Source: Figure 3 · p. 5, §4.5.2 · p. 5

• CLBT substantially outperforms the previous state of the art in zero-shot cross-lingual dependency parsing.Source: §5 · p. 5
• Compared with XLM, CLBT uses much less parallel data and training time while achieving comparable performance.Source: §5 · p. 5
• The authors identify unsupervised cross-lingual alignment as future work requiring deeper understanding of multilingual contextual embedding geometry.Source: §5 · p. 5

• The appendix section is titled as statistics for UD version 2.2 treebanks.Source: §A.1 · p. 8, Table 3 · p. 8
• Table 3 summarizes statistics for the UD treebanks selected for the experiments.Source: Table 3 · p. 8, §A.1 · p. 8
• The table’s language-family abbreviation IE means Indo-European.Source: Table 3 · p. 8

• The parser re-implementation excludes learned embeddings to isolate the effect of pre-trained embeddings, and uses universal POS tags throughout.Source: §A.2 · p. 8
• The experiments use 768-dimensional contextualized embeddings from English and multilingual base BERT models.Source: §A.2 · p. 8
• The GD-based transformation is optimized with Adam at learning rate 0.001, β1 = 0.9, and β2 = 0.999.Source: §A.2 · p. 8

A.3 Full Results on UD Treebanks

• Table 4 compares test-set LAS for baseline models and proposed models, including a model that combines CLBT with cross-lingual FastText.The combined embeddings are concatenated as input to the parser.Source: §A.3 · p. 8, Table 4 · p. 9
• Table 4 groups languages by family and defines AVG. as the average of results from all target languages.Source: Table 4 · p. 9
• Table 5 reports test-set UAS for baseline and proposed models across target languages.AVG. denotes the average across all target languages.Source: Table 5 · p. 9