PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Dependency Parsing with Unsupervised Cross-Lingual Bootstrapping

Lalita Lowphansirikul, Attapol Rutherford, Jian Gang Ngui, Sarana Nutanong, Peerat Limkonchotiwat

arXiv:2609.37883v1cs.CL

TL;DR

Dependency parsing is difficult to transfer across languages because syntactic tasks depend on structural similarity, and previous approaches have relied on supervised fine-tuning. This paper bootstraps multilingual PLMs through unsupervised contrastive learning with syntax-aware augmentation, MLM, and layer aggregation. It reports improved zero-shot parsing in low-resource settings and higher scores in parameter-free tree probing.

  • Problem

    Cross-lingual dependency parsing transfer depends on syntactic similarity, while prior approaches to reducing this constraint remain limited to supervised fine-tuning.

    Source: §1 · p. 1

  • Method

    The method bootstraps multilingual PLMs with unsupervised contrastive learning, syntax-aware sentence augmentation, an MLM auxiliary loss, and multi-layer representation aggregation.

    Source: §1 · pp. 1–2

  • Results

    In the (0,50) data bucket, the bootstrapped model with last-layer representation and OV augmentation improves average UAS by +1.27 and LAS by +1.29; probing also finds higher UAS and UUAS.

    Source: §4 · pp. 3–4

  • Takeaways & Limitations

    On seen languages, uplift in probed UUAS is positively correlated with downstream LAS for bootstrapped models with MLM loss (λ = 0.1).

    Source: §4 · p. 4

    The bootstrapping study uses only the UD English EWT treebank as its single source and leaves more diverse text sources and non-English data unexplored.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Pre-trained language models (PLMs) with encoder-based architectures have shown impressive capabilities in zero-shot cross-lingual transfer for various language understanding tasks. However, applying this technique to dependency parsing remains a significant challenge due to its syntactic nature. To boost model generalizability across linguistic typologies, we propose a cross-lingual unsupervised bootstrapping method to improve syntactic knowledge within the PLM. We show that our method achieves a significant improvement in zero-shot parsing performance in low-resource languages. Analysis of these bootstrapped models uncovers increased robustness in recognizing syntactic structures, evidenced by higher scores in parameter-free tree probing tests.

1 Introduction

Cross-lingual dependency parsing transfer depends on syntactic similarity, while prior approaches have remained limited to supervised fine-tuning. This paper proposes unsupervised bootstrapping to improve syntactic representations and zero-shot parsing.

1 Introduction

  • Syntactic transfer depends on structural similarity, including shared subject–verb–object word order across languages.Earlier approaches include joint training, difficulty-based selection, meta-learning, adapters, and word permutation, but remain limited to supervised fine-tuning.Source: §1 · p. 1
  • The proposed bootstrapping combines syntax-aware augmentation, masked-language modeling, and multi-layer representation aggregation.The method uses contrastive learning to improve representation similarity between sentences and syntactically modified counterparts.Source: §1 · pp. 1–2
  • Bootstrapped multilingual PLMs improve average UAS and LAS, especially on languages represented in pre-training data.The paper also reports higher UAS and UUAS in tree-probing tests than for vanilla pre-trained models.Source: §1 · p. 2

2 Proposed Method

The method bootstraps multilingual PLMs through unsupervised contrastive learning on monolingual English sentences and their syntax-altered counterparts. It combines subtree-rotation augmentation, an MLM auxiliary loss, and representations aggregated across layers.

2 Proposed Method

  • The framework uses unsupervised contrastive learning with monolingual English data and syntax-altered positive samples that retain the sentence’s intended meaning.Its three components are syntax-aware augmentation, contrastive learning with an MLM auxiliary loss, and multi-layer representation aggregation.Source: §2 · p. 2

2.1 Syntax-aware Sentence Augmentation

  • Subtree rotation changes word order while retaining intended meaning; Object-Verb augmentation turns “The cat is scratching a fluffy mat” into “The cat a fluffy mat is scratching.”Each augmentation is paired one-to-one with its original sentence, and the resulting sentences serve as positive-sample candidates.Source: §2.1 · p. 2, Figure 2 · p. 2

2.2 Contrastive Learning with MLM Auxiliary Loss

  • Contrastive learning uses original sentences as anchors, augmented sentences as positives, and the other in-batch samples as negatives.Masked-language-model prediction is added as a weighted auxiliary loss to learn grammatical understanding from the original sentence.Source: §2.2 · p. 3

2.3 Multi-layer Representation Aggregation

  • Multi-layer aggregation averages mean-pooled representations from selected layers rather than using only the last layer.The authors describe this as producing a more robust overall representation without additional training cost.Source: §2.3 · p. 3

3 Experimental Setup

Experiments use multilingual encoders and a UDapter-based parser, then evaluate zero-shot parsing across low-resource languages alongside parameter-free tree probing.

3 Experimental Setup

  • Experiments use mBERT and XLM-R with Object-Verb augmentation, contrastive learning, an MLM auxiliary-loss weight, and last- and sixth-layer representation aggregation.Other tested encoders include XLM-100, XLMR, DistilBERT, and MiniLM; combinations were tuned for similarity on English EWT data.Source: §3 · p. 3
  • A UDapter-based parser is fine-tuned with vanilla or bootstrapped models across 13 high-resource treebanks, with model weights frozen.The adapters are contextualized by language typological features.Source: §3 · p. 3
  • Evaluation covers 28 low-resource languages using average UAS and LAS, plus parameter-free tree probing to assess captured syntax.The probing evaluation uses unsupervised tree parsing.Source: §3 · p. 3

4 Results and Discussion

Bootstrapping improves zero-shot parsing overall and especially in low-resource groups, while probing results are positive but gains vary across settings and word-order groups.

  • For L12 + SynAug OV + MLM, overall zero-shot parsing gains were +0.16 UAS and +0.25 LAS.Source: §4 · p. 3, Table 1 · p. 4
  • In the (0,50) low-resource group, last-layer OV bootstrapping improved averages by +1.27 UAS and +1.29 LAS; multi-layer learning gave almost similar improvement without penalty.Source: §4 · p. 3, Table 1 · p. 4
  • Gains varied by word order: results were mixed for OV languages but mostly positive for VO and languages with no dominant order.Source: §4 · p. 4, Figure 3 · p. 4
  • Bootstrapped mBERT scored higher on probed UAS and UUAS than vanilla mBERT in single- and multi-layer settings; Spearman correlations between probing-UUAS uplift and downstream LAS were 78.33 and 66.67, respectively (p < 0.05).Source: §4 · p. 4, Table 2 · p. 4
  • Wikipedia-pretrained multilingual encoders showed higher gains than CC-100-pretrained encoders, while bootstrapped XLMR parsing did not improve and its probed results declined.Source: §4 · p. 4, Table 3 · p. 4

5 Conclusion

The paper bootstraps multilingual PLMs with syntax-aware augmentation, MLM auxiliary loss, and layer-wise representation aggregation. It reports zero-shot parsing gains and a positive link between parsing and probing results.

  • The proposed contrastive-learning method combines syntax-aware sentence augmentation, contextual knowledge preservation, and multi-layer representation aggregation.Source: §5 · p. 4
  • 1.27 UAS and 1.29 LAS gains on seen languages demonstrate zero-shot parsing improvements with mBERT, with gains also reported for other multilingual PLMs.Source: §5 · p. 4
  • Parameter-free tree probing reveals a positive link between parsing and downstream performance, motivating further study of intrinsic properties in encoder- and decoder-based multilingual PLMs.Source: §5 · pp. 4–5

Limitations

The study is limited to bootstrapping from one English treebank, and several augmentation choices and broader evaluations remain unexplored.

Limitations

  • Bootstrapping uses only the UD English EWT treebank, leaving integration of diverse text sources and non-English sources unexplored.Other augmentation categories, sample weights, and multiple modifications to the same sentence are also left for future work.Source: Limitations · p. 5
  • The effects on tasks such as relation extraction and question answering, and on other target-task fine-tuning approaches, remain future investigations.The authors also report variability across multilingual encoders and call for a systematic, quantifiable way to identify combinations that enhance syntactic generalizability.Source: Limitations · p. 5

A.1 Evaluation Setup

The evaluation bootstraps multilingual encoders, fine-tunes a UDapter-based parser, and tests zero-shot parsing and parameter-free tree probing.

A.1 Evaluation Setup

  • mBERT, XLM-R, XLM-100, DistilBERT, and MiniLM were bootstrapped, with representations aggregated from the last layer alone or the last and sixth layers.Object-Verb augmentation and MLM auxiliary-loss weights were selected through tuning; model selection used similarity between original and syntax-modified English sentences.Source: §A.1 · p. 7
  • Bootstrapped models initialized UDapter-based parsers fine-tuned across 13 high-resource treebanks, then evaluated on 28 low-resource languages using average UAS and LAS.UDapter uses typology-conditioned trainable adapters while freezing the model weights.Source: §A.1 · p. 7
  • Parameter-free tree probing decodes trees from token-representation similarities to evaluate captured syntactic structure.Bootstrapping and validation sentences came from English EWT train and test sets.Source: §A.1 · p. 7

A.2 Syntax-aware sentence augmentation on English

The augmentation procedure reorders words within their parsed subtrees using three syntactic features: verb–object order, adposition–noun order, and adjective–noun order.

A.2 Syntax-aware sentence augmentation on English

  • Words are rotated according to three syntactic features in parsed English sentences, with swapped words kept within their respective subtrees.Figure 4 illustrates Noun-Adj and adposition reordering; dashed arcs mark relations associated with reordering.Source: §A.2 · p. 7, Figure 4 · p. 7
  • Verb–object reordering identifies objects through core and selected non-core relations, and includes verb dependents such as auxiliaries and adverbs.The listed relations include iobj, obj, xcomp, ccomp, cop, expl, obl, compound, aux, and advmod.Source: §A.2 · p. 7
  • Adposition–noun and adjective–noun order are identified using the case and amod relations, respectively.The figure shows examples of Noun-Adj and adposition reordering.Source: §A.2 · p. 7

A.3 Model training details

The appendix describes training details and reports additional parsing comparisons across augmentation settings and multilingual encoders.

A.3 Model training details

  • Training varied learning rate, epoch count, batch size, and warmup ratio; batch-size search was restricted for XLM-R and XLM-100 because of memory constraints.Reported run times differ by encoder, and parser fine-tuning took approximately 31–38 hours for the listed models, except XLM-100 at 38 hours.Source: §A.3 · pp. 7–8

A.4 Additional experiments on the choices of augmentation function and MLM weight

  • Additional mBERT results compare average UAS/LAS changes across augmentation choices and MLM-loss weights; Assyrian and Amharic are excluded because most tokens are unknown to its tokenizer.Table 4 describes augmentation comparisons against the baseline for bootstrapped single-layer mBERT without MLM.Source: §A.4 · p. 8, Table 4 · p. 8

A.5 Zero-shot dependency parsing performance of other multilingual encoders.

  • The appendix reports zero-shot parsing comparisons for multilingual encoders on 28 target languages, with average UAS/LAS changes relative to UDapter.The reported tables cover Distil-mBERT, XLM-R, and Multilingual MiniLM, as well as encoders pretrained on Wikipedia or CC-100.Source: §A.5 · p. 8, Table 6 · p. 9, Table 5 · p. 9
  • Per-language probing UAS/UUAS and zero-shot UAS/LAS comparisons between vanilla and bootstrapped mBERT are also tabulated.The captions identify test languages and mark whether languages were included in mBERT pretraining; the supplied passages do not give cell values.Source: Table 8 · p. 10, Table 9 · p. 11

Related papers

Loading 2609.37883v1…