GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction

1 Introduction

• Cross-lingual relation and event extraction aims to produce language-agnostic representations that transfer from a model trained in one language to others.These extraction tasks identify semantic relationships in narratives and support applications such as knowledge graph completion and question answering.Source: §1 · p. 1
• Dependency structure can shorten syntactic distances between words and help model long-range dependencies, while mitigating differences in language word order.The passage contrasts sequential and syntactic distances of 15 and 4 between “fire” and “hospitalized” in one example.Source: §1 · p. 1
• GCNs aggregate information only across a fixed number of dependency hops, which may mismatch path-length distributions across source and target languages.GCNs also perform poorly on long-distance dependencies or words disconnected in the dependency tree.Source: §1 · p. 1
• GATE allows attention between all words and uses pairwise syntactic distances to weight attention, avoiding reliance on deeper layer stacking.The authors note that stacking more layers is infeasible for relatively low-resource information extraction tasks.Source: §1 · p. 2
• GATE’s self-attention encoder models structured contextual representations and propagates information among attention heads based on syntactic distances.The authors describe the encoder as suitable for transfer across typologically diverse languages because it is agnostic to word order.Source: §1 · p. 2
• GATE outperforms three recently proposed relation and event extraction methods on ACE 2005 transfer among English, Chinese, and Arabic.The analysis also finds GATE less sensitive to source-language characteristics, including word order and sentence structure.Source: §1 · p. 2

2 Task Description

• Relation extraction predicts a relation type or None for an ordered pair of subject and object entity mentions in a sentence.The relation types come from a predefined set.Source: §2 · p. 2
• Event extraction includes event detection, which identifies event triggers and their types, and event argument role labeling, which assigns roles to participating mentions.The example identifies “firing” as an Attack trigger and “Terrorists” and “hotel” as arguments with Attacker and Place roles.Source: §2 · p. 2
• The work studies sentence-level relation extraction and event argument role labeling, assuming event triggers are provided for the latter.Zero-shot transfer means no labeled examples are available in the target language; models are trained on one or more source languages and deployed in target languages.Source: §2 · p. 2
• The transfer pipeline parses each sentence into a universal dependency tree, builds multilingual word representations with universal features, encodes them with GATE, and predicts relation and role labels.The word representations combine multilingual contextual encodings with POS, dependency-relation, and entity-type embeddings; a pair of classifiers predicts the target labels.Source: §2 · p. 2

3.1 Transformer Encoder

• The Transformer encoder projects each layer's word representations into queries, keys, and values, then aggregates them with multi-head self-attention.Its attention mask is initially a zero matrix.Source: §3.1 · p. 3

3.2 Graph Attention Transformer Encoder

• GATE computes pairwise syntactic distances from universal dependency parses and uses them to guide attention, favoring tokens that are closer in the parse tree.The distance matrix uses undirected shortest paths, and the attention weights are modified with a distance-based normalization.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 Relation Extractor

• For relation extraction, GATE max-pools subject, object, and sentence representations, concatenating them for a linear classifier and Softmax prediction.The relation extractor is trained with cross-entropy loss.Source: §3.3 · p. 4

3.4 Event Argument Role Labeler

• For event argument role labeling, GATE max-pools trigger, candidate argument, and sentence representations, then concatenates them for a linear classifier and Softmax prediction.The role labeler is optimized with cross-entropy loss.Source: §3.4 · p. 4

Dataset

• ACE 2005 provides manual relation and event-argument annotations in English, Chinese, and Arabic, with dataset statistics and average sequential-versus-syntactic distances reported for candidate mentions.The distance averages ignore mention order; preprocessing follows Subburathinam et al. (2019).Source: Dataset · p. 4, Table 1 · p. 4

Evaluation Criteria

• A relation mention is correct when its type and both entity-head offsets match; an event argument role is correct when event type, offsets, and role label match a reference.Source: Evaluation Criteria · p. 4

Baseline Models

• Five baselines cover lexical mapping, GCN and recurrent encoders, standard Transformer attention, and relative-position representations; the latter uses sequential rather than syntactic distances.The three IE baselines were reimplemented because their source code was unavailable; CL Trans GCN could not use multilingual contextual encoders by design.Source: Baseline Models · pp. 4–5

Implementation Details

• Inputs combine frozen multilingual BERT features with universal POS, dependency-label, and entity-type embeddings; the encoder uses size 512, one layer, and eight heads.The distance threshold is tuned per attention head and source language; implementation uses Zhang, Qi, and Manning (2018), OpenNMT, and transformers.Source: Implementation Details · p. 5

Single-source transfer

• GATE beat all baselines in four of six transfer directions and improved over Transformer by 4.7% (EARL) and 1.3% (RE) on average.The comparison covers single-source transfer on both tasks.Source: Single-source transfer · p. 5

Multi-source transfer

• In multi-source transfer, GATE outperformed the three prior IE approaches except for RE from English–Arabic to Chinese, and remained competitive with Transformer variants.Models trained on language pairs observe more examples than in single-source transfer.Source: Multi-source transfer · p. 6

Encoding dependency structure

• GATE’s syntactic-distance attention outperformed structural position encoding on EARL, especially Arabic; dropping directionality raised English-to-Arabic F-score from 47.1 to 52.2.The authors suggest learned structural positions can be source-language sensitive; GATE models syntactic distance directly.Source: Encoding dependency structure · p. 6

Sensitivity towards source language

• GATE made roughly similar numbers of correct predictions on English and translated Chinese inputs, while CL GCN and CL RNN did better on translations.GATE performed substantially better than the baselines as the target and source sentences’ structural distance increased.Source: Figure 3 · p. 6, Sensitivity towards source language · p. 7

Ablation study

• M-BERT performed better on average than XLM-R as a word-feature source; POS and dependency-relation embeddings contributed little, while language-universal features helped with minimal training data.M-BERT and XLM-R performed better in Chinese and Arabic, respectively, and were comparable in English; the authors suggest tokenization errors may explain the limited contribution of POS and dependency labels.Source: Ablation study · p. 7

6 Related Work

• Earlier cross-lingual approaches used annotation projection, bilingual dictionaries, parallel data, machine translation, or learned common patterns.Examples include Kim et al. (2010a); Kim and Lee (2012), Hsi et al. (2016); Ni and Florian (2019), and Lin, Liu, and Sun (2017).Source: §6 · p. 7
• Subburathinam et al. (2019) and Liu et al. (2019) used GCNs for multilingual structured representations, but GCNs struggle with long-range or disconnected words.GCNs were introduced by Kipf and Welling (2017).Source: §6 · p. 7
• GATE uses syntactic distances to weight multi-head attention, addressing GCN limitations and the word-order differences that hinder cross-lingual transfer.The paper also situates this challenge alongside prior work on dependency modeling, source reordering, adversarial training, and constrained inference.Source: §6 · p. 7

7 Conclusion

• GATE models fine-grained syntactic structural information from sentence dependency parses to generate structured contextual representations.Source: §7 · p. 7
• Experiments on three languages demonstrate GATE’s effectiveness in cross-lingual relation and event extraction.Source: §7 · p. 7
• The authors identify exploring other sources of language-universal information as future work.Source: §7 · p. 7

A Dataset Details

• The experiments use the publicly available ACE05 dataset.Source: A Dataset Details · p. 10
• Table 8 lists the dataset statistics.Source: A Dataset Details · p. 10
• Table 9 reports sequential and shortest-path distances for relation and event mentions and their arguments.Source: A Dataset Details · p. 10

B Hyper-parameter Details

• The hyperparameters for all baselines and GATE are detailed in Table 10.Source: B Hyper-parameter Details · p. 10

C Tuning δ (shown in Eq. (3))

• For both tasks, δ was tuned on validation data over [1, 2, 4, 8], with the selected per-head values used in all experiments.The selected values were [2, 2, 4, 4, ∞, ∞, ∞, ∞] for event argument role labeling and [1, 1, 2, 2, ∞, ∞, ∞, ∞] for relation extraction; the authors report comparably better test-set performance with these choices.Source: C Tuning δ (shown in Eq. (3)) · p. 10

• Google Cloud Translate was used for English-to-Arabic and English-to-Chinese translation, with special symbols marking relation and event mentions and candidate arguments.About 10% of examples were dropped when the mentions could not be identified after translation.Source: D Translation Experiment · p. 10

E Error Analysis

• The comparison uses models trained on English and evaluated on Chinese for event argument role labeling and relation extraction.Event trigger type was not used as a feature in event argument role-labeling training.Source: E Error Analysis · p. 10
• GATE makes more correct predictions overall, while on event argument role labeling it produces more false positives and fewer false negatives than self-attention.The event argument role-labeling prediction rates are aggregated from the confusion matrices; the supplied passages do not give their values.Source: E Error Analysis · p. 11, Table 12 · p. 11, Figure 5 · p. 12
• For English-to-Chinese event argument role labeling, GATE notably improves predictions for Destination, Entity, Person, and Place role types.The authors relate these types to syntactic distances between event triggers and argument mentions, which they say supports their distance-based dependency hypothesis for cross-lingual transfer.Source: E Error Analysis · p. 11
• Manual inspection of 50 examples attributed errors to ACE annotation errors, knowledge absent from the input sentence, and test mentions or phrases rarely seen in training.Source: E Error Analysis · p. 11
• The confusion matrices cover event argument role labeling and relation extraction for GATE and self-attention in English-to-Chinese transfer.Source: Figure 5 · p. 12, Figure 6 · p. 13, Figure 7 · p. 14