ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

• In real-world extraction, relations in new sentences may be absent from training, motivating models that predict previously unobserved classes.Source: §1 · p. 1
• Prior zero-shot approaches require manually defined templates or repeated binary classifications over relation descriptions, which do not make relations comparable.Source: §1 · pp. 1–2
• ZS-BERT projects sentences and relation descriptions into an embedding space and aligns their vectors to represent relations semantically.It uses relation descriptions to generate representations without crowdsourcing or annotation of class attributes.Source: §1 · p. 2
• The paper formulates zero-shot relation extraction around text descriptions of both seen and unseen relations.Source: §1 · p. 2
• On two well-known datasets, ZS-BERT significantly outperforms state-of-the-art methods for unseen-relation prediction under zero-shot learning.Source: §1 · p. 2

• BERT-based relation-extraction models discussed here target supervised or few-shot settings rather than zero-shot extraction.Source: §2 · pp. 2–3
• Levy et al. (2017) use manually defined question templates for zero-shot relation extraction, requiring human effort for unseen relations.Source: §2 · p. 3
• Obamuyide and Vlachos (2018) frame zero-shot extraction as textual entailment by pairing each sentence with every relation description.Source: §2 · p. 3

• Seen and unseen relation labels form disjoint sets, with n and m denoting their respective numbers of relations.Source: §3 · p. 3
• The objective is to train a model on samples from seen relations that predicts labels from the unseen set.Source: §3 · p. 3
• Learned functions project sentences and relation descriptions into an embedding space for alignment, while new sentences are matched to projected unseen relations by nearest neighbor.Source: §3 · p. 3

4 The Proposed ZS-BERT Model

• ZS-BERT jointly aligns sentence and relation-description embeddings and classifies seen relations, then uses nearest-neighbor search to predict unseen relations.At test time, it embeds new sentences and unseen relation descriptions in the shared space.Source: §4 · pp. 3–4

4.1 Learning Relation Attribute Vectors

• Relation descriptions are encoded by Sentence-BERT into fixed attribute vectors that serve as semantic representations for seen and unseen relations.The paper gives the relation “publisher” as an example and uses its description to generate its attribute vector.Source: §4.1 · p. 4

4.2 Input Sentence Encoder

• The sentence encoder combines BERT’s contextual [CLS] and entity representations through learned layers to produce a sentence embedding.Entity representations are formed by averaging the hidden states of each entity’s tokens.Source: §4.2 · p. 4

4.3 Model Training

• Training minimizes a margin-based ranking loss that separates positive sentence–relation pairs from the closest mismatched relations, while also classifying seen relations.The margin is intended to prevent embedding collapse; mismatched relations within the batch provide negative samples.Source: §4.3 · p. 5

4.4 Generating Zero-Shot Prediction

• For zero-shot prediction, the model embeds each new sentence and unseen relation description, then selects the relation whose attribute vector is nearest.Distance is computed using negative inner product.Source: §4.4 · p. 5

5.1 Evaluation Settings

• Wiki-ZSL is filtered from Wiki-KB by removing “none” instances and relations occurring fewer than 300 times, while crowd-filtered FewRel is also used.FewRel supports zero-shot evaluation when training and testing relation labels are disjoint.Source: §5.1 · pp. 5–6
• The protocol randomly selects m unseen relations, excludes them from training, repeats random selection and splitting five times, and reports average precision, recall, and F1.The study varies m to examine its effect on performance.Source: §5.1 · p. 6
• Supervised relation-extraction baselines are adapted to produce embeddings for nearest-neighbor prediction, while ESIM and CIM treat sentence-description pairs as entailment inputs.The supervised baselines include CNN, Bi-LSTM, attentional Bi-LSTM, and R-BERT; the paper notes that Levy et al. (2017) requires predefined question templates and is not a fair comparison.Source: §5.1 · p. 6

5.2 Experimental Results

• Across two datasets and varying unseen-relation counts, ZS-BERT steadily outperforms competing methods, with a more significant advantage at m=5.The results also report that ESIM and CIM perform below ZS-BERT.Source: Table 2 · p. 6, §5.2 · pp. 6–7
• About 80% F1 is reached with 2% unseen-instance supervision, and 10% yields 90% F1 on Wiki-ZSL.F1 rises as more instances from unseen relations are made available for training.Source: Figure 4 · p. 7, §5.2 · p. 8
• At m=10, γ=7.5 and α=0.4 are recommended; α=1.0 sharply reduces performance, and inner product is identified as a suitable distance function.The paper attributes the margin parameter’s effect to the distance between positive and negative pairs.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 Case Study

• Similar relations sometimes mingle in the embeddings, while dissimilar relations separate more clearly; ZS-BERT shows larger inter-relation distances than R-BERT.The case study uses t-SNE to compare sentence embeddings from the two methods.Source: Figure 5 · p. 8, §5.3 · pp. 8–9
• The four error cases comprise wrong entity targeting, distant-supervision noise, semantically neighboring relations, and unrelated predictions potentially due to noise or information loss.The examples include a prediction suitable for other entities in the sentence and a “manufacturer” prediction for a “publisher” relation.Source: §5.3 · p. 8

6 Conclusions

• ZS-BERT steadily outperforms existing relation-extraction models under zero-shot settings.Source: §6 · p. 9
• ZS-BERT combines multi-task learning and contextual representation learning to embed input sentences and improve performance.Source: §6 · p. 9
• Effective relation embeddings might also support semi-supervised or few-shot learning by serving as auxiliary relation prototypes.Source: §6 · p. 9