PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Meng Qu, Tianyu Gao, Louis-Pascal A. C. Xhonneux, Jian Tang

arXiv:2007.02387v1cs.LGstat.ML

TL;DR

Few-shot relation extraction must predict relations from only a few labeled examples, and the paper considers how relationships among relations can help generalization. It combines graph-informed Bayesian prototype learning with stochastic gradient Langevin dynamics, reporting effectiveness against competitive baselines in few-shot and zero-shot settings on two benchmark datasets.

  • Problem

    Few-shot relation extraction has limited labeled examples, and the paper investigates whether a global graph of relations can provide useful additional information.

    Source: §3 · p. 3

  • Method

    The approach learns posterior distributions over relation prototypes from support examples and a graph-based prior, using stochastic gradient Langevin dynamics to sample and optimize them.

    Source: §1 · p. 2

  • Results

    Experiments on two benchmark datasets show effectiveness against competitive baselines in both few-shot and zero-shot relation extraction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The global relation graph provides prior knowledge that supports transferring supervision and generalizing to relations without labeled sentences.

    Source: §1 · p. 1

    The approach assumes a global graph describing relationships among all possible relations is available as an additional data source.

    Source: §3 · p. 3

Abstract

from arXiv · show

This paper studies few-shot relation extraction, which aims at predicting the relation for a pair of entities in a sentence by training with a few labeled examples in each relation. To more effectively generalize to new relations, in this paper we study the relationships between different relations and propose to leverage a global relation graph. We propose a novel Bayesian meta-learning approach to effectively learn the posterior distribution of the prototype vectors of relations, where the initial prior of the prototype vectors is parameterized with a graph neural network on the global relation graph. Moreover, to effectively optimize the posterior distribution of the prototype vectors, we propose to use the stochastic gradient Langevin dynamics, which is related to the MAML algorithm but is able to handle the uncertainty of the prototype vectors. The whole framework can be effectively and efficiently optimized in an end-to-end fashion. Experiments on two benchmark datasets prove the effectiveness of our proposed approach against competitive baselines in both the few-shot and zero-shot settings.

1. Introduction

Few-shot relation extraction is constrained by limited labeled data, while distant supervision can introduce noisy labels. The paper uses a global relation graph to inform Bayesian prototype distributions, optimized with stochastic gradient Langevin dynamics, and reports effectiveness in few-shot and zero-shot settings.

  • Relation extraction has limited labeled data because annotation is costly, while distant supervision can generate noisy labels when entity pairs have multiple possible relations.Source: §1 · p. 1
  • Prior meta-learning methods use diverse few-example tasks to support adaptation to new relations, but performance remains unsatisfactory with limited training information.Source: §1 · p. 1
  • A global relation graph supplies prior knowledge about relations, supporting supervision transfer and generalization even without labeled sentences.Source: §1 · pp. 1–2
  • The Bayesian approach infers posterior distributions of relation prototypes from labeled support sentences and the global graph, rather than estimating a single prototype vector.Source: §1 · p. 2
  • Stochastic gradient Langevin dynamics samples the prototype posterior with noisy gradient updates to model uncertainty, and the framework is optimized end to end.Source: §1 · p. 2
  • Experiments on two benchmark datasets show the approach is effective against competitive baselines in both few-shot and zero-shot relation extraction.Source: §1 · p. 2

2. Related Work

Few-shot learning and meta-learning train models across tasks with few demonstrations, using either prototype-based or optimization-based approaches. The paper positions its relation-extraction approach against methods that omit relation connections and Bayesian methods with restricted posterior approximations or sampling requirements.

2.1. Few-shot Learning and Meta-learning

  • Few-shot and meta-learning methods learn from diverse tasks with few demonstrations, using metric-based prototypes or optimization-based adaptation.Optimization-based approaches adapt shared parameters through gradient descent, whereas the paper treats prototypes or parameters as random variables with posterior distributions.Source: §2.1 · p. 2
  • Existing Bayesian meta-learning methods either ignore class relationships, approximate posteriors with Gaussian distributions, or use SVGD, which requires a suitably designed kernel.The paper contrasts these choices with graph neural networks over a global class graph and stochastic gradient Langevin dynamics for sampling.Source: §2.1 · p. 2

2.2. Relation Extraction

  • Few-shot relation extraction reduces reliance on abundant labeled sentences, but recent methods combining meta-learning and BERT ignore connections among relations.The paper uses a global relation graph as prior knowledge to model those connections.Source: §2.2 · pp. 2–3

3. Problem Definition

Few-shot relation extraction seeks better results from only a few labeled examples, motivating the use of a global graph as an additional data source. The task predicts labels for query sentences using support examples and a subset of relations drawn from the full relation set.

  • Few-shot relation extraction remains challenging because its examples contain limited information, and the paper proposes another data source to improve results.Source: §3 · p. 3
  • The formulation assumes a global graph covering all possible relations, with links indicating relations likely to have similar semantic meanings.Source: §3 · p. 3
  • For a subset of relations, the classifier uses labeled support sentences and the global graph to predict labels for unlabeled query sentences.Source: §3 · p. 3

4. Model

The model treats each relation prototype as uncertain and conditions its posterior on support sentences and a global relation graph. It uses graph-derived priors and stochastic-gradient Langevin sampling for end-to-end optimization and prediction.

4.1. A Probabilistic Formalization

  • Query-label probabilities integrate over prototype vectors conditioned on support examples and the global graph, with a softmax scoring each query encoding against relation prototypes.The encoder–prototype dot product estimates how likely a sentence expresses a relation.Source: §4.1 · p. 3

4.2. Parameterization of the Posterior Distribution

  • The posterior combines a graph-conditioned prior with a support-sentence likelihood; a GNN embeds each relation using its graph neighbors, and those embeddings set independent Gaussian prior means.The support likelihood adapts the prior to the target relations, combining graph knowledge with the few labeled examples.Source: §4.2 · p. 4

4.3. Optimization and Prediction

  • Because posterior integration is intractable, Monte Carlo samples approximate the query log probability; graph- and support-encoding-based initialization is used to accelerate convergence.The initialization combines each relation’s graph embedding and support-set mean encoding, then subtracts the overall support-set mean encoding.Source: §4.3 · p. 5
  • Stochastic gradient Langevin dynamics draws multiple noisy prototype samples, adapting them toward support likelihood while the graph prior guides updates and preserving prototype uncertainty.Its support-likelihood update is related to MAML, but the added graph prior and noise yield posterior samples rather than a single maximum-posterior estimate.Source: §4.3 · p. 5

5. Experiment

Experiments evaluate the approach on FewRel and NYT-25 in few-shot and zero-shot settings. The approach outperforms competitive baselines, while ablations examine the graph-based prior and Langevin optimization.

5.4.1. Comparison with Baseline Methods

  • The approach outperforms competitive baselines on FewRel test, FewRel validation, and NYT-25 test results using the same BERTBASE sentence encoder.The authors attribute gains over MAML and Proto to the global relation graph and Bayesian treatment of prototype uncertainty, and gains over Versa and BMAML to the graph-based prior and Langevin optimization.Source: Table 2 · p. 6, Table 3 · p. 7, Table 4 · p. 7

5.1. Datasets

  • Evaluation uses FewRel and NYT-25; most FewRel analyses use its validation set because its test set is private, with only a final test number reported.NYT-25 is split into 10 training relations, 5 validation relations, and 10 test relations.Source: §5 · p. 5, §5.1 · p. 6

5.2. Compared Algorithms

  • The comparison includes relation-extraction and meta-learning baselines, including Pair, MTB, Proto, MAML, Versa, and BMAML.Meta-learning algorithms use BERTBASE sentence encodings and a linear softmax classifier, with the algorithms learning classifier parameters corresponding to relation prototype vectors.Source: §5.2 · p. 6

5.3. Parameter Settings

  • The graph-based prior uses a one-layer graph convolutional network, while the default Langevin setup draws 10 samples and performs 5 updates.The relation graph is built from 512-dimensional Wikidata relation embeddings using a 10-nearest-neighbor graph; other hyperparameters are selected on FewRel validation.Source: §5.3 · p. 6, §5.1 · p. 6

5.4.2. Analysis of the Graph-based Prior

  • Removing the graph-based prior significantly worsens FewRel validation results, and the graph-neural-network prior beats a feedforward variant on both datasets.The approach also achieves results described as impressive in zero-shot settings, without labeled demonstration sentences.Source: §5.4.2 · pp. 7–8, Figure 2 · p. 8

5.4.3. Analysis of the Optimization Algorithm

  • Langevin dynamics performs better than a Gaussian posterior variant using amortized variational inference on FewRel validation.The compared Gaussian variant initializes its mean from the same values used to initialize the Langevin samples.Source: §5.4.3 · p. 8, Table 6 · p. 8
  • On FewRel 5-way 1-shot validation, performance converges with around 8–10 samples and after 4–5 update steps.With only one or two samples, results are worse than without sample updates; the authors attribute this to high variance in the log-probability estimate.Source: §5.4.3 · p. 8, Figure 3 · p. 8

6. Conclusion

The paper uses a global relation graph to inform Bayesian meta-learning of relation-prototype posteriors, optimized with stochastic gradient Langevin dynamics. Experiments on two datasets support the approach, and future work will study automatic graph-structure learning and application to other tasks.

6. Conclusion

  • The approach models posterior distributions of relation prototypes, with a graph neural network on the global relation graph parameterizing the prior and Langevin dynamics optimizing the posterior.Source: §6 · p. 8
  • Experiments on two datasets support the effectiveness of the proposed approach.Source: §6 · p. 8
  • Future work will study automatic relation-graph structure learning and apply the approach to other applications, including few-shot image classification.Source: §6 · p. 8

A. Justification of the Initialization in the Langevin Dynamics

The initialization strategy seeks a high-posterior starting point for Langevin sampling, which can otherwise spend a long time in burn-in. It uses a lower bound on the prototype posterior to derive an initialization from support-set sentence encodings.

A. Justification of the Initialization in the Langevin Dynamics

  • Langevin sampling requires burn-in, so the method initializes prototype samples in a high-posterior-probability region to help them converge faster.The passage motivates this choice as a way to avoid exploring low-density areas.Source: A. Justification of the Initialization… · pp. 11–12
  • The N-way K-shot posterior combines a K-normalized support-set likelihood with the prototype prior conditioned on the global graph.The 1/K normalization is intended to keep the log-likelihood numerically stable as K increases.Source: A. Justification of the Initialization… · p. 11
  • Because the posterior log-probability contains a difficult log-partition function, the method approximates it with a lower bound derived using an inequality.Source: A. Justification of the Initialization… · p. 11
  • Under the condition that exp(E(xs)·vr) lies in [a, b], the lower bound yields an initialization based on the mean support-set encoding and each relation’s mean encoding.The authors state that this initialization has high posterior probability and is likely to speed Langevin convergence.Source: A. Justification of the Initialization… · pp. 11–12

B. Details on the Sentence Encoder

The sentence encoder marks the two entity mentions, processes the marked sentence with BERTBASE, and concatenates the embeddings of the two starting markers as its encoding.

B. Details on the Sentence Encoder

C. Comparison of Similarity Measures

The comparison finds that the preferable similarity measure depends on the number of support sentences per relation: dot product at very low shot counts and Euclidean distance at higher counts.

C. Comparison of Similarity Measures

Loading 2007.02387v1…