Source-linked AI summary
Contrastive Triple Extraction with Generative Transformer
Hongbin Ye, Ningyu Zhang, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang, Huajun Chen
TL;DR
Triple extraction must capture long-term dependencies and avoid generating triples unsupported by the input. CGT addresses this with a shared transformer, triplet contrastive learning, dynamic masking, and calibration, and outperforms strong baselines across three datasets.
Problem
Existing triple extraction methods can lose long-term dependencies and generate unfaithful triples unsupported by the source text.
Method
CGT uses a shared transformer for sequence generation, triplet contrastive learning, batch-wise dynamic attention masking, and triple-wise calibration.
Results
CGT outperforms substantially strong baselines on NYT, WebNLG, and MIE and performs better on long sentences.
Takeaways & Limitations
Contrastive learning and calibration provide a framework for generating more faithful triples while retaining end-to-end sequence generation.
Takeaways & Limitations
Generated triples can have incorrect entity boundaries because end-to-end generation makes fine-grained boundary capture difficult without sequence token information.
Abstract
from arXiv · showhide
Triple extraction is an essential task in information extraction for natural language processing and knowledge graph construction. In this paper, we revisit the end-to-end triple extraction task for sequence generation. Since generative triple extraction may struggle to capture long-term dependencies and generate unfaithful triples, we introduce a novel model, contrastive triple extraction with a generative transformer. Specifically, we introduce a single shared transformer module for encoder-decoder-based generation. To generate faithful results, we propose a novel triplet contrastive training object. Moreover, we introduce two mechanisms to further improve model performance (i.e., batch-wise dynamic attention-masking and triple-wise calibration). Experimental results on three datasets (i.e., NYT, WebNLG, and MIE) show that our approach achieves better performance than that of baselines.
Introduction
CGT revisits triple extraction as sequence generation to address long-term dependency loss and unfaithful generated triples. It combines a shared generative transformer with contrastive learning and calibration, outperforming strong baselines across three datasets.
- Introduction: CGT addresses recurrent models’ difficulty capturing long-term dependencies and sequence-to-sequence models’ generation of triples unsupported by the input.These limitations hinder extraction from longer texts and can produce logically plausible but textually ungrounded facts.
- Introduction: CGT uses a single shared transformer module for encoder-decoder generation and requires no additional parameters beyond the pretrained language model.The model concatenates input and target sequences and uses partial causal masking to distinguish their representations.
- Introduction: Triplet contrastive learning treats ground-truth triples as positives and randomly corrupted triples as negatives to encourage faithful generation.The contrastive objective is jointly optimized with generation through batch-wise dynamic attention masking.
- Introduction: Batch-wise dynamic attention masking enables joint optimization, while triple-wise calibration filters remaining false triples during inference.The two mechanisms target optimization across different objects and faithfulness at inference time.
- Introduction: CGT empirically outperforms substantially strong baselines on NYT, WebNLG, and MIE, while improving long-sentence performance through better long-term dependency capture.The reported evaluation covers three benchmark datasets and compares CGT with existing triple extraction approaches.
Related Work
Prior triple extraction work includes pipeline, joint-learning, and generative approaches. These methods address task correlations and output flexibility, but faithful generation remains difficult for unconditional sequence generation.
- Related Work: Pipeline methods extract entities before relations, but separating the tasks introduces error propagation and omits their correlations.Joint-learning methods instead model task correlations implicitly or through mechanisms such as attention and reinforcement learning.
- Related Work: RNN-based encoder-decoder methods struggle with long-term dependencies, motivating transformer-based generation methods.The limitation is linked to loss of important information in longer texts.
- Related Work: Unconditional generation makes sequence faithfulness difficult to judge, motivating approaches that use factual constraints or entity-aware structures.CGT differs by using contrastive learning to encourage faithful triples.
Overview
CGT formulates triple extraction as sequence-to-sequence generation, encoding the input and serializing multiple triples into one target sequence. Its framework combines an input encoder, generative transformer, and contrastive mechanisms for faithful output.
- Preliminary: The source is the input sentence, while the target serializes each head, relation, and tail triple with [S2S SEQ], bounded by [SOS] and [EOS].The target sequence can contain multiple triples, and h_i, r_i, and t_i denote the i-th head, relation, and tail.
- Preliminary: CGT models cross-dependencies between entities and relations by treating extraction as a sequence-to-sequence task.The input and output are represented as source and target token sequences.
- Framework: The framework consists of an input encoder, a generative transformer using partial causal masking, and triplet contrastive learning.Beam search generates multiple triples during inference.
- Framework: Triplet contrastive learning is jointly optimized with generation through batch-wise dynamic attention masking and supported by triple-wise calibration for faithful generation.These mechanisms are presented as components of the CGT framework.
Our Model
The model uses a shared transformer for triple generation and combines generative training with triplet contrastive learning to improve faithfulness. Dynamic attention masking separates the two training tasks, while triple-wise calibration filters unfaithful outputs during inference.
- Generative Transformer: A 12-layer shared transformer encodes contextual features for encoder-decoder-based triple generation.The architecture uses stacked self-attention layers and a single transformer module for both encoding and decoding.
- Generative Transformer: Partial causal masking blocks attention from the source segment to the target segment while preserving source-side contextual attention.The mask matrix controls which tokens can attend to one another during triple generation.
- Triplet Contrastive Learning: Triplet contrastive learning distinguishes gold triples from corrupt triples formed by randomly replacing an entity.Each input sentence is paired with one triple, and the classifier uses the [CLS] representation to produce logits for contrastive optimization.
- Batch-wise Dynamic Attention Masking: Batch-wise dynamic attention masking assigns sampled training instances either partial causal masking for generation or all-zero masking for contrastive learning.This prevents contrastive learning from exposing generated labels through unrestricted attention when both objectives are optimized jointly.
- Training Objective: The joint objective combines generative and contrastive losses, weighted by hyperparameter α.The optimization objective is lossgenerative + αlosscontrastive.
- Triple-wise Calibration: During inference, beam-generated triples are filtered by contrastive matching scores and heuristic rules for reasonable triple structure.Triples below threshold θ are removed, and heuristics constrain patterns such as a relation following the head entity.
Experiment
CGT is evaluated on NYT, WebNLG, and MIE against generative and extractive baselines using precision, recall, and F1. It improves over generation-based baselines, remains comparable to extractive methods, and shows gains in long-sentence and ablation analyses.
- Main Results: Relative 10.2 and 27.0 F1 score improvements over PNDec and CopyMTL, respectively, were reported on NYT, while CGT achieved significant gains over generation-based baselines on NYT and WebNLG.Table 3 compares extractive models, generative approaches, and CGT settings; the reported comparisons favor CGT over all generation-based baselines.
- Main Results: CGT achieved comparable performance to extractive models such as CasRel despite the larger search space of generative models.The passage contrasts generative models’ broader output capability with extractive models’ easier optimization.
- Main Results: Relative 13.02 F1 score improvement over all baselines was reported for CGT on the MIE dataset.MIE is described as a challenging dialogue-based information-extraction dataset.
- Ablation Study: Removing the contrastive object reduced performance, supporting its role in enhancing the faithfulness of generated triples; randomly initialized CGT still outperformed generative baselines across all three datasets.This indicates the reported gains were attributed to the architecture as well as pretraining.
- Long-Sentence Analysis: All models degraded as sentence length increased, while the analysis evaluates whether CGT better captures long-term dependencies on longer inputs.The supplied passage establishes the length-related performance decay but does not include the complete CGT comparison outcome.
- Error Analysis: Error analysis classifies incorrect outputs into three categories, including distract-context, wrong-boundary, and wrong-triple cases.Wrong boundaries reflect difficulty capturing fine-grained entity boundaries in end-to-end generation.
Conclusion and Future Work
The paper revisits triple extraction as sequence generation and proposes CGT to jointly extract entities and relations while addressing long-term dependence and faithfulness. Experiments on three datasets support the approach, while future work targets stronger transformers, ontology knowledge, and additional extraction scenarios.
- Conclusion: CGT revisits triple extraction as sequence generation and jointly extracts entities and relations.The model is proposed to address long-term dependence and generate faithful triples.
- Conclusion: Experiments on three datasets demonstrate the efficacy of the proposed approach.
- Future Work: Future work will explore stronger transformer architectures, ontology knowledge injection, and applications to event extraction.Longformer is given as an example of a stronger transformer architecture.