Source-linked AI summary
CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning
Daojian Zeng, Ranran Haoran Zhang, Qianying Liu
TL;DR
CopyRE’s joint extraction is limited by unstable head–tail entity copying and incomplete multi-token entities. CopyMTL analyzes the copying issue, separates head and tail predictions, and adds multi-task sequence labeling; experiments report state-of-the-art performance with large-margin gains over previous approaches.
Problem
CopyRE suffers from unstable entity copying and incomplete outputs when target entities contain multiple tokens, limiting joint entity and relation extraction.
Method
CopyMTL combines a new architecture with separate head and tail entity distributions and multi-task sequence labeling to predict multi-token entities.
Results
CopyMTL achieves state-of-the-art results and outperforms previous approaches by a large margin.
Takeaways & Limitations
CopyMTL provides a strong baseline for future studies of joint entity and relation extraction.
Takeaways & Limitations
The current model can extract only a fixed number of triplets.
Abstract
from arXiv · showhide
Joint extraction of entities and relations has received significant attention due to its potential of providing higher performance for both tasks. Among existing methods, CopyRE is effective and novel, which uses a sequence-to-sequence framework and copy mechanism to directly generate the relation triplets. However, it suffers from two fatal problems. The model is extremely weak at differing the head and tail entity, resulting in inaccurate entity extraction. It also cannot predict multi-token entities (e.g. \textit{Steven Jobs}). To address these problems, we give a detailed analysis of the reasons behind the inaccurate entity extraction problem, and then propose a simple but extremely effective model structure to solve this problem. In addition, we propose a multi-task learning framework equipped with copy mechanism, called CopyMTL, to allow the model to predict multi-token entities. Experiments reveal the problems of CopyRE and show that our model achieves significant improvement over the current state-of-the-art method by 9% in NYT and 16% in WebNLG (F1 score). Our code is available at https://github.com/WindChimeRan/CopyMTL
Introduction
Joint entity–relation extraction methods avoid some weaknesses of pipeline systems, but CopyRE remains limited by unstable head–tail copying and incomplete multi-token entities. CopyMTL addresses these problems through separate head/tail distributions and multi-task learning.
- Background: Pipeline models suffer error cascading, while joint models can benefit from entity–relation interdependencies.Joint extraction is positioned as an alternative to separately recognizing entities and classifying relations.
- Prior approach: CopyRE uses a Seq2Seq framework with a decoder copy mechanism to generate relation triplets and avoid the OOV problem.It is described as a leading Seq2Seq-based joint extraction method.
- Problems of CopyRE: CopyRE’s shared head–tail distribution makes entity copying unstable and requires an unnatural mask to distinguish the two entities.Without masking, the decoder can select the same token for the tail; masking instead selects the second-highest probability token.
- Problems of CopyRE: CopyRE cannot extract multi-token entities because its decoder points only to the last token, such as predicting “Jobs” instead of “Steven Jobs.”The paper notes that multi-token entities are common and that this limitation can reduce performance.
- CopyMTL: CopyMTL separates head and tail distributions, lets tail prediction receive head information, and adds encoder-side NER to predict entity starts.The decoder points to the last token while the sequence-labeling task predicts the start token, with losses optimized jointly.
- Contributions: CopyMTL introduces a multi-task framework for multi-token entities and achieves state-of-the-art results, outperforming previous approaches by a large margin.The reported result is stated at the level of overall experimental performance rather than a specific dataset metric.
Background
CopyRE uses an encoder–decoder sequence-to-sequence architecture to generate relation-entity triplets, with attention and copying selecting source positions for entities. Its training uses softmax-based predictions and cross-entropy, but entity copying remains unstable and incomplete for multi-token entities.
- CopyRE architecture: CopyRE encodes the sentence and decodes relation-entity triplets from left to right using a one-direction LSTM initialized from the encoder.Attention combines encoder hidden states with decoder states before subsequent decoding steps.
- CopyRE architecture: Every three decoder steps predict a relation, the last token of the head, and the last token of the tail, respectively.Entity copying selects source positions, so the decoder represents entities through their predicted final tokens.
- CopyRE architecture: A mask records the predicted head entity and prevents the decoder from predicting it again at the tail step.The mask is based on the assumption that an entity cannot be both head and tail in the same triplet, but it does not contribute to minimizing cross-entropy loss.
- Training objective: At relation steps, softmax scores range over relation types; at entity steps, they range over source-sentence positions.The resulting predictions are trained by minimizing cross-entropy between outputs and gold labels.
- Training objective: CopyRE uses padding triplets (NA, NA, NA) during training and computes their shared NA confidence through a learned parameter.These padding triplets represent cases without valid relations or entities.
- Problems of CopyRE: CopyRE cannot fit its training data well, with approximate F1 scores of 0.75 and 0.40 on two datasets, and entity copying is identified as its bottleneck.The model can reach 0.84 relation F1 while entity F1 drops to 0.64, and errors often mix up head-tail order.
- Problems of CopyRE: Because CopyRE predicts only an entity’s last token, multi-token entities produce incomplete outputs rather than the full entity span.The proposed direction is to use sequence labeling and multi-task learning so entity recognition can calibrate copied entities.
Our Method
CopyMTL addresses CopyRE’s unstable head-tail copying and incomplete multi-token entities with a new decoder structure and a CRF-based multi-task tagging layer. It jointly trains entity tagging and triplet generation to stabilize copying and complete extracted entities.
- Our Method: CopyMTL combines a new model structure with an encoder sequence-labeling task to address CopyRE’s entity-copying and multi-token entity problems.The added tagging layer uses BIO labels and a CRF to identify entity spans for post-processing.
- New Structure for Entity Copying: CopyRE produces identical head and tail copying distributions across decoder time steps, making head-tail prediction depend on masking.The model selects the highest-probability token as the head and the second-highest after masking it as the tail.
- New Structure for Entity Copying: A nonlinear fused feature layer makes entity copying depend on both token position and decoder time, removing the need to rank head and tail through masking.The nonlinear transformation prevents the reduction that caused identical distributions and yields one target output to maximize.
- Sequence Labeling Layer: The CRF uses BIO tagging to recover complete multi-token entities from CopyRE’s decoded last-token outputs during inference.For an I tag, post-processing searches backward until it finds the corresponding B tag.
- Training: The encoder adds a CRF loss while the decoder uses cross-entropy loss for generated triplets, combined through a weighted multi-task objective.The tagging-loss weight is represented by λ.
Experiments
Experiments evaluate CopyMTL against prior joint-extraction methods on NYT and WebNLG using stricter exact entity-token and relation-type matching. CopyMTL achieves the best results, while revised entity copying explains major gains and multi-task learning outperforms entity-length prediction.
- Datasets and baselines: CopyMTL is evaluated on NYT and WebNLG against CopyRE, NovelTagging, and GraphRel, including OneDecoder and MultiDecoder variants.The comparison follows CopyRE settings and evaluates alternative parameter-sharing strategies.
- Evaluation: The evaluation counts outputs as correct only when relation types and all entity tokens are correct.This stricter metric addresses incomplete multi-token entities and supports fairer comparison with NovelTagging and GraphRel.
- Comparison of baselines: CopyMTL is best on both datasets, improving over GraphRel-2p by 8.8% precision and 9.2% recall on NYT, and 13.1% precision and 19% recall on WebNLG.Compared with CopyRE, CopyMTL adds 33% and 19% F1 enhancement in WebNLG and NYT, respectively.
- Comparison of baselines: CopyMTL’s decoder choice is dataset-dependent: MultiDecoder is better than OneDecoder on NYT but worse on WebNLG.The authors associate this difference with NYT’s larger dataset and MultiDecoder’s greater parameter count.
- Effects of the revised entity copying method: Replacing CopyRE’s entity-copying equation with the revised architecture yields 13% F1 boost in NYT and 31% F1 boost in WebNLG.These models use less strict evaluation and do not consider multi-token entities.
- Effects of the revised entity copying method: The revised architecture raises entity-recognition F1 by 10% on NYT and 19% on WebNLG, while relation-classification F1 increases by less than 3%.The authors state that more precise entity predictions aid relation classification during decoding.
- Training analysis: CopyRE’ reaches 97% training F1 on both datasets and continues improving through epoch 40, whereas CopyRE saturates earlier at 75% on NYT and 40% on WebNLG.Figure 4 compares the overall F1 training curves of CopyRE and CopyRE’.
- Multi-token entity models: CopyRE’5 is worse than CopyMTL in all evaluations, although both outperform GraphRel.CopyRE’5 adds entity-length prediction alongside relation classification and entity recognition.
Related Work
Joint entity-relation extraction methods include pipeline, table-filling, tagging, and Seq2Seq approaches. Each addresses the task differently but has limitations involving error propagation, computational redundancy, overlapping relations, or incomplete entities.
- Pipeline methods: Pipeline models recognize entities first and then classify relations, but component errors cascade and later stages cannot correct earlier mistakes.They also fail to capture explicit interactions between entity recognition and relation classification.
- Table filling: Table-filling methods enumerate word-pair combinations in a sentence, creating redundant computation and sparse tables that hinder learning.GraphRel is identified as a state-of-the-art table-filling model using a two-phase GCN.
- Tagging: Tagging methods assign labels to tokens, but overlapping relations can require different relation tags for one token.NovelTagging combines entity and relation information in a single tag per word.
- Seq2Seq: CopyRE uses a Seq2Seq framework with a copy mechanism to handle overlapping relations, but it cannot predict entire multi-token entities and has weak performance.The paper positions CopyMTL as resolving these problems.
Conclusions and Future Work
CopyMTL addresses CopyRE’s inaccurate entity copying and incomplete multi-token entities through a new architecture and multi-task learning framework. The authors report strong effectiveness while identifying fixed triplet capacity as future work.
- Conclusions: CopyMTL analyzes CopyRE’s inaccurate entity copying and incomplete entity generation, then addresses these problems with a new architecture and multi-task learning.The architecture targets head-tail distinction, while multi-task learning completes multi-token entities.
- Future Work: CopyMTL currently extracts only a fixed number of triplets, motivating future work to support any number of triplets.The authors also position CopyMTL as a strong baseline for future studies.