ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

1 Introduction

• Conventional pre-training objectives do not explicitly model relational facts, although those facts are important for understanding whole texts.At least 40.7% of relational facts in a human-annotated Wikipedia-document corpus require extraction from multiple sentences.Source: §1 · p. 1
• Understanding an entity requires considering its connections to other entities and its diverse relations across the document.Culiacán appears in several sentences and is used to find the answer to where Guadalajara is.Source: §1 · p. 2
• Understanding relations can require multi-hop reasoning that combines clues from multiple sentences.The example infers that the Panamerican Highway is in Mexico by linking two clues.Source: §1 · p. 2
• ERICA uses contrastive learning with entity discrimination to identify an inferable tail entity and relation discrimination to distinguish semantically close relations.Document-level distantly supervised entity pairs let the relation task account implicitly for complex reasoning chains.Source: §1 · p. 2
• ERICA improves BERT and RoBERTa over baselines on relation extraction, entity typing, and question answering, especially in low-resource settings.The experiments are presented as evidence that ERICA improves entity and relation understanding and captures in-text relational facts.Source: §1 · p. 2

2 Related Work

• PLMs are pre-trained on unlabeled text and fine-tuned for tasks, but typically treat words as basic units and overlook entities and their relations.Earlier approaches include universal language representations (Dai and Le, 2015; Howard and Ruder, 2018) and Transformer-based PLMs such as BERT and XLNet (Devlin et al., 2018; Yang et al., 2019).Source: §2 · p. 2
• Knowledge-guided PLMs add KG facts but can miss textual relations and knowledge about out-of-KG entities.Prior work adds entity information through pre-training objectives (Xiong et al., 2019; Wang et al., 2019; Sun et al., 2020; Yamada et al., 2020) or fuses text and KG information in model structures (Zhang et al., 2019; Peters et al., 2019; Wang et al., 2020).Source: §2 · p. 2
• Text-focused pre-training has modeled entity mentions or within-sentence relations, whereas ERICA considers interactions among multiple entities and relations at document level.Earlier methods include span-based pre-training (Sun et al., 2019; Joshi et al., 2020; Kong et al., 2020; Ye et al., 2020) and relation-aware modeling (Soares et al., 2019; Peng et al., 2020).Source: §2 · pp. 2–3

3.1 Notations

• KG distant supervision labels document entity pairs, yielding positive tuples that include intra- and inter-sentence cases requiring cross-sentence, multi-hop, or coreferential reasoning.Pairs without relation annotations in the KG are assigned no_relation.Source: §3.1 · p. 3

3.2 Entity & Relation Representation

• ERICA mean-pools tokens for each entity mention, averages its occurrence representations into a global entity representation, and concatenates two entity representations for a relation.Source: §3.2 · p. 3

3.3 Entity Discrimination Task

• ED conditions on a head entity and relation to identify the true tail among document entities, training positive entity pairs to be closer than negatives.The model receives the relation name and head mention before the document.Source: §3.3 · p. 3

3.4 Relation Discrimination Task

• RD pulls text-based representations of entity pairs with the same relation closer, using document-level supervision and negatives sampled from all tuples T.The authors position this against sentence-level relation supervision and aim to cover complex reasoning chains.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5 Overall Objective

• Entity masking yielded no performance gain; the authors conjecture document-level supervision makes mention memorization harder, while joint masked-LM training aims to avoid forgetting.Source: §3.5 · p. 4

4 Experiments

• Adding no_relation entity pairs as negative samples further improves performance, which the authors attribute to greater training-pair diversity.Source: §4 · p. 4

4.1 Distantly Supervised Dataset Construction

• The distant-supervision dataset contains 1,000,000 Wikipedia documents with more than 4,000 relations annotated from Wikidata.Documents with fewer than 128 words, 4 entities, or 4 relational triples are excluded, and entity pairs in RE and QA test sets are omitted to avoid leakage.Source: §4.1 · p. 5

4.2 Pre-training Details

• ERICABERT and ERICARoBERTa are trained for 2,500 steps with batch size 2,048 and learning rate 3 × 10^-5.The setup uses AdamW, warmup for the first 20% of steps, linear decay, and up to 64 negative samples per document for LRD.Source: §4.2 · p. 5

4.3 Relation Extraction

• ERICA significantly outperforms all baselines on DocRED across training splits, with larger margins on smaller training sets.The authors attribute the gains to better document-level relational reasoning.Source: Table 1 · p. 5, §4.3 · p. 6
• ERICA achieves almost comparable results to CP on sentence-level relation extraction and does not impair PLMs’ sentence-level relation understanding.The evaluation covers TACRED and SemEval-2010 Task 8.Source: Table 2 · p. 5, §4.3 · p. 6

4.4 Entity Typing

• ERICA outperforms all baselines on FIGER entity typing.The authors associate this result with improved entity representations from entity-level and relation-level contrastive learning.Source: Table 3 · p. 6, §4.4 · p. 6

4.5 Question Answering

• ERICA outperforms baselines on WikiHop in both standard and masked settings, with significant improvements in the masked setting.The authors interpret masked-setting gains as evidence of stronger multi-hop reasoning rather than reliance on entity-mention shortcuts.Source: Table 4 · p. 6, §4.5 · p. 7
• ERICA outperforms all baselines on extractive QA across SQuAD, TriviaQA, and NaturalQA.The reported testbed uses 10% and 100% training splits; 1% results are in the appendix.Source: Table 5 · p. 6, §4.5 · p. 7

5.1 Ablation Study

• Combining LED and LRD further improves DocRED and FIGER, while on WikiHop LED dominates and LRD slightly hurts; extra -NSP pretraining contributes little.Both intra- and inter-sentence pairs contribute to LRD; adding the objectives only during fine-tuning gave no gain, while distant-supervision pretraining increases training-example diversity.Source: §5.1 · pp. 7–8

5.2 Effects of Domain Shifting

• Pre-training with entities retagged to match DocRED’s entity distribution performs better than original ERICA; performance also improves as relation-domain diversity increases.ERICA is less competent at handling relations unseen in its corpus, whose relations are distantly annotated through a knowledge graph with a predefined relation set.Source: §5.2 · p. 8, Table 7 · p. 8, Figure 4 · p. 8

5.3 Effects of Pre-training Data’s Size

• ERICA performs better as the proportion of its original pre-training dataset increases.The comparison varies the data amount from 10% to 70%.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Effects of Methods for Entity Encoding

• ERICA improves performance with both Mean Pool and Entity Marker; Entity Marker performs better at larger training scale, while Mean Pool is stronger in low-resource settings.Training with a dataset sharing similar entity distributions is more helpful for Mean Pool.Source: Table 8 · p. 8, §5.4 · p. 9

5.5 Embedding Visualization

• On DocRED dev-set t-SNE plots, ERICABERT’s entity and relation embeddings cluster more compactly by category than BERT’s, whose distributions appear random.The visualization labels entity and relation categories; it includes the 10 most frequent relations.Source: §5.5 · p. 9, Figure 6 · p. 9

• ERICA is a general framework that uses contrastive learning to improve PLMs’ entity and relation understanding.Source: §6 · p. 9
• ERICA outperforms baselines on relation extraction, entity typing, and question answering, especially under low-resource settings.Source: §6 · p. 9
• The authors conclude that ERICA helps PLMs capture in-text relational facts and synthesize information about entities and their relations.Source: §6 · p. 9

• The section introduces training details for relation extraction, entity typing, and question answering.Source: A Training Details for Downstream Tasks · p. 13
• The downstream tasks covered are relation extraction, entity typing, and question answering.Source: A Training Details for Downstream Tasks · p. 13
• All models are implemented based on Huggingface transformers.Source: A Training Details for Downstream Tasks · p. 13

A.1 Relation Extraction

• Relation extraction experiments use DocRED, TACRED, and SemEval-2010 Task 8, with training partitions of 1%, 10%, and 100%.Source: §A.1 · p. 13

A.2 Entity Tying

• Entity typing uses the distantly supervised FIGER dataset and an ERNIE-based implementation, fine-tuned for three epochs.Source: §A.2 · p. 13

A.3 Question Answering

• Question-answering evaluation covers multi-choice WikiHop and extractive MRQA datasets SQuAD, TriviaQA, and NaturalQA; reported training splits vary by task.Source: §A.3 · p. 13

B Generalized Language Understanding (GLUE)

• ERICABERT and ERICARoBERTa achieve comparable GLUE dev-set performance to their original models, suggesting LED and LRD do not hurt general language understanding.Source: B Generalized Language Understanding… · p. 13

C Full results of ablation study

• Table 10 lists the full ablation results for DocRED, WikiHop, and FIGER.The passage identifies the results table but does not provide its values or comparisons.Source: C Full results of ablation study · p. 13

D Joint Named Entity Recognition and Relation Extraction

• Joint NER and RE aims to identify entities in text and the relations between them.Source: D Joint Named Entity Recognition and… · p. 13
• ERICA outperforms all baselines on CoNLL04 and ADE in joint NER and RE experiments.The passage says the experiments replace SpERT's BERT and RoBERTa encoders with ERICABERT and ERICARoBERTa, respectively.Source: D Joint Named Entity Recognition and… · p. 14, Table 12 · p. 14
• SpERT (Eberts and Ulges, 2019) is the base model, evaluated on CoNLL04 and ADE.The implementation is modified while keeping all settings the same.Source: D Joint Named Entity Recognition and… · p. 14