PaperlayerView exact revision on arXiv

Source-linked AI summary

Packed Levitated Marker for Entity and Relation Extraction

Deming Ye, Yankai Lin, Peng Li, Maosong Sun

arXiv:2109.06067v5cs.CL

TL;DR

Existing span and span-pair extraction methods often ignore interrelations among spans, limiting their representations. PL-Marker packs levitated markers with neighborhood-oriented and subject-oriented strategies to model these interrelations, and it improves NER and relation-extraction results across standard benchmarks, including 4.1%-4.3% strict relation F1 gains on ACE05 and ACE04.

  • Problem

    Existing span and span-pair extraction methods focus on better PLM representations but independently process related spans, ignoring their interrelations.

  • Method

    PL-Marker strategically packs levitated markers using neighborhood-oriented packing for neighboring spans and subject-oriented packing for each subject with its candidate objects.

  • Results

    PL-Marker advances baselines on six NER benchmarks and improves strict relation F1 by 4.1%-4.3% over previous state-of-the-art models on ACE05 and ACE04 with higher speed.

  • Takeaways & Limitations

    Modeling interrelations among spans and same-subject or same-object span pairs supports state-of-the-art F1 scores and promising efficiency across NER and RE.

Abstract

from arXiv · show

Recent entity and relation extraction works focus on investigating how to obtain a better span representation from the pre-trained encoder. However, a major limitation of existing works is that they ignore the interrelation between spans (pairs). In this work, we propose a novel span representation approach, named Packed Levitated Markers (PL-Marker), to consider the interrelation between the spans (pairs) by strategically packing the markers in the encoder. In particular, we propose a neighborhood-oriented packing strategy, which considers the neighbor spans integrally to better model the entity boundary information. Furthermore, for those more complicated span pair classification tasks, we design a subject-oriented packing strategy, which packs each subject and all its objects to model the interrelation between the same-subject span pairs. The experimental results show that, with the enhanced marker feature, our model advances baselines on six NER benchmarks, and obtains a 4.1%-4.3% strict relation F1 improvement with higher speed over previous state-of-the-art models on ACE04 and ACE05.

1 Introduction

PL-Marker addresses the limitation that marker-based span methods ignore interrelations among spans or span pairs. It packs related markers to improve entity-boundary modeling and same-subject span-pair representation, yielding gains on NER and relation extraction.

  • Motivation and contribution: PL-Marker models interrelations among spans or span pairs by strategically packing levitated markers during encoding.Prior marker methods process span pairs independently during training, whereas PL-Marker packs related spans to model them integrally.
  • Packing strategies: The neighborhood-oriented strategy packs neighboring spans together to help distinguish entity boundaries, while subject-oriented packing groups each subject with its candidate objects.The subject is emphasized with solid markers and its objects use levitated markers.
  • Empirical findings: The model advances the T-Concat baseline on six NER benchmarks and improves strict relation F1 by 4.1%-4.3% over prior state-of-the-art models on ACE04 and ACE05 with higher speed.It also achieves better performance on SciERC.

2 Related Work

Prior span-representation work enhances PLM features through span pre-training, external knowledge, or structural extensions. PL-Marker instead packs levitated markers to model multiple related spans during training and inference, including NER.

  • Existing approaches: Span-representation methods are grouped into span pre-training, knowledge infusion, and structural extension approaches.These categories respectively modify pre-training objectives, incorporate external knowledge, or add reasoning modules.
  • Span Pre-training: Span pre-training masks contiguous spans or stores span information in boundary tokens for downstream tasks.This approach enhances span representations through span-level pre-training objectives.
  • Knowledge Infusion: Knowledge-infusion methods incorporate entity information from knowledge graphs, synonym resources, or Wikipedia anchor texts.The cited approaches learn or pre-train with external entity-related knowledge.
  • Structural Extension: Structural extensions add modules such as biaffine attention, graph propagation, or memory flow, while marker methods achieve strong RE results but struggle to specify more than two span pairs.PL-Marker differs from PURE by packing multiple span pairs during both training and inference and is the first cited work to apply levitated markers to NER.

3 Method

PL-Marker uses levitated markers and packing schemes to derive span and span-pair representations efficiently. Neighborhood-oriented packing groups related spans, while subject-oriented packing jointly represents a subject and its candidate objects for relation extraction.

  • Levitated markers: Levitated markers share positions with span-boundary tokens and use directional attention so paired markers aggregate span information without attending to other markers or text tokens.The markers enable parallel processing of multiple entity pairs.
  • Neighborhood-oriented Packing for Span: Neighborhood-oriented packing sorts candidate spans by start and end positions, groups them up to a maximum size, and processes the groups in multiple runs.This groups spans with the same start token to model their connections while controlling input length.
  • Neighborhood-oriented Packing for Span: For NER, PL-Marker combines packed-marker span features with T-Concat features to classify each candidate span as an entity type or non-entity.The approach targets overlapping span classification by representing candidate spans through the pretrained encoder.
  • Subject-oriented Packing for Span Pair: Subject-oriented packing places solid markers around a subject, applies levitated markers to all candidate objects, and packs the resulting spans into one instance.This fusion avoids binding two levitated-marker pairs with directional attention while modeling same-subject span pairs integrally.
  • Subject-oriented Packing for Span Pair: The relation-extraction pipeline first filters candidate entity spans with a NER model, then classifies filtered span pairs and adds an auxiliary object-entity-type loss.PL-Marker is used as a post-processing module in a two-stage model to reduce inference cost.

4 Experiment

Across flat and nested NER and end-to-end RE benchmarks, PL-Marker improves span-based baselines while examining accuracy, efficiency, and feature contributions.

  • 4.2.2 Results: PL-Marker improves T-Concat by 0.1%-1.1% absolute F1 across all six NER benchmarks and outperforms SeqTagger by 0.4%, 0.7%, and 1.9% on CoNLL03, OntoNotes 5.0, and Few-NERD.These benchmarks include both flat and nested NER settings.
  • 4.2.2 Results: Neighborhood-oriented packing outperforms random packing on all three flat NER datasets, including a 9.4% improvement on Few-NERD.Few-NERD averages 325 candidate spans, compared with 90 for CoNLL03 and 174 for OntoNotes 5.0.
  • 4.3.2 Results: 4.1%-4.3% strict relation F1 improvements over PURE are achieved on ACE05 and ACE04, respectively.With BERTBASE, PL-Marker also exceeds previous methods by 1.7% on ACE05 and 2.5% on ACE04.
  • 4.4.1 Speed of Span Model: A two-stage model matches PL-Marker on Few-NERD with 3.1x speedup, while PL-Marker sacrifices 60% speed versus SeqTagger on CoNLL03.Increasing group size to 512 slows PL-Marker because of higher Transformer complexity; the practical group size is 256.
  • 4.4.1 Speed of Span Model: Compared with PURE (Full), PL-Marker achieves 2.2x-2.8x speedup and better performance on ACE05 and SciERC.Against PURE (Approx.), it improves relation F1 for boundaries by 2.8%-4.0% on ACE05 and SciERC.
  • 4.6 Ablation Study: Removing solid markers drops F1 by 2.0%-3.8% on ACE05 and SciERC, while removing inverse relation or entity-type loss drops F1 by 0.9%-1.1% or 0.4%-0.7%, respectively.The ablations use gold entities for these comparisons.

5 Conclusion

PL-Marker combines neighborhood-oriented and subject-oriented packing to model span and span-pair representations. It achieves state-of-the-art F1 scores with promising efficiency across six NER and RE benchmarks.

  • PL-Marker uses neighborhood-oriented and subject-oriented packing strategies to obtain span and span-pair representations.
  • The model considers interrelations between spans and span pairs while maintaining promising efficiency.
  • PL-Marker achieves state-of-the-art F1 scores across six standard NER and RE benchmarks.

A Training Configuration

The experiments use Adam with warmup, size-dependent learning rates, and five random seeds. Sequence lengths, span lengths, and selected batch sizes vary by task and dataset.

  • Training Configuration: All models use Adam, 10% warmup, and five seeds: 42, 43, 44, 45, and 46.BASE models use a 2e-5 learning rate, while LARGE and XXLARGE models use 1e-5.
  • Training Configuration: Maximum expanded-sentence length is 512 for NER, while RE uses 256 for ACE05 and SciERC and 384 for ACE04.
  • Training Configuration: Maximum span length is 16 for OntoNote 5.0 and Few-NERD and 8 for the remaining datasets.
  • Training Configuration: The selected batch size is 8 for CoNLL03 NER and ACE05 RE after searching sizes from 4 to 32.

B Prompt Initialization for NER

For NER, PL-Marker marker embeddings are initialized with meaningful words rather than random values. The chosen pair uses [MASK] and entity, producing slight improvements across all six NER benchmarks.

  • Prompt Initialization for NER: Meaningful marker initialization uses the words [MASK] and entity instead of random embeddings.
  • Prompt Initialization for NER: Prompt-based marker initialization yields a slight improvement on all six NER benchmarks.

C Refine Entity Type

The RE model can refine NER entity-type predictions, but its benefit varies by dataset. The authors therefore apply this refinement only to ACE04 and ACE05, where relation predictions are more tied to entity types than in SciERC.

  • Refine Entity Type: +0.5% strict relation F1 occurs on ACE05, whereas SciERC changes by -0.9% when RE entity types refine NER predictions.Table 9 reports entity F1 and strict relation F1 for refinement versus no refinement.
  • Refine Entity Type: The most frequent entity-type pair covers 48.5% of relations in ACE05, 52.0% in ACE04, and 19.1% in SciERC.These proportions indicate that relations are more relevant to entity type in ACE05 than in SciERC.
  • Refine Entity Type: Levitated Marker classification uses two marker features, [M] and [/M], rather than the edge-token representations used by T-Concat.
  • Refine Entity Type: Table 10 compares the RE model with PURE using relation F1 on boundaries and speed under the same PURE entity input.
  • Refine Entity Type: The RE model is used to refine NER results for ACE04 and ACE05 but not SciERC.

D Inverse Relation on Baseline

Bidirectional prediction improves the symmetrical PURE baseline, while Table 10 reports relation F1 comparisons on ACE05 and SciERC.

  • D Inverse Relation on Baseline: The PURE w. InvRel. model is formed by applying the inverse relation and bidirectional prediction to baseline PURE.
  • D Inverse Relation on Baseline: Overall entity F1 scores for baselines and PL-Marker are reported on CoNLL03, OntoNotes 5.0, and Few-NERD.Table 11 reports averages across five random seeds with standard deviations as subscripts.
  • D Inverse Relation on Baseline: Overall entity and relation F1 scores for PL-Marker are reported on ACE04, ACE05, and SciERC.Table 12 reports averages across five random seeds with standard deviations as subscripts and identifies BERTBASE and ALBERTXXLARGE abbreviations.
  • D Inverse Relation on Baseline: 0.7%-2.4% relation F1 improvement is obtained on ACE05 and SciERC when bidirectional prediction is applied to symmetrical PURE.The comparison excludes the authors’ asymmetric framework.

E Detailed NER Results

The NER experiments illustrate the span representation used by NER models and report average baseline and PL-Marker scores for flat NER.

  • E Detailed NER Results: Figure 3 illustrates the span representation adopted by the NER models.
  • E Detailed NER Results: Table 11 reports average baseline and PL-Marker scores with standard deviations for flat NER.

F Detailed RE Results

The RE experiments report PL-Marker’s average performance with standard deviations across the evaluated datasets.

  • F Detailed RE Results: Average PL-Marker entity and relation F1 scores are reported for RE on ACE04, ACE05, and SciERC.The scores include standard deviations.
Loading 2109.06067v5…