PaperlayerView exact revision on arXiv

Source-linked AI summary

Packed Levitated Marker for Entity and Relation Extraction

Deming Ye, Yankai Lin, Peng Li, Maosong Sun

arXiv:2109.06067v5cs.CL

TL;DR

Existing span and span-pair extraction methods often ignore interrelations among spans, limiting their representations. PL-Marker packs levitated markers with neighborhood-oriented and subject-oriented strategies to model these interrelations, and it improves NER and relation-extraction results across standard benchmarks, including 4.1%-4.3% strict relation F1 gains on ACE05 and ACE04.

  • Problem

    Existing span and span-pair extraction methods focus on better PLM representations but independently process related spans, ignoring their interrelations.

    Source: §1 · p. 2

  • Method

    PL-Marker strategically packs levitated markers using neighborhood-oriented packing for neighboring spans and subject-oriented packing for each subject with its candidate objects.

    Source: §1 · p. 2

  • Results

    PL-Marker advances baselines on six NER benchmarks and improves strict relation F1 by 4.1%-4.3% over previous state-of-the-art models on ACE05 and ACE04 with higher speed.

    Source: §1 · p. 2, §4.3.2 · p. 6

  • Takeaways & Limitations

    Modeling interrelations among spans and same-subject or same-object span pairs supports state-of-the-art F1 scores and promising efficiency across NER and RE.

    Source: §4.3.2 · p. 6, §5 · p. 9

Abstract

from arXiv · show

Recent entity and relation extraction works focus on investigating how to obtain a better span representation from the pre-trained encoder. However, a major limitation of existing works is that they ignore the interrelation between spans (pairs). In this work, we propose a novel span representation approach, named Packed Levitated Markers (PL-Marker), to consider the interrelation between the spans (pairs) by strategically packing the markers in the encoder. In particular, we propose a neighborhood-oriented packing strategy, which considers the neighbor spans integrally to better model the entity boundary information. Furthermore, for those more complicated span pair classification tasks, we design a subject-oriented packing strategy, which packs each subject and all its objects to model the interrelation between the same-subject span pairs. The experimental results show that, with the enhanced marker feature, our model advances baselines on six NER benchmarks, and obtains a 4.1%-4.3% strict relation F1 improvement with higher speed over previous state-of-the-art models on ACE04 and ACE05.

1 Introduction

PL-Marker addresses the limitation that marker-based span methods ignore interrelations among spans or span pairs. It packs related markers to improve entity-boundary modeling and same-subject span-pair representation, yielding gains on NER and relation extraction.

Motivation and contribution

  • PL-Marker models interrelations among spans or span pairs by strategically packing levitated markers during encoding.Prior marker methods process span pairs independently during training, whereas PL-Marker packs related spans to model them integrally.Source: §1 · p. 2

Packing strategies

  • The neighborhood-oriented strategy packs neighboring spans together to help distinguish entity boundaries, while subject-oriented packing groups each subject with its candidate objects.The subject is emphasized with solid markers and its objects use levitated markers.Source: §1 · p. 2

Empirical findings

  • The model advances the T-Concat baseline on six NER benchmarks and improves strict relation F1 by 4.1%-4.3% over prior state-of-the-art models on ACE04 and ACE05 with higher speed.It also achieves better performance on SciERC.Source: §1 · p. 2

2 Related Work

Prior span-representation work enhances PLM features through span pre-training, external knowledge, or structural extensions. PL-Marker instead packs levitated markers to model multiple related spans during training and inference, including NER.

Existing approaches

  • Span-representation methods are grouped into span pre-training, knowledge infusion, and structural extension approaches.These categories respectively modify pre-training objectives, incorporate external knowledge, or add reasoning modules.Source: §2 · pp. 2–3

Span Pre-training

  • Span pre-training masks contiguous spans or stores span information in boundary tokens for downstream tasks.This approach enhances span representations through span-level pre-training objectives.Source: §2 · p. 2

Knowledge Infusion

  • Knowledge-infusion methods incorporate entity information from knowledge graphs, synonym resources, or Wikipedia anchor texts.The cited approaches learn or pre-train with external entity-related knowledge.Source: §2 · pp. 2–3

Structural Extension

  • Structural extensions add modules such as biaffine attention, graph propagation, or memory flow, while marker methods achieve strong RE results but struggle to specify more than two span pairs.PL-Marker differs from PURE by packing multiple span pairs during both training and inference and is the first cited work to apply levitated markers to NER.Source: §2 · p. 3

3 Method

PL-Marker uses levitated markers and packing schemes to derive span and span-pair representations efficiently. Neighborhood-oriented packing groups related spans, while subject-oriented packing jointly represents a subject and its candidate objects for relation extraction.

Levitated markers

  • Levitated markers share positions with span-boundary tokens and use directional attention so paired markers aggregate span information without attending to other markers or text tokens.The markers enable parallel processing of multiple entity pairs.Source: §3.1 · p. 3, §3.2 · p. 3

Neighborhood-oriented Packing for Span

  • Neighborhood-oriented packing sorts candidate spans by start and end positions, groups them up to a maximum size, and processes the groups in multiple runs.This groups spans with the same start token to model their connections while controlling input length.Source: §3.2 · p. 3
  • For NER, PL-Marker combines packed-marker span features with T-Concat features to classify each candidate span as an entity type or non-entity.The approach targets overlapping span classification by representing candidate spans through the pretrained encoder.Source: §3.2 · p. 4

Subject-oriented Packing for Span Pair

  • Subject-oriented packing places solid markers around a subject, applies levitated markers to all candidate objects, and packs the resulting spans into one instance.This fusion avoids binding two levitated-marker pairs with directional attention while modeling same-subject span pairs integrally.Source: §3.3 · p. 4
  • The relation-extraction pipeline first filters candidate entity spans with a NER model, then classifies filtered span pairs and adds an auxiliary object-entity-type loss.PL-Marker is used as a post-processing module in a two-stage model to reduce inference cost.Source: §3.3 · pp. 4–5, §3.4 · p. 5

4 Experiment

Across flat and nested NER and end-to-end RE benchmarks, PL-Marker improves span-based baselines while examining accuracy, efficiency, and feature contributions.

4.2.2 Results

  • PL-Marker improves T-Concat by 0.1%-1.1% absolute F1 across all six NER benchmarks and outperforms SeqTagger by 0.4%, 0.7%, and 1.9% on CoNLL03, OntoNotes 5.0, and Few-NERD.These benchmarks include both flat and nested NER settings.Source: §4.1.1 · p. 5, §4.2.2 · p. 6
  • Neighborhood-oriented packing outperforms random packing on all three flat NER datasets, including a 9.4% improvement on Few-NERD.Few-NERD averages 325 candidate spans, compared with 90 for CoNLL03 and 174 for OntoNotes 5.0.Source: §4.1.1 · p. 5, §4.2.2 · p. 6, Table 2 · p. 6

4.3.2 Results

  • 4.1%-4.3% strict relation F1 improvements over PURE are achieved on ACE05 and ACE04, respectively.With BERTBASE, PL-Marker also exceeds previous methods by 1.7% on ACE05 and 2.5% on ACE04.Source: §4.3.2 · p. 6

4.4.1 Speed of Span Model

  • A two-stage model matches PL-Marker on Few-NERD with 3.1x speedup, while PL-Marker sacrifices 60% speed versus SeqTagger on CoNLL03.Increasing group size to 512 slows PL-Marker because of higher Transformer complexity; the practical group size is 256.Source: §4.4.1 · p. 7
  • Compared with PURE (Full), PL-Marker achieves 2.2x-2.8x speedup and better performance on ACE05 and SciERC.Against PURE (Approx.), it improves relation F1 for boundaries by 2.8%-4.0% on ACE05 and SciERC.Source: §4.4.2 · pp. 7–8

4.6 Ablation Study

  • Removing solid markers drops F1 by 2.0%-3.8% on ACE05 and SciERC, while removing inverse relation or entity-type loss drops F1 by 0.9%-1.1% or 0.4%-0.7%, respectively.The ablations use gold entities for these comparisons.Source: §4.6 · p. 8

5 Conclusion

PL-Marker combines neighborhood-oriented and subject-oriented packing to model span and span-pair representations. It achieves state-of-the-art F1 scores with promising efficiency across six NER and RE benchmarks.

  • PL-Marker uses neighborhood-oriented and subject-oriented packing strategies to obtain span and span-pair representations.Source: §5 · p. 9
  • The model considers interrelations between spans and span pairs while maintaining promising efficiency.Source: §5 · p. 9
  • PL-Marker achieves state-of-the-art F1 scores across six standard NER and RE benchmarks.Source: §5 · p. 9

A Training Configuration

The experiments use Adam with warmup, size-dependent learning rates, and five random seeds. Sequence lengths, span lengths, and selected batch sizes vary by task and dataset.

Training Configuration

B Prompt Initialization for NER

For NER, PL-Marker marker embeddings are initialized with meaningful words rather than random values. The chosen pair uses [MASK] and entity, producing slight improvements across all six NER benchmarks.

Prompt Initialization for NER

C Refine Entity Type

The RE model can refine NER entity-type predictions, but its benefit varies by dataset. The authors therefore apply this refinement only to ACE04 and ACE05, where relation predictions are more tied to entity types than in SciERC.

Refine Entity Type

  • +0.5% strict relation F1 occurs on ACE05, whereas SciERC changes by -0.9% when RE entity types refine NER predictions.Table 9 reports entity F1 and strict relation F1 for refinement versus no refinement.Source: C Refine Entity Type · p. 13, Table 9 · p. 13
  • The most frequent entity-type pair covers 48.5% of relations in ACE05, 52.0% in ACE04, and 19.1% in SciERC.These proportions indicate that relations are more relevant to entity type in ACE05 than in SciERC.Source: C Refine Entity Type · p. 13
  • Levitated Marker classification uses two marker features, [M] and [/M], rather than the edge-token representations used by T-Concat.Source: Figure 3 · p. 13
  • Table 10 compares the RE model with PURE using relation F1 on boundaries and speed under the same PURE entity input.Source: Table 10 · p. 13
  • The RE model is used to refine NER results for ACE04 and ACE05 but not SciERC.Source: C Refine Entity Type · p. 13

D Inverse Relation on Baseline

Bidirectional prediction improves the symmetrical PURE baseline, while Table 10 reports relation F1 comparisons on ACE05 and SciERC.

D Inverse Relation on Baseline

  • The PURE w. InvRel. model is formed by applying the inverse relation and bidirectional prediction to baseline PURE.Source: D Inverse Relation on Baseline · p. 13
  • Overall entity F1 scores for baselines and PL-Marker are reported on CoNLL03, OntoNotes 5.0, and Few-NERD.Table 11 reports averages across five random seeds with standard deviations as subscripts.Source: Table 11 · p. 14
  • Overall entity and relation F1 scores for PL-Marker are reported on ACE04, ACE05, and SciERC.Table 12 reports averages across five random seeds with standard deviations as subscripts and identifies BERTBASE and ALBERTXXLARGE abbreviations.Source: Table 12 · p. 14
  • 0.7%-2.4% relation F1 improvement is obtained on ACE05 and SciERC when bidirectional prediction is applied to symmetrical PURE.The comparison excludes the authors’ asymmetric framework.Source: D Inverse Relation on Baseline · p. 14

E Detailed NER Results

The NER experiments illustrate the span representation used by NER models and report average baseline and PL-Marker scores for flat NER.

E Detailed NER Results

F Detailed RE Results

The RE experiments report PL-Marker’s average performance with standard deviations across the evaluated datasets.

F Detailed RE Results

  • Average PL-Marker entity and relation F1 scores are reported for RE on ACE04, ACE05, and SciERC.The scores include standard deviations.Source: F Detailed RE Results · p. 14

Related papers

Loading 2109.06067v5…