PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-NER : Contextual Phrase Generation at Scale

Himanshu Gupta, Shreyas Verma, Santosh Mashetty, Swaroop Mishra

arXiv:2109.08079v4cs.IRcs.CLcs.LG

TL;DR

NER research has focused on entity recognition, with less attention to generating relevant entity context that may not appear in a sentence. The paper introduces CONTEXT-NER and the EDGAR10-Q financial-report dataset, then evaluates task models and EDGAR10-Q pre-finetuning on finance benchmarks. EDGAR-T5 outperforms vanilla T5 and BloombergGPT-50B on the reported downstream finance datasets.

  • Problem

    Research has largely focused on NER performance, while generating relevant entity context—including phrases absent from the sentence—has received limited attention.

    Source: §1 · p. 1

  • Method

    The paper defines CONTEXT-NER and introduces EDGAR10-Q, and uses phrase generation to create questions that a transformer machine-reading-comprehension model answers.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    EDGAR-T5 outperforms vanilla T5 by 10.81 points and BloombergGPT-50B by 15.81 points on downstream finance datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EDGAR10-Q provides a challenging resource for systematic study of generating relevant phrases that describe entities in sentences.

    Source: §1 · p. 2

    Limited computational resources prevented fine-tuning models larger than 1B parameters, and budget constraints reduced ChatGPT evaluation to a subset of the test set.

    Source: §8 · p. 9

Abstract

from arXiv · show

Named Entity Recognition (NER) has seen significant progress in recent years, with numerous state-of-the-art (SOTA) models achieving high performance. However, very few studies have focused on the generation of entities' context. In this paper, we introduce CONTEXT-NER, a task that aims to generate the relevant context for entities in a sentence, where the context is a phrase describing the entity but not necessarily present in the sentence. To facilitate research in this task, we also present the EDGAR10-Q dataset, which consists of annual and quarterly reports from the top 1500 publicly traded companies. The dataset is the largest of its kind, containing 1M sentences, 2.8M entities, and an average of 35 tokens per sentence, making it a challenging dataset. We propose a baseline approach that combines a phrase generation algorithm with inferencing using a 220M language model, achieving a ROUGE-L score of 27% on the test split. Additionally, we perform a one-shot inference with ChatGPT, which obtains a 30% ROUGE-L, highlighting the difficulty of the dataset. We also evaluate models such as T5 and BART, which achieve a maximum ROUGE-L of 49% after supervised finetuning on EDGAR10-Q. We also find that T5-large, when pre-finetuned on EDGAR10-Q, achieve SOTA results on downstream finance tasks such as Headline, FPB, and FiQA SA, outperforming vanilla version by 10.81 points. To our surprise, this 66x smaller pre-finetuned model also surpasses the finance-specific LLM BloombergGPT-50B by 15 points. We hope that our dataset and generated artifacts will encourage further research in this direction, leading to the development of more sophisticated language models for financial text analysis

1 Introduction

NER research has concentrated on identifying entities, while generating relevant descriptions for them—even when those phrases are absent from the sentence—has received limited attention. The paper introduces CONTEXT-NER and EDGAR10-Q, evaluates baseline and supervised approaches, and examines downstream finance tasks and instruction tuning.

1 Introduction

  • CONTEXT-NER generates a relevant phrase describing an entity, whether or not the phrase appears in its sentence.EDGAR10-Q contains 1M sentences and 2.78M entities from financial reports; its sentences average approximately 35 tokens and include difficult numerical entities.Source: §1 · p. 2
  • EDGAR10-Q contains 1M sentences and 2.78M entities; its sentences average approximately 35 tokens, are complex and uncommon in benchmark datasets, and pose a challenge for SOTA models.Contextual phrases can be difficult to retrieve without adequate domain knowledge.Source: §1 · p. 2
  • 49.01% is the highest reported supervised ROUGE-L score, compared with 30.31% for one-shot ChatGPT and 27.59 for the baseline.The baseline uses sentence syntax to generate questions and identify relevant phrases; the supervised result is reported across generative models.Source: §1 · p. 2
  • EDGAR-T5 outperforms vanilla T5 by 10.81 points and BloombergGPT-50B by 15.81 points on downstream finance datasets.The paper also examines CONTEXT-NER versus OpenIE and instruction tuning on EDGAR10-Q.Source: §1 · pp. 2–3
  • The paper also investigates CONTEXT-NER versus OpenIE, downstream finance tasks, and instruction tuning on EDGAR10-Q.Instruction tuning improved performance by 2% points.Source: §1 · pp. 2–3

2 EDGAR10-Q Dataset

EDGAR10-Q is a large financial NER-context dataset built from SEC filings, with entity labels and sentence-level examples. Its scale and sentence complexity distinguish it from benchmark datasets, and the train and test sets are described as roughly equal in difficulty.

Dataset Description

  • Nearly 650× more documents and 780× more entities than Wikicoref are reported for EDGAR10-Q.Table 5 compares EDGAR10-Q with other NER datasets.Source: Table 5 · p. 3, §2 · p. 3

Dataset Creation

  • EDGAR10-Q was created from quarterly 10-Q and annual 10-K reports from 2019–2021, with entities labeled using NER context labels.The reports are curated, and their labels serve as gold labels for the dataset.Source: §2 · p. 3

Dataset Description

  • 49% of sentences contain one entity, while 2% contain five or more; average sentence length rises as entity count increases.The dataset contains nearly 18.7K documents and 1M sentences.Source: §2 · p. 3
  • The train and test sets are described as roughly equal in difficulty, and Table 6 presents their supervised-learning split.Source: §2 · p. 3, Table 6 · p. 3

3 Baseline Approach

The baseline generates candidate phrases from sentence syntax, frames questions around entities, and uses machine reading comprehension to identify relevant descriptions; a one-shot ChatGPT evaluation provides a comparison.

3 Baseline Approach

  • OpenIE's verb-centered extraction is poorly suited to financial sentences where auxiliary verbs such as “is” convey little entity information, motivating phrase generation and machine reading comprehension.The baseline uses generated questions to identify relevant entity descriptions.Source: §3 · pp. 3–4

3.1 Phrase generation

  • Phrase generation extracts simple subject-object phrases and complex phrases around prepositions, using nouns or pronouns and their modifiers.The method uses part-of-speech tags and gives “loan amount” and “connection with refinance” as extracted phrases.Source: §3.1 · p. 4

3.2 Machine Reading Comprehension for Baseline

  • Questions are framed by entity type and noun phrases, then MRC selects an answer containing the target entity, preferring the highest-confidence duplicate; failures can leave no response.When generated questions fail, the method tries “what is” plus the entity; if responses identify different entities, it returns no response.Source: §3.2 · pp. 4–5

3.3 ChatGPT Evaluation

  • ChatGPT is evaluated in a one-shot setting without finetuning, after receiving a task definition and example.The evaluation uses the test set directly.Source: §3.3 · p. 5

4 Experiments and Results

The experiments compare a rule-based baseline and one-shot ChatGPT with supervised generative models, using ROUGE-L and exact match to assess output quality. Supervised models achieve higher F1 than the baselines, while exact-match and no-match analyses reveal remaining errors.

4.1 Experimental Setup

  • Experiments use BERT base for the baseline and one-shot ChatGPT; ROUGE-L precision, recall, F1, and exact match assess generated outputs.The ChatGPT setup specifies gpt-3.5-turbo with a 256-token maximum.Source: §4.1 · p. 5

4.2 Results

  • ChatGPT's 30.31% F1 exceeded the baseline's 27.59%, and their performance difference grew with sentence length.ChatGPT was evaluated on a subset of the test set because of budget limitations; the subset's distribution was similar to the full test set.Source: §4.2 · pp. 5–6
  • 47.67%, 47.77%, and 49.01% F1 for T5, Flan, and BART Base exceeded ChatGPT's 30.31%.Table 9 reports supervised learning scores for different models.Source: §4.2 · p. 6, Table 9 · p. 6
  • 49.23% T5 Large and 47.70% Tk-Instruct Large nearly matched BART's 49.01% F1, with no significant gains as parameter counts rose.The passage reports these large-model scores as nearly the same as BART's.Source: §4.2 · p. 6
  • Exact-match scores remained low for the baseline and ChatGPT, while supervised scores were higher; Flan and Tk-Instruct led supervised models, but overall exact match remained low.Tables 10 and 11 report exact-match results for the baseline and ChatGPT, and for supervised models, respectively.Source: §4.2 · p. 6, Table 10 · p. 6, Table 11 · p. 6
  • 47.96% vs. 19% no-match for baseline versus ChatGPT; supervised models also had around 20% no-match.Tables 12 and 13 report no-match results for the baseline and ChatGPT, and for supervised models, respectively.Source: §4.2 · p. 6, Table 12 · p. 6, Table 13 · p. 6

5 Analysis

The analysis compares CONTEXT-NER with OpenIE, tests instruction tuning, and evaluates whether EDGAR10-Q pre-finetuning improves finance-task performance.

5.1 CONTEXT-NER vs. OpenIE

  • On the evaluated EDGAR10-Q examples, Stanford and AllenNLP OpenIE failed to identify relations, contextual phrases, or entities, with long-range dependencies posing difficulty.The comparison presents OpenIE outputs for EDGAR10-Q examples in Table 14.Source: §5.1 · p. 7, Table 14 · p. 7

5.2 Marginal Improvement with Instruction Tuning

  • Instruction tuning Tk-Instruct improves scores by roughly 2% across sentence categories and by 4% in sentences with 5+ categories.Figure 2 reports ROUGE-L F1 across sentence categories and overall.Source: §5.2 · p. 7, Figure 2 · p. 8

5.3 Effect of EDGAR10-Q on downstream tasks

  • EDGAR-T5 establishes SOTA on FiQA, FPB, and Headline, outperforming vanilla T5 and BloombergGPT-50B across all three tasks.On FiQA, it beats vanilla T5 and BloombergGPT by 5.53 and 5.35% points; on FPB, by 23.92 and 28.62% points; Headline F1 is 93.55% versus 90.55% and 82.20%.Source: §5.3 · p. 8, Table 16 · p. 8

6 Related Work

Related work spans NER, relation extraction, and information extraction approaches, while this paper targets contextual phrases for entities.

6 Related Work

  • Etzioni et al. introduced a schemaless OpenIE approach to extract facts from text, but its assumption that relations link two entities poses challenges for financial data.Source: §6 · p. 8
  • Prior relation-extraction work includes zero-shot MRC for unseen relations [34], multi-turn question answering [40], and joint entity-relation models [57].The passage notes that performance of one joint approach suffers on unseen relations [57].Source: §6 · p. 8
  • The paper aims to extract entities and their contexts, a more complex scenario than relation-based approaches.The related work also mentions decaNLP [52] and entity-extraction frameworks including Stanford CoreNLP, Spacy, NLTK, and Flair [44].Source: §6 · p. 8

7 Conclusion

The conclusion presents CONTEXT-NER and EDGAR10-Q as resources for studying entity-context phrase extraction. It reports promising baseline and supervised experiments and identifies further work as a path for continued investigation.

7 Conclusion

  • CONTEXT-NER targets extracting relevant phrases for entities, and EDGAR10-Q is presented as a large, complex finance dataset for research.The paper describes its baseline as demonstrating feasibility and its supervised experiments as promising.Source: §7 · pp. 8–9
  • The comparison with GPT-3 showcased challenges posed by the dataset.The conclusion also reports promising results from finetuning pretrained language models in a supervised setting.Source: §7 · p. 9
  • The authors expect the dataset and study to encourage further investigation and advancement in the field.The conclusion characterizes EDGAR10-Q as a valuable research resource in finance.Source: §7 · pp. 8–9

8 Limitations and Future Work

The authors identify compute, ChatGPT evaluation, and dataset coverage as boundaries on the study and suggest corresponding future research directions.

8 Limitations and Future Work

  • Limited compute prevented finetuning models larger than 1B on EDGAR10-Q.Future work could test other instruction-based or chain-of-thought reasoning approaches and more powerful models.Source: §8 · p. 9
  • Budget constraints made the ChatGPT evaluation set smaller than the test set and led the authors to omit few-shot evaluation.The authors note that investigating few-shot ChatGPT evaluation could be interesting.Source: §8 · p. 9
  • The dataset covers reports from limited markets and years, so the authors propose expanding it to study generalizability and temporal dynamics.The proposed expansion includes reports from different markets and more recent years.Source: §8 · p. 9

9 Ethical Considerations

The authors report that source-document licenses permit research use, modification, and redistribution; the dataset contains no real-life names, and the methods do not specifically emphasize or reduce sociopolitical bias.

9 Ethical Considerations

  • Source-document licenses permit use, modification, and redistribution in a research context.Source: §9 · p. 9
  • The authors report that the dataset contains no real-life names.Source: §9 · p. 9
  • The methods do not specifically emphasize or reduce sociopolitical bias.Source: §9 · p. 9

Checklist

The checklist points readers to the paper’s training and compute details and reports that new dataset-related assets are available through public repositories and released under stated terms.

Checklist

  • Training details are provided in §4.1, but the authors did not estimate total compute.Source: Checklist · p. 16

A Dataset Checklist

B.1 Dataset documentation and intended uses

The documentation describes EDGAR10-Q as a benchmark for generating entity-associated phrases from financial reports, with a large sentence/entity collection and documented intended uses, scope, and construction.

B.1 Dataset documentation and intended uses

  • EDGAR10-Q targets CONTEXT-NER: generating a phrase that describes an entity and may be absent from its sentence, using complex financial text and numerical entities.Source: §B.1 · p. 18

B.2 Composition

  • The dataset contains approximately 1M instances and 2.78M entities; its standardized train, validation, and test splits use an 80/10/10 ratio.Source: §B.2 · p. 19
  • The annual and quarterly reports may contain spelling and grammatical mistakes, and personally identifiable information was removed.Source: §B.2 · p. 19

B.3 Collection process

  • The collection process scrapes financial documents, and the dataset documentation describes cleaning and labeling procedures by reference to §E.Source: §B.3 · p. 20, §B.4 · p. 20

B.5 Uses

  • The dataset could support traditional NER and related tasks, but it captures only the US market.Source: §B.5 · p. 21

C Extended Related Work

D EDGAR10-Q Datacard

  • EDGAR10-Q comprises 1M sentences and 2.8M entities from reports of the top 1500 publicly traded companies, with an average of 35 tokens per sentence.Source: D EDGAR10-Q Datacard · p. 22

E Dataset Creation

  • The documented creation workflow retrieves SEC filings, parses and filters them, extracts sentences, entities, phrases, and question-answer records, then writes company-level CSV files.Source: E Dataset Creation · p. 23

F.1 Raw Data

The appendix describes EDGAR10-Q’s record fields and supervised examples, then details a baseline that generates noun phrases, turns them into questions, and uses machine reading comprehension to associate phrases with entities.

F.1 Raw Data

  • Each raw record includes sentence, entity, entity type, labels, extracted phrases, generated questions, selected key, confidence score, and MRC answer fields.Source: §F.1 · p. 24

F.2 Supervised modeling data

  • Supervised modeling concatenates the entity with its sentence as input and uses one of the labeled phrases as the output.Source: §F.2 · p. 25

G Baseline approach

  • The baseline uses noun phrases because financial sentences often contain uninformative auxiliary verbs such as “is,” unlike relation-centered OpenIE.Source: §G.1 · p. 26

G.1 Phrase Generation

  • Simple phrase extraction finds noun or pronoun subjects and objects and adds compounds or adjectives; complex extraction starts from prepositions and searches on both sides.Source: §G.1 · p. 26

G.2 Machine Reading Comprehension Model

  • The baseline frames entity-type-specific questions from extracted phrases and uses a pretrained distilled BERT MRC model to associate entities with descriptions without further MRC training.Source: §G.2 · pp. 27–28

G.3 Baseline Method Code:

  • The implementation combines sentence parsing, Flair entity tagging, noun-phrase and preposition-phrase extraction, and filtering of phrases that contain entities.Source: §G.3 · p. 28

H Other Results

Figure 5 reports precision and recall scores for instruction tuning Tk-Instruct on EDGAR10-Q, with trends similar to its F1 score.

H Other Results

  • Figure 5 reports precision and recall scores for instruction tuning Tk-Instruct on EDGAR10-Q.The passage does not provide the score values.Source: H Other Results · p. 29
  • Precision and recall show trends similar to the F1 score.Source: H Other Results · p. 29
  • Figure 5’s caption describes a zero-shot open information extraction process using question generation and reading comprehension.Source: Figure 5 · p. 29

Related papers

Loading 2109.08079v4…