Source-linked AI summary
Building Language Models for Text with Named Entities
Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang
TL;DR
Language models can struggle to predict infrequent named entities. This paper uses entity types in a two-part language model and reports that it outperforms state-of-the-art neural language models on its benchmarks.
Problem
Named entities are often infrequent and numerous, making them difficult for language models to predict accurately.
Source: §3 · p. 3Method
The model predicts an entity type from context, then uses that type as a prior to predict the specific entity name.
Source: §1 · p. 1Results
The proposed method significantly outperforms state-of-the-art neural language models and models that add type information as a feature on the evaluated benchmarks.
Source: §1 · p. 2Takeaways & Limitations
The authors’ evaluation and case studies indicate that entity-type information also captures inherent text features.
Source: §6 · p. 9The decomposition assumes that the next token’s type can be predicted from the types of context words.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Text in many domains involves a significant amount of named entities. Predict- ing the entity names is often challenging for a language model as they appear less frequent on the training corpus. In this paper, we propose a novel and effective approach to building a discriminative language model which can learn the entity names by leveraging their entity type information. We also introduce two benchmark datasets based on recipes and Java programming codes, on which we evalu- ate the proposed model. Experimental re- sults show that our model achieves 52.2% better perplexity in recipe generation and 22.06% on code generation than the state-of-the-art language models.
1 Introduction
The paper addresses language models’ difficulty predicting diverse, infrequent entity names by using entity-type information. It proposes a two-part model and evaluates it on recipe and Java-code benchmarks.
1 Introduction
- Named entities’ wide variation and rarity make them difficult to predict, while diverse ingredient names hinder recipe generation in particular.The paper gives different oil varieties as an example of ingredient-name diversity.Source: §1 · p. 1
- A type model predicts the next entity type, then an entity composite model uses that type as a prior to predict the specific name.The model combines the two models’ probabilities through joint inference.Source: §1 · pp. 1–2
- The proposed method significantly outperforms state-of-the-art neural language models and models that add type information as a feature.The paper notes that adding type as a feature alone does not guarantee improvement.Source: §1 · p. 2
- The evaluation uses recipe and Java-code benchmarks, with auxiliary corpora providing entity-type information for both datasets.The code corpus was curated, and the code identifiers’ types were collected using an AST-based approach.Source: §1 · pp. 1–2
- The paper presents the model as a new baseline on the benchmarks and releases its code and datasets.It also describes the model’s goal as learning entity names by leveraging entity-type information.Source: §1 · p. 2
2 Related Work and Background
The section distinguishes the paper’s discriminative, neural approach from prior class-based and entity-generation models, then gives code-generation context and the LSTM setup used as an example.
Class Based Language Models
- Existing class-based language models are generative n-gram models, whereas this paper uses a discriminative neural model with joint inference over two models.The section also distinguishes its named-entity focus from word-cluster approaches.Source: §2 · p. 2
Language Models for Named Entities
- Prior entity-generation systems often copy names from input or reference material, while this work targets general free-form language generation (Gu et al., 2016; Wiseman et al., 2017).The section also contrasts its goal with task-specific recipe generation.Source: §2 · p. 3
Code Generation
- Code has open vocabulary, and the paper addresses variable names by leveraging types and scope; local regularities can also be captured by cache-based models.The section notes code’s local regularities and lower ambiguity relative to natural language.Source: §2 · p. 3
LSTM Language Model
- The paper uses LSTM language models as running examples but states that its approach can be applied with other language-model types.Recipe modeling uses Merity et al. (2017) as a black box; code modeling uses forward and backward LSTMs.Source: §2 · p. 3
3 A Probabilistic Model for Text with Named Entities
Named entities are difficult to predict because their vocabulary is large and they occur less frequently than common words. The model decomposes next-word prediction into predicting an entity type and then predicting the word using that type.
3 A Probabilistic Model for Text with Named Entities
- The proposed model separates next-word prediction into a type model and an entity composite model.The type model predicts the next entity type; the entity composite model predicts the next word given its type.Source: §3 · p. 4
3.1 Type model
- The type model replaces entities in the training corpus with their types, while retaining non-entity words, and models the resulting corpus with a language-modeling technique.The resulting vocabulary includes the original text vocabulary and entity types.Source: §3.1 · p. 4
3.2 Entity Composite Model
- The entity composite model augments word representations with type embeddings and normalizes its Softmax over words sharing the target word’s type.It predicts the actual word conditioned on the context and type information.Source: §3.2 · pp. 4–5
3.3 Training and Inference Strategies
- The two models are trained independently, then their generated probabilities are combined at inference to produce probabilities over candidate words.The type model uses type context; the entity composite model predicts the actual word given context and candidate-word type information.Source: §3.3 · p. 5
4 Experiments
The experiments evaluate language generation on recipe and source-code tasks. The supplied experiment passages also report that joint inference can use a strong language model without type information in place of the entity composite model.
4 Experiments
- The proposed model is evaluated on recipe generation and source-code generation, and the authors release the code and datasets.The recipe corpus contains individual recipes with ingredients; the source-code task is also described as involving many entity names.Source: §4 · p. 5
- Experiments using Merity et al. (2017) validate that a strong state-of-the-art language model can replace the entity composite model in joint inference.The model without type information can be combined with the type model for the final probability distribution.Source: §4 · p. 5
- Algorithm 1 takes a token sequence, word types, and an integer m as input, and lists model parameters and W1 through Wm as outputs.The algorithm includes a test or generation phase.Source: §4 · p. 5
13 end
The proposed language model uses entity-type information for recipe and Java code generation, with type acquisition and evaluation tailored to each corpus. Its reported results improve perplexity over the stated baselines, while the recipe and code experiments impose specific coverage and context boundaries.
13 end
- For recipe ingredients, types are manually annotated; Java token types are extracted with Eclipse JDT, and other applications may use ontology or lexical resources.The authors note that manually assigning types may be appropriate when automatic extraction is unavailable.Source: §13 · p. 5
4.1 Recipe Generation
- Recipe typing groups ingredients into eight super-ingredient categories and covers 76% of ingredient mentions among the 1,224 most frequent unique ingredients.The experiments omit 14,881 less-frequent unique ingredients, including some misspellings.Source: §4.1 · p. 6
- 52.2% improvement over the baseline in recipe-generation perplexity; adding type as a simple feature does not guarantee significant improvement.The comparison uses the recipe corpus.Source: Table 1 · p. 7, §4.1 · p. 7
4.2 Code Generation
- For evolving-code evaluation, projects are divided into six-month versions, and testing is restricted to previously seen methods containing new tokens.The task models edits to existing methods rather than generating entirely new methods from scratch.Source: §4.2 · p. 7
- Code prediction operates at method level, excluding code outside methods and removing user-defined String tokens from the dataset.Token types are extracted from Java ASTs using Eclipse JDT; keywords such as for and if have no associated type.Source: §4.2 · p. 7
- 22.06% better perplexity than SLP-Core, while the proposed method also outperforms forward and backward LSTM baselines.The reported improvements over the original backward and forward LSTMs are 40.3% and 63.14% lower perplexity, respectively.Source: Table 2 · p. 8, §4.2 · p. 8
5 Quantitative Error Analysis
The case studies test recipe prediction with fill-in-the-blank tasks and code prediction against a model with project-level context. Recipe accuracy gains are larger for less-frequent ingredients, while the code comparison reports lower perplexity for the proposed backward LSTM than for SLP-Core.
5 Quantitative Error Analysis
- The case studies assess generation performance and interpret the meaning of the reported perplexity results.Source: §5 · p. 8
5.1 Case Study-1: Recipe Generation
- Because lower perplexity need not imply higher accuracy, the authors test ingredient prediction with fill-in-the-blank examples and a six-choice MCQ variant.They also examine how accuracy improvement depends on ingredient training frequency.Source: §5.1 · p. 8
- The model outperforms the baselines in both free-form and MCQ ingredient fill-in-the-blank tasks, with larger accuracy gains for less-frequent ingredients.The frequency pattern is reported for ingredients such as Apple and Tomato.Source: §5.1 · pp. 8–9, Table 3 · p. 9
5.2 Case Study-2: Code Generation
- 3.40 perplexity for SLP-Core versus 2.65 for the backward LSTM on the same test methods, although SLP-Core received full project information.The paper notes that the Java-method instances provide only local context.Source: §5.2 · p. 9
6 Conclusion
The conclusion attributes improved language-model performance to using entity types to model names that are difficult to predict. It describes a type-based prediction stage followed by conditional prediction of the actual entity.
6 Conclusion
- The type model replaces same-type entity names with their shared type, reducing vocabulary size and improving type prediction accuracy.Source: §6 · p. 9
- The entity composite model uses entity type as a prior to predict the actual entity name from its conditional probability distribution.Source: §6 · p. 9
- Evaluation and case studies support the conclusion that entity-type information captures text features and improves overall language-model performance.Source: §6 · p. 9