Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation

1 Introduction

• The baseline SotA model scores less than 28 F1 on tail entities, versus 86 F1 across all entities, while resolving rare entities poses a growing scalability challenge.Wikidata has 15 times more entities than Wikipedia; seeing each tail entity 100 times would require training data 1,500 times Wikipedia’s size.Source: §1 · pp. 1–2
• Bootleg meets or exceeds state-of-the-art on three NED benchmarks and beats prior state-of-the-art by more than 40 F1 points on Wikipedia tail entities.Its reasoning patterns yield tail-example lifts of 18, 56, 62, and 45 F1 points for entity memorization, type consistency, KG relation, and type affordance, respectively.Source: §1 · p. 3
• A 2D regularization scheme combines entity, type, and relation signals according to entity frequency, improving unseen-entity performance by 13.6 F1 points over standard regularization.The motivation is that tail entities often have non-tail types and relations, so the model should balance signals differently for each entity.Source: §1 · p. 3
• Weak labeling increases Wikipedia training mentions by 1.7x and yields a 2.6 F1-point lift on unseen entities.The method uses pronouns and alternative names on an entity’s Wikipedia page to label mentions.Source: §1 · p. 3
• Bootleg’s learned representations improve TACRED relation extraction by 1.0 F1 point and lift optimized production search and assistant tasks by 8%.The paper also reports that relation-only and type-only models each reach 3.3x the prior SotA baseline’s performance on unseen entities using 1% of the space.Source: §1 · p. 3

2 NED Overview and Reasoning Patterns

• Tail entities occur fewer than 11 times in training, torso entities 11–1,000 times, and head entities more than 1,000 times; the baseline scores under 28 F1 on the tail and 86 F1 overall.Entities seen more often permit memorization of distinguishing contextual cues.Source: §2 · p. 3

2.1 Four Reasoning Patterns

• Entity memorization matches text to facts about a specific entity, while type consistency uses textual cues to favor entities of a shared type; type consistency covers 12% of tail examples.Examples include “Nebraska” distinguishing Lincoln, Nebraska and “or” suggesting Lincoln and Ford are both car companies.Source: §2.1 · p. 4
• KG relation reasoning combines textual cues with known links between candidates, covering 23% of tail examples; type affordance uses text associated with an entity type and covers 76%.For example, “in” cues the capital-of relation for Lincoln and Logan County, while “ordered” favors a drink over a location for Manhattan.Source: §2.1 · p. 4
• Entity, relation, and type signals are available in structured knowledge bases, allowing rare entities to leverage textual patterns learned from other entities sharing their types or relations.The challenge is combining these more general signals with discriminative entity signals.Source: §2.1 · p. 4

3 Bootleg Architecture for Tail Disambiguation

• Bootleg’s three modules target entity memorization, type consistency, KG relations, and type affordance by processing entity representations with word context and entity connections.The architecture combines Ent2Ent, Phrase2Ent, and KG2Ent; each aims to encode the reasoning patterns.Source: §3 · p. 4, Figure 2 · p. 5, §3.2 · p. 6

3.1 Encoding the Signals

• For each candidate, Bootleg combines its entity, type, and relation embeddings into a vector, forming entity matrix E alongside contextual word matrix W.A multilayer perceptron forms each candidate vector from its entity, type, and relation embeddings.Source: §3.1 · p. 5

3.2 Bootleg Model Architecture

• Phrase2Ent attends from entity representations to text, Ent2Ent models dependencies between entities, and KG2Ent uses pairwise connectivity to transfer representations between related entities.Final scoring uses the highest score from the graph-updated representation or the text-attended representation.Source: §3.2 · p. 6

3.3 Improving Tail Generalization

• The regularization scheme masks the full entity embedding with probability inversely related to popularity, encouraging entity-specific signals for frequent entities and general type and relation patterns for rare ones.When masked, the model receives type and relation embeddings without the entity embedding.Source: §3.3 · p. 6, §3.3.1 · p. 7

3.3.1 Regularization

• Inverse-popularity 2D regularization improves unseen-entity performance by 13.6 F1 points over standard regularization.Fixed 80% masking improves by over 11.3 F1, while masking proportional to popularity improves by 3.8 F1; standard regularization also leaves the combined-signal model 10 F1 points below type-only and relation-only models on unseen entities.Source: §3.3.1 · p. 7

3.3.2 Weakly Supervised Data Labeling

• Weak labeling uses matching gendered pronouns and known alternative names on entity Wikipedia pages, increasing labeled mentions 1.7x and lifting unseen-entity performance by 2.6 F1.Wikipedia’s internal links provide the original self-supervised mention labels, but an estimated 68% of named entities are unlabeled.Source: §3.3.2 · p. 7

4.1 Experimental Setup

• The Wikipedia setup contains 5.3M entities, allows up to 30 candidates per mention, and reports micro-average F1 on true anchor links with filtered evaluation slices.Candidate lists are mined from Wikipedia anchor links and Wikidata aliases, with first and last names added for people.Source: §4.1 · pp. 7–8

4.2 Bootleg Performance

• Bootleg scores up to 5.8 F1 points above prior reported benchmark numbers, and beats NED-Base by 41.2 F1 on tail and 50 F1 on unseen entities.NED-Base reaches 85.9 F1 overall, within 5.4 F1 points of Bootleg.Source: §4.2 · pp. 8–9

4.3 Downstream Evaluation

• Bootleg representations reach 80.3 micro-average test F1 on TACRED-Revisited, 1.0 above KnowBERT and 2.3 above SpanBERT.The Overton production system shows positive relative quality with Bootleg embeddings across four languages, with improvements most visible on tail slices; the paper reports up to an 8% lift on production search and assistant tasks.Source: §4 · p. 7, §4.3 · pp. 9–10

4.4 Memory Usage

• Using only the top 5% of popularity-ranked entity embeddings cuts embedding memory by 95%, costs 0.8 F1 overall, and raises tail F1 by 2 points.The top 5% are ranked by training-data popularity; remaining entities use a fixed unseen-entity embedding.Source: §4.4 · p. 11, Figure 3 · p. 11

4.5 Ablation Study

• Type-only or KG-only models improve tail F1 by over 25 points and unseen F1 by over 46 points versus Ent-only, but neither matches full Bootleg on validation.The Ent-only model has stronger tail performance than NED-Base, which the authors attribute to Ent2Ent memorizing entity co-occurrences.Source: §4.5 · p. 11
• Inverse-popularity regularization achieves the highest overall F1, while regularizing popular entities more leads to a 9.8 F1-point drop versus inverse popularity.Fixed 80% regularization achieves the highest unseen-entity F1 among the fixed regularization settings.Source: §4.5 · pp. 11–12, Table 6 · p. 12

• Across reasoning-pattern slices, Bootleg generally combines entity, relation, and type signals effectively, although KG-only performs well on the KG-relation slice.The slices are representative but may not cover every example exhibiting a pattern.Source: §5 · p. 12, Table 7 · p. 13
• Granularity errors—predictions that are too general or specific—make up 12% of overall and 7% of tail errors.Source: §5 · p. 12
• Bootleg struggles with numerical tokens; year-bearing entity titles occur in 14% of overall and 25% of tail errors.The authors suggest BERT's subword representation of some numbers may contribute to this difficulty.Source: §5 · pp. 12–13
• Multi-hop cases account for 6% of overall and 7% of tail errors; Bootleg encodes only direct knowledge-graph connections.The authors describe the absence of multi-hop reasoning as a fundamental limitation.Source: §5 · p. 13
• Among cases where the BERT baseline is correct and Bootleg is not, 28% contain an exact mention-title match and 32% contain a missed title keyword.The authors attribute the drop to regularization discouraging use of entity-level information.Source: §5 · p. 13

• Earlier NED systems used rules or engineered statistical features [50], while recent state-of-the-art models use contextual word and entity embeddings [46] [51].The authors report that these deep models perform well on popular entities but struggle on the tail.Source: §6 · p. 13
• Tail-focused NED studies use phrase-based language models, while Bootleg investigates structural reasoning patterns for tail disambiguation.The cited tail studies do not fuse type or knowledge-graph information for disambiguation.Source: §6 · pp. 13, 15
• Prior work uses type features, type prediction, or type ontologies for disambiguation [9], among other approaches.Bootleg builds on this line with fine- and coarse-grained entity-level type embeddings and a mention-level type prediction task.Source: §6 · p. 14
• Knowledge-graph approaches use graph embeddings, hyperlink co-occurrences, latent relations [30], or textualized Wikidata triples [8] [33].Bootleg's KG2Ent connections support collective resolution rather than serving only as additional features.Source: §6 · p. 14
• Related work adds entity knowledge to downstream language tasks [6] [41] [54]; Bootleg also studies weakly labeled Wikipedia data for tail NED.The authors characterize Bootleg as the first to investigate the lift from weakly labeled Wikipedia data over the tail.Source: §6 · pp. 14–15

7 Conclusion

• Bootleg improves over the baseline state-of-the-art model by over 40 F1 points on Wikipedia's tail and transfers to relation extraction and production tasks.Its embeddings improve TACRED relation extraction by 1.0 F1 points and yield an 8% lift on highly optimized production tasks at a major technology company.Source: §7 · p. 15

A Extended Model Details

• A coarse mention-type prediction task infers types from the mention embedding and adds predicted types to each candidate's type embedding.The model predicts a distribution over coarse types and uses a type-prediction loss alongside the disambiguation loss.Source: A Extended Model Details · p. 19
• Bootleg adds sinusoidal positional encodings for mention boundaries to candidate embeddings so the model can use absolute and relative word positions.The authors illustrate the role of relative position with “America” referring to different entities depending on whether the sentence says “in Indiana” or “in America.”Source: A Extended Model Details · p. 19

• The section presents the experimental setup and training.Source: B Extended Results · p. 19
• It reports extended results on the regularization scheme and model ablations.Source: B Extended Results · p. 19
• It extends the error analysis to assess Bootleg's ability to reason over the four patterns.Source: B Extended Results · p. 19

• The Wikipedia training set uses the November 2019 dump, 3.3M linked entities, and 5.7M sentences after weak labelling.Source: §B.1 · p. 19
• Wikipedia experiments use page-level 80/10/10 splits for micro ablations and sentence-level 96/2/2 splits for benchmark training.Source: §B.1 · p. 19
• KORE50 contains 144 difficult mentions, RSS500 has 520 news mentions, and AIDA CoNLL-YAGO is a document-based news dataset.For KORE50, the authors retain 144 mentions for comparability despite one lacking a Wikipedia page; four RSS500 mentions lack entities in the knowledge base.Source: §B.1 · p. 20
• AIDA documents are converted into sentences by prefixing document titles, with mention-boundary expansion and up to 30 candidates selected by text similarity.Candidate generation searches n-grams from longest to shortest and ranks candidate Wikipedia pages by similarity to proper nouns in the sentence.Source: §B.1 · p. 20
• Bootleg draws structural resources from Wikidata triples and assigns entities types from Wikidata and HYENA hierarchies.The relation embeddings use 1,197 relations; filtered Wikidata types number 27K, while HYENA provides 505 types and five coarse categories.Source: §B.1 · p. 20

• All Bootleg models use 30 candidates per mention and structural resources, with T = 3 types and R = 50 relations assigned per entity.Source: §B.2 · p. 20
• Full-Wikipedia models use hidden dimension 512, while the micro-data model uses 256; both use 256-dimensional u and 128-dimensional other type and relation embeddings.Source: §B.2 · p. 20
• The benchmark model adds log sentence co-occurrence weights for entity pairs occurring together at least 10 times in Wikipedia.Pairs occurring fewer than 10 times receive weight zero; the authors report that this feature helped the model learn entity co-occurrences.Source: §B.2 · p. 20
• The benchmark model also appends entity-title BERT embeddings and a feature counting sentence entities appearing on each entity's Wikipedia page.The page co-occurrence feature performs similarly as an entity feature or a KG2Ent module, while reducing runtime as an entity feature.Source: §B.2 · pp. 20–21
• Training uses Adam at learning rate 0.0001, dropout 0.1, 16 attention heads, and a frozen BERT encoder stack, except for NED-Base.Source: §B.2 · p. 21
• Full-Wikipedia benchmark and ablation models train for one and two epochs respectively, while the micro model trains for eight epochs.The corresponding setups use batch sizes 512, 512, and 96; full-Wikipedia training runs on eight V100 GPUs and micro training on one P100.Source: §B.2 · p. 21

• The reported ablation model sizes exclude BERT parameters because NED-Base fine-tunes BERT while Bootleg does not.Table 10 reports sizes in MB for NED-Base, full Bootleg, and entity-only, KG-only, and type-only variants.Source: §B.3 · p. 21, Table 10 · p. 22
• Power-law inverse-popularity regularization performs best on tail and unseen entities, while fixed 0.5 regularization is marginally better on torso and all entities.The fixed-0.5 model is 4.5 F1 points worse on unseen entities than the best model; the authors suggest its lower regularization better exploits memorized entity patterns.Source: §B.3 · p. 22
• 2.6 F1 points on unseen entities and 0.3 F1 points on tail entities are gained with weak labelling, while torso performance is 0.5 F1 points better without it.Source: §B.3 · p. 22
• 4% of mentions without weak labelling versus 14% with it share a type with another sentence mention, and the model predicts consistent answers 4% versus 13% of the time.These figures use a weaker consistency definition than the main-text analysis and include weakly labelled mentions.Source: §B.3 · p. 22
• 14% of torso errors with weak labelling involve consistency reasoning when the correct answer violates that pattern, compared with 5% without weak labelling.Source: §B.3 · p. 23

C Extended Downstream Details

• TACRED examples provide text and subject/object positions, with 41 relation labels plus a no-relation option and NER, POS, and contextual Bootleg embeddings as features.Source: C Extended Downstream Details · p. 23
• For TACRED, Bootleg mention extraction searches known candidate-map mentions over sentence n-grams from longest to shortest.The ablation Bootleg model with entity, KG, and type signals adds fine-tuned BERT word embeddings and is trained on Wikipedia sentences relevant to TACRED candidates.Source: C Extended Downstream Details · p. 23
• The downstream classifier concatenates frozen Bootleg embeddings with SpanBERT encodings, then uses four Transformer layers and a softmax relation score.SpanBERT embeddings are fine-tuned; the model is trained with cross-entropy loss.Source: C Extended Downstream Details · p. 23
• 1.20x, 1.18x, and 1.20x worse are the baseline error rates relative to Bootleg when entity, relation, or type signals are present, respectively.The relative gap between BERT and Bootleg errors is also larger on above-median signal-count slices by 1.10x, 4.67x, and 1.35x for entity, relation, and type signals.Source: C Extended Downstream Details · p. 23

D.1 Reasoning Patterns

• 88% of tail entities have non-tail types, and 90% have non-tail relations, showing that entity, type, and relation tails are distinct.For example, the head type “country” includes rare entities such as Palaú and Belgium–France border.Source: §D.1 · p. 24
• Bootleg beats NED-Base by 9.4 F1 points for examples with more-than-median-frequency gold types and 20.3 F1 points for those with less-than-median-frequency gold types.For gold relations, the corresponding gains are 7.8 F1 points and 13.7 F1 points.Source: §D.1 · p. 24
• TF-IDF affordance keywords appear in 88% of examples whose gold entity has a particular type.The passage describes full coverage for one example type, but does not identify that type in the supplied text.Source: §D.1 · p. 24
• Tail types and tail relations are defined as those appearing 1–10 times in training data.This is the paper's frequency threshold for these signal categories.Source: §D.1 · p. 24
• Bootleg has the lowest error rates across types and relations regardless of rare-entity proportion, while baseline and Entity-Only errors rise as that proportion increases.The type trend is flat and the relation trend slopes upward; Bootleg's advantage over the baseline increases with rare proportion.Source: §D.1 · p. 24, Figure 4 · p. 25
• Each of 77 validation examples containing a “cafe” entity had an affordance keyword, while types with under 50% keyword coverage had a median of one validation occurrence.The passage cites this contrast as support for combining knowledge signals with distinct tails to address rare examples.Source: §D.1 · p. 25