UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition

1 INTRODUCTION

• Targeted distillation uses mission-focused instruction tuning to focus student models on a broad application class rather than all possible applications.The motivation is that generic distillation can produce only a shallow approximation under limited compute.Source: §1 · p. 1
• Sparse annotations, emerging entity types, annotation costs, and limited generalization of supervised NER models motivate open NER across domains and types.The paper notes that expert annotation is especially costly in high-value domains such as biomedicine.Source: §1 · p. 2
• The paper presents a general recipe for targeted distillation and demonstrates it by using ChatGPT-generated instruction data to tune LLaMA into UniversalNER models for open-domain NER.The generated data come from broad-coverage unlabeled web text.Source: §1 · p. 2
• Over 30 absolute average F1 points over Vicuna and 7–9 over ChatGPT: UniversalNER performs strongly across tens of thousands of entity types, also surpassing InstructUIE by a large margin.The UniversalNER benchmark comprises 43 datasets across 9 domains; the paper reports no direct supervision for these results.Source: §1 · p. 2

2 RELATED WORK

• Alpaca and Vicuna distill general abilities from LLMs, while other approaches target specialized tasks or reasoning, often within particular datasets or domains.Examples include instruction generation for Alpaca, ShareGPT conversations for Vicuna, and specialized-task distillation.Source: §2 · p. 2
• Instruction-tuning systems such as FLAN, T0, and Tk-Instruct convert supervised datasets into instruction-following formats, while smaller LLM-tuned models can still struggle across diverse tasks.The passage also describes a substantial targeted-benchmark gap between smaller models and ChatGPT.Source: §2 · pp. 2–3
• The paper distinguishes mission-focused tuning by varying task labels, such as entity types, rather than relying on instruction diversity to cover many unrelated tasks.It presents this as a recipe for a broad application class and reports closing the targeted-task gap with ChatGPT; the cited comparison says supervised-only InstructUIE performs subpar to ChatGPT.Source: §2 · p. 3

3 MISSION-FOCUSED INSTRUCTION TUNING

• Mission-focused instruction tuning targets generalization across semantic types and domains by diversifying inputs rather than instructions.The NER case study maps inputs and predefined entity types to entities of a requested type.Source: §3.1 · p. 3, §3 · p. 3

3.1 DATA CONSTRUCTION

• To cover unknown test-domain distributions, the method samples passages from the Pile and uses ChatGPT to generate entity mentions and types without specifying target types.It randomly samples 50K passages, each at most 256 tokens, from a corpus comprising 22 English subdatasets.Source: §3.1 · p. 3
• After filtering, the generated data contain 45,889 input-output pairs, 240,725 entities, and 13,020 distinct types; the top 1% of entities account for 74% of total frequency.The types span domains and vary in granularity, including subset relations such as COUNTY within LOCATION.Source: §3.1 · pp. 3–4
• Definition-based construction generates 353,092 entity types and makes the tuned model less sensitive to type paraphrasing, but performs worse on standard NER benchmarks.The prompts ask ChatGPT to define entity types in short sentences.Source: §3.1 · p. 4

3.2 INSTRUCTION TUNING

• Conversation-style tuning turns each entity type into a natural-language query and trains the model to return a JSON list of its mentions; preliminary experiments favor it over traditional NER-style tuning.The template uses the passage as input and applies the language-modeling objective to the gold answer tokens.Source: §3.2 · p. 4, Figure 2 · p. 4
• The method also considers querying all entity types at once instead of using one type per query.The passage refers detailed results and discussion to Section 5.2.Source: §3.2 · p. 4
• Frequency-weighted negative entity-type queries with empty JSON outputs improve instruction-tuning results.These examples address the mismatch between open-world generated data and closed-world queries for types absent from a passage.Source: §3.2 · pp. 4–5
• For supervised tuning across datasets, adding the dataset name to the input lets the model learn dataset-specific label semantics amid conflicting definitions.At inference, the dataset name is included for the supervised setting and omitted for zero-shot use; ACE and MultiNERD differ on whether pronouns count as PERSON.Source: §3.2 · p. 5

4 UNIVERSAL NER BENCHMARK

• Dataset-specific tuning templates add the dataset name to the input to resolve conflicts in label definitions.The dataset name is the additional input field shown in the template.Source: Figure 3 · p. 5
• The benchmark covers 43 NER datasets across 9 domains, including general, biomedical, clinical, STEM, programming, social media, law, finance, and transportation.The passage points to Figure 4 for the data distribution and an appendix table for dataset statistics.Source: §4 · p. 5
• Benchmark processing converts labels into natural-language forms, removes labels judged inconsistent across sources, and splits long document-level instances into sentences.The authors state that some labels and guidelines are not optimized as ground truth for holistic evaluation, while long contexts may exceed model input limits.Source: §4 · p. 5

5.1 EXPERIMENTAL SETTINGS

• Evaluation uses strict entity-level micro-F1, requiring exact matches of both entity type and boundary; models are based on LLaMA2.For large test datasets, evaluation samples up to 200,000 passage-query pairs per dataset.Source: §5.1 · p. 6

5.2 DISTILLATION

• UniNER-7B and UniNER-13B achieve average F1 scores of 41.7% and 43.4%, respectively, versus 34.9% for ChatGPT, and both outperform ChatGPT across all evaluated domains.Source: §5 · p. 6, §5.2 · p. 7

5.3 SUPERVISED FINETUNING

• 60.0% average F1 is the best out-of-domain result after continual supervised fine-tuning; supervised-only training from scratch reaches 57.2%.Without fine-tuning, instruction-tuned UniNER-7B and 13B already surpass ChatGPT, Vicuna, and InstructUIE-11B by a large margin.Source: §5.3 · p. 7, Table 3 · p. 8
• 84.78% average F1 on 20 datasets for UniNER-7B, surpassing BERT-base by 4.69% and InstructUIE-11B by 3.62%.The in-domain supervised setting uses the same training data as InstructUIE.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 ANALYSIS

• Frequency-based negative sampling outperforms no sampling by 21.9% and uniform sampling by 5.7%.The compared UniNER-7B models are instruction-tuned on Pile.Source: §5.4 · p. 8, Table 4 · p. 8
• Data-specific templates outperform original templates on most datasets, with larger gains under label overlap and gains above 10% for FACILITY (22.0%) and TIME (12.4%) amid inconsistent definitions.The analysis suggests these improvements are particularly effective at resolving label conflicts.Source: §5.4 · pp. 8–9, Figure 6 · p. 8

5.5 CASE STUDY

• UniNER-7B-definition is more robust to entity-type paraphrasing than UniNER-7B-type, despite performing worse on the Universal NER benchmark.The paper notes that definitions may improve performance for less populous entity types while lowering standard benchmark performance.Source: §5.5 · p. 9
• In a code example, UniNER extracts diverse entity types, including local variables, methods, and classes, though it misses from_pretrained as a method.Figure 8 presents the code snippet used to assess diverse entity extraction.Source: §5.5 · p. 9

6 CONCLUSION

• The paper presents targeted distillation through mission-focused instruction tuning, using NER as a case study for training smaller, more efficient open-domain NER models.The approach distills ChatGPT into UniversalNER without direct supervision and reports accuracy across a wide range of domains and entity types.Source: §6 · p. 9
• UniversalNER retains ChatGPT’s NER capabilities and surpasses ChatGPT and other state-of-the-art systems in NER performance.The conclusion describes this outcome as applying across a wide range of domains and entity types.Source: §6 · p. 9
• The paper releases its distillation recipe, data, and UniversalNER models to facilitate future research on targeted distillation.The release is stated as the paper’s contribution to future research.Source: §6 · p. 9

Sensitivity to entity type paraphrasing

• UniNER-7B-type sometimes misses entities with similar meanings, while UniNER-7B-definition is more robust to entity-type paraphrasing despite lower benchmark performance.Definitions may improve performance for less populous entity types, although they lower performance on standard NER benchmarks.Source: §A.1 · p. 16, Figure 7 · p. 16

Recognition of diverse entity types

• For the code example, UniNER identifies library, local-variable, method, and class entities, including transformers, input_text, encode, and GPT2Tokenizer.The queries also return no global variables in the snippet.Source: §A.1 · pp. 16–17
• The code case study demonstrates extraction of diverse entity types, but from_pretrained is missed as a method.Figure 8 presents the code snippet used for this assessment.Source: Figure 8 · p. 16, §A.1 · p. 17

B FULL EVALUATION RESULTS

• Full evaluation results for ChatGPT, UniNER-7B-type, and UniNER-7B-sup+type are shown in Figure 9.The passage identifies the models and figure but does not state the results.Source: B FULL EVALUATION RESULTS · p. 17

• Table 6 reports the number of instances in each dataset's train, dev, and test splits.Source: C DATA STATISTICS · p. 17, Table 6 · p. 19
• Table 6 reports the number of entity types in each benchmark dataset.Source: C DATA STATISTICS · p. 17, Table 6 · p. 19
• Table 6 reports average input-text tokens and average entities per instance for each dataset.Source: C DATA STATISTICS · p. 17, Table 6 · p. 19