LLMaAA: Making Large Language Models as Active Annotators

1 Introduction

• Task-specific models require extensive labeled data, while generated text can be low quality or domain-shifted and LLM annotations can be noisy.The introduction also notes the cost of LLM annotation, especially under restricted budgets.Source: §1 · pp. 1–2
• LLMaAA integrates active learning into LLM annotation and improves annotation and training through prompt engineering and learnable example reweighting.It retrieves k-NN demonstrations and reweights silver training samples to prioritize informative examples and reduce noisy-label effects.Source: §1 · p. 2
• On NER and RE, task-specific models outperform teacher LLMs within hundreds of annotated samples.The evaluation uses about 100 gold examples for demonstration and validation, alongside hundreds of silver samples.Source: §1 · p. 2
• The authors present LLMaAA as a cost-effective, privacy-conscious alternative with task-specific performance and lower inference costs.They describe these task-specific models as offering advantages in performance, data privacy, and inference costs.Source: §1 · p. 2

2 Related Work

• LLM prompting research studies how prompts and instructions steer model outputs, including few-shot, zero-shot, and chain-of-thought approaches.The passage describes prompt engineering as improving controllability and performance across few-shot and zero-shot settings.Source: §2 · p. 2
• Prior work uses LLMs for both annotation and text generation, while generation-based methods still face low-quality and domain-shift issues and do not consider generation cost efficiency.The discussion names ZEROGEN (Ye et al., 2022) and SUNGEN (Gao et al., 2023) among zero-shot generation approaches.Source: §2 · p. 3
• Pool-based active learning selects examples using uncertainty, diversity, or hybrid criteria, with selected examples typically labeled by humans and added iteratively.The figure presents LLMaAA as an iterative combination of LLM pseudo-labeling, active data selection, and automatic reweighting.Source: §2 · p. 3, Figure 2 · p. 3

3 LLM as Active Annotator

• LLMaAA iterates LLM pseudo-labeling, active data acquisition, and automatic reweighting to gradually produce stronger task-specific models.The goal is to learn robustly from minimal LLM-generated labels while leveraging abundant unlabeled data.Source: §3 · p. 3

3.1 Optimizing LLM as Better Annotator

• For better annotations, k-NN retrieval selects nearby demonstrations, while label verbalizers turn potentially ambiguous classification labels into natural-language descriptions.The paper uses Sentence-BERT representations for retrieval and predefined templates for verbalizers.Source: §3.1 · p. 4

3.2 Active Data Acquisition

• The pool-based loop trains a model, acquires and labels a batch, then adds it to the labeled set; compared strategies include random, uncertainty, and k-means diversity sampling.Uncertainty methods include maximum entropy and least confidence.Source: §3.2 · p. 4

3.3 Robust Learning with Noisy Labels

• Automatic reweighting adaptively weights training examples to reduce noisy-label bias, using a small clean validation set and an online approximation to minimize validation loss.The assumed validation set is much smaller than the unlabeled pool.Source: §3.3 · p. 4

4 Tasks

• LLMaAA is instantiated for named entity recognition and relation extraction, using BERT-family encoders and two simple task models.The authors leave other task-model choices for future study.Source: §4 · pp. 4–5
• Automatic reweighting samples noisy training and clean validation batches, computes training loss, and updates model parameters through gradient descent.The algorithm outputs trained parameters after S iterations.Source: §4 · p. 5

4.1 Named Entity Recognition

• For flat NER without overlapping entities, the task is formulated as BIO sequence tagging; uncertainty scores can be pooled by average, sum, or max.Average and sum pooling are adopted in practice for better empirical performance; BERT features feed a linear classifier for token labels.Source: §4.1 · p. 5

4.2 Relation Extraction

• Relation extraction classifies a subject–object pair into a predefined relation set or NA, using marked entity spans and a BERT-based classifier.The model encloses entity spans with special tokens and classifies the concatenated subject and object embeddings.Source: §4.2 · p. 5

5.1 Setup

• With ~100 gold examples for demonstration and validation, LLMAAA trains task-adapted models using 500 silver samples; experiments cover Chinese OntoNotes 4.0, English CoNLL03, and Re-TACRED-subset.NER uses k-NN example retrieval and RE uses a label verbalizer, with k set to 5.Source: §5.1 · pp. 5–6

5.2 Overall Results

• 74.00%, 82.84%, and 80.79% F1 on Chinese OntoNotes 4.0, English CoNLL03, and Re-TACRED-subset, respectively; least-confidence LLMAAA outperforms all comparative baselines.LLMAAA improves on PROMPTING by 4% in average score and surpasses SUPERVISED, while using task-specific models with fewer parameters and lower inference latency.Source: Table 1 · p. 6, §5.2 · p. 6
• Generation-based ZEROGEN and FEWGEN fail to match LLMAAA, even with 10× more data in the zero-shot setting.On Re-TACRED, ZEROGEN tends to produce templated text that deviates from the news domain; FEWGEN improves with examples but still trails LLMAAA.Source: §5.2 · p. 6

5.3.1 Effects of Prompt Engineering

• Zero-shot NER loses 21% F1 on OntoNotes and 25% on CoNLL without k-NN examples, while the RE label verbalizer improves F1 from 70.94% to 73.77%.The results indicate that demonstrations help NER and aligned label descriptions help RE.Source: §5.3.1 · p. 7, Table 3 · p. 7

5.3.2 Accelerating with Active Learning

• Uncertainty-based acquisition outperforms random selection, matches random with only 30%–40% of training data, and surpasses PROMPTING within 500 annotated samples.K-means beats random only on OntoNotes and yields similar results on CoNLL03 and Re-TACRED; it may require more data to learn informative representations.Source: §5.3.2 · pp. 7–8, Figure 3 · p. 7

5.3.3 Reweighting Helps Robust Training

• Automatic reweighting consistently improves performance across datasets and methods, with larger gains on OntoNotes and Re-TACRED than on CoNLL03.The authors attribute its role in poorer-annotator settings to estimating an unbiased validation distribution and helping prevent overfitting to noisy labels.Source: Figure 4 · p. 7, §5.3.3 · p. 8

6.1 LLMAAA with Different Annotators

• On OntoNotes, TAMs consistently outperform GPT-3 and GPT-4; the student exceeds GPT-3 by 27% in F1, and the gap narrows with stronger teachers.The experiments use a fixed budget of 500 annotated samples; the authors suggest more annotation budget and more powerful TAMs could extend the limit.Source: §6.1 · p. 8

6.2 Why Can Students Outperform Teachers?

• Students can outperform teachers even without sample reweighting, an observation the authors relate to earlier knowledge-distillation and pseudo-label learning findings.The paper notes that those prior settings are similar to, but slightly different from, LLMAAA.Source: §6.2 · p. 8
• In a simplified binary task with teacher accuracy p > 0.5, minimizing the student objective yields S(x; θ) = p, so a 0.5 threshold predicts the correct label.The authors say the same conclusion holds when a temperature-zero teacher is deterministic for each x but statistically random across D(x).Source: §6.2 · pp. 8–9
• This account is a relaxed intuitive explanation of hard-label distillation, not a rigorous theoretical analysis.The authors leave rigorous analysis for future work.Source: §6.2 · p. 9

• LLMAAA uses active learning to select informative samples for LLM annotation, with prompt engineering and automatic reweighting to improve annotation quality and reduce noisy-label impact.Source: §7 · p. 9
• Across NER and RE, TAMs trained with hundreds of LLM-annotated samples substantially outperform their teacher LLMs.Source: §7 · p. 9
• The authors characterize LLMAAA as more efficient than data-generation methods that usually need orders of magnitude more synthetic data, and as a cost-effective, privacy-ensured practical solution.Source: §7 · p. 9

• LLMAAA relies on an unlabeled pool and oracle-annotated demonstration and validation data; the authors use 100 gold samples for both purposes and regard pool availability as practical in many real-world settings.Source: Limitations · p. 9
• A weak teacher may not produce application-ready students, while a student architecture imposes a ceiling that makes outperforming a stronger teacher theoretically impossible.Source: Limitations · p. 9
• The experiments use proprietary, black-box GPT annotators, a practice the authors say may raise concerns such as potential exposure to test data.The authors state that their analysis in § 6.1 does not affect their view of the method's effectiveness.Source: Limitations · p. 9

• Generated data may contain bias or toxic content, and downstream NER or RE may produce mis-extractions, false information, or failures; the authors recommend considering debiasing or manual checks.Source: Ethics Statement · p. 9

A Dataset Statistics

• Chinese OntoNotes uses four entity types and 512-token chunks, with 15,724/4,301/4,346 training/validation/testing samples.The split follows Che et al. (2013).Source: A Dataset Statistics · pp. 13–14
• English CoNLL 2003 contains PER, LOC, ORG, and MISC labels, with 14,041/3,250/3,453 training/validation/testing sentences.The dataset is the English CoNLL 2003 shared task (Tjong Kim Sang and De Meulder, 2003).Source: A Dataset Statistics · p. 14
• The Re-TACRED subset keeps personal-relationship instances, balances NA examples to their original portion, and avoids all 40 relation types because longer prompts may exceed API limits and increase latency.Relation-type statistics are listed in Table 5.Source: A Dataset Statistics · p. 14, Table 5 · p. 14
• For all three datasets, 100 original validation examples serve as both demonstrations and validation data, while the initial labeled seed contains 50 randomly sampled training examples.Source: A Dataset Statistics · p. 14

B.1 LLM Inference APIs

• The authors access OpenAI APIs through Azure, and ChatGPT or GPT-4 responses may vary over time even at temperature 0.Table 6 lists the API used for each model.Source: §B.1 · p. 14, Table 6 · p. 14

B.2 Training Task-Specific Models

• Training runs for LLMaAA and the listed baselines use three random seeds, with results reported as mean and standard deviation.Source: §B.2 · p. 14
• TAMs use BERT encoders, AdamW, batch size 8, and 40 epochs, with the checkpoint selected by validation performance.English uses bert-base-cased at 5e-5; Chinese uses chinese-bert-base-wwm at 2e-5, and other parameters use 1e-4.Source: §B.2 · p. 14

B.3 Prompts

• Annotation prompts place the target sample last and leave its output blank; Re-TACRED prompts are provided with and without verbalized labels.Each demonstration inserts its text into the input and its label into the output.Source: §B.3 · p. 14, Table 7 · p. 15
• Zero-shot generation prompts include a default example to show the desired output format.The generation prompts are used similarly to the annotation prompts.Source: §B.3 · p. 14, Table 8 · p. 16

C Annotation Examples

• The CoNLL 2003 examples include a correct annotation and a partially wrong one that hallucinates April, March, and Thursday as entities.The examples appear in Listing 1.Source: C Annotation Examples · pp. 14–15