PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

Yubo Ma, Yixin Cao, YongChing Hong, Aixin Sun

arXiv:2303.08559v2cs.CLcs.AI

TL;DR

The paper investigates whether LLMs are competitive few-shot information extractors relative to fine-tuned SLMs. Through experiments across nine datasets and four IE tasks, it finds that LLMs are generally less effective but can improve hard-sample handling when used as rerankers after SLM filtering. The resulting paradigm achieves an average 2.4% F1 gain with limited additional latency and budget cost.

  • Problem

    Whether LLMs perform competitively as few-shot information extractors remains unresolved because prior studies report inconsistent conclusions.

  • Method

    The paper conducts experiments across nine datasets and four IE tasks, then combines SLM filtering with LLM reranking of difficult samples.

  • Results

    LLMs generally underperform fine-tuned SLMs except in extremely low-resource settings, but filter-then-rerank produces an average 2.4% F1 gain across few-shot IE tasks.

  • Takeaways & Limitations

    LLMs are not effective few-shot IE extractors in general, yet can complement SLMs by improving performance on hard samples.

  • Takeaways & Limitations

    The adaptive filter relies on maximum SLM probabilities and assumes that SLMs are well-calibrated, an assumption the paper calls imperfect.

Abstract

from arXiv · show

Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs are competitive few-shot solvers for information extraction (IE) tasks, however, remains an open problem. In this work, we aim to provide a thorough answer to this question. Through extensive experiments on nine datasets across four IE tasks, we demonstrate that current advanced LLMs consistently exhibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. Therefore, we conclude that LLMs are not effective few-shot information extractors in general. Nonetheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle challenging samples that SLMs struggle with. And moreover, we propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs serve as filters and LLMs serve as rerankers. By prompting LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements (2.4% F1-gain on average) on various IE tasks, with an acceptable time and cost investment.

1 Introduction

The paper asks whether LLMs are competitive few-shot information extractors and compares them with fine-tuned SLMs across diverse settings. It finds that LLMs generally underperform, but can complement SLMs on hard samples through filter-then-rerank.

  • Empirical study: The study compares in-context-learning LLMs with fine-tuned SLMs across nine datasets covering four information extraction tasks.The tasks are named entity recognition, relation extraction, event detection, and event argument extraction.
  • Empirical findings: LLMs outperform SLMs only when label types and samples per label are extremely scarce; with hundreds of samples, SLMs significantly outperform LLMs.The comparison spans settings from 1-shot to 20-shot or more, with varied instructions, demonstrations, selection strategies, and prompt formats.
  • Sample difficulty: LLMs are good at hard samples but often perform worse than fine-tuned SLMs on easy negative samples because of strong false-positive tendencies.Sample difficulty is categorized using SLM confidence scores.
  • Proposed framework: The paper proposes filter-then-rerank: SLMs filter samples and LLMs rerank candidate labels for the hardest cases.For easy samples, the SLM prediction remains final; only a small subset receives LLM reranking.
  • Proposed framework: The combined system yields an average 2.4% higher F1 than previous methods across various few-shot information extraction tasks with modest additional cost.The framework is designed to combine the complementary strengths of SLMs and LLMs.

2 Related Work

Prior few-shot IE studies reach inconsistent conclusions about whether LLMs are competitive, partly because they use different tasks, datasets, and settings. This paper addresses the gap with broader experiments and a framework combining LLMs with SLMs.

  • LLMs for Information Extraction: Recent IE research uses LLMs either to generate pseudolabeled data for SLM training or directly for inference through in-context learning.Instruction-finetuning approaches can require diverse samples and perform suboptimally in low-resource scenarios.
  • Few-shot IE with ICL: Few-shot IE studies disagree: some favor LLMs as competent extractors, while others dispute that conclusion.This disagreement leaves LLM competitiveness unresolved.
  • Few-shot IE with ICL: The paper attributes prior disagreement to the absence of a comprehensive unified benchmark and variation in tasks, datasets, and few-shot settings.It also notes that overly simplistic datasets may exaggerate LLM effectiveness.
  • Proposed approach: The proposed filter-then-rerank paradigm integrates SLMs and LLMs in one framework and supports multiple IE tasks by providing candidate spans in the text.It uses prompting strategies that transform IE tasks into multiple-choice questions.

3 Large LMs v.s. Small LMs

Across nine datasets and four IE tasks, the study compares few-shot LLMs with fine-tuned SLMs under varied low-resource settings and prompt designs. LLMs generally trail SLMs in realistic settings, despite occasional advantages with extremely limited annotations.

  • Main results: LLMs only outperform SLMs when labels and samples are highly restricted, whereas SLMs are generally more effective.The authors conclude that LLMs are less effective than supervised SLMs for few-shot IE under real scenarios involving fine-grained tasks and 10–20 annotations per label.
  • Main results: Under extremely low-resource 1-shot or 5-shot settings, LLMs sometimes perform better, but SLMs improve more steeply as sample size grows.LLM performance tends to plateau with modest increases in sample size, while SLM trajectories rise more sharply.
  • Efficiency: LLMs are much slower than SLMs because of larger parameter counts, longer input contexts, and additional response delay in API-based inference.Inference speed is compared over 500 sentences; LLaMA is especially slow under a batch-size-one memory constraint.
  • Evaluation setup: The evaluation varies instructions, demonstration number and selection, prompt format, and other prompt-context factors.The study examines text and code prompts and uses retrieved demonstrations for LLM in-context learning.
  • Prompt sensitivity: Prompt variations do not reveal a lottery prompt that substantially changes the conclusion that LLMs are not good few-shot IE solvers.The reported conclusion follows the prompt-variant investigation.

4 LLMs are Good Few-shot Reranker

The proposed filter-then-rerank framework uses SLM confidence to route samples: SLMs handle easy cases, while LLMs rerank candidates for hard cases. This targets complementary strengths while avoiding unnecessary LLM use on easy samples.

  • Filter-then-rerank paradigm: The framework shifts LLMs from sentence-level extraction to sample-level candidate selection and reframes each decision as a multi-choice question.SLM-generated candidate answers provide the options that LLMs rerank.
  • Difficulty estimation: Low SLM confidence defines hard samples, while higher confidence defines easy samples using the maximum predicted label probability.The confidence score is the highest SLM probability across labels for a sample.
  • Complementary abilities: LLM reranking enhances performance on hard samples but degrades performance on easy samples, especially for very easy cases.The reported hard-sample gains are associated with external knowledge and complex reasoning correcting SLM errors.
  • Why easy samples fail: Easy samples contain higher negative-sample ratios across diverse IE tasks, and LLM false-positive tendencies therefore make reranking harmful there.Negative samples are those labeled None; SLMs predict them with relatively high confidence and accuracy.

5 Adaptive Filter-then-rerank Paradigm

The adaptive filter-then-rerank paradigm uses SLMs to identify easy and hard samples, retaining SLM predictions for easy cases and asking LLMs to rerank SLM candidates for hard cases. Across experiments, this selective design improves performance while reducing the cost and latency of direct LLM inference.

  • Architecture: SLMs filter samples by confidence, keeping their predictions for easy cases and sending top-N candidates from hard cases to LLMs for reranking.LLMs use multiple-choice prompts with demonstrations to produce the final answer for hard samples.
  • Experimental Setup: The experiments evaluate filters based on FSLS or KnowPrompt and rerankers including Vicuna-13B, InstructGPT, and GPT-4.The setup uses a validation-optimized difficulty threshold, top-3 SLM predictions plus None, and 4-shot demonstrations.
  • Main Results: 2.4% average F1 gain is achieved with InstructGPT across three datasets and nine settings without SLM ensemble.Using ensemble SLMs as the filter still produces a 2.1% average gain.
  • Analysis: Only 0.5%~10% of samples are reranked, yet reranking yields 10%~25% absolute F1 gains on those hard samples.The selective design improves overall performance while limiting LLM usage.
  • Analysis: Approximately 80%~90% reductions in budget and latency arise from fewer LLM calls and shorter prompts than direct ICL.The cost reduction comes from reranking only hard samples and using fewer candidate labels and demonstrations.
  • Ablation Study: Demos with explanations improve reranking across datasets, while label filtering improves performance and reduces demonstration length.Removing adaptive difficulty filtering causes performance to collapse, supporting the integrated SLM-LLM design.

6 Conclusion

The study finds that LLMs are generally ineffective few-shot information extractors despite advantages in extreme low-resource settings. It combines SLM filters with LLM rerankers to improve hard-sample predictions, achieving a 2.4% average F1 gain while minimizing latency and budget costs.

  • Conclusion: The study evaluates nine datasets across four IE tasks and finds that LLMs are not effective few-shot information extractors in general.LLMs remain superior in extreme low-resource scenarios but struggle with IE prompts, limited demonstration capacity, and high inference costs.
  • Conclusion: The adaptive filter-then-rerank paradigm combines SLM strengths on routine cases with LLM improvements on hard samples.SLMs filter samples while LLMs rerank difficult cases.
  • Conclusion: 2.4% average F1 gain is achieved across multiple few-shot IE tasks while latency and budget costs are minimized.

Limitations

The study acknowledges uncertainty about whether unexplored prompts could improve LLM performance and notes that benchmark contamination remains a potential evaluation risk. The adaptive paradigm also relies on an imperfect assumption that SLM confidence is calibrated for assessing sample difficulty.

  • Prompt Scope: Unexplored prompt variants may outperform those tested, although the authors consider a substantial change to the conclusions unlikely.The study explored different LLMs, demonstration strategies, and prompt formats without finding significant prompt effects.
  • Evaluation Risk: Public benchmark memorization may overestimate LLM abilities, despite the authors’ use of earlier and stable model versions to mitigate contamination.The authors report that their primary conclusion remains unchanged even if contamination exists.
  • Adaptive Filtering: Sample-difficulty assessment uses maximum SLM probabilities and assumes that SLMs are well-calibrated, an imperfect assumption.The authors suggest calibrating SLM filters or developing better difficulty metrics as future work.

A.2 Details of Few-shot IE Datasets

The paper constructs few-shot IE datasets by sampling examples per label across tasks with varying schema complexity, then uses validation splits or cross-validation depending on dataset size. It evaluates supervised SLM methods using standardized training configurations.

  • Sampling: The K-shot sampling strategy selects at least K samples for each label, using K values of 1, 5, 10, 20, 50, and 100 for RE and 1, 5, 10, and 20 for other tasks.Sentence-level sampling handles examples containing multiple entities, events, or arguments.
  • Sampling: Greedy sampling orders labels by frequency and updates counts across all event types while constructing subsets.The procedure is designed to satisfy per-label shot requirements.
  • Validation: Datasets with more than 300 sentences use a 10% validation split, while smaller datasets use 5-fold cross-validation.This split strategy aims to reduce overfitting when constructing training and validation data.
  • SLM Baselines: The SLM evaluation compares fine-tuning, FSLS, KnowPrompt, and PAIE using RoBERTa-large backbones.These methods cover supervised and task-specific few-shot IE approaches.
  • Implementation: KnowPrompt uses up to 10 epochs for 50- and 100-shot datasets and 50 epochs for other datasets, with maximum input length 800 and learning rate 1e-4 for PAIE.

C LLMs Implementations

The study evaluates five LLMs using in-context learning, including proprietary and open-source models, under controlled inference settings. Main experiments use greedy decoding and compare prompt configurations without fine-tuning.

  • Models: Five LLMs are evaluated: CODEX, InstructGPT, ChatGPT, LLaMA-13B, and Vicuna-13B.The first three are proprietary models, while LLaMA-13B and Vicuna-13B are open-source models.
  • Open-source implementation: Open-source LLM experiments use maximum input length 2048, batch size 1, and a single NVIDIA V100 GPU.Inference uses Accelerate, fp16, maximum output length 96, greedy decoding, and zero frequency and presence penalties.
  • OpenAI implementation: OpenAI-model experiments use official APIs with maximum input length 3600, except 7000 for CODEX on relation extraction.Maximum output length is 32 for relation extraction and 96 for the other tasks; decoding is greedy.
  • Decoding: The main experiments set sampling temperature t = 0 for structured information extraction outputs.Table 8 is used to validate higher generated quality at t = 0 than at t ≠ 0; self-consistency is not considered.

D.2 Automatic Chain-of-thought

The paper investigates Automatic Chain-of-thought for few-shot information extraction by inserting generated rationales into demonstrations. It reports substantial degradation and attributes this outcome to longer examples, inconsistent demonstrations, and low-quality rationales.

  • Procedure: Automatic Chain-of-thought generates rationales for samples with positive labels before inserting them between demonstrations’ sentences and ground-truth answers.The rationale prompt asks why a span belongs to a label, while sentences without positive labels retain the vanilla format.
  • Results: Auto-CoT degrades performance with a large margin.The comparison uses InstructGPT-generated rationales and CODEX as the inference backbone.
  • Analysis: The authors speculate that longer rationales reduce the number of examples that fit in demonstrations.This is attributed to rationale-induced sample-length increases under fixed context limits.
  • Analysis: The rationale setup creates a discrepancy between samples with positive labels and samples without labels.Rationales are supplied only for positive-label sentences because explaining the absence of labels is difficult.
  • Analysis: Some automatically generated rationales are low quality, especially for relation extraction.The paper identifies better use of automatically generated rationales as future work.

E.1 LLMs struggle on Fine-grained Datasets

The paper examines why LLM performance varies across fine-grained IE datasets and across prompting choices. It finds that LLMs fare relatively worse as label inventories grow, while simple text prompts and sentence-embedding demo selection are preferred for the main experiments.

  • Fine-grained datasets: Under 5-shot settings, LLMs’ performance difference relative to SLMs negatively correlates with label number across IE tasks.With more label types, LLMs perform relatively worse than SLMs, leading the authors to conclude that LLMs struggle on fine-grained datasets.
  • Instruction format: Six instruction variants show no significant correlation between instruction complexity and ChatGPT performance on four 20-shot IE tasks.The instruction-free prompt performs comparably to, or better than, complex instructions, so the main experiments use simple instruction I1.
  • Demo number: Relation extraction performance consistently improves with more demonstrations, whereas NER and event detection stabilize or degrade as demo numbers increase.The latter tasks appear limited before reaching the maximum input length.
  • Demo selection: Sentence-embedding and Efficient Prompt Retriever selection both surpass random sampling by a large margin.Sentence embedding is adopted for the main experiment because of its simplicity.
  • Prompt format: Text and code prompts have comparable performance across formats, so the simpler text prompt is selected for the main experiment.The comparison is reported in Table 14.
  • Reranking examples: LLM rerankers correct errors involving missing external knowledge and limited reasoning abilities.Examples include identifying Triptolemus as a Greek-mythology figure and recognizing that “fought” need not indicate physical violence.

F.2 Easy Samples

On easy samples, LLMs often make false-positive predictions, including hallucinated entities and mismatched spans. The paper illustrates these errors with NER examples and contrasts them with the candidate-level reranking setup.

  • Error sources: LLMs’ inferior performance on easy samples is attributed to false-positive predictions for negative samples.The authors speculate that hallucination and span boundary mismatch are the main reasons.
  • LLM hallucination: Without IE-specific fine-tuning, LLMs sometimes hallucinate verbs, prepositions, or special symbols as entities.The paper presents hallucination as a distinctive error in in-context learning IE systems.
  • LLM hallucination: LLMs predict “of” as organization-other even though the sentence uses it as a preposition relating two institutions.The analysis selects the candidate indicating that “of” does not pertain to a known entity category.
  • Span boundary mismatch: Sample-level reranking considers each candidate independently and enables multiple-choice prompts, but aggravates span-boundary mismatches in NER and event detection.Easy samples suffer more from this mismatch than hard samples.
  • Span boundary mismatch: LLMs predict “Baron” as a person entity even though the gold entity is the longer span “Baron Carl Hurleman.”This example demonstrates a span boundary mismatch rather than a failure to identify the surrounding person mention.
Loading 2303.08559v2…