PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors

Kai Zhang, Bernal Jiménez Gutiérrez, Yu Su

arXiv:2305.11159v1cs.CL

TL;DR

Instruction-tuned LLMs still underperform small-LM methods on relation extraction, a task that is rare in instruction-tuning data. QA4RE reformulates RE as multiple-choice question answering. Across four datasets, it improves LLM performance and enables models to exceed prior zero-shot RE SoTA.

  • Problem

    Advanced instruction-tuned LLMs fail to outperform small-LM methods on RE, while RE is barely represented in instruction-tuning datasets.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    QA4RE reformulates RE as multiple-choice QA to leverage a task widely covered in instruction-tuning datasets.

    Source: §7 · pp. 8–9

  • Results

    Across four datasets, QA4RE improves over vanilla RE and enables text-davinci-003 and FLAN-T5-XXLarge to exceed prior zero-shot RE SoTA, with average absolute F1 gains of 8.2% and 8.6%, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study presents alignment with frequent instruction-tuning tasks such as QA as a promising direction for adapting LLMs to infrequent, challenging tasks.

    Source: §1 · p. 2

    Few-shot exploration is limited, leaving unclear whether more training examples improve RE performance or whether zero-shot GPT-3 trends hold in few-shot settings.

    Source: §8 · p. 9

Abstract

from arXiv · show

Recent work has shown that fine-tuning large language models (LLMs) on large-scale instruction-following datasets substantially improves their performance on a wide range of NLP tasks, especially in the zero-shot setting. However, even advanced instruction-tuned LLMs still fail to outperform small LMs on relation extraction (RE), a fundamental information extraction task. We hypothesize that instruction-tuning has been unable to elicit strong RE capabilities in LLMs due to RE's low incidence in instruction-tuning datasets, making up less than 1% of all tasks (Wang et al., 2022). To address this limitation, we propose QA4RE, a framework that aligns RE with question answering (QA), a predominant task in instruction-tuning datasets. Comprehensive zero-shot RE experiments over four datasets with two series of instruction-tuned LLMs (six LLMs in total) demonstrate that our QA4RE framework consistently improves LLM performance, strongly verifying our hypothesis and enabling LLMs to outperform strong zero-shot baselines by a large margin. Additionally, we provide thorough experiments and discussions to show the robustness, few-shot effectiveness, and strong transferability of our QA4RE framework. This work illustrates a promising way of adapting LLMs to challenging and underrepresented tasks by aligning these tasks with more common instruction-tuning tasks like QA.

1 Introduction

Advanced instruction-tuned LLMs underperform small-LM zero-shot methods on relation extraction (RE), which is rare in instruction-tuning datasets. QA4RE reformulates RE as multiple-choice question answering, and experiments report gains over vanilla RE and prior zero-shot SoTA.

1 Introduction

  • Even advanced instruction-tuned LLMs fail to outperform the small-LM-based state-of-the-art zero-shot RE method using standard RE.This is reported for FLAN-T5 XXL and text-davinci-003 on general-domain RE.Source: §1 · p. 1
  • QA4RE casts the input sentence as a question and relation types as multiple-choice options, aligning RE with the more prevalent QA task.The paper reports QA at around 12–15% of tasks in two instruction-tuning datasets.Source: §1 · p. 2
  • 8.2% and 8.6% absolute F1 improvements for text-davinci-003 and FLAN-T5-XXLarge, respectively, with QA4RE; it also surpasses prior zero-shot RE SoTA.Results are averaged over four RE datasets.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors report robustness and few-shot effectiveness, as well as transferability across instruction-tuned models ranging from 80M to 175B parameters.The reported experiments include models from the FLAN-T5 and GPT-3.5 series.Source: §1 · p. 2

2 Related Work

Related work covers instruction tuning, low-resource RE reformulations, and earlier investigations of LLMs on RE. It describes both the gains attributed to instruction tuning and the limited prior evidence on LLM relation extraction.

Instruction Tuning

  • Instruction tuning on many downstream tasks has been associated with improved accuracy, robustness, and generalization to unseen tasks.The cited work discusses both human-instruction alignment strategies and task reformulation as instruction following.Source: §2 · pp. 2–3

Low-Resource Relation Extraction

  • Low-resource RE methods recast extraction as NLI or summarization, while other approaches use logical-rule prompts or learned entity-type tokens.The authors use relation templates from prior studies in their work.Source: §2 · p. 3

LLMs for Relation Extraction

  • Prior biomedical RE work found LLMs underperformed fine-tuned small LMs in few-shot settings and identified poor handling of NoTA as a major culprit.RE-like tasks made up about 0.5% of Super Natural Instruction, and none were selected for model evaluation.Source: §2 · p. 3

3 Methodology

The paper defines RE as identifying the relation between two entities in a sentence from predefined types. QA4RE turns this task into multiple-choice question answering using relation templates as options.

3.1 Problem Statement

  • RE identifies the relation between two entities in a sentence from a predefined set of relation types.An example consists of sentence S and head and tail entities Eh and Et within S.Source: §3.1 · p. 3

3.2 Relation Templates

  • Prior low-resource RE methods use relation-entailed templates as label verbalizations, including an NLI formulation based on manually constructed templates.The paper notes that NLI is a current state-of-the-art low-resource method.Source: §3.2 · p. 3
  • The framework reuses templates and type constraints developed in earlier work to construct its QA answer options.The paper also discusses applying the NLI formulation to LLMs.Source: §3.2 · p. 3

3.3 QA4RE Framework

  • QA4RE inserts the head and tail entities into relation templates and presents them as multiple-choice options, with the sentence as context.When applicable, type constraints eliminate options incompatible with the entity types; the model predicts an answer index rather than verbalizing the relation.Source: §3.3 · p. 4

4 Experiment Setup

Experiments use four RE datasets, compare zero-shot and few-shot baselines, and apply shared prompting and template choices across datasets and models. The implementation includes GPT-3.5 and FLAN-T5 models with series-specific decoding controls.

4.1 Datasets

  • The evaluation uses TACRED, RETACRED, TACREV, and SemEval, reporting micro-averaged F1 without the none-of-the-above relation.To control OpenAI API costs, the test set samples 1,000 examples from each dataset's test split.Source: §4.1 · p. 4

4.2 Baselines

  • Zero-shot baselines include NLI and SuRE, while few-shot comparisons on TACRED include NLI, standard fine-tuning, PTR, and KnowPrompt.NLI uses its own templates on TACRED and TACREV and SuRE templates on RETACRED and SemEval; all zero-shot methods apply entity-type constraints when available.Source: §4.2 · p. 4

4.3 QA4RE Implementation Details

  • The study uses shared task instructions and prompt formats across datasets and LLMs, with SuRE templates for QA4RE on all four datasets.GPT-3.5 Text Completion models constrain outputs with logit bias, while FLAN-T5 XLarge (3B) and XXLarge (11B) use greedy decoding; the authors could not replicate SuRE few-shot results with its provided code.Source: §4.3 · pp. 4–5

5 Results

QA4RE improves zero-shot relation extraction over vanilla RE across all LLMs and most datasets, with the strongest models surpassing NLIDeBERTa by a large margin. The results also indicate robustness to template variation, improved handling of NoTA, and few-shot gains over zero-shot performance.

5.1 Zero-Shot Results

  • QA4RE improves vanilla RE across all LLMs and most datasets, with text-davinci-003 and FLAN-T5 XL and XXL outperforming NLIDeBERTa by a large margin.The two FLAN-T5 models also benefit significantly; the largest gains occur for the strongest model in each series.Source: §5.1 · p. 5
  • On SemEval, where absent type constraints make QA a 19-choice task, QA4RE still substantially improves results, particularly for text-davinci-003 and FLAN-T5 XXL.Source: §5.1 · pp. 5–6

5.2 Robustness to Verbalization Templates

  • Across four TACRED templates, QA4RE outperforms vanilla RE and small-LM baselines, including with label-only templates, and is less dependent on expert-crafted verbalizations.The authors note this robustness may ease transfer to domains where expert annotations are costly, such as biomedical or financial settings.Source: §5.2 · p. 6

5.3 None-of-the-Above Relation Evaluation

  • For NoTA evaluation, text-davinci-003 with QA4RE improves NoTA-included metrics and outperforms small-LM NLI methods by adding an entity-filled NoTA option, without threshold tuning.Source: §5.3 · p. 6

5.4 Few-Shot Results

  • Vanilla RE gains no improvement from labeled examples, and its few-shot demonstrations might bias predictions toward incorrect contextual relations.Source: §5.4 · p. 7
  • In few-shot RE, QA4RE yields gains over zero-shot results with random example selection, remains template-robust, and performs around 2% below the best NLI model.The few-shot QA4RE result does not surpass DeBERTa-based NLI using its own TEMP1 templates.Source: §5.4 · p. 7

6 Discussions

The discussion finds that QA4RE's gains come from reformulating RE as QA rather than simply adding relation explanations. It also compares QA4RE with NLI4RE and examines transfer across model sizes.

6.1 Are Relation Templates All LLMs Need?

  • QA4RE's gains mainly come from the QA reformulation: adding relation explanations to vanilla RE did not yield consistent or significant improvements.The comparison uses the same templates and type constraints on TACRED.Source: §6.1 · p. 7, Table 5 · p. 8

6.2 QA4RE vs. NLI4RE

  • NLI4RE underperforms vanilla RE for text-davinci-003, and evaluating each candidate relation requires multiple inference runs.The paper attributes the poor performance partly to the 0.5 threshold and the model not seeing the full relation space at once; the repeated runs add cost.Source: §6.2 · p. 8, Table 6 · p. 8

6.3 QA4RE & Model Size

  • QA4RE improves instruction-tuned models from FLAN-T5 Small (80M) through models up to 175B, but gains vary across model families and sizes.Larger FLAN-T5 models benefit more, while that trend does not continue for larger GPT-3.5 models; text-davinci-003 and FLAN-T5-XXL are highlighted as showing strong improvement.Source: §6.3 · p. 8, Table 7 · p. 8

7 Conclusions and Future Work

The paper concludes that casting RE as multiple-choice QA can leverage QA's broad coverage in instruction-tuning datasets. Experiments report improved zero-shot RE performance, with robustness, few-shot effectiveness, and transfer also explored.

7 Conclusions and Future Work

  • The paper reformulates RE as multiple-choice QA to leverage a task widely represented in instruction-tuning datasets, unlike RE.The authors report that experiments demonstrate QA4RE's effectiveness as a zero-shot relation extractor, especially for text-davinci-003 and FLAN-T5 XXL.Source: §7 · pp. 8–9
  • The paper reports experiments exploring QA4RE's robustness, few-shot effectiveness, and the training scenarios in which it is most effective.These are described as areas studied, not as a claim that every setting improves.Source: §7 · p. 9
  • Future work will explore aligning other underrepresented tasks with common instruction-tuning tasks and applying QA4RE to OPT-series and PaLM models.The authors say these models were not included because of limited computational resources and/or access.Source: §7 · p. 9

8 Limitations

The authors report that QA4RE did not improve an earlier LLM without strong instruction tuning, and that their few-shot exploration was limited. They leave the effects of more few-shot examples and cross-model trends in that setting unresolved.

8 Limitations

  • QA4RE produced no improvement for the earlier, not strongly instruction-tuned text-davinci-001 model.The reported gains therefore do not extend to every instruction-tuned LLM generation examined.Source: §8 · p. 9
  • The paper's few-shot exploration is more limited than its zero-shot experiments.Source: §8 · p. 9
  • It remains unclear whether adding more training examples improves few-shot RE performance or whether zero-shot trends across GPT-3 models hold in few-shot settings.Source: §8 · p. 9

9 Ethics Statement

The authors state that they anticipate no ethical issues from the research. Their dataset review finds RE rare and QA dominant in instruction-tuning datasets.

9 Ethics Statement

  • The authors state that they do not anticipate ethical issues regarding the research topics.Source: §9 · p. 9

A Instruction Dataset Portion

B.1 Hyperparameters for Few-Shot Methods

The appendix describes few-shot hyperparameter selection, prompt formats, and relation-verbalization templates used in the experiments.

B.1 Hyperparameters for Few-Shot Methods

  • Few-shot results average three random training subsets per K, while 100 randomly selected dev examples are used for hyperparameter search.For LLMs, the dev set selects among 1, 2, or 5 in-context examples; the demonstrations are type-constrained and sampled from the train set.Source: §B.1 · p. 13
  • Small-LM baselines use published code and hyperparameters where available; SuRE was not reproducible with its default hyperparameters.Fine-tuning, PTR, and KnowPrompt use dev-set grid searches, with training-token learning rates specified in Table 10.Source: §B.1 · p. 13, Table 10 · p. 13
  • Experiments used 8 NVIDIA GeForce RTX 2080 Ti and 2 NVIDIA RTX A6000 GPUs, with GPU hours and OpenAI API costs listed in Table 11..Source: §B.1 · p. 13, Table 11 · p. 13

B.2 Prompts for LLMs

  • The appendix lists the LLM prompt formats for vanilla, TEMP, NLI4RE, and QA4RE frameworks in Table 12.NLI4RE is shown with one template for simplicity.Source: §B.2 · p. 13

B.3 Relation Verbalization Templates

  • The four relation-verbalization styles include concrete examples, semantic relationships, straightforward relation labels, and word translation.For org:top_members/employees, examples include “{Eh} is a chairman/ president/director of {Et}” and “{Eh} is a high level member of {Et}.”Source: §B.3 · p. 13
  • The concrete-example templates come from Sainz et al. (2021), while the other three styles were explored by Lu et al. (2022).The authors use templates from the works’ official GitHub repositories and list LLM templates in Tables 13–15.Source: §B.3 · p. 14

C Full Test Results on FLAN-T5

Full-test results show consistent average gains from QA4RE across FLAN-T5 models, with larger models benefiting more. Smaller models do not improve on SemEval, where long QA4RE inputs may be difficult to understand.

C Full Test Results on FLAN-T5

  • QA4RE yields consistent, significant average improvements over vanilla RE across FLAN-T5 models on the four full-test datasets, with larger models benefiting more.Table 8 reports dataset results and marks the best and second-best scores, with F1 improvements over vanilla RE highlighted.Source: C Full Test Results on FLAN-T5 · p. 14
  • Smaller FLAN-T5 models do not improve on SemEval, a 19-choice QA task, possibly because they struggle with QA4RE’s long inputs..Source: C Full Test Results on FLAN-T5 · p. 14

Related papers

Loading 2305.11159v1…