PaperlayerView exact revision on arXiv

Source-linked AI summary

GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, Eneko Agirre

arXiv:2310.03668v5cs.CL

TL;DR

IE models struggle with unseen schemas because annotation guidelines contain task-specific definitions and exceptions that label names do not capture. GoLLIE fine-tunes an LLM to use those guidelines through structured representations and regularization, and comprehensive zero-shot evaluation shows stronger results than prior approaches. The paper concludes that guidelines are valuable for zero-shot IE, while ambiguous labels and domain-specific needs remain boundaries.

  • Problem

    IE models transfer poorly to new schemas because task-specific annotation guidelines contain definitions and exceptions beyond label names.

  • Method

    GoLLIE fine-tunes an LLM to follow annotation guidelines encoded in structured Python representations, using training regularization to discourage memorization.

  • Results

    GoLLIE outperforms previous zero-shot IE attempts and achieves similar supervised results to the baseline.

  • Takeaways & Limitations

    Annotation guidelines are valuable for LLM-based zero-shot IE, and detailed definitions and representative candidates complement each other.

  • Takeaways & Limitations

    GoLLIE remains challenged by ambiguous or coarse labels and can require task-specific techniques on datasets such as WNUT and NCBIDisease.

Abstract

from arXiv · show

Large Language Models (LLMs) combined with instruction tuning have made significant progress when generalizing to unseen tasks. However, they have been less successful in Information Extraction (IE), lagging behind task-specific models. Typically, IE tasks are characterized by complex annotation guidelines that describe the task and give examples to humans. Previous attempts to leverage such information have failed, even with the largest models, as they are not able to follow the guidelines out of the box. In this paper, we propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. Comprehensive evaluation empirically demonstrates that GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous attempts at zero-shot information extraction. The ablation study shows that detailed guidelines are key for good results.

1 INTRODUCTION

Information Extraction is difficult because schemas contain detailed, exception-filled guidelines, while current models transfer poorly to new schemas. GoLLIE addresses this open challenge by fine-tuning an LLM to follow guidelines for zero-shot IE.

  • IE guidelines contain granular definitions and numerous exceptions that human annotators must follow.
  • Performance of current state-of-the-art IE models decreases substantially on new annotation schemas.
  • Different schemas assign different meanings to identical labels, so label names alone are insufficient for IE.
  • Current LLMs fail to follow annotation guidelines out of the box, with prompted gpt-3.5-turbo reaching around 20 F1 on Music and Politics.
  • GoLLIE is fine-tuned to attend to guidelines on a small set of established IE tasks and outperforms prior zero-shot IE methods.

2 RELATED WORK

Although LLMs generalize to unseen tasks through instruction tuning, IE systems have lagged behind task-specific models because prompts often represent schemas only through label names. Annotation guidelines are therefore central to transferring IE behavior across schemas.

  • Instruction tuning has become a leading approach for enabling LLM generalization to unseen tasks.
  • Encoder-only models remain highly effective for IE, while LLM and natural-language-instruction approaches have generally lagged behind them.
  • The paper defines a schema as the set of labels and their definitions.
  • Many IE instruction-tuning methods prompt with label names alone, despite datasets assigning different meanings to the same labels.

3 APPROACH

GoLLIE represents IE inputs and outputs as structured Python code and incorporates annotation guidelines directly into that representation. Training regularization is used to make the model attend to guideline details rather than memorize datasets or labels.

  • GoLLIE forces attention to guideline details and is designed to perform robustly on schemas unseen during training.
  • Python code provides a unified, human-readable representation for inputs and outputs across IE tasks.
  • 3.1 INPUT-OUTPUT REPRESENTATION: The input format combines schema definitions, input text, and output annotations, with labels represented as classes and guidelines as docstrings.
  • 3.2 GUIDELINES ENHANCED REPRESENTATION: Guideline-enhanced representations encode label definitions as class docstrings and representative candidates as comments.
  • 3.3 TRAINING REGULARIZATION: Training noise includes class-order shuffling, class dropout, guideline paraphrasing, and representative-candidate manipulation to discourage memorization.

4 EXPERIMENTAL SETUP

The experiments separate training and evaluation mainly by domain, train across five IE tasks, and evaluate three of them in zero-shot settings. The setup includes dataset-specific modifications and comparisons with closely related baselines.

  • 4.1 DATA: Training uses mostly News and Biomedical datasets, while evaluation uses datasets from diverse domains to reduce similarity between splits.
  • 4.1 DATA: GoLLIE is trained on NER, RE, EE, EAE, and SF, but evaluated on NER, EE, and EAE.
  • 4.1 DATA: Dataset modifications make the system incomparable with state-of-the-art results on some supervised tasks, while CASIE evaluation uses event-category and partial argument matching.
  • 4.1 DATA: Guidelines come from dataset authors when available; otherwise, human experts create them from development annotations.
  • 4.2 LANGUAGE MODELS AND BASELINES: The model is a fine-tuned Code-LLaMA, with QLoRA used for training.
  • 4.2 LANGUAGE MODELS AND BASELINES: Instruct-UIE is the closest comparison system because it uses a similar zero-shot IE setting without guidelines.

5 RESULTS

GoLLIE improves zero-shot IE when using annotation guidelines, substantially outperforming prior zero-shot systems while remaining close to the supervised baseline. Results also show that task-specific techniques remain useful for some datasets, and representative annotation items contribute strongly to performance.

  • Supervised evaluation: GoLLIE remains close to the supervised baseline, with an average absolute difference of 0.3 F1 points.This result holds despite noise introduced during GoLLIE fine-tuning to promote guideline generalization.
  • Supervised evaluation: Task-specific techniques are still needed on WNUT and NCBIDisease, where GoLLIE underperforms significantly.External knowledge and biomedical-domain pretraining are cited as complementary improvements.
  • Zero-shot evaluation: 13 F1 points on average separate guideline-based results from the baseline across almost every zero-shot dataset.The reported difference is an absolute improvement over the baseline.
  • Zero-shot evaluation: GoLLIE surpasses Instruct-UIE and Entailment-based IE by a large margin in zero-shot information extraction.The comparison uses fewer training datasets while incorporating guideline information.
  • Zero-shot evaluation: GoLLIE performs significantly better than PromptNER across all compared datasets, despite both approaches incorporating guideline information.PromptNER adds definitions to prompts, whereas GoLLIE is fine-tuned to follow guidelines.
  • Ablation study: Definitions and representative candidates in guidelines complement each other, while class dropout yields only small significant improvements.Class order shuffling, guideline paraphrasing, and class name masking show no significant contribution.

6 ERROR ANALYSIS

GoLLIE generally follows detailed annotation guidelines, but errors remain where labels are ambiguous, conflict in granularity, or trigger strong pretraining-based expectations.

  • GoLLIE successfully follows guidelines for polysemous labels such as MEDIA, VULNERABILITYPATCH, TRAILER, and TASK, while the baseline struggles with insufficient label information.The label names alone do not determine the intended categorization.
  • GoLLIE labels years as TIME according to MultiNERD’s guidelines, although the dataset annotations omit those entities.This creates an evaluation mismatch between the model’s guideline-compliant output and the dataset labels.
  • GoLLIE also fails on ambiguous labels such as MISCELLANEOUS and PLOT, whose broad meanings hinder consistent tagging.MISCELLANEOUS covers entities outside predefined categories, while PLOT spans events, characters, and countries of origin.
  • GoLLIE can assign fine-grained labels such as WRITER, but may also assign the overlapping coarse PERSON label to scientists.The paper considers this output not technically incorrect because every scientist is a person.
  • GoLLIE outperforms the baseline on POLITICAL PARTY, yet often predicts ORGANIZATION because its fine-tuning data lacked the POLITICAL PARTY entity.The authors propose expanding the number and diversity of fine-tuning datasets to address this preconception.
  • The authors expect GoLLIE to perform best with well-defined guidelines and identify ambiguous or coarse labels as continuing challenges.They propose more specific instructions and more diverse pre-training data as future improvements.

7 CONCLUSIONS

The paper concludes that fine-tuning GoLLIE to comply with annotation guidelines improves zero-shot information extraction and supports generalization to unseen IE tasks. Future work targets broader training data, more complex tasks, and better handling of ambiguous labels.

  • GoLLIE is fine-tuned to comply with annotation guidelines, and comprehensive zero-shot evaluation shows better results than prior zero-shot IE approaches.The compared approaches either do not leverage guidelines or use models not fine-tuned to follow them.
  • The authors plan to expand and diversify pre-training datasets and add more complex tasks to improve GoLLIE’s flexibility.They also aim to expand instructions for ambiguous and coarse labels.
  • GoLLIE’s training includes NER, EE, EAE, RE, and SF data, with SF prompts adding slot definitions, optional arguments, fine-grained types, and query constraints.These design choices condition the model to produce the requested structured information.

A.1 EXAMPLE OF GENERALIZATION TO NEW CUSTOM TASKS

GoLLIE can generalize to user-defined annotation schemas expressed in Python code, including structured templates with typed attributes and arguments.

  • Users can define custom annotation schemas in Python code, including Template types with additional arguments beyond ordinary entities.The example defines Launcher and Mission templates with arguments such as the launcher’s space company or crew.
  • The model’s output satisfies schema constraints by filling string attributes with strings and list attributes such as crew with lists.It correctly analyzes a sentence under the newly created schema.
  • Figure 6 presents a TACRED example converted into a Slot Filling task represented as code.

B PERFORMANCE IN SEEN VS UNSEEN LABELS: FURTHER ANALYSIS

GoLLIE is evaluated on seen and unseen labels, where annotation-guideline differences matter alongside label overlap. It is more robust to label preconceptions and produces very few hallucinated or unparsable outputs.

  • Seen versus unseen labels: Table 6 categorizes zero-shot labels as seen or unseen using overlap with training labels and similar labels such as GEOPOLITICAL entity.The classification treats COUNTRY as exposed during training because a similar label appeared there.
  • Seen versus unseen labels: Micro-F1 is reported separately for seen and unseen labels across zero-shot datasets.
  • Seen versus unseen labels: GoLLIE generally performs better on both seen and unseen labels and is more robust than the baseline to strong label preconceptions.Guideline differences can make an overlapping label behave like an unseen one, while GoLLIE’s guideline adherence improves generalization.
  • Model hallucinations: Less than 1% of predictions contain either outputs impossible to parse or labels hallucinated outside the input classes.The evaluation removes hallucinated labels and treats unparsable outputs as empty lists.
  • Training objective: Training regularization randomly shuffles guideline order, masks names, or drops classes, so loss is computed only on output tokens to prevent guideline-token loss from dominating.

D.2 DATASET DETAILTS

The experiments use standard dataset splits and a controlled training setup, while comparing full-model fine-tuning with LoRA-layer training and documenting computational resources.

  • Training setup: OntoNotes was sampled at 30,000 random examples per epoch because it is orders of magnitude larger than the other datasets.Models were trained for 3 epochs with an effective batch size of 32 and a learning rate of 3e-4, yielding 15,485 training steps.
  • Dataset splits: Training used standard train, development, and test splits, with ACE and CASIE receiving dataset-specific split procedures.ACE followed Lin et al. (2020), while CASIE used the first 200 instances for validation and the last 2000 for testing.
  • Resource accounting: The study documents per-run training resources and estimates carbon footprint using 400W per GPU and a carbon intensity of 0.141 kg/kWh.All experiments were conducted on the authors’ private infrastructure.
  • Parameter-efficient tuning: Preliminary experiments compared QLoRA with training all model parameters using LLaMA2 7B under an identical experimental setup with guideline prompts.The comparison included training loss and F1 scores at the end of the first and third epochs.

E HANDLING DATASETS WITH HUNDREDS OF LABELS AND CODE-STYLE

GoLLIE’s code-style prompts add token overhead, but datasets with many labels primarily strain the input context through lengthy guideline definitions; batching labels can mitigate this limitation.

  • Many-label datasets: Datasets with hundreds of labels can exceed current LLM context sizes when label guidelines are included, limiting GoLLIE’s scope.The study focuses on datasets with fewer than 20 labels and identifies large-label datasets as a limitation.
  • Many-label datasets: Batching labels into multiple inputs provides a proposed mitigation for datasets whose label guidelines do not fit in one context.The paper illustrates splitting 100 labels into ten inputs of 10 labels and combining the outputs.
  • Code-style prompts: Python-based code-style prompts require additional input tokens, but detailed guidelines can occupy far more space than the code structure.For generic OntoNotes PERSON guidelines, code accounts for almost half the input characters; for detailed HarveyNER POINT guidelines, it is only a small portion.
  • Adapting to new datasets: Applying GoLLIE to new datasets requires defining labels and guidelines within predefined task templates.Datasets lacking public guidelines may require human experts to construct them from development examples.
  • Evaluation caveat: The evaluation may be affected by possible Code-LLaMA data contamination because the authors could not inspect its pre-training data.Comparisons with the baseline retain their relative validity because both models use the same backbone and pre-training.
Loading 2310.03668v5…