Code4Struct: Code Generation for Few-Shot Event Structure Prediction

1 Introduction

• Code can represent target structures more straightforwardly than natural language, which can require careful prompt engineering.Programming-language features such as type annotations, argument defaults, and inheritance can constrain outputs and reuse parent-event components while preserving child-event properties.Source: §1 · pp. 1–2
• For EAE, Code4Struct converts the entity and event ontology into Python classes, then prompts an LLM with relevant class definitions and the input sentence to generate an event instance.The generated code can be mapped to an instance graph of a Transport event.Source: Figure 1 · p. 1, §1 · p. 2
• Code4Struct proposes code generation for structured prediction and uses Event Argument Extraction as its case study.EAE extracts event structures from unstructured text.Source: §1 · p. 2
• With sufficient in-context examples, the code prompt is generally advantageous over two text-prompt variants across different LLMs.The experiments compare code-based and text-based prompts on different LLMs.Source: §1 · p. 2
• 20-shot CODE4STRUCT rivals models trained on 4,202 instances and beats a SOTA approach by 29.5% absolute F1 with equal 20-shot data.Zero-shot CODE4STRUCT also outperforms the SOTA on both 20- and 50-shot settings.Source: §1 · p. 2
• Using 10-shot sibling-event examples improves zero-resource event-type prediction by 12% F1 over the zero-shot baseline.The approach integrates the event ontology hierarchy through class inheritance.Source: §1 · p. 2

2 Code Generation Prompt Construction

• The prompt combines ontology code, optional k-shot examples, and a task prompt containing a partial event-class instantiation for completion.The three components are ordered as ontology, examples, then task prompt.Source: §2 · p. 3, Figure 2 · p. 3

2.1 Ontology Code Representation

• The ontology representation concatenates base, entity, and event class definitions; entity classes inherit from Entity and include ACE-based descriptions as docstrings.Entity type names serve as class names, such as GPE.Source: §2.1 · p. 3

2.1.1 Event Class Definition

• Each event type is represented as a class that inherits from its parent event class, or from Event when it has no parent.ACE's event hierarchy is mirrored in the class definitions.Source: §2.1.1 · p. 3
• Argument roles become constructor inputs with type annotations and same-named class members, while event-description templates replace role names with member variables.For example, a role can accept a list of entities drawn from specified types; the templates are modified from Li et al. (2021).Source: §2.1.1 · p. 3

2.2 Task Prompt

• The task prompt pairs an instruction and input sentence with incomplete event-instantiation code, and marks the ground-truth trigger words with asterisks.The partial instantiation prompts the model to complete the event class.Source: §2.2 · pp. 3–4
• Generation stops when the model emits a triple quote, class, print, or comment marker, to limit output to the EAE task.Source: §2.2 · p. 4

2.3 In-context Learning

• For k-shot prompting, examples pair task prompts with ground-truth event instantiations and are selected from training instances of the same event type in training-set order.Source: §2.3 · p. 4

3 Why Represent Event Structure in PL?

• EAE prompts trade off effort spent defining the output format against the benefit of language-model pretraining.Text2Event uses a concise but unnatural output format, while manual templates ask models to fill in text.Source: §3 · p. 4
• Code4Struct maps event structures into code, using code-LLMs' text-to-code ability and programming-language features to describe structures in a format aligned with pretraining.The cited code-LLM examples are Chen et al. (2021) and Nijkamp et al. (2022).Source: §3 · p. 4
• The authors report non-trivial zero-shot performance on unseen event types and comparable performance to fully supervised methods with 20 examples per event type.Source: §3 · p. 4

4.1 Experiment Setup

• Experiments use ACE05-E with gold event types and triggers, comparing three LLMs under greedy decoding and evaluating argument identification (Arg-I) and classification (Arg-C).ACE05-E has 8 parent and 33 child event types, with 14 child types having fewer than 50 training instances; generation uses temperature 0 and a 128-token limit.Source: §4.1 · p. 5

4.2 Comparison with Text Prompt

• Code prompts outperform both text variants on Arg-C F1 for all tested LLMs except text-davinci-003 when k ≥5; text prompt T (2) performs poorly at k ≤1.The low-shot T (2) gap is attributed primarily to difficulty producing the requested structure from a language-described format, highlighting the need for prompt engineering with language prompts in low-shot settings.Source: §4.2 · p. 5

4.3 Comparison with different LLM

• At 20-shot, text-davinci-003 reaches 60.4% Arg-C F1 versus 56.5% for text-davinci-002, while code prompts are generally stronger than text prompts except on text-davinci-003.The code–text Arg-C gap grows with more examples for the *-davinci-002 models; text-davinci-003 can perform on par with or slightly below T (2) at k ≥5.Source: §4.2 · p. 5, §4.3 · p. 6

4.4 Comparison with different k

• The first in-context example brings the largest reported Arg-C F1 gains (+11.8, +10.2, and +6.9 for the three LLMs), while scores generally rise with diminishing returns as k increases.For code-davinci-002 at k ≥20, Arg-I and Arg-C plateau or slightly decline; only 19 of 33 event types have more than 50 examples, and performance may plateau as additional examples add little sentence variability.Source: §4.4 · pp. 6–7, Figure 3 · p. 6

4.5 Comparison with Supervised Models

• 20-shot Code4Struct with text-davinci-003 exceeds DEGREE by +29.5% Arg-C F1 and is comparable to supervised models trained on 4,202 instances.Zero-shot Code4Struct with text-davinci-003 also exceeds DEGREE’s 20-shot and 50-shot Arg-I and Arg-C performance; the comparison table notes averages of 18 and 39 examples per type for 20-shot and 50-shot prompts.Source: §4.5 · p. 7, Table 4 · p. 7

4.6 Event Type Hierarchy Improves Zero-resource EAE

• Sibling-type examples match same-type performance at 1-shot and 10-shot, with +9% and +12% Arg-C F1 gains over zero-shot, respectively.Random non-sibling examples also help, but less than sibling examples at 10-shot; Table 5 aggregates 23 test event types with more than 10 training instances.Source: §4.6 · p. 8, Table 5 · p. 8

Code-LLM for Structured Task

• Code4Struct uses inheritance and type annotation to introduce information and constraints for structured prediction, a capability the authors say prior code-based structured-task work largely overlooks.Prior work cited includes code for embodied-agent plans, commonsense reasoning graphs, and math or symbolic reasoning delegated to an interpreter.Source: §5 · p. 8

Event Extraction

• Event-extraction approaches include joint classification, reading comprehension, and text generation, while Text2Event generates structures in a manually specified linearized format.The authors contrast this with generating structure directly in programming-language code to use LLM knowledge of programming languages.Source: §5 · p. 8

Conclusions

• With 20-shot data, Code4Struct is comparable to fully supervised models trained on 4,202 instances and exceeds a state-of-the-art model by 29.5% absolute F1.With 10-shot sibling-type examples, it improves zero-shot performance for zero-resource event types by 12% absolute F1.Source: §6 · p. 8
• Code4Struct applies code generation to event argument extraction by converting natural-language sentences into event structures.The paper frames it as a structured prediction approach for NLP using LLMs trained on language and code.Source: §6 · p. 8

Future Work

• Future work will extend Code4Struct to relation prediction and schema matching and explore executable programming-language features for structured prediction.Source: §6 · p. 8

• The method assumes event triggers and argument templates are given, so it requires an event detection system to provide triggers and event types before argument generation.Source: Limitations · p. 9
• Only two hierarchy levels from ACE05-E are explored, limiting coverage of complex real-world event hierarchies; the ontology is also human-curated.Source: Limitations · p. 9
• LLM inference is relatively slow and costly compared with supervised approaches, and the models used generally have over 100x as many parameters.Source: Limitations · p. 9

• Toxic-language risk is relatively low when the test text is not toxic, but discrimination and bias remain possible in the foundational LLMs used.Source: Ethical Considerations · p. 9

• With zero-shot prompting, the model inferred plausible arguments absent from the text, including a U.S. retirement location and a court for a fine event.Source: §A.1 · p. 11
• Increasing examples from zero to 50 shots inhibited argument generation, including commonsense inferences and the correctly predicted SEC argument.Source: §A.1 · p. 11

A.2 Prompt Component Analysis

• The qualitative prompt examples show AMR appended after the input sentence, keywords incorporated into event definitions, and hierarchical definitions with sibling-event examples.Source: §A.2 · p. 11, Figure A.7 · p. 17, Figure A.8 · p. 17
• Event-keyword and AMR augmentations perform slightly better than CODE4STRUCT on all metrics in zero-shot settings.Source: §A.2 · p. 11, Table A.1 · p. 12

A.3 Ablation Study

• Removing event-trigger marking consistently degrades performance on all metrics across varying numbers of in-context examples.Source: §A.3 · p. 12, Table A.2 · p. 12
• Removing event descriptions raises zero-shot Arg-I recall from 78.7 to 81.8, but yields consistently worse few-shot Arg-C than CODE4STRUCT.Source: §A.3 · pp. 12–13, Table A.2 · p. 12
• Type annotation consistently helps when k ≥20, while hierarchical event definitions benefit performance at k ≥20 and perform on par with CODE4STRUCT below 20 shots.Source: Table A.2 · p. 12, §A.3 · p. 13

A.4 In-context Example Variability Analysis

• Mean example variability increases with diminishing returns as k grows, and its positive correlation with Arg-C F1 suggests limited variability gains may contribute to performance plateaus.Source: §A.4 · p. 13, Figure A.2 · p. 13, Figure A.3 · p. 13

A.2 Prompt Component Analysis

• The text-prompt comparison includes code-imitation and BART-Gen-style prompts; 50-shot results for text prompts are unavailable because their prompts exceed input-token limits.Source: Table A.3 · p. 14, Figure A.5 · p. 15, Figure A.6 · p. 16
• A text-prompt example assigns Anne-Marie to both adjudicator and plaintiff roles in a Justice:Sue event, illustrating a role-assignment error to compare with the code prompt.Source: Figure A.4 · p. 14