PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models of Code are Few-Shot Commonsense Learners

Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, Graham Neubig

arXiv:2210.07128v3cs.CLcs.LG

TL;DR

Structured commonsense reasoning asks language models to generate graphs from natural-language inputs, but serialized graphs differ from their pretraining data. COCOGEN converts those structures into Python code for code-generation models, and across three tasks CODEX outperforms fine-tuned and few-shot natural-language baselines.

  • Problem

    Structured commonsense reasoning requires generating structured outputs from natural-language inputs, while serialized graphs strongly diverge from language-model pretraining corpora.

  • Method

    COCOGEN converts each output graph into semantically equivalent Python code, uses Code-LLMs to generate it from text, and converts the result back into a graph.

  • Results

    Across three structured commonsense generation tasks, COCOGEN with CODEX outperforms fine-tuned and few-shot natural-language models, including T5 and GPT-3.

  • Takeaways & Limitations

    Code-LLMs can serve as structured commonsense reasoners when desired graph predictions are represented in a code-like format.

  • Takeaways & Limitations

    Experiments use non-open-source models whose pretraining data, size, and biases are not fully documented, limiting precise characterization of those models.

Abstract

from arXiv · show

We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event -- or a reasoning-graph. To employ large language models (LMs) for this task, existing approaches ``serialize'' the output graph as a flat list of nodes and edges. Although feasible, these serialized graphs strongly deviate from the natural language corpora that LMs were pre-trained on, hindering LMs from generating them correctly. In this paper, we show that when we instead frame structured commonsense reasoning tasks as code generation tasks, pre-trained LMs of code are better structured commonsense reasoners than LMs of natural language, even when the downstream task does not involve source code at all. We demonstrate our approach across three diverse structured commonsense reasoning tasks. In all these natural language tasks, we show that using our approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task (e.g., T5) and other strong LMs such as GPT-3 in the few-shot setting.

1 Introduction

Structured commonsense reasoning requires generating graphs from natural-language inputs, but serialized graph formats diverge from language-model pretraining. COCOGEN instead represents structures as Python code and evaluates code LMs across three tasks.

  • Structured commonsense reasoning generates graphs or other structures from natural-language inputs, requiring both learned language knowledge and complex structured prediction.
  • Serialized graphs are loosely structured, verbose, and unlike typical pretraining text, causing topology, structural, and semantic errors.
  • COCOGEN converts desired commonsense graphs into Python code, aligning their representation with code-model pretraining data.
  • Code-LLMs are presented as better structured commonsense reasoners than NL-LLMs when graph predictions are represented as code.
  • The evaluation spans three structured commonsense generation tasks and compares COCOGEN with fine-tuned and few-shot natural-language models.

2 COCOGEN: Representing Commonsense structures with code

COCOGEN transforms each text–structure pair into semantically equivalent Python code, prompts a code-generation model with such examples, and converts the completion back into the target graph.

  • COCOGEN transforms an output graph G into a semantically equivalent program Gc, generates Gc from text T, then converts it back into G.
  • Python is used because it is popular in modern Code-LLM training data, although the approach is programming-language agnostic.
  • A text–structure pair is encoded as a Python class or function using attributes, comments, hashmaps, object attributes, or function calls.
  • For script generation, a Tree class stores the goal as an attribute, nodes as Node objects, and edges through children attributes.
  • The empirically most effective representation is relatively simple, especially for larger models.
  • Few-shot prompts contain k Python classes, while inference appends a partial class specifying the goal for CODEX to complete with nodes and dependencies.
  • CODEX nearly always generates syntactically valid Python, enabling straightforward conversion to graphs and evaluation with original dataset metrics.

3 Evaluation

The evaluation tests COCOGEN with CODEX against natural-language few-shot and fine-tuned baselines across diverse structured commonsense generation tasks. Results cover script generation, edge prediction, and entity state tracking, using semantic and structural metrics with controlled few-shot prompts.

  • Tasks: The evaluation spans script generation, edge prediction, and entity state tracking, which require different structured outputs and reasoning.The tasks are PROSCRIPT, PROSCRIPT edge prediction, and PROPARA entity state tracking.
  • Evaluation setup: The evaluation controls prompt variation by repeating experiments with at least three randomly sampled prompts, with k ranging from 5 to 30 examples.Prompt sizes depend on task-specific input length and model context limits.
  • Models and baselines: COCOGEN uses CODEX with Python prompts, while baselines include few-shot CURIE and DAVINCI and fine-tuned T5 models.Few-shot models receive sampled input-output examples, whereas fine-tuned models observe the entire downstream training data.
  • Results: For script generation, COCOGEN with 15 prompt examples outperforms fine-tuned T5 trained on all 3500 samples and few-shot CURIE across semantic and structural metrics.COCOGEN also beats DAVINCI on semantic metrics, although DAVINCI is slightly better on two structural metrics.
  • Results: For PROSCRIPT edge prediction, COCOGEN significantly outperforms CURIE and DAVINCI and exceeds fine-tuned T5 using only 15 examples versus T5’s 100.Because the node set is given, this task evaluates prediction of graph structure independently of natural-language content generation.
  • Results: For PROPARA, COCOGEN achieves 5.0 higher average F1 than DAVINCI across five prompts and reaches F1 63.0 against CURIE.With three prompt examples, COCOGEN is state-of-the-art among few-shot settings and is within 10 F1 points of the current state of the art.

EXPLAGRAPHS

EXPLAGRAPHS generates explanation graphs from beliefs, arguments, and stances, using typed edges to represent support or counter relations. COCOGEN achieves strong performance with few-shot code generation.

  • EXPLAGRAPHS generates a graph connecting a belief and an argument to support or counter the belief.
  • The dataset provides the stance, using a stance prediction model's output while focusing evaluation on argument-graph generation.
  • COCOGEN converts beliefs, arguments, and stances into Python variables and represents typed graph edges with add_edge function calls.
  • Structural accuracy measures connected DAGs with two belief and two argument concepts, while semantic correctness evaluates whether the stance is inferred correctly.
  • COCOGEN with 30 examples outperforms T5 fine-tuned on 1500 examples across all metrics and beats DAVINCI and CURIE by about 50%-100%.

4 Analysis

The analysis separates the contributions of code models, structured prompts, retrieval, and Python formatting. Results show complementary gains from code models and code prompts, while retrieval and format choices depend on task and model size.

  • Structured Prompts vs. Code-LLMs: Both Code-LLMs and structured code prompts improve performance, with CODEX using a code prompt outperforming alternatives across all metrics.
  • Structured Prompts vs. Code-LLMs: The combined use of a Code-LLM and code prompt improves performance more than the sum of either component alone.
  • Dynamic prompt selection: Dynamic prompt selection helps PROSCRIPT edge prediction and script generation, where similar scripts provide useful information.
  • Dynamic prompt selection: EXPLAGRAPHS contains near-duplicate training examples, reducing diversity and effective prompt size when prompts are dynamically selected.
  • Python Formatting: No Python class design works uniformly across tasks; larger models are less sensitive, and formats resembling typical code generally help most.
  • Human evaluation: Human evaluation closely correlates with automated metrics; COCOGEN is more relevant and correct for EXPLAGRAPHS and generally more relevant for PROSCRIPT.

5 Related work

Prior work commonly serializes structured outputs as strings, whereas this paper translates reasoning tasks into Python and uses code-generation models to preserve structure during generation.

  • Existing structured commonsense methods flatten output graphs into strings and struggle to generate well-formed outputs.
  • COCOGEN addresses structured generation by translating the task into Python code and generating that code with large code-generation models.
  • The paper extends code representations beyond procedural knowledge tasks to broader structured commonsense reasoning.
  • Unlike prior work converting natural-language mathematics into formal structures, COCOGEN translates reasoning outputs into Python to match code-model pre-training data.
  • Structured commonsense reasoning requires both broader information than the prompt and capabilities for reasoning and symbolic manipulation.

6 Conclusion

The paper presents COCOGEN as a simple approach that converts commonsense structures into Python code for Code-LLMs. Its results support code generation as a direction for structured commonsense reasoning and related NLP tasks.

  • The paper presents the first use of large language models of code for structured commonsense generation.
  • COCOGEN converts commonsense structures into Python code to leverage Code-LLMs' code-generation abilities for structured generation.
  • The authors believe the paper's principles and methods apply to additional NLP tasks requiring language understanding and structured prediction.

Limitations

The paper reports limitations involving opaque proprietary models, uncertain future access, English-only evaluation, and mixed results from dynamic prompt retrieval.

  • Model access and transparency: Some experiments use proprietary DAVINCI, CURIE, and CODEX models whose pretraining data, sizes, and biases are not fully documented.The authors provide only educational guesses about these details.
  • Model access and transparency: Future research access to CODEX is uncertain because its free research availability may not continue.The authors release code and model outputs to support reproducibility.
  • Language scope: The evaluation is limited to English datasets, leaving cross-lingual structured generation unexplored.The authors identify cross-lingual evaluation as future work.
  • Model comparison: CODEX and DAVINCI have comparable capacities, supporting a fair comparison between them.DAVINCI is stronger on text generation, whereas CODEX is stronger on code generation.
  • Prompt retrieval: Dynamic knowledge-similarity retrieval yields marginal gains for PROSCRIPT but hurts EXPLAGRAPHS when nearest training examples overlap.Overlapping examples reduce prompt diversity and effective information.

C Human Evaluation

Human evaluation compares COCOGEN and DAVINCI graphs for relevance and correctness, complementing automated metrics. COCOGEN is stronger on EXPLAGRAPHS, while the models show complementary strengths on PROSCRIPT.

  • Evaluation setup: Three authors evaluated 20 examples by choosing which COCOGEN or DAVINCI graph was more relevant and correct.Annotators could also select no preference.
  • Findings: Human judgments closely correlate with automated metrics, and annotators preferred COCOGEN graphs for EXPLAGRAPHS relevance and correctness.The cited results identify EXPLAGRAPHS as the clearest human-evaluation advantage.
  • Prompt-selection context: Dynamic prompt retrieval produces marginal PROSCRIPT gains but hurts EXPLAGRAPHS when similar training examples reduce prompt diversity.These results contextualize why prompt-selection strategies may not uniformly improve human-evaluated outputs.
  • Findings: DAVINCI often recovers relevant events but arranges semantic relations incorrectly in argument graphs.The example contrasts correctly structured COCOGEN relations with incorrect DAVINCI relations.
  • Findings: For PROSCRIPT, human evaluation indicates that COCOGEN and DAVINCI have complementary strengths.The supplied passage does not specify the exact preference breakdown.

D Dataset statistics

The appendix reports dataset statistics for the structured commonsense tasks and notes split availability for EXPLAGRAPHS and PROSCRIPT.

  • Dataset splits: Because the EXPLAGRAPHS test split is unavailable, evaluation uses its validation split.The PROSCRIPT test splits were obtained from the dataset authors.
  • Dataset splits: For PROSCRIPT, the authors obtained test splits directly from the dataset authors.
  • Dataset statistics: Dataset statistics for the tasks used in the work are reported in Table 12.The supplied passages do not include the table’s numerical values.

E Sample outputs

COCOGEN represents structured outputs through Python-oriented formats and provides task examples, format comparisons, and reproducibility materials. The outputs are usually syntactically valid Python graphs, and larger CODEX models are less prompt-sensitive.

  • Sample outputs: COCOGEN uses CODEX with Python prompts to generate structured outputs for the evaluated tasks.The prompts and model outputs are publicly referenced for reproduction.
  • Sample outputs: COCOGEN generates syntactically valid Python graphs similar to task graphs or tables in nearly 100% of cases.
  • Output formats: Python representations include NetworkX classes, DOT-like classes, and tree formats for PROSCRIPT.Figure 7 also presents three designs for EXPLAGRAPHS.
  • Output formats: Table 14 compares CODEX-001 and CODEX-002 across PROSCRIPT edge-prediction formats, with the literal format performing best for CODEX-002.
  • Model size and prompts: As model size increases from CODEX-001 to CODEX-002, prompt sensitivity decreases, suggesting prompt design may become easier for very large models.
  • Experimental variation: Experiments use three random seeds to vary example order, with minimal variance across runs and statistically significant COCOGEN improvements over DAVINCI.The reported significance threshold is p-value < 0.001.
Loading 2210.07128v3…