Source-linked AI summary
Generated Knowledge Prompting for Commonsense Reasoning
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi
TL;DR
The paper examines whether external knowledge remains useful for commonsense reasoning with large pretrained models and whether it can be integrated flexibly. It introduces generated knowledge prompting, which generates question-related statements and adds them to inference prompts. The method improves zero-shot and finetuned models across four benchmarks and reaches state-of-the-art results on three.
Problem
It remains unclear whether external knowledge benefits commonsense reasoning with increasingly large pretrained models, while existing integration methods face coverage and task-specific supervision constraints.
Method
Generated Knowledge Prompting generates flexible natural-language knowledge statements from a language model and provides them as additional input when answering questions.
Results
The method improves zero-shot and finetuned models across four commonsense benchmarks and sets new state-of-the-art results on three datasets.
Takeaways & Limitations
The results highlight large language models as flexible sources of external knowledge for commonsense reasoning.
Abstract
from arXiv · showhide
It remains an open question whether incorporating external knowledge benefits commonsense reasoning while maintaining the flexibility of pretrained sequence models. To investigate this question, we develop generated knowledge prompting, which consists of generating knowledge from a language model, then providing the knowledge as additional input when answering a question. Our method does not require task-specific supervision for knowledge integration, or access to a structured knowledge base, yet it improves performance of large-scale, state-of-the-art models on four commonsense reasoning tasks, achieving state-of-the-art results on numerical commonsense (NumerSense), general commonsense (CommonsenseQA 2.0), and scientific commonsense (QASC) benchmarks. Generated knowledge prompting highlights large-scale language models as flexible sources of external knowledge for improving commonsense reasoning. Our code is available at https://github.com/liujch1998/GKP
1 Introduction
The paper asks whether external knowledge still benefits commonsense reasoning with large pretrained models, despite limited knowledge-base coverage and task-specific integration requirements. It proposes generated knowledge prompting and reports improvements across commonsense benchmarks.
- External knowledge may help commonsense reasoning, but its benefits may diminish as pretrained models grow larger and absorb more raw text.
- Prior approaches face limited knowledge-base coverage and often require task-specific supervision for integrating knowledge.
- The paper investigates external knowledge on large state-of-the-art pretrained models across four commonsense benchmarks without structured knowledge bases or joint integration finetuning.
- Generated Knowledge Prompting generates question-related knowledge statements with a language model and supplies them as input alongside the question.
- Table 1 presents examples in which generated knowledge changes prediction scores and rectifies model answers.
- Few-shot prompting elicits flexible natural-language knowledge statements beyond predefined templates, supporting varied settings without finetuning.
- The method improves zero-shot and finetuned models across numerical, general, and scientific commonsense benchmarks, achieving new state-of-the-art results on three datasets.
- Performance depends on knowledge quality, knowledge quantity, and the inference-time integration strategy; generated statements can support explicit reasoning such as deduction.
2 Generated Knowledge Prompting
Generated Knowledge Prompting answers multiple-choice commonsense questions by generating question-conditioned knowledge and integrating it into inference. The method uses few-shot demonstrations, knowledge-augmented prompts, and confidence-based answer selection without further knowledge-prompting finetuning.
- 2 Generated Knowledge Prompting: A multiple-choice commonsense task predicts an answer from a finite, question-dependent set of text choices.
- 2 Generated Knowledge Prompting: The method uses two steps: knowledge generation followed by knowledge integration during inference.
- 2 Generated Knowledge Prompting: Knowledge generation samples M question-conditioned natural-language statements from a language model.
- 2 Generated Knowledge Prompting: Each knowledge statement is a variable-length text sequence intended to contain information helpful for answering the question.
- 2.1 Knowledge Generation: Few-shot generation prompts contain an instruction, five human-written question-knowledge demonstrations, and a placeholder for the new question.
- 2.1 Knowledge Generation: For a new question, the model repeatedly samples prompt continuations to form Kq = {k1, k2, . . . , kM}.
- 2.2 Knowledge Integration via Prompting: Integration forms knowledge-augmented questions by concatenating each generated statement with the original question.
- 2.2 Knowledge Integration via Prompting: The inference model scores answer choices using the augmented question that best supports each choice, favoring strongly supportive knowledge statements.
3 Experimental Setup
The study evaluates generated knowledge prompting across four commonsense reasoning datasets using zero-shot and finetuned inference models, with task-specific setup choices and baseline comparisons.
- Implementation Details: Knowledge generation uses GPT-3 few-shot prompting to produce 20 statements per question, with a CSQA2 exception using five statements and longer generations.The standard generation uses nucleus sampling p = 0.5 and stops at 64 tokens or a newline; CSQA2 allows up to 128 tokens.
- Datasets and Task Setup: The evaluation covers NumerSense, CommonsenseQA, CommonsenseQA 2.0, and QASC, spanning numerical, general, binary, and scientific commonsense formats.NumerSense uses masked number-word recovery; CommonsenseQA is 5-way multiple choice; CommonsenseQA 2.0 is binary true-or-false classification; QASC is 8-way science QA.
- Datasets and Task Setup: NumerSense uses zero-shot T5 and GPT-3 inference, selecting answers by normalized likelihood over candidate choices.T5 uses a text-infilling setup, while GPT-3 computes the generative probability of each completed sentence.
- Datasets and Task Setup: CommonsenseQA uses zero-shot and finetuned T5 models, while CommonsenseQA 2.0 uses only finetuned Unicorn because zero-shot calibration is poor.UnifiedQA is included among the finetuned T5 systems for CommonsenseQA and QASC; Unicorn is the finetuned model used for CommonsenseQA 2.0.
- Datasets and Task Setup: QASC provides two background-knowledge pieces per question and is evaluated with zero-shot and finetuned T5 models using the CommonsenseQA setup.The two knowledge pieces jointly answer each question.
- Knowledge Generation Baselines: The study compares generated knowledge with no knowledge, random sentences, question-context continuations, template-generated knowledge, and retrieval-based knowledge.The no-knowledge condition is the vanilla baseline, while random and context conditions use the same generation implementation and hyperparameters.
4 Experimental Results
Generated Knowledge Prompting improves commonsense reasoning across zero-shot and finetuned models, achieving state-of-the-art results on multiple benchmarks. Performance depends on knowledge quality, quantity, integration strategy, and model scale, while human evaluation finds most selected knowledge helpful and factual.
- Overall Performance: Generated Knowledge Prompting sets new state-of-the-art results on NumerSense, CommonsenseQA 2.0, and QASC.It also works under both zero-shot and finetuned settings and performs comparably to retrieval-based systems.
- Overall Performance: 7% to 10% gains improve zero-shot inference across NumerSense, CSQA, and QASC.The reported changes are 64.05 →72.47, 39.89 →47.26, and 44.89 →55.00, respectively.
- Knowledge Generation Methods: Generated knowledge consistently improves performance, whereas random sentences barely help or hurt and context sentences provide only some gain.This comparison supports the reported importance of knowledge quality.
- Knowledge Generation Methods: Knowledge quantity improves performance until saturation at M = 20, after which additional statements cause decline.The decline may reflect the generation of more noisy knowledge.
- Knowledge Generation Methods: Adaptive knowledge integration performs best among the evaluated integration methods.The alternatives are Mixture-of-Experts and Product-of-Experts.
- Analysis: Smaller inference models receive larger gains, while relatively large knowledge-generation models are needed for useful and reliable knowledge.On T5-11b inference, the 6.7B knowledge model gives a 5.0% improvement versus 10.5% for the 175B model; 1.3B and 0.4B models show no significant improvement.
- Human Evaluation: Human evaluation finds 83% of selected knowledge factually correct, 72% helpful, and 13% harmful.Among statements rectifying model predictions, 93% were judged helpful; among misleading statements, 39% were judged harmful.
- Human Evaluation: Generated knowledge prompting can turn commonsense question answering into explicit procedures such as deduction, analogy, and numerical reasoning.Examples also include paraphrasing, induction, abduction, logical elimination, and negation.
5 Related Work
Prior commonsense-reasoning methods use pretrained models, external knowledge bases, graph reasoning, or generated text, but knowledge-base coverage and integration flexibility remain limitations.
- Pretrained language models implicitly contain knowledge that can be queried through conditional generation and can perform commonsense inference directly.
- External knowledge can be injected through knowledge-base pretraining, retrieval-based finetuning, or graph-based reasoning.
- Many knowledge-based methods require high-quality, high-coverage, in-domain commonsense knowledge bases.
- Knowledge-base methods face bottlenecks when datasets lack suitable resources or queries fall outside represented domains.
- Recent methods improve commonsense reasoning by augmenting questions with generated clarifications, explanations, or implications.
6 Conclusion
The paper introduces generated knowledge prompting, which elicits knowledge from language models and integrates it during inference without finetuning for knowledge integration. It improves commonsense reasoning across multiple datasets and highlights language models as flexible knowledge sources.
- Generated knowledge prompting elicits and integrates language-model knowledge to improve commonsense reasoning performance.
- Knowledge statements are generated from task-specific, human-written, few-shot question-knowledge demonstrations.
- The method plugs generated knowledge into inference without requiring finetuning for knowledge integration.
- The approach is effective across multiple datasets, establishes new state-of-the-art results on three commonsense reasoning tasks, and works under varied settings.
- Its success highlights language models as flexible, high-quality knowledge sources for commonsense reasoning.
A.1 Comparison with Prior Methods
Compared with prior methods, generated knowledge prompting uses few-shot demonstrations for knowledge generation and applies to finetuned inference models without joint finetuning with knowledge.
- Generated knowledge prompting uniquely uses few-shot demonstrations to prompt knowledge generation.
- The method applies to finetuned inference models without joint finetuning with generated knowledge.
- The evaluated tasks include NumerSense, CSQA, CSQA2, and QASC, for which the appendix provides full knowledge-generation prompts.
A.3 Human Evaluation Guidelines
The appendix documents the guidelines used to evaluate generated knowledge with human judgments.
- Human evaluation of generated knowledge uses detailed guidelines.
- The guidelines are presented in Tables 11 and 12.
- The documented guidelines support evaluation of the generated knowledge statements.
B.1 Limitations and Risks
The method’s scope is limited to a representative selection of commonsense reasoning tasks and datasets, and transferring it elsewhere may require task-specific prompt expertise. Poorly designed prompts may also reduce system performance.
- The evaluation covers a representative selection of commonsense reasoning tasks and datasets, so broader task coverage remains untested.
- Applying the method to other tasks may require moderate expertise to craft a task-specific prompt.
- Improper implementation or badly designed prompts may lower commonsense reasoning performance.
B.2 Computation
The paper does not train a new model; inference uses Quadro RTX 8000 GPUs, while knowledge generation uses the OpenAI GPT-3 API. The prompts combine questions with generated knowledge and include task-specific demonstrations or examples.
- The paper trains no new model and conducts inference on Quadro RTX 8000 GPUs.Inference costs about 200 GPU hours in total.
- Knowledge generation uses the OpenAI GPT-3 API at an approximate cost of $500.
- The implementation uses PyTorch and the Huggingface Transformers library.
- The method compares knowledge generators and inference models by whether they are task-finetuned, template-prompted, demonstration-prompted, or zero-shot.
- NumerSense prompts generate numerical facts and demonstrate knowledge statements for questions about wings, sides, feet, states, and limbs.
- The inference prompt places the question before a Knowledge field for generated knowledge.
- Prompt designs use task-specific instructions and examples, including manually written or dataset-selected demonstrations and task-relevant knowledge.NumerSense demonstrations are manually written; CSQA and CSQA2 demonstrations are selected from training data, while QASC uses gold separate facts.
- Human evaluation guidelines are presented across two continued tables.