Source-linked AI summary
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, Yejin Choi
TL;DR
Commonsense QA needs relevant knowledge, but existing retrieval is incomplete and language-model generation can be inconsistent or require very large models. RAINIER learns a question-focused knowledge introspector with reinforcement learning, optimizing generated knowledge through its effect on fixed QA models without gold knowledge labels. It improves QA across 9 benchmarks, including unseen datasets, and outperforms knowledge elicited from a few-shot GPT-3 despite being 16x smaller.
Problem
Commonsense QA lacks a systematic source of high-quality, question-relevant knowledge because retrieval coverage is incomplete and generated knowledge can be inconsistent or require large models.
Method
RAINIER generates question-specific knowledge and uses reinforcement learning to optimize its effect on fixed QA models, without gold knowledge labels.
Results
RAINIER improves QA models across 9 commonsense benchmarks, including 5 seen and 4 unseen datasets, with larger and more consistent gains than few-shot GPT-3 despite being 16x smaller.
Takeaways & Limitations
Moderately sized models can generate fluent, high-quality, diverse commonsense knowledge that usefully complements QA models and can outperform knowledge elicited from GPT-3.
Takeaways & Limitations
RAINIER's performance on non-commonsense applications is unknown, and its generated knowledge can express inappropriate social values, culture-specific content, or ethical risks.
Abstract
from arXiv · showhide
Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent. We present Rainier, or Reinforced Knowledge Introspector, that learns to generate contextually relevant knowledge in response to given questions. Our approach starts by imitating knowledge generated by GPT-3, then learns to generate its own knowledge via reinforcement learning where rewards are shaped based on the increased performance on the resulting question answering. Rainier demonstrates substantial and consistent performance gains when tested over 9 different commonsense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.
1 Introduction
RAINIER addresses the challenge of finding high-quality commonsense knowledge by generating question-relevant statements and optimizing them for downstream QA performance. Reinforcement learning trains the introspector without gold knowledge labels, yielding broad benchmark gains and smaller-model advantages over GPT-3.
- 1 Introduction: Existing approaches either retrieve incomplete in-domain knowledge or generate knowledge with domain-specific engineering and very large models, leaving systematic high-quality knowledge discovery open.The cited generation approaches include GPT-3 as a large model source.
- 1 Introduction: The method uses reinforcement learning to optimize generated knowledge for resulting QA performance rather than relying on unavailable gold knowledge labels.PPO is adapted because the generated knowledge consists of discrete, nondifferentiable word tokens.
- 1 Introduction: RAINIER is trained multitask on 8 commonsense QA datasets spanning general, scientific, physical, and social commonsense to support generalization to unseen benchmarks.
- 1 Introduction: RAINIER improves QA performance across 9 commonsense benchmarks, including 5 seen and 4 unseen datasets, with larger and more consistent gains than few-shot GPT-3 despite being 16x smaller.It also improves QA models it was not trained against and can boost a QA model 4x larger than itself.
- 1 Introduction: RAINIER generates fluent, useful knowledge prompts whose effects on QA align with human judgments and whose content is diverse across domains, relations, and syntax.Reported domains include scientific and social commonsense; relations include part of, member of, and purpose.
2 Method
RAINIER generates question-specific commonsense knowledge in two stages: GPT-3 imitation initializes a knowledge generator, then reinforcement learning optimizes knowledge for a fixed QA model’s predictions. Its PPO-based training uses QA-derived rewards while constraining the policy to remain near the imitation model.
- Method overview: RAINIER generates question-specific knowledge for a fixed QA model, complementing supervised QA training when gold knowledge labels are unavailable.The model is a sequence-to-sequence generator trained to produce knowledge statements conditioned on each question.
- Two-stage training: Training proceeds through GPT-3 imitation followed by reinforcement learning that makes generated knowledge more useful while preserving fluency and meaning.GPT-3 supplies silver question-knowledge pairs for Stage I; Stage II directly optimizes the knowledge’s effect on QA predictions.
- Reinforcement-learning formulation: Knowledge generation is modeled as sequential decision making, with tokens as actions and the generated statement’s effect on QA prediction defining the reward.The state contains the question and previously decoded tokens, while the reward is assigned from the completed knowledge statement.
- PPO optimization: RAINIER adds a KL-based penalty against the imitation policy and uses PPO with policy and value losses to optimize discrete knowledge sequences.The policy ratio compares the current model with a lagging policy, and the value model estimates values for partially decoded states.
- Reward shaping: The reward targets changes in the QA model’s prediction, rather than only absolute score changes, rewarding corrections from incorrect to correct and penalizing the reverse.The reward uses the correct answer’s score and the competing answers’ scores to provide a smoothed polarity based on prediction correctness and margin.
3 Experiments
The experiments evaluate RAINIER across seen and unseen commonsense QA datasets, using UnifiedQA-large with several knowledge-generation baselines. The setup also notes an input-length constraint that excludes two reading-comprehension datasets.
- Datasets: RAINIER is evaluated on 8 seen multiple-choice datasets and 4 unseen datasets not used for training.The seen datasets are used for imitation and reinforcement learning, whereas the unseen datasets are reserved for evaluation.
- Baselines: All experiments use UnifiedQA-large as the QA model, with optional knowledge supplied by alternative generation methods.The baselines include vanilla UnifiedQA, few-shot GPT-3, Self-talk, and DREAM.
- Experimental scope: MCTest and RACE are excluded because most questions are too long to fit into the model’s input.This imposes an input-length boundary on the evaluated reading-comprehension datasets.
- Seen-dataset evaluation: Table 1 reports results on seen datasets, comparing UnifiedQA-large with optional knowledge from the evaluated generation models.Skipped baselines are marked with “–”.
- Unseen-dataset evaluation: Table 2 reports results on the four unseen datasets, testing generalization beyond the datasets used for training.The unseen datasets are NumerSense, RiddleSense, QuaRTz, and HellaSwag.
4 Results
RAINIER improves commonsense QA across seen and unseen benchmarks, while its generated knowledge remains useful across QA models and benefits from both imitation and reinforcement-learning stages.
- Main Results: +12% on QASC and +6% on CommonsenseQA; RAINIER improves five seen benchmarks but not OpenBookQA, ARC, or AI2Science.The authors conjecture that the latter datasets provide a strong baseline because UnifiedQA was already trained on them.
- Main Results: RAINIER outperforms GPT-3-based knowledge generators despite using 0.77B rather than 13B parameters.This corresponds to a 16x smaller model.
- Main Results: RAINIER substantially improves QA performance on all four unseen datasets, demonstrating generalization beyond the training benchmarks.
- Main Results: RAINIER improves every evaluated QA model, including one four times larger, suggesting its knowledge is generally useful rather than reward-hacking artifacts.The largest gains occur for QA models with weak but non-trivial performance.
- Ablations: Removing imitation training eliminates improvement, while imitation alone is weaker than the full pipeline, establishing the importance of both training stages.
- Ablations: The proposed reward shaping performs best on unseen datasets and generalizes better than a reward with slightly stronger seen-dataset performance.
- Analysis: 64% of generated knowledge is factually correct and 58% is judged helpful, while 25% is incorrect and 24% harmful.Among prediction-rectifying knowledge, 84% is human-judged helpful; among misleading knowledge, 62% is harmful.
- Analysis: Generated knowledge spans benchmark-relevant domains, multiple relations, and varied syntax, with domain distributions reflecting each benchmark.The domain aspect is especially diverse for general-commonsense benchmarks.
5 Related Work
Prior commonsense QA work generates explanations or clarifications through supervision, templates, or prompting, while RAINIER applies reinforcement learning to knowledge introspection using QA-model feedback.
- Explicit reasoning for commonsense QA: Earlier methods generate explanations or clarifications through supervised learning, predefined templates, or in-context prompting, often imposing constraints on generated text.
- Reinforcement learning for NLP: Related reinforcement-learning applications span question answering, summarization, translation, grounded and controlled generation, and prompt generation.
- Explicit reasoning for commonsense QA: Table 5 provides examples of RAINIER knowledge that rectify UnifiedQA-large predictions and are annotated as relevant, factual, and helpful.
- Reinforcement learning for NLP: RAINIER differs by using PPO to optimize a reward derived purely from neural QA-model decisions for knowledge introspection.
6 Conclusion
RAINIER introspects relevant knowledge for commonsense QA through reinforcement learning, generating fluent and diverse prompts that improve QA on seen and unseen benchmarks. Its knowledge can outperform few-shot GPT-3-derived knowledge despite being 16x smaller.
- 6 Conclusion: RAINIER improves QA performance on both seen and unseen commonsense benchmarks using generated knowledge prompts.Its generated knowledge is fluent, meaningful, high-quality, and diverse across domains and relations.
- 6 Conclusion: RAINIER uses a novel reinforcement-learning adaptation and does not require gold knowledge labels that are difficult to obtain.
- 6 Conclusion: RAINIER-generated knowledge outperforms knowledge elicited from few-shot GPT-3, which is 16x larger.The effect of the generated knowledge on QA aligns well with human judgments.
Limitations
RAINIER’s demonstrated scope is limited to commonsense QA, and its generated knowledge raises unresolved safety and deployment concerns. The authors therefore caution against treating the resulting system as ready for real-world or critical applications.
- Limitations: RAINIER’s performance on non-commonsense applications is unknown and requires further investigation.
- Limitations: The resulting model is not ready for real-world applications because a substantial gap remains between commonsense-task performance and human performance.
- Limitations: Generated knowledge is limited in length, has not been tested for long coherent text, and may express inappropriate, culture-specific, or ethically risky content.The authors recommend extra care in production environments, especially for critical decisions or direct exposure to end users.
A.2 Baselines
The baselines generate multiple candidate knowledge statements per question using GPT-3-Curie self-talk or DREAM scene elaborations. These baselines vary the prompting strategy and source model rather than directly supervising gold knowledge.
- A.2 Baselines: Self-talk generates M = 10 knowledge candidates per question with GPT-3-Curie using ten question-answer template pairs.Each candidate is produced through sequential question and answer generation with nucleus sampling.
- A.2 Baselines: The baseline comparison therefore includes both GPT-3-Curie self-talk and DREAM-based scene elaboration as alternative knowledge-generation sources.
- A.2 Baselines: DREAM generates M = 10 scene elaborations per question across motivation, emotion, rule-of-thumb, and consequence types.Two or three elaborations are generated for each controlled scene-elaboration type.
B Additional Analysis
Additional analyses document experimental settings, baseline behavior, prompt templates, and examples of both useful and problematic generated knowledge. One reported result qualifies the broader gains: on three other seen datasets, RAINIER did not improve over vanilla QA.
- B Additional Analysis: The analysis identifies semantically problematic, socially value-laden, culture-specific, and potentially ethically risky generated knowledge.These categories are represented in Tables 9–12 and accompanying examples.
- B Additional Analysis: The appendix provides hyperparameter settings, self-talk templates, dataset-specific prompts, and examples spanning OpenBookQA, ARC, AI2Science, CommonsenseQA, QASC, and PhysicalIQA.
- B Additional Analysis: RAINIER-generated knowledge does not improve over the vanilla QA baseline on three other seen datasets using UnifiedQA-large.
- B Additional Analysis: Examples illustrate generated knowledge connecting questions to answers across science, commonsense, physical reasoning, and social situations.Examples include links such as sunlight to water’s state, friction to heat, and messy rooms to dirty clothes.