Crystal: Introspective Reasoners Reinforced with Self-Feedback

1 Introduction

• Commonsense reasoning depends on underlying knowledge that is often obscure, and CoT’s effectiveness is marginal because its prompts do not account for this challenge.Knowledge-augmented reasoning requires both introspecting relevant, high-quality knowledge and using it faithfully to make informed predictions.Source: §1 · p. 1
• Crystal addresses the mutual adaptation between knowledge introspection and knowledge-grounded reasoning that existing methods do not comprehensively optimize.The paper presents this as a route toward a strong, interpretable, and generalizable commonsense reasoner.Source: §1 · p. 2
• Crystal generates question-relevant knowledge statements before using them with the question to predict an answer.Its two modes are trained with reinforcement learning to improve their synergy, using feedback based on whether generated knowledge can fix downstream prediction errors.Source: §1 · p. 2
• Crystal improves absolute accuracy by 1.5% to 2.5% over direct QA models trained on the same data, generalizes to unseen benchmarks, and substantially outperforms CoT-distilled models.The paper also reports improved interpretability across experiments on 25 commonsense QA benchmarks, including 10 seen and 15 unseen benchmarks.Source: §1 · p. 2

2.1 Introspective Reasoning

• Introspective reasoning makes commonsense knowledge explicit before prediction, exposing reasoning paths that users can observe and allowing multiple paths to be ensembled.CRYSTAL generates knowledge such as “Cows produce milk,” then uses it to answer the question.Source: §2.1 · pp. 2–3

2.2 CRYSTAL

• CRYSTAL is one model with knowledge-introspection and knowledge-grounded-reasoning modes; it aggregates predictions from available reasoning paths into a final answer.The first mode takes a question and produces relevant knowledge; the second takes the question and that knowledge to predict an answer.Source: §2.2 · p. 3

2.3 Training Objectives

• Training adapts both modes: PPO rewards knowledge by downstream reasoning feedback, while the model omits a direct QA objective that could invite shortcuts.The paper reports that including direct QA loss hurts performance, probably by allowing shortcuts around the knowledge.Source: §2.3 · p. 4

2.4 PPO and Model Sharing

• CRYSTAL shares parameters across PPO’s policy, reward, and value roles, using an LM head for policy and reward modeling and a value-regression head for value modeling.The authors report that this sharing improves PPO training’s memory and time efficiency.Source: §2.4 · p. 4

2.5 Two-Staged Training

• Stage I trains knowledge generation by imitating few-shot GPT-3 statements alongside knowledge-grounded reasoning; stage II adapts the two modes to each other.The stage-I reasoning loss uses silver knowledge statements generated by few-shot GPT-3.Source: §2.5 · p. 4

2.6 Interleaved Optimization Schedule

• Rather than jointly optimizing losses, each stage alternates knowledge-generation and reasoning updates; the authors liken this schedule to EM, with knowledge as the hidden variable.Stage I alternates LQK and LQKA; stage II alternates LPPO and LQKA.Source: §2.6 · pp. 4–5, Figure 2 · p. 4

3 Experimental Setup

• CRYSTAL is trained on 10 datasets and evaluated on their development sets for seen evaluation and 15 additional datasets for unseen evaluation.Training-set silver knowledge comes from few-shot GPT-3.Source: §3 · p. 5
• Models are initialized from T5-large, T5-3b, or T5-11b, with the value regression head initialized from scratch at stage II.Source: §3 · p. 5
• The primary baseline is Direct QA, trained on the same datasets with the standard QA objective and corresponding T5 sizes.Source: §3 · p. 5
• The comparisons also include Rainier and CoT-distilled models, whose reported results are shown only on datasets they were trained on.The cited CoT-distilled approaches include fine-tune-CoT, SCoTD, and SCOTT.Source: §3 · p. 5
• Retrieval-based methods are excluded because CRYSTAL does not retrieve from extra sources; the study reports non-retrieval state-of-the-art performance for each seen dataset.Source: §3 · p. 5

4.1 Performance

• On seen datasets, CRYSTAL beats direct QA by 1.5%∼2.5% across model sizes, and on unseen benchmarks gains 2.0%∼4.3% average accuracy over direct QA.Source: §4.1 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6
• CRYSTAL is competitive with similar-sized CoT-distilled models, with 27%/23% CommonsenseQA gains over Fine-tune-CoT for CRYSTAL-large/3b and 5%/9% gains over SCOTT for CRYSTAL-3b on CommonsenseQA/QASC.CRYSTAL-large is comparable to SCoTD on OpenBookQA and CommonsenseQA.Source: §4.1 · p. 6
• CRYSTAL closely matches non-retrieval SOTA, sets new SOTA on CommonsenseQA and QASC, and has less than 3% gap on OpenBookQA, PIQA, SIQA, and Winogrande.The cited comparisons describe CRYSTAL as a single model performing strongly across these benchmarks.Source: §4.1 · p. 6

4.2 Interpretability

• Expert annotations found non-trivial knowledge support in 34% of cases, trivial entailment in 19%, related knowledge in 31%, repetition in 5%, and unrelated or contradictory knowledge in 11%.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3 Qualitative Examples

• Examples show introspected facts can support correct answers, as with cows producing milk, but grounded reasoning sometimes ignores correct knowledge, as in the anemometer example.Source: §4.3 · pp. 7–8

4.4 Memory and Time Efficiency

• Compared with Rainier's conventional PPO, CRYSTAL shares policy, value, and reward parameters and stores two models rather than four, with less GPU memory use and faster training.Source: §4.4 · p. 8

4.5 Ablations

• Omitting RL stage II lowers seen-dataset performance by approximately 0.5% to 0.6% across model sizes.Source: §4.5 · p. 8, Table 7 · p. 9
• Adding a direct QA loss to stage I hurts performance by 0.8%, so CRYSTAL excludes that loss from its training objective.Source: §4.5 · p. 9

5 Related Work

• CRYSTAL’s distinctive contribution is jointly accounting for mutual adaptation between knowledge generation and knowledge-grounded reasoning in one model.The paper contrasts this with knowledge-augmented methods that use task-specific engineering and may be difficult to generalize to unseen domains.Source: §5 · p. 9
• CRYSTAL shares a distillation stage with chain-of-thought distillation, but adds introspective knowledge and reinforcement learning to improve synergy between reasoning paths and predictions.The section also describes prior knowledge-augmented methods as grounding reasoning in model-generated knowledge.Source: §5 · p. 9
• CRYSTAL is presented as the first method to use reinforcement learning from self-feedback, unlike prior self-feedback approaches that use supervised loss.The authors contrast it with an inference-time feedback method that relies on emergent behavior in large language models.Source: §5 · p. 9

6 Conclusion

• The method is reported to achieve superior performance and good interpretability on commonsense reasoning tasks.The authors emphasize accounting for both introspective knowledge and mutual adaptation between the two reasoning modes.Source: §6 · p. 9
• CRYSTAL comprehensively addresses the introspective nature of commonsense knowledge and mutual adaptation between knowledge introspection and grounded reasoning.Source: §6 · p. 9
• The authors conclude that training neural models with self-feedback is feasible and beneficial.Source: §6 · p. 9

Limitations

• CRYSTAL is intended for commonsense QA; its performance on non-commonsense applications remains unknown.The authors say further investigation is needed.Source: Limitations · p. 10
• The experimental setting limits the length of generated knowledge, and long, coherent text generation has not been tested.Source: Limitations · p. 10
• Production use calls for extra care, especially for critical decisions or when generated content is exposed directly to end users.Source: Limitations · p. 10

A More on Method

• CRYSTAL’s input format adds a mode marker: “Knowledge:” for introspection and “Answer:” for knowledge-grounded reasoning.Source: A More on Method · p. 14
• The output is the letter for the predicted answer choice, rather than the answer choice’s content.Source: A More on Method · p. 14
• The input and output text are not lowercased.Source: A More on Method · p. 14

• Table 9 lists the training and evaluation datasets, while Table 10 reports the hyperparameters.Source: B More on Experimental Setup · p. 14

• Table 11 reports the detailed instructions for human evaluation.Source: C More on Results · p. 14

• Table 12 compares Crystal with existing knowledge-augmented commonsense reasoning methods and specifies criteria for unified models and cross-adaptation.The criteria distinguish whether a method uses a unified model, trains knowledge-grounded reasoning to adapt to knowledge generation, or trains knowledge generation to adapt to reasoning.Source: D More on Related Work · p. 14, Table 12 · p. 16
• Table 9 gives dataset details and links to the sources from which the datasets were retrieved.Source: Table 9 · p. 14
• Table 10 reports the hyperparameter settings.Source: Table 10 · p. 15
• Table 11 provides the instructions for human evaluation.Source: Table 11 · p. 16