Source-linked AI summary
REFINER: Reasoning Feedback on Intermediate Representations
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, Boi Faltings
TL;DR
REFINER addresses the challenge of refining erroneous intermediate reasoning in language-model reasoning tasks. It trains a generator to interact iteratively with a critic that supplies fine-grained structured feedback. Across three tasks, REFINER improves over comparable baselines, and the trained critic also improves GPT-3.5 without fine-tuning the reasoner.
Problem
Intermediate reasoning representations can contain inappropriate deductions, while meaningful refinement of those representations remains unclear and conventional correction can require costly additional data and fine-tuning.
Method
REFINER trains separate generator and critic models so the generator produces intermediate reasoning steps and iteratively incorporates the critic’s structured error feedback.
Results
+13.1, +3.2, and +15 pts. are reported gains over comparably sized strong fine-tuned LM baselines across three reasoning tasks, while GPT-3.5 gains +3.5 and +6.8 pts. from trained-critic interaction.
Takeaways & Limitations
Structured fine-grained feedback can improve intermediate representations and reasoning performance, including when a trained critic assists GPT-3.5 without modifying the reasoner.
Takeaways & Limitations
The framework was evaluated on only three reasoning tasks, and its critic and fine-grained error types are task-specific rather than general across tasks.
Abstract
from arXiv · showhide
Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences, e.g., chain-of-thought prompting. However, these intermediate inference steps may be inappropriate deductions from the initial context and lead to incorrect final predictions. Here we introduce REFINER, a framework for finetuning LMs to explicitly generate intermediate reasoning steps while interacting with a critic model that provides automated feedback on the reasoning. Specifically, the critic provides structured feedback that the reasoning LM uses to iteratively improve its intermediate arguments. Empirical evaluations of REFINER on three diverse reasoning tasks show significant improvements over baseline LMs of comparable scale. Furthermore, when using GPT-3.5 or ChatGPT as the reasoner, the trained critic significantly improves reasoning without finetuning the reasoner. Finally, our critic model is trained without expensive human-in-the-loop data but can be substituted with humans at inference time.
1 Introduction
REFINER addresses the difficulty of meaningfully refining erroneous intermediate reasoning by having a generator iteratively use fine-grained critic feedback. Across three reasoning tasks, it reports gains over comparable baselines and shows benefits from structured feedback and trained critics.
- Motivation: The motivation is that intermediate representations can improve reasoning performance and interpretability yet may contain inappropriate deductions that undermine final predictions.Prior correction through additional annotation and fine-tuning is data- and resource-intensive and may generalize insufficiently in complex reasoning tasks.
- Framework: REFINER uses a generator and critic to iteratively refine intermediate reasoning steps through structured feedback about errors.The critic is trained on automatically constructed incorrect representations and corresponding fine-grained feedback, then interacts with the generator during training and inference.
- Results: +13.1, +3.2, and +15 pts. are reported gains over comparably sized strong fine-tuned LM baselines across three reasoning tasks.The evaluated tasks are math word problems, synthetic natural language reasoning, and moral action generation.
- Results: Fine-grained structured feedback can benefit more than scalar value feedback for moral action generation and math word problems.The paper also reports that REFINER outperforms refinement methods using feedback from large language models, such as self-refine.
2 Related Work
Prior work establishes the value of intermediate reasoning and richer feedback, while REFINER differs by training smaller generators and integrating a critic into iterative reasoning feedback.
- Intermediate Representations: Intermediate representations and rationales have supported mathematical skills, program execution computations, and general reasoning outputs.These representations expose aspects of model reasoning and can support task performance.
- Intermediate Representations: REFINER fine-tunes smaller models to generate intermediate steps rather than only prompting a large model.It also accommodates tasks without unique closed-form correct answers, including the Moral Norm task.
- Natural Language Feedback: Natural-language feedback has been explored for alignment and task improvement, but automated critics for reasoning tasks have proved difficult.Prior approaches include scalar rewards, richer feedback, and direct correction of answers or reasoning.
- Natural Language Feedback: REFINER integrates a secondary critic into an interaction loop during generator training and provides feedback at intermediate reasoning steps.This distinguishes it from correctors that improve an initial proposition by learning generator mistakes and fixes.
3 REFINER
REFINER decomposes reasoning into intermediate hypotheses and final outputs, then trains a critic and generator to interact through structured feedback that corrects reasoning errors.
- Problem Formulation: REFINER models natural-language reasoning as generating an intermediate hypothesis z from context x and then output y conditioned on x and z.The formulation is p(y|x) = p(y|x,z)p(z|x).
- Tasks: The study covers math word problems, synthetic natural-language reasoning, and moral action generation, spanning logical and normative reasoning.The tasks use different intermediate representations, including equations and inference rules.
- REFINER: The framework uses separate CRITIC and GENERATOR models, with the critic providing structured feedback and the generator improving its intermediate reasoning.The design supports feedback during training and test-time interaction with either a trained critic or a human.
- CRITIC Model: The critic is trained on plausible and automatically perturbed implausible hypotheses paired with structured error feedback.Feedback has the form ⟨error type, position, hint⟩, with optional position and hint fields.
- GENERATOR Model: During exploration, the generator samples multiple hypotheses and the critic randomly selects one to provide feedback, increasing the range of training feedback.A warm-up phase first fine-tunes the generator on a small supervised dataset before feedback-based learning.
- GENERATOR Model: At inference, the trained generator and critic produce a trajectory of refined hypotheses until the generator or critic signals completion.The described stopping signals are generated feedback or “No hint.”
4 Experimental Setup
REFINER is evaluated on three diverse reasoning tasks using supervised and few-shot language-model baselines, task-specific metrics, and controlled training procedures.
- Datasets: The experiments evaluate math word problems on MAWPs for training and SVAMP and GSM8K for evaluation, measuring equation generation and answer prediction.The MWP setup replaces numeric values with variable names for equation generation.
- Results: Table 2 compares REFINER with baselines on SVAMP and reports average scores over three runs with p<0.05.For models other than GPT-3.5, final answers are obtained by symbolic execution of generated equations.
- Training Details: Training uses UQA-base critics, nucleus sampling with p = 0.5 for exploration, Adam at 1e−4, 20 epochs with early stopping, and three random seeds.All models were trained on one A100 GPU; inference uses greedy decoding.
- Evaluation: The three tasks use exact match for intermediate steps and accuracy for final answers on MWP and sNLR, while moral action generation uses manual evaluation.Critic feedback data is generated with the rule-based perturbation strategy.
- Baselines: The baselines include UQA-base, UQA-large, GPT-3.5, ChatGPT, PPO, and chain-of-thought prompting, with comparisons adjusted to task and model scale.The sNLR comparison uses UQA-base, while moral action generation focuses on UQA-large because human annotation is expensive.
5 Results
REFINER improves intermediate reasoning and final predictions across math word problems, synthetic natural-language reasoning, and moral action generation. Its trained critic also benefits external refinement settings and remains useful during inference.
- Intermediate Steps: REFINER significantly outperforms comparable baseline models on intermediate-step generation across evaluated reasoning tasks.For MWP, the model explores equation-only and joint equation-answer generation; UQA-base gains +13.1 EM when adding a critic.
- Final Answer Prediction: On moral stories, human evaluation shows a 20-point relevance improvement over UQA-large for generated norms and moral actions.Three human judges evaluated 100 randomly sampled moral-story examples, with results summarized using relevance categories and agreement scores.
- Final Answer Prediction: REFINER improves final answer prediction by +3.5, +3.2, and +15 points on MWP, sNLR, and MS, respectively.On sNLR, GPT-3.5 gains +6.8 EM for inference-step generation but only +1.5 for consequent prediction.
- Comparison with Refinement Methods: REFINER improves GPT-3.5 and ChatGPT over self-refine by +3.3 on SVAMP and +2.2 on GSM8K.The trained critic also improves Self-Consistency by +2.02 and ReACT by +2.9.
- Ablation: Removing the critic during inference lowers performance from 47.2 to 39.8, while removing exploration costs +3.3 relative to the baseline.The ablation supports using critic feedback at inference and an exploration phase during generator training.
- Ablation: A perfect critic can fix generator mistakes, while a noisy critic remains beneficial and supports AI-AI or human-AI interaction.The framework is presented as a tool for refining reasoning through feedback rather than requiring only autonomous generation.
6 Analysis
The analysis examines error patterns, critic quality, and robustness to noisy feedback. It finds that critic quality at inference is crucial, while moderate training noise can remain tolerable.
- Error Analysis: Incorrect Numbers in MWP and Semantic Misalignment in MS are the most frequent errors identified by fine-grained analysis.The authors relate these patterns to sensitivity to number order in MWP and difficulty grounding norms in MS contexts.
- Noise Sensitivity: Training with more than 75% critic noise makes the generator ignore the critic, eliminating differences between trained-critic and oracle-critic inference.
- Noise Sensitivity: Training with less than 50% noise does not significantly harm performance relative to training with an oracle critic.
- Noise Sensitivity: Critic quality during inference has a huge impact on REFINER’s performance.Figure 4 compares training-noise effects under oracle and automatic critics and separately varies inference-time critic noise.
- Qualitative Analysis: REFINER failures occur when the generator ignores correct feedback or when the critic supplies incomplete or partially correct feedback.The critic can also be ambiguous, and errors involving multiple mistakes are especially difficult for it to identify correctly.
- Quality of the feedback: The trained UQA critic comprehensively outperforms GPT-3.5 on exact-match feedback quality across evaluated task instances.The comparison uses 500 instances per task and includes cases where the critic should indicate that a solution is correct.
7 Conclusion
REFINER improves language-model reasoning through an iterative feedback loop between a generator and a critic. Across three reasoning tasks, structured fine-grained feedback yields significant gains, including when the critic is noisy.
- REFINER uses an iterative feedback loop between a generator and a critic to improve language-model reasoning.
- Across three reasoning tasks, structured and fine-grained feedback on intermediate reasoning errors produces significant performance gains and surpasses scalar value feedback.
- A trained critic can improve language-model intermediate representations even when its feedback is noisy.
Limitations
The evaluation covers only three reasoning tasks, while the critic and fine-grained error types are task-specific. Generalizing them remains a main limitation and future direction.
- REFINER was evaluated on only three reasoning tasks because the number of applicable downstream tasks made comprehensive evaluation infeasible.
- The critic and fine-grained reasoning errors are specific to each task rather than general across reasoning tasks.
- Generalizing the critic and fine-grained error types is identified as a main limitation and direction for future work.
Ethical Considerations
The ethical discussion notes that REFINER does not mitigate biases inherited from language models and that the moral-action dataset reflects Western cultural norms. Deployment therefore requires harm checks and mitigation.
- REFINER does not provide a way to mitigate language-model biases about race, gender, or other demographic attributes.
- Models improved with REFINER may still exhibit the harmful behaviors normally associated with the underlying language models.
- The authors recommend checking off-the-shelf deployments for harm toward protected groups and applying appropriate mitigation.
- The MS dataset heavily skews toward Western culture and social norms.Human evaluation for MS used adequately paid AMT workers based in the US.
- Automatic evaluation of semi-structured critic feedback uses sentence-level matching to oracle answers and is not order-sensitive.
- ReAct uses separate GPT-3.5 reason and action models and was treated as approximately 3–4 times more expensive than GPT-3.5 plus CoT.The experiments assumed three reasoning steps for ReAct and five samples for self-consistency to control cost.
A.3 More results on SVAMP dataset
This appendix compares REFINER with established baselines on SVAMP and outlines the algorithms, model setup, and supporting dataset statistics used for evaluation.
- SVAMP comparison: REFINER is compared with Graph2Tree, GTS, and DeductReasoner on the SVAMP answer-prediction task.The cited baselines use quantity relations, sequence-to-tree decoding, or bottom-up DAG-structured decoding.
- Reported results: Table 9 reports the SVAMP comparison, while Table 10 provides dataset statistics.
- Algorithms and models: The appendix includes separate training and inference algorithms for REFINER.
- Algorithms and models: REFINER training uses a supervised critic that takes context and plausible or implausible hypotheses as input and produces textual feedback.The generator is trained to produce plausible hypotheses from the context.
D Training Details
REFINER uses task-specific critics, automatically generated perturbation data, and controlled training and inference settings. The reported experiments include qualitative correction examples and English-language datasets.
- Training setup: Each task uses a UnifiedQA-T5-base critic trained with Adam at learning rate 1e−4 for up to 20 epochs with early stopping.All models were trained on one A100 GPU, using three random seeds and reporting average results.
- Inference and evaluation: Inference uses greedy decoding with T = 1 for the automatic critic and T = 3 for the oracle critic.MWP and sNLR use exact match for intermediate steps and accuracy for final answers; Moral Stories uses manual evaluation.
- Inference and evaluation: Qualitative examples show that structured feedback can correct operators in multistep equations, while dataset details and model information are reported separately.
- Feedback data: Critic training data pairs plausible and implausible intermediate hypotheses with structured feedback generated from reasoning-error perturbations.Perturbations include omitting, replacing, or adding tokens or rules.
- Feedback data: For MWP and sNLR, rule-based feedback generation targets errors such as logical invalidity, missing links, and missing implicit knowledge steps.
- Feedback data: For moral norm generation, the critic targets logical contradiction and semantic misalignment, including incoherent or uninformative hypotheses.Contradictions are created by inverse moral judgments, while semantic misalignment alters the action phrase.
G Human Evaluation on Moral Stories
The Moral Stories human evaluation asks annotators to judge generated norms and moral actions using Likert ratings and majority voting. The appendix documents the evaluation materials and reports complete result breakdowns.
- Evaluation procedure: Annotators rate the relevance of generated norms and moral actions on a five-point Likert scale from strongly disagree to strongly agree.
- Evaluation procedure: Ratings of 4 or 5 are treated as Relevant, ratings of 1 or 2 as Irrelevant, and rating 3 remains Unsure.
- Evaluation procedure: Three judges evaluate each example, and majority voting assigns Unsure when no clear majority winner exists.
- Evaluation caveat: Low and moderate Krippendorff’s α values indicate that judging the plausibility of moral norms and actions is challenging.
- Reported results: Figures 8 and 9 provide complete breakdowns for human evaluations of moral norms and moral actions.