Source-linked AI summary
Generating Sequences by Learning to Self-Correct
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, Yejin Choi
TL;DR
Sequence generators often violate semantic constraints and lack a way to revise imperfect outputs, while adapting large or inaccessible models can be impractical. Self-Correction separates an imperfect base generator from a learned corrector trained with scalar or natural-language feedback, and improves performance across three diverse tasks, including when the corrector is much smaller.
Problem
Language models frequently violate semantic constraints and lack a mechanism to iteratively revise imperfect generations.
Method
Self-Correction decouples an imperfect base generator from a separate corrector trained online to improve intermediate generations using scalar or natural-language feedback.
Results
Self-Correction improves the base generator on mathematical program synthesis, lexically constrained generation, and toxicity control, including with a much smaller corrector.
Takeaways & Limitations
Self-correction provides an efficient, transferable way to improve and control outputs from small and large generators without modifying the underlying generator.
Takeaways & Limitations
Training assumes access to a scalar value function for measuring output quality.
Abstract
from arXiv · showhide
Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs. Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation. We present Self-Correction, an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations. To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations. We show that Self-Correction improves upon the base generator in three diverse generation tasks - mathematical program synthesis, lexically-constrained generation, and toxicity control - even when the corrector is much smaller than the base generator.
1 INTRODUCTION
Self-Correction replaces single-pass generation with a base generator that proposes an initial hypothesis and a corrector that iteratively improves it. The approach improves generation across diverse tasks and can transfer correction to larger generators.
- Motivation: Single-pass generation discards useful partial work when an output is slightly sub-optimal, forcing models to start over.Examples include omitted keywords or an incorrect hypothesis that retains a useful problem-solving structure.
- Approach: Self-Correction separates generation into a base generator and a corrector that can improve outputs over multiple passes.Neither module must solve the full task in one pass.
- Findings: Self-Correction improves the base generator on mathematical program synthesis, lexically constrained generation, and toxicity reduction.The corrector can also be applied to a larger generator with similar performance to training a new corrector.
- Findings: The approach supports efficient and transferable correction networks that build on existing generation models and their outputs.The corrector may be much smaller than the base generator.
2 SELF-CORRECTING SEQUENCE GENERATORS
Self-correcting sequence generators decompose generation into an initial hypothesis and repeated correction, optionally guided by feedback. Self-corrective learning trains the corrector online from value-improving, similar generation pairs.
- Architecture: A base generator produces an initial hypothesis, and a corrector refines it repeatedly to form a Self-Corrector.Inference follows a trajectory from y0 through corrected outputs, with the final generation selected as the output.
- Architecture: Separating modules allows independent parameterization, including a corrector that is much smaller than the generator.The generator can remain a general-purpose language model while the corrector targets task-specific requirements.
- Feedback: Explicit feedback can condition correction, including sentences or compiler traces describing properties of the intermediate generation.The paper reports learning to exploit natural-language feedback for better performance.
- Assumption: Training assumes access to a scalar value function that measures output quality, such as a classifier.At inference, correction continues for a fixed number of steps or until a target value is reached when value is available.
- Training: Self-corrective learning builds a datapool of generated hypotheses with scalar values and optional feedback, then adds corrections generated by the current corrector.The initial datapool is created from multiple generator outputs, and exploration expands it during training.
- Training: The algorithm forms pairs where a correction has higher value than its hypothesis, then samples pairs using value improvement and hypothesis–correction similarity.The corrector is updated with cross-entropy loss, while α emphasizes value gains and β retains similar targets.
3 EXPERIMENTS
Experiments evaluate Self-Correction across mathematical program synthesis, lexically constrained generation, and toxicity reduction, including modularity, feedback, and iterative-correction analyses. Across these settings, correctors improve generators while preserving or improving relevant generation qualities.
- Mathematical program synthesis: Self-Correction improves mathematical program synthesis across three settings, including GSM, Multiarith, and Multitask arithmetic.On Multiarith, correction reaches 98–99%, a 38-point gain; on GSM, performance rises from 8.57% to 21%, or 24% when correcting only incorrect solutions.
- Lexically constrained generation: SELF-CORRECT improves lexical constraint coverage on COMMONGEN and E2E while maintaining or improving language quality.On COMMONGEN, it paired with NeuroLogic achieves the best results and is an order of magnitude faster than NeuroLogic-A*; on E2E, it outperforms NeuroLogic-A* with standard beam search.
- Toxicity reduction: SELF-CORRECT substantially reduces toxic generations while maintaining fluency and diversity, outperforming inference-time and reinforcement-learning baselines.The approach controls toxicity without modifying the underlying generator.
- Changing modules: Self-correctors improve generators much larger than the corrector through test-time swapping or training with the larger generator.For GPT-3 Instruct, Multitask performance increases from 84.90 to 90.90 with a corrector and to 92.75 after training with GPT-3 Instruct; GSM increases from 36.80 to 45.00 and then 45.92.
- Feedback: Explicit natural-language feedback improves performance in all three tasks by helping correctors focus on useful corrections.For toxicity, fine-grained attributes such as identity attack and profanity outperform using only a scalar toxicity score.
- Additional ablations and analysis: Multiple corrections improve toxicity reduction and math performance, with feedback making repeated corrections more beneficial in the Multitask setting.After two to three corrections in the math task, the model needs additional guidance; exploration also improves performance on all three math datasets.
4 RELATED WORK
Self-correction differs from prior editing, denoising, reinforcement-learning, and rationale-generation approaches by learning an expressive corrector online to improve generations without supervised edits or critiques.
- Self-correction trains an expressive text-to-text corrector online to improve a quality measure without supervised edits or critiques.
- Unlike reinforcement learning applied to a base generator, self-correction can improve models that are inaccessible or infeasible to update and can complement RL-tuned generators.
- Self-correction decomposes generation into reusable intermediate outputs, unlike rationale methods whose intermediate steps precede a final response.
5 CONCLUSION
The paper introduces self-correctors that separate initial generation from correction and reports applicability across generator sizes, quality objectives, and feedback settings.
- Self-correctors decompose generation into initial generation and correction steps.
- A fixed base generator paired with a trained corrector improves and controls outputs from both small and large generators.
- The learning framework supports natural language feedback and motivates future work on alternative self-correctors.
A.1 MATHEMATICAL PROGRAM SYNTHESIS
The appendix describes GPT-Neo initialization and residual-pair training, with an additional feedback-based training format using GPT-3-generated feedback.
- GPT-Neo 1.3B is fine-tuned as the initial generator, using default Huggingface hyperparameters except for evaluation steps.
- The corrector is initialized from the fine-tuned generator and trained on sequences containing a problem, current generation, and residual pair.
- A feedback variant uses six demonstrations and GPT-3 text-davinci-002 to insert natural language feedback before the correction target.
A.2 LEXICALLY-CONSTRAINED GENERATION
The appendix reports dataset and metric references for CommonGen and E2E, alongside a human fluency evaluation protocol for E2E generations.
- Table 8 and Table 9 provide hyperparameters for CommonGen and E2E.
- Human fluency evaluation samples 100 E2E instances and uses three Amazon Mechanical Turk annotators per instance.
- Annotators rate fluency on a 3-point Likert scale, aggregated by majority vote, with US or Canadian participants meeting a 98% approval threshold.
B ADDITIONAL RESULTS
The additional results report evaluations of toxicity reduction and lexically constrained generation on COMMONGEN and E2E, alongside E2E hyperparameters and metrics including BLEU-4, CIDER, coverage, and runtime.
- The E2E experiments use a separately reported set of hyperparameters.
- Additional experiments evaluate toxicity reduction using natural language feedback.
- The reported evaluation metrics include BLEU-4, CIDER, coverage, and runtime.
- Lexically constrained generation is evaluated on both COMMONGEN and E2E.
C QUALITATIVE EXAMPLES
Qualitative examples show the corrector repairing mathematical programs, adding missing lexical constraints, and addressing toxic language in generated sequences.
- Mathematical program synthesis: The corrector fixes GSM programs by correcting weekday and weekend counts or removing an incorrect final addition.
- Mathematical program synthesis: The corrector repairs GSM programs by appending a missing discount and performing multipart variable and expression corrections.
- Lexically constrained generation: For constraints ‘table’, ‘paper’, and ‘read’, the corrector changes a reading sentence to include all three required words.
- Lexically constrained generation: The constrained-generation figures distinguish COMMONGEN from E2E and mark previously missing constraints in green.
- Toxicity control: The toxicity-control figure marks toxic language in red.