PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Sequences by Learning to Self-Correct

Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, Yejin Choi

arXiv:2211.00053v1cs.CL

TL;DR

Single-pass generation offers no mechanism to revise imperfect outputs that may contain useful structure. The paper introduces Self-Correction, separating a base generator from a trained corrector that iteratively improves its outputs. It improves base generators across mathematical program synthesis, lexically constrained generation, and toxicity reduction.

  • Problem

    Single-pass generation offers no mechanism to iteratively revise imperfect outputs, even when those outputs contain useful structure.

    Source: §1 · p. 1

  • Method

    Self-Correction separates an imperfect base generator from a trained corrector that iteratively improves its outputs.

    Source: §1 · p. 1

  • Results

    Self-Correction improves base generators across mathematical program synthesis, lexically constrained generation, and toxicity reduction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The trained corrector can transfer to a larger generator with similar performance to training a new corrector.

    Source: §1 · p. 2

    The scalar-quality training setup assumes access to a value function at training time.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs. Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation. We present Self-Correction, an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations. To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations. We show that Self-Correction improves upon the base generator in three diverse generation tasks - mathematical program synthesis, lexically-constrained generation, and toxicity control - even when the corrector is much smaller than the base generator.

1 INTRODUCTION

Self-Correction replaces single-pass generation with iterative refinement by a separately trained corrector. The introduction reports gains across three tasks, transfer to a larger generator, and promising results from natural-language feedback.

1 INTRODUCTION

  • Single-pass generation discards useful work when an output is slightly suboptimal, motivating refinement from the generated sequence rather than starting from scratch.An initial output may satisfy most requirements or contain useful structure despite omissions or errors.Source: §1 · p. 1
  • The system pairs a base generator’s initial hypothesis with a separately trained corrector that can iteratively improve it over multiple passes.Neither module must solve the full task in one pass; corrector training pairs generator outputs with carefully selected targets.Source: §1 · p. 1, Figure 1 · p. 2
  • Self-Correction improves the base generator on mathematical program synthesis, lexically constrained generation, and toxicity reduction.A trained corrector also achieves similar performance when applied to a larger generator as training a new corrector; adding a third module for natural-language feedback shows promising results.Source: §1 · p. 2

2 SELF-CORRECTING SEQUENCE GENERATORS

A self-corrector separates an initial generator from a corrector that can iteratively refine its outputs, optionally using feedback. The section also introduces self-corrective learning, which trains the corrector from value-improving generations.

2 SELF-CORRECTING SEQUENCE GENERATORS

  • A self-corrector uses a base generator to produce an initial hypothesis and a separate corrector to refine it, allowing the modules to be parameterized independently.The corrector can be trained for task-specific objectives while the generator remains a general-purpose language model.Source: §2 · p. 2
  • The corrector can condition later generations on explicit feedback about intermediate outputs, including natural-language comments or compiler traces.Unlike a single-pass generator, it can learn to exploit such feedback to improve performance.Source: §2 · p. 2

2.1 LEARNING A CORRECTOR

  • Self-corrective learning seeds a datapool with base-generator outputs and forms training pairs in which one hypothesis has a higher scalar value than another.The value function is assumed to be available during training; the corrector is trained to improve quality while staying close to the original hypothesis.Source: §2.1 · p. 3
  • Training samples value-improving pairs according to their value gain and similarity, updates the corrector with cross-entropy, and adds current-corrector generations to the datapool.Hypotheses for exploration can come from the datapool or another source; the experiments use datapool hypotheses.Source: §2.1 · p. 4
  • At inference, the system generates an initial output and repeatedly applies the trained corrector, stopping after a fixed number of steps or when a target value is reached.The final output is the last generation in the trajectory.Source: §2.1 · p. 4

3 EXPERIMENTS

Across tasks ranging from exact program correctness to partial lexical coverage and open-ended toxicity, the experiments assess self-correction, smaller correctors paired with larger generators, and explicit feedback.

3.1 MATHEMATICAL PROGRAM SYNTHESIS

  • 98-99% MultiArith accuracy after correction was a 38-point gain over the generator, while GSM accuracy rose from 8.57% to 21% (24% with incorrect-only correction), exceeding cited larger-model baselines.The learned corrector also made value changes, repaired operation order and unit conversions, and performed larger revisions.Source: Table 1 · p. 5, §3.1 · p. 6

3.2 LEXICALLY CONSTRAINED GENERATION

  • The corrector substantially improved constraint coverage on COMMONGEN and E2E while maintaining or improving language quality; on COMMONGEN it beat NeuroLogic-A* with NeuroLogic decoding, and on E2E it beat NeuroLogic-A* with standard beam search.The COMMONGEN approach was an order of magnitude faster than NeuroLogic-A*.Source: Table 2 · p. 6, §3.2 · p. 6

3.3 TOXICITY REDUCTION

  • SELF-CORRECT substantially reduced toxic generations while maintaining fluency and diversity, outperforming inference-time baselines and reinforcement-learning methods without modifying the underlying generator.The compared methods included PPLM, GeDi, DExpert, PPO, and Quark.Source: Table 3 · p. 7, §3.3 · p. 7

3.4 CHANGING MODULES – CORRECTING GPT-3

  • With GPT-3 Instruct, a corrector raised Multitask from 84.90 to 90.90 (92.75 when trained with GPT-3 Instruct) and GSM from 36.80 to 45.00 (45.92 when trained with GPT-3 Instruct).The experiments also found substantially reduced toxicity when a corrector trained in prior experiments was paired with larger generators; the study considered swapping in a large generator at test time or training with it.Source: §3.4 · pp. 7–8, Table 4 · p. 8

3.5 LEVERAGING EXPLICIT FEEDBACK

  • Explicit natural-language feedback improved performance across toxicity, lexical constraints, and math; fine-grained toxicity attributes improved on scalar-score feedback.Feedback used toxicity attributes, elaborated missing lexical constraints, or language-model-generated math guidance; math feedback used a gold solution, creating a risk of solution leakage at inference.Source: §3.5 · pp. 8–9, Table 5 · p. 9

3.6 ADDITIONAL ABLATIONS AND ANALYSIS

  • More correction iterations improved toxicity reduction and Multitask math performance, with feedback making multiple math corrections more beneficial; proportional sampling, value-based pairing, and exploration also improved performance.After 2–3 math corrections, the model needed additional guidance. Ablations found benefits from sampling pairs proportionally, pairing sequences that improve value, and training on correction pairs beyond those induced by the base generator.Source: §3.6 · p. 9

4 RELATED WORK

Self-correction differs from prior editing and denoising approaches by training an expressive text-to-text corrector online to improve a quality measure without supervised edits or critiques. It also complements RL-tuned generators and produces reusable intermediate generations in multiple steps.

4 RELATED WORK

  • Unlike prior text-editing methods, self-correction trains an expressive text-to-text corrector online to improve a quality measure without supervised edits or critiques.Prior work models supervised Wikipedia edits (Faltings et al., 2021; Schick et al., 2022), perturbs sequences with simple operations (Miao et al., 2019; Liu et al., 2020), trains on human-written critiques, or iteratively updates continuous variables (Lee et al., 2020; Qin et al., 2022).Source: §4 · pp. 9–10
  • Unlike ground-truth denoising pretraining, self-correction denoises generations to improve a scalar quality measure, and complements RL-tuned generators by learning the difference between generations and solutions.RL improves a base generator using scalar feedback, but is infeasible for many models accessed through an API; RL-tuned generators can also be used within a self-corrector.Source: §4 · p. 10
  • Self-correction resembles rationale-generation methods in producing intermediate steps, but each step has the output’s form, allowing previous generations to be reused.Rationale-generation methods are cited by Wei et al. (2022) and Dohan et al. (2022).Source: §4 · p. 10

5 CONCLUSION

The paper introduces self-correctors that separate initial generation from correction, using a trained corrector with a fixed base generator. The authors find the approach effective for improving and controlling outputs, including through natural-language feedback.

5 CONCLUSION

  • Self-correctors decompose generation into initial generation and correction, with a corrector trained to improve outputs from a fixed base generator using a scalar quality measure.The paper presents a simple, general procedure for training the corrector.Source: §5 · p. 10
  • Self-correction is applicable and effective for improving performance and controlling outputs from both small and large generators.Source: §5 · p. 10
  • The learning framework offers a promising mechanism for using natural-language feedback to improve generation, while alternative self-correctors remain an avenue for future work.Source: §5 · p. 10

APPENDIX · A ADDITIONAL EXPERIMENTAL DETAILS · A.1 MATHEMATICAL PROGRAM SYNTHESIS

The mathematical program synthesis appendix describes how GPT-Neo 1.3B was fine-tuned as an initial generator and used to initialize a corrector, with a feedback-conditioned training variant also specified.

A.1 MATHEMATICAL PROGRAM SYNTHESIS

  • GPT-Neo 1.3B was fine-tuned with Huggingface default hyperparameters, except evaluation steps were reduced to select a strong checkpoint for each dataset.Source: §A.1 · p. 15
  • The fine-tuned initial generator was used to initialize the corrector.Source: §A.1 · p. 15
  • Without feedback, corrector training sequences combine a problem x with a residual pair yi and yj, using [SC], [CURR], [START], and [END] tokens.Source: §A.1 · p. 15
  • For this setup, the loss is computed on tokens after [START].Source: §A.1 · p. 15
  • The feedback variant uses six demonstrations from training problems and GPT-Neo base generations, with GPT-3 (text-davinci-002) as the feedback model.Source: §A.1 · p. 15
  • Feedback-conditioned sequences add [FEEDBACK]F(x,yi) before [START], and the loss covers tokens after [FEEDBACK].Source: §A.1 · p. 15

A.2 LEXICALLY-CONSTRAINED GENERATION

This subsection reports hyperparameter settings for CommonGen and E2E and describes a human evaluation of E2E fluency.

A.2 LEXICALLY-CONSTRAINED GENERATION

  • Hyperparameters for CommonGen and E2E are reported in Tables 8 and 9.Table 8 is captioned as covering CommonGen.Source: §A.2 · p. 15, Table 8 · p. 15
  • E2E fluency evaluation used 100 sampled instances, three AMT annotators per instance, and majority-vote aggregation of 3-point Likert ratings.Annotators were US- or Canada-based and had a 98% approval rate across at least 5,000 prior annotations.Source: §A.2 · p. 15

B ADDITIONAL RESULTS

The section lists toxicity-reduction results with natural-language feedback and lexical-constraint evaluation results on CommonGen and E2E.

B ADDITIONAL RESULTS

  • Table 9 lists hyperparameters for E2E.The supplied passage gives no hyperparameter values.Source: Table 9 · p. 15
  • Table 10 presents toxicity-reduction evaluation results using natural-language feedback.The supplied passage gives the table's scope but no result values.Source: Table 10 · p. 15
  • The section lists Bleu-4, CIDER, Coverage, and Runtime as evaluation metrics.The supplied passage does not identify which table or task these metric labels belong to.Source: B ADDITIONAL RESULTS · p. 16
  • Table 11 presents lexical-constraint evaluation results on COMMONGEN.The supplied passage gives the table's scope but no result values.Source: Table 11 · p. 16
  • Table 12 presents lexical-constraint evaluation results on E2E.The supplied passage gives the table's scope but no result values.Source: Table 12 · p. 16

C QUALITATIVE EXAMPLES

The qualitative examples show correctors revising imperfect generations in grade-school math, constrained generation, and toxicity control.

C QUALITATIVE EXAMPLES

  • In GSM examples, the corrector fixes weekday/weekend counts, removes an incorrect addition, adds an omitted discount, and makes multipart variable corrections.Figure 7 shows fixes to weekday/weekend counts and an incorrect final addition; Figure 8 shows an added discount and coordinated changes to incorrect assignments and variable names.Source: Figure 7 · p. 17, Figure 8 · p. 17
  • In COMMONGEN and E2E examples, the corrector adds previously missing constraint words.Green text indicates the constraints that were missing.Source: Figure 9 · p. 18
  • The toxicity-control examples mark toxic language in red.The figure caption identifies red words as toxic language.Source: Figure 10 · p. 18

Related papers

Loading 2211.00053v1…