PaperlayerView exact revision on arXiv

Source-linked AI summary

LeTI: Learning to Generate from Textual Interactions

Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

arXiv:2305.10314v2cs.CLcs.AIcs.SE

TL;DR

Existing fine-tuning often relies on input-output pairs or numerical rewards, while textual interactions can also identify and explain errors. LETI iteratively fine-tunes on generated code and interpreter feedback, improving generation and sample efficiency over binary feedback and applying to code-formulated NLP tasks.

  • Problem

    Fine-tuning commonly uses input-output pairs or numerical rewards, whereas textual interactions can assess correctness while also pinpointing and explaining errors.

    Source: §1 · p. 1

  • Method

    LETI iteratively fine-tunes on language instructions, model-generated programs, and evaluator feedback, prefixed with a binary reward token, without ground-truth solutions.

    Source: §2.3 · p. 3

  • Results

    Textual feedback improves generation quality and sample efficiency over binary feedback, and LETI also improves code generation on MBPP and applies to event argument extraction formulated as code generation.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    Interpreter feedback can support iterative language-model improvement without ground-truth code, and the approach also applies to natural-language tasks formulated as code generation.

    Source: §5 · p. 9

    The study uses only Python-interpreter feedback, which may not be useful for every error type; AssertError feedback can amount to binary feedback, and human language feedback was not investigated.

    Source: Limitations and Future Work · p. 9

Abstract

from arXiv · show

Fine-tuning pre-trained language models (LMs) is essential for enhancing their capabilities. Existing techniques commonly fine-tune on input-output pairs (e.g., instruction tuning) or with numerical rewards that gauge the output quality (e.g., RLHF). We explore LMs' potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback. Our focus is the code generation task, where the model produces code based on natural language instructions. This setting invites a natural and scalable way to acquire textual feedback: the error messages and stack traces from code execution using a Python interpreter. LETI iteratively fine-tunes the model, using the LM objective, on a concatenation of natural language instructions, LM-generated programs, and textual feedback. Prepended to this fine-tuning text, a binary reward token is used to differentiate correct and buggy solutions. LETI requires no ground-truth outputs for training and even outperforms a fine-tuned baseline that does. LETI not only improves the performance of LMs on a code generation dataset MBPP, but also generalizes to other datasets. Trained on MBPP, it achieves comparable or better performance than the base LMs on unseen problems in HumanEval. Furthermore, compared to binary feedback, we observe that textual feedback leads to improved generation quality and sample efficiency, achieving the same performance with fewer than half of the gradient steps. LETI is equally applicable in natural language tasks when they can be formulated as code generation, which we empirically verified on event argument extraction.

1 Introduction

LETI explores whether language models can learn from textual interactions that both assess correctness and explain errors, using interpreter feedback to iteratively fine-tune generated code. It improves code-generation performance without ground-truth code, while textual feedback is more sample-efficient than binary feedback and the approach also applies to code-formulated NLP tasks.

1 Introduction

  • Existing fine-tuning signals commonly assess correctness or output quality, while textual interactions can also pinpoint errors and explain why they occurred.The paper hypothesizes that language models can harness these richer signals from an environment such as a Python interpreter.Source: §1 · p. 1
  • LETI learns from interpreter error messages and stack traces by fine-tuning on instructions, generated programs, and textual feedback, without instruction-code paired data.A binary reward token distinguishes correct from buggy solutions.Source: §1 · p. 2
  • 63.2% more syntactically correct and executable code was generated by LETI than by the pretrained 2B model, without ground-truth code.With postprocessing, LETI reduced NameError issues from 10% to 1% in two iterations and also improved HumanEval performance.Source: §1 · p. 2
  • Textual feedback improves generation quality and sample efficiency over binary feedback, reaching the same 2B-model performance with about half the gradient steps.LETI was also empirically applied to event argument extraction when formulated as code generation.Source: §1 · p. 3

2 LETI: Learning from Textual Interactions

LETI repeatedly samples programs, evaluates them for correctness and error feedback, and fine-tunes the language model on the resulting feedback-conditioned sequences. The approach uses no ground-truth solutions and interleaves this training with continued pre-training to help preserve the model’s original capabilities.

2 LETI: Learning from Textual Interactions

  • LETI iterates by generating candidate programs, evaluating them, and using the feedback to fine-tune the generator.The method assumes no ground-truth solutions; its language model is pretrained on natural language and code and samples candidate solutions for each problem.Source: §2 · p. 3, §2.1 · p. 3

2.2 Solution Evaluator

  • The solution evaluator returns a binary correctness label and, for errors, textual feedback comprising interpreter stack traces and error messages.The evaluator is a Python interpreter in the code-generation experiments; the paper notes that an evaluator can be implemented differently for other problems.Source: §2.2 · p. 3

2.3 Feedback-conditioned Fine-tuning

  • Feedback-conditioned fine-tuning trains on each generated solution conditioned on its binary and textual evaluator feedback.The feedback sequence begins with a reward token, with textual feedback included when available.Source: §2.3 · p. 3

2.4 Regularization with Continued Pre-training

  • LETI interleaves feedback-conditioned optimization with the language-model objective on pre-training data to alleviate distribution shifts and preserve capabilities on untrained tasks.The paper reports that continued-pretraining regularization is essential to maintain the language model’s original capability on tasks it was not trained on.Source: §2.4 · p. 4

3 Experimental Results

Across MBPP, HumanEval, reasoning benchmarks, and event argument extraction, LETI improves several code-generation outcomes without ground-truth solutions, while results depend on model size, feedback, and evaluator design. Textual feedback improves sample efficiency over binary feedback, and evaluator choices can shape which task-specific metrics improve.

3.1 Experiment Setup

  • The MBPP setup uses 974 short Python problems and test suites, with CodeGen-mono models of 350M and 2B parameters.The study tests both with and without stop-word-based post-processing, defaulting to no post-processing unless noted.Source: §3.2.1 · p. 4, §3.1 · p. 4

3.2.1 Mostly Basic Python Problems (MBPP)

  • Without ground-truth solutions, LETI improves MBPP test Pass@1 across iterations and outperforms supervised fine-tuning for the 2B model.With post-processing, the 2B model improves from 26.89% to 29.53% within two iterations.Source: Figure 2 · p. 5, §3.2.1 · p. 5
  • On MBPP, LETI reduces SyntaxError counts from 5179 to 652, while AssertionError remains difficult to fix from interpreter feedback alone.With post-processing, NameError counts fall from 810 to 94 within two iterations.Source: §3.2.1 · p. 5

3.2.2 HumanEval

  • On HumanEval, LETI improves Pass@10 and Pass@100 across both model sizes, while 2B Pass@1 declines versus the pretrained model.The evaluation uses post-processing for generated solutions.Source: §3.2.2 · pp. 5–6, Table 2 · p. 6

3.3 Learning from Textual Feedback is More Sample-efficient

  • Textual feedback yields better final performance than binary feedback, with the 2B model improving 2.24x faster per iteration.The reported improvement-rate advantage is 1.57x for the 350M model.Source: §3.3 · p. 6, Table 3 · p. 6

3.4 LETI Retains Reasoning and Chain-of-Thought Performance

  • LETI causes no significant degradation on GSM8K or BBH overall; for 2B, BBH chain-of-thought performance and its gain over direct prompting improve.The 350M model shows some BBH chain-of-thought degradation despite continued-pretraining regularization.Source: §3.4 · p. 7, Table 5 · p. 7
  • Removing continued-pretraining regularization significantly degrades 350M performance on PaL-prompted GSM8K and BBH chain-of-thought, but not MBPP performance.The reported MBPP comparison shows no significant difference with versus without regularization.Source: §3.4 · p. 7

3.5 LETI is applicable to NLP tasks like Event Argument Extraction (EAE)

  • For EAE, three LETI iterations increase Arg-I F1 by 12.3% (21.2% →33.5%) and Arg-C F1 by 2.6% (8% →10.6%).The rule-based evaluator’s design can influence which errors LETI prioritizes; its Pass@1 correlation is stronger for precision than for F1.Source: §3.5 · p. 8, Figure 4 · p. 8

4 Related Work

Prior code-generation work uses interpreter feedback for filtering, ranking, fine-tuning, or reinforcement learning, while other approaches use human feedback or condition on coarse labels. LETI directly trains the underlying LM from textual feedback.

Using feedback to improve code generation

Improving LMs with reinforcement learning

5 Conclusion

LETI fine-tunes language models to learn from textual interactions, using automatic Python-interpreter feedback to improve code generation. Textual feedback outperforms binary-feedback baselines in generation quality and sample efficiency, and LETI also applies to NLP tasks formulated as code generation.

Conclusion

  • LETI uses automatic textual feedback from a Python interpreter to improve language models on code-generation tasks.The paper presents LETI as a fine-tuning paradigm for learning from textual interactions.Source: §5 · p. 9
  • Textual feedback outperforms binary-feedback baselines in both generation quality and sample efficiency.The conclusion reports this comparison without specifying a numerical result.Source: §5 · p. 9
  • LETI also applies to NLP tasks formulated as code generation, as empirically verified on Event Argument Extraction.The conclusion names Event Argument Extraction as the demonstrated task.Source: §5 · p. 9

Limitations and Future Work

The study leaves real-world human feedback unexplored and notes that interpreter feedback can be uninformative for functional errors. In the ablation, LETI consistently benefits from sampling more solutions per problem.

Limitations and Future Work

  • Interpreter feedback can help with SyntaxError and NameError, but AssertError traces may add no information beyond binary feedback.The study did not investigate real-world human-language feedback, which may offer greater linguistic diversity and helpfulness.Source: Limitations and Future Work · p. 9
  • The study leaves combining interpreter feedback with more helpful feedback, using stronger or larger backbone LMs, and extending to multi-turn settings for future work.LLM-simulated feedback is named as one possible source of more helpful feedback.Source: Limitations and Future Work · p. 9

A.1 Does the number of solutions generated per problem matter?

  • In Fig. A.7, LETI consistently benefits from larger n for each problem, indicating that more exploration helps; the paper uses n = 128 for all other experiments.Source: §A.1 · p. 12

A.2 Does the number of training problems |P| matters?

The number of MBPP training problems affects test-set performance: larger training sets generally yield faster and greater improvements. The 350M model shows net positive test-set improvement only once the training set reaches at least 128 problems.

A.2 Does the number of training problems |P| matters?

  • Larger training sets generally produce faster and more significant improvements on test sets.The comparison includes a complete MBPP set of |P| = 374 and subsets of |P| = 16, 64, and 128.Source: §A.2 · p. 12
  • For the 350M model, net positive test-set improvement appears only when the number of training problems reaches |P| ≥128.This threshold is reported for the 350M model; the passage reports generally smaller improvement rates for the 2B model with smaller |P|.Source: §A.2 · p. 12

A.3 How do reward tokens impact performance?

Reward-token effects differ across tasks: both tokens outperform no token on MBPP, but not on HumanEval, GSM8K, or Big-Bench-Hard overall. PaL on GSM8K is an exception, with a pattern resembling MBPP.

A.3 How do reward tokens impact performance?

  • On MBPP, both <|good|> and <|bad|> outperform no reward token, as the fine-tuning objective separates correct and incorrect solutions into token-conditioned sequences.The paper proposes that this partitioning may shift problem-related solutions away from the unconditioned distribution.Source: §A.3 · pp. 12–13
  • With 128 or 374 training problems, LETI with textual feedback achieves higher performance and improvement rates than a baseline without textual feedback.The figure caption characterizes this as more efficient use of samples.Source: Figure A.5 · p. 13
  • On HumanEval, neither reward token improves performance; the gap between no token and either token is large, while the difference between tokens is minimal.The authors suggest HumanEval solutions differ from in-domain MBPP solutions, with good performance coming from the original unconditioned distribution.Source: §A.3 · p. 13
  • On GSM8K and Big-Bench-Hard, performance is generally similar across reward-token conditions, except that PaL on GSM8K favors <|good|> over <|bad|>.PaL solutions to some GSM8K problems resemble MBPP solutions; the authors suggest token-conditioned performance differences may indicate similarity between problems.Source: §A.3 · p. 13

A.4 Does the performance gain come from more pre-training steps?

The authors test whether LETI’s gains come from continued pre-training regularization rather than feedback-conditioned fine-tuning. In their 350M-model experiment, pre-training data alone does not improve MBPP test performance.

A.4 Does the performance gain come from more pre-training steps?

  • LETI without post-processing consistently benefits from generating more candidate solutions per problem.The comparison varies the number of candidates, n.Source: Figure A.7 · p. 14
  • Training only on additional pre-training data does not improve MBPP test performance in the 350M-model experiment, supporting attribution of LETI’s gains to FCFT rather than regularization.LETI training alternates FCFT batches with continued pre-training batches.Source: §A.4 · pp. 13–14

B LETI Training Details

The appendix specifies LETI’s training settings for MBPP and event argument extraction, along with the pass@k estimator and iteration reporting. It also describes how training-problem count relates to improvement speed and magnitude.

B LETI Training Details

  • For each LETI iteration, FCFT runs for three epochs; learning rates are 1e-5 for the 350M model and 5e-6 for the 2B model, with weight decay 0.01 and batch size 128.An iteration takes approximately 4 hours for 350M and 22 hours for 2B; the lower 2B learning rate addresses training instability.Source: B LETI Training Details · p. 14
  • MBPP has 374 training problems and 500 test problems; each iteration samples 128 solutions per training problem, while evaluation samples 16 per test problem.Training and test sampling temperatures are 1.0 and 0.1, respectively.Source: B LETI Training Details · p. 14
  • For event argument extraction on ACE-05, the authors sample 64 solutions per training example and omit continued pre-training regularization for computational efficiency.They state that regularization mainly helps maintain out-of-domain performance, which is not the main focus of this experiment.Source: B LETI Training Details · p. 14

B.1 Metrics Details

  • Pass@k is estimated with the unbiased estimator from Chen et al. (2021b), using n solutions where n > k.Table A.7 lists the iteration numbers for the reported LETI-optimized performance.Source: §B.1 · p. 14, Table A.7 · p. 15
  • With more training problems, LETI produces faster and more significant improvements.The figure compares training-set sizes.Source: Figure A.8 · p. 15

B.2 Evaluation Details

The appendix gives evaluation sampling and generation settings for several datasets, and details the MBPP fine-tuned baseline. It also reports that pre-training data regularization alone does not significantly change MBPP test performance.

B.2 Evaluation Details

  • Evaluation generations are not conditioned on reward tokens such as <|good|> or <|bad|>.This applies to the evaluation datasets described in this section.Source: §B.2 · p. 15
  • For GSM-8K, evaluation uses 40 samples per problem, temperature 0.7, top-p 0.95, and up to 1,536 generated tokens.Source: §B.2 · p. 15
  • For Big-Bench-Hard, evaluation uses one deterministic sample per prompt and allows up to 1,536 tokens for direct prompts or 2,048 for chain-of-thought prompts.The scores of 250 chain-of-thought prompts exceeding 2,048 tokens are set to zero.Source: §B.2 · p. 15
  • For HumanEval, evaluation samples 256 solutions per problem at temperatures 0.2, 0.6, and 0.8, then reports the maximum performance across temperatures.Generation is capped at 768 tokens per problem.Source: §B.2 · p. 15

B.3 Fine-tuned Baseline Details

  • The MBPP fine-tuned baseline trains 350M and 2B CodeGen-Mono models on 374 examples for 30 epochs and reports the best pass@1 checkpoint without post-processing.Checkpoints are evaluated every six epochs; ground-truth solutions end with an <eos> token.Source: §B.3 · p. 15
  • Pre-training data regularization alone yields no significant difference in MBPP test performance, while regularization without FCFT does not improve performance.Source: Figure A.9 · p. 16, Figure A.10 · p. 16
  • For generated code that repeats the prompt and code, the cited evaluation heuristic keeps only the first code block for execution and evaluation.The example involves pre-trained 2B CodeGen-Mono generating up to 512 tokens.Source: Figure A.11 · p. 17

Related papers

Loading 2305.10314v2…