Improving Code Generation by Training with Natural Language Feedback

• Pre-trained LLMs still generate incorrect code because web-scale code contains bugs and training supplies offline demonstrations without trial-and-error guidance.The authors hypothesize that feedback on models’ own outputs provides more targeted supervision.Source: §1 · pp. 1–2
• Natural-language feedback is proposed as expressive, sample-efficient training supervision that can improve models without requiring the same feedback at test time.The authors frame this as a weak form of scalable oversight because overseers evaluate outputs rather than manually write new demonstrations.Source: §1 · p. 2
• ILF trains a refinement model on failed programs and human feedback, then fine-tunes the original model on passing refinements.The procedure can be iterated and is formally connected to minimizing expected KL divergence from a ground-truth distribution.Source: §1 · p. 2
• 38% relative (10% absolute) pass@1 improvement on MBPP over zero-shot performance demonstrates ILF’s main code-generation result.ILF also outperforms fine-tuning on MBPP-provided code by 64% relative (14% absolute).Source: §1 · p. 2
• ILF’s effectiveness depends on feedback quality: irrelevant feedback causes steep correctness drops, while LLM-generated feedback produces lower final pass rates than human-written feedback.These findings support the method’s reliance on accurate, useful feedback.Source: §1 · p. 2

2.1. Preliminaries

• The preliminaries define programs as token sequences, tasks as descriptions paired with unit tests, EVAL as a binary test-pass indicator, and FINETUNE as gradient-based optimization.These components establish the task and verification machinery used by ILF.Source: §2.1 · p. 2

2.2. Imitation Learning From Language Feedback

• ILF approximates a ground-truth program distribution by fitting the base model with supervised cross-entropy, using Monte Carlo samples rather than enumerating all programs.The target distribution cannot be sampled directly, motivating proposal-based sampling.Source: §2.2 · pp. 2–3
• ILF samples failed programs, obtains human feedback, generates feedback-informed refinements, keeps passing refinements, and fine-tunes the base model on them.The algorithm’s explicit pipeline is summarized in Algorithm 1.Source: §2.2 · p. 3, §2.3 · p. 3

2.3. Proposal Distribution q

• The proposal distribution raises probability on higher-quality programs by combining failed-model samples, human feedback, refinement generation, and a filter that retains only test-passing outputs.Unit-test verification is used directly as the reward, though other signals such as stack traces or compiler outputs could also be used.Source: §2.3 · p. 3, §2.2 · p. 3
• ILF assumes a sufficiently capable πRefine model can turn an incorrect program and feedback into a passing refinement, while feedback may be less laborious than manual repair.In the implementation, πRefine is a fine-tuned CodeGen-Mono 6.1B model trained with repaired examples.Source: §2.3 · p. 4

3. Experiments and Results

• The experiments use 195 filtered human-annotated triples and evaluate final performance on MBPPTest with pass@k, where correctness requires passing all task unit tests.Annotations are retained only when refinements pass all tests, feedback is correct, and edits remain below the specified distance threshold.Source: §3 · pp. 4–5

3.1. CODEGEN-MONO 6.1B Incorporates Feedback

• 13.8% pass@10 on relevant feedback falls to 4.0% with irrelevant feedback, a 71% relative decrease showing that zero-shot repairs use feedback content.This evaluation targets tasks where zero-shot CODEGEN-MONO 6.1B previously produced no correct programs.Source: Table 2 · p. 5, §3.1 · p. 6
• Fine-tuning πRefine raises pass@1 from 2% to 19% and pass@10 from 13.8% to 47%, with correct refinements for 61% of evaluation tasks.The resulting 78 correct refinements are used to train the final model πθ∗.Source: §3.1 · p. 6, Table 3 · p. 6

3.2. ILF Yields Pass Rates Higher Than Fine-Tuning on Gold Data or Human-Written Programs Alone

• 10% absolute (38% relative) pass@1 improvement over zero-shot makes ILF the strongest final method, exceeding MBPP-gold and human-refinement fine-tuning baselines.ILF also exceeds those baselines by 14% absolute (64% relative) and 3% absolute (9% relative), respectively, on pass@1.Source: §3.2 · p. 7, Table 4 · p. 7
• MBPP gold programs contain more high-perplexity examples than refinement datasets, suggesting refinements are easier for CODEGEN-MONO 6.1B to learn because they remain closer to its original distribution.High perplexity is defined here as perplexity ≥102.Source: §3.2 · p. 7, Figure 3 · p. 7

3.3. Scaling Up Model Feedback Does Not Offer the Same Benefits As Human Feedback

• 122 human-feedback pieces outperform larger quantities of InstructGPT feedback, whose increasing dataset sizes yield only modest pass-rate improvements.The 122 pieces comprise 44 feedback samples for training πRefine and 78 for generating refinements used to train πθ∗.Source: §3.3 · p. 8, Figure 4 · p. 8

3.4. Human Feedback Is More Informative Than InstructGPT Feedback

• Human feedback addresses more bugs on average and avoids irrelevant feedback, whereas InstructGPT often gives no, incorrect, irrelevant, or task-restating feedback.InstructGPT refinements can still be correct even when their feedback is not.Source: Table 5 · p. 8, Table 6 · p. 8, §3.4 · p. 9

3.5. πRefine Struggles To Incorporate Feedback Addressing Many Bugs

• The average pass rate of πRefine refinements decreases as feedback addresses more bugs, motivating decomposition into multiple repair steps.The authors connect this direction to evidence that CODEGEN models follow instructions more effectively across multiple turns.Source: §3.5 · p. 9, Figure 5 · p. 9

LLMs for Program Synthesis

• Prior program-synthesis research uses general-purpose, code-only, and mixed text-code pretrained language models to generate programs.These models have demonstrated strong code-challenge performance, but the related work spans different pretraining data compositions.Source: §4 · p. 9

Learning from Human Feedback

• Natural-language feedback can improve code-generation performance at inference time, but ILF learns from that feedback during training instead.The paper distinguishes ILF from approaches that prompt models with feedback only at test time.Source: §4 · p. 9
• ILF is related to reinforcement learning from human feedback but uses natural-language feedback rather than preference comparisons between alternatives.Prior RLHF methods train models to satisfy human preferences, whereas ILF follows work using written feedback directly.Source: §4 · p. 9
• Other training-time feedback methods use language feedback within RLHF pipelines, but reported code-generation gains may not exceed those from prompting alone.In that approach, feedback solicits responses that crowdworkers rank for training a preference model.Source: §4 · p. 9
• ILF also extends beyond code generation: related work reports that it outperforms supervised fine-tuning baselines for text summarization and connects to explanation- and RL-based natural-language supervision.The broader literature includes supervision through explanations and natural language embedded in reinforcement-learning systems.Source: §4 · pp. 9–10

5. Conclusion

• ILF is theoretically justified as minimizing expected KL divergence between πθ and a target ground-truth distribution using natural-language feedback.Source: §5 · p. 10
• ILF yields 38% and 64% relative pass@1 increases over zero-shot and MBPP fine-tuning baselines while training on only 78 examples.The approach also generates training data targeting bugs likely to occur in the model’s own outputs.Source: §5 · p. 10
• ILF can use any base model with a sufficiently capable refinement model and can be applied across multiple rounds as new information or bugs emerge.Source: §5 · p. 10
• The approach supplies an online, process-based training signal missing from offline pre-training and may reduce reward hacking while improving intermediate reasoning correctness.Source: §5 · p. 10
• Further work is required to extend ILF, including automating feedback generation to reduce the time and cost of collecting human feedback.Source: §5 · p. 10

A.1. Training Details

• All reported Table 4 pass rates are evaluated on MBPPTest with hyperparameters selected through sweeps on MBPPRefine.Source: §A.1 · p. 15
• The sweeps vary learning rate across 1.0−6, 5.0−6, and 1.0−5, batch size across 32, 64, and 128, and epochs across 1, 2, and 5.Source: §A.1 · p. 15
• ILF reuses the best hyperparameters from the MBPP-program sweep instead of sweeping over ILF-generated programs because MBPPRefine tasks train πRefine.Source: §A.1 · p. 15
• Section 3.3 separately tunes hyperparameters for each dataset size using MBPPTrain tasks, MBPPRefine validation tasks, and InstructGPT refinements that pass unit tests.Source: §A.1 · p. 15
• The experimental pipelines use HuggingFace transformers v4.12.5, HuggingFace datasets v2.7.1, and PyTorch v1.11.Source: §A.1 · p. 15

A.2. Annotator Instructions

• Figure 6 shows an example of the instructions given to Surge annotators.Source: Figure 6 · p. 16

A.3. Examples of Human Versus InstructGPT Feedback

• Human and InstructGPT feedback are presented for incorrect programs generated by CODEGEN-MONO 6.1B.Source: Table 7 · p. 17, §A.3 · pp. 18–20
• Additional feedback corrects complex-number construction by replacing separate arguments with an expression using +, and recommends atan2 for complex-number angles.Source: §A.3 · p. 17
• Feedback identifies an unnecessary loop that overcounts matches and recommends checking corresponding elements across lists.Source: §A.3 · p. 18
• Another critique flags inefficient enumeration of all element-pair combinations and recommends tracking seen pairs with a set.Source: §A.3 · p. 18
• Several examples diagnose input-shape and tuple-unpacking errors, including single-list handling, counting lists rather than tuple items, and dereferencing tuple keys.Source: §A.3 · pp. 17–18
• The magic-square example explains that correctness requires equal row, column, and diagonal sums rather than testing whether sums are squares.Source: §A.3 · p. 19
• The character-position example identifies that the code counts uppercase characters instead of comparing each string character with its alphabetic counterpart.Source: §A.3 · p. 20