When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback

1 Introduction

• Static benchmarks typically use well-defined questions and one-shot completions, while prior interactive evaluations assume a single natural-language feedback form.The paper motivates broader testing because real programming interactions include varied feedback, from correctness judgments to suggested code changes.Source: §1 · pp. 1–2
• The pipeline transforms static benchmarks into interactive tasks by obfuscating inputs, simulating a user, and testing four feedback types.The four types are CODE FEEDBACK, QUERY REPHRASING, PARAGRAPH, and SENTENCE.Source: §1 · p. 2
• Across 10 models and 3 benchmarks, relative performance often changes between static and interactive settings.The evaluation examines multiple feedback types, whose effects on model behavior also differ.Source: §1 · p. 2
• Higher-quality CODE FEEDBACK more often worsens stronger models’ outputs than lower-quality feedback, while PARAGRAPH yields more surface-level edits than CODE FEEDBACK.These behavior differences can occur even when feedback types are similarly effective for performance.Source: §1 · p. 2
• The authors release a pipeline designed to make it easy to convert additional static benchmarks into interactive evaluations.They present this as an approach for investigating downstream effects of interactive programming settings on model behavior.Source: §1 · pp. 2–3

2 Methodology

• Each question runs through up to five refinement steps, with the code model receiving the obfuscated question, its previous code, and the latest user feedback.The loop ends early if the model reaches a correct solution.Source: §2 · p. 3

2.1 Dataset Transformation

• Sonnet-3.5 compresses full questions into underspecified one-sentence prompts; sampled datasets include APPS, ClassEval, and LiveCodeBench.The datasets were selected for challenging, lengthy problem descriptions and available ground-truth solutions.Source: §2.1 · p. 3

2.2 Feedback Types

• Four feedback modes comprise sentence or paragraph critiques, similar-length query rephrasing, and code-line corrections with suggested alternatives.Query rephrasing is length-constrained to prevent copying the full question.Source: §2.2 · pp. 3–4

2.3 Code Models

• Ten open and closed models span a range of capabilities and sizes; prompts include only the latest solution and user feedback, not the full interaction history.The selected model set includes six open-source and four closed models.Source: §2.3 · p. 4

2.4 User Models

• The simulated user receives the full question and ground-truth solution to support higher-quality feedback, but is instructed not to reveal the exact solution.Sonnet-3.5 is the primary user model, and the authors report that GPT-4o-mini can also improve performance over the self-critique baseline.Source: §2.4 · p. 4

3 Static vs. Interactive Performance

• The evaluation compares static performance with interactive feedback and a five-iteration self-critique baseline, which receives no added problem information or user input.Static evaluation uses the original fully specified question and treats the first attempt as final.Source: §3 · p. 4

3.1 Performance Metrics

• Performance is measured with test case accuracy, while ranking changes are measured using normalized Spearman’s Footrule, where 0 indicates perfectly correlated rankings and 1 perfectly anti-correlated rankings.In ClassEval, test case accuracy combines function and class tests.Source: §3.1 · pp. 4–5

3.2 Results

• Feedback often recovers static performance and can exceed it on APPS and ClassEval, while SELF-CRITIQUE BASELINE usually underperforms static evaluation.On LiveCodeBench, feedback can recover performance comparable with the static setting.Source: Table 1 · p. 5, §3.2 · p. 5
• CODE FEEDBACK and PARAGRAPH are consistently the most effective feedback types, while SENTENCE and QUERY REPHRASING are weaker, especially on LiveCodeBench.The authors suggest longer feedback may contain more helpful information; query rephrasing may help on popular datasets when models recognize the question.Source: §3.2 · p. 5
• Relative model rankings generally change between static and interactive settings, with some LiveCodeBench models shifting four ranks.Across datasets, CODE FEEDBACK ranking distances range from 0.222 to 0.346; top models tend to remain high while weaker models vary more.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
• Models can improve after directionally incorrect feedback, and stronger models sometimes worsen after directionally correct CODE FEEDBACK.The figure distinguishes post-feedback gains from losses across feedback types and directional correctness.Source: Figure 4 · p. 6

4.1 Quality Metrics

• Directional correctness labels whether feedback accurately says a solution is correct or incorrect; the authors automate this classification with GPT-4o.A solution is considered correct when TCA = 1 and incorrect when TCA < 1; 2QUERY REPHRASING is excluded because it does not directly assess the solution.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Results

• PARAGRAPH and CODE FEEDBACK lead performance, although PARAGRAPH tends to have the highest directional correctness and CODE FEEDBACK the lowest.Directional correctness is often above 0.8 and varies little across models and feedback types.Source: §4.2 · p. 6
• PARAGRAPH feedback produces the most behavioral and surface-level changes, while CODE FEEDBACK induces more behavioral changes with fewer aesthetic changes.Figure 5 averages steerability across models on APPS and LiveCodeBench.Source: Figure 5 · p. 7
• Directionally incorrect feedback still often improves performance; Sonnet-3.5 has equal improvement rates after correct and incorrect PARAGRAPH and SENTENCE feedback.Directionally correct feedback has a higher improvement rate overall, but incorrect feedback also produces substantial improvement.Source: §4.2 · p. 7
• For stronger models, directionally correct CODE FEEDBACK is more likely to worsen solutions than directionally incorrect CODE FEEDBACK, unlike PARAGRAPH and SENTENCE feedback.PARAGRAPH and SENTENCE also reduce the proportion of worse post-feedback solutions.Source: §4.2 · p. 7

5 Model Steerability

• Performance drops in interactive settings are attributed to ineffective incorporation of feedback rather than models outright ignoring it.Source: §5 · p. 7

5.1 Steerability Metrics

• Surface-level steerability measures Levenshtein edit distance between code versions; behavioral steerability counts test cases that flip between incorrect and correct, or vice versa.Source: §5.1 · p. 7

5.2 Results

• PARAGRAPH feedback has the highest surface-level steerability, with an average edit distance of 445.6 characters, and shares the highest behavioral steerability with CODE FEEDBACK, changing 21.8% and 19.3% of test cases on average.PARAGRAPH is also one of the highest-performing feedback styles.Source: §5.2 · p. 7
• Weaker models often make many surface-level edits with little behavioral effect, whereas stronger models can make relatively small edits that substantially change code behavior.Figure 6 plots surface-level steerability against behavioral steerability during the first refinement step.Source: §5.2 · p. 7, Figure 6 · p. 8
• ClassEval is excluded because its evaluation utilities do not report exactly which test cases pass or fail.Source: §5.2 · p. 7

Related work

• Static benchmarks such as HumanEval and MBPP are widely used to assess coding capabilities, while live benchmarks aim to reduce contamination risks.The pipeline can convert many static benchmarks into interactive evaluations; the authors demonstrate it on three datasets.Source: §6 · p. 8
• Human studies examine how programmers use chat assistants, usually with no more than three models, while large-scale preference platforms leave fine-grained feedback effects challenging to study.Source: §6 · p. 8
• Prior simulated-user benchmarks cover coding and other tasks; this benchmark adds diverse feedback forms and input obfuscation to require collaboration on underspecified problems.The design draws on existing human studies for common feedback types and underspecified inputs.Source: §6 · p. 8

7 Conclusion

• The pipeline requires code models to collaborate with simulated users on underspecified coding problems using different feedback types.Source: §7 · p. 8
• Relative model performance changes radically between static and interactive settings, and the paper analyzes feedback quality and steerability to examine feedback’s effects on model behavior and effectiveness.Source: §7 · pp. 8–9

8 Limitations

• The experiments cover three code benchmarks, so results may not extend to other datasets, including non-Python coding questions.Because LLM responses are not explicitly compared with human-generated feedback on the modified questions, the authors report behavioral trends rather than specific degrees of change.Source: §8 · p. 9
• The feedback types are not comprehensive, and simulated user feedback is not claimed to represent actual human users.Feedback type stays fixed across refinement rounds, and the user initiates feedback rather than the model asking clarification questions.Source: §8 · p. 9

8.1 Potential Risks

• The pipeline is a testbed for examining model responses to feedback, not an approximation of actual human use.The authors caution against using it as a proxy for how people use models.Source: §8.1 · p. 9
• The included programming questions are in Python, and the feedback is in English.Source: §8.1 · p. 9

A.1 Additional Details on Datasets

• ClassEval problems are underspecified by summarizing method docstrings, while LiveCodeBench ground truths are generated by sampling Sonnet-3.5 twice.LiveCodeBench questions without a correct generated solution are excluded because the pipeline requires a ground-truth solution in the user prompt.Source: §A.1 · p. 12
• For APPS and LiveCodeBench, the reported averages in Tables 1–2 and Figure 3 combine difficulty levels.Source: §A.1 · p. 12

• The examples show Sonnet-3.5 feedback responding to a proposed solution.Source: §A.2 · p. 12
• All feedback types respond to the same question, identified in Table 3.Source: §A.2 · p. 12

A.3 Additional Details on Models

• Gemma-7B-it, Llama-3.1-8B-Instruct, and Qwen2.5-Coder-7B-Instruct run on one L40S GPU with temperature 0.9, sampling enabled, and a 4096-token limit.Source: §A.3 · p. 12

A.4 Prompts

• Paragraph and sentence feedback are length-constrained; code feedback identifies incorrect lines, while query rephrasing adds missing problem details.The appendix also provides prompts for generating question summaries and ClassEval docstring summaries.Source: §A.4 · p. 13

A.5 Performance Metrics

• Normalized Spearman’s Footrule distance treats ranking difference as a normalized distance, with zero for identical rankings and one for maximum distance.For uncorrelated rankings of length 10, the normalized distance is approximately 0.73.Source: §A.5 · p. 14

A.6 Additional Details on Measuring Feedback Quality

• Directional feedback correctness is classified using GPT-4o and rule-based corrections for feedback that acknowledges missing edge cases or formatting.Attempts to measure feedback quality through probability changes or a 1–5 relevance rating were considered too noisy or hard to define.Source: §A.6 · p. 14

A.7 Performance Tables for Static vs. Interactive Settings

• Tables 4–7 report average test case accuracy for APPS Interview, APPS Introductory, LiveCodeBench, and ClassEval.The supplied table descriptions identify these measures and datasets, but do not provide their cell values.Source: §A.7 · p. 15, Table 4 · p. 16, Table 5 · p. 16

A.8 Additional Tables

• Supplementary tables cover directional correctness and steps to reach 100% TCA, edit-based and behavioral steerability, and ranking distance from the static setting.The appendix also supplies prompts used for the feedback and code-model settings.Source: §A.8 · p. 15, Table 12 · p. 20, Table 13 · p. 21